[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 433381066 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524592K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002309] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.006567] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009133] LSM: Security Framework initializing [ 0.010070] Yama: becoming mindful. [ 0.011039] SELinux: Initializing. [ 0.012075] *** VALIDATE selinux *** [ 0.020497] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024784] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026122] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028107] *** VALIDATE tmpfs *** [ 0.029479] *** VALIDATE proc *** [ 0.030224] *** VALIDATE cgroup *** [ 0.031009] *** VALIDATE cgroup2 *** [ 0.033106] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034167] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036031] Spectre V2 : User space: Vulnerable [ 0.037010] Speculative Store Bypass: Vulnerable [ 0.039809] debug: unmapping init [mem 0xffffffff93859000-0xffffffff93860fff] [ 0.041819] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042733] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043031] ... version: 2 [ 0.044016] ... bit width: 48 [ 0.045014] ... generic registers: 4 [ 0.046016] ... value mask: 0000ffffffffffff [ 0.047020] ... max period: 00007fffffffffff [ 0.048019] ... fixed-purpose events: 3 [ 0.049016] ... event mask: 000000070000000f [ 0.050329] rcu: Hierarchical SRCU implementation. [ 0.052471] smp: Bringing up secondary CPUs ... [ 0.053667] x86: Booting SMP configuration: [ 0.054028] .... node #0, CPUs: #1 #2 #3 [ 0.058279] smp: Brought up 1 node, 4 CPUs [ 0.060019] smpboot: Max logical packages: 1 [ 0.061018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.250506] node 0 deferred pages initialised in 189ms [ 0.254207] devtmpfs: initialized [ 0.255327] x86/mm: Memory block size: 128MB [ 0.259097] gcov: version magic: 0x41383552 [ 0.262046] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.265083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.268288] pinctrl core: initialized pinctrl subsystem [ 0.270315] [ 0.270915] ************************************************************* [ 0.273023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.275017] ** ** [ 0.277020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.279020] ** ** [ 0.281018] ** This means that this kernel is built to expose internal ** [ 0.282015] ** IOMMU data structures, which may compromise security on ** [ 0.285019] ** your system. ** [ 0.286018] ** ** [ 0.288016] ** If you see this message and you are not debugging the ** [ 0.290022] ** kernel, report this immediately to your vendor! ** [ 0.292022] ** ** [ 0.294022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.298039] ************************************************************* [ 0.299707] NET: Registered protocol family 16 [ 0.302435] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.305065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.308070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.312109] cpuidle: using governor menu [ 0.313617] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.315723] PCI: Using configuration type 1 for base access [ 0.318179] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.327125] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.330057] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.333081] cryptd: max_cpu_qlen set to 1000 [ 0.336281] ACPI: Added _OSI(Module Device) [ 0.338029] ACPI: Added _OSI(Processor Device) [ 0.339028] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.341027] ACPI: Added _OSI(Processor Aggregator Device) [ 0.346180] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.352507] ACPI: Interpreter enabled [ 0.354088] ACPI: PM: (supports S0 S3 S4 S5) [ 0.356022] ACPI: Using IOAPIC for interrupt routing [ 0.358182] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.361423] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.372347] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.374059] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.375018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.378114] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.382494] acpiphp: Slot [2] registered [ 0.383147] acpiphp: Slot [5] registered [ 0.385126] acpiphp: Slot [6] registered [ 0.386151] acpiphp: Slot [3] registered [ 0.387150] acpiphp: Slot [4] registered [ 0.388121] acpiphp: Slot [7] registered [ 0.390124] acpiphp: Slot [8] registered [ 0.391136] acpiphp: Slot [9] registered [ 0.392146] acpiphp: Slot [10] registered [ 0.394117] acpiphp: Slot [11] registered [ 0.395062] acpiphp: Slot [12] registered [ 0.396017] acpiphp: Slot [13] registered [ 0.396979] acpiphp: Slot [14] registered [ 0.398106] acpiphp: Slot [15] registered [ 0.399138] acpiphp: Slot [16] registered [ 0.400114] acpiphp: Slot [17] registered [ 0.402162] acpiphp: Slot [18] registered [ 0.403141] acpiphp: Slot [19] registered [ 0.404164] acpiphp: Slot [20] registered [ 0.406112] acpiphp: Slot [21] registered [ 0.407125] acpiphp: Slot [22] registered [ 0.408103] acpiphp: Slot [23] registered [ 0.409111] acpiphp: Slot [24] registered [ 0.410144] acpiphp: Slot [25] registered [ 0.412146] acpiphp: Slot [26] registered [ 0.413139] acpiphp: Slot [27] registered [ 0.415158] acpiphp: Slot [28] registered [ 0.416177] acpiphp: Slot [29] registered [ 0.418150] acpiphp: Slot [30] registered [ 0.420152] acpiphp: Slot [31] registered [ 0.421082] PCI host bridge to bus 0000:00 [ 0.422024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.424034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.426037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.428034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.430025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.432070] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.433254] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.437100] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.440342] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.446540] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.451052] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.454032] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.456021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.458027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.460748] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.463597] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.464049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.466783] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.469542] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.476018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.482028] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.486691] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.494018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.500022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.517022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.528835] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.540027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.543919] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.551028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.560818] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.564440] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.566278] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.567291] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.569204] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.573171] iommu: Default domain type: Passthrough [ 0.574475] SCSI subsystem initialized [ 0.575000] ACPI: bus type USB registered [ 0.576165] usbcore: registered new interface driver usbfs [ 0.580103] usbcore: registered new interface driver hub [ 0.582120] usbcore: registered new device driver usb [ 0.584186] pps_core: LinuxPPS API ver. 1 registered [ 0.586014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.589075] PTP clock support registered [ 0.592056] EDAC MC: Ver: 3.0.0 [ 0.594170] PCI: Using ACPI for IRQ routing [ 0.595681] NetLabel: Initializing [ 0.596013] NetLabel: domain hash size = 128 [ 0.597011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.598126] NetLabel: unlabeled traffic allowed by default [ 0.600123] vgaarb: loaded [ 0.602157] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.603011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.608356] clocksource: Switched to clocksource kvm-clock [ 0.707586] VFS: Disk quotas dquot_6.6.0 [ 0.708943] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.711456] *** VALIDATE ramfs *** [ 0.712722] *** VALIDATE hugetlbfs *** [ 0.714207] pnp: PnP ACPI init [ 0.716365] pnp: PnP ACPI: found 6 devices [ 0.738624] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.741059] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.742675] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.744379] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.746110] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.747796] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.750151] NET: Registered protocol family 2 [ 0.752274] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.756508] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.759219] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.763666] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.766623] TCP: Hash tables configured (established 65536 bind 65536) [ 0.769720] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.772927] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.776133] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.779195] NET: Registered protocol family 1 [ 0.781817] RPC: Registered named UNIX socket transport module. [ 0.784145] RPC: Registered udp transport module. [ 0.785874] RPC: Registered tcp transport module. [ 0.787649] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.790084] NET: Registered protocol family 44 [ 0.791811] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.794030] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.796439] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.798696] PCI: CLS 0 bytes, default 64 [ 0.800289] Unpacking initramfs... [ 2.046241] debug: unmapping init [mem 0xffff98a67cc64000-0xffff98a67ffcffff] [ 2.049195] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.050777] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.052878] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.543207] Initialise system trusted keyrings [ 2.544935] Key type blacklist registered [ 2.546647] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.555482] zbud: loaded [ 2.558241] *** VALIDATE nfs *** [ 2.559470] *** VALIDATE nfs4 *** [ 2.561036] pstore: using deflate compression [ 2.564539] Platform Keyring initialized [ 2.659626] NET: Registered protocol family 38 [ 2.661211] Key type asymmetric registered [ 2.662823] Asymmetric key parser 'x509' registered [ 2.664658] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.667568] io scheduler mq-deadline registered [ 2.669567] io scheduler kyber registered [ 2.670970] io scheduler bfq registered [ 2.673086] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.676267] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.678684] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.681254] ACPI: Power Button [PWRF] [ 2.686376] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.691805] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.699780] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.726224] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.751124] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.755260] Non-volatile memory driver v1.3 [ 2.756611] Linux agpgart interface v0.103 [ 2.780553] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.783215] vda: detected capacity change from 0 to 74702848 [ 2.797250] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.799966] vdb: detected capacity change from 0 to 1073741824 [ 2.805435] libphy: Fixed MDIO Bus: probed [ 2.809639] usbcore: registered new interface driver usbserial_generic [ 2.811204] usbserial: USB Serial support registered for generic [ 2.812797] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.815652] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.816804] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.818511] mousedev: PS/2 mouse device common for all mice [ 2.820436] rtc_cmos 00:05: RTC can wake from S4 [ 2.822521] rtc_cmos 00:05: registered as rtc0 [ 2.824353] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.825341] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.826959] intel_pstate: CPU model not supported [ 2.832692] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.835890] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.835995] hid: raw HID events driver (C) Jiri Kosina [ 2.839775] usbcore: registered new interface driver usbhid [ 2.840973] usbhid: USB HID core driver [ 2.842260] drop_monitor: Initializing network drop monitor service [ 2.844522] Initializing XFRM netlink socket [ 2.846306] NET: Registered protocol family 10 [ 2.849169] Segment Routing with IPv6 [ 2.850548] NET: Registered protocol family 17 [ 2.852348] mpls_gso: MPLS GSO support [ 2.858058] RAS: Correctable Errors collector initialized. [ 2.859949] AVX version of gcm_enc/dec engaged. [ 2.861526] AES CTR mode by8 optimization enabled [ 2.929615] sched_clock: Marking stable (2929542097, 0)->(3791012970, -861470873) [ 2.933381] registered taskstats version 1 [ 2.935240] Loading compiled-in X.509 certificates [ 2.937673] zswap: loaded using pool lzo/zbud [ 2.963321] Key type big_key registered [ 2.974293] Key type encrypted registered [ 2.975630] ima: No TPM chip found, activating TPM-bypass! [ 2.977868] ima: Allocated hash algorithm: sha1 [ 2.979770] ima: No architecture policies found [ 2.981598] evm: Initialising EVM extended attributes: [ 2.983728] evm: security.selinux [ 2.984909] evm: security.ima [ 2.986055] evm: security.capability [ 2.987456] evm: HMAC attrs: 0x1 [ 2.990444] rtc_cmos 00:05: setting system clock to 2026-08-15 05:21:04 UTC (1786771264) [ 2.996664] debug: unmapping init [mem 0xffffffff94803000-0xffffffff949fffff] [ 2.999800] debug: unmapping init [mem 0xffffffff93582000-0xffffffff93858fff] [ 3.009067] Write protecting the kernel read-only data: 28672k [ 3.012898] debug: unmapping init [mem 0xffffffff91c03000-0xffffffff91dfffff] [ 3.015356] debug: unmapping init [mem 0xffffffff92514000-0xffffffff925fffff] [ 3.046232] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.054508] systemd[1]: Detected virtualization kvm. [ 3.056052] systemd[1]: Detected architecture x86-64. [ 3.057624] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.081150] systemd[1]: No hostname configured. [ 3.082379] systemd[1]: Set hostname to . [ 3.084024] random: systemd: uninitialized urandom read (16 bytes read) [ 3.085841] systemd[1]: Initializing machine ID from random generator. [ 3.124668] random: ln: uninitialized urandom read (6 bytes read) [ 3.198250] random: systemd: uninitialized urandom read (16 bytes read) [ 3.201205] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.205656] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.210203] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.707435] device-mapper: uevent: version 1.0.3 [ 3.709383] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] [ 4.253160] random: fast init done Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.285442] virtio_net virtio0 ens2: renamed from eth0 [ 4.326186] scsi host0: ata_piix [ 4.329738] scsi host1: ata_piix [ 4.330941] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.332530] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.550847] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.492991] random: crng init done [ 9.494238] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.646116] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.656211] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.226222] SELinux: Disabled at runtime. [ 11.414690] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.432098] systemd[1]: Detected virtualization kvm. [ 11.437199] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.379598] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.400485] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.425359] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.439067] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.453246] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.482754] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.515553] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ 14.421815] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 15.322883] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 16.469320] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 16.558106] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.101815] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 17.220858] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 21.269571] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit)[ 21.947112] hrtimer: interrupt took 4104034 ns [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 22.224991] NFS: Registering the id_resolver key type [ 22.228270] Key type id_resolver registered [ 22.230791] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg154-client login: [ 92.996132] libcfs: loading out-of-tree module taints kernel. [ 93.310191] Key type ._llcrypt registered [ 93.313969] Key type .llcrypt registered [ 94.142554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 94.166394] alg: No test for adler32 (adler32-zlib) [ 95.911959] Lustre: Lustre: Build Version: 2.17.57_1_g05abe65 [ 97.264582] LNet: Added LNI 192.168.201.54@tcp [8/256/0/180] [ 99.147183] Key type lgssc registered [ 100.888283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 275.229693] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 280.429488] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 296.161695] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing check_logdir /tmp/testlogs/ [ 301.023337] Lustre: lustre-OST0000-osc-ffff98a6d0319000: disconnect after 23s idle [ 303.260667] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing yml_node [ 309.211221] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 313.120891] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 316.855271] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 318.949369] Lustre: DEBUG MARKER: -----============= acceptance-small: recovery-small ============----- Sat Aug 15 01:26:18 EDT 2026 [ 337.902651] Lustre: DEBUG MARKER: excepting tests: 136 [ 340.312782] Lustre: DEBUG MARKER: === recovery-small: start setup 01:26:39 (1786771599) === [ 345.800680] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing check_config_client /mnt/lustre [ 369.940258] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 385.388483] Lustre: DEBUG MARKER: === recovery-small: finish setup 01:27:25 (1786771645) === [ 386.923378] Lustre: DEBUG MARKER: == recovery-small test 1: create, chmod, stat: drop req, drop rep ========================================================== 01:27:27 (1786771647) [ 404.448718] Lustre: 10848:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786771649/real 1786771649] req@ffff98a6d0f7b100 x1873565563502848/t0(0) o700->lustre-MDT0000-mdc-ffff98a6d0319000@192.168.201.154@tcp:30/10 lens 264/248 e 0 to 1 dl 1786771665 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 404.486663] Lustre: lustre-MDT0000-mdc-ffff98a6d0319000: Connection to lustre-MDT0000 (at 192.168.201.154@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 404.531527] Lustre: lustre-MDT0000-mdc-ffff98a6d0319000: Connection restored to 192.168.201.154@tcp (at 192.168.201.154@tcp) [ 423.391398] Lustre: 10869:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786771668/real 1786771668] req@ffff98a6d0f79880 x1873565563505152/t0(0) o36->lustre-MDT0000-mdc-ffff98a6d0319000@192.168.201.154@tcp:12/10 lens 520/576 e 0 to 1 dl 1786771684 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 423.440320] Lustre: lustre-MDT0000-mdc-ffff98a6d0319000: Connection to lustre-MDT0000 (at 192.168.201.154@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 423.496226] Lustre: lustre-MDT0000-mdc-ffff98a6d0319000: Connection restored to 192.168.201.154@tcp (at 192.168.201.154@tcp) [ 442.336169] Lustre: 10895:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786771687/real 1786771687] req@ffff98a6d0f4f480 x1873565563506944/t0(0) o101->lustre-MDT0000-mdc-ffff98a6d0319000@192.168.201.154@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786771703 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'tchmod.0' uid:0 gid:0 projid:0 [ 442.370940] Lustre: lustre-MDT0000-mdc-ffff98a6d0319000: Connection to lustre-MDT0000 (at 192.168.201.154@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 442.432437] Lustre: lustre-MDT0000-mdc-ffff98a6d0319000: Connection restored to 192.168.201.154@tcp (at 192.168.201.154@tcp) [ 462.304523] Lustre: 10915:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786771707/real 1786771707] req@ffff98a6d0f7a680 x1873565563509760/t0(0) o36->lustre-MDT0000-mdc-ffff98a6d0319000@192.168.201.154@tcp:12/10 lens 488/512 e 0 to 1 dl 1786771723 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'tchmod.0' uid:0 gid:0 projid:4294967295 [ 462.358658] Lustre: lustre-MDT0000-mdc-ffff98a6d0319000: Connection to lustre-MDT0000 (at 192.168.201.154@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 462.409106] Lustre: lustre-MDT0000-mdc-ffff98a6d0319000: Connection restored to 192.168.201.154@tcp (at 192.168.201.154@tcp) [ 482.271199] Lustre: 10941:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786771727/real 1786771727] req@ffff98a6d0e78000 x1873565563511552/t0(0) o34->lustre-MDT0000-mdc-ffff98a6d0319000@192.168.201.154@tcp:12/10 lens 472/728 e 0 to 1 dl 1786771743 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'statone.0' uid:0 gid:0 projid:0 [ 482.281526] Lustre: lustre-MDT0000-mdc-ffff98a6d0319000: Connection to lustre-MDT0000 (at 192.168.201.154@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 482.307760] Lustre: lustre-MDT0000-mdc-ffff98a6d0319000: Connection restored to 192.168.201.154@tcp (at 192.168.201.154@tcp) [ 501.216796] Lustre: 10962:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786771746/real 1786771746] req@ffff98a6d0f7b100 x1873565563513344/t0(0) o34->lustre-MDT0000-mdc-ffff98a6d0319000@192.168.201.154@tcp:12/10 lens 472/728 e 0 to 1 dl 1786771762 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'statone.0' uid:0 gid:0 projid:0 [ 501.242053] Lustre: lustre-MDT0000-mdc-ffff98a6d0319000: Connection to lustre-MDT0000 (at 192.168.201.154@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 501.282267] Lustre: lustre-MDT0000-mdc-ffff98a6d0319000: Connection restored to 192.168.201.154@tcp (at 192.168.201.154@tcp) [ 508.540756] Lustre: DEBUG MARKER: == recovery-small test 4: open: drop req, drop rep ======= 01:29:28 (1786771768) [ 526.304248] Lustre: 11574:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786771771/real 1786771771] req@ffff98a6d0f78000 x1873565563516288/t0(0) o101->lustre-MDT0000-mdc-ffff98a6d0319000@192.168.201.154@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786771787 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'cat.0' uid:0 gid:0 projid:0 [ 526.335911] Lustre: lustre-MDT0000-mdc-ffff98a6d0319000: Connection to lustre-MDT0000 (at 192.168.201.154@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 526.378554] Lustre: lustre-MDT0000-mdc-ffff98a6d0319000: Connection restored to 192.168.201.154@tcp (at 192.168.201.154@tcp) [ 555.135197] Lustre: DEBUG MARKER: == recovery-small test 5: rename: drop req, drop rep ===== 01:30:15 (1786771815) [ 572.383578] Lustre: 12202:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786771817/real 1786771817] req@ffff98a6d0f4d500 x1873565563523968/t0(0) o101->lustre-MDT0000-mdc-ffff98a6d0319000@192.168.201.154@tcp:12/10 lens 576/1152 e 0 to 1 dl 1786771833 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'mv.0' uid:0 gid:0 projid:0 [ 572.413216] Lustre: 12202:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 572.418916] Lustre: lustre-MDT0000-mdc-ffff98a6d0319000: Connection to lustre-MDT0000 (at 192.168.201.154@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 572.432953] Lustre: Skipped 1 previous similar message [ 572.463842] Lustre: lustre-MDT0000-mdc-ffff98a6d0319000: Connection restored to 192.168.201.154@tcp (at 192.168.201.154@tcp) [ 572.470461] Lustre: Skipped 1 previous similar message [ 604.252414] Lustre: DEBUG MARKER: == recovery-small test 6: link, unlink: drop req, drop rep ========================================================== 01:31:03 (1786771863) [ 621.537122] Lustre: lustre-OST0001-osc-ffff98a6d0319000: disconnect after 23s idle [ 621.541623] Lustre: Skipped 1 previous similar message [ 640.479395] Lustre: 12865:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786771885/real 1786771885] req@ffff98a6d0f4e680 x1873565563537024/t0(0) o36->lustre-MDT0000-mdc-ffff98a6d0319000@192.168.201.154@tcp:12/10 lens 512/440 e 0 to 1 dl 1786771901 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'link.0' uid:0 gid:0 projid:4294967295 [ 640.496090] Lustre: 12865:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 640.501344] Lustre: lustre-MDT0000-mdc-ffff98a6d0319000: Connection to lustre-MDT0000 (at 192.168.201.154@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 640.510219] Lustre: Skipped 2 previous similar messages [ 640.544623] Lustre: lustre-MDT0000-mdc-ffff98a6d0319000: Connection restored to 192.168.201.154@tcp (at 192.168.201.154@tcp) [ 640.558711] Lustre: Skipped 2 previous similar messages [ 686.141864] Lustre: DEBUG MARKER: == recovery-small test 8: touch: drop rep (bug 1423) ===== 01:32:26 (1786771946) [ 714.081528] Lustre: DEBUG MARKER: == recovery-small test 9: pause bulk on OST (bug 1420) === 01:32:54 (1786771974) [ 730.430303] Lustre: DEBUG MARKER: == recovery-small test 10a: finish request on server after client eviction (bug 1521) ========================================================== 01:33:10 (1786771990) [ 730.794516] Lustre: *** cfs_fail_loc=305, val=0*** [ 746.414787] Lustre: *** cfs_fail_loc=305, val=0*** [ 749.535837] Lustre: lustre-OST0001-osc-ffff98a6d0319000: disconnect after 22s idle [ 762.792441] Lustre: *** cfs_fail_loc=305, val=0*** [ 778.174350] Lustre: *** cfs_fail_loc=305, val=0*** [ 794.593349] Lustre: *** cfs_fail_loc=305, val=0*** [ 809.975814] Lustre: *** cfs_fail_loc=305, val=0*** [ 826.417387] Lustre: *** cfs_fail_loc=305, val=0*** [ 843.133176] LustreError: lustre-MDT0000-mdc-ffff98a6d0319000: operation ldlm_enqueue to node 192.168.201.154@tcp failed: rc = -107 [ 843.143206] Lustre: lustre-MDT0000-mdc-ffff98a6d0319000: Connection to lustre-MDT0000 (at 192.168.201.154@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 843.156499] Lustre: Skipped 3 previous similar messages [ 843.183259] LustreError: lustre-MDT0000-mdc-ffff98a6d0319000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 843.225811] Lustre: lustre-MDT0000-mdc-ffff98a6d0319000: Connection restored to 192.168.201.154@tcp (at 192.168.201.154@tcp) [ 843.245400] Lustre: Skipped 3 previous similar messages [ 852.882994] Lustre: DEBUG MARKER: == recovery-small test 10b: re-send BL AST =============== 01:35:12 (1786772112) [ 877.545430] Lustre: DEBUG MARKER: == recovery-small test 10c: re-send BL AST vs reconnect race (LU-5569) ========================================================== 01:35:37 (1786772137) [ 878.186677] Lustre: *** cfs_fail_loc=305, val=0*** [ 878.188324] Lustre: Skipped 1 previous similar message [ 886.921561] Lustre: DEBUG MARKER: == recovery-small test 10d: test failed blocking ast ===== 01:35:46 (1786772146) [ 889.614110] Lustre: Unmounted lustre-client [ 890.131415] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 890.673326] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 891.868096] LustreError: lustre-OST0000-osc-ffff98a6c8f05800: operation ldlm_enqueue to node 192.168.201.154@tcp failed: rc = -107 [ 891.878591] LustreError: lustre-OST0000-osc-ffff98a6c8f05800: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 891.892200] Lustre: 2395:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.154@tcp:/lustre/fid: [0x200000404:0x1:0x0]/ may get corrupted (rc -108) [ 894.455455] Lustre: Unmounted lustre-client [ 900.303277] Lustre: DEBUG MARKER: == recovery-small test 10e: re-send BL AST vs reconnect race 2 ========================================================== 01:36:00 (1786772160) [ 901.648603] Lustre: DEBUG MARKER: SKIP: recovery-small test_10e need two clients [ 903.016069] Lustre: DEBUG MARKER: == recovery-small test 11: wake up a thread waiting for completion after eviction (b=2460) ========================================================== 01:36:03 (1786772163) [ 930.926879] Lustre: DEBUG MARKER: == recovery-small test 12: recover from timed out resend in ptlrpcd (b=2494) ========================================================== 01:36:30 (1786772190) [ 931.191812] Lustre: DEBUG MARKER: multiop /mnt/lustre/f12.recovery-small OS_c [ 947.679202] Lustre: 18575:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786772193/real 1786772193] req@ffff98a6d0f4f480 x1873565563618432/t0(0) o35->lustre-MDT0000-mdc-ffff98a6c8f05800@192.168.201.154@tcp:23/10 lens 392/624 e 0 to 1 dl 1786772209 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 [ 947.727113] Lustre: 18575:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 975.812029] Lustre: DEBUG MARKER: == recovery-small test 13: mdc_readpage restart test (bug 1138) ========================================================== 01:37:15 (1786772235) [ 1003.691291] Lustre: DEBUG MARKER: == recovery-small test 14: mdc_readpage resend test (bug 1138) ========================================================== 01:37:43 (1786772263) [ 1012.576848] Lustre: DEBUG MARKER: == recovery-small test 15: failed open (-ENOMEM) ========= 01:37:52 (1786772272) [ 1020.266247] Lustre: DEBUG MARKER: == recovery-small test 16: timeout bulk put, don't evict client (2732) ========================================================== 01:38:00 (1786772280) [ 1068.119276] Lustre: DEBUG MARKER: == recovery-small test 17a: timeout bulk get, don't evict client (2732) ========================================================== 01:38:47 (1786772327) [ 1127.552608] Lustre: DEBUG MARKER: == recovery-small test 17b: timeout bulk get, dont evict client (3582) ========================================================== 01:39:47 (1786772387) [ 1128.974924] Lustre: DEBUG MARKER: SKIP: recovery-small test_17b Needs multiple clients [ 1131.533859] Lustre: DEBUG MARKER: == recovery-small test 18a: manual ost invalidate clears page cache immediately ========================================================== 01:39:50 (1786772390) [ 1132.428152] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1132.529990] Lustre: lustre-OST0001-osc-ffff98a6c8f05800: Connection to lustre-OST0001 (at 192.168.201.154@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1132.539633] Lustre: Skipped 5 previous similar messages [ 1132.559605] LustreError: lustre-OST0001-osc-ffff98a6c8f05800: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1132.577158] Lustre: lustre-OST0001-osc-ffff98a6c8f05800: Connection restored to 192.168.201.154@tcp (at 192.168.201.154@tcp) [ 1132.588317] Lustre: Skipped 4 previous similar messages [ 1140.980711] Lustre: DEBUG MARKER: == recovery-small test 18b: eviction and reconnect clears page cache (2766) ========================================================== 01:40:00 (1786772400) [ 1145.365149] LustreError: lustre-OST0000-osc-ffff98a6c8f05800: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1175.153280] Lustre: DEBUG MARKER: == recovery-small test 18c: Dropped connect reply after eviction handing (14755) ========================================================== 01:40:34 (1786772434) [ 1179.628878] LustreError: lustre-OST0000-osc-ffff98a6c8f05800: operation ost_statfs to node 192.168.201.154@tcp failed: rc = -107 [ 1189.914339] Lustre: Evicted from lustre-OST0000_UUID (at 192.168.201.154@tcp) after server handle changed from 0x984d325c3974dff5 to 0x984d325c3974e13e [ 1189.935425] LustreError: lustre-OST0000-osc-ffff98a6c8f05800: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1200.418974] Lustre: DEBUG MARKER: == recovery-small test 19a: test expired_lock_main on mds (2867) ========================================================== 01:41:00 (1786772460) [ 1201.058561] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 1218.527171] Lustre: 2393:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786772464/real 1786772464] req@ffff98a6c8e10380 x1873565563686272/t0(0) o103->lustre-MDT0000-mdc-ffff98a6c8f05800@192.168.201.154@tcp:17/18 lens 328/224 e 0 to 1 dl 1786772480 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 [ 1218.549046] Lustre: 2393:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1305.600218] LustreError: lustre-MDT0000-mdc-ffff98a6c8f05800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1309.091376] Lustre: Unmounted lustre-client [ 1316.750479] Lustre: DEBUG MARKER: == recovery-small test 19b: test expired_lock_main on ost (2867) ========================================================== 01:42:56 (1786772576) [ 1317.175656] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 1424.902299] LustreError: lustre-OST0000-osc-ffff98a6c8f05800: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1427.666190] Lustre: Unmounted lustre-client [ 1436.868930] Lustre: DEBUG MARKER: == recovery-small test 19c: check reconnect and lock resend do not trigger expired_lock_main ========================================================== 01:44:56 (1786772696) [ 1437.701428] Lustre: Mounted lustre-client - version 2.17.57_1_g05abe65 [ 1443.659962] Lustre: Unmounted lustre-client [ 1457.864618] Lustre: DEBUG MARKER: == recovery-small test 20a: ldlm_handle_enqueue error (should return error) ========================================================== 01:45:17 (1786772717) [ 1459.762532] LustreError: lustre-OST0000-osc-ffff98a6c8f05800: operation ldlm_enqueue to node 192.168.201.154@tcp failed: rc = -12 [ 1467.428920] Lustre: DEBUG MARKER: == recovery-small test 20b: ldlm_handle_enqueue error (should return error) ========================================================== 01:45:27 (1786772727) [ 1469.295969] LustreError: lustre-OST0000-osc-ffff98a6c8f05800: operation ldlm_enqueue to node 192.168.201.154@tcp failed: rc = -12 [ 1476.919945] Lustre: DEBUG MARKER: == recovery-small test 21a: drop close request while close and open are both in flight ========================================================== 01:45:36 (1786772736) [ 1508.861238] Lustre: DEBUG MARKER: == recovery-small test 21b: drop open request while close and open are both in flight ========================================================== 01:46:08 (1786772768) [ 1651.679272] Lustre: lustre-MDT0000-mdc-ffff98a6c8f05800: Connection to lustre-MDT0000 (at 192.168.201.154@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1651.688069] Lustre: Skipped 17 previous similar messages [ 1651.709669] Lustre: lustre-MDT0000-mdc-ffff98a6c8f05800: Connection restored to 192.168.201.154@tcp (at 192.168.201.154@tcp) [ 1651.714711] Lustre: Skipped 14 previous similar messages [ 1658.934942] Lustre: DEBUG MARKER: == recovery-small test 21c: drop both request while close and open are both in flight ========================================================== 01:48:39 (1786772919) [ 1690.564983] Lustre: DEBUG MARKER: == recovery-small test 21d: drop close reply while close and open are both in flight ========================================================== 01:49:10 (1786772950) [ 1719.596334] Lustre: DEBUG MARKER: == recovery-small test 21e: drop open reply while close and open are both in flight ========================================================== 01:49:39 (1786772979) [ 1861.600818] Lustre: 30738:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786772982/real 1786772982] req@ffff98a6d0e78700 x1873565563821440/t0(0) o36->lustre-MDT0000-mdc-ffff98a6c8f05800@192.168.201.154@tcp:12/10 lens 488/512 e 0 to 1 dl 1786773123 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 1861.671847] Lustre: 30738:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [ 1863.033625] Lustre: DEBUG MARKER: == recovery-small test 21f: drop both reply while close and open are both in flight ========================================================== 01:52:03 (1786773123) [ 1888.635420] Lustre: DEBUG MARKER: == recovery-small test 21g: drop open reply and close request while close and open are both in flight ========================================================== 01:52:28 (1786773148) [ 1915.778696] Lustre: DEBUG MARKER: == recovery-small test 21h: drop open request and close reply while close and open are both in flight ========================================================== 01:52:55 (1786773175) [ 1942.394518] Lustre: DEBUG MARKER: == recovery-small test 22: drop close request and do mknod ========================================================== 01:53:22 (1786773202) [ 1968.264905] Lustre: DEBUG MARKER: == recovery-small test 23: client hang when close a file after mds crash ========================================================== 01:53:47 (1786773227) [ 1997.282258] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 192.168.201.154@tcp) was lost; in progress operations using this service will fail [ 1997.336980] Lustre: Evicted from MGS (at 192.168.201.154@tcp) after server handle changed from 0x984d325c3974d639 to 0x984d325c3974fc9d [ 1998.902889] Lustre: 16830:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.154@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 2020.387339] Lustre: DEBUG MARKER: oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2022.669768] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2033.632234] Lustre: DEBUG MARKER: == recovery-small test 24a: fsync error (should return error) ========================================================== 01:54:53 (1786773293) [ 2035.647595] LustreError: lustre-OST0000-osc-ffff98a6c8f05800: operation ost_write to node 192.168.201.154@tcp failed: rc = -107 [ 2035.667410] LustreError: lustre-OST0000-osc-ffff98a6c8f05800: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2035.685897] Lustre: 2395:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.154@tcp:/lustre/fid: [0x240000402:0x8:0x0]// may get corrupted (rc -5) [ 2043.392023] Lustre: DEBUG MARKER: == recovery-small test 24b: test dirty page discard due to client eviction ========================================================== 01:55:03 (1786773303) [ 2044.891937] LustreError: lustre-OST0000-osc-ffff98a6c8f05800: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2044.905480] Lustre: 2394:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.154@tcp:/lustre/fid: [0x200000405:0x2e:0x0]// may get corrupted (rc -5) [ 2044.914092] Lustre: 2395:0:(llite_lib.c:4340:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.201.154@tcp:/lustre/fid: [0x200000405:0x2f:0x0]// may get corrupted (rc -108) [ 2052.531626] Lustre: DEBUG MARKER: == recovery-small test 26a: evict dead exports =========== 01:55:11 (1786773311) [ 2055.036867] Lustre: DEBUG MARKER: SKIP: recovery-small test_26a msg and ost1 are at the same node [ 2057.516618] Lustre: DEBUG MARKER: == recovery-small test 26b: evict dead exports =========== 01:55:17 (1786773317) [ 2059.412675] Lustre: DEBUG MARKER: SKIP: recovery-small test_26b msg and ost1 are at the same node [ 2061.029361] Lustre: DEBUG MARKER: == recovery-small test 27: fail LOV while using OSC's ==== 01:55:21 (1786773321) [ 2064.582928] LustreError: lustre-MDT0000-mdc-ffff98a6c8f05800: operation ldlm_enqueue to node 192.168.201.154@tcp failed: rc = -19 [ 2064.594587] LustreError: Skipped 1 previous similar message [ 2084.319404] LustreError: MGC192.168.201.154@tcp: Connection to MGS (at 192.168.201.154@tcp) was lost; in progress operations using this service will fail [ 2084.350187] Lustre: Evicted from MGS (at 192.168.201.154@tcp) after server handle changed from 0x984d325c3974fc9d to 0x984d325c39751715 [ 2086.251693] Lustre: 16830:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.154@tcp: IR log lustre-cliir failed, not fatal: rc = -2 [ 2094.219331] BUG: spinlock recursion on CPU#2, ptlrpcd_rcv/2391 [ 2094.223020] lock: 0xffff98a6d1bc8ae0, .magic: dead4ead, .owner: ptlrpcd_rcv/2391, .owner_cpu: 2 [ 2094.227163] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 2094.232031] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2094.236279] Call Trace: [ 2094.237303] ? dump_stack+0xbb/0x10e [ 2094.240674] ? spin_dump.cold.3+0x32/0x65 [ 2094.242881] ? do_raw_spin_lock+0x10f/0x190 [ 2094.244490] ? _raw_spin_lock+0x21/0x30 [ 2094.246003] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2094.248116] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2094.250941] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2094.252928] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2094.255295] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2094.257949] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2094.260099] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2094.262059] ? __next_timer_interrupt+0x160/0x160 [ 2094.263752] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2094.266148] ? do_wait_intr+0xf0/0xf0 [ 2094.267562] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2094.269914] ? kthread+0x1d1/0x200 [ 2094.271336] ? set_kthread_struct+0x70/0x70 [ 2094.272841] ? ret_from_fork+0x1f/0x30 [ 2120.052114] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2120.055324] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 2120.057385] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 2120.057398] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2120.057467] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 2120.057473] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2120.057475] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2120.057490] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2120.057495] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2120.057499] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2120.057502] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2120.057504] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2120.057507] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2120.057509] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2120.057512] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2120.057515] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2120.057517] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2120.057526] Call Trace: [ 2120.057531] [ 2120.057536] ? show_regs.cold.9+0x22/0x2f [ 2120.057546] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2120.057553] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2120.057557] ? watchdog+0x40/0x40 [ 2120.057561] ? hrtimer_interrupt+0x128/0x2c0 [ 2120.057565] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2120.057571] ? apic_timer_interrupt+0xf/0x20 [ 2120.057575] [ 2120.057578] ? native_safe_halt+0x2e/0x50 [ 2120.057584] kvm_wait+0xcd/0xe0 [ 2120.057591] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2120.057598] do_raw_spin_lock+0x177/0x190 [ 2120.057603] _raw_spin_lock+0x21/0x30 [ 2120.057608] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2120.058093] Modules linked in: lustre(O) [ 2120.058702] obd_mod_free+0x46/0x140 [ptlrpc] [ 2120.058927] mdc_commit_open+0x91/0xb0 [mdc] [ 2120.059282] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2120.059521] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2120.059984] after_reply+0xe2c/0x1780 [ptlrpc] [ 2120.060194] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2120.060411] ? __next_timer_interrupt+0x160/0x160 [ 2120.060421] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2120.061449] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2120.061521] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 2120.061527] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2120.061530] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2120.061541] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2120.060622] ? do_wait_intr+0xf0/0xf0 [ 2120.061761] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2120.063391] kthread+0x1d1/0x200 [ 2120.063401] ? set_kthread_struct+0x70/0x70 [ 2120.063405] ret_from_fork+0x1f/0x30 [ 2120.069938] osp(O) [ 2120.093878] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2120.093893] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2120.093895] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2120.093897] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2120.093900] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2120.093902] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2120.093905] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2120.093908] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2120.093910] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2120.093919] Call Trace: [ 2120.093927] [ 2120.093933] ? show_regs.cold.9+0x22/0x2f [ 2120.093948] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2120.093954] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2120.093958] ? watchdog+0x40/0x40 [ 2120.093963] ? hrtimer_interrupt+0x128/0x2c0 [ 2120.093966] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2120.093975] ? apic_timer_interrupt+0xf/0x20 [ 2120.093980] [ 2120.093982] ? native_safe_halt+0x2e/0x50 [ 2120.093987] kvm_wait+0xcd/0xe0 [ 2120.093994] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2120.094004] do_raw_spin_lock+0x177/0x190 [ 2120.094008] _raw_spin_lock+0x21/0x30 [ 2120.094014] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2120.094470] ? do_raw_spin_unlock+0x75/0x190 [ 2120.094476] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2120.095011] ? do_wait_intr+0xf0/0xf0 [ 2120.095021] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2120.102142] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2120.102211] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2120.102219] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2120.102226] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2120.102573] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2120.102583] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2120.102588] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2120.102593] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2120.102595] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2120.102598] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2120.102600] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2120.102603] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2120.102606] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2120.102609] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2120.102612] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2120.102621] Call Trace: [ 2120.102627] [ 2120.102632] ? show_regs.cold.9+0x22/0x2f [ 2120.102642] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2120.102650] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2120.102655] ? watchdog+0x40/0x40 [ 2120.102659] ? hrtimer_interrupt+0x128/0x2c0 [ 2120.102663] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2120.102670] ? apic_timer_interrupt+0xf/0x20 [ 2120.102674] [ 2120.102677] ? native_safe_halt+0x2e/0x50 [ 2120.102681] kvm_wait+0xcd/0xe0 [ 2120.102688] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2120.102695] ? newidle_balance.isra.93+0x207/0x580 [ 2120.102702] do_raw_spin_lock+0x177/0x190 [ 2120.102707] _raw_spin_lock+0x21/0x30 [ 2120.102711] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2120.106698] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2120.107243] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2120.107561] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2120.107674] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2120.107712] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2120.107739] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2120.107845] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2120.107929] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2120.108539] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2120.108566] lmv_intent_open+0x338/0x1150 [lmv] [ 2120.108597] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2120.108681] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2120.108831] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2120.108853] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2120.108936] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2120.109017] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2120.109173] ? do_raw_spin_unlock+0x75/0x190 [ 2120.109180] ? _raw_spin_unlock+0x12/0x30 [ 2120.109187] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2120.109213] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2120.109234] ll_atomic_open+0x36a/0xd70 [lustre] [ 2120.110916] process_one_work+0x2c8/0x700 [ 2120.116820] lookup_open+0x356/0xb40 [ 2120.126983] worker_thread+0x296/0x6e0 [ 2120.130490] do_last+0x313/0xfc0 [ 2120.134982] ? rescuer_thread+0x570/0x570 [ 2120.135002] kthread+0x1d1/0x200 [ 2120.135007] ? set_kthread_struct+0x70/0x70 [ 2120.135011] ret_from_fork+0x1f/0x30 [ 2120.722407] ? nd_jump_root+0xe5/0x160 [ 2120.724707] ? path_init+0x437/0x520 [ 2120.727358] path_openat+0xf7/0x500 [ 2120.729418] ? do_raw_spin_unlock+0x75/0x190 [ 2120.732002] do_filp_open+0x99/0x140 [ 2120.734519] ? getname_flags+0x6e/0x330 [ 2120.735882] ? __check_object_size+0xff/0x256 [ 2120.738776] ? do_raw_spin_unlock+0x75/0x190 [ 2120.744814] ? _raw_spin_unlock+0x12/0x30 [ 2120.748814] do_sys_openat2+0x2b4/0x410 [ 2120.750925] do_sys_open+0x73/0xa0 [ 2120.752899] __x64_sys_openat+0x24/0x30 [ 2120.759639] do_syscall_64+0xc1/0x440 [ 2120.765714] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2120.768053] RIP: 0033:0x7f53425a5332 [ 2120.770916] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2120.785021] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2120.788876] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2120.792437] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2120.795651] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2120.798979] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2120.805546] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2148.053100] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2148.055196] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 2148.056250] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2148.056325] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2148.056331] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2148.056335] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2148.056658] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2148.056672] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2148.056677] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2148.056682] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2148.056684] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2148.056687] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2148.056689] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2148.056691] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2148.056696] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2148.056698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2148.056701] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2148.056710] Call Trace: [ 2148.056718] [ 2148.056726] ? show_regs.cold.9+0x22/0x2f [ 2148.056739] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2148.056746] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2148.056750] ? watchdog+0x40/0x40 [ 2148.056753] ? hrtimer_interrupt+0x128/0x2c0 [ 2148.056757] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2148.056765] ? apic_timer_interrupt+0xf/0x20 [ 2148.056769] [ 2148.056772] ? native_safe_halt+0x2e/0x50 [ 2148.056776] kvm_wait+0xcd/0xe0 [ 2148.056785] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2148.056794] ? newidle_balance.isra.93+0x207/0x580 [ 2148.056800] do_raw_spin_lock+0x177/0x190 [ 2148.056804] _raw_spin_lock+0x21/0x30 [ 2148.056808] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2148.057021] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 2148.057037] Modules linked in: [ 2148.057037] process_one_work+0x2c8/0x700 [ 2148.057041] lustre(O) osp(O) [ 2148.057053] worker_thread+0x296/0x6e0 [ 2148.057072] ? rescuer_thread+0x570/0x570 [ 2148.057077] kthread+0x1d1/0x200 [ 2148.057081] ? set_kthread_struct+0x70/0x70 [ 2148.057085] ret_from_fork+0x1f/0x30 [ 2148.057199] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2148.057320] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2148.057327] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2148.057329] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2148.057342] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2148.057346] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2148.057350] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2148.057352] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2148.057354] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2148.057356] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2148.057358] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2148.057360] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2148.057364] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2148.057366] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2148.057374] Call Trace: [ 2148.057378] [ 2148.057383] ? show_regs.cold.9+0x22/0x2f [ 2148.057396] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2148.057403] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2148.057462] ? watchdog+0x40/0x40 [ 2148.057468] ? hrtimer_interrupt+0x128/0x2c0 [ 2148.057473] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2148.057481] ? apic_timer_interrupt+0xf/0x20 [ 2148.057486] [ 2148.057489] ? native_safe_halt+0x2e/0x50 [ 2148.057493] kvm_wait+0xcd/0xe0 [ 2148.057500] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2148.057507] do_raw_spin_lock+0x177/0x190 [ 2148.057512] _raw_spin_lock+0x21/0x30 [ 2148.057518] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2148.057856] obd_mod_free+0x46/0x140 [ptlrpc] [ 2148.062409] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver [ 2148.100092] mdc_commit_open+0x91/0xb0 [mdc] [ 2148.106484] intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2148.106656] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2148.106667] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2148.106672] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2148.106693] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2148.106696] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2148.106701] RAX: 0000000000007f87 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2148.106703] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2148.106705] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2148.106707] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2148.106709] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2148.106713] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2148.106715] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2148.106717] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2148.106726] Call Trace: [ 2148.106733] [ 2148.106742] ? show_regs.cold.9+0x22/0x2f [ 2148.106754] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2148.106761] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2148.106765] ? watchdog+0x40/0x40 [ 2148.106769] ? hrtimer_interrupt+0x128/0x2c0 [ 2148.106772] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2148.106781] ? apic_timer_interrupt+0xf/0x20 [ 2148.106785] [ 2148.106788] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2148.106796] do_raw_spin_lock+0x177/0x190 [ 2148.106800] _raw_spin_lock+0x21/0x30 [ 2148.106806] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2148.107163] ? do_raw_spin_unlock+0x75/0x190 [ 2148.107169] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2148.107359] ? do_wait_intr+0xf0/0xf0 [ 2148.107368] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2148.107690] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2148.107969] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2148.108188] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2148.108322] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2148.108374] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2148.108403] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2148.108427] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2148.108511] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2148.109061] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2148.109100] lmv_intent_open+0x338/0x1150 [lmv] [ 2148.109134] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2148.109228] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2148.109313] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2148.109344] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2148.109428] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2148.109515] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2148.109670] ? do_raw_spin_unlock+0x75/0x190 [ 2148.109677] ? _raw_spin_unlock+0x12/0x30 [ 2148.109686] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2148.109712] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2148.109731] ll_atomic_open+0x36a/0xd70 [lustre] [ 2148.109813] lookup_open+0x356/0xb40 [ 2148.109839] do_last+0x313/0xfc0 [ 2148.109845] ? nd_jump_root+0xe5/0x160 [ 2148.109849] ? path_init+0x437/0x520 [ 2148.109854] path_openat+0xf7/0x500 [ 2148.109859] ? do_raw_spin_unlock+0x75/0x190 [ 2148.109864] do_filp_open+0x99/0x140 [ 2148.109871] ? getname_flags+0x6e/0x330 [ 2148.109875] ? __check_object_size+0xff/0x256 [ 2148.109883] ? do_raw_spin_unlock+0x75/0x190 [ 2148.109887] ? _raw_spin_unlock+0x12/0x30 [ 2148.109892] do_sys_openat2+0x2b4/0x410 [ 2148.109896] do_sys_open+0x73/0xa0 [ 2148.109900] __x64_sys_openat+0x24/0x30 [ 2148.109904] do_syscall_64+0xc1/0x440 [ 2148.109910] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2148.109916] RIP: 0033:0x7f53425a5332 [ 2148.109925] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2148.109928] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2148.109934] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2148.109936] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2148.109938] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2148.109940] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2148.109942] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2148.651968] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2148.653785] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2148.655473] after_reply+0xe2c/0x1780 [ptlrpc] [ 2148.657000] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2148.658619] ? __next_timer_interrupt+0x160/0x160 [ 2148.660034] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2148.661851] ? do_wait_intr+0xf0/0xf0 [ 2148.662800] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2148.664957] kthread+0x1d1/0x200 [ 2148.665943] ? set_kthread_struct+0x70/0x70 [ 2148.666933] ret_from_fork+0x1f/0x30 [ 2154.426136] rcu: INFO: rcu_sched self-detected stall on CPU [ 2154.428476] rcu: 2-....: (33434 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=10817 [ 2154.435977] (t=60000 jiffies g=274185 q=27884) [ 2154.437958] Sending NMI from CPU 2 to CPUs 1: [ 2154.439887] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 2154.441789] NMI backtrace for cpu 2 [ 2154.446470] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2154.451948] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2154.460530] Call Trace: [ 2154.462635] [ 2154.463382] ? dump_stack+0xbb/0x10e [ 2154.464673] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2154.467426] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2154.470909] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2154.475369] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2154.479132] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2154.483292] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2154.486440] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2154.488501] ? update_process_times+0x6d/0xc0 [ 2154.490322] ? tick_sched_handle.isra.18+0x27/0x90 [ 2154.492117] ? tick_sched_timer+0x53/0xb0 [ 2154.493517] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2154.495467] ? tick_sched_handle.isra.18+0x90/0x90 [ 2154.497257] ? hrtimer_interrupt+0x128/0x2c0 [ 2154.498825] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2154.500589] ? apic_timer_interrupt+0xf/0x20 [ 2154.502330] [ 2154.503182] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2154.505254] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2154.507272] ? do_raw_spin_lock+0x177/0x190 [ 2154.508852] ? _raw_spin_lock+0x21/0x30 [ 2154.510286] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2154.512359] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2154.514294] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2154.516301] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2154.518867] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2154.521984] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2154.524489] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2154.527131] ? __next_timer_interrupt+0x160/0x160 [ 2154.528652] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2154.530672] ? do_wait_intr+0xf0/0xf0 [ 2154.532194] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2154.534471] ? kthread+0x1d1/0x200 [ 2154.535683] ? set_kthread_struct+0x70/0x70 [ 2154.537613] ? ret_from_fork+0x1f/0x30 [ 2176.052022] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2176.055656] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2176.078439] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2176.083600] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2176.088062] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2176.091170] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2176.093233] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2176.101698] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2176.105384] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2176.108527] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2176.111787] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2176.115056] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2176.118464] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2176.121564] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2176.125188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2176.127662] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2176.131133] Call Trace: [ 2176.132016] [ 2176.133091] ? show_regs.cold.9+0x22/0x2f [ 2176.134568] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2176.137161] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2176.139025] ? watchdog+0x40/0x40 [ 2176.141193] ? hrtimer_interrupt+0x128/0x2c0 [ 2176.143443] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2176.145909] ? apic_timer_interrupt+0xf/0x20 [ 2176.147831] [ 2176.148579] ? native_safe_halt+0x2e/0x50 [ 2176.150480] kvm_wait+0xcd/0xe0 [ 2176.151949] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2176.153843] ? newidle_balance.isra.93+0x207/0x580 [ 2176.155650] do_raw_spin_lock+0x177/0x190 [ 2176.157159] _raw_spin_lock+0x21/0x30 [ 2176.161132] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2176.164441] process_one_work+0x2c8/0x700 [ 2176.166225] worker_thread+0x296/0x6e0 [ 2176.168597] ? rescuer_thread+0x570/0x570 [ 2176.170190] kthread+0x1d1/0x200 [ 2176.171890] ? set_kthread_struct+0x70/0x70 [ 2176.173294] ret_from_fork+0x1f/0x30 [ 2180.055016] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 2180.061100] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2180.062297] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 2180.085482] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2180.085498] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2180.085502] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2180.085518] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2180.085523] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2180.085529] RAX: 0000000000000e8e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2180.085531] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2180.085534] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2180.085536] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2180.085538] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2180.085542] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2180.085545] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2180.085548] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2180.085557] Call Trace: [ 2180.085565] [ 2180.085573] ? show_regs.cold.9+0x22/0x2f [ 2180.085585] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2180.085593] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2180.085597] ? watchdog+0x40/0x40 [ 2180.085601] ? hrtimer_interrupt+0x128/0x2c0 [ 2180.085604] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2180.085614] ? apic_timer_interrupt+0xf/0x20 [ 2180.085618] [ 2180.085620] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2180.085628] do_raw_spin_lock+0x177/0x190 [ 2180.085633] _raw_spin_lock+0x21/0x30 [ 2180.085639] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2180.093065] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2180.093161] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2180.093172] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2180.093174] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2180.093191] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2180.093206] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2180.093211] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2180.093213] RDX: 0000000000000f80 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2180.093216] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2180.093218] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2180.093221] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2180.093225] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2180.093228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2180.093230] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2180.093239] Call Trace: [ 2180.093245] [ 2180.093250] ? show_regs.cold.9+0x22/0x2f [ 2180.093274] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2180.093281] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2180.093284] ? watchdog+0x40/0x40 [ 2180.093288] ? hrtimer_interrupt+0x128/0x2c0 [ 2180.093291] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2180.093299] ? apic_timer_interrupt+0xf/0x20 [ 2180.093303] [ 2180.093305] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2180.093309] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2180.093313] do_raw_spin_lock+0x177/0x190 [ 2180.093317] _raw_spin_lock+0x21/0x30 [ 2180.093323] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2180.093623] obd_mod_free+0x46/0x140 [ptlrpc] [ 2180.093831] mdc_commit_open+0x91/0xb0 [mdc] [ 2180.093869] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2180.094073] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2180.094438] after_reply+0xe2c/0x1780 [ptlrpc] [ 2180.094649] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2180.094848] ? __next_timer_interrupt+0x160/0x160 [ 2180.094860] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2180.095070] ? do_wait_intr+0xf0/0xf0 [ 2180.095079] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2180.100025] ? do_raw_spin_unlock+0x75/0x190 [ 2180.100058] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2180.100492] ? do_wait_intr+0xf0/0xf0 [ 2180.100505] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2180.100707] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2180.100925] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2180.101144] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2180.101473] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2180.101527] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2180.101562] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2180.101591] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2180.101681] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2180.101924] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2180.101955] lmv_intent_open+0x338/0x1150 [lmv] [ 2180.101996] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2180.102083] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2180.102163] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2180.102185] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2180.109169] kthread+0x1d1/0x200 [ 2180.109188] ? set_kthread_struct+0x70/0x70 [ 2180.109193] ret_from_fork+0x1f/0x30 [ 2180.459173] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2180.464678] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2180.471392] ? do_raw_spin_unlock+0x75/0x190 [ 2180.474892] ? _raw_spin_unlock+0x12/0x30 [ 2180.478574] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2180.480231] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2180.486789] ll_atomic_open+0x36a/0xd70 [lustre] [ 2180.493173] lookup_open+0x356/0xb40 [ 2180.498136] do_last+0x313/0xfc0 [ 2180.501774] ? nd_jump_root+0xe5/0x160 [ 2180.504428] ? path_init+0x437/0x520 [ 2180.507043] path_openat+0xf7/0x500 [ 2180.510699] ? do_raw_spin_unlock+0x75/0x190 [ 2180.514909] do_filp_open+0x99/0x140 [ 2180.516747] ? getname_flags+0x6e/0x330 [ 2180.518189] ? __check_object_size+0xff/0x256 [ 2180.520932] ? do_raw_spin_unlock+0x75/0x190 [ 2180.524826] ? _raw_spin_unlock+0x12/0x30 [ 2180.528809] do_sys_openat2+0x2b4/0x410 [ 2180.531488] do_sys_open+0x73/0xa0 [ 2180.534879] __x64_sys_openat+0x24/0x30 [ 2180.540335] do_syscall_64+0xc1/0x440 [ 2180.544655] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2180.549375] RIP: 0033:0x7f53425a5332 [ 2180.552153] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2180.575435] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2180.584130] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2180.593170] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2180.608458] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2180.612720] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2180.621432] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2204.052106] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2204.055992] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2204.076419] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2204.082033] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2204.086648] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2204.090588] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2204.093230] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2204.100984] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2204.103901] RAX: 0000000000006f91 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2204.106975] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2204.110257] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2204.113883] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2204.117340] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2204.120777] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2204.124377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2204.127112] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2204.130459] Call Trace: [ 2204.131497] [ 2204.132436] ? show_regs.cold.9+0x22/0x2f [ 2204.133753] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2204.135668] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2204.137586] ? watchdog+0x40/0x40 [ 2204.139031] ? hrtimer_interrupt+0x128/0x2c0 [ 2204.140542] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2204.142554] ? apic_timer_interrupt+0xf/0x20 [ 2204.144294] [ 2204.145226] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2204.147237] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2204.149121] ? newidle_balance.isra.93+0x207/0x580 [ 2204.151047] do_raw_spin_lock+0x177/0x190 [ 2204.152599] _raw_spin_lock+0x21/0x30 [ 2204.154072] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2204.156047] process_one_work+0x2c8/0x700 [ 2204.157382] worker_thread+0x296/0x6e0 [ 2204.158819] ? rescuer_thread+0x570/0x570 [ 2204.160173] kthread+0x1d1/0x200 [ 2204.161301] ? set_kthread_struct+0x70/0x70 [ 2204.162613] ret_from_fork+0x1f/0x30 [ 2208.055894] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 2208.057847] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 2208.058253] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2208.058326] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2208.058331] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2208.058334] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2208.058347] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2208.058351] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2208.058355] RAX: 0000000000007f68 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2208.058357] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2208.058359] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2208.058361] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2208.058363] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2208.058366] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2208.058368] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2208.058370] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2208.058379] Call Trace: [ 2208.058384] [ 2208.058389] ? show_regs.cold.9+0x22/0x2f [ 2208.058400] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2208.058405] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2208.058409] ? watchdog+0x40/0x40 [ 2208.058413] ? hrtimer_interrupt+0x128/0x2c0 [ 2208.058416] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2208.058425] ? apic_timer_interrupt+0xf/0x20 [ 2208.058429] [ 2208.058431] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2208.058438] do_raw_spin_lock+0x177/0x190 [ 2208.058442] _raw_spin_lock+0x21/0x30 [ 2208.058447] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2208.058714] ? do_raw_spin_unlock+0x75/0x190 [ 2208.058720] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2208.067928] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2208.068015] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2208.068023] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2208.068026] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2208.068040] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2208.068261] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 [ 2208.097089] ? do_wait_intr+0xf0/0xf0 [ 2208.097113] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2208.097578] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2208.098203] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2208.099081] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2208.099194] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2208.099236] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2208.103714] ORIG_RAX: ffffffffffffff13 [ 2208.103725] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2208.103727] RDX: 00000000000030d6 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2208.103731] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2208.103733] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2208.103735] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2208.103738] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2208.103740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2208.103742] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2208.103750] Call Trace: [ 2208.103757] [ 2208.103762] ? show_regs.cold.9+0x22/0x2f [ 2208.112132] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2208.112186] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2208.112299] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2208.117083] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2208.117104] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2208.117108] ? watchdog+0x40/0x40 [ 2208.117113] ? hrtimer_interrupt+0x128/0x2c0 [ 2208.117118] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2208.117127] ? apic_timer_interrupt+0xf/0x20 [ 2208.117132] [ 2208.117137] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2208.117143] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2208.117148] do_raw_spin_lock+0x177/0x190 [ 2208.117154] _raw_spin_lock+0x21/0x30 [ 2208.117162] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2208.117459] obd_mod_free+0x46/0x140 [ptlrpc] [ 2208.118231] mdc_commit_open+0x91/0xb0 [mdc] [ 2208.118268] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2208.118732] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2208.119130] after_reply+0xe2c/0x1780 [ptlrpc] [ 2208.119326] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2208.120081] ? __next_timer_interrupt+0x160/0x160 [ 2208.120092] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2208.120295] ? do_wait_intr+0xf0/0xf0 [ 2208.120306] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2208.140025] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2208.142444] kthread+0x1d1/0x200 [ 2208.154103] lmv_intent_open+0x338/0x1150 [lmv] [ 2208.154153] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2208.154271] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2208.154356] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2208.155571] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2208.155702] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2208.155958] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2208.156050] ? do_raw_spin_unlock+0x75/0x190 [ 2208.156060] ? _raw_spin_unlock+0x12/0x30 [ 2208.156067] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2208.156096] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2208.156118] ll_atomic_open+0x36a/0xd70 [lustre] [ 2208.156201] lookup_open+0x356/0xb40 [ 2208.156210] do_last+0x313/0xfc0 [ 2208.156215] ? nd_jump_root+0xe5/0x160 [ 2208.156219] ? path_init+0x437/0x520 [ 2208.156223] path_openat+0xf7/0x500 [ 2208.156227] ? do_raw_spin_unlock+0x75/0x190 [ 2208.156232] do_filp_open+0x99/0x140 [ 2208.156237] ? getname_flags+0x6e/0x330 [ 2208.156242] ? __check_object_size+0xff/0x256 [ 2208.156248] ? do_raw_spin_unlock+0x75/0x190 [ 2208.156252] ? _raw_spin_unlock+0x12/0x30 [ 2208.156256] do_sys_openat2+0x2b4/0x410 [ 2208.156261] do_sys_open+0x73/0xa0 [ 2208.156269] __x64_sys_openat+0x24/0x30 [ 2208.156273] do_syscall_64+0xc1/0x440 [ 2208.156279] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2208.156284] RIP: 0033:0x7f53425a5332 [ 2208.156292] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2208.156296] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2208.156300] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2208.156303] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2208.156305] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2208.156307] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2208.156309] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2208.634089] ? set_kthread_struct+0x70/0x70 [ 2208.636515] ret_from_fork+0x1f/0x30 [ 2212.832598] INFO: task writemany:37225 blocked for more than 120 seconds. [ 2212.840666] Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2212.848560] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2212.855651] task:writemany state:D stack:0 pid:37225 ppid:37222 flags:0x80004000 [ 2212.860790] Call Trace: [ 2212.862343] __schedule+0x351/0xcb0 [ 2212.864498] schedule+0xc0/0x180 [ 2212.866318] schedule_preempt_disabled+0x21/0x40 [ 2212.868258] rwsem_down_write_slowpath+0x5d7/0xa40 [ 2212.869557] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2212.872413] down_write+0x80/0xd0 [ 2212.873919] do_last+0x2eb/0xfc0 [ 2212.875562] ? nd_jump_root+0xe5/0x160 [ 2212.877598] ? path_init+0x437/0x520 [ 2212.879522] path_openat+0xf7/0x500 [ 2212.881596] ? do_raw_spin_unlock+0x75/0x190 [ 2212.885819] do_filp_open+0x99/0x140 [ 2212.887591] ? getname_flags+0x6e/0x330 [ 2212.889556] ? __check_object_size+0xff/0x256 [ 2212.891626] ? do_raw_spin_unlock+0x75/0x190 [ 2212.894256] ? _raw_spin_unlock+0x12/0x30 [ 2212.896098] do_sys_openat2+0x2b4/0x410 [ 2212.897767] do_sys_open+0x73/0xa0 [ 2212.899461] __x64_sys_openat+0x24/0x30 [ 2212.904183] do_syscall_64+0xc1/0x440 [ 2212.905304] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2212.910181] RIP: 0033:0x7f53425a5332 [ 2212.914353] Code: Unable to access opcode bytes at RIP 0x7f53425a5308. [ 2212.919468] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2212.923476] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f53425a5332 [ 2212.926487] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2212.932815] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2212.935585] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2212.938409] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2212.946269] INFO: task writemany:37226 blocked for more than 120 seconds. [ 2212.949548] Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2212.958425] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2212.966293] task:writemany state:D stack:0 pid:37226 ppid:37222 flags:0x80004000 [ 2212.976340] Call Trace: [ 2212.978502] __schedule+0x351/0xcb0 [ 2212.982294] schedule+0xc0/0x180 [ 2212.984021] schedule_preempt_disabled+0x21/0x40 [ 2212.989812] rwsem_down_write_slowpath+0x5d7/0xa40 [ 2212.995443] ? __might_sleep+0x59/0xc0 [ 2213.003369] down_write+0x80/0xd0 [ 2213.011314] do_unlinkat+0x184/0x460 [ 2213.018630] __x64_sys_unlink+0x37/0x50 [ 2213.021673] do_syscall_64+0xc1/0x440 [ 2213.032537] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2213.036271] RIP: 0033:0x7f53424b010b [ 2213.040585] Code: Unable to access opcode bytes at RIP 0x7f53424b00e1. [ 2213.046441] RSP: 002b:00007ffff8ec31a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2213.056369] RAX: ffffffffffffffda RBX: 00000000000015e0 RCX: 00007f53424b010b [ 2213.063604] RDX: 0000000000000038 RSI: 00007ffff8ec3610 RDI: 00007ffff8ec3210 [ 2213.067625] RBP: 00007ffff8ec3610 R08: 00007f5342848238 R09: 00007f5342848260 [ 2213.074200] R10: 00007ffff8ec2f52 R11: 0000000000000246 R12: 0000000000000003 [ 2213.084808] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2213.095737] INFO: task writemany:37228 blocked for more than 120 seconds. [ 2213.098540] Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2213.108631] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2213.113905] task:writemany state:D stack:0 pid:37228 ppid:37222 flags:0x80000000 [ 2213.120376] Call Trace: [ 2213.125388] __schedule+0x351/0xcb0 [ 2213.129586] schedule+0xc0/0x180 [ 2213.136439] schedule_preempt_disabled+0x21/0x40 [ 2213.140736] rwsem_down_write_slowpath+0x5d7/0xa40 [ 2213.148524] ? __might_sleep+0x59/0xc0 [ 2213.154474] down_write+0x80/0xd0 [ 2213.156776] do_unlinkat+0x184/0x460 [ 2213.162674] __x64_sys_unlink+0x37/0x50 [ 2213.165459] do_syscall_64+0xc1/0x440 [ 2213.169550] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2213.175579] RIP: 0033:0x7f53424b010b [ 2213.179290] Code: Unable to access opcode bytes at RIP 0x7f53424b00e1. [ 2213.180841] RSP: 002b:00007ffff8ec31a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2213.183360] RAX: ffffffffffffffda RBX: 000000000002c258 RCX: 00007f53424b010b [ 2213.187635] RDX: 0000000000000038 RSI: 00007ffff8ec3610 RDI: 00007ffff8ec3210 [ 2213.196588] RBP: 00007ffff8ec3610 R08: 00007f534284821c R09: 00007f5342848260 [ 2213.205067] R10: 00007ffff8ec2f52 R11: 0000000000000246 R12: 0000000000000003 [ 2213.207440] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2213.252769] INFO: task writemany:37229 blocked for more than 120 seconds. [ 2213.257367] Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2213.264515] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2213.268784] task:writemany state:D stack:0 pid:37229 ppid:37222 flags:0x80000000 [ 2213.272827] Call Trace: [ 2213.273862] __schedule+0x351/0xcb0 [ 2213.275670] schedule+0xc0/0x180 [ 2213.277416] schedule_preempt_disabled+0x21/0x40 [ 2213.279884] rwsem_down_write_slowpath+0x5d7/0xa40 [ 2213.282312] ? __might_sleep+0x59/0xc0 [ 2213.283887] down_write+0x80/0xd0 [ 2213.285335] do_unlinkat+0x184/0x460 [ 2213.286783] __x64_sys_unlink+0x37/0x50 [ 2213.288399] do_syscall_64+0xc1/0x440 [ 2213.289781] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2213.291857] RIP: 0033:0x7f53424b010b [ 2213.293731] Code: Unable to access opcode bytes at RIP 0x7f53424b00e1. [ 2213.297518] RSP: 002b:00007ffff8ec31a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2213.302800] RAX: ffffffffffffffda RBX: 000000000008ebc0 RCX: 00007f53424b010b [ 2213.306790] RDX: 0000000000000038 RSI: 00007ffff8ec3610 RDI: 00007ffff8ec3210 [ 2213.312443] RBP: 00007ffff8ec3610 R08: 00007f53428481f8 R09: 00007f5342848260 [ 2213.317819] R10: 00007ffff8ec2f52 R11: 0000000000000246 R12: 0000000000000003 [ 2213.320405] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2232.052637] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2232.055331] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2232.089309] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2232.105679] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2232.113261] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2232.121768] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2232.125240] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2232.142226] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2232.149078] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2232.156044] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2232.163628] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2232.170307] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2232.172874] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2232.176262] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2232.180713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2232.183190] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2232.190850] Call Trace: [ 2232.192921] [ 2232.193793] ? show_regs.cold.9+0x22/0x2f [ 2232.198930] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2232.204064] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2232.209813] ? watchdog+0x40/0x40 [ 2232.211111] ? hrtimer_interrupt+0x128/0x2c0 [ 2232.213616] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2232.217356] ? apic_timer_interrupt+0xf/0x20 [ 2232.220090] [ 2232.221218] ? native_safe_halt+0x2e/0x50 [ 2232.223084] kvm_wait+0xcd/0xe0 [ 2232.224685] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2232.228294] ? newidle_balance.isra.93+0x207/0x580 [ 2232.230709] do_raw_spin_lock+0x177/0x190 [ 2232.232655] _raw_spin_lock+0x21/0x30 [ 2232.234237] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2232.237230] process_one_work+0x2c8/0x700 [ 2232.239323] worker_thread+0x296/0x6e0 [ 2232.241258] ? rescuer_thread+0x570/0x570 [ 2232.243270] kthread+0x1d1/0x200 [ 2232.244860] ? set_kthread_struct+0x70/0x70 [ 2232.246736] ret_from_fork+0x1f/0x30 [ 2236.059763] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 2236.060684] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 2236.068153] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2236.068238] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2236.068245] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2236.068248] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2236.068264] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2236.068268] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2236.068273] RAX: 0000000000006c86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2236.068276] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2236.068278] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2236.069255] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2236.069258] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2236.069261] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2236.069264] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2236.069266] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2236.069337] Call Trace: [ 2236.069346] [ 2236.069354] ? show_regs.cold.9+0x22/0x2f [ 2236.069371] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2236.069379] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2236.069384] ? watchdog+0x40/0x40 [ 2236.069389] ? hrtimer_interrupt+0x128/0x2c0 [ 2236.069393] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2236.069404] ? apic_timer_interrupt+0xf/0x20 [ 2236.069408] [ 2236.069410] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2236.069420] do_raw_spin_lock+0x177/0x190 [ 2236.069424] _raw_spin_lock+0x21/0x30 [ 2236.069430] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2236.069919] ? do_raw_spin_unlock+0x75/0x190 [ 2236.069925] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2236.071377] ? do_wait_intr+0xf0/0xf0 [ 2236.071389] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2236.072457] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2236.072988] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2236.074054] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2236.074182] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2236.074221] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2236.074250] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2236.074291] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2236.074380] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2236.087939] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2236.088068] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2236.088076] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2236.088078] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2236.088094] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2236.088098] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2236.088103] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2236.088106] RDX: 0000000000006ac7 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2236.088108] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2236.088110] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2236.088112] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2236.088115] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2236.088117] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2236.088119] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2236.088128] Call Trace: [ 2236.088134] [ 2236.088140] ? show_regs.cold.9+0x22/0x2f [ 2236.088150] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2236.088157] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2236.088161] ? watchdog+0x40/0x40 [ 2236.088166] ? hrtimer_interrupt+0x128/0x2c0 [ 2236.088170] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2236.088178] ? apic_timer_interrupt+0xf/0x20 [ 2236.088183] [ 2236.088185] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2236.088190] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2236.088194] do_raw_spin_lock+0x177/0x190 [ 2236.088199] _raw_spin_lock+0x21/0x30 [ 2236.088205] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2236.088634] obd_mod_free+0x46/0x140 [ptlrpc] [ 2236.088967] mdc_commit_open+0x91/0xb0 [mdc] [ 2236.089068] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2236.089354] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2236.089632] after_reply+0xe2c/0x1780 [ptlrpc] [ 2236.089916] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2236.090217] ? __next_timer_interrupt+0x160/0x160 [ 2236.090227] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2236.090511] ? do_wait_intr+0xf0/0xf0 [ 2236.090520] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2236.090898] kthread+0x1d1/0x200 [ 2236.090904] ? set_kthread_struct+0x70/0x70 [ 2236.090908] ret_from_fork+0x1f/0x30 [ 2236.611220] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2236.619254] lmv_intent_open+0x338/0x1150 [lmv] [ 2236.625934] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2236.628343] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2236.632847] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2236.641915] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2236.650070] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2236.655348] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2236.659678] ? do_raw_spin_unlock+0x75/0x190 [ 2236.666502] ? _raw_spin_unlock+0x12/0x30 [ 2236.671870] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2236.676214] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2236.681063] ll_atomic_open+0x36a/0xd70 [lustre] [ 2236.688870] lookup_open+0x356/0xb40 [ 2236.692798] do_last+0x313/0xfc0 [ 2236.694665] ? nd_jump_root+0xe5/0x160 [ 2236.699465] ? path_init+0x437/0x520 [ 2236.701351] path_openat+0xf7/0x500 [ 2236.703512] ? do_raw_spin_unlock+0x75/0x190 [ 2236.706504] do_filp_open+0x99/0x140 [ 2236.708793] ? getname_flags+0x6e/0x330 [ 2236.711399] ? __check_object_size+0xff/0x256 [ 2236.713810] ? do_raw_spin_unlock+0x75/0x190 [ 2236.716251] ? _raw_spin_unlock+0x12/0x30 [ 2236.718694] do_sys_openat2+0x2b4/0x410 [ 2236.720424] do_sys_open+0x73/0xa0 [ 2236.722168] __x64_sys_openat+0x24/0x30 [ 2236.725410] do_syscall_64+0xc1/0x440 [ 2236.727532] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2236.730835] RIP: 0033:0x7f53425a5332 [ 2236.732991] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2236.750840] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2236.754659] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2236.758878] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2236.763133] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2236.766834] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2236.775624] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2260.055853] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2260.063784] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2260.130589] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2260.141779] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2260.151803] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2260.165692] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2260.177011] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2260.200717] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2260.208104] RAX: 0000000000006186 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2260.212442] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2260.216518] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2260.222089] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2260.226443] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2260.233648] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2260.244693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2260.253024] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2260.263103] Call Trace: [ 2260.265135] [ 2260.265941] ? show_regs.cold.9+0x22/0x2f [ 2260.267449] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2260.285182] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2260.289900] ? watchdog+0x40/0x40 [ 2260.292180] ? hrtimer_interrupt+0x128/0x2c0 [ 2260.294663] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2260.297467] ? apic_timer_interrupt+0xf/0x20 [ 2260.299713] [ 2260.300695] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2260.303605] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2260.306466] ? newidle_balance.isra.93+0x207/0x580 [ 2260.309459] do_raw_spin_lock+0x177/0x190 [ 2260.313788] _raw_spin_lock+0x21/0x30 [ 2260.316025] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2260.321780] process_one_work+0x2c8/0x700 [ 2260.327761] worker_thread+0x296/0x6e0 [ 2260.330366] ? rescuer_thread+0x570/0x570 [ 2260.331780] kthread+0x1d1/0x200 [ 2260.341619] ? set_kthread_struct+0x70/0x70 [ 2260.350376] ret_from_fork+0x1f/0x30 [ 2264.057015] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 2264.057318] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 2264.060311] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2264.060396] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2264.060404] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2264.060407] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2264.060422] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2264.060426] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2264.060430] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2264.060433] RDX: 0000000000003806 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2264.060435] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2264.060438] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2264.060440] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2264.060444] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2264.060447] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2264.060449] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2264.060458] Call Trace: [ 2264.060463] [ 2264.060469] ? show_regs.cold.9+0x22/0x2f [ 2264.060478] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2264.060484] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2264.060488] ? watchdog+0x40/0x40 [ 2264.060492] ? hrtimer_interrupt+0x128/0x2c0 [ 2264.060496] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2264.060505] ? apic_timer_interrupt+0xf/0x20 [ 2264.060509] [ 2264.060513] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2264.060517] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2264.060522] do_raw_spin_lock+0x177/0x190 [ 2264.060527] _raw_spin_lock+0x21/0x30 [ 2264.060532] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2264.061070] obd_mod_free+0x46/0x140 [ptlrpc] [ 2264.061358] mdc_commit_open+0x91/0xb0 [mdc] [ 2264.061394] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2264.062133] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2264.062341] after_reply+0xe2c/0x1780 [ptlrpc] [ 2264.062551] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2264.064486] ? __next_timer_interrupt+0x160/0x160 [ 2264.064500] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2264.065021] ? do_wait_intr+0xf0/0xf0 [ 2264.065033] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2264.065244] kthread+0x1d1/0x200 [ 2264.065251] ? set_kthread_struct+0x70/0x70 [ 2264.065255] ret_from_fork+0x1f/0x30 [ 2264.448647] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2264.512617] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2264.522319] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2264.530586] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2264.535574] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2264.547464] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2264.556569] RAX: 0000000000007f07 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2264.563329] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2264.568835] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2264.574705] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2264.579392] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2264.586413] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2264.594265] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2264.601408] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2264.614684] Call Trace: [ 2264.619027] [ 2264.620162] ? show_regs.cold.9+0x22/0x2f [ 2264.622663] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2264.631346] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2264.637772] ? watchdog+0x40/0x40 [ 2264.645488] ? hrtimer_interrupt+0x128/0x2c0 [ 2264.652797] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2264.658730] ? apic_timer_interrupt+0xf/0x20 [ 2264.662681] [ 2264.666475] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2264.680957] do_raw_spin_lock+0x177/0x190 [ 2264.685419] _raw_spin_lock+0x21/0x30 [ 2264.691503] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2264.697882] ? do_raw_spin_unlock+0x75/0x190 [ 2264.700177] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2264.703778] ? do_wait_intr+0xf0/0xf0 [ 2264.705655] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2264.708332] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2264.710957] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2264.719343] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2264.721354] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2264.724437] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2264.726527] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2264.727989] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2264.730282] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2264.732937] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2264.735147] lmv_intent_open+0x338/0x1150 [lmv] [ 2264.736889] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2264.739334] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2264.741432] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2264.743427] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2264.745382] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2264.747705] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2264.749499] ? do_raw_spin_unlock+0x75/0x190 [ 2264.750910] ? _raw_spin_unlock+0x12/0x30 [ 2264.752691] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2264.753824] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2264.754899] ll_atomic_open+0x36a/0xd70 [lustre] [ 2264.759224] lookup_open+0x356/0xb40 [ 2264.760954] do_last+0x313/0xfc0 [ 2264.762935] ? nd_jump_root+0xe5/0x160 [ 2264.764663] ? path_init+0x437/0x520 [ 2264.766221] path_openat+0xf7/0x500 [ 2264.767683] ? do_raw_spin_unlock+0x75/0x190 [ 2264.769370] do_filp_open+0x99/0x140 [ 2264.770367] ? getname_flags+0x6e/0x330 [ 2264.771359] ? __check_object_size+0xff/0x256 [ 2264.772497] ? do_raw_spin_unlock+0x75/0x190 [ 2264.774451] ? _raw_spin_unlock+0x12/0x30 [ 2264.776224] do_sys_openat2+0x2b4/0x410 [ 2264.777454] do_sys_open+0x73/0xa0 [ 2264.778588] __x64_sys_openat+0x24/0x30 [ 2264.779587] do_syscall_64+0xc1/0x440 [ 2264.780573] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2264.782281] RIP: 0033:0x7f53425a5332 [ 2264.783130] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2264.792604] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2264.799673] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2264.802537] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2264.804635] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2264.806587] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2264.809526] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2288.052036] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2288.054667] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2288.075610] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2288.083115] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2288.089658] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2288.091682] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2288.093522] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2288.102634] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2288.108572] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2288.117644] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2288.121840] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2288.125506] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2288.128886] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2288.132646] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2288.136795] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2288.139937] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2288.144070] Call Trace: [ 2288.148081] [ 2288.148743] ? show_regs.cold.9+0x22/0x2f [ 2288.150544] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2288.152797] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2288.161503] ? watchdog+0x40/0x40 [ 2288.164644] ? hrtimer_interrupt+0x128/0x2c0 [ 2288.167047] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2288.169464] ? apic_timer_interrupt+0xf/0x20 [ 2288.171142] [ 2288.172160] ? native_safe_halt+0x2e/0x50 [ 2288.175109] kvm_wait+0xcd/0xe0 [ 2288.177167] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2288.180455] ? newidle_balance.isra.93+0x207/0x580 [ 2288.184989] do_raw_spin_lock+0x177/0x190 [ 2288.188018] _raw_spin_lock+0x21/0x30 [ 2288.191509] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2288.197636] process_one_work+0x2c8/0x700 [ 2288.200494] worker_thread+0x296/0x6e0 [ 2288.203415] ? rescuer_thread+0x570/0x570 [ 2288.206060] kthread+0x1d1/0x200 [ 2288.208010] ? set_kthread_struct+0x70/0x70 [ 2288.210720] ret_from_fork+0x1f/0x30 [ 2292.055007] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 2292.063246] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 2292.071218] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2292.071325] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2292.071337] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2292.071340] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2292.071358] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2292.071366] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2292.071370] RAX: 0000000000001b39 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2292.071373] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2292.071374] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2292.071376] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2292.071378] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2292.071381] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2292.071383] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2292.071389] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2292.071397] Call Trace: [ 2292.071407] [ 2292.071415] ? show_regs.cold.9+0x22/0x2f [ 2292.071426] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2292.071436] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2292.071440] ? watchdog+0x40/0x40 [ 2292.071444] ? hrtimer_interrupt+0x128/0x2c0 [ 2292.071448] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2292.071456] ? apic_timer_interrupt+0xf/0x20 [ 2292.071461] [ 2292.071464] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2292.071470] do_raw_spin_lock+0x177/0x190 [ 2292.071476] _raw_spin_lock+0x21/0x30 [ 2292.071480] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2292.071901] ? do_raw_spin_unlock+0x75/0x190 [ 2292.071907] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2292.072251] ? do_wait_intr+0xf0/0xf0 [ 2292.072267] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2292.072605] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2292.073113] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2292.084215] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2292.085058] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2292.085068] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2292.085072] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2292.085092] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2292.085097] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2292.085101] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2292.085288] RDX: 0000000000005bc9 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2292.085419] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2292.085422] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2292.085564] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2292.085568] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2292.085571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2292.085577] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2292.085584] Call Trace: [ 2292.085590] [ 2292.085652] ? show_regs.cold.9+0x22/0x2f [ 2292.086152] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2292.167409] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2292.168093] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2292.168133] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2292.168179] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2292.168218] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2292.168590] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2292.171838] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2292.171885] lmv_intent_open+0x338/0x1150 [lmv] [ 2292.171926] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2292.172059] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2292.172140] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2292.172163] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2292.172247] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2292.172344] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2292.173736] ? do_raw_spin_unlock+0x75/0x190 [ 2292.173752] ? _raw_spin_unlock+0x12/0x30 [ 2292.173760] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2292.173787] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2292.173809] ll_atomic_open+0x36a/0xd70 [lustre] [ 2292.173930] lookup_open+0x356/0xb40 [ 2292.173939] do_last+0x313/0xfc0 [ 2292.173944] ? nd_jump_root+0xe5/0x160 [ 2292.173949] ? path_init+0x437/0x520 [ 2292.173954] path_openat+0xf7/0x500 [ 2292.173959] ? do_raw_spin_unlock+0x75/0x190 [ 2292.173963] do_filp_open+0x99/0x140 [ 2292.173970] ? getname_flags+0x6e/0x330 [ 2292.173975] ? __check_object_size+0xff/0x256 [ 2292.173982] ? do_raw_spin_unlock+0x75/0x190 [ 2292.173987] ? _raw_spin_unlock+0x12/0x30 [ 2292.173992] do_sys_openat2+0x2b4/0x410 [ 2292.173996] do_sys_open+0x73/0xa0 [ 2292.174001] __x64_sys_openat+0x24/0x30 [ 2292.174004] do_syscall_64+0xc1/0x440 [ 2292.174011] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2292.174018] RIP: 0033:0x7f53425a5332 [ 2292.174026] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2292.174031] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2292.174038] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2292.174041] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2292.174043] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2292.174046] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2292.174048] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2292.937402] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2292.940254] ? watchdog+0x40/0x40 [ 2292.944419] ? hrtimer_interrupt+0x128/0x2c0 [ 2292.949282] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2292.950943] ? apic_timer_interrupt+0xf/0x20 [ 2292.955068] [ 2292.955838] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2292.957561] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2292.966182] do_raw_spin_lock+0x177/0x190 [ 2292.969475] _raw_spin_lock+0x21/0x30 [ 2292.973149] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2292.976361] obd_mod_free+0x46/0x140 [ptlrpc] [ 2292.981041] mdc_commit_open+0x91/0xb0 [mdc] [ 2292.985012] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2292.988565] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2292.990703] after_reply+0xe2c/0x1780 [ptlrpc] [ 2292.998453] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2293.000142] ? __next_timer_interrupt+0x160/0x160 [ 2293.002840] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2293.005087] ? do_wait_intr+0xf0/0xf0 [ 2293.007158] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2293.020034] kthread+0x1d1/0x200 [ 2293.021219] ? set_kthread_struct+0x70/0x70 [ 2293.024562] ret_from_fork+0x1f/0x30 [ 2316.052032] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2316.058544] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2316.105877] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2316.114806] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2316.123146] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2316.133653] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2316.139150] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2316.159979] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2316.166140] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2316.174360] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2316.186813] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2316.193424] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2316.200777] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2316.207180] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2316.214806] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2316.220644] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2316.230665] Call Trace: [ 2316.234129] [ 2316.234686] ? show_regs.cold.9+0x22/0x2f [ 2316.235898] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2316.238690] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2316.243066] ? watchdog+0x40/0x40 [ 2316.247969] ? hrtimer_interrupt+0x128/0x2c0 [ 2316.250824] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2316.252787] ? apic_timer_interrupt+0xf/0x20 [ 2316.255824] [ 2316.257113] ? native_safe_halt+0x2e/0x50 [ 2316.260048] kvm_wait+0xcd/0xe0 [ 2316.262153] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2316.266291] ? newidle_balance.isra.93+0x207/0x580 [ 2316.269164] do_raw_spin_lock+0x177/0x190 [ 2316.273738] _raw_spin_lock+0x21/0x30 [ 2316.276633] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2316.279402] process_one_work+0x2c8/0x700 [ 2316.287145] worker_thread+0x296/0x6e0 [ 2316.288954] ? rescuer_thread+0x570/0x570 [ 2316.290693] kthread+0x1d1/0x200 [ 2316.291901] ? set_kthread_struct+0x70/0x70 [ 2316.293840] ret_from_fork+0x1f/0x30 [ 2320.055047] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 2320.057341] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 2320.058260] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2320.058336] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2320.058343] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2320.058346] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2320.058361] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2320.058365] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2320.058369] RAX: 000000000000201c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2320.058372] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2320.058374] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2320.058376] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2320.058379] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2320.058382] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2320.058384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2320.058386] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2320.058396] Call Trace: [ 2320.058402] [ 2320.058407] ? show_regs.cold.9+0x22/0x2f [ 2320.058416] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2320.058422] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2320.058426] ? watchdog+0x40/0x40 [ 2320.058430] ? hrtimer_interrupt+0x128/0x2c0 [ 2320.058434] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2320.058442] ? apic_timer_interrupt+0xf/0x20 [ 2320.058447] [ 2320.058450] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2320.058456] do_raw_spin_lock+0x177/0x190 [ 2320.058461] _raw_spin_lock+0x21/0x30 [ 2320.058466] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2320.058764] ? do_raw_spin_unlock+0x75/0x190 [ 2320.058771] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2320.058980] ? do_wait_intr+0xf0/0xf0 [ 2320.059176] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2320.059381] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2320.059599] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2320.059828] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2320.060087] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2320.060125] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2320.060152] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2320.060179] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2320.060259] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2320.060482] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2320.060512] lmv_intent_open+0x338/0x1150 [lmv] [ 2320.060543] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2320.060629] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2320.060710] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2320.060733] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2320.060833] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2320.060922] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2320.061077] ? do_raw_spin_unlock+0x75/0x190 [ 2320.061084] ? _raw_spin_unlock+0x12/0x30 [ 2320.061090] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2320.061116] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2320.061135] ll_atomic_open+0x36a/0xd70 [lustre] [ 2320.061219] lookup_open+0x356/0xb40 [ 2320.061227] do_last+0x313/0xfc0 [ 2320.061232] ? nd_jump_root+0xe5/0x160 [ 2320.061236] ? path_init+0x437/0x520 [ 2320.061241] path_openat+0xf7/0x500 [ 2320.061246] ? do_raw_spin_unlock+0x75/0x190 [ 2320.061251] do_filp_open+0x99/0x140 [ 2320.061256] ? getname_flags+0x6e/0x330 [ 2320.061261] ? __check_object_size+0xff/0x256 [ 2320.061269] ? do_raw_spin_unlock+0x75/0x190 [ 2320.061273] ? _raw_spin_unlock+0x12/0x30 [ 2320.061277] do_sys_openat2+0x2b4/0x410 [ 2320.061282] do_sys_open+0x73/0xa0 [ 2320.061285] __x64_sys_openat+0x24/0x30 [ 2320.061289] do_syscall_64+0xc1/0x440 [ 2320.061296] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2320.061303] RIP: 0033:0x7f53425a5332 [ 2320.061310] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2320.061314] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2320.061319] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2320.061321] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2320.061324] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2320.061326] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2320.061328] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2320.390352] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2320.417674] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2320.423383] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2320.429066] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2320.432185] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2320.442137] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2320.446617] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2320.449421] RDX: 0000000000000625 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2320.453014] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2320.466132] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2320.472598] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2320.478073] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2320.485371] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2320.490834] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2320.494160] Call Trace: [ 2320.495396] [ 2320.496533] ? show_regs.cold.9+0x22/0x2f [ 2320.498180] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2320.500097] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2320.501423] ? watchdog+0x40/0x40 [ 2320.502496] ? hrtimer_interrupt+0x128/0x2c0 [ 2320.504463] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2320.507046] ? apic_timer_interrupt+0xf/0x20 [ 2320.508612] [ 2320.509754] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2320.512151] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2320.514010] do_raw_spin_lock+0x177/0x190 [ 2320.515357] _raw_spin_lock+0x21/0x30 [ 2320.517119] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2320.519016] obd_mod_free+0x46/0x140 [ptlrpc] [ 2320.521433] mdc_commit_open+0x91/0xb0 [mdc] [ 2320.523561] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2320.526939] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2320.532116] after_reply+0xe2c/0x1780 [ptlrpc] [ 2320.534280] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2320.536549] ? __next_timer_interrupt+0x160/0x160 [ 2320.538479] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2320.540508] ? do_wait_intr+0xf0/0xf0 [ 2320.544782] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2320.548222] kthread+0x1d1/0x200 [ 2320.558432] ? set_kthread_struct+0x70/0x70 [ 2320.561797] ret_from_fork+0x1f/0x30 [ 2334.431360] rcu: INFO: rcu_sched self-detected stall on CPU [ 2334.437906] rcu: 2-....: (127772 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=42208 [ 2334.447773] (t=240005 jiffies g=274185 q=61675) [ 2334.454316] Sending NMI from CPU 2 to CPUs 1: [ 2334.461137] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 1.414 msecs [ 2334.463752] NMI backtrace for cpu 1 [ 2334.463771] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2334.463774] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2334.463775] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2334.463777] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2334.463779] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [ 2334.463782] RAX: 000000000000646c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2334.463784] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2334.463785] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2334.463787] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2334.463788] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2334.463789] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2334.463791] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2334.463792] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2334.463793] Call Trace: [ 2334.463794] [ 2334.463795] ? show_regs.cold.9+0x22/0x2f [ 2334.463796] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2334.463797] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2334.463798] ? nmi_handle+0xbd/0x210 [ 2334.463799] ? default_do_nmi+0x4b/0x1b0 [ 2334.463800] ? do_nmi+0x280/0x380 [ 2334.463801] ? end_repeat_nmi+0x16/0x1f [ 2334.463802] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2334.463804] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2334.463805] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2334.463806] [ 2334.463807] do_raw_spin_lock+0x177/0x190 [ 2334.463808] _raw_spin_lock+0x21/0x30 [ 2334.463809] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2334.463810] ? do_raw_spin_unlock+0x75/0x190 [ 2334.463811] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2334.463812] ? do_wait_intr+0xf0/0xf0 [ 2334.463813] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2334.463814] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2334.463815] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2334.463816] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2334.463817] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2334.463818] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2334.463819] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2334.463820] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2334.463821] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2334.463822] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2334.463824] lmv_intent_open+0x338/0x1150 [lmv] [ 2334.463825] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2334.463826] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2334.463827] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2334.463828] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2334.463829] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2334.463830] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2334.463831] ? do_raw_spin_unlock+0x75/0x190 [ 2334.463833] ? _raw_spin_unlock+0x12/0x30 [ 2334.463834] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2334.463835] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2334.463836] ll_atomic_open+0x36a/0xd70 [lustre] [ 2334.463837] lookup_open+0x356/0xb40 [ 2334.463838] do_last+0x313/0xfc0 [ 2334.463839] ? nd_jump_root+0xe5/0x160 [ 2334.463840] ? path_init+0x437/0x520 [ 2334.463841] path_openat+0xf7/0x500 [ 2334.463842] ? do_raw_spin_unlock+0x75/0x190 [ 2334.463843] do_filp_open+0x99/0x140 [ 2334.463844] ? getname_flags+0x6e/0x330 [ 2334.463845] ? __check_object_size+0xff/0x256 [ 2334.463846] ? do_raw_spin_unlock+0x75/0x190 [ 2334.463847] ? _raw_spin_unlock+0x12/0x30 [ 2334.463848] do_sys_openat2+0x2b4/0x410 [ 2334.463850] do_sys_open+0x73/0xa0 [ 2334.463851] __x64_sys_openat+0x24/0x30 [ 2334.463852] do_syscall_64+0xc1/0x440 [ 2334.463853] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2334.463854] RIP: 0033:0x7f53425a5332 [ 2334.463856] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2334.463858] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2334.463861] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2334.463863] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2334.463864] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2334.463865] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2334.463867] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2334.463871] NMI backtrace for cpu 2 [ 2334.819996] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2334.836110] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2334.845293] Call Trace: [ 2334.847389] [ 2334.849402] ? dump_stack+0xbb/0x10e [ 2334.852547] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2334.857683] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2334.861422] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2334.864192] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2334.871934] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2334.873743] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2334.876990] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2334.880415] ? update_process_times+0x6d/0xc0 [ 2334.887204] ? tick_sched_handle.isra.18+0x27/0x90 [ 2334.889918] ? tick_sched_timer+0x53/0xb0 [ 2334.895356] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2334.898923] ? tick_sched_handle.isra.18+0x90/0x90 [ 2334.905777] ? hrtimer_interrupt+0x128/0x2c0 [ 2334.912546] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2334.927952] ? apic_timer_interrupt+0xf/0x20 [ 2334.935346] [ 2334.936111] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2334.939148] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2334.941937] ? do_raw_spin_lock+0x177/0x190 [ 2334.944324] ? _raw_spin_lock+0x21/0x30 [ 2334.946184] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2334.949211] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2334.952398] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2334.954171] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2334.956388] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2334.958804] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2334.960936] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2334.963390] ? __next_timer_interrupt+0x160/0x160 [ 2334.965220] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2334.967208] ? do_wait_intr+0xf0/0xf0 [ 2334.968828] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2334.972366] ? kthread+0x1d1/0x200 [ 2334.973792] ? set_kthread_struct+0x70/0x70 [ 2334.975523] ? ret_from_fork+0x1f/0x30 [ 2335.712093] INFO: task khugepaged:48 blocked for more than 120 seconds. [ 2335.716531] Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2335.725396] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2335.732551] task:khugepaged state:D stack:0 pid:48 ppid:2 flags:0x80004000 [ 2335.742911] Call Trace: [ 2335.744065] __schedule+0x351/0xcb0 [ 2335.747264] ? wait_for_completion+0xae/0x1e0 [ 2335.752194] schedule+0xc0/0x180 [ 2335.758367] schedule_timeout+0x126/0x190 [ 2335.766906] ? try_to_wake_up+0x80/0x710 [ 2335.779618] ? __prepare_to_swait+0x5b/0x90 [ 2335.785454] ? do_raw_spin_unlock+0x75/0x190 [ 2335.798325] wait_for_completion+0xf0/0x1e0 [ 2335.808777] flush_work+0x1f3/0x390 [ 2335.813376] ? flush_workqueue_prep_pwqs+0x200/0x200 [ 2335.820645] ? do_raw_spin_unlock+0x75/0x190 [ 2335.833133] ? __pagevec_release+0x50/0x50 [ 2335.837690] lru_add_drain_all+0x1d4/0x280 [ 2335.847306] khugepaged+0xa2/0x16f0 [ 2335.848552] ? woken_wake_function+0x30/0x30 [ 2335.857041] ? collapse_pte_mapped_thp+0x8a0/0x8a0 [ 2335.863046] kthread+0x1d1/0x200 [ 2335.868301] ? set_kthread_struct+0x70/0x70 [ 2335.871469] ret_from_fork+0x1f/0x30 [ 2335.884451] INFO: task writemany:37225 blocked for more than 120 seconds. [ 2335.897050] Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2335.903057] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2335.911348] task:writemany state:D stack:0 pid:37225 ppid:37222 flags:0x80004004 [ 2335.917865] Call Trace: [ 2335.923104] __schedule+0x351/0xcb0 [ 2335.925984] schedule+0xc0/0x180 [ 2335.931241] schedule_preempt_disabled+0x21/0x40 [ 2335.933284] rwsem_down_write_slowpath+0x5d7/0xa40 [ 2335.937964] ? lprocfs_counter_add+0x15b/0x210 [obdclass] [ 2335.944255] down_write+0x80/0xd0 [ 2335.947982] do_last+0x2eb/0xfc0 [ 2335.949115] ? nd_jump_root+0xe5/0x160 [ 2335.953187] ? path_init+0x437/0x520 [ 2335.955300] path_openat+0xf7/0x500 [ 2335.956361] ? do_raw_spin_unlock+0x75/0x190 [ 2335.961650] do_filp_open+0x99/0x140 [ 2335.965205] ? getname_flags+0x6e/0x330 [ 2335.969499] ? __check_object_size+0xff/0x256 [ 2335.971419] ? do_raw_spin_unlock+0x75/0x190 [ 2335.978183] ? _raw_spin_unlock+0x12/0x30 [ 2335.986057] do_sys_openat2+0x2b4/0x410 [ 2335.987442] do_sys_open+0x73/0xa0 [ 2335.992523] __x64_sys_openat+0x24/0x30 [ 2335.995358] do_syscall_64+0xc1/0x440 [ 2336.004409] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2336.008341] RIP: 0033:0x7f53425a5332 [ 2336.018290] Code: Unable to access opcode bytes at RIP 0x7f53425a5308. [ 2336.020838] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2336.023873] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f53425a5332 [ 2336.026404] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2336.029081] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2336.031565] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2336.033833] R13: 00000000ffffffff R14: 0000000000000014 R15: 0000000000000000 [ 2336.038321] INFO: task writemany:37226 blocked for more than 120 seconds. [ 2336.040896] Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2336.043590] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2336.045674] task:writemany state:D stack:0 pid:37226 ppid:37222 flags:0x80004004 [ 2336.049873] Call Trace: [ 2336.051021] __schedule+0x351/0xcb0 [ 2336.052633] schedule+0xc0/0x180 [ 2336.053858] schedule_preempt_disabled+0x21/0x40 [ 2336.055507] rwsem_down_write_slowpath+0x5d7/0xa40 [ 2336.057247] ? __might_sleep+0x59/0xc0 [ 2336.058700] down_write+0x80/0xd0 [ 2336.059920] do_unlinkat+0x184/0x460 [ 2336.062347] __x64_sys_unlink+0x37/0x50 [ 2336.064135] do_syscall_64+0xc1/0x440 [ 2336.065224] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2336.067083] RIP: 0033:0x7f53424b010b [ 2336.068498] Code: Unable to access opcode bytes at RIP 0x7f53424b00e1. [ 2336.070554] RSP: 002b:00007ffff8ec31a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2336.072896] RAX: ffffffffffffffda RBX: 00000000000015e0 RCX: 00007f53424b010b [ 2336.075211] RDX: 0000000000000038 RSI: 00007ffff8ec3610 RDI: 00007ffff8ec3210 [ 2336.078188] RBP: 00007ffff8ec3610 R08: 00007f5342848238 R09: 00007f5342848260 [ 2336.082608] R10: 00007ffff8ec2f52 R11: 0000000000000246 R12: 0000000000000003 [ 2336.085776] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2336.089393] INFO: task writemany:37228 blocked for more than 120 seconds. [ 2336.092443] Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2336.095272] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2336.098160] task:writemany state:D stack:0 pid:37228 ppid:37222 flags:0x80000004 [ 2336.102342] Call Trace: [ 2336.103408] __schedule+0x351/0xcb0 [ 2336.104964] schedule+0xc0/0x180 [ 2336.107377] schedule_preempt_disabled+0x21/0x40 [ 2336.109713] rwsem_down_write_slowpath+0x5d7/0xa40 [ 2336.111314] ? __might_sleep+0x59/0xc0 [ 2336.118228] down_write+0x80/0xd0 [ 2336.119437] do_unlinkat+0x184/0x460 [ 2336.126313] __x64_sys_unlink+0x37/0x50 [ 2336.132013] do_syscall_64+0xc1/0x440 [ 2336.133986] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2336.136493] RIP: 0033:0x7f53424b010b [ 2336.139264] Code: Unable to access opcode bytes at RIP 0x7f53424b00e1. [ 2336.142575] RSP: 002b:00007ffff8ec31a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2336.150301] RAX: ffffffffffffffda RBX: 000000000002c258 RCX: 00007f53424b010b [ 2336.154186] RDX: 0000000000000038 RSI: 00007ffff8ec3610 RDI: 00007ffff8ec3210 [ 2336.157348] RBP: 00007ffff8ec3610 R08: 00007f534284821c R09: 00007f5342848260 [ 2336.160363] R10: 00007ffff8ec2f52 R11: 0000000000000246 R12: 0000000000000003 [ 2336.163134] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2336.185088] INFO: task writemany:37229 blocked for more than 120 seconds. [ 2336.192014] Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2336.198366] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2336.204818] task:writemany state:D stack:0 pid:37229 ppid:37222 flags:0x80000004 [ 2336.209978] Call Trace: [ 2336.210958] __schedule+0x351/0xcb0 [ 2336.213089] schedule+0xc0/0x180 [ 2336.215008] schedule_preempt_disabled+0x21/0x40 [ 2336.217239] rwsem_down_write_slowpath+0x5d7/0xa40 [ 2336.220113] ? __might_sleep+0x59/0xc0 [ 2336.222269] down_write+0x80/0xd0 [ 2336.224131] do_unlinkat+0x184/0x460 [ 2336.226301] __x64_sys_unlink+0x37/0x50 [ 2336.229256] do_syscall_64+0xc1/0x440 [ 2336.231162] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2336.234136] RIP: 0033:0x7f53424b010b [ 2336.235703] Code: Unable to access opcode bytes at RIP 0x7f53424b00e1. [ 2336.239116] RSP: 002b:00007ffff8ec31a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 2336.243601] RAX: ffffffffffffffda RBX: 000000000008ebc0 RCX: 00007f53424b010b [ 2336.248139] RDX: 0000000000000038 RSI: 00007ffff8ec3610 RDI: 00007ffff8ec3210 [ 2336.252179] RBP: 00007ffff8ec3610 R08: 00007f53428481f8 R09: 00007f5342848260 [ 2336.256344] R10: 00007ffff8ec2f52 R11: 0000000000000246 R12: 0000000000000003 [ 2336.260053] R13: 00000000ffffffff R14: 0000000000000038 R15: 0000000000000000 [ 2344.052250] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 2344.058952] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2344.093219] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2344.104986] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2344.112640] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2344.117064] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2344.119186] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2344.132617] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2344.136932] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2344.139153] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2344.142383] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2344.146989] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2344.150750] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2344.154787] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2344.159888] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2344.162746] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2344.166344] Call Trace: [ 2344.168475] [ 2344.169356] ? show_regs.cold.9+0x22/0x2f [ 2344.171767] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2344.175224] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2344.179226] ? watchdog+0x40/0x40 [ 2344.181300] ? hrtimer_interrupt+0x128/0x2c0 [ 2344.184288] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2344.188247] ? apic_timer_interrupt+0xf/0x20 [ 2344.194871] [ 2344.196223] ? native_safe_halt+0x2e/0x50 [ 2344.198227] kvm_wait+0xcd/0xe0 [ 2344.199307] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2344.203016] ? newidle_balance.isra.93+0x207/0x580 [ 2344.214234] do_raw_spin_lock+0x177/0x190 [ 2344.218372] _raw_spin_lock+0x21/0x30 [ 2344.221282] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2344.231064] process_one_work+0x2c8/0x700 [ 2344.235292] worker_thread+0x296/0x6e0 [ 2344.239176] ? rescuer_thread+0x570/0x570 [ 2344.240684] kthread+0x1d1/0x200 [ 2344.242905] ? set_kthread_struct+0x70/0x70 [ 2344.245056] ret_from_fork+0x1f/0x30 [ 2360.055193] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 2360.060989] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 2360.063059] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2360.063543] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2360.063553] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2360.063557] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2360.063573] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2360.063576] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2360.063581] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2360.063583] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2360.063585] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2360.063588] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2360.063590] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2360.063593] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2360.063597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2360.063599] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2360.063607] Call Trace: [ 2360.063613] [ 2360.063620] ? show_regs.cold.9+0x22/0x2f [ 2360.063629] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2360.063636] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2360.063640] ? watchdog+0x40/0x40 [ 2360.063643] ? hrtimer_interrupt+0x128/0x2c0 [ 2360.063647] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2360.063653] ? apic_timer_interrupt+0xf/0x20 [ 2360.063657] [ 2360.063660] ? native_safe_halt+0x2e/0x50 [ 2360.063663] kvm_wait+0xcd/0xe0 [ 2360.063670] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2360.063678] do_raw_spin_lock+0x177/0x190 [ 2360.063682] _raw_spin_lock+0x21/0x30 [ 2360.063686] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2360.069682] Modules linked in: lustre(O) [ 2360.123607] ? do_raw_spin_unlock+0x75/0x190 [ 2360.123630] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2360.123907] ? do_wait_intr+0xf0/0xf0 [ 2360.123917] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2360.124318] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2360.129767] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2360.129846] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2360.129853] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2360.129856] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2360.129870] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2360.129874] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2360.129879] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2360.129881] RDX: 000000000000402c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2360.129884] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2360.129886] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2360.129889] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2360.129892] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2360.129895] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2360.129898] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2360.129905] Call Trace: [ 2360.129910] [ 2360.129915] ? show_regs.cold.9+0x22/0x2f [ 2360.129924] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2360.129930] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2360.129934] ? watchdog+0x40/0x40 [ 2360.129938] ? hrtimer_interrupt+0x128/0x2c0 [ 2360.129942] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2360.129950] ? apic_timer_interrupt+0xf/0x20 [ 2360.129954] [ 2360.129957] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2360.129962] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2360.129967] do_raw_spin_lock+0x177/0x190 [ 2360.129971] _raw_spin_lock+0x21/0x30 [ 2360.129977] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2360.140605] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2360.148354] obd_mod_free+0x46/0x140 [ptlrpc] [ 2360.167514] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2360.167639] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2360.167680] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2360.167707] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2360.167733] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2360.171956] mdc_commit_open+0x91/0xb0 [mdc] [ 2360.172028] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2360.172444] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2360.172659] after_reply+0xe2c/0x1780 [ptlrpc] [ 2360.172885] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2360.173509] ? __next_timer_interrupt+0x160/0x160 [ 2360.173522] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2360.173730] ? do_wait_intr+0xf0/0xf0 [ 2360.173739] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2360.173943] kthread+0x1d1/0x200 [ 2360.173948] ? set_kthread_struct+0x70/0x70 [ 2360.173952] ret_from_fork+0x1f/0x30 [ 2360.523782] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2360.526995] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2360.533060] lmv_intent_open+0x338/0x1150 [lmv] [ 2360.536232] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2360.539781] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2360.543918] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2360.547280] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2360.551226] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2360.554721] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2360.557691] ? do_raw_spin_unlock+0x75/0x190 [ 2360.560028] ? _raw_spin_unlock+0x12/0x30 [ 2360.563382] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2360.566149] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2360.568814] ll_atomic_open+0x36a/0xd70 [lustre] [ 2360.572698] lookup_open+0x356/0xb40 [ 2360.574258] do_last+0x313/0xfc0 [ 2360.576000] ? nd_jump_root+0xe5/0x160 [ 2360.577948] ? path_init+0x437/0x520 [ 2360.579885] path_openat+0xf7/0x500 [ 2360.584030] ? do_raw_spin_unlock+0x75/0x190 [ 2360.585537] do_filp_open+0x99/0x140 [ 2360.589702] ? getname_flags+0x6e/0x330 [ 2360.597583] ? __check_object_size+0xff/0x256 [ 2360.601878] ? do_raw_spin_unlock+0x75/0x190 [ 2360.605989] ? _raw_spin_unlock+0x12/0x30 [ 2360.609727] do_sys_openat2+0x2b4/0x410 [ 2360.611968] do_sys_open+0x73/0xa0 [ 2360.613863] __x64_sys_openat+0x24/0x30 [ 2360.615837] do_syscall_64+0xc1/0x440 [ 2360.617826] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2360.620792] RIP: 0033:0x7f53425a5332 [ 2360.623809] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2360.655613] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2360.662820] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2360.668933] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2360.671165] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2360.672915] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2360.681965] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2372.052026] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 2372.056619] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2372.121510] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2372.127759] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2372.134232] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2372.145794] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2372.148659] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2372.162272] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2372.173389] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2372.179030] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2372.184869] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2372.188804] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2372.198194] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2372.205852] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2372.214645] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2372.221720] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2372.228265] Call Trace: [ 2372.233526] [ 2372.235344] ? show_regs.cold.9+0x22/0x2f [ 2372.237224] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2372.240484] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2372.245263] ? watchdog+0x40/0x40 [ 2372.248034] ? hrtimer_interrupt+0x128/0x2c0 [ 2372.251538] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2372.254848] ? apic_timer_interrupt+0xf/0x20 [ 2372.256072] [ 2372.256551] ? native_safe_halt+0x2e/0x50 [ 2372.261690] kvm_wait+0xcd/0xe0 [ 2372.265671] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2372.269888] ? newidle_balance.isra.93+0x207/0x580 [ 2372.272062] do_raw_spin_lock+0x177/0x190 [ 2372.273376] _raw_spin_lock+0x21/0x30 [ 2372.274410] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2372.276413] process_one_work+0x2c8/0x700 [ 2372.278163] worker_thread+0x296/0x6e0 [ 2372.279229] ? rescuer_thread+0x570/0x570 [ 2372.280619] kthread+0x1d1/0x200 [ 2372.281746] ? set_kthread_struct+0x70/0x70 [ 2372.283238] ret_from_fork+0x1f/0x30 [ 2388.057046] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 2388.058277] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 2388.059873] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2388.059966] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2388.059976] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2388.059979] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2388.060001] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2388.060006] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2388.060011] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2388.060013] RDX: 0000000000002778 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2388.060016] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2388.060018] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2388.060021] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2388.060025] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2388.060028] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2388.060033] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2388.060043] Call Trace: [ 2388.060053] [ 2388.060060] ? show_regs.cold.9+0x22/0x2f [ 2388.060074] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2388.060082] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2388.060087] ? watchdog+0x40/0x40 [ 2388.060091] ? hrtimer_interrupt+0x128/0x2c0 [ 2388.060096] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2388.060106] ? apic_timer_interrupt+0xf/0x20 [ 2388.060110] [ 2388.060113] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2388.060119] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2388.060124] do_raw_spin_lock+0x177/0x190 [ 2388.060129] _raw_spin_lock+0x21/0x30 [ 2388.060136] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2388.060500] obd_mod_free+0x46/0x140 [ptlrpc] [ 2388.061803] mdc_commit_open+0x91/0xb0 [mdc] [ 2388.062046] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2388.062808] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2388.063323] after_reply+0xe2c/0x1780 [ptlrpc] [ 2388.063706] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2388.064173] ? __next_timer_interrupt+0x160/0x160 [ 2388.064186] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2388.064449] ? do_wait_intr+0xf0/0xf0 [ 2388.064461] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2388.064738] kthread+0x1d1/0x200 [ 2388.064745] ? set_kthread_struct+0x70/0x70 [ 2388.064748] ret_from_fork+0x1f/0x30 [ 2388.209124] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2388.226606] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2388.229483] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2388.232360] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2388.234807] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2388.242108] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2388.245143] RAX: 0000000000007f86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2388.247974] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2388.250570] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2388.252588] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2388.254501] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2388.256600] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2388.259170] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2388.260675] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2388.262684] Call Trace: [ 2388.263326] [ 2388.263992] ? show_regs.cold.9+0x22/0x2f [ 2388.265317] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2388.267142] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2388.269069] ? watchdog+0x40/0x40 [ 2388.270164] ? hrtimer_interrupt+0x128/0x2c0 [ 2388.271390] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2388.273113] ? apic_timer_interrupt+0xf/0x20 [ 2388.274318] [ 2388.275019] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2388.276800] do_raw_spin_lock+0x177/0x190 [ 2388.277998] _raw_spin_lock+0x21/0x30 [ 2388.279240] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2388.281434] ? do_raw_spin_unlock+0x75/0x190 [ 2388.282891] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2388.284379] ? do_wait_intr+0xf0/0xf0 [ 2388.285458] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2388.287200] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2388.288609] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2388.290525] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2388.292298] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2388.293714] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2388.294844] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2388.295916] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2388.297236] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2388.298785] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2388.300371] lmv_intent_open+0x338/0x1150 [lmv] [ 2388.301487] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2388.302929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2388.304213] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2388.305298] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2388.306640] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2388.308188] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2388.309455] ? do_raw_spin_unlock+0x75/0x190 [ 2388.310404] ? _raw_spin_unlock+0x12/0x30 [ 2388.312908] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2388.314288] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2388.316911] ll_atomic_open+0x36a/0xd70 [lustre] [ 2388.318157] lookup_open+0x356/0xb40 [ 2388.319211] do_last+0x313/0xfc0 [ 2388.320012] ? nd_jump_root+0xe5/0x160 [ 2388.320893] ? path_init+0x437/0x520 [ 2388.322154] path_openat+0xf7/0x500 [ 2388.323406] ? do_raw_spin_unlock+0x75/0x190 [ 2388.325025] do_filp_open+0x99/0x140 [ 2388.326608] ? getname_flags+0x6e/0x330 [ 2388.328180] ? __check_object_size+0xff/0x256 [ 2388.329824] ? do_raw_spin_unlock+0x75/0x190 [ 2388.332592] ? _raw_spin_unlock+0x12/0x30 [ 2388.337191] do_sys_openat2+0x2b4/0x410 [ 2388.343159] do_sys_open+0x73/0xa0 [ 2388.346297] __x64_sys_openat+0x24/0x30 [ 2388.348346] do_syscall_64+0xc1/0x440 [ 2388.350889] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2388.353589] RIP: 0033:0x7f53425a5332 [ 2388.355227] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2388.364334] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2388.366815] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2388.368858] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2388.372272] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2388.377202] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2388.380427] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2400.052046] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 2400.063878] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2400.113104] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2400.133127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2400.137676] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2400.141614] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2400.143726] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2400.152074] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2400.155428] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2400.162762] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2400.167865] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2400.169832] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2400.176797] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2400.182308] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2400.193323] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2400.195323] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2400.199714] Call Trace: [ 2400.200876] [ 2400.201864] ? show_regs.cold.9+0x22/0x2f [ 2400.203869] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2400.206097] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2400.208625] ? watchdog+0x40/0x40 [ 2400.210400] ? hrtimer_interrupt+0x128/0x2c0 [ 2400.213742] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2400.216144] ? apic_timer_interrupt+0xf/0x20 [ 2400.218268] [ 2400.219181] ? native_safe_halt+0x2e/0x50 [ 2400.220932] kvm_wait+0xcd/0xe0 [ 2400.222254] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2400.224684] ? newidle_balance.isra.93+0x207/0x580 [ 2400.226800] do_raw_spin_lock+0x177/0x190 [ 2400.229235] _raw_spin_lock+0x21/0x30 [ 2400.230911] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2400.236137] process_one_work+0x2c8/0x700 [ 2400.238659] worker_thread+0x296/0x6e0 [ 2400.241779] ? rescuer_thread+0x570/0x570 [ 2400.247989] kthread+0x1d1/0x200 [ 2400.249405] ? set_kthread_struct+0x70/0x70 [ 2400.251006] ret_from_fork+0x1f/0x30 [ 2416.058350] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 2416.060700] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 2416.072437] Modules linked in: [ 2416.076009] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2416.076097] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2416.076104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2416.076107] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2416.076122] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2416.076126] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2416.076130] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2416.076132] RDX: 0000000000007a8e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2416.076134] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2416.076148] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2416.076151] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2416.076155] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2416.076158] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2416.076160] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2416.076167] Call Trace: [ 2416.076172] [ 2416.076178] ? show_regs.cold.9+0x22/0x2f [ 2416.076187] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2416.076193] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2416.076197] ? watchdog+0x40/0x40 [ 2416.076201] ? hrtimer_interrupt+0x128/0x2c0 [ 2416.076204] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2416.076212] ? apic_timer_interrupt+0xf/0x20 [ 2416.076216] [ 2416.076219] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2416.076223] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2416.076228] do_raw_spin_lock+0x177/0x190 [ 2416.076232] _raw_spin_lock+0x21/0x30 [ 2416.076239] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2416.076560] obd_mod_free+0x46/0x140 [ptlrpc] [ 2416.076772] mdc_commit_open+0x91/0xb0 [mdc] [ 2416.076807] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2416.078385] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2416.078457] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2416.078464] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2416.078466] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2416.078477] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2416.078481] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2416.078485] RAX: 0000000000001a8d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2416.078487] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2416.078490] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2416.078492] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2416.078495] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2416.078498] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2416.078501] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2416.078504] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2416.078511] Call Trace: [ 2416.078514] [ 2416.078518] ? show_regs.cold.9+0x22/0x2f [ 2416.078526] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2416.078531] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2416.078536] ? watchdog+0x40/0x40 [ 2416.078540] ? hrtimer_interrupt+0x128/0x2c0 [ 2416.078544] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2416.078551] ? apic_timer_interrupt+0xf/0x20 [ 2416.078555] [ 2416.078558] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2416.078564] do_raw_spin_lock+0x177/0x190 [ 2416.078568] _raw_spin_lock+0x21/0x30 [ 2416.078573] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2416.078835] ? do_raw_spin_unlock+0x75/0x190 [ 2416.078840] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2416.079044] ? do_wait_intr+0xf0/0xf0 [ 2416.079053] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2416.079422] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2416.079646] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2416.080445] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2416.080570] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2416.080609] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2416.080640] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2416.080665] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2416.080744] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2416.080957] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2416.080984] lmv_intent_open+0x338/0x1150 [lmv] [ 2416.081011] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2416.081094] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2416.115291] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2416.115575] after_reply+0xe2c/0x1780 [ptlrpc] [ 2416.115776] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2416.115969] ? __next_timer_interrupt+0x160/0x160 [ 2416.115979] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2416.118385] ? do_wait_intr+0xf0/0xf0 [ 2416.119211] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2416.119522] kthread+0x1d1/0x200 [ 2416.119529] ? set_kthread_struct+0x70/0x70 [ 2416.119533] ret_from_fork+0x1f/0x30 [ 2416.382066] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2416.383583] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2416.385190] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2416.386931] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2416.388759] ? do_raw_spin_unlock+0x75/0x190 [ 2416.390662] ? _raw_spin_unlock+0x12/0x30 [ 2416.391891] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2416.393607] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2416.395335] ll_atomic_open+0x36a/0xd70 [lustre] [ 2416.396695] lookup_open+0x356/0xb40 [ 2416.397639] do_last+0x313/0xfc0 [ 2416.398736] ? nd_jump_root+0xe5/0x160 [ 2416.399958] ? path_init+0x437/0x520 [ 2416.400980] path_openat+0xf7/0x500 [ 2416.402234] ? do_raw_spin_unlock+0x75/0x190 [ 2416.403661] do_filp_open+0x99/0x140 [ 2416.404797] ? getname_flags+0x6e/0x330 [ 2416.406000] ? __check_object_size+0xff/0x256 [ 2416.407841] ? do_raw_spin_unlock+0x75/0x190 [ 2416.409066] ? _raw_spin_unlock+0x12/0x30 [ 2416.410282] do_sys_openat2+0x2b4/0x410 [ 2416.411751] do_sys_open+0x73/0xa0 [ 2416.412851] __x64_sys_openat+0x24/0x30 [ 2416.413926] do_syscall_64+0xc1/0x440 [ 2416.415105] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2416.416601] RIP: 0033:0x7f53425a5332 [ 2416.417591] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2416.422559] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2416.424629] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2416.426696] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2416.428781] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2416.430885] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2416.432941] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2428.052880] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 2428.058225] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2428.110096] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2428.119652] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2428.129096] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2428.137523] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2428.142313] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2428.164527] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2428.169013] RAX: 0000000000007ec3 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2428.173214] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2428.176697] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2428.179618] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2428.182813] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2428.186443] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2428.190378] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2428.192818] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2428.194785] Call Trace: [ 2428.195408] [ 2428.196183] ? show_regs.cold.9+0x22/0x2f [ 2428.197303] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2428.198747] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2428.200360] ? watchdog+0x40/0x40 [ 2428.201294] ? hrtimer_interrupt+0x128/0x2c0 [ 2428.202978] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2428.204874] ? apic_timer_interrupt+0xf/0x20 [ 2428.206822] [ 2428.207666] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2428.209991] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2428.212366] ? newidle_balance.isra.93+0x207/0x580 [ 2428.214400] do_raw_spin_lock+0x177/0x190 [ 2428.216376] _raw_spin_lock+0x21/0x30 [ 2428.218379] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2428.221218] process_one_work+0x2c8/0x700 [ 2428.223046] worker_thread+0x296/0x6e0 [ 2428.224784] ? rescuer_thread+0x570/0x570 [ 2428.226363] kthread+0x1d1/0x200 [ 2428.228355] ? set_kthread_struct+0x70/0x70 [ 2428.230688] ret_from_fork+0x1f/0x30 [ 2444.055012] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 2444.059390] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2444.060953] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 2444.088407] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2444.088425] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2444.088428] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2444.088446] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2444.088451] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2444.088458] RAX: 0000000000006497 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2444.088460] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2444.088463] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2444.088465] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2444.088467] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2444.088470] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2444.088474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2444.088476] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2444.088484] Call Trace: [ 2444.088492] [ 2444.088500] ? show_regs.cold.9+0x22/0x2f [ 2444.088512] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2444.088519] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2444.088523] ? watchdog+0x40/0x40 [ 2444.088841] ? hrtimer_interrupt+0x128/0x2c0 [ 2444.088848] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2444.088858] ? apic_timer_interrupt+0xf/0x20 [ 2444.088862] [ 2444.088864] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2444.088873] do_raw_spin_lock+0x177/0x190 [ 2444.088878] _raw_spin_lock+0x21/0x30 [ 2444.088883] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2444.089272] ? do_raw_spin_unlock+0x75/0x190 [ 2444.089278] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2444.103725] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2444.103815] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2444.103822] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2444.103825] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2444.103839] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2444.103843] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2444.103854] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2444.103856] RDX: 00000000000016b9 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2444.103858] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2444.103861] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2444.103863] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2444.103867] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2444.103870] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2444.103872] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2444.103880] Call Trace: [ 2444.103885] [ 2444.103889] ? show_regs.cold.9+0x22/0x2f [ 2444.103898] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2444.103904] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2444.103909] ? watchdog+0x40/0x40 [ 2444.103913] ? hrtimer_interrupt+0x128/0x2c0 [ 2444.103917] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2444.103925] ? apic_timer_interrupt+0xf/0x20 [ 2444.103929] [ 2444.103932] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2444.103936] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2444.103941] do_raw_spin_lock+0x177/0x190 [ 2444.103945] _raw_spin_lock+0x21/0x30 [ 2444.103950] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2444.115706] ? do_wait_intr+0xf0/0xf0 [ 2444.123965] obd_mod_free+0x46/0x140 [ptlrpc] [ 2444.125292] mdc_commit_open+0x91/0xb0 [mdc] [ 2444.125336] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2444.125580] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2444.126014] after_reply+0xe2c/0x1780 [ptlrpc] [ 2444.126745] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2444.127312] ? __next_timer_interrupt+0x160/0x160 [ 2444.127324] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2444.128074] ? do_wait_intr+0xf0/0xf0 [ 2444.128084] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2444.129393] kthread+0x1d1/0x200 [ 2444.129401] ? set_kthread_struct+0x70/0x70 [ 2444.129405] ret_from_fork+0x1f/0x30 [ 2444.471227] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2444.473099] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2444.474815] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2444.476816] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2444.478315] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2444.479779] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2444.481119] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2444.482360] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2444.483809] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2444.485408] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2444.487001] lmv_intent_open+0x338/0x1150 [lmv] [ 2444.488249] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2444.489509] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2444.490975] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2444.492107] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2444.493329] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2444.494778] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2444.496071] ? do_raw_spin_unlock+0x75/0x190 [ 2444.497168] ? _raw_spin_unlock+0x12/0x30 [ 2444.498266] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2444.499456] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2444.500812] ll_atomic_open+0x36a/0xd70 [lustre] [ 2444.502109] lookup_open+0x356/0xb40 [ 2444.502938] do_last+0x313/0xfc0 [ 2444.503602] ? nd_jump_root+0xe5/0x160 [ 2444.504574] ? path_init+0x437/0x520 [ 2444.505402] path_openat+0xf7/0x500 [ 2444.506261] ? do_raw_spin_unlock+0x75/0x190 [ 2444.507311] do_filp_open+0x99/0x140 [ 2444.508226] ? getname_flags+0x6e/0x330 [ 2444.509153] ? __check_object_size+0xff/0x256 [ 2444.510399] ? do_raw_spin_unlock+0x75/0x190 [ 2444.512125] ? _raw_spin_unlock+0x12/0x30 [ 2444.513784] do_sys_openat2+0x2b4/0x410 [ 2444.515773] do_sys_open+0x73/0xa0 [ 2444.517170] __x64_sys_openat+0x24/0x30 [ 2444.518706] do_syscall_64+0xc1/0x440 [ 2444.520214] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2444.522396] RIP: 0033:0x7f53425a5332 [ 2444.523986] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2444.531466] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2444.534447] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2444.537463] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2444.540277] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2444.542954] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2444.547170] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2456.058090] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2456.068074] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2456.114534] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2456.123594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2456.127848] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2456.137701] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2456.140739] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2456.153157] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2456.157208] RAX: 0000000000007b91 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2456.166602] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2456.175197] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2456.184016] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2456.189786] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2456.194712] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2456.197204] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2456.199885] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2456.203361] Call Trace: [ 2456.204597] [ 2456.205541] ? show_regs.cold.9+0x22/0x2f [ 2456.207697] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2456.214363] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2456.216201] ? watchdog+0x40/0x40 [ 2456.217522] ? hrtimer_interrupt+0x128/0x2c0 [ 2456.219170] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2456.222373] ? apic_timer_interrupt+0xf/0x20 [ 2456.225768] [ 2456.226248] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2456.228319] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2456.229926] ? newidle_balance.isra.93+0x207/0x580 [ 2456.232107] do_raw_spin_lock+0x177/0x190 [ 2456.233839] _raw_spin_lock+0x21/0x30 [ 2456.235643] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2456.238327] process_one_work+0x2c8/0x700 [ 2456.240113] worker_thread+0x296/0x6e0 [ 2456.241269] ? rescuer_thread+0x570/0x570 [ 2456.242877] kthread+0x1d1/0x200 [ 2456.243936] ? set_kthread_struct+0x70/0x70 [ 2456.245142] ret_from_fork+0x1f/0x30 [ 2458.592350] INFO: task khugepaged:48 blocked for more than 120 seconds. [ 2458.602469] Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2458.618127] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2458.627089] task:khugepaged state:D stack:0 pid:48 ppid:2 flags:0x80004000 [ 2458.636239] Call Trace: [ 2458.639248] __schedule+0x351/0xcb0 [ 2458.643126] ? wait_for_completion+0xae/0x1e0 [ 2458.648475] schedule+0xc0/0x180 [ 2458.650524] schedule_timeout+0x126/0x190 [ 2458.657454] ? try_to_wake_up+0x80/0x710 [ 2458.660808] ? __prepare_to_swait+0x5b/0x90 [ 2458.664480] ? do_raw_spin_unlock+0x75/0x190 [ 2458.666763] wait_for_completion+0xf0/0x1e0 [ 2458.668713] flush_work+0x1f3/0x390 [ 2458.670345] ? flush_workqueue_prep_pwqs+0x200/0x200 [ 2458.672565] ? do_raw_spin_unlock+0x75/0x190 [ 2458.675783] ? __pagevec_release+0x50/0x50 [ 2458.677599] lru_add_drain_all+0x1d4/0x280 [ 2458.681958] khugepaged+0xa2/0x16f0 [ 2458.683879] ? woken_wake_function+0x30/0x30 [ 2458.686601] ? collapse_pte_mapped_thp+0x8a0/0x8a0 [ 2458.689276] kthread+0x1d1/0x200 [ 2458.690587] ? set_kthread_struct+0x70/0x70 [ 2458.693187] ret_from_fork+0x1f/0x30 [ 2472.055456] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 2472.057221] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 2472.059104] Modules linked in: [ 2472.061808] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2472.061886] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2472.061893] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2472.061896] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2472.061910] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2472.061913] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2472.061918] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2472.061921] RDX: 0000000000002fdd RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2472.061923] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2472.061926] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2472.061928] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2472.061931] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2472.061934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2472.061936] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2472.061944] Call Trace: [ 2472.061949] [ 2472.061954] ? show_regs.cold.9+0x22/0x2f [ 2472.061963] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2472.061969] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2472.061973] ? watchdog+0x40/0x40 [ 2472.061977] ? hrtimer_interrupt+0x128/0x2c0 [ 2472.061981] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2472.061989] ? apic_timer_interrupt+0xf/0x20 [ 2472.061994] [ 2472.061996] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2472.062001] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2472.062005] do_raw_spin_lock+0x177/0x190 [ 2472.062009] _raw_spin_lock+0x21/0x30 [ 2472.062015] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2472.062402] obd_mod_free+0x46/0x140 [ptlrpc] [ 2472.062737] mdc_commit_open+0x91/0xb0 [mdc] [ 2472.062775] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2472.062976] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2472.063288] after_reply+0xe2c/0x1780 [ptlrpc] [ 2472.063484] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2472.063743] ? __next_timer_interrupt+0x160/0x160 [ 2472.063753] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2472.063945] ? do_wait_intr+0xf0/0xf0 [ 2472.063954] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2472.064355] kthread+0x1d1/0x200 [ 2472.064360] ? set_kthread_struct+0x70/0x70 [ 2472.064364] ret_from_fork+0x1f/0x30 [ 2472.189942] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2472.204047] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2472.207376] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2472.209887] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2472.211772] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2472.223657] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2472.229527] RAX: 0000000000006b0c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2472.232709] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2472.235742] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2472.241486] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2472.245933] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2472.252666] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2472.257913] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2472.261785] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2472.266116] Call Trace: [ 2472.267360] [ 2472.268388] ? show_regs.cold.9+0x22/0x2f [ 2472.270501] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2472.272722] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2472.275052] ? watchdog+0x40/0x40 [ 2472.276825] ? hrtimer_interrupt+0x128/0x2c0 [ 2472.278993] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2472.281689] ? apic_timer_interrupt+0xf/0x20 [ 2472.283832] [ 2472.284837] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2472.287665] do_raw_spin_lock+0x177/0x190 [ 2472.291130] _raw_spin_lock+0x21/0x30 [ 2472.293552] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2472.296223] ? do_raw_spin_unlock+0x75/0x190 [ 2472.298767] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2472.302762] ? do_wait_intr+0xf0/0xf0 [ 2472.304905] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2472.310831] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2472.315146] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2472.318882] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2472.322998] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2472.328761] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2472.331902] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2472.333922] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2472.336730] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2472.339720] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2472.342011] lmv_intent_open+0x338/0x1150 [lmv] [ 2472.343380] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2472.346770] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2472.349234] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2472.351126] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2472.354122] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2472.361025] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2472.366391] ? do_raw_spin_unlock+0x75/0x190 [ 2472.371112] ? _raw_spin_unlock+0x12/0x30 [ 2472.374971] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2472.381618] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2472.384485] ll_atomic_open+0x36a/0xd70 [lustre] [ 2472.386767] lookup_open+0x356/0xb40 [ 2472.388503] do_last+0x313/0xfc0 [ 2472.392850] ? nd_jump_root+0xe5/0x160 [ 2472.394739] ? path_init+0x437/0x520 [ 2472.397643] path_openat+0xf7/0x500 [ 2472.398846] ? do_raw_spin_unlock+0x75/0x190 [ 2472.401722] do_filp_open+0x99/0x140 [ 2472.404655] ? getname_flags+0x6e/0x330 [ 2472.406893] ? __check_object_size+0xff/0x256 [ 2472.410085] ? do_raw_spin_unlock+0x75/0x190 [ 2472.413713] ? _raw_spin_unlock+0x12/0x30 [ 2472.416771] do_sys_openat2+0x2b4/0x410 [ 2472.419082] do_sys_open+0x73/0xa0 [ 2472.420620] __x64_sys_openat+0x24/0x30 [ 2472.422879] do_syscall_64+0xc1/0x440 [ 2472.425110] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2472.428433] RIP: 0033:0x7f53425a5332 [ 2472.430954] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2472.440743] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2472.443702] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2472.446507] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2472.448854] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2472.451751] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2472.454701] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2484.052701] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2484.057425] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2484.132568] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2484.147311] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2484.159196] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2484.165720] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2484.171910] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2484.182556] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2484.187694] RAX: 0000000000007f84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2484.193101] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2484.200100] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2484.209065] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2484.216280] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2484.218644] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2484.227766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2484.234620] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2484.238394] Call Trace: [ 2484.239761] [ 2484.242347] ? show_regs.cold.9+0x22/0x2f [ 2484.245301] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2484.246912] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2484.255778] ? watchdog+0x40/0x40 [ 2484.263284] ? hrtimer_interrupt+0x128/0x2c0 [ 2484.268733] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2484.272912] ? apic_timer_interrupt+0xf/0x20 [ 2484.281790] [ 2484.286545] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2484.288178] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2484.294376] ? newidle_balance.isra.93+0x207/0x580 [ 2484.296022] do_raw_spin_lock+0x177/0x190 [ 2484.300235] _raw_spin_lock+0x21/0x30 [ 2484.305744] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2484.311870] process_one_work+0x2c8/0x700 [ 2484.321579] worker_thread+0x296/0x6e0 [ 2484.322941] ? rescuer_thread+0x570/0x570 [ 2484.332264] kthread+0x1d1/0x200 [ 2484.336247] ? set_kthread_struct+0x70/0x70 [ 2484.339825] ret_from_fork+0x1f/0x30 [ 2500.055150] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 2500.057007] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 2500.058187] Modules linked in: [ 2500.062465] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2500.062543] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2500.062550] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2500.062628] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2500.062643] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2500.062646] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2500.062650] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2500.062652] RDX: 0000000000007102 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2500.062654] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2500.062657] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2500.062659] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2500.062662] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2500.062664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2500.062667] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2500.062684] Call Trace: [ 2500.062691] [ 2500.062697] ? show_regs.cold.9+0x22/0x2f [ 2500.062707] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2500.062714] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2500.062717] ? watchdog+0x40/0x40 [ 2500.062721] ? hrtimer_interrupt+0x128/0x2c0 [ 2500.062724] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2500.062733] ? apic_timer_interrupt+0xf/0x20 [ 2500.062737] [ 2500.062741] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2500.062746] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2500.062750] do_raw_spin_lock+0x177/0x190 [ 2500.062755] _raw_spin_lock+0x21/0x30 [ 2500.062761] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2500.064257] lustre(O) [ 2500.089191] obd_mod_free+0x46/0x140 [ptlrpc] [ 2500.098066] osp(O) [ 2500.101154] mdc_commit_open+0x91/0xb0 [mdc] [ 2500.108105] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2500.108199] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2500.108208] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2500.108213] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2500.123900] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2500.131091] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2500.131163] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2500.131172] RAX: 00000000000062da RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2500.131175] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2500.131177] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2500.131179] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2500.131182] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2500.131186] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2500.131188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2500.131191] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2500.131200] Call Trace: [ 2500.131208] [ 2500.131217] ? show_regs.cold.9+0x22/0x2f [ 2500.131239] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2500.131247] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2500.131251] ? watchdog+0x40/0x40 [ 2500.131255] ? hrtimer_interrupt+0x128/0x2c0 [ 2500.131259] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2500.131271] ? apic_timer_interrupt+0xf/0x20 [ 2500.131276] [ 2500.131279] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2500.131290] do_raw_spin_lock+0x177/0x190 [ 2500.131296] _raw_spin_lock+0x21/0x30 [ 2500.131301] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2500.131626] ? do_raw_spin_unlock+0x75/0x190 [ 2500.131633] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2500.132077] ? do_wait_intr+0xf0/0xf0 [ 2500.132089] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2500.132353] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2500.132643] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2500.133058] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2500.133259] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2500.133302] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2500.133329] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2500.133358] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2500.133441] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2500.133775] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2500.133959] lmv_intent_open+0x338/0x1150 [lmv] [ 2500.133997] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2500.134060] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2500.134117] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2500.134138] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2500.134215] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2500.134270] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2500.134325] ? do_raw_spin_unlock+0x75/0x190 [ 2500.134331] ? _raw_spin_unlock+0x12/0x30 [ 2500.134337] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2500.134514] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2500.134527] ll_atomic_open+0x36a/0xd70 [lustre] [ 2500.144055] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2500.147542] lookup_open+0x356/0xb40 [ 2500.147573] do_last+0x313/0xfc0 [ 2500.147578] ? nd_jump_root+0xe5/0x160 [ 2500.147583] ? path_init+0x437/0x520 [ 2500.147589] path_openat+0xf7/0x500 [ 2500.147594] ? do_raw_spin_unlock+0x75/0x190 [ 2500.147602] do_filp_open+0x99/0x140 [ 2500.147610] ? getname_flags+0x6e/0x330 [ 2500.147615] ? __check_object_size+0xff/0x256 [ 2500.147625] ? do_raw_spin_unlock+0x75/0x190 [ 2500.147629] ? _raw_spin_unlock+0x12/0x30 [ 2500.147638] do_sys_openat2+0x2b4/0x410 [ 2500.147644] do_sys_open+0x73/0xa0 [ 2500.147649] __x64_sys_openat+0x24/0x30 [ 2500.147652] do_syscall_64+0xc1/0x440 [ 2500.147661] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2500.156978] after_reply+0xe2c/0x1780 [ptlrpc] [ 2500.157275] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2500.157464] ? __next_timer_interrupt+0x160/0x160 [ 2500.157474] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2500.157685] ? do_wait_intr+0xf0/0xf0 [ 2500.157695] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2500.157894] kthread+0x1d1/0x200 [ 2500.157900] ? set_kthread_struct+0x70/0x70 [ 2500.157904] ret_from_fork+0x1f/0x30 [ 2500.655255] RIP: 0033:0x7f53425a5332 [ 2500.656857] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2500.670448] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2500.676736] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2500.683910] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2500.688052] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2500.695737] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2500.701859] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2512.052011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2512.055883] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2512.073966] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2512.076589] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2512.078562] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2512.084875] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2512.087690] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2512.108475] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2512.112132] RAX: 00000000000001b7 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2512.123614] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2512.133943] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2512.146029] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2512.156983] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2512.162434] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2512.165312] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2512.175768] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2512.186765] Call Trace: [ 2512.188288] [ 2512.192311] ? show_regs.cold.9+0x22/0x2f [ 2512.196911] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2512.202295] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2512.206771] ? watchdog+0x40/0x40 [ 2512.208058] ? hrtimer_interrupt+0x128/0x2c0 [ 2512.215833] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2512.221684] ? apic_timer_interrupt+0xf/0x20 [ 2512.224849] [ 2512.226690] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2512.233183] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2512.240810] ? newidle_balance.isra.93+0x207/0x580 [ 2512.242565] do_raw_spin_lock+0x177/0x190 [ 2512.251634] _raw_spin_lock+0x21/0x30 [ 2512.254864] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2512.262842] process_one_work+0x2c8/0x700 [ 2512.265225] worker_thread+0x296/0x6e0 [ 2512.271704] ? rescuer_thread+0x570/0x570 [ 2512.278444] kthread+0x1d1/0x200 [ 2512.282422] ? set_kthread_struct+0x70/0x70 [ 2512.287060] ret_from_fork+0x1f/0x30 [ 2514.434087] rcu: INFO: rcu_sched self-detected stall on CPU [ 2514.435874] rcu: 2-....: (232701 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=75121 [ 2514.442132] (t=420008 jiffies g=274185 q=63188) [ 2514.444575] Sending NMI from CPU 2 to CPUs 1: [ 2514.451165] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 2514.453063] NMI backtrace for cpu 2 [ 2514.459578] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2514.471161] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2514.481677] Call Trace: [ 2514.484692] [ 2514.486874] ? dump_stack+0xbb/0x10e [ 2514.495808] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2514.503122] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2514.511397] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2514.519852] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2514.529627] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2514.536175] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2514.537743] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2514.546777] ? update_process_times+0x6d/0xc0 [ 2514.552809] ? tick_sched_handle.isra.18+0x27/0x90 [ 2514.560834] ? tick_sched_timer+0x53/0xb0 [ 2514.568138] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2514.573360] ? tick_sched_handle.isra.18+0x90/0x90 [ 2514.579472] ? hrtimer_interrupt+0x128/0x2c0 [ 2514.586611] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2514.592796] ? apic_timer_interrupt+0xf/0x20 [ 2514.599343] [ 2514.605788] ? native_safe_halt+0x2e/0x50 [ 2514.610677] ? kvm_wait+0xcd/0xe0 [ 2514.612740] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2514.615717] ? do_raw_spin_lock+0x177/0x190 [ 2514.618572] ? _raw_spin_lock+0x21/0x30 [ 2514.620663] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2514.624388] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2514.627280] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2514.629996] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2514.632661] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2514.636678] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2514.640621] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2514.643698] ? __next_timer_interrupt+0x160/0x160 [ 2514.646220] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2514.648077] ? do_wait_intr+0xf0/0xf0 [ 2514.650758] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2514.654477] ? kthread+0x1d1/0x200 [ 2514.657304] ? set_kthread_struct+0x70/0x70 [ 2514.662711] ? ret_from_fork+0x1f/0x30 [ 2540.053019] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2540.056768] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 2540.058030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 2540.058041] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2540.058195] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2540.058201] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2540.058205] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2540.058222] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2540.058226] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2540.058232] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2540.058236] RDX: 0000000000007f23 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2540.058239] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2540.058242] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2540.058244] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2540.058247] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2540.058250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2540.058252] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2540.058262] Call Trace: [ 2540.058270] [ 2540.058276] ? show_regs.cold.9+0x22/0x2f [ 2540.058288] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2540.058294] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2540.058299] ? watchdog+0x40/0x40 [ 2540.058378] ? hrtimer_interrupt+0x128/0x2c0 [ 2540.058385] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2540.058394] ? apic_timer_interrupt+0xf/0x20 [ 2540.058398] [ 2540.058401] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2540.058406] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2540.058411] do_raw_spin_lock+0x177/0x190 [ 2540.058416] _raw_spin_lock+0x21/0x30 [ 2540.058422] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2540.058718] obd_mod_free+0x46/0x140 [ptlrpc] [ 2540.058916] mdc_commit_open+0x91/0xb0 [mdc] [ 2540.058956] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2540.060052] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2540.060571] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2540.060646] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2540.060653] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2540.060659] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2540.060865] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2540.060874] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2540.060878] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2540.060882] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2540.060885] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2540.060887] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2540.060888] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2540.060890] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2540.060893] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2540.060896] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2540.060898] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2540.060906] Call Trace: [ 2540.060910] [ 2540.060913] ? show_regs.cold.9+0x22/0x2f [ 2540.060922] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2540.060929] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2540.060933] ? watchdog+0x40/0x40 [ 2540.060937] ? hrtimer_interrupt+0x128/0x2c0 [ 2540.060941] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2540.060948] ? apic_timer_interrupt+0xf/0x20 [ 2540.060953] [ 2540.060956] ? native_safe_halt+0x2e/0x50 [ 2540.060960] kvm_wait+0xcd/0xe0 [ 2540.060967] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2540.060987] ? newidle_balance.isra.93+0x207/0x580 [ 2540.060992] do_raw_spin_lock+0x177/0x190 [ 2540.060996] _raw_spin_lock+0x21/0x30 [ 2540.061000] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2540.061382] process_one_work+0x2c8/0x700 [ 2540.061389] worker_thread+0x296/0x6e0 [ 2540.061394] ? rescuer_thread+0x570/0x570 [ 2540.061398] kthread+0x1d1/0x200 [ 2540.061402] ? set_kthread_struct+0x70/0x70 [ 2540.061406] ret_from_fork+0x1f/0x30 [ 2540.060568] after_reply+0xe2c/0x1780 [ptlrpc] [ 2540.061531] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2540.061740] ? __next_timer_interrupt+0x160/0x160 [ 2540.061750] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2540.065539] Modules linked in: [ 2540.071957] ? do_wait_intr+0xf0/0xf0 [ 2540.107550] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2540.107631] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2540.107640] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2540.107643] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2540.107661] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2540.107665] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2540.107671] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2540.107673] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2540.107675] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2540.107677] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2540.107680] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2540.107683] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2540.107686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2540.107688] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2540.107699] Call Trace: [ 2540.107706] [ 2540.107714] ? show_regs.cold.9+0x22/0x2f [ 2540.107726] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2540.107733] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2540.107738] ? watchdog+0x40/0x40 [ 2540.107743] ? hrtimer_interrupt+0x128/0x2c0 [ 2540.107746] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2540.107753] ? apic_timer_interrupt+0xf/0x20 [ 2540.107758] [ 2540.107761] ? native_safe_halt+0x2e/0x50 [ 2540.107765] kvm_wait+0xcd/0xe0 [ 2540.107773] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2540.107781] do_raw_spin_lock+0x177/0x190 [ 2540.107786] _raw_spin_lock+0x21/0x30 [ 2540.107790] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2540.108131] ? do_raw_spin_unlock+0x75/0x190 [ 2540.108137] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2540.108334] ? do_wait_intr+0xf0/0xf0 [ 2540.108343] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2540.108545] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2540.108764] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2540.128595] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2540.128922] kthread+0x1d1/0x200 [ 2540.128931] ? set_kthread_struct+0x70/0x70 [ 2540.128936] ret_from_fork+0x1f/0x30 [ 2540.812359] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2540.815702] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2540.824625] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2540.827351] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2540.833006] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2540.836386] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2540.846388] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2540.848336] lmv_intent_open+0x338/0x1150 [lmv] [ 2540.856561] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2540.863888] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2540.868771] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2540.871238] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2540.874331] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2540.876849] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2540.879641] ? do_raw_spin_unlock+0x75/0x190 [ 2540.882572] ? _raw_spin_unlock+0x12/0x30 [ 2540.885781] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2540.889060] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2540.891713] ll_atomic_open+0x36a/0xd70 [lustre] [ 2540.896935] lookup_open+0x356/0xb40 [ 2540.903703] do_last+0x313/0xfc0 [ 2540.905038] ? nd_jump_root+0xe5/0x160 [ 2540.908792] ? path_init+0x437/0x520 [ 2540.913369] path_openat+0xf7/0x500 [ 2540.916248] ? do_raw_spin_unlock+0x75/0x190 [ 2540.918346] do_filp_open+0x99/0x140 [ 2540.922052] ? getname_flags+0x6e/0x330 [ 2540.929239] ? __check_object_size+0xff/0x256 [ 2540.938819] ? do_raw_spin_unlock+0x75/0x190 [ 2540.944855] ? _raw_spin_unlock+0x12/0x30 [ 2540.948250] do_sys_openat2+0x2b4/0x410 [ 2540.950950] do_sys_open+0x73/0xa0 [ 2540.952800] __x64_sys_openat+0x24/0x30 [ 2540.955515] do_syscall_64+0xc1/0x440 [ 2540.966545] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2540.971304] RIP: 0033:0x7f53425a5332 [ 2540.972950] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2540.996744] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2541.000818] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2541.017463] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2541.022580] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2541.029643] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2541.035228] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2568.052634] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2568.055037] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 2568.058791] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 2568.058806] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2568.058875] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2568.058882] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2568.058885] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2568.058900] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2568.058905] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2568.058910] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2568.058912] RDX: 0000000000003797 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2568.058914] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2568.058916] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2568.058918] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2568.058922] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2568.058925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2568.058927] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2568.058938] Call Trace: [ 2568.058944] [ 2568.058949] ? show_regs.cold.9+0x22/0x2f [ 2568.058959] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2568.058966] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2568.058971] ? watchdog+0x40/0x40 [ 2568.058975] ? hrtimer_interrupt+0x128/0x2c0 [ 2568.058979] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2568.058987] ? apic_timer_interrupt+0xf/0x20 [ 2568.058992] [ 2568.058995] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2568.059000] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2568.059004] do_raw_spin_lock+0x177/0x190 [ 2568.059009] _raw_spin_lock+0x21/0x30 [ 2568.059250] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2568.059311] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2568.059316] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2568.059016] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2568.059658] obd_mod_free+0x46/0x140 [ptlrpc] [ 2568.059855] mdc_commit_open+0x91/0xb0 [mdc] [ 2568.059897] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2568.059321] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2568.060263] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2568.061229] [ 2568.061234] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2568.061248] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2568.061252] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2568.061229] after_reply+0xe2c/0x1780 [ptlrpc] [ 2568.061337] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2568.061341] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2568.061343] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2568.061345] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2568.061347] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2568.061350] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2568.061352] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2568.061355] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2568.061363] Call Trace: [ 2568.061367] [ 2568.061370] ? show_regs.cold.9+0x22/0x2f [ 2568.061380] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2568.061386] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2568.061392] ? watchdog+0x40/0x40 [ 2568.061396] ? hrtimer_interrupt+0x128/0x2c0 [ 2568.061400] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2568.061407] ? apic_timer_interrupt+0xf/0x20 [ 2568.061412] [ 2568.061415] ? native_safe_halt+0x2e/0x50 [ 2568.061419] kvm_wait+0xcd/0xe0 [ 2568.061427] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2568.061433] ? newidle_balance.isra.93+0x207/0x580 [ 2568.061443] do_raw_spin_lock+0x177/0x190 [ 2568.061449] _raw_spin_lock+0x21/0x30 [ 2568.061440] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2568.061453] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2568.062286] ? __next_timer_interrupt+0x160/0x160 [ 2568.062297] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2568.063251] process_one_work+0x2c8/0x700 [ 2568.063395] worker_thread+0x296/0x6e0 [ 2568.063399] ? rescuer_thread+0x570/0x570 [ 2568.063403] kthread+0x1d1/0x200 [ 2568.063408] ? set_kthread_struct+0x70/0x70 [ 2568.063411] ret_from_fork+0x1f/0x30 [ 2568.071424] Modules linked in: [ 2568.076525] ? do_wait_intr+0xf0/0xf0 [ 2568.076544] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2568.098658] lustre(O) [ 2568.106999] kthread+0x1d1/0x200 [ 2568.107015] ? set_kthread_struct+0x70/0x70 [ 2568.107018] ret_from_fork+0x1f/0x30 [ 2568.527208] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2568.565810] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2568.576708] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2568.588090] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2568.589778] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2568.607077] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2568.611712] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2568.621154] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2568.633173] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2568.640962] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2568.651131] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2568.659393] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2568.664207] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2568.668042] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2568.673971] Call Trace: [ 2568.675223] [ 2568.676129] ? show_regs.cold.9+0x22/0x2f [ 2568.682373] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2568.683733] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2568.685309] ? watchdog+0x40/0x40 [ 2568.695973] ? hrtimer_interrupt+0x128/0x2c0 [ 2568.701578] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2568.710481] ? apic_timer_interrupt+0xf/0x20 [ 2568.711945] [ 2568.716959] ? native_safe_halt+0x2e/0x50 [ 2568.718387] kvm_wait+0xcd/0xe0 [ 2568.719537] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2568.732262] do_raw_spin_lock+0x177/0x190 [ 2568.733679] _raw_spin_lock+0x21/0x30 [ 2568.739359] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2568.747170] ? do_raw_spin_unlock+0x75/0x190 [ 2568.754076] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2568.760133] ? do_wait_intr+0xf0/0xf0 [ 2568.762365] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2568.767165] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2568.768959] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2568.773417] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2568.778370] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2568.787158] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2568.794872] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2568.796904] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2568.799062] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2568.801718] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2568.803654] lmv_intent_open+0x338/0x1150 [lmv] [ 2568.806296] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2568.810438] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2568.813456] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2568.816395] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2568.821008] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2568.828198] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2568.836945] ? do_raw_spin_unlock+0x75/0x190 [ 2568.844829] ? _raw_spin_unlock+0x12/0x30 [ 2568.847459] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2568.853088] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2568.860866] ll_atomic_open+0x36a/0xd70 [lustre] [ 2568.866812] lookup_open+0x356/0xb40 [ 2568.868084] do_last+0x313/0xfc0 [ 2568.870616] ? nd_jump_root+0xe5/0x160 [ 2568.878318] ? path_init+0x437/0x520 [ 2568.884663] path_openat+0xf7/0x500 [ 2568.887676] ? do_raw_spin_unlock+0x75/0x190 [ 2568.889266] do_filp_open+0x99/0x140 [ 2568.892861] ? getname_flags+0x6e/0x330 [ 2568.894207] ? __check_object_size+0xff/0x256 [ 2568.898970] ? do_raw_spin_unlock+0x75/0x190 [ 2568.903168] ? _raw_spin_unlock+0x12/0x30 [ 2568.907777] do_sys_openat2+0x2b4/0x410 [ 2568.909657] do_sys_open+0x73/0xa0 [ 2568.911460] __x64_sys_openat+0x24/0x30 [ 2568.915804] do_syscall_64+0xc1/0x440 [ 2568.917545] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2568.919744] RIP: 0033:0x7f53425a5332 [ 2568.924795] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2568.936430] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2568.942043] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2568.948597] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2568.956749] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2568.962395] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2568.967137] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2596.055118] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 2596.056749] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2596.058783] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 2596.058799] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2596.058872] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2596.058878] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2596.058881] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2596.058919] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2596.058923] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2596.058928] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2596.058931] RDX: 0000000000007b0c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2596.058932] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2596.058934] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2596.058937] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2596.058939] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2596.058942] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2596.058945] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2596.058954] Call Trace: [ 2596.058973] [ 2596.058980] ? show_regs.cold.9+0x22/0x2f [ 2596.058990] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2596.059006] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2596.059011] ? watchdog+0x40/0x40 [ 2596.059015] ? hrtimer_interrupt+0x128/0x2c0 [ 2596.059020] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2596.059029] ? apic_timer_interrupt+0xf/0x20 [ 2596.059033] [ 2596.059037] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2596.059041] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2596.059046] do_raw_spin_lock+0x177/0x190 [ 2596.059050] _raw_spin_lock+0x21/0x30 [ 2596.059058] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2596.062483] obd_mod_free+0x46/0x140 [ptlrpc] [ 2596.062681] mdc_commit_open+0x91/0xb0 [mdc] [ 2596.062748] Modules linked in: lustre(O) [ 2596.062717] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2596.062912] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2596.063236] after_reply+0xe2c/0x1780 [ptlrpc] [ 2596.063432] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2596.063635] ? __next_timer_interrupt+0x160/0x160 [ 2596.063646] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2596.063848] ? do_wait_intr+0xf0/0xf0 [ 2596.063858] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2596.064063] kthread+0x1d1/0x200 [ 2596.065043] ? set_kthread_struct+0x70/0x70 [ 2596.065048] ret_from_fork+0x1f/0x30 [ 2596.071922] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2596.072147] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2596.072155] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2596.072161] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2596.074256] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2596.074272] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2596.074277] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2596.074281] RAX: 00000000000075c3 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2596.074283] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2596.074285] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2596.074287] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2596.074289] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2596.074292] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2596.074295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2596.074297] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2596.074305] Call Trace: [ 2596.074310] [ 2596.074316] ? show_regs.cold.9+0x22/0x2f [ 2596.074325] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2596.074384] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2596.074388] ? watchdog+0x40/0x40 [ 2596.074392] ? hrtimer_interrupt+0x128/0x2c0 [ 2596.074397] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2596.074406] ? apic_timer_interrupt+0xf/0x20 [ 2596.074411] [ 2596.074414] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2596.074419] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2596.074423] ? newidle_balance.isra.93+0x207/0x580 [ 2596.074428] do_raw_spin_lock+0x177/0x190 [ 2596.074433] _raw_spin_lock+0x21/0x30 [ 2596.074439] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2596.074668] process_one_work+0x2c8/0x700 [ 2596.074675] worker_thread+0x296/0x6e0 [ 2596.074679] ? rescuer_thread+0x570/0x570 [ 2596.074682] kthread+0x1d1/0x200 [ 2596.074686] ? set_kthread_struct+0x70/0x70 [ 2596.074689] ret_from_fork+0x1f/0x30 [ 2596.515621] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2596.551397] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2596.560890] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2596.570517] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2596.572043] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2596.595314] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2596.602525] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2596.606112] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2596.615631] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2596.620795] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2596.628475] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2596.637079] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2596.641416] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2596.647908] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2596.660217] Call Trace: [ 2596.661093] [ 2596.661759] ? show_regs.cold.9+0x22/0x2f [ 2596.674396] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2596.678461] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2596.684338] ? watchdog+0x40/0x40 [ 2596.687800] ? hrtimer_interrupt+0x128/0x2c0 [ 2596.689829] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2596.694699] ? apic_timer_interrupt+0xf/0x20 [ 2596.699566] [ 2596.701185] ? native_safe_halt+0x2e/0x50 [ 2596.703462] kvm_wait+0xcd/0xe0 [ 2596.705128] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2596.708312] do_raw_spin_lock+0x177/0x190 [ 2596.713348] _raw_spin_lock+0x21/0x30 [ 2596.714798] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2596.718289] ? do_raw_spin_unlock+0x75/0x190 [ 2596.723912] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2596.726933] ? do_wait_intr+0xf0/0xf0 [ 2596.729592] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2596.732653] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2596.736387] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2596.744569] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2596.748586] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2596.752204] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2596.756994] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2596.759350] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2596.765660] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2596.769959] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2596.775848] lmv_intent_open+0x338/0x1150 [lmv] [ 2596.779905] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2596.785478] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2596.787934] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2596.790807] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2596.794909] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2596.803975] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2596.807928] ? do_raw_spin_unlock+0x75/0x190 [ 2596.811860] ? _raw_spin_unlock+0x12/0x30 [ 2596.823295] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2596.826638] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2596.835850] ll_atomic_open+0x36a/0xd70 [lustre] [ 2596.841913] lookup_open+0x356/0xb40 [ 2596.849419] do_last+0x313/0xfc0 [ 2596.851803] ? nd_jump_root+0xe5/0x160 [ 2596.853781] ? path_init+0x437/0x520 [ 2596.857624] path_openat+0xf7/0x500 [ 2596.862292] ? do_raw_spin_unlock+0x75/0x190 [ 2596.866876] do_filp_open+0x99/0x140 [ 2596.869321] ? getname_flags+0x6e/0x330 [ 2596.871787] ? __check_object_size+0xff/0x256 [ 2596.876889] ? do_raw_spin_unlock+0x75/0x190 [ 2596.883307] ? _raw_spin_unlock+0x12/0x30 [ 2596.886671] do_sys_openat2+0x2b4/0x410 [ 2596.889409] do_sys_open+0x73/0xa0 [ 2596.891449] __x64_sys_openat+0x24/0x30 [ 2596.896148] do_syscall_64+0xc1/0x440 [ 2596.899581] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2596.902477] RIP: 0033:0x7f53425a5332 [ 2596.906336] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2596.919666] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2596.927953] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2596.935802] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2596.943235] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2596.945456] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2596.950631] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2624.052192] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2624.055147] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2624.055290] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 2624.057119] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 2624.057131] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2624.057200] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2624.057205] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2624.057207] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2624.057270] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2624.057274] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2624.057278] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2624.057280] RDX: 0000000000001e65 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2624.057282] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2624.057284] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2624.057286] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2624.057288] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2624.057291] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2624.057305] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2624.057313] Call Trace: [ 2624.057330] [ 2624.057336] ? show_regs.cold.9+0x22/0x2f [ 2624.057346] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2624.057352] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2624.057356] ? watchdog+0x40/0x40 [ 2624.057360] ? hrtimer_interrupt+0x128/0x2c0 [ 2624.057364] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2624.057371] ? apic_timer_interrupt+0xf/0x20 [ 2624.057375] [ 2624.057378] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2624.057392] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2624.057398] do_raw_spin_lock+0x177/0x190 [ 2624.057402] _raw_spin_lock+0x21/0x30 [ 2624.057408] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2624.057687] obd_mod_free+0x46/0x140 [ptlrpc] [ 2624.058084] mdc_commit_open+0x91/0xb0 [mdc] [ 2624.058137] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2624.058366] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2624.058610] after_reply+0xe2c/0x1780 [ptlrpc] [ 2624.059008] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2624.059226] ? __next_timer_interrupt+0x160/0x160 [ 2624.059242] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2624.059685] ? do_wait_intr+0xf0/0xf0 [ 2624.059695] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2624.061174] kthread+0x1d1/0x200 [ 2624.061360] ? set_kthread_struct+0x70/0x70 [ 2624.061366] ret_from_fork+0x1f/0x30 [ 2624.080041] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2624.084708] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2624.084787] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2624.084794] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2624.084797] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2624.084810] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2624.084814] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2624.084818] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2624.084820] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2624.084822] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2624.084824] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2624.084826] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2624.084829] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2624.084831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2624.084833] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2624.084842] Call Trace: [ 2624.084847] [ 2624.084852] ? show_regs.cold.9+0x22/0x2f [ 2624.084861] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2624.084867] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2624.084871] ? watchdog+0x40/0x40 [ 2624.084876] ? hrtimer_interrupt+0x128/0x2c0 [ 2624.084880] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2624.084886] ? apic_timer_interrupt+0xf/0x20 [ 2624.084890] [ 2624.084892] ? native_safe_halt+0x2e/0x50 [ 2624.084896] kvm_wait+0xcd/0xe0 [ 2624.084902] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2624.084911] do_raw_spin_lock+0x177/0x190 [ 2624.084915] _raw_spin_lock+0x21/0x30 [ 2624.084919] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2624.085245] ? do_raw_spin_unlock+0x75/0x190 [ 2624.085251] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2624.085838] ? do_wait_intr+0xf0/0xf0 [ 2624.085847] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2624.086113] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2624.086728] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2624.086945] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2624.093107] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2624.140363] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2624.140492] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2624.140537] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2624.151784] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2624.152166] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2624.152181] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2624.152186] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2624.152191] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2624.152193] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2624.152196] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2624.152198] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2624.152200] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2624.152203] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2624.152206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2624.152209] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2624.152216] Call Trace: [ 2624.152224] [ 2624.152231] ? show_regs.cold.9+0x22/0x2f [ 2624.152243] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2624.152250] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2624.152254] ? watchdog+0x40/0x40 [ 2624.152258] ? hrtimer_interrupt+0x128/0x2c0 [ 2624.152262] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2624.152272] ? apic_timer_interrupt+0xf/0x20 [ 2624.152277] [ 2624.152280] ? native_safe_halt+0x2e/0x50 [ 2624.152284] kvm_wait+0xcd/0xe0 [ 2624.152292] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2624.152300] ? newidle_balance.isra.93+0x207/0x580 [ 2624.152306] do_raw_spin_lock+0x177/0x190 [ 2624.152310] _raw_spin_lock+0x21/0x30 [ 2624.152315] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2624.152536] process_one_work+0x2c8/0x700 [ 2624.152545] worker_thread+0x296/0x6e0 [ 2624.152549] ? rescuer_thread+0x570/0x570 [ 2624.152553] kthread+0x1d1/0x200 [ 2624.152559] ? set_kthread_struct+0x70/0x70 [ 2624.152564] ret_from_fork+0x1f/0x30 [ 2624.594042] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2624.596012] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2624.598180] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2624.600370] lmv_intent_open+0x338/0x1150 [lmv] [ 2624.602286] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2624.604763] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2624.606368] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2624.607694] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2624.609422] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2624.611562] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2624.613445] ? do_raw_spin_unlock+0x75/0x190 [ 2624.615337] ? _raw_spin_unlock+0x12/0x30 [ 2624.617450] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2624.620223] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2624.623480] ll_atomic_open+0x36a/0xd70 [lustre] [ 2624.626197] lookup_open+0x356/0xb40 [ 2624.628337] do_last+0x313/0xfc0 [ 2624.629443] ? nd_jump_root+0xe5/0x160 [ 2624.631483] ? path_init+0x437/0x520 [ 2624.636292] path_openat+0xf7/0x500 [ 2624.640683] ? do_raw_spin_unlock+0x75/0x190 [ 2624.645744] do_filp_open+0x99/0x140 [ 2624.654261] ? getname_flags+0x6e/0x330 [ 2624.657687] ? __check_object_size+0xff/0x256 [ 2624.660080] ? do_raw_spin_unlock+0x75/0x190 [ 2624.664269] ? _raw_spin_unlock+0x12/0x30 [ 2624.666974] do_sys_openat2+0x2b4/0x410 [ 2624.668028] do_sys_open+0x73/0xa0 [ 2624.671150] __x64_sys_openat+0x24/0x30 [ 2624.672977] do_syscall_64+0xc1/0x440 [ 2624.675038] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2624.681281] RIP: 0033:0x7f53425a5332 [ 2624.684991] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2624.694435] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2624.698952] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2624.702796] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2624.707934] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2624.710446] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2624.714745] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2652.053349] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2652.057045] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 2652.057386] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 2652.057393] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2652.057553] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2652.057559] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2652.057562] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2652.057599] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2652.057604] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2652.057609] RAX: 0000000000007e83 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2652.057612] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2652.057615] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2652.057617] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2652.057620] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2652.057624] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2652.057627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2652.057630] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2652.057767] Call Trace: [ 2652.057787] [ 2652.057794] ? show_regs.cold.9+0x22/0x2f [ 2652.057803] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2652.057810] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2652.057814] ? watchdog+0x40/0x40 [ 2652.057819] ? hrtimer_interrupt+0x128/0x2c0 [ 2652.057823] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2652.057831] ? apic_timer_interrupt+0xf/0x20 [ 2652.057836] [ 2652.057839] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2652.057846] do_raw_spin_lock+0x177/0x190 [ 2652.057851] _raw_spin_lock+0x21/0x30 [ 2652.057856] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2652.059157] ? do_raw_spin_unlock+0x75/0x190 [ 2652.059165] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2652.059554] ? do_wait_intr+0xf0/0xf0 [ 2652.059569] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2652.060104] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2652.060567] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2652.061284] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2652.062081] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2652.062134] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2652.062171] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2652.062210] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2652.062342] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2652.065070] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2652.065115] lmv_intent_open+0x338/0x1150 [lmv] [ 2652.065149] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2652.065267] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2652.065350] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2652.065394] Modules linked in: lustre(O) osp(O) [ 2652.065373] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2652.065462] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2652.065553] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2652.065638] ? do_raw_spin_unlock+0x75/0x190 [ 2652.065646] ? _raw_spin_unlock+0x12/0x30 [ 2652.065654] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2652.065689] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2652.065710] ll_atomic_open+0x36a/0xd70 [lustre] [ 2652.065795] lookup_open+0x356/0xb40 [ 2652.065807] do_last+0x313/0xfc0 [ 2652.065813] ? nd_jump_root+0xe5/0x160 [ 2652.065817] ? path_init+0x437/0x520 [ 2652.065822] path_openat+0xf7/0x500 [ 2652.065827] ? do_raw_spin_unlock+0x75/0x190 [ 2652.065832] do_filp_open+0x99/0x140 [ 2652.065839] ? getname_flags+0x6e/0x330 [ 2652.065844] ? __check_object_size+0xff/0x256 [ 2652.065853] ? do_raw_spin_unlock+0x75/0x190 [ 2652.065857] ? _raw_spin_unlock+0x12/0x30 [ 2652.065862] do_sys_openat2+0x2b4/0x410 [ 2652.065867] do_sys_open+0x73/0xa0 [ 2652.065872] __x64_sys_openat+0x24/0x30 [ 2652.065876] do_syscall_64+0xc1/0x440 [ 2652.065884] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2652.065891] RIP: 0033:0x7f53425a5332 [ 2652.065937] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2652.065942] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2652.065947] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2652.065950] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2652.065953] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2652.065955] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2652.065957] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2652.071186] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2652.071259] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2652.071267] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2652.071271] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2652.071284] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2652.071299] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2652.071304] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2652.071307] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2652.071309] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2652.071311] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2652.071313] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2652.071316] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2652.071318] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2652.071320] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2652.071328] Call Trace: [ 2652.071334] [ 2652.071339] ? show_regs.cold.9+0x22/0x2f [ 2652.071350] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2652.071358] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2652.071362] ? watchdog+0x40/0x40 [ 2652.071365] ? hrtimer_interrupt+0x128/0x2c0 [ 2652.071369] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2652.071375] ? apic_timer_interrupt+0xf/0x20 [ 2652.071379] [ 2652.071382] ? native_safe_halt+0x2e/0x50 [ 2652.071386] kvm_wait+0xcd/0xe0 [ 2652.071393] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2652.071399] do_raw_spin_lock+0x177/0x190 [ 2652.071403] _raw_spin_lock+0x21/0x30 [ 2652.071410] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2652.071780] obd_mod_free+0x46/0x140 [ptlrpc] [ 2652.072375] mdc_commit_open+0x91/0xb0 [mdc] [ 2652.072414] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2652.073155] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2652.080543] ofd(O) [ 2652.115219] after_reply+0xe2c/0x1780 [ptlrpc] [ 2652.115907] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2652.136442] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2652.136518] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2652.136526] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2652.136533] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2652.147260] ? __next_timer_interrupt+0x160/0x160 [ 2652.147281] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2652.147556] ? do_wait_intr+0xf0/0xf0 [ 2652.147567] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2652.147763] kthread+0x1d1/0x200 [ 2652.147770] ? set_kthread_struct+0x70/0x70 [ 2652.147774] ret_from_fork+0x1f/0x30 [ 2652.928505] [ 2652.929418] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2652.933992] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2652.964991] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2652.982743] RAX: 0000000000007d88 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2652.985756] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2653.002212] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2653.011346] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2653.023614] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2653.025229] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2653.031701] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2653.035813] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2653.043208] Call Trace: [ 2653.045395] [ 2653.047328] ? show_regs.cold.9+0x22/0x2f [ 2653.050954] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2653.053533] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2653.057600] ? watchdog+0x40/0x40 [ 2653.061330] ? hrtimer_interrupt+0x128/0x2c0 [ 2653.064609] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2653.067827] ? apic_timer_interrupt+0xf/0x20 [ 2653.071763] [ 2653.073715] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2653.078362] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2653.085188] ? newidle_balance.isra.93+0x207/0x580 [ 2653.096157] do_raw_spin_lock+0x177/0x190 [ 2653.097722] _raw_spin_lock+0x21/0x30 [ 2653.105048] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2653.117877] process_one_work+0x2c8/0x700 [ 2653.119571] worker_thread+0x296/0x6e0 [ 2653.120904] ? rescuer_thread+0x570/0x570 [ 2653.130570] kthread+0x1d1/0x200 [ 2653.137540] ? set_kthread_struct+0x70/0x70 [ 2653.139601] ret_from_fork+0x1f/0x30 [ 2660.060836] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 2660.066583] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2660.109329] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2660.124719] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2660.130003] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2660.134855] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2660.156523] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2660.169644] RAX: 0000000000003b13 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2660.176609] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 2660.182388] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2660.190992] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 2660.204344] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 2660.210989] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 2660.219665] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2660.223632] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 2660.229347] Call Trace: [ 2660.232482] [ 2660.233024] ? show_regs.cold.9+0x22/0x2f [ 2660.237139] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2660.240976] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2660.243758] ? watchdog+0x40/0x40 [ 2660.244923] ? hrtimer_interrupt+0x128/0x2c0 [ 2660.248326] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2660.251271] ? apic_timer_interrupt+0xf/0x20 [ 2660.253844] [ 2660.254664] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2660.264687] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2660.268671] do_raw_spin_lock+0x177/0x190 [ 2660.272242] _raw_spin_lock+0x21/0x30 [ 2660.273572] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2660.283140] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2660.286820] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2660.291640] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2660.293951] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2660.295692] ? woken_wake_function+0x30/0x30 [ 2660.307785] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2660.316223] kthread+0x1d1/0x200 [ 2660.319254] ? set_kthread_struct+0x70/0x70 [ 2660.323180] ret_from_fork+0x1f/0x30 [ 2668.000053] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 30s! [ 2668.003601] Showing busy workqueues and worker pools: [ 2668.005183] workqueue events: flags=0x0 [ 2668.006196] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 2668.008794] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4] [ 2668.013212] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2668.015575] pending: cpu_latency_work [ptlrpc] [ 2668.017707] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2668.020120] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 2668.022273] workqueue events_long: flags=0x0 [ 2668.023757] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2668.025895] pending: lu_objects_flush [obdclass] [ 2668.027660] workqueue events_unbound: flags=0x2 [ 2668.030361] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/512 refcnt=3 [ 2668.032706] pending: flush_memcg_stats_dwork [ 2668.034397] workqueue events_power_efficient: flags=0x80 [ 2668.035853] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2668.038255] pending: check_lifetime, do_cache_clean [ 2668.039625] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2668.041568] pending: neigh_periodic_work, neigh_periodic_work [ 2668.043540] workqueue mm_percpu_wq: flags=0x8 [ 2668.045250] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2668.047353] pending: vmstat_update, lru_add_drain_per_cpu [ 2668.048884] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2668.051245] pending: vmstat_update, lru_add_drain_per_cpu [ 2668.053098] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2668.055389] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2668.057280] workqueue knbd0-recv: flags=0x1a [ 2668.058376] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2668.060224] in-flight: 651:recv_work [ 2668.065474] workqueue ptlrpc_pinger: flags=0x2 [ 2668.070631] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2668.077437] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 2668.087439] pool 8: cpus=0-3 flags=0x4 nice=0 hung=30s workers=4 idle: 15979 31824 8 [ 2668.093634] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2658s workers=2 idle: 654 [ 2680.055111] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 2680.059806] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2680.062306] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 2680.083214] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2680.083231] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2680.083239] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2680.083256] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2680.083260] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2680.083266] RAX: 0000000000007317 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2680.083268] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2680.083271] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2680.083273] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2680.083274] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2680.083278] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2680.083280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2680.083282] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2680.083293] Call Trace: [ 2680.083300] [ 2680.083306] ? show_regs.cold.9+0x22/0x2f [ 2680.083316] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2680.083322] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2680.083326] ? watchdog+0x40/0x40 [ 2680.083330] ? hrtimer_interrupt+0x128/0x2c0 [ 2680.083334] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2680.083348] ? apic_timer_interrupt+0xf/0x20 [ 2680.083353] [ 2680.083356] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2680.083365] do_raw_spin_lock+0x177/0x190 [ 2680.083370] _raw_spin_lock+0x21/0x30 [ 2680.083374] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2680.083939] ? do_raw_spin_unlock+0x75/0x190 [ 2680.083946] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2680.084223] ? do_wait_intr+0xf0/0xf0 [ 2680.084234] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2680.084551] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2680.084776] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2680.085152] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2680.085301] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2680.085345] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2680.085375] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2680.085587] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2680.085670] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2680.085908] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2680.085937] lmv_intent_open+0x338/0x1150 [lmv] [ 2680.086116] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2680.086215] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2680.086313] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2680.086336] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2680.086505] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2680.103976] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2680.104058] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2680.104065] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2680.104068] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2680.104081] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2680.104085] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2680.104096] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2680.104099] RDX: 00000000000038e4 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2680.104101] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2680.104103] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2680.104106] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2680.104109] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2680.104112] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2680.104114] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2680.104122] Call Trace: [ 2680.104127] [ 2680.104132] ? show_regs.cold.9+0x22/0x2f [ 2680.104141] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2680.104147] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2680.104152] ? watchdog+0x40/0x40 [ 2680.104156] ? hrtimer_interrupt+0x128/0x2c0 [ 2680.104165] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2680.104185] ? apic_timer_interrupt+0xf/0x20 [ 2680.104190] [ 2680.104193] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2680.104198] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2680.104203] do_raw_spin_lock+0x177/0x190 [ 2680.104212] _raw_spin_lock+0x21/0x30 [ 2680.104218] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2680.104741] obd_mod_free+0x46/0x140 [ptlrpc] [ 2680.105023] mdc_commit_open+0x91/0xb0 [mdc] [ 2680.105070] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2680.105678] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2680.106771] after_reply+0xe2c/0x1780 [ptlrpc] [ 2680.108094] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2680.109157] ? __next_timer_interrupt+0x160/0x160 [ 2680.109259] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2680.110238] ? do_wait_intr+0xf0/0xf0 [ 2680.110320] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2680.111026] kthread+0x1d1/0x200 [ 2680.111035] ? set_kthread_struct+0x70/0x70 [ 2680.111039] ret_from_fork+0x1f/0x30 [ 2680.727893] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2680.730197] ? do_raw_spin_unlock+0x75/0x190 [ 2680.733725] ? _raw_spin_unlock+0x12/0x30 [ 2680.739403] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2680.741301] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2680.753240] ll_atomic_open+0x36a/0xd70 [lustre] [ 2680.758268] lookup_open+0x356/0xb40 [ 2680.760902] do_last+0x313/0xfc0 [ 2680.765629] ? nd_jump_root+0xe5/0x160 [ 2680.773062] ? path_init+0x437/0x520 [ 2680.778883] path_openat+0xf7/0x500 [ 2680.782128] ? do_raw_spin_unlock+0x75/0x190 [ 2680.787533] do_filp_open+0x99/0x140 [ 2680.793019] ? getname_flags+0x6e/0x330 [ 2680.796067] ? __check_object_size+0xff/0x256 [ 2680.800495] ? do_raw_spin_unlock+0x75/0x190 [ 2680.802111] ? _raw_spin_unlock+0x12/0x30 [ 2680.803497] do_sys_openat2+0x2b4/0x410 [ 2680.813691] do_sys_open+0x73/0xa0 [ 2680.823940] __x64_sys_openat+0x24/0x30 [ 2680.825459] do_syscall_64+0xc1/0x440 [ 2680.826798] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2680.832076] RIP: 0033:0x7f53425a5332 [ 2680.835125] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2680.857244] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2680.863253] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2680.874919] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2680.880360] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2680.886271] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2680.893871] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2688.059666] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 2688.070479] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2688.139510] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2688.155246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2688.172181] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2688.185225] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2688.207460] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2688.212134] RAX: 0000000000005805 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2688.216089] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 2688.219226] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2688.222772] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 2688.225998] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 2688.233466] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 2688.241254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2688.249782] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 2688.255261] Call Trace: [ 2688.257984] [ 2688.259228] ? show_regs.cold.9+0x22/0x2f [ 2688.261585] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2688.263860] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2688.266378] ? watchdog+0x40/0x40 [ 2688.267810] ? hrtimer_interrupt+0x128/0x2c0 [ 2688.270031] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2688.271858] ? apic_timer_interrupt+0xf/0x20 [ 2688.274028] [ 2688.275091] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2688.277799] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2688.280127] do_raw_spin_lock+0x177/0x190 [ 2688.281864] _raw_spin_lock+0x21/0x30 [ 2688.283862] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2688.286911] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2688.289770] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2688.293609] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2688.296774] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2688.300586] ? woken_wake_function+0x30/0x30 [ 2688.302793] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2688.307204] kthread+0x1d1/0x200 [ 2688.309192] ? set_kthread_struct+0x70/0x70 [ 2688.311462] ret_from_fork+0x1f/0x30 [ 2694.442876] rcu: INFO: rcu_sched self-detected stall on CPU [ 2694.446390] rcu: 2-....: (332632 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 2694.452658] (t=600016 jiffies g=274185 q=63727) [ 2694.457155] rcu: rcu_sched kthread starved for 58382 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 2694.464056] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2694.470241] rcu: RCU grace-period kthread stack dump: [ 2694.480599] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2694.486520] Call Trace: [ 2694.487354] __schedule+0x351/0xcb0 [ 2694.495180] schedule+0xc0/0x180 [ 2694.497439] schedule_timeout+0xb4/0x190 [ 2694.504970] ? do_raw_spin_unlock+0x75/0x190 [ 2694.507209] ? __next_timer_interrupt+0x160/0x160 [ 2694.516506] ? prepare_to_swait_event+0xf3/0x140 [ 2694.518453] rcu_gp_kthread+0x8dd/0x15b0 [ 2694.519796] ? force_qs_rnp+0x2c0/0x2c0 [ 2694.528002] kthread+0x1d1/0x200 [ 2694.532874] ? set_kthread_struct+0x70/0x70 [ 2694.541068] ret_from_fork+0x1f/0x30 [ 2694.544350] rcu: Stack dump where RCU GP kthread last ran: [ 2694.552047] Sending NMI from CPU 2 to CPUs 1: [ 2694.564806] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 8.308 msecs [ 2694.570408] NMI backtrace for cpu 1 [ 2694.570417] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2694.570420] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2694.570421] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2694.570423] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2694.570425] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [ 2694.570428] RAX: 000000000000760a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2694.570429] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2694.570431] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2694.570432] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2694.570433] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2694.570434] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2694.570435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2694.570437] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2694.570438] Call Trace: [ 2694.570439] [ 2694.570439] ? show_regs.cold.9+0x22/0x2f [ 2694.570440] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2694.570441] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2694.570442] ? nmi_handle+0xbd/0x210 [ 2694.570443] ? default_do_nmi+0x4b/0x1b0 [ 2694.570444] ? do_nmi+0x280/0x380 [ 2694.570445] ? end_repeat_nmi+0x16/0x1f [ 2694.570447] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2694.570448] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2694.570449] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2694.570450] [ 2694.570451] do_raw_spin_lock+0x177/0x190 [ 2694.570452] _raw_spin_lock+0x21/0x30 [ 2694.570453] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2694.570454] ? do_raw_spin_unlock+0x75/0x190 [ 2694.570455] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2694.570457] ? do_wait_intr+0xf0/0xf0 [ 2694.570458] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2694.570459] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2694.570460] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2694.570461] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2694.570462] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2694.570463] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2694.570464] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2694.570465] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2694.570466] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2694.570467] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2694.570468] lmv_intent_open+0x338/0x1150 [lmv] [ 2694.570470] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2694.570471] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2694.570472] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2694.570473] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2694.570474] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2694.570475] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2694.570476] ? do_raw_spin_unlock+0x75/0x190 [ 2694.570477] ? _raw_spin_unlock+0x12/0x30 [ 2694.570478] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2694.570479] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2694.570480] ll_atomic_open+0x36a/0xd70 [lustre] [ 2694.570481] lookup_open+0x356/0xb40 [ 2694.570482] do_last+0x313/0xfc0 [ 2694.570483] ? nd_jump_root+0xe5/0x160 [ 2694.570484] ? path_init+0x437/0x520 [ 2694.570485] path_openat+0xf7/0x500 [ 2694.570486] ? do_raw_spin_unlock+0x75/0x190 [ 2694.570487] do_filp_open+0x99/0x140 [ 2694.570488] ? getname_flags+0x6e/0x330 [ 2694.570489] ? __check_object_size+0xff/0x256 [ 2694.570490] ? do_raw_spin_unlock+0x75/0x190 [ 2694.570492] ? _raw_spin_unlock+0x12/0x30 [ 2694.570492] do_sys_openat2+0x2b4/0x410 [ 2694.570493] do_sys_open+0x73/0xa0 [ 2694.570494] __x64_sys_openat+0x24/0x30 [ 2694.570495] do_syscall_64+0xc1/0x440 [ 2694.570496] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2694.570497] RIP: 0033:0x7f53425a5332 [ 2694.570499] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2694.570501] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2694.570504] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2694.570505] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2694.570507] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2694.570508] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2694.570509] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2694.570515] Sending NMI from CPU 2 to CPUs 1: [ 2695.110638] NMI backtrace for cpu 1 [ 2695.110646] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2695.110648] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2695.110649] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2695.110651] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2695.110654] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [ 2695.110657] RAX: 0000000000002789 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2695.110658] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2695.110660] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2695.110662] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2695.110664] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2695.110665] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2695.110667] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2695.110668] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2695.110670] Call Trace: [ 2695.110670] [ 2695.110671] ? show_regs.cold.9+0x22/0x2f [ 2695.110673] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2695.110674] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2695.110675] ? nmi_handle+0xbd/0x210 [ 2695.110676] ? default_do_nmi+0x4b/0x1b0 [ 2695.110677] ? do_nmi+0x280/0x380 [ 2695.110677] ? end_repeat_nmi+0x16/0x1f [ 2695.110678] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2695.110679] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2695.110680] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2695.110681] [ 2695.110682] do_raw_spin_lock+0x177/0x190 [ 2695.110683] _raw_spin_lock+0x21/0x30 [ 2695.110683] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2695.110684] ? do_raw_spin_unlock+0x75/0x190 [ 2695.110685] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2695.110686] ? do_wait_intr+0xf0/0xf0 [ 2695.110687] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2695.110687] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2695.110688] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2695.110689] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2695.110690] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2695.110690] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2695.110691] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2695.110692] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2695.110693] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2695.110694] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2695.110695] lmv_intent_open+0x338/0x1150 [lmv] [ 2695.110696] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2695.110698] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2695.110698] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2695.110699] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2695.110700] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2695.110701] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2695.110702] ? do_raw_spin_unlock+0x75/0x190 [ 2695.110703] ? _raw_spin_unlock+0x12/0x30 [ 2695.110704] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2695.110705] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2695.110706] ll_atomic_open+0x36a/0xd70 [lustre] [ 2695.110707] lookup_open+0x356/0xb40 [ 2695.110707] do_last+0x313/0xfc0 [ 2695.110708] ? nd_jump_root+0xe5/0x160 [ 2695.110709] ? path_init+0x437/0x520 [ 2695.110710] path_openat+0xf7/0x500 [ 2695.110711] ? do_raw_spin_unlock+0x75/0x190 [ 2695.110712] do_filp_open+0x99/0x140 [ 2695.110713] ? getname_flags+0x6e/0x330 [ 2695.110713] ? __check_object_size+0xff/0x256 [ 2695.110714] ? do_raw_spin_unlock+0x75/0x190 [ 2695.110715] ? _raw_spin_unlock+0x12/0x30 [ 2695.110716] do_sys_openat2+0x2b4/0x410 [ 2695.110717] do_sys_open+0x73/0xa0 [ 2695.110718] __x64_sys_openat+0x24/0x30 [ 2695.110719] do_syscall_64+0xc1/0x440 [ 2695.110719] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2695.110720] RIP: 0033:0x7f53425a5332 [ 2695.110722] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2695.110724] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2695.110726] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2695.110728] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2695.110729] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2695.110730] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2695.110731] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2695.112451] NMI backtrace for cpu 2 [ 2695.489464] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2695.494178] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2695.499398] Call Trace: [ 2695.500375] [ 2695.501054] ? dump_stack+0xbb/0x10e [ 2695.503738] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2695.507083] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2695.509507] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2695.512065] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2695.514891] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2695.520961] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2695.524133] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2695.533205] ? update_process_times+0x6d/0xc0 [ 2695.536941] ? tick_sched_handle.isra.18+0x27/0x90 [ 2695.541034] ? tick_sched_timer+0x53/0xb0 [ 2695.546071] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2695.549096] ? tick_sched_handle.isra.18+0x90/0x90 [ 2695.554074] ? hrtimer_interrupt+0x128/0x2c0 [ 2695.565721] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2695.569945] ? apic_timer_interrupt+0xf/0x20 [ 2695.571464] [ 2695.575073] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2695.579235] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2695.586476] ? do_raw_spin_lock+0x177/0x190 [ 2695.591736] ? _raw_spin_lock+0x21/0x30 [ 2695.596975] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2695.607842] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2695.614219] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2695.618012] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2695.629415] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2695.638198] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2695.640190] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2695.651554] ? __next_timer_interrupt+0x160/0x160 [ 2695.656893] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2695.667002] ? do_wait_intr+0xf0/0xf0 [ 2695.676939] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2695.678963] ? kthread+0x1d1/0x200 [ 2695.680166] ? set_kthread_struct+0x70/0x70 [ 2695.697125] ? ret_from_fork+0x1f/0x30 [ 2696.053062] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2696.058749] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2696.078557] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2696.083552] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2696.088964] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2696.093724] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2696.096702] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2696.108920] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2696.111699] RAX: 000000000000284f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2696.115690] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2696.120750] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2696.125502] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2696.127642] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2696.137486] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2696.143153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2696.147000] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2696.151445] Call Trace: [ 2696.152601] [ 2696.159799] ? show_regs.cold.9+0x22/0x2f [ 2696.161911] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2696.165977] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2696.172872] ? watchdog+0x40/0x40 [ 2696.173922] ? hrtimer_interrupt+0x128/0x2c0 [ 2696.177753] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2696.179302] ? apic_timer_interrupt+0xf/0x20 [ 2696.186671] [ 2696.191555] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2696.195305] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2696.206973] ? newidle_balance.isra.93+0x207/0x580 [ 2696.208783] do_raw_spin_lock+0x177/0x190 [ 2696.210126] _raw_spin_lock+0x21/0x30 [ 2696.218686] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2696.223337] process_one_work+0x2c8/0x700 [ 2696.225122] worker_thread+0x296/0x6e0 [ 2696.228795] ? rescuer_thread+0x570/0x570 [ 2696.232577] kthread+0x1d1/0x200 [ 2696.234503] ? set_kthread_struct+0x70/0x70 [ 2696.237951] ret_from_fork+0x1f/0x30 [ 2698.208376] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 61s! [ 2698.217893] Showing busy workqueues and worker pools: [ 2698.220872] workqueue events: flags=0x0 [ 2698.222539] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2698.228815] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 2698.237736] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2698.245983] pending: cpu_latency_work [ptlrpc] [ 2698.248219] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2698.257351] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 2698.262775] workqueue events_long: flags=0x0 [ 2698.265876] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2698.271691] pending: lu_objects_flush [obdclass] [ 2698.275343] workqueue events_unbound: flags=0x2 [ 2698.280962] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2698.288047] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2698.291015] workqueue events_power_efficient: flags=0x80 [ 2698.296061] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2698.299680] pending: check_lifetime, do_cache_clean [ 2698.303233] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2698.306990] pending: neigh_periodic_work, neigh_periodic_work [ 2698.310856] workqueue mm_percpu_wq: flags=0x8 [ 2698.317675] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2698.323694] pending: vmstat_update, lru_add_drain_per_cpu [ 2698.325194] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2698.328014] pending: vmstat_update, lru_add_drain_per_cpu [ 2698.330784] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2698.333648] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2698.336163] workqueue knbd0-recv: flags=0x1a [ 2698.337863] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2698.340550] in-flight: 651:recv_work [ 2698.341738] workqueue ptlrpc_pinger: flags=0x2 [ 2698.343801] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2698.346353] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 2698.348853] pool 8: cpus=0-3 flags=0x4 nice=0 hung=61s workers=4 idle: 15979 31824 8 [ 2698.352308] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2688s workers=2 idle: 654 [ 2716.058516] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 2716.062261] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2716.100675] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2716.107492] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2716.114064] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2716.117345] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2716.137342] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2716.142585] RAX: 0000000000006d9a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2716.148961] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 2716.157592] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2716.161305] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 2716.168886] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 2716.179720] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 2716.191469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2716.199694] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 2716.207786] Call Trace: [ 2716.209502] [ 2716.210226] ? show_regs.cold.9+0x22/0x2f [ 2716.213737] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2716.218835] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2716.229295] ? watchdog+0x40/0x40 [ 2716.236457] ? hrtimer_interrupt+0x128/0x2c0 [ 2716.247455] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2716.256249] ? apic_timer_interrupt+0xf/0x20 [ 2716.262578] [ 2716.272232] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2716.287196] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2716.295501] do_raw_spin_lock+0x177/0x190 [ 2716.308871] _raw_spin_lock+0x21/0x30 [ 2716.328266] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2716.334700] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2716.360749] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2716.372728] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2716.380237] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2716.382360] ? woken_wake_function+0x30/0x30 [ 2716.390803] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2716.402361] kthread+0x1d1/0x200 [ 2716.406354] ? set_kthread_struct+0x70/0x70 [ 2716.412254] ret_from_fork+0x1f/0x30 [ 2720.055074] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 2720.057285] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2720.058226] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 2720.078225] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2720.078240] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2720.084312] Modules linked in: [ 2720.088558] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2720.092225] lustre(O) [ 2720.093330] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2720.095227] osp(O) [ 2720.096306] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 [ 2720.107793] ofd(O) [ 2720.108716] ORIG_RAX: ffffffffffffff13 [ 2720.111360] lod(O) [ 2720.112323] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2720.115458] mdt(O) [ 2720.116322] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2720.130229] mdd(O) [ 2720.131167] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2720.131173] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2720.131176] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2720.131179] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2720.131183] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2720.131185] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2720.131193] Call Trace: [ 2720.131201] [ 2720.131208] ? show_regs.cold.9+0x22/0x2f [ 2720.131220] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2720.131228] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2720.131232] ? watchdog+0x40/0x40 [ 2720.131237] ? hrtimer_interrupt+0x128/0x2c0 [ 2720.131241] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2720.131249] ? apic_timer_interrupt+0xf/0x20 [ 2720.131254] [ 2720.131258] ? native_safe_halt+0x2e/0x50 [ 2720.131262] kvm_wait+0xcd/0xe0 [ 2720.131270] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2720.131279] do_raw_spin_lock+0x177/0x190 [ 2720.131284] _raw_spin_lock+0x21/0x30 [ 2720.131288] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2720.142743] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2720.142832] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2720.142839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2720.142842] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2720.142858] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2720.142863] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2720.142868] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2720.142870] RDX: 0000000000007e32 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2720.142872] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2720.142874] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2720.142877] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2720.142880] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2720.142883] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2720.142885] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2720.142895] Call Trace: [ 2720.142900] [ 2720.142907] ? show_regs.cold.9+0x22/0x2f [ 2720.142919] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2720.142926] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2720.142930] ? watchdog+0x40/0x40 [ 2720.142935] ? hrtimer_interrupt+0x128/0x2c0 [ 2720.142939] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2720.142948] ? apic_timer_interrupt+0xf/0x20 [ 2720.142953] [ 2720.142957] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2720.142962] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2720.142966] do_raw_spin_lock+0x177/0x190 [ 2720.142970] _raw_spin_lock+0x21/0x30 [ 2720.142976] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2720.146448] ? do_raw_spin_unlock+0x75/0x190 [ 2720.152074] obd_mod_free+0x46/0x140 [ptlrpc] [ 2720.152358] mdc_commit_open+0x91/0xb0 [mdc] [ 2720.152397] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2720.153495] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2720.153702] after_reply+0xe2c/0x1780 [ptlrpc] [ 2720.154102] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2720.154306] ? __next_timer_interrupt+0x160/0x160 [ 2720.154316] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2720.154614] ? do_wait_intr+0xf0/0xf0 [ 2720.154624] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2720.164635] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2720.164933] ? do_wait_intr+0xf0/0xf0 [ 2720.164945] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2720.165195] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2720.165408] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2720.165570] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2720.165658] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2720.165687] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2720.165706] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2720.165723] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2720.165778] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2720.165980] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2720.165999] lmv_intent_open+0x338/0x1150 [lmv] [ 2720.166080] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2720.166166] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2720.166249] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2720.169564] kthread+0x1d1/0x200 [ 2720.169584] ? set_kthread_struct+0x70/0x70 [ 2720.169589] ret_from_fork+0x1f/0x30 [ 2720.555813] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2720.561387] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2720.566303] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2720.567901] ? do_raw_spin_unlock+0x75/0x190 [ 2720.578013] ? _raw_spin_unlock+0x12/0x30 [ 2720.586423] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2720.589482] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2720.592336] ll_atomic_open+0x36a/0xd70 [lustre] [ 2720.596069] lookup_open+0x356/0xb40 [ 2720.598447] do_last+0x313/0xfc0 [ 2720.600440] ? nd_jump_root+0xe5/0x160 [ 2720.603228] ? path_init+0x437/0x520 [ 2720.605615] path_openat+0xf7/0x500 [ 2720.607713] ? do_raw_spin_unlock+0x75/0x190 [ 2720.613484] do_filp_open+0x99/0x140 [ 2720.615106] ? getname_flags+0x6e/0x330 [ 2720.616423] ? __check_object_size+0xff/0x256 [ 2720.621485] ? do_raw_spin_unlock+0x75/0x190 [ 2720.622998] ? _raw_spin_unlock+0x12/0x30 [ 2720.632902] do_sys_openat2+0x2b4/0x410 [ 2720.634214] do_sys_open+0x73/0xa0 [ 2720.635405] __x64_sys_openat+0x24/0x30 [ 2720.637708] do_syscall_64+0xc1/0x440 [ 2720.638987] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2720.645638] RIP: 0033:0x7f53425a5332 [ 2720.646932] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2720.654722] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2720.657401] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2720.659294] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2720.668265] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2720.671674] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2720.677391] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2724.052307] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2724.060930] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2724.094668] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2724.109054] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2724.114608] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2724.119868] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2724.124465] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2724.139097] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2724.141963] RAX: 0000000000004b7b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2724.145273] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2724.148098] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2724.151271] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2724.154373] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2724.158739] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2724.162705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2724.165195] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2724.168158] Call Trace: [ 2724.169909] [ 2724.170541] ? show_regs.cold.9+0x22/0x2f [ 2724.172278] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2724.176887] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2724.178679] ? watchdog+0x40/0x40 [ 2724.180937] ? hrtimer_interrupt+0x128/0x2c0 [ 2724.182494] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2724.185503] ? apic_timer_interrupt+0xf/0x20 [ 2724.186655] [ 2724.187733] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2724.190366] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2724.192455] ? newidle_balance.isra.93+0x207/0x580 [ 2724.194773] do_raw_spin_lock+0x177/0x190 [ 2724.198157] _raw_spin_lock+0x21/0x30 [ 2724.199180] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2724.201762] process_one_work+0x2c8/0x700 [ 2724.204483] worker_thread+0x296/0x6e0 [ 2724.205816] ? rescuer_thread+0x570/0x570 [ 2724.211215] kthread+0x1d1/0x200 [ 2724.213648] ? set_kthread_struct+0x70/0x70 [ 2724.216806] ret_from_fork+0x1f/0x30 [ 2728.418950] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 91s! [ 2728.428297] Showing busy workqueues and worker pools: [ 2728.432431] workqueue events: flags=0x0 [ 2728.438444] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2728.442257] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 2728.454498] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2728.460839] pending: cpu_latency_work [ptlrpc] [ 2728.466361] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2728.476192] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 2728.478466] workqueue events_long: flags=0x0 [ 2728.490783] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2728.496656] pending: lu_objects_flush [obdclass] [ 2728.504527] workqueue events_unbound: flags=0x2 [ 2728.506909] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2728.514106] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2728.519975] workqueue events_power_efficient: flags=0x80 [ 2728.524612] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2728.533827] pending: check_lifetime, do_cache_clean [ 2728.542573] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2728.549388] pending: neigh_periodic_work, neigh_periodic_work [ 2728.555605] workqueue mm_percpu_wq: flags=0x8 [ 2728.571371] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2728.577136] pending: vmstat_update, lru_add_drain_per_cpu [ 2728.580978] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2728.583241] pending: vmstat_update, lru_add_drain_per_cpu [ 2728.594516] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2728.599203] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2728.615882] workqueue knbd0-recv: flags=0x1a [ 2728.617381] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2728.626336] in-flight: 651:recv_work [ 2728.632556] workqueue ptlrpc_pinger: flags=0x2 [ 2728.639667] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2728.650933] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 2728.657612] pool 8: cpus=0-3 flags=0x4 nice=0 hung=91s workers=4 idle: 15979 31824 8 [ 2728.671428] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2719s workers=2 idle: 654 [ 2744.058773] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 2744.065014] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2744.091822] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2744.100298] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2744.106175] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2744.108043] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2744.124955] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2744.131806] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2744.141479] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 2744.152995] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2744.162015] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 2744.165384] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 2744.167880] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 2744.174914] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2744.180885] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 2744.184746] Call Trace: [ 2744.186008] [ 2744.186929] ? show_regs.cold.9+0x22/0x2f [ 2744.190318] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2744.192398] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2744.194718] ? watchdog+0x40/0x40 [ 2744.195886] ? hrtimer_interrupt+0x128/0x2c0 [ 2744.197950] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2744.199655] ? apic_timer_interrupt+0xf/0x20 [ 2744.208322] [ 2744.210916] ? native_safe_halt+0x2e/0x50 [ 2744.216573] kvm_wait+0xcd/0xe0 [ 2744.217750] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2744.223313] do_raw_spin_lock+0x177/0x190 [ 2744.224893] _raw_spin_lock+0x21/0x30 [ 2744.226887] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2744.235011] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2744.241042] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2744.247823] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2744.259018] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2744.273317] ? woken_wake_function+0x30/0x30 [ 2744.280384] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2744.286773] kthread+0x1d1/0x200 [ 2744.288167] ? set_kthread_struct+0x70/0x70 [ 2744.290871] ret_from_fork+0x1f/0x30 [ 2748.057137] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 2748.058189] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 2748.065774] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2748.065852] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2748.065861] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2748.065864] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2748.065880] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2748.065884] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2748.065889] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2748.065891] RDX: 00000000000067c0 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2748.065894] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2748.065895] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2748.065898] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2748.065901] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2748.065904] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2748.065906] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2748.065915] Call Trace: [ 2748.065921] [ 2748.065926] ? show_regs.cold.9+0x22/0x2f [ 2748.065936] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2748.065942] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2748.065947] ? watchdog+0x40/0x40 [ 2748.065951] ? hrtimer_interrupt+0x128/0x2c0 [ 2748.065956] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2748.065965] ? apic_timer_interrupt+0xf/0x20 [ 2748.065969] [ 2748.065972] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2748.065977] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2748.065981] do_raw_spin_lock+0x177/0x190 [ 2748.065986] _raw_spin_lock+0x21/0x30 [ 2748.065992] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2748.066257] obd_mod_free+0x46/0x140 [ptlrpc] [ 2748.067129] mdc_commit_open+0x91/0xb0 [mdc] [ 2748.067167] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2748.067614] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2748.068598] after_reply+0xe2c/0x1780 [ptlrpc] [ 2748.068922] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2748.069729] ? __next_timer_interrupt+0x160/0x160 [ 2748.069742] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2748.069946] ? do_wait_intr+0xf0/0xf0 [ 2748.069957] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2748.074765] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2748.075123] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2748.075131] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2748.075133] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2748.075148] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2748.075153] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2748.075158] RAX: 0000000000007a91 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2748.075160] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2748.075162] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2748.075164] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2748.075166] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2748.075169] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2748.075171] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2748.075173] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2748.075182] Call Trace: [ 2748.075189] [ 2748.075195] ? show_regs.cold.9+0x22/0x2f [ 2748.075206] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2748.075213] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2748.075217] ? watchdog+0x40/0x40 [ 2748.075222] ? hrtimer_interrupt+0x128/0x2c0 [ 2748.075225] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2748.075234] ? apic_timer_interrupt+0xf/0x20 [ 2748.075240] [ 2748.075242] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2748.075249] do_raw_spin_lock+0x177/0x190 [ 2748.075254] _raw_spin_lock+0x21/0x30 [ 2748.075258] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2748.123718] kthread+0x1d1/0x200 [ 2748.123739] ? set_kthread_struct+0x70/0x70 [ 2748.123744] ret_from_fork+0x1f/0x30 [ 2748.444579] ? do_raw_spin_unlock+0x75/0x190 [ 2748.446129] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2748.448013] ? do_wait_intr+0xf0/0xf0 [ 2748.449194] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2748.451280] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2748.460159] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2748.463461] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2748.464700] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2748.469428] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2748.470932] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2748.472300] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2748.474786] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2748.477010] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2748.479162] lmv_intent_open+0x338/0x1150 [lmv] [ 2748.481543] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2748.492064] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2748.500005] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2748.515235] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2748.522568] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2748.532979] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2748.538277] ? do_raw_spin_unlock+0x75/0x190 [ 2748.541214] ? _raw_spin_unlock+0x12/0x30 [ 2748.548470] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2748.554295] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2748.557424] ll_atomic_open+0x36a/0xd70 [lustre] [ 2748.563350] lookup_open+0x356/0xb40 [ 2748.569590] do_last+0x313/0xfc0 [ 2748.571847] ? nd_jump_root+0xe5/0x160 [ 2748.578275] ? path_init+0x437/0x520 [ 2748.580085] path_openat+0xf7/0x500 [ 2748.581421] ? do_raw_spin_unlock+0x75/0x190 [ 2748.585689] do_filp_open+0x99/0x140 [ 2748.588394] ? getname_flags+0x6e/0x330 [ 2748.590898] ? __check_object_size+0xff/0x256 [ 2748.594255] ? do_raw_spin_unlock+0x75/0x190 [ 2748.596890] ? _raw_spin_unlock+0x12/0x30 [ 2748.598955] do_sys_openat2+0x2b4/0x410 [ 2748.601027] do_sys_open+0x73/0xa0 [ 2748.602941] __x64_sys_openat+0x24/0x30 [ 2748.604960] do_syscall_64+0xc1/0x440 [ 2748.611194] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2748.613064] RIP: 0033:0x7f53425a5332 [ 2748.615647] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2748.637864] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2748.643540] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2748.648722] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2748.652537] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2748.655067] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2748.657232] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2756.052126] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2756.060641] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2756.101965] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2756.113096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2756.123648] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2756.129465] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2756.132669] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2756.151731] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2756.156017] RAX: 0000000000005a05 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2756.165608] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2756.175129] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2756.187175] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2756.192962] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2756.202630] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2756.208792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2756.212774] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2756.217642] Call Trace: [ 2756.218791] [ 2756.222119] ? show_regs.cold.9+0x22/0x2f [ 2756.224568] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2756.228069] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2756.230504] ? watchdog+0x40/0x40 [ 2756.231989] ? hrtimer_interrupt+0x128/0x2c0 [ 2756.235784] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2756.238577] ? apic_timer_interrupt+0xf/0x20 [ 2756.241713] [ 2756.242527] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2756.245662] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2756.247696] ? newidle_balance.isra.93+0x207/0x580 [ 2756.253088] do_raw_spin_lock+0x177/0x190 [ 2756.254975] _raw_spin_lock+0x21/0x30 [ 2756.257639] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2756.261536] process_one_work+0x2c8/0x700 [ 2756.266041] worker_thread+0x296/0x6e0 [ 2756.270916] ? rescuer_thread+0x570/0x570 [ 2756.272389] kthread+0x1d1/0x200 [ 2756.282515] ? set_kthread_struct+0x70/0x70 [ 2756.284027] ret_from_fork+0x1f/0x30 [ 2759.137336] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 122s! [ 2759.149939] Showing busy workqueues and worker pools: [ 2759.153566] workqueue events: flags=0x0 [ 2759.162105] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2759.172807] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 2759.181273] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2759.185258] pending: cpu_latency_work [ptlrpc] [ 2759.189594] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2759.199233] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 2759.207485] workqueue events_long: flags=0x0 [ 2759.209025] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2759.220368] pending: lu_objects_flush [obdclass] [ 2759.230087] workqueue events_unbound: flags=0x2 [ 2759.231638] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2759.246547] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2759.263675] workqueue events_power_efficient: flags=0x80 [ 2759.274242] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2759.286807] pending: check_lifetime, do_cache_clean [ 2759.294689] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2759.302572] pending: neigh_periodic_work, neigh_periodic_work [ 2759.309268] workqueue mm_percpu_wq: flags=0x8 [ 2759.310868] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2759.314930] pending: vmstat_update, lru_add_drain_per_cpu [ 2759.318154] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2759.321692] pending: vmstat_update, lru_add_drain_per_cpu [ 2759.324367] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2759.327300] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2759.330583] workqueue ipv6_addrconf: flags=0x40008 [ 2759.332594] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2759.335134] pending: addrconf_verify_work [ 2759.336434] workqueue knbd0-recv: flags=0x1a [ 2759.337628] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2759.339664] in-flight: 651:recv_work [ 2759.340815] workqueue ptlrpc_pinger: flags=0x2 [ 2759.342171] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2759.344041] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 2759.346165] pool 8: cpus=0-3 flags=0x4 nice=0 hung=122s workers=4 idle: 15979 31824 8 [ 2759.349402] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2749s workers=2 idle: 654 [ 2772.059624] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 2772.067004] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2772.107296] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2772.120564] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2772.129595] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2772.131838] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2772.150359] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2772.162131] RAX: 000000000000752f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2772.168167] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 2772.173220] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2772.177397] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 2772.180826] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 2772.184085] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 2772.188716] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2772.192647] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 2772.196355] Call Trace: [ 2772.197614] [ 2772.198760] ? show_regs.cold.9+0x22/0x2f [ 2772.202602] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2772.207845] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2772.209905] ? watchdog+0x40/0x40 [ 2772.214597] ? hrtimer_interrupt+0x128/0x2c0 [ 2772.222854] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2772.227866] ? apic_timer_interrupt+0xf/0x20 [ 2772.232579] [ 2772.235831] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2772.238549] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2772.240399] do_raw_spin_lock+0x177/0x190 [ 2772.242122] _raw_spin_lock+0x21/0x30 [ 2772.243956] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2772.246492] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2772.253126] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2772.255631] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2772.257778] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2772.259952] ? woken_wake_function+0x30/0x30 [ 2772.261246] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2772.266688] kthread+0x1d1/0x200 [ 2772.268507] ? set_kthread_struct+0x70/0x70 [ 2772.271104] ret_from_fork+0x1f/0x30 [ 2776.059489] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 2776.059490] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 2776.059506] Modules linked in: lustre(O) [ 2776.063122] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2776.063200] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2776.063207] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2776.063210] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2776.063224] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2776.063229] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2776.063233] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2776.063236] RDX: 0000000000007f09 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2776.063238] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2776.063241] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2776.063242] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2776.063245] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2776.063248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2776.063250] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2776.063258] Call Trace: [ 2776.063264] [ 2776.063270] ? show_regs.cold.9+0x22/0x2f [ 2776.063280] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2776.063286] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2776.063292] ? watchdog+0x40/0x40 [ 2776.063296] ? hrtimer_interrupt+0x128/0x2c0 [ 2776.063301] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2776.063309] ? apic_timer_interrupt+0xf/0x20 [ 2776.063458] [ 2776.063461] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2776.063467] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2776.063471] do_raw_spin_lock+0x177/0x190 [ 2776.063476] _raw_spin_lock+0x21/0x30 [ 2776.063483] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2776.063761] obd_mod_free+0x46/0x140 [ptlrpc] [ 2776.070233] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2776.070643] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2776.070651] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2776.070654] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2776.070669] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2776.070674] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2776.070679] RAX: 0000000000007d84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2776.070681] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2776.070683] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2776.070685] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2776.070687] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2776.070690] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2776.070693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2776.070695] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2776.070704] Call Trace: [ 2776.070709] [ 2776.070715] ? show_regs.cold.9+0x22/0x2f [ 2776.070724] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2776.070730] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2776.070735] ? watchdog+0x40/0x40 [ 2776.070739] ? hrtimer_interrupt+0x128/0x2c0 [ 2776.070744] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2776.070753] ? apic_timer_interrupt+0xf/0x20 [ 2776.070757] [ 2776.070816] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2776.070825] do_raw_spin_lock+0x177/0x190 [ 2776.070830] _raw_spin_lock+0x21/0x30 [ 2776.070834] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2776.071145] ? do_raw_spin_unlock+0x75/0x190 [ 2776.071152] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2776.071866] ? do_wait_intr+0xf0/0xf0 [ 2776.071876] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2776.072096] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2776.072654] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2776.074660] mdc_commit_open+0x91/0xb0 [mdc] [ 2776.074710] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2776.074981] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2776.075600] after_reply+0xe2c/0x1780 [ptlrpc] [ 2776.076037] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2776.076254] ? __next_timer_interrupt+0x160/0x160 [ 2776.076265] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2776.076647] ? do_wait_intr+0xf0/0xf0 [ 2776.076657] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2776.077046] kthread+0x1d1/0x200 [ 2776.077053] ? set_kthread_struct+0x70/0x70 [ 2776.077058] ret_from_fork+0x1f/0x30 [ 2776.468211] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2776.470429] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2776.472153] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2776.473631] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2776.474981] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2776.476570] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2776.479767] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2776.481078] lmv_intent_open+0x338/0x1150 [lmv] [ 2776.481814] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2776.483224] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2776.484625] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2776.485801] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2776.486994] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2776.488230] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2776.489474] ? do_raw_spin_unlock+0x75/0x190 [ 2776.490475] ? _raw_spin_unlock+0x12/0x30 [ 2776.491396] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2776.492663] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2776.493817] ll_atomic_open+0x36a/0xd70 [lustre] [ 2776.495016] lookup_open+0x356/0xb40 [ 2776.495905] do_last+0x313/0xfc0 [ 2776.496633] ? nd_jump_root+0xe5/0x160 [ 2776.497590] ? path_init+0x437/0x520 [ 2776.498417] path_openat+0xf7/0x500 [ 2776.499237] ? do_raw_spin_unlock+0x75/0x190 [ 2776.500395] do_filp_open+0x99/0x140 [ 2776.501390] ? getname_flags+0x6e/0x330 [ 2776.502275] ? __check_object_size+0xff/0x256 [ 2776.503284] ? do_raw_spin_unlock+0x75/0x190 [ 2776.504309] ? _raw_spin_unlock+0x12/0x30 [ 2776.505327] do_sys_openat2+0x2b4/0x410 [ 2776.507135] do_sys_open+0x73/0xa0 [ 2776.508590] __x64_sys_openat+0x24/0x30 [ 2776.510243] do_syscall_64+0xc1/0x440 [ 2776.511725] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2776.513582] RIP: 0033:0x7f53425a5332 [ 2776.515223] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2776.523085] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2776.526039] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2776.528948] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2776.531902] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2776.534530] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2776.537081] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2784.052057] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2784.058173] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2784.098474] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2784.104595] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2784.109449] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2784.112946] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2784.119385] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2784.146062] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2784.149598] RAX: 0000000000005e8b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2784.154156] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2784.160199] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2784.164074] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2784.171535] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2784.178043] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2784.183177] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2784.189619] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2784.201745] Call Trace: [ 2784.202627] [ 2784.205026] ? show_regs.cold.9+0x22/0x2f [ 2784.210343] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2784.214293] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2784.217292] ? watchdog+0x40/0x40 [ 2784.219725] ? hrtimer_interrupt+0x128/0x2c0 [ 2784.222307] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2784.230557] ? apic_timer_interrupt+0xf/0x20 [ 2784.247371] [ 2784.248353] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2784.256327] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2784.267313] ? newidle_balance.isra.93+0x207/0x580 [ 2784.274405] do_raw_spin_lock+0x177/0x190 [ 2784.277282] _raw_spin_lock+0x21/0x30 [ 2784.283262] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2784.300817] process_one_work+0x2c8/0x700 [ 2784.312232] worker_thread+0x296/0x6e0 [ 2784.314348] ? rescuer_thread+0x570/0x570 [ 2784.326429] kthread+0x1d1/0x200 [ 2784.336192] ? set_kthread_struct+0x70/0x70 [ 2784.347933] ret_from_fork+0x1f/0x30 [ 2789.873065] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 152s! [ 2789.879648] Showing busy workqueues and worker pools: [ 2789.888028] workqueue events: flags=0x0 [ 2789.890278] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2789.897563] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 2789.909476] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2789.919550] pending: cpu_latency_work [ptlrpc] [ 2789.924374] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2789.932183] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 2789.940854] workqueue events_long: flags=0x0 [ 2789.945468] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2789.953338] pending: lu_objects_flush [obdclass] [ 2789.959258] workqueue events_unbound: flags=0x2 [ 2789.964794] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2789.968347] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2789.979890] workqueue events_power_efficient: flags=0x80 [ 2789.990097] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2790.001593] pending: check_lifetime, do_cache_clean [ 2790.004825] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2790.017878] pending: neigh_periodic_work, neigh_periodic_work [ 2790.028176] workqueue mm_percpu_wq: flags=0x8 [ 2790.034959] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2790.049140] pending: vmstat_update, lru_add_drain_per_cpu [ 2790.051860] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2790.064219] pending: vmstat_update, lru_add_drain_per_cpu [ 2790.068651] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2790.075268] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2790.079189] workqueue ipv6_addrconf: flags=0x40008 [ 2790.081379] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2790.084570] pending: addrconf_verify_work [ 2790.089085] workqueue knbd0-recv: flags=0x1a [ 2790.091213] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2790.096453] in-flight: 651:recv_work [ 2790.104310] workqueue ptlrpc_pinger: flags=0x2 [ 2790.106940] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2790.124334] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 2790.141080] pool 8: cpus=0-3 flags=0x4 nice=0 hung=153s workers=4 idle: 15979 31824 8 [ 2790.144824] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2780s workers=2 idle: 654 [ 2800.058697] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 2800.061570] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2800.081208] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2800.086834] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2800.089796] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2800.094676] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2800.111154] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2800.118132] RAX: 0000000000002174 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2800.129067] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 2800.135148] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2800.143560] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 2800.147598] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 2800.150989] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 2800.153832] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2800.155978] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 2800.158346] Call Trace: [ 2800.159837] [ 2800.160447] ? show_regs.cold.9+0x22/0x2f [ 2800.165571] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2800.169727] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2800.171235] ? watchdog+0x40/0x40 [ 2800.173165] ? hrtimer_interrupt+0x128/0x2c0 [ 2800.174246] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2800.176341] ? apic_timer_interrupt+0xf/0x20 [ 2800.179278] [ 2800.181486] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2800.183248] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2800.184873] do_raw_spin_lock+0x177/0x190 [ 2800.188345] _raw_spin_lock+0x21/0x30 [ 2800.190179] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2800.193154] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2800.195631] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2800.198104] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2800.203801] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2800.209053] ? woken_wake_function+0x30/0x30 [ 2800.211089] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2800.213251] kthread+0x1d1/0x200 [ 2800.214143] ? set_kthread_struct+0x70/0x70 [ 2800.215783] ret_from_fork+0x1f/0x30 [ 2804.055023] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 2804.059256] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2804.062842] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 2804.112630] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2804.112646] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2804.112648] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2804.112661] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2804.112665] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2804.112670] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2804.112673] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2804.112675] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2804.112676] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2804.112679] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2804.112681] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2804.112684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2804.112686] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2804.112694] Call Trace: [ 2804.112701] [ 2804.112706] ? show_regs.cold.9+0x22/0x2f [ 2804.112717] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2804.112724] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2804.112728] ? watchdog+0x40/0x40 [ 2804.112732] ? hrtimer_interrupt+0x128/0x2c0 [ 2804.112736] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2804.112742] ? apic_timer_interrupt+0xf/0x20 [ 2804.112746] [ 2804.112749] ? native_safe_halt+0x2e/0x50 [ 2804.112752] kvm_wait+0xcd/0xe0 [ 2804.112761] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2804.112845] do_raw_spin_lock+0x177/0x190 [ 2804.112852] _raw_spin_lock+0x21/0x30 [ 2804.112857] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2804.113463] ? do_raw_spin_unlock+0x75/0x190 [ 2804.113469] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2804.113749] ? do_wait_intr+0xf0/0xf0 [ 2804.113759] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2804.114027] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2804.114877] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2804.115117] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2804.115916] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2804.115955] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2804.115986] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2804.116014] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2804.123675] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2804.123752] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2804.123760] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2804.123763] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2804.134597] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2804.134901] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2804.134942] lmv_intent_open+0x338/0x1150 [lmv] [ 2804.134975] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2804.135085] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2804.152689] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2804.152704] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2804.152711] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2804.152713] RDX: 0000000000002000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2804.152715] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2804.152718] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2804.152720] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2804.152723] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2804.152726] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2804.152728] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2804.152736] Call Trace: [ 2804.152742] [ 2804.152747] ? show_regs.cold.9+0x22/0x2f [ 2804.152761] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2804.152767] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2804.152771] ? watchdog+0x40/0x40 [ 2804.152776] ? hrtimer_interrupt+0x128/0x2c0 [ 2804.152780] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2804.152789] ? apic_timer_interrupt+0xf/0x20 [ 2804.152794] [ 2804.152796] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2804.152802] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2804.152806] do_raw_spin_lock+0x177/0x190 [ 2804.152810] _raw_spin_lock+0x21/0x30 [ 2804.152817] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2804.153095] obd_mod_free+0x46/0x140 [ptlrpc] [ 2804.153290] mdc_commit_open+0x91/0xb0 [mdc] [ 2804.153327] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2804.153542] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2804.153749] after_reply+0xe2c/0x1780 [ptlrpc] [ 2804.153941] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2804.154131] ? __next_timer_interrupt+0x160/0x160 [ 2804.154140] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2804.154339] ? do_wait_intr+0xf0/0xf0 [ 2804.154348] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2804.154568] kthread+0x1d1/0x200 [ 2804.154573] ? set_kthread_struct+0x70/0x70 [ 2804.154577] ret_from_fork+0x1f/0x30 [ 2804.574210] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2804.577985] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2804.581111] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2804.584224] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2804.588215] ? do_raw_spin_unlock+0x75/0x190 [ 2804.591613] ? _raw_spin_unlock+0x12/0x30 [ 2804.593973] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2804.599722] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2804.601771] ll_atomic_open+0x36a/0xd70 [lustre] [ 2804.608594] lookup_open+0x356/0xb40 [ 2804.610181] do_last+0x313/0xfc0 [ 2804.612200] ? nd_jump_root+0xe5/0x160 [ 2804.615325] ? path_init+0x437/0x520 [ 2804.617264] path_openat+0xf7/0x500 [ 2804.618607] ? do_raw_spin_unlock+0x75/0x190 [ 2804.619959] do_filp_open+0x99/0x140 [ 2804.622723] ? getname_flags+0x6e/0x330 [ 2804.626751] ? __check_object_size+0xff/0x256 [ 2804.630449] ? do_raw_spin_unlock+0x75/0x190 [ 2804.632107] ? _raw_spin_unlock+0x12/0x30 [ 2804.636010] do_sys_openat2+0x2b4/0x410 [ 2804.639217] do_sys_open+0x73/0xa0 [ 2804.640755] __x64_sys_openat+0x24/0x30 [ 2804.642090] do_syscall_64+0xc1/0x440 [ 2804.646051] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2804.648335] RIP: 0033:0x7f53425a5332 [ 2804.649790] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2804.664146] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2804.670335] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2804.674324] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2804.678141] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2804.686715] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2804.691536] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2816.052008] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2816.054563] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2816.079489] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2816.085837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2816.095651] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2816.100594] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2816.104538] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2816.115320] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2816.128560] RAX: 000000000000685d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2816.132724] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2816.138155] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2816.145562] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2816.152576] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2816.159751] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2816.163736] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2816.171766] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2816.181461] Call Trace: [ 2816.183626] [ 2816.185360] ? show_regs.cold.9+0x22/0x2f [ 2816.186977] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2816.191465] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2816.193140] ? watchdog+0x40/0x40 [ 2816.198437] ? hrtimer_interrupt+0x128/0x2c0 [ 2816.202066] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2816.207101] ? apic_timer_interrupt+0xf/0x20 [ 2816.210741] [ 2816.211793] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2816.217764] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2816.223844] ? newidle_balance.isra.93+0x207/0x580 [ 2816.228746] do_raw_spin_lock+0x177/0x190 [ 2816.230826] _raw_spin_lock+0x21/0x30 [ 2816.233518] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2816.239870] process_one_work+0x2c8/0x700 [ 2816.242344] worker_thread+0x296/0x6e0 [ 2816.248471] ? rescuer_thread+0x570/0x570 [ 2816.253514] kthread+0x1d1/0x200 [ 2816.255664] ? set_kthread_struct+0x70/0x70 [ 2816.257811] ret_from_fork+0x1f/0x30 [ 2820.575130] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 183s! [ 2820.579664] Showing busy workqueues and worker pools: [ 2820.582475] workqueue events: flags=0x0 [ 2820.584656] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2820.590781] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 2820.601729] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2820.605531] pending: cpu_latency_work [ptlrpc] [ 2820.608489] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2820.612655] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 2820.616759] workqueue events_long: flags=0x0 [ 2820.618882] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2820.622376] pending: lu_objects_flush [obdclass] [ 2820.628607] workqueue events_unbound: flags=0x2 [ 2820.633835] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2820.641023] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2820.647424] workqueue events_power_efficient: flags=0x80 [ 2820.650579] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2820.653622] pending: check_lifetime, do_cache_clean [ 2820.660638] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2820.663943] pending: neigh_periodic_work, neigh_periodic_work [ 2820.668021] workqueue mm_percpu_wq: flags=0x8 [ 2820.669985] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2820.679597] pending: vmstat_update, lru_add_drain_per_cpu [ 2820.684642] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2820.694058] pending: vmstat_update, lru_add_drain_per_cpu [ 2820.696204] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2820.703954] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2820.707792] workqueue ipv6_addrconf: flags=0x40008 [ 2820.711551] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2820.715860] pending: addrconf_verify_work [ 2820.722575] workqueue knbd0-recv: flags=0x1a [ 2820.731545] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2820.735896] in-flight: 651:recv_work [ 2820.742461] workqueue ptlrpc_pinger: flags=0x2 [ 2820.748958] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2820.754444] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 2820.757918] pool 8: cpus=0-3 flags=0x4 nice=0 hung=183s workers=4 idle: 15979 31824 8 [ 2820.761835] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2811s workers=2 idle: 654 [ 2828.059254] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 2828.073395] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2828.137477] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2828.154654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2828.176868] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2828.186875] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2828.221446] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2828.225430] RAX: 0000000000007299 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2828.232690] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 2828.246612] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2828.257610] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 2828.269848] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 2828.285319] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 2828.287230] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2828.303841] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 2828.311342] Call Trace: [ 2828.313300] [ 2828.314282] ? show_regs.cold.9+0x22/0x2f [ 2828.317299] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2828.320813] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2828.326377] ? watchdog+0x40/0x40 [ 2828.332565] ? hrtimer_interrupt+0x128/0x2c0 [ 2828.341789] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2828.351777] ? apic_timer_interrupt+0xf/0x20 [ 2828.354518] [ 2828.355302] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2828.369641] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2828.381325] do_raw_spin_lock+0x177/0x190 [ 2828.385583] _raw_spin_lock+0x21/0x30 [ 2828.390393] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2828.395737] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2828.397815] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2828.405306] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2828.409731] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2828.416509] ? woken_wake_function+0x30/0x30 [ 2828.419926] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2828.427758] kthread+0x1d1/0x200 [ 2828.429594] ? set_kthread_struct+0x70/0x70 [ 2828.431106] ret_from_fork+0x1f/0x30 [ 2832.055013] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 2832.058226] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2832.065424] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 2832.082729] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2832.082744] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2832.082747] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2832.082761] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2832.082765] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2832.082771] RAX: 0000000000005791 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2832.082773] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2832.082775] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2832.082777] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2832.082780] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2832.082783] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2832.082786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2832.082788] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2832.082796] Call Trace: [ 2832.082802] [ 2832.082808] ? show_regs.cold.9+0x22/0x2f [ 2832.082817] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2832.082823] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2832.082827] ? watchdog+0x40/0x40 [ 2832.082832] ? hrtimer_interrupt+0x128/0x2c0 [ 2832.082836] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2832.082845] ? apic_timer_interrupt+0xf/0x20 [ 2832.082849] [ 2832.082852] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2832.082860] do_raw_spin_lock+0x177/0x190 [ 2832.082865] _raw_spin_lock+0x21/0x30 [ 2832.082939] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2832.083220] ? do_raw_spin_unlock+0x75/0x190 [ 2832.083226] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2832.089218] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2832.089295] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2832.089302] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2832.089304] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2832.089317] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2832.089320] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2832.089325] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2832.089327] RDX: 0000000000007af9 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2832.089329] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2832.089331] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2832.089333] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2832.089336] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2832.089338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2832.089341] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2832.089349] Call Trace: [ 2832.089355] [ 2832.089360] ? show_regs.cold.9+0x22/0x2f [ 2832.089368] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2832.089375] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2832.089379] ? watchdog+0x40/0x40 [ 2832.089383] ? hrtimer_interrupt+0x128/0x2c0 [ 2832.089388] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2832.089397] ? apic_timer_interrupt+0xf/0x20 [ 2832.089402] [ 2832.089405] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2832.089410] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2832.089414] do_raw_spin_lock+0x177/0x190 [ 2832.089418] _raw_spin_lock+0x21/0x30 [ 2832.089424] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2832.089974] obd_mod_free+0x46/0x140 [ptlrpc] [ 2832.090241] mdc_commit_open+0x91/0xb0 [mdc] [ 2832.090279] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2832.090541] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2832.090838] after_reply+0xe2c/0x1780 [ptlrpc] [ 2832.096487] ? do_wait_intr+0xf0/0xf0 [ 2832.096509] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2832.096792] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2832.097252] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2832.097478] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2832.097589] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2832.097627] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2832.097654] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2832.097695] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2832.097778] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2832.098227] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2832.098260] lmv_intent_open+0x338/0x1150 [lmv] [ 2832.098288] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2832.098374] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2832.098456] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2832.098476] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2832.098559] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2832.098641] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2832.098798] ? do_raw_spin_unlock+0x75/0x190 [ 2832.098806] ? _raw_spin_unlock+0x12/0x30 [ 2832.098813] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2832.098839] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2832.098861] ll_atomic_open+0x36a/0xd70 [lustre] [ 2832.100494] lookup_open+0x356/0xb40 [ 2832.100505] do_last+0x313/0xfc0 [ 2832.100510] ? nd_jump_root+0xe5/0x160 [ 2832.100514] ? path_init+0x437/0x520 [ 2832.100518] path_openat+0xf7/0x500 [ 2832.100523] ? do_raw_spin_unlock+0x75/0x190 [ 2832.100529] do_filp_open+0x99/0x140 [ 2832.100535] ? getname_flags+0x6e/0x330 [ 2832.100540] ? __check_object_size+0xff/0x256 [ 2832.100548] ? do_raw_spin_unlock+0x75/0x190 [ 2832.100552] ? _raw_spin_unlock+0x12/0x30 [ 2832.100558] do_sys_openat2+0x2b4/0x410 [ 2832.100563] do_sys_open+0x73/0xa0 [ 2832.100567] __x64_sys_openat+0x24/0x30 [ 2832.100571] do_syscall_64+0xc1/0x440 [ 2832.100577] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2832.100583] RIP: 0033:0x7f53425a5332 [ 2832.100591] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2832.100595] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2832.100601] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2832.100603] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2832.100605] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2832.100607] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2832.100609] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2832.401611] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2832.404388] ? __next_timer_interrupt+0x160/0x160 [ 2832.407093] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2832.409613] ? do_wait_intr+0xf0/0xf0 [ 2832.411791] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2832.415231] kthread+0x1d1/0x200 [ 2832.417167] ? set_kthread_struct+0x70/0x70 [ 2832.419613] ret_from_fork+0x1f/0x30 [ 2848.052023] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2848.056783] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2848.082576] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2848.087543] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2848.091975] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2848.095084] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2848.097263] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2848.105455] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 2848.108751] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2848.111863] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2848.115149] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2848.118094] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2848.121095] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2848.124334] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2848.127971] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2848.133982] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2848.136326] Call Trace: [ 2848.138040] [ 2848.138607] ? show_regs.cold.9+0x22/0x2f [ 2848.142598] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2848.145695] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2848.147658] ? watchdog+0x40/0x40 [ 2848.149023] ? hrtimer_interrupt+0x128/0x2c0 [ 2848.151272] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2848.153304] ? apic_timer_interrupt+0xf/0x20 [ 2848.155051] [ 2848.155681] ? native_safe_halt+0x2e/0x50 [ 2848.157475] kvm_wait+0xcd/0xe0 [ 2848.158786] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2848.160826] ? newidle_balance.isra.93+0x207/0x580 [ 2848.163722] do_raw_spin_lock+0x177/0x190 [ 2848.165311] _raw_spin_lock+0x21/0x30 [ 2848.166879] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2848.170269] process_one_work+0x2c8/0x700 [ 2848.172353] worker_thread+0x296/0x6e0 [ 2848.176354] ? rescuer_thread+0x570/0x570 [ 2848.178683] kthread+0x1d1/0x200 [ 2848.179930] ? set_kthread_struct+0x70/0x70 [ 2848.181870] ret_from_fork+0x1f/0x30 [ 2850.786030] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 213s! [ 2850.793336] Showing busy workqueues and worker pools: [ 2850.796940] workqueue events: flags=0x0 [ 2850.798355] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2850.805227] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 2850.810514] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2850.813006] pending: cpu_latency_work [ptlrpc] [ 2850.816022] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2850.822785] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 2850.827944] workqueue events_long: flags=0x0 [ 2850.829967] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2850.832859] pending: lu_objects_flush [obdclass] [ 2850.834992] workqueue events_unbound: flags=0x2 [ 2850.836658] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2850.839852] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2850.846603] workqueue events_power_efficient: flags=0x80 [ 2850.849145] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2850.856985] pending: check_lifetime, do_cache_clean [ 2850.859503] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2850.863217] pending: neigh_periodic_work, neigh_periodic_work [ 2850.867841] workqueue mm_percpu_wq: flags=0x8 [ 2850.869865] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2850.873977] pending: vmstat_update, lru_add_drain_per_cpu [ 2850.876602] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2850.879380] pending: vmstat_update, lru_add_drain_per_cpu [ 2850.881828] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2850.885131] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2850.889686] workqueue ipv6_addrconf: flags=0x40008 [ 2850.892030] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2850.897871] pending: addrconf_verify_work [ 2850.900429] workqueue knbd0-recv: flags=0x1a [ 2850.901836] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2850.904042] in-flight: 651:recv_work [ 2850.905481] workqueue ptlrpc_pinger: flags=0x2 [ 2850.908461] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2850.914097] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 2850.917467] pool 8: cpus=0-3 flags=0x4 nice=0 hung=213s workers=4 idle: 15979 31824 8 [ 2850.922861] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2841s workers=2 idle: 654 [ 2856.058032] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 2856.063028] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2856.099846] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2856.105130] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2856.109278] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2856.112275] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2856.120264] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2856.123442] RAX: 0000000000007d85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2856.126989] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 2856.130381] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2856.134854] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 2856.139401] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 2856.145298] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 2856.151308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2856.155736] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 2856.159520] Call Trace: [ 2856.161160] [ 2856.162183] ? show_regs.cold.9+0x22/0x2f [ 2856.167216] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2856.170278] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2856.173658] ? watchdog+0x40/0x40 [ 2856.175317] ? hrtimer_interrupt+0x128/0x2c0 [ 2856.178590] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2856.182159] ? apic_timer_interrupt+0xf/0x20 [ 2856.185188] [ 2856.188249] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2856.196891] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2856.198671] do_raw_spin_lock+0x177/0x190 [ 2856.200592] _raw_spin_lock+0x21/0x30 [ 2856.202354] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2856.205604] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2856.208432] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2856.211546] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2856.214385] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2856.217101] ? woken_wake_function+0x30/0x30 [ 2856.220246] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2856.223219] kthread+0x1d1/0x200 [ 2856.225062] ? set_kthread_struct+0x70/0x70 [ 2856.228174] ret_from_fork+0x1f/0x30 [ 2860.055352] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 2860.057662] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 2860.058569] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2860.058640] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2860.058645] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2860.058648] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2860.058661] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2860.058666] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2860.058671] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2860.058673] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2860.058675] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2860.058678] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2860.058680] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2860.058683] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2860.058686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2860.058689] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2860.058697] Call Trace: [ 2860.058703] [ 2860.058709] ? show_regs.cold.9+0x22/0x2f [ 2860.073218] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2860.073286] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2860.073294] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2860.073297] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2860.073311] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2860.073315] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2860.073319] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2860.073322] RDX: 0000000000005d07 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2860.073324] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2860.073326] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2860.073328] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2860.073331] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2860.073334] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2860.073336] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2860.073345] Call Trace: [ 2860.073351] [ 2860.073356] ? show_regs.cold.9+0x22/0x2f [ 2860.073365] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2860.073371] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2860.073765] ? watchdog+0x40/0x40 [ 2860.073771] ? hrtimer_interrupt+0x128/0x2c0 [ 2860.073773] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2860.073780] ? apic_timer_interrupt+0xf/0x20 [ 2860.073783] [ 2860.073786] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2860.073789] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2860.073792] do_raw_spin_lock+0x177/0x190 [ 2860.073795] _raw_spin_lock+0x21/0x30 [ 2860.073800] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2860.074010] obd_mod_free+0x46/0x140 [ptlrpc] [ 2860.074182] mdc_commit_open+0x91/0xb0 [mdc] [ 2860.074211] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2860.074356] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2860.114486] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2860.114508] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2860.114513] ? watchdog+0x40/0x40 [ 2860.114517] ? hrtimer_interrupt+0x128/0x2c0 [ 2860.114521] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2860.114531] ? apic_timer_interrupt+0xf/0x20 [ 2860.114535] [ 2860.114540] ? native_safe_halt+0x2e/0x50 [ 2860.114545] kvm_wait+0xcd/0xe0 [ 2860.114554] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2860.114564] do_raw_spin_lock+0x177/0x190 [ 2860.114569] _raw_spin_lock+0x21/0x30 [ 2860.114573] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2860.123305] after_reply+0xe2c/0x1780 [ptlrpc] [ 2860.131825] ? do_raw_spin_unlock+0x75/0x190 [ 2860.131848] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2860.132310] ? do_wait_intr+0xf0/0xf0 [ 2860.132319] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2860.132535] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2860.132871] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2860.133184] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2860.133303] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2860.133340] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2860.133369] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2860.133821] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2860.133920] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2860.141874] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2860.142180] ? __next_timer_interrupt+0x160/0x160 [ 2860.142191] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2860.142405] ? do_wait_intr+0xf0/0xf0 [ 2860.142415] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2860.142617] kthread+0x1d1/0x200 [ 2860.142623] ? set_kthread_struct+0x70/0x70 [ 2860.142627] ret_from_fork+0x1f/0x30 [ 2860.504898] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2860.506932] lmv_intent_open+0x338/0x1150 [lmv] [ 2860.508886] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2860.511188] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2860.513896] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2860.518446] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2860.520242] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2860.527363] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2860.533202] ? do_raw_spin_unlock+0x75/0x190 [ 2860.536022] ? _raw_spin_unlock+0x12/0x30 [ 2860.537309] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2860.541463] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2860.542715] ll_atomic_open+0x36a/0xd70 [lustre] [ 2860.545076] lookup_open+0x356/0xb40 [ 2860.546399] do_last+0x313/0xfc0 [ 2860.556528] ? nd_jump_root+0xe5/0x160 [ 2860.562435] ? path_init+0x437/0x520 [ 2860.570198] path_openat+0xf7/0x500 [ 2860.572068] ? do_raw_spin_unlock+0x75/0x190 [ 2860.574119] do_filp_open+0x99/0x140 [ 2860.579989] ? getname_flags+0x6e/0x330 [ 2860.582610] ? __check_object_size+0xff/0x256 [ 2860.587561] ? do_raw_spin_unlock+0x75/0x190 [ 2860.589625] ? _raw_spin_unlock+0x12/0x30 [ 2860.592156] do_sys_openat2+0x2b4/0x410 [ 2860.594896] do_sys_open+0x73/0xa0 [ 2860.597312] __x64_sys_openat+0x24/0x30 [ 2860.600308] do_syscall_64+0xc1/0x440 [ 2860.603097] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2860.605143] RIP: 0033:0x7f53425a5332 [ 2860.606200] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2860.616163] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2860.628996] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2860.633392] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2860.640874] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2860.647055] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2860.651637] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2874.445111] rcu: INFO: rcu_sched self-detected stall on CPU [ 2874.449566] rcu: 2-....: (416461 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 2874.455978] (t=780019 jiffies g=274185 q=63789) [ 2874.461917] rcu: rcu_sched kthread starved for 238385 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 2874.472448] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 2874.482842] rcu: RCU grace-period kthread stack dump: [ 2874.488852] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 2874.492902] Call Trace: [ 2874.493644] __schedule+0x351/0xcb0 [ 2874.501673] schedule+0xc0/0x180 [ 2874.504167] schedule_timeout+0xb4/0x190 [ 2874.508521] ? do_raw_spin_unlock+0x75/0x190 [ 2874.512567] ? __next_timer_interrupt+0x160/0x160 [ 2874.515668] ? prepare_to_swait_event+0xf3/0x140 [ 2874.519297] rcu_gp_kthread+0x8dd/0x15b0 [ 2874.523115] ? force_qs_rnp+0x2c0/0x2c0 [ 2874.526838] kthread+0x1d1/0x200 [ 2874.529487] ? set_kthread_struct+0x70/0x70 [ 2874.534302] ret_from_fork+0x1f/0x30 [ 2874.538428] rcu: Stack dump where RCU GP kthread last ran: [ 2874.546413] Sending NMI from CPU 2 to CPUs 3: [ 2874.550849] NMI backtrace for cpu 3 [ 2874.551053] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2874.551055] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2874.551056] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2874.551058] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2874.551059] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 [ 2874.551062] RAX: 0000000000007d15 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2874.551064] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 2874.551065] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2874.551067] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 2874.551068] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 2874.551069] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 2874.551070] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2874.551071] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 2874.551072] Call Trace: [ 2874.551073] [ 2874.551074] ? show_regs.cold.9+0x22/0x2f [ 2874.551075] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2874.551076] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2874.551077] ? nmi_handle+0xbd/0x210 [ 2874.551078] ? default_do_nmi+0x4b/0x1b0 [ 2874.551079] ? do_nmi+0x280/0x380 [ 2874.551080] ? end_repeat_nmi+0x16/0x1f [ 2874.551081] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2874.551082] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2874.551083] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2874.551084] [ 2874.551085] do_raw_spin_lock+0x177/0x190 [ 2874.551086] _raw_spin_lock+0x21/0x30 [ 2874.551087] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2874.551088] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2874.551089] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2874.551090] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2874.551091] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2874.551092] ? woken_wake_function+0x30/0x30 [ 2874.551094] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2874.551095] kthread+0x1d1/0x200 [ 2874.551096] ? set_kthread_struct+0x70/0x70 [ 2874.551097] ret_from_fork+0x1f/0x30 [ 2874.553724] Sending NMI from CPU 2 to CPUs 1: [ 2874.691876] NMI backtrace for cpu 1 [ 2874.691883] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2874.691885] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2874.691886] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2874.691888] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2874.691889] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [ 2874.691892] RAX: 0000000000007586 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2874.691893] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2874.691895] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2874.691896] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2874.691897] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2874.691898] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2874.692004] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2874.692006] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2874.692008] Call Trace: [ 2874.692008] [ 2874.692009] ? show_regs.cold.9+0x22/0x2f [ 2874.692010] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 2874.692011] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 2874.692012] ? nmi_handle+0xbd/0x210 [ 2874.692013] ? default_do_nmi+0x4b/0x1b0 [ 2874.692014] ? do_nmi+0x280/0x380 [ 2874.692015] ? end_repeat_nmi+0x16/0x1f [ 2874.692016] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2874.692017] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2874.692018] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2874.692019] [ 2874.692019] do_raw_spin_lock+0x177/0x190 [ 2874.692020] _raw_spin_lock+0x21/0x30 [ 2874.692021] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2874.692022] ? do_raw_spin_unlock+0x75/0x190 [ 2874.692023] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2874.692024] ? do_wait_intr+0xf0/0xf0 [ 2874.692025] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2874.692026] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2874.692027] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2874.692028] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2874.692029] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2874.692030] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2874.692031] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2874.692032] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2874.692034] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2874.692035] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2874.692036] lmv_intent_open+0x338/0x1150 [lmv] [ 2874.692037] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2874.692038] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2874.692039] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2874.692040] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2874.692041] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2874.692042] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2874.692043] ? do_raw_spin_unlock+0x75/0x190 [ 2874.692044] ? _raw_spin_unlock+0x12/0x30 [ 2874.692045] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2874.692046] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2874.692047] ll_atomic_open+0x36a/0xd70 [lustre] [ 2874.692048] lookup_open+0x356/0xb40 [ 2874.692049] do_last+0x313/0xfc0 [ 2874.692050] ? nd_jump_root+0xe5/0x160 [ 2874.692051] ? path_init+0x437/0x520 [ 2874.692052] path_openat+0xf7/0x500 [ 2874.692053] ? do_raw_spin_unlock+0x75/0x190 [ 2874.692054] do_filp_open+0x99/0x140 [ 2874.692055] ? getname_flags+0x6e/0x330 [ 2874.692056] ? __check_object_size+0xff/0x256 [ 2874.692057] ? do_raw_spin_unlock+0x75/0x190 [ 2874.692058] ? _raw_spin_unlock+0x12/0x30 [ 2874.692059] do_sys_openat2+0x2b4/0x410 [ 2874.692060] do_sys_open+0x73/0xa0 [ 2874.692061] __x64_sys_openat+0x24/0x30 [ 2874.692062] do_syscall_64+0xc1/0x440 [ 2874.692063] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2874.692064] RIP: 0033:0x7f53425a5332 [ 2874.692065] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2874.692068] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2874.692070] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2874.692071] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2874.692073] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2874.692074] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2874.692075] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2874.692922] NMI backtrace for cpu 2 [ 2875.126486] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2875.130522] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2875.138852] Call Trace: [ 2875.140464] [ 2875.142830] ? dump_stack+0xbb/0x10e [ 2875.144159] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 2875.150090] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 2875.153593] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 2875.157799] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 2875.162994] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 2875.164971] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 2875.168425] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 2875.173998] ? update_process_times+0x6d/0xc0 [ 2875.176322] ? tick_sched_handle.isra.18+0x27/0x90 [ 2875.178077] ? tick_sched_timer+0x53/0xb0 [ 2875.179932] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2875.182030] ? tick_sched_handle.isra.18+0x90/0x90 [ 2875.184065] ? hrtimer_interrupt+0x128/0x2c0 [ 2875.185705] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2875.188348] ? apic_timer_interrupt+0xf/0x20 [ 2875.189986] [ 2875.190864] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2875.193252] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2875.195415] ? do_raw_spin_lock+0x177/0x190 [ 2875.197261] ? _raw_spin_lock+0x21/0x30 [ 2875.198678] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2875.200853] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 2875.202860] ? mdc_commit_open+0x91/0xb0 [mdc] [ 2875.204668] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2875.207376] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2875.210092] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 2875.212610] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2875.215236] ? __next_timer_interrupt+0x160/0x160 [ 2875.217005] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2875.219261] ? do_wait_intr+0xf0/0xf0 [ 2875.220713] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2875.223386] ? kthread+0x1d1/0x200 [ 2875.225421] ? set_kthread_struct+0x70/0x70 [ 2875.227365] ? ret_from_fork+0x1f/0x30 [ 2876.053286] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2876.057325] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2876.088477] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2876.096680] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2876.104053] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2876.106897] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2876.110729] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2876.120425] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2876.122890] RAX: 0000000000006909 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2876.128478] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2876.131991] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2876.139408] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2876.145933] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2876.149423] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2876.155489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2876.160041] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2876.162804] Call Trace: [ 2876.164095] [ 2876.164937] ? show_regs.cold.9+0x22/0x2f [ 2876.168852] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2876.170426] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2876.172563] ? watchdog+0x40/0x40 [ 2876.173741] ? hrtimer_interrupt+0x128/0x2c0 [ 2876.175063] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2876.178472] ? apic_timer_interrupt+0xf/0x20 [ 2876.179667] [ 2876.183943] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2876.187213] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2876.189742] ? newidle_balance.isra.93+0x207/0x580 [ 2876.196840] do_raw_spin_lock+0x177/0x190 [ 2876.199802] _raw_spin_lock+0x21/0x30 [ 2876.201668] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2876.205090] process_one_work+0x2c8/0x700 [ 2876.208032] worker_thread+0x296/0x6e0 [ 2876.210074] ? rescuer_thread+0x570/0x570 [ 2876.211857] kthread+0x1d1/0x200 [ 2876.215014] ? set_kthread_struct+0x70/0x70 [ 2876.216693] ret_from_fork+0x1f/0x30 [ 2880.993229] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 243s! [ 2880.996466] Showing busy workqueues and worker pools: [ 2880.999326] workqueue events: flags=0x0 [ 2881.001242] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2881.004506] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 2881.010457] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2881.013365] pending: cpu_latency_work [ptlrpc] [ 2881.016746] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2881.020344] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 2881.023575] workqueue events_long: flags=0x0 [ 2881.025714] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2881.030308] pending: lu_objects_flush [obdclass] [ 2881.033330] workqueue events_unbound: flags=0x2 [ 2881.035364] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2881.039465] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2881.042224] workqueue events_power_efficient: flags=0x80 [ 2881.045949] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2881.049915] pending: check_lifetime, do_cache_clean [ 2881.053520] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2881.056086] pending: neigh_periodic_work, neigh_periodic_work [ 2881.059743] workqueue mm_percpu_wq: flags=0x8 [ 2881.069604] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2881.072481] pending: vmstat_update, lru_add_drain_per_cpu [ 2881.075484] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2881.079142] pending: vmstat_update, lru_add_drain_per_cpu [ 2881.088370] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2881.097770] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2881.104778] workqueue ipv6_addrconf: flags=0x40008 [ 2881.106174] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2881.110140] pending: addrconf_verify_work [ 2881.111742] workqueue knbd0-recv: flags=0x1a [ 2881.119980] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2881.123278] in-flight: 651:recv_work [ 2881.125513] workqueue ptlrpc_pinger: flags=0x2 [ 2881.134901] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2881.139595] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 2881.141912] pool 8: cpus=0-3 flags=0x4 nice=0 hung=244s workers=4 idle: 15979 31824 8 [ 2881.149576] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2871s workers=2 idle: 654 [ 2900.058346] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 2900.059811] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 2900.061624] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 2900.061634] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2900.061717] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2900.061723] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2900.061726] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2900.061740] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2900.061744] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2900.061748] RAX: 0000000000005c83 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2900.061750] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2900.061753] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2900.061755] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2900.061757] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2900.061761] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2900.061764] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2900.061766] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2900.061775] Call Trace: [ 2900.061781] [ 2900.061788] ? show_regs.cold.9+0x22/0x2f [ 2900.061797] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2900.061803] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2900.061808] ? watchdog+0x40/0x40 [ 2900.061812] ? hrtimer_interrupt+0x128/0x2c0 [ 2900.061816] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2900.061824] ? apic_timer_interrupt+0xf/0x20 [ 2900.061829] [ 2900.061832] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2900.061839] do_raw_spin_lock+0x177/0x190 [ 2900.061843] _raw_spin_lock+0x21/0x30 [ 2900.061848] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2900.062134] ? do_raw_spin_unlock+0x75/0x190 [ 2900.062140] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2900.062331] ? do_wait_intr+0xf0/0xf0 [ 2900.062340] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2900.064160] Modules linked in: [ 2900.066050] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2900.066331] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2900.067909] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2900.068034] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2900.068077] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2900.068107] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2900.068134] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2900.068217] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2900.068464] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2900.069224] lmv_intent_open+0x338/0x1150 [lmv] [ 2900.069257] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2900.069353] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2900.069439] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2900.069462] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2900.070146] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2900.070231] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2900.070313] ? do_raw_spin_unlock+0x75/0x190 [ 2900.070320] ? _raw_spin_unlock+0x12/0x30 [ 2900.070328] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2900.070358] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2900.070379] ll_atomic_open+0x36a/0xd70 [lustre] [ 2900.070463] lookup_open+0x356/0xb40 [ 2900.070471] do_last+0x313/0xfc0 [ 2900.070477] ? nd_jump_root+0xe5/0x160 [ 2900.070481] ? path_init+0x437/0x520 [ 2900.070486] path_openat+0xf7/0x500 [ 2900.071145] ? do_raw_spin_unlock+0x75/0x190 [ 2900.071154] do_filp_open+0x99/0x140 [ 2900.071162] ? getname_flags+0x6e/0x330 [ 2900.071166] ? __check_object_size+0xff/0x256 [ 2900.071173] ? do_raw_spin_unlock+0x75/0x190 [ 2900.071177] ? _raw_spin_unlock+0x12/0x30 [ 2900.071183] do_sys_openat2+0x2b4/0x410 [ 2900.071187] do_sys_open+0x73/0xa0 [ 2900.071192] __x64_sys_openat+0x24/0x30 [ 2900.071195] do_syscall_64+0xc1/0x440 [ 2900.071202] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2900.071208] RIP: 0033:0x7f53425a5332 [ 2900.071217] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2900.071221] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2900.071226] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2900.071229] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2900.071231] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2900.071234] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2900.071236] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2900.071324] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2900.071390] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2900.071395] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2900.071398] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2900.071405] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2900.071408] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2900.071412] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2900.071414] RDX: 000000000000667d RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2900.071416] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2900.071418] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2900.071420] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2900.071423] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2900.071426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2900.071428] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2900.071436] Call Trace: [ 2900.071440] [ 2900.071443] ? show_regs.cold.9+0x22/0x2f [ 2900.071451] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2900.071457] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2900.071461] ? watchdog+0x40/0x40 [ 2900.071465] ? hrtimer_interrupt+0x128/0x2c0 [ 2900.071469] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2900.071474] ? apic_timer_interrupt+0xf/0x20 [ 2900.083257] lustre(O) osp(O) ofd(O) lod(O) [ 2900.123979] [ 2900.123993] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2900.124006] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2900.124011] do_raw_spin_lock+0x177/0x190 [ 2900.124018] _raw_spin_lock+0x21/0x30 [ 2900.124027] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2900.124351] obd_mod_free+0x46/0x140 [ptlrpc] [ 2900.124645] mdc_commit_open+0x91/0xb0 [mdc] [ 2900.124752] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2900.124957] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2900.125161] after_reply+0xe2c/0x1780 [ptlrpc] [ 2900.125376] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2900.134007] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2900.134086] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2900.134094] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2900.134097] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2900.134111] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2900.134114] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2900.134119] RAX: 0000000000001c13 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2900.134121] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 2900.134123] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2900.134125] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 2900.134127] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 2900.134130] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 2900.134132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2900.134134] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 2900.134142] Call Trace: [ 2900.134148] [ 2900.134154] ? show_regs.cold.9+0x22/0x2f [ 2900.134164] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2900.134170] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2900.134175] ? watchdog+0x40/0x40 [ 2900.134179] ? hrtimer_interrupt+0x128/0x2c0 [ 2900.134183] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2900.134192] ? apic_timer_interrupt+0xf/0x20 [ 2900.134196] [ 2900.134199] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2900.134204] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2900.134210] do_raw_spin_lock+0x177/0x190 [ 2900.134216] _raw_spin_lock+0x21/0x30 [ 2900.134220] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2900.134483] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2900.134915] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2900.135358] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2900.136275] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2900.136566] ? woken_wake_function+0x30/0x30 [ 2900.136577] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2900.136866] kthread+0x1d1/0x200 [ 2900.136872] ? set_kthread_struct+0x70/0x70 [ 2900.136876] ret_from_fork+0x1f/0x30 [ 2900.935908] ? __next_timer_interrupt+0x160/0x160 [ 2900.940341] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2900.945994] ? do_wait_intr+0xf0/0xf0 [ 2900.950027] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2900.958787] kthread+0x1d1/0x200 [ 2900.959827] ? set_kthread_struct+0x70/0x70 [ 2900.969429] ret_from_fork+0x1f/0x30 [ 2908.052062] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 2908.061144] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2908.105986] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2908.110779] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2908.115559] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2908.119486] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2908.123335] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2908.129135] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2908.133008] RAX: 0000000000007908 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2908.142347] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2908.147743] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2908.151078] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2908.155240] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2908.161165] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2908.165989] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2908.169106] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2908.172807] Call Trace: [ 2908.173849] [ 2908.175001] ? show_regs.cold.9+0x22/0x2f [ 2908.177249] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2908.179822] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2908.181924] ? watchdog+0x40/0x40 [ 2908.183702] ? hrtimer_interrupt+0x128/0x2c0 [ 2908.185948] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2908.190057] ? apic_timer_interrupt+0xf/0x20 [ 2908.192871] [ 2908.194154] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2908.198081] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2908.200338] ? newidle_balance.isra.93+0x207/0x580 [ 2908.203517] do_raw_spin_lock+0x177/0x190 [ 2908.205203] _raw_spin_lock+0x21/0x30 [ 2908.207164] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2908.210139] process_one_work+0x2c8/0x700 [ 2908.215298] worker_thread+0x296/0x6e0 [ 2908.219623] ? rescuer_thread+0x570/0x570 [ 2908.220915] kthread+0x1d1/0x200 [ 2908.224350] ? set_kthread_struct+0x70/0x70 [ 2908.227956] ret_from_fork+0x1f/0x30 [ 2911.203256] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 274s! [ 2911.210864] Showing busy workqueues and worker pools: [ 2911.214636] workqueue events: flags=0x0 [ 2911.219607] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2911.223219] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 2911.231736] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2911.237092] pending: cpu_latency_work [ptlrpc] [ 2911.243318] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2911.247948] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 2911.258903] workqueue events_long: flags=0x0 [ 2911.260466] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2911.267127] pending: lu_objects_flush [obdclass] [ 2911.271636] workqueue events_unbound: flags=0x2 [ 2911.274737] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2911.278293] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2911.282189] workqueue events_power_efficient: flags=0x80 [ 2911.286216] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2911.292701] pending: check_lifetime, do_cache_clean [ 2911.295799] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2911.302958] pending: neigh_periodic_work, neigh_periodic_work [ 2911.304913] workqueue mm_percpu_wq: flags=0x8 [ 2911.315633] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2911.321694] pending: vmstat_update, lru_add_drain_per_cpu [ 2911.325329] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2911.330271] pending: vmstat_update, lru_add_drain_per_cpu [ 2911.340401] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2911.348259] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2911.351917] workqueue ipv6_addrconf: flags=0x40008 [ 2911.353565] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2911.363096] pending: addrconf_verify_work [ 2911.370027] workqueue knbd0-recv: flags=0x1a [ 2911.373689] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2911.378169] in-flight: 651:recv_work [ 2911.379422] workqueue ptlrpc_pinger: flags=0x2 [ 2911.387071] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2911.396998] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 2911.405391] pool 8: cpus=0-3 flags=0x4 nice=0 hung=274s workers=4 idle: 15979 31824 8 [ 2911.409944] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2901s workers=2 idle: 654 [ 2928.056146] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 2928.058396] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 2928.058452] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 2928.058460] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2928.058532] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2928.058539] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2928.058542] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2928.058558] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2928.058562] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2928.058567] RAX: 0000000000007f89 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2928.058569] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 2928.058571] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2928.058573] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 2928.058575] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 2928.058579] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 2928.058581] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2928.058584] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 2928.058592] Call Trace: [ 2928.058599] [ 2928.058604] ? show_regs.cold.9+0x22/0x2f [ 2928.058614] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2928.058620] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2928.058625] ? watchdog+0x40/0x40 [ 2928.058630] ? hrtimer_interrupt+0x128/0x2c0 [ 2928.058634] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2928.058650] ? apic_timer_interrupt+0xf/0x20 [ 2928.058654] [ 2928.058657] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2928.058662] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2928.058669] do_raw_spin_lock+0x177/0x190 [ 2928.058674] _raw_spin_lock+0x21/0x30 [ 2928.058679] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2928.058956] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2928.061027] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2928.062569] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2928.062798] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2928.063025] ? woken_wake_function+0x30/0x30 [ 2928.063035] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2928.063282] kthread+0x1d1/0x200 [ 2928.063289] ? set_kthread_struct+0x70/0x70 [ 2928.063293] ret_from_fork+0x1f/0x30 [ 2928.063451] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2928.063640] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2928.063646] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2928.063649] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2928.063660] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2928.063663] RSP: 0018:ffffbb5381503500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2928.063667] RAX: 0000000000007f08 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2928.063670] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2928.063672] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2928.063674] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2928.063676] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2928.063679] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2928.063681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2928.063683] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2928.063692] Call Trace: [ 2928.063695] [ 2928.063698] ? show_regs.cold.9+0x22/0x2f [ 2928.063707] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2928.063714] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2928.063718] ? watchdog+0x40/0x40 [ 2928.063722] ? hrtimer_interrupt+0x128/0x2c0 [ 2928.063726] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2928.063731] ? apic_timer_interrupt+0xf/0x20 [ 2928.063735] [ 2928.063738] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2928.063743] do_raw_spin_lock+0x177/0x190 [ 2928.063747] _raw_spin_lock+0x21/0x30 [ 2928.063751] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2928.063976] ? do_raw_spin_unlock+0x75/0x190 [ 2928.063981] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2928.064173] ? do_wait_intr+0xf0/0xf0 [ 2928.064181] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2928.064392] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2928.064612] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2928.064824] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2928.064943] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2928.064980] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2928.065008] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2928.065034] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2928.065115] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2928.071836] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr [ 2928.075580] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2928.075638] lmv_intent_open+0x338/0x1150 [lmv] [ 2928.075673] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2928.077920] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2928.078017] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2928.078057] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2928.078147] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2928.119671] intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2928.119712] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2928.119720] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2928.119722] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2928.119737] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2928.119742] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2928.119746] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2928.119749] RDX: 0000000000006038 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2928.119751] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2928.119753] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2928.119755] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2928.119758] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2928.119760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2928.119762] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2928.119771] Call Trace: [ 2928.119777] [ 2928.119783] ? show_regs.cold.9+0x22/0x2f [ 2928.119793] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2928.119798] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2928.119803] ? watchdog+0x40/0x40 [ 2928.119808] ? hrtimer_interrupt+0x128/0x2c0 [ 2928.119812] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2928.119822] ? apic_timer_interrupt+0xf/0x20 [ 2928.119826] [ 2928.119829] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2928.119833] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2928.119838] do_raw_spin_lock+0x177/0x190 [ 2928.119842] _raw_spin_lock+0x21/0x30 [ 2928.119850] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2928.130912] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2928.131048] ? do_raw_spin_unlock+0x75/0x190 [ 2928.131056] ? _raw_spin_unlock+0x12/0x30 [ 2928.131063] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2928.131088] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2928.131107] ll_atomic_open+0x36a/0xd70 [lustre] [ 2928.131203] lookup_open+0x356/0xb40 [ 2928.131213] do_last+0x313/0xfc0 [ 2928.131218] ? nd_jump_root+0xe5/0x160 [ 2928.131222] ? path_init+0x437/0x520 [ 2928.131226] path_openat+0xf7/0x500 [ 2928.131231] ? do_raw_spin_unlock+0x75/0x190 [ 2928.131235] do_filp_open+0x99/0x140 [ 2928.131241] ? getname_flags+0x6e/0x330 [ 2928.131245] ? __check_object_size+0xff/0x256 [ 2928.131252] ? do_raw_spin_unlock+0x75/0x190 [ 2928.131256] ? _raw_spin_unlock+0x12/0x30 [ 2928.131260] do_sys_openat2+0x2b4/0x410 [ 2928.131263] do_sys_open+0x73/0xa0 [ 2928.131267] __x64_sys_openat+0x24/0x30 [ 2928.131270] do_syscall_64+0xc1/0x440 [ 2928.131277] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2928.131282] RIP: 0033:0x7f53425a5332 [ 2928.131290] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2928.131293] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2928.131298] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2928.131301] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2928.131303] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2928.131305] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2928.131307] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2928.912424] obd_mod_free+0x46/0x140 [ptlrpc] [ 2928.915000] mdc_commit_open+0x91/0xb0 [mdc] [ 2928.916565] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2928.921314] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2928.924322] after_reply+0xe2c/0x1780 [ptlrpc] [ 2928.926513] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2928.928815] ? __next_timer_interrupt+0x160/0x160 [ 2928.930956] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2928.933007] ? do_wait_intr+0xf0/0xf0 [ 2928.934641] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2928.937678] kthread+0x1d1/0x200 [ 2928.938750] ? set_kthread_struct+0x70/0x70 [ 2928.940721] ret_from_fork+0x1f/0x30 [ 2936.052025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 2936.056080] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2936.089022] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2936.092935] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2936.095003] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2936.106543] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2936.112048] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2936.150404] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2936.159450] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2936.167372] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2936.175561] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2936.181551] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2936.188805] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2936.194947] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2936.211063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2936.216899] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2936.223981] Call Trace: [ 2936.224962] [ 2936.225802] ? show_regs.cold.9+0x22/0x2f [ 2936.227631] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2936.231779] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2936.234913] ? watchdog+0x40/0x40 [ 2936.240145] ? hrtimer_interrupt+0x128/0x2c0 [ 2936.244437] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2936.250109] ? apic_timer_interrupt+0xf/0x20 [ 2936.251739] [ 2936.252305] ? native_safe_halt+0x2e/0x50 [ 2936.255913] kvm_wait+0xcd/0xe0 [ 2936.257146] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2936.260022] ? newidle_balance.isra.93+0x207/0x580 [ 2936.263209] do_raw_spin_lock+0x177/0x190 [ 2936.264793] _raw_spin_lock+0x21/0x30 [ 2936.265998] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2936.271041] process_one_work+0x2c8/0x700 [ 2936.272523] worker_thread+0x296/0x6e0 [ 2936.274399] ? rescuer_thread+0x570/0x570 [ 2936.275994] kthread+0x1d1/0x200 [ 2936.277954] ? set_kthread_struct+0x70/0x70 [ 2936.281506] ret_from_fork+0x1f/0x30 [ 2941.926453] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 304s! [ 2941.930536] Showing busy workqueues and worker pools: [ 2941.932610] workqueue events: flags=0x0 [ 2941.934138] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2941.936748] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 2941.944842] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2941.947312] pending: cpu_latency_work [ptlrpc] [ 2941.956142] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2941.959055] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 2941.984239] workqueue events_long: flags=0x0 [ 2941.985800] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2942.000729] pending: lu_objects_flush [obdclass] [ 2942.009132] workqueue events_unbound: flags=0x2 [ 2942.013671] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2942.018197] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2942.022181] workqueue events_power_efficient: flags=0x80 [ 2942.035904] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2942.046628] pending: check_lifetime, do_cache_clean [ 2942.052517] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2942.068129] pending: neigh_periodic_work, neigh_periodic_work [ 2942.072355] workqueue mm_percpu_wq: flags=0x8 [ 2942.074696] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2942.081742] pending: vmstat_update, lru_add_drain_per_cpu [ 2942.085372] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2942.089233] pending: vmstat_update, lru_add_drain_per_cpu [ 2942.091742] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2942.098892] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2942.105523] workqueue ipv6_addrconf: flags=0x40008 [ 2942.109627] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2942.114133] pending: addrconf_verify_work [ 2942.118603] workqueue knbd0-recv: flags=0x1a [ 2942.120409] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2942.123247] in-flight: 651:recv_work [ 2942.125025] workqueue ptlrpc_pinger: flags=0x2 [ 2942.127030] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2942.129837] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 2942.132766] pool 8: cpus=0-3 flags=0x4 nice=0 hung=305s workers=4 idle: 15979 31824 8 [ 2942.136829] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2932s workers=2 idle: 654 [ 2956.056445] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 2956.057305] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 2956.058781] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 2956.058794] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2956.058867] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2956.058873] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2956.058876] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2956.058954] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2956.058959] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2956.058963] RAX: 0000000000007b20 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2956.058965] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 2956.058967] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2956.058969] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 2956.058972] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 2956.058975] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 2956.058978] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2956.058980] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 2956.058989] Call Trace: [ 2956.059004] [ 2956.059012] ? show_regs.cold.9+0x22/0x2f [ 2956.059022] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2956.059030] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2956.059035] ? watchdog+0x40/0x40 [ 2956.059039] ? hrtimer_interrupt+0x128/0x2c0 [ 2956.059044] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2956.059053] ? apic_timer_interrupt+0xf/0x20 [ 2956.059058] [ 2956.059061] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2956.059066] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2956.059074] do_raw_spin_lock+0x177/0x190 [ 2956.059079] _raw_spin_lock+0x21/0x30 [ 2956.059085] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2956.059427] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2956.060005] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2956.060652] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2956.061906] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2956.062121] ? woken_wake_function+0x30/0x30 [ 2956.062132] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2956.062674] kthread+0x1d1/0x200 [ 2956.062682] ? set_kthread_struct+0x70/0x70 [ 2956.062686] ret_from_fork+0x1f/0x30 [ 2956.078288] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) [ 2956.083340] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2956.083416] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2956.083425] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2956.083428] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2956.083442] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 2956.083446] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2956.083451] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2956.083453] RDX: 0000000000004aa0 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2956.083456] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2956.083458] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2956.083460] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2956.083464] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2956.083467] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2956.083470] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2956.083478] Call Trace: [ 2956.083484] [ 2956.083490] ? show_regs.cold.9+0x22/0x2f [ 2956.083499] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2956.083506] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2956.083509] ? watchdog+0x40/0x40 [ 2956.083513] ? hrtimer_interrupt+0x128/0x2c0 [ 2956.083518] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2956.083529] ? apic_timer_interrupt+0xf/0x20 [ 2956.083534] [ 2956.083537] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 2956.083542] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2956.083547] do_raw_spin_lock+0x177/0x190 [ 2956.083552] _raw_spin_lock+0x21/0x30 [ 2956.083558] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2956.086255] obd_mod_free+0x46/0x140 [ptlrpc] [ 2956.086523] mdc_commit_open+0x91/0xb0 [mdc] [ 2956.086565] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2956.090085] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2956.096847] lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2956.096930] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2956.096940] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2956.096942] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2956.096957] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2956.096961] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 2956.096966] RAX: 0000000000000513 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2956.096968] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2956.096970] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2956.096973] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2956.096975] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2956.096978] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2956.096981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2956.096983] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2956.096992] Call Trace: [ 2956.096997] [ 2956.097003] ? show_regs.cold.9+0x22/0x2f [ 2956.097014] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2956.097021] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2956.097025] ? watchdog+0x40/0x40 [ 2956.097029] ? hrtimer_interrupt+0x128/0x2c0 [ 2956.097033] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2956.097042] ? apic_timer_interrupt+0xf/0x20 [ 2956.097046] [ 2956.097049] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2956.097056] do_raw_spin_lock+0x177/0x190 [ 2956.097060] _raw_spin_lock+0x21/0x30 [ 2956.097065] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2956.097917] ? do_raw_spin_unlock+0x75/0x190 [ 2956.097924] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2956.098128] ? do_wait_intr+0xf0/0xf0 [ 2956.098137] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2956.098804] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2956.099093] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2956.099539] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2956.099661] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2956.099857] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2956.099890] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2956.100477] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2956.100572] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2956.101017] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2956.101046] lmv_intent_open+0x338/0x1150 [lmv] [ 2956.101078] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2956.131441] after_reply+0xe2c/0x1780 [ptlrpc] [ 2956.144918] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2956.149561] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2956.157829] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2956.157882] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2956.165220] ? __next_timer_interrupt+0x160/0x160 [ 2956.165241] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2956.165539] ? do_wait_intr+0xf0/0xf0 [ 2956.165547] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2956.165741] kthread+0x1d1/0x200 [ 2956.165747] ? set_kthread_struct+0x70/0x70 [ 2956.165751] ret_from_fork+0x1f/0x30 [ 2956.721002] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2956.723668] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2956.726237] ? do_raw_spin_unlock+0x75/0x190 [ 2956.728206] ? _raw_spin_unlock+0x12/0x30 [ 2956.730113] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2956.732190] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2956.734528] ll_atomic_open+0x36a/0xd70 [lustre] [ 2956.736829] lookup_open+0x356/0xb40 [ 2956.738511] do_last+0x313/0xfc0 [ 2956.740047] ? nd_jump_root+0xe5/0x160 [ 2956.741865] ? path_init+0x437/0x520 [ 2956.745208] path_openat+0xf7/0x500 [ 2956.748807] ? do_raw_spin_unlock+0x75/0x190 [ 2956.751992] do_filp_open+0x99/0x140 [ 2956.753667] ? getname_flags+0x6e/0x330 [ 2956.757154] ? __check_object_size+0xff/0x256 [ 2956.760477] ? do_raw_spin_unlock+0x75/0x190 [ 2956.762102] ? _raw_spin_unlock+0x12/0x30 [ 2956.767999] do_sys_openat2+0x2b4/0x410 [ 2956.772858] do_sys_open+0x73/0xa0 [ 2956.776279] __x64_sys_openat+0x24/0x30 [ 2956.779021] do_syscall_64+0xc1/0x440 [ 2956.780823] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2956.785067] RIP: 0033:0x7f53425a5332 [ 2956.791071] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2956.800243] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2956.808036] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2956.812963] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2956.816013] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2956.819673] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2956.823136] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2968.054300] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 2968.069737] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2968.111322] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2968.121857] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2968.134278] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 2968.141901] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2968.148446] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 2968.175590] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2968.179848] RAX: 0000000000006f15 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2968.182117] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 2968.191268] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 2968.195613] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 2968.200280] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 2968.204396] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 2968.208483] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2968.212304] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 2968.216239] Call Trace: [ 2968.217850] [ 2968.219380] ? show_regs.cold.9+0x22/0x2f [ 2968.222348] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2968.225139] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2968.226915] ? watchdog+0x40/0x40 [ 2968.228575] ? hrtimer_interrupt+0x128/0x2c0 [ 2968.231179] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2968.233931] ? apic_timer_interrupt+0xf/0x20 [ 2968.237170] [ 2968.237925] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 2968.240823] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2968.243984] ? newidle_balance.isra.93+0x207/0x580 [ 2968.245826] do_raw_spin_lock+0x177/0x190 [ 2968.247233] _raw_spin_lock+0x21/0x30 [ 2968.248727] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 2968.251774] process_one_work+0x2c8/0x700 [ 2968.255211] worker_thread+0x296/0x6e0 [ 2968.256983] ? rescuer_thread+0x570/0x570 [ 2968.260246] kthread+0x1d1/0x200 [ 2968.261514] ? set_kthread_struct+0x70/0x70 [ 2968.263945] ret_from_fork+0x1f/0x30 [ 2972.641528] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 335s! [ 2972.645819] Showing busy workqueues and worker pools: [ 2972.649290] workqueue events: flags=0x0 [ 2972.650664] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 2972.654595] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 2972.660928] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2972.664629] pending: cpu_latency_work [ptlrpc] [ 2972.670473] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2972.675074] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 2972.678608] workqueue events_long: flags=0x0 [ 2972.679750] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 2972.686319] pending: lu_objects_flush [obdclass] [ 2972.688766] workqueue events_unbound: flags=0x2 [ 2972.692813] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 2972.695108] pending: flush_memcg_stats_dwork, idle_cull_fn [ 2972.698776] workqueue events_power_efficient: flags=0x80 [ 2972.703624] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2972.706510] pending: check_lifetime, do_cache_clean [ 2972.708481] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2972.710974] pending: neigh_periodic_work, neigh_periodic_work [ 2972.712876] workqueue mm_percpu_wq: flags=0x8 [ 2972.714069] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2972.716682] pending: vmstat_update, lru_add_drain_per_cpu [ 2972.718800] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 2972.722093] pending: vmstat_update, lru_add_drain_per_cpu [ 2972.725623] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 2972.729753] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 2972.732587] workqueue xprtiod: flags=0xa [ 2972.733897] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 2972.736064] pending: xprt_autoclose [ 2972.737556] workqueue ipv6_addrconf: flags=0x40008 [ 2972.739525] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 2972.745586] pending: addrconf_verify_work [ 2972.754522] workqueue knbd0-recv: flags=0x1a [ 2972.756862] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 2972.769284] in-flight: 651:recv_work [ 2972.770720] workqueue ptlrpc_pinger: flags=0x2 [ 2972.780740] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 2972.788483] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 2972.797738] pool 8: cpus=0-3 flags=0x4 nice=0 hung=335s workers=4 idle: 15979 31824 8 [ 2972.807716] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2963s workers=2 idle: 654 [ 2984.055102] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 2984.057097] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 2984.058025] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 2984.058033] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2984.058113] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2984.058119] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2984.058122] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2984.058134] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2984.058138] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2984.058143] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2984.058145] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 2984.058148] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2984.058150] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 2984.058153] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 2984.058156] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 2984.058159] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2984.058161] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 2984.058170] Call Trace: [ 2984.058176] [ 2984.058182] ? show_regs.cold.9+0x22/0x2f [ 2984.058192] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2984.058198] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2984.058203] ? watchdog+0x40/0x40 [ 2984.058208] ? hrtimer_interrupt+0x128/0x2c0 [ 2984.058212] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2984.058228] ? apic_timer_interrupt+0xf/0x20 [ 2984.058233] [ 2984.058236] ? native_safe_halt+0x2e/0x50 [ 2984.058240] kvm_wait+0xcd/0xe0 [ 2984.058247] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2984.058255] do_raw_spin_lock+0x177/0x190 [ 2984.058260] _raw_spin_lock+0x21/0x30 [ 2984.058264] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 2984.058653] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 2984.058856] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 2984.059076] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 2984.059366] Modules linked in: lustre(O) [ 2984.059292] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 2984.059501] ? woken_wake_function+0x30/0x30 [ 2984.059509] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 2984.060880] kthread+0x1d1/0x200 [ 2984.060888] ? set_kthread_struct+0x70/0x70 [ 2984.060893] ret_from_fork+0x1f/0x30 [ 2984.062455] Modules linked in: lustre(O) [ 2984.065682] osp(O) [ 2984.086699] osp(O) [ 2984.091915] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2984.091985] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2984.091993] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2984.091995] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2984.092009] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2984.092013] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 2984.092017] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 2984.092020] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 2984.092022] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 2984.092024] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 2984.092027] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 2984.092030] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 2984.092033] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2984.092035] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 2984.092487] Call Trace: [ 2984.092495] [ 2984.092501] ? show_regs.cold.9+0x22/0x2f [ 2984.092512] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2984.092519] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2984.092522] ? watchdog+0x40/0x40 [ 2984.092527] ? hrtimer_interrupt+0x128/0x2c0 [ 2984.092531] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2984.092539] ? apic_timer_interrupt+0xf/0x20 [ 2984.092544] [ 2984.092546] ? native_safe_halt+0x2e/0x50 [ 2984.092550] kvm_wait+0xcd/0xe0 [ 2984.092558] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 2984.092573] do_raw_spin_lock+0x177/0x190 [ 2984.092631] _raw_spin_lock+0x21/0x30 [ 2984.092638] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 2984.092932] ? do_raw_spin_unlock+0x75/0x190 [ 2984.092938] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 2984.093357] ? do_wait_intr+0xf0/0xf0 [ 2984.093366] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 2984.093559] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 2984.093772] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2984.104106] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 2984.104180] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 2984.104187] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 2984.104190] RIP: 0010:native_safe_halt+0x2e/0x50 [ 2984.104202] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 2984.104207] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 2984.104229] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 2984.104231] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 2984.104233] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 2984.104235] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 2984.104237] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 2984.106505] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2984.124339] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 2984.124349] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2984.124350] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 2984.124357] Call Trace: [ 2984.124364] [ 2984.124368] ? show_regs.cold.9+0x22/0x2f [ 2984.124379] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 2984.124384] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 2984.124387] ? watchdog+0x40/0x40 [ 2984.124390] ? hrtimer_interrupt+0x128/0x2c0 [ 2984.124392] ? smp_apic_timer_interrupt+0xa6/0x270 [ 2984.124399] ? apic_timer_interrupt+0xf/0x20 [ 2984.124402] [ 2984.124404] ? native_safe_halt+0x2e/0x50 [ 2984.124406] kvm_wait+0xcd/0xe0 [ 2984.124412] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 2984.124417] do_raw_spin_lock+0x177/0x190 [ 2984.124420] _raw_spin_lock+0x21/0x30 [ 2984.124425] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 2984.128841] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2984.128891] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 2984.128924] mdc_intent_lock+0x353/0x7e0 [mdc] [ 2984.128954] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2984.137636] obd_mod_free+0x46/0x140 [ptlrpc] [ 2984.137913] mdc_commit_open+0x91/0xb0 [mdc] [ 2984.147086] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 2984.147931] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 2984.147973] lmv_intent_open+0x338/0x1150 [lmv] [ 2984.148004] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2984.148115] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2984.148271] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 2984.148298] ll_intent_lock+0xc5/0xaa0 [lustre] [ 2984.148388] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 2984.154945] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 2984.160520] ll_lookup_it+0xe0e/0x2270 [lustre] [ 2984.160667] ? do_raw_spin_unlock+0x75/0x190 [ 2984.160678] ? _raw_spin_unlock+0x12/0x30 [ 2984.160687] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 2984.160717] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 2984.160737] ll_atomic_open+0x36a/0xd70 [lustre] [ 2984.160826] lookup_open+0x356/0xb40 [ 2984.160837] do_last+0x313/0xfc0 [ 2984.160842] ? nd_jump_root+0xe5/0x160 [ 2984.160847] ? path_init+0x437/0x520 [ 2984.160851] path_openat+0xf7/0x500 [ 2984.160856] ? do_raw_spin_unlock+0x75/0x190 [ 2984.160860] do_filp_open+0x99/0x140 [ 2984.160867] ? getname_flags+0x6e/0x330 [ 2984.160871] ? __check_object_size+0xff/0x256 [ 2984.160878] ? do_raw_spin_unlock+0x75/0x190 [ 2984.160882] ? _raw_spin_unlock+0x12/0x30 [ 2984.160886] do_sys_openat2+0x2b4/0x410 [ 2984.160890] do_sys_open+0x73/0xa0 [ 2984.160894] __x64_sys_openat+0x24/0x30 [ 2984.160898] do_syscall_64+0xc1/0x440 [ 2984.160906] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2984.160912] RIP: 0033:0x7f53425a5332 [ 2984.160970] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 2984.160976] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 2984.160982] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 2984.160985] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 2984.160987] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 2984.160989] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 2984.160991] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 2984.901741] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 2984.906857] after_reply+0xe2c/0x1780 [ptlrpc] [ 2984.911959] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 2984.918475] ? __next_timer_interrupt+0x160/0x160 [ 2984.921102] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 2984.926719] ? do_wait_intr+0xf0/0xf0 [ 2984.927847] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 2984.933518] kthread+0x1d1/0x200 [ 2984.935063] ? set_kthread_struct+0x70/0x70 [ 2984.937117] ret_from_fork+0x1f/0x30 [ 3000.052058] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 3000.057296] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3000.103316] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3000.112080] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3000.127220] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3000.131586] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3000.134489] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3000.147488] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3000.153149] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3000.157280] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3000.163963] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3000.169732] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3000.183314] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3000.193229] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3000.201643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3000.203558] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3000.214433] Call Trace: [ 3000.216505] [ 3000.218145] ? show_regs.cold.9+0x22/0x2f [ 3000.219888] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3000.222554] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3000.229702] ? watchdog+0x40/0x40 [ 3000.231247] ? hrtimer_interrupt+0x128/0x2c0 [ 3000.233267] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3000.235763] ? apic_timer_interrupt+0xf/0x20 [ 3000.250145] [ 3000.251257] ? native_safe_halt+0x2e/0x50 [ 3000.259365] kvm_wait+0xcd/0xe0 [ 3000.265267] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3000.271925] ? newidle_balance.isra.93+0x207/0x580 [ 3000.277407] do_raw_spin_lock+0x177/0x190 [ 3000.278757] _raw_spin_lock+0x21/0x30 [ 3000.285744] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3000.295003] process_one_work+0x2c8/0x700 [ 3000.298341] worker_thread+0x296/0x6e0 [ 3000.299373] ? rescuer_thread+0x570/0x570 [ 3000.307022] kthread+0x1d1/0x200 [ 3000.308049] ? set_kthread_struct+0x70/0x70 [ 3000.316252] ret_from_fork+0x1f/0x30 [ 3002.848176] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 365s! [ 3002.862674] Showing busy workqueues and worker pools: [ 3002.871025] workqueue events: flags=0x0 [ 3002.872512] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3002.887416] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3002.906787] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3002.915008] pending: cpu_latency_work [ptlrpc] [ 3002.927029] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3002.936733] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3002.953044] workqueue events_long: flags=0x0 [ 3002.954636] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3002.960391] pending: lu_objects_flush [obdclass] [ 3002.970803] workqueue events_unbound: flags=0x2 [ 3002.975802] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3002.981696] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3002.992042] workqueue events_power_efficient: flags=0x80 [ 3002.993965] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3003.010234] pending: check_lifetime, do_cache_clean [ 3003.012856] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3003.026701] pending: neigh_periodic_work, neigh_periodic_work [ 3003.029559] workqueue mm_percpu_wq: flags=0x8 [ 3003.040470] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3003.044603] pending: vmstat_update, lru_add_drain_per_cpu [ 3003.055331] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3003.065818] pending: vmstat_update, lru_add_drain_per_cpu [ 3003.072766] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3003.081197] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3003.087857] workqueue xprtiod: flags=0xa [ 3003.093666] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3003.109695] pending: xprt_autoclose [ 3003.122069] workqueue ipv6_addrconf: flags=0x40008 [ 3003.128806] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3003.136729] pending: addrconf_verify_work [ 3003.146486] workqueue knbd0-recv: flags=0x1a [ 3003.148073] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3003.169204] in-flight: 651:recv_work [ 3003.170516] workqueue ptlrpc_pinger: flags=0x2 [ 3003.172183] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3003.174467] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3003.178663] pool 8: cpus=0-3 flags=0x4 nice=0 hung=366s workers=4 idle: 15979 31824 8 [ 3003.183829] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=2993s workers=2 idle: 654 [ 3012.055431] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 3012.057005] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 3012.058116] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 3012.058131] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3012.058523] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3012.058529] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3012.058532] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3012.058550] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3012.058554] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3012.058559] RAX: 0000000000007e93 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3012.058561] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3012.058563] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3012.058565] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3012.058568] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3012.058571] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3012.058574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3012.058576] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3012.058874] Call Trace: [ 3012.058883] [ 3012.058891] ? show_regs.cold.9+0x22/0x2f [ 3012.059037] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3012.059045] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3012.059049] ? watchdog+0x40/0x40 [ 3012.059053] ? hrtimer_interrupt+0x128/0x2c0 [ 3012.059114] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3012.059127] ? apic_timer_interrupt+0xf/0x20 [ 3012.059132] [ 3012.059135] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3012.059140] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3012.059148] do_raw_spin_lock+0x177/0x190 [ 3012.059153] _raw_spin_lock+0x21/0x30 [ 3012.059259] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3012.059573] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3012.059999] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3012.060419] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3012.060639] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3012.060957] ? woken_wake_function+0x30/0x30 [ 3012.060969] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3012.061201] kthread+0x1d1/0x200 [ 3012.061208] ? set_kthread_struct+0x70/0x70 [ 3012.061212] ret_from_fork+0x1f/0x30 [ 3012.066786] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3012.067530] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3012.067537] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3012.067540] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3012.067553] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3012.067558] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3012.067562] RAX: 000000000000490c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3012.067564] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3012.067567] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3012.067569] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3012.067571] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3012.067574] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3012.067577] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3012.067579] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3012.067587] Call Trace: [ 3012.067592] [ 3012.067598] ? show_regs.cold.9+0x22/0x2f [ 3012.067608] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3012.067613] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3012.067617] ? watchdog+0x40/0x40 [ 3012.067622] ? hrtimer_interrupt+0x128/0x2c0 [ 3012.067626] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3012.067635] ? apic_timer_interrupt+0xf/0x20 [ 3012.067639] [ 3012.067642] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3012.067650] do_raw_spin_lock+0x177/0x190 [ 3012.067655] _raw_spin_lock+0x21/0x30 [ 3012.067660] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3012.068516] ? do_raw_spin_unlock+0x75/0x190 [ 3012.068521] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3012.068742] ? do_wait_intr+0xf0/0xf0 [ 3012.068750] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3012.080170] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3012.080243] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3012.080264] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3012.080266] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3012.080281] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3012.080285] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3012.080290] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3012.080292] RDX: 0000000000002751 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3012.080294] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3012.080296] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3012.080299] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3012.080301] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3012.080304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3012.080306] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3012.080314] Call Trace: [ 3012.080319] [ 3012.080324] ? show_regs.cold.9+0x22/0x2f [ 3012.080332] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3012.080337] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3012.080341] ? watchdog+0x40/0x40 [ 3012.080345] ? hrtimer_interrupt+0x128/0x2c0 [ 3012.080349] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3012.080358] ? apic_timer_interrupt+0xf/0x20 [ 3012.080362] [ 3012.080364] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3012.080369] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3012.080373] do_raw_spin_lock+0x177/0x190 [ 3012.080377] _raw_spin_lock+0x21/0x30 [ 3012.080382] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3012.080645] obd_mod_free+0x46/0x140 [ptlrpc] [ 3012.082299] mdc_commit_open+0x91/0xb0 [mdc] [ 3012.082339] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3012.082544] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3012.083182] after_reply+0xe2c/0x1780 [ptlrpc] [ 3012.083454] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3012.096136] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3012.102424] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3012.163502] ? __next_timer_interrupt+0x160/0x160 [ 3012.181695] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3012.181835] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3012.181876] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3012.181907] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3012.181934] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3012.182020] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3012.185035] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3012.185076] lmv_intent_open+0x338/0x1150 [lmv] [ 3012.185108] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3012.185628] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3012.185709] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3012.186175] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3012.186316] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3012.186396] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3012.186974] ? do_raw_spin_unlock+0x75/0x190 [ 3012.186995] ? _raw_spin_unlock+0x12/0x30 [ 3012.187002] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3012.187029] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3012.187049] ll_atomic_open+0x36a/0xd70 [lustre] [ 3012.187165] lookup_open+0x356/0xb40 [ 3012.187180] do_last+0x313/0xfc0 [ 3012.187203] ? nd_jump_root+0xe5/0x160 [ 3012.187209] ? path_init+0x437/0x520 [ 3012.187214] path_openat+0xf7/0x500 [ 3012.187224] ? do_raw_spin_unlock+0x75/0x190 [ 3012.187229] do_filp_open+0x99/0x140 [ 3012.187236] ? getname_flags+0x6e/0x330 [ 3012.187245] ? __check_object_size+0xff/0x256 [ 3012.187252] ? do_raw_spin_unlock+0x75/0x190 [ 3012.187256] ? _raw_spin_unlock+0x12/0x30 [ 3012.187260] do_sys_openat2+0x2b4/0x410 [ 3012.187269] do_sys_open+0x73/0xa0 [ 3012.187273] __x64_sys_openat+0x24/0x30 [ 3012.187276] do_syscall_64+0xc1/0x440 [ 3012.187282] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3012.187289] RIP: 0033:0x7f53425a5332 [ 3012.187295] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3012.187299] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3012.187308] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3012.187310] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3012.187312] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3012.187317] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3012.187319] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3013.022031] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3013.024081] ? do_wait_intr+0xf0/0xf0 [ 3013.025966] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3013.028284] kthread+0x1d1/0x200 [ 3013.030380] ? set_kthread_struct+0x70/0x70 [ 3013.031923] ret_from_fork+0x1f/0x30 [ 3028.052171] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 3028.058058] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3028.076490] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3028.081243] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3028.086404] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3028.089256] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3028.091192] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3028.098576] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3028.101735] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3028.104561] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3028.108404] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3028.112896] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3028.116129] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3028.119204] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3028.122769] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3028.125124] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3028.128084] Call Trace: [ 3028.129143] [ 3028.130107] ? show_regs.cold.9+0x22/0x2f [ 3028.131888] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3028.133740] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3028.135134] ? watchdog+0x40/0x40 [ 3028.136780] ? hrtimer_interrupt+0x128/0x2c0 [ 3028.138346] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3028.140841] ? apic_timer_interrupt+0xf/0x20 [ 3028.142767] [ 3028.144090] ? native_safe_halt+0x2e/0x50 [ 3028.145891] kvm_wait+0xcd/0xe0 [ 3028.147207] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3028.149593] ? newidle_balance.isra.93+0x207/0x580 [ 3028.151514] do_raw_spin_lock+0x177/0x190 [ 3028.153096] _raw_spin_lock+0x21/0x30 [ 3028.154982] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3028.157418] process_one_work+0x2c8/0x700 [ 3028.159101] worker_thread+0x296/0x6e0 [ 3028.160767] ? rescuer_thread+0x570/0x570 [ 3028.162516] kthread+0x1d1/0x200 [ 3028.163859] ? set_kthread_struct+0x70/0x70 [ 3028.165632] ret_from_fork+0x1f/0x30 [ 3033.567047] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 396s! [ 3033.570502] Showing busy workqueues and worker pools: [ 3033.572929] workqueue events: flags=0x0 [ 3033.574787] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3033.578153] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3033.585907] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3033.590427] pending: cpu_latency_work [ptlrpc] [ 3033.593751] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3033.603911] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3033.608564] workqueue events_long: flags=0x0 [ 3033.610789] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3033.614275] pending: lu_objects_flush [obdclass] [ 3033.617239] workqueue events_unbound: flags=0x2 [ 3033.620330] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3033.624119] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3033.628663] workqueue events_power_efficient: flags=0x80 [ 3033.631445] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3033.635902] pending: check_lifetime, do_cache_clean [ 3033.638817] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3033.643765] pending: neigh_periodic_work, neigh_periodic_work [ 3033.649672] workqueue mm_percpu_wq: flags=0x8 [ 3033.651978] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3033.656697] pending: vmstat_update, lru_add_drain_per_cpu [ 3033.658769] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3033.665442] pending: vmstat_update, lru_add_drain_per_cpu [ 3033.667786] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3033.671783] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3033.675204] workqueue xprtiod: flags=0xa [ 3033.676856] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3033.680571] pending: xprt_autoclose [ 3033.683203] workqueue ipv6_addrconf: flags=0x40008 [ 3033.684339] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3033.686948] pending: addrconf_verify_work [ 3033.688489] workqueue knbd0-recv: flags=0x1a [ 3033.690489] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3033.692173] in-flight: 651:recv_work [ 3033.693513] workqueue ptlrpc_pinger: flags=0x2 [ 3033.694851] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3033.699411] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3033.701540] pool 8: cpus=0-3 flags=0x4 nice=0 hung=396s workers=4 idle: 15979 31824 8 [ 3033.704733] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3024s workers=2 idle: 654 [ 3040.057778] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 3040.058017] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 3040.059654] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 3040.059663] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3040.059749] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3040.060175] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3040.060180] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3040.060200] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3040.060204] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3040.060209] RAX: 0000000000007188 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3040.060212] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3040.060214] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3040.060216] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3040.060219] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3040.060222] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3040.060225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3040.060231] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3040.060247] Call Trace: [ 3040.060252] [ 3040.060258] ? show_regs.cold.9+0x22/0x2f [ 3040.060269] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3040.060281] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3040.060285] ? watchdog+0x40/0x40 [ 3040.060290] ? hrtimer_interrupt+0x128/0x2c0 [ 3040.060295] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3040.060303] ? apic_timer_interrupt+0xf/0x20 [ 3040.060309] [ 3040.060312] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3040.060319] do_raw_spin_lock+0x177/0x190 [ 3040.060329] _raw_spin_lock+0x21/0x30 [ 3040.060334] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3040.061311] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) [ 3040.061344] ? do_raw_spin_unlock+0x75/0x190 [ 3040.061375] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3040.061600] ? do_wait_intr+0xf0/0xf0 [ 3040.061609] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3040.063133] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3040.063407] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3040.063641] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3040.063784] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3040.064280] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3040.064313] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3040.064343] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3040.064440] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3040.064705] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3040.064736] lmv_intent_open+0x338/0x1150 [lmv] [ 3040.064975] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3040.065042] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3040.065049] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3040.065052] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3040.065064] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3040.065068] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3040.065072] RAX: 000000000000605c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3040.065075] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3040.065077] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3040.065079] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3040.065082] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3040.065085] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3040.065088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3040.065090] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3040.065099] Call Trace: [ 3040.065103] [ 3040.065106] ? show_regs.cold.9+0x22/0x2f [ 3040.065434] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3040.065440] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3040.065444] ? watchdog+0x40/0x40 [ 3040.065449] ? hrtimer_interrupt+0x128/0x2c0 [ 3040.065453] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3040.065462] ? apic_timer_interrupt+0xf/0x20 [ 3040.065466] [ 3040.065469] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3040.065474] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3040.065482] do_raw_spin_lock+0x177/0x190 [ 3040.065486] _raw_spin_lock+0x21/0x30 [ 3040.065491] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3040.065786] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3040.065978] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3040.064767] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3040.067024] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3040.067119] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3040.067153] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3040.067449] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3040.067705] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3040.068453] ? do_raw_spin_unlock+0x75/0x190 [ 3040.068464] ? _raw_spin_unlock+0x12/0x30 [ 3040.068494] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3040.068531] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3040.068557] ll_atomic_open+0x36a/0xd70 [lustre] [ 3040.068665] lookup_open+0x356/0xb40 [ 3040.068688] do_last+0x313/0xfc0 [ 3040.068693] ? nd_jump_root+0xe5/0x160 [ 3040.068698] ? path_init+0x437/0x520 [ 3040.068702] path_openat+0xf7/0x500 [ 3040.068707] ? do_raw_spin_unlock+0x75/0x190 [ 3040.068712] do_filp_open+0x99/0x140 [ 3040.068720] ? getname_flags+0x6e/0x330 [ 3040.068724] ? __check_object_size+0xff/0x256 [ 3040.068732] ? do_raw_spin_unlock+0x75/0x190 [ 3040.068736] ? _raw_spin_unlock+0x12/0x30 [ 3040.068746] do_sys_openat2+0x2b4/0x410 [ 3040.068751] do_sys_open+0x73/0xa0 [ 3040.068756] __x64_sys_openat+0x24/0x30 [ 3040.068760] do_syscall_64+0xc1/0x440 [ 3040.068771] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3040.068778] RIP: 0033:0x7f53425a5332 [ 3040.068785] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3040.068790] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3040.068796] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3040.068798] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3040.068811] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3040.068813] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3040.068816] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3040.073154] ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3040.073217] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3040.073225] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3040.073228] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3040.073306] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3040.073311] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3040.073316] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3040.073318] RDX: 000000000000512c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3040.073321] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3040.073323] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3040.073326] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3040.073329] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3040.073332] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3040.073334] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3040.073343] Call Trace: [ 3040.073348] [ 3040.073355] ? show_regs.cold.9+0x22/0x2f [ 3040.073365] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3040.073371] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3040.073375] ? watchdog+0x40/0x40 [ 3040.073380] ? hrtimer_interrupt+0x128/0x2c0 [ 3040.073384] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3040.073392] ? apic_timer_interrupt+0xf/0x20 [ 3040.073397] [ 3040.073400] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3040.073405] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3040.073410] do_raw_spin_lock+0x177/0x190 [ 3040.073415] _raw_spin_lock+0x21/0x30 [ 3040.073421] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3040.073693] obd_mod_free+0x46/0x140 [ptlrpc] [ 3040.096837] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3040.103202] mdc_commit_open+0x91/0xb0 [mdc] [ 3040.103261] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3040.103535] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3040.103756] after_reply+0xe2c/0x1780 [ptlrpc] [ 3040.103966] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3040.104181] ? __next_timer_interrupt+0x160/0x160 [ 3040.104192] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3040.104429] ? do_wait_intr+0xf0/0xf0 [ 3040.104438] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3040.104659] kthread+0x1d1/0x200 [ 3040.104665] ? set_kthread_struct+0x70/0x70 [ 3040.104669] ret_from_fork+0x1f/0x30 [ 3040.476528] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3040.479220] ? woken_wake_function+0x30/0x30 [ 3040.480559] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3040.484958] kthread+0x1d1/0x200 [ 3040.487459] ? set_kthread_struct+0x70/0x70 [ 3040.489340] ret_from_fork+0x1f/0x30 [ 3054.451360] rcu: INFO: rcu_sched self-detected stall on CPU [ 3054.456114] rcu: 2-....: (515330 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 3054.463325] (t=960025 jiffies g=274185 q=63810) [ 3054.466156] rcu: rcu_sched kthread starved for 418391 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 3054.473125] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3054.480061] rcu: RCU grace-period kthread stack dump: [ 3054.482834] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3054.488882] Call Trace: [ 3054.490732] __schedule+0x351/0xcb0 [ 3054.492835] schedule+0xc0/0x180 [ 3054.494414] schedule_timeout+0xb4/0x190 [ 3054.496762] ? do_raw_spin_unlock+0x75/0x190 [ 3054.499395] ? __next_timer_interrupt+0x160/0x160 [ 3054.502553] ? prepare_to_swait_event+0xf3/0x140 [ 3054.505747] rcu_gp_kthread+0x8dd/0x15b0 [ 3054.508674] ? force_qs_rnp+0x2c0/0x2c0 [ 3054.510123] kthread+0x1d1/0x200 [ 3054.511145] ? set_kthread_struct+0x70/0x70 [ 3054.515323] ret_from_fork+0x1f/0x30 [ 3054.518524] rcu: Stack dump where RCU GP kthread last ran: [ 3054.525621] Sending NMI from CPU 2 to CPUs 0: [ 3054.530865] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 3054.533949] Sending NMI from CPU 2 to CPUs 1: [ 3054.552222] NMI backtrace for cpu 1 [ 3054.552230] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3054.552233] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3054.552234] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3054.552235] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3054.552238] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [ 3054.552240] RAX: 0000000000003307 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3054.552242] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3054.552243] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3054.552245] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3054.552246] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3054.552248] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3054.552249] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3054.552250] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3054.552251] Call Trace: [ 3054.552252] [ 3054.552253] ? show_regs.cold.9+0x22/0x2f [ 3054.552254] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3054.552255] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3054.552256] ? nmi_handle+0xbd/0x210 [ 3054.552257] ? default_do_nmi+0x4b/0x1b0 [ 3054.552258] ? do_nmi+0x280/0x380 [ 3054.552259] ? end_repeat_nmi+0x16/0x1f [ 3054.552260] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3054.552261] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3054.552262] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3054.552263] [ 3054.552263] do_raw_spin_lock+0x177/0x190 [ 3054.552264] _raw_spin_lock+0x21/0x30 [ 3054.552265] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3054.552266] ? do_raw_spin_unlock+0x75/0x190 [ 3054.552267] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3054.552268] ? do_wait_intr+0xf0/0xf0 [ 3054.552269] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3054.552270] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3054.552271] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3054.552272] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3054.552273] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3054.552274] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3054.552275] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3054.552276] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3054.552277] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3054.552278] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3054.552279] lmv_intent_open+0x338/0x1150 [lmv] [ 3054.552281] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3054.552282] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3054.552283] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3054.552284] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3054.552286] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3054.552287] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3054.552288] ? do_raw_spin_unlock+0x75/0x190 [ 3054.552289] ? _raw_spin_unlock+0x12/0x30 [ 3054.552290] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3054.552291] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3054.552292] ll_atomic_open+0x36a/0xd70 [lustre] [ 3054.552293] lookup_open+0x356/0xb40 [ 3054.552294] do_last+0x313/0xfc0 [ 3054.552295] ? nd_jump_root+0xe5/0x160 [ 3054.552296] ? path_init+0x437/0x520 [ 3054.552297] path_openat+0xf7/0x500 [ 3054.552298] ? do_raw_spin_unlock+0x75/0x190 [ 3054.552299] do_filp_open+0x99/0x140 [ 3054.552300] ? getname_flags+0x6e/0x330 [ 3054.552301] ? __check_object_size+0xff/0x256 [ 3054.552302] ? do_raw_spin_unlock+0x75/0x190 [ 3054.552302] ? _raw_spin_unlock+0x12/0x30 [ 3054.552304] do_sys_openat2+0x2b4/0x410 [ 3054.552305] do_sys_open+0x73/0xa0 [ 3054.552306] __x64_sys_openat+0x24/0x30 [ 3054.552307] do_syscall_64+0xc1/0x440 [ 3054.552308] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3054.552309] RIP: 0033:0x7f53425a5332 [ 3054.552310] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3054.552312] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3054.552314] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3054.552315] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3054.552316] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3054.552318] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3054.552319] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3054.557391] NMI backtrace for cpu 2 [ 3055.011392] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3055.016404] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3055.020187] Call Trace: [ 3055.021319] [ 3055.022224] ? dump_stack+0xbb/0x10e [ 3055.023821] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3055.025436] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3055.027356] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3055.029635] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3055.031774] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3055.033918] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3055.036246] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3055.038705] ? update_process_times+0x6d/0xc0 [ 3055.040583] ? tick_sched_handle.isra.18+0x27/0x90 [ 3055.042794] ? tick_sched_timer+0x53/0xb0 [ 3055.044603] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3055.046992] ? tick_sched_handle.isra.18+0x90/0x90 [ 3055.048938] ? hrtimer_interrupt+0x128/0x2c0 [ 3055.050982] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3055.053055] ? apic_timer_interrupt+0xf/0x20 [ 3055.055208] [ 3055.056100] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3055.058604] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3055.060777] ? do_raw_spin_lock+0x177/0x190 [ 3055.062720] ? _raw_spin_lock+0x21/0x30 [ 3055.064534] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3055.067155] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3055.069802] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3055.077629] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3055.082792] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3055.086063] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3055.088608] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3055.091404] ? __next_timer_interrupt+0x160/0x160 [ 3055.095071] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3055.097735] ? do_wait_intr+0xf0/0xf0 [ 3055.099301] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3055.102261] ? kthread+0x1d1/0x200 [ 3055.103400] ? set_kthread_struct+0x70/0x70 [ 3055.105407] ? ret_from_fork+0x1f/0x30 [ 3063.782114] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 426s! [ 3063.787138] Showing busy workqueues and worker pools: [ 3063.789279] workqueue events: flags=0x0 [ 3063.791257] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3063.795053] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3063.799926] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3063.803280] pending: cpu_latency_work [ptlrpc] [ 3063.806064] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3063.809337] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3063.812297] workqueue events_long: flags=0x0 [ 3063.814585] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3063.818325] pending: lu_objects_flush [obdclass] [ 3063.825340] workqueue events_unbound: flags=0x2 [ 3063.827374] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3063.833107] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3063.835919] workqueue events_power_efficient: flags=0x80 [ 3063.838618] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3063.846219] pending: check_lifetime, do_cache_clean [ 3063.849361] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3063.857375] pending: neigh_periodic_work, neigh_periodic_work [ 3063.861161] workqueue mm_percpu_wq: flags=0x8 [ 3063.864816] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3063.868937] pending: vmstat_update, lru_add_drain_per_cpu [ 3063.871325] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3063.878254] pending: vmstat_update, lru_add_drain_per_cpu [ 3063.886126] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3063.891323] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3063.897388] workqueue xprtiod: flags=0xa [ 3063.903244] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3063.909335] pending: xprt_autoclose [ 3063.910392] workqueue ipv6_addrconf: flags=0x40008 [ 3063.915146] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3063.919439] pending: addrconf_verify_work [ 3063.921229] workqueue knbd0-recv: flags=0x1a [ 3063.923137] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3063.927282] in-flight: 651:recv_work [ 3063.930225] workqueue ptlrpc_pinger: flags=0x2 [ 3063.932984] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3063.935284] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3063.938036] pool 8: cpus=0-3 flags=0x4 nice=0 hung=426s workers=4 idle: 15979 31824 8 [ 3063.944231] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3054s workers=2 idle: 654 [ 3068.058891] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 3068.062867] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3068.102238] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3068.115313] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3068.119094] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3068.124462] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3068.147801] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3068.159046] RAX: 0000000000007f09 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3068.162171] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3068.170384] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3068.176487] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3068.183862] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3068.191920] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3068.198085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3068.205561] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3068.209367] Call Trace: [ 3068.210556] [ 3068.213519] ? show_regs.cold.9+0x22/0x2f [ 3068.219703] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3068.221373] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3068.225291] ? watchdog+0x40/0x40 [ 3068.230616] ? hrtimer_interrupt+0x128/0x2c0 [ 3068.234849] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3068.236582] ? apic_timer_interrupt+0xf/0x20 [ 3068.238326] [ 3068.239589] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3068.246431] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3068.249356] do_raw_spin_lock+0x177/0x190 [ 3068.251110] _raw_spin_lock+0x21/0x30 [ 3068.254535] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3068.259001] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3068.267737] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3068.276214] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3068.283096] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3068.284670] ? woken_wake_function+0x30/0x30 [ 3068.291332] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3068.302659] kthread+0x1d1/0x200 [ 3068.303707] ? set_kthread_struct+0x70/0x70 [ 3068.311505] ret_from_fork+0x1f/0x30 [ 3080.055209] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 3080.057028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 3080.058894] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3080.058978] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3080.058987] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3080.058990] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3080.059005] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3080.059011] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [ 3080.067367] Modules linked in: lustre(O) [ 3080.093091] ORIG_RAX: ffffffffffffff13 [ 3080.093103] RAX: 00000000000079f6 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3080.093106] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3080.093108] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3080.093110] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3080.093113] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3080.093116] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3080.093120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3080.093122] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3080.093131] Call Trace: [ 3080.093139] [ 3080.093144] ? show_regs.cold.9+0x22/0x2f [ 3080.093165] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3080.093172] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3080.093389] ? watchdog+0x40/0x40 [ 3080.093395] ? hrtimer_interrupt+0x128/0x2c0 [ 3080.093399] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3080.093414] ? apic_timer_interrupt+0xf/0x20 [ 3080.093419] [ 3080.093422] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3080.093435] do_raw_spin_lock+0x177/0x190 [ 3080.093444] _raw_spin_lock+0x21/0x30 [ 3080.093451] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3080.094043] ? do_raw_spin_unlock+0x75/0x190 [ 3080.094050] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3080.094277] ? do_wait_intr+0xf0/0xf0 [ 3080.094287] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3080.094486] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3080.095007] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3080.095274] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3080.095404] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3080.108677] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3080.108758] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3080.108765] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3080.108769] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3080.108781] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3080.108786] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3080.108790] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3080.108793] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3080.108795] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3080.108797] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3080.108799] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3080.108802] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3080.108805] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3080.108808] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3080.108815] Call Trace: [ 3080.108820] [ 3080.108825] ? show_regs.cold.9+0x22/0x2f [ 3080.108835] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3080.108841] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3080.108845] ? watchdog+0x40/0x40 [ 3080.108849] ? hrtimer_interrupt+0x128/0x2c0 [ 3080.108854] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3080.108860] ? apic_timer_interrupt+0xf/0x20 [ 3080.108864] [ 3080.108868] ? native_safe_halt+0x2e/0x50 [ 3080.115793] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3080.117821] kvm_wait+0xcd/0xe0 [ 3080.117841] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3080.117849] do_raw_spin_lock+0x177/0x190 [ 3080.117853] _raw_spin_lock+0x21/0x30 [ 3080.117860] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3080.130009] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3080.130070] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3080.130204] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3080.130487] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3080.130518] lmv_intent_open+0x338/0x1150 [lmv] [ 3080.130625] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3080.131087] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3080.131170] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3080.131194] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3080.131285] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3080.131367] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3080.131448] ? do_raw_spin_unlock+0x75/0x190 [ 3080.131457] ? _raw_spin_unlock+0x12/0x30 [ 3080.131464] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3080.131490] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3080.131508] ll_atomic_open+0x36a/0xd70 [lustre] [ 3080.137907] obd_mod_free+0x46/0x140 [ptlrpc] [ 3080.138211] mdc_commit_open+0x91/0xb0 [mdc] [ 3080.138254] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3080.138451] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3080.138851] after_reply+0xe2c/0x1780 [ptlrpc] [ 3080.139052] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3080.139249] ? __next_timer_interrupt+0x160/0x160 [ 3080.139260] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3080.139476] ? do_wait_intr+0xf0/0xf0 [ 3080.139487] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3080.142014] lookup_open+0x356/0xb40 [ 3080.142031] do_last+0x313/0xfc0 [ 3080.142036] ? nd_jump_root+0xe5/0x160 [ 3080.142041] ? path_init+0x437/0x520 [ 3080.142046] path_openat+0xf7/0x500 [ 3080.142050] ? do_raw_spin_unlock+0x75/0x190 [ 3080.142057] do_filp_open+0x99/0x140 [ 3080.142064] ? getname_flags+0x6e/0x330 [ 3080.142068] ? __check_object_size+0xff/0x256 [ 3080.142075] ? do_raw_spin_unlock+0x75/0x190 [ 3080.142079] ? _raw_spin_unlock+0x12/0x30 [ 3080.142085] do_sys_openat2+0x2b4/0x410 [ 3080.142089] do_sys_open+0x73/0xa0 [ 3080.142094] __x64_sys_openat+0x24/0x30 [ 3080.142097] do_syscall_64+0xc1/0x440 [ 3080.142104] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3080.142110] RIP: 0033:0x7f53425a5332 [ 3080.142119] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3080.142123] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3080.142129] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3080.142132] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3080.142134] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3080.142136] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3080.142138] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3080.564793] kthread+0x1d1/0x200 [ 3080.567329] ? set_kthread_struct+0x70/0x70 [ 3080.570185] ret_from_fork+0x1f/0x30 [ 3088.052142] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 3088.056517] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3088.104049] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3088.111839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3088.121256] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3088.126085] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3088.131620] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3088.144113] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3088.146749] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3088.150094] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3088.155583] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3088.160881] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3088.166847] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3088.176839] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3088.181373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3088.186730] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3088.200573] Call Trace: [ 3088.201133] [ 3088.202931] ? show_regs.cold.9+0x22/0x2f [ 3088.206517] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3088.219378] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3088.230199] ? watchdog+0x40/0x40 [ 3088.235727] ? hrtimer_interrupt+0x128/0x2c0 [ 3088.238766] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3088.244605] ? apic_timer_interrupt+0xf/0x20 [ 3088.249099] [ 3088.251136] ? native_safe_halt+0x2e/0x50 [ 3088.255121] kvm_wait+0xcd/0xe0 [ 3088.258227] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3088.263067] ? newidle_balance.isra.93+0x207/0x580 [ 3088.267585] do_raw_spin_lock+0x177/0x190 [ 3088.269312] _raw_spin_lock+0x21/0x30 [ 3088.274593] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3088.277873] process_one_work+0x2c8/0x700 [ 3088.282677] worker_thread+0x296/0x6e0 [ 3088.285609] ? rescuer_thread+0x570/0x570 [ 3088.287073] kthread+0x1d1/0x200 [ 3088.288287] ? set_kthread_struct+0x70/0x70 [ 3088.291593] ret_from_fork+0x1f/0x30 [ 3093.985751] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 456s! [ 3094.001854] Showing busy workqueues and worker pools: [ 3094.008708] workqueue events: flags=0x0 [ 3094.011047] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3094.013862] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3094.018215] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3094.027761] pending: cpu_latency_work [ptlrpc] [ 3094.032757] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3094.048273] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3094.060018] workqueue events_long: flags=0x0 [ 3094.064030] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3094.068036] pending: lu_objects_flush [obdclass] [ 3094.076773] workqueue events_unbound: flags=0x2 [ 3094.078408] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3094.097680] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3094.101859] workqueue events_power_efficient: flags=0x80 [ 3094.104018] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3094.125622] pending: check_lifetime, do_cache_clean [ 3094.133847] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3094.137723] pending: neigh_periodic_work, neigh_periodic_work [ 3094.140723] workqueue mm_percpu_wq: flags=0x8 [ 3094.147891] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3094.151125] pending: vmstat_update, lru_add_drain_per_cpu [ 3094.154755] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3094.161856] pending: vmstat_update, lru_add_drain_per_cpu [ 3094.164743] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3094.173979] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3094.176560] workqueue xprtiod: flags=0xa [ 3094.177624] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3094.186990] pending: xprt_autoclose [ 3094.190055] workqueue ipv6_addrconf: flags=0x40008 [ 3094.193119] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3094.200715] pending: addrconf_verify_work [ 3094.208698] workqueue knbd0-recv: flags=0x1a [ 3094.214174] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3094.222630] in-flight: 651:recv_work [ 3094.224236] workqueue ptlrpc_pinger: flags=0x2 [ 3094.230522] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3094.233847] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3094.236931] pool 8: cpus=0-3 flags=0x4 nice=0 hung=457s workers=4 idle: 15979 31824 8 [ 3094.240525] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3084s workers=2 idle: 654 [ 3096.061209] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 3096.075907] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3096.098940] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3096.104205] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3096.109706] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3096.113141] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3096.129983] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3096.135093] RAX: 0000000000001d10 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3096.145687] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3096.149876] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3096.158431] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3096.161205] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3096.166934] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3096.171585] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3096.174730] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3096.178219] Call Trace: [ 3096.179730] [ 3096.182417] ? show_regs.cold.9+0x22/0x2f [ 3096.183663] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3096.185235] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3096.187498] ? watchdog+0x40/0x40 [ 3096.188909] ? hrtimer_interrupt+0x128/0x2c0 [ 3096.190756] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3096.193811] ? apic_timer_interrupt+0xf/0x20 [ 3096.195950] [ 3096.197181] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3096.200019] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3096.202721] do_raw_spin_lock+0x177/0x190 [ 3096.206075] _raw_spin_lock+0x21/0x30 [ 3096.210847] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3096.213650] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3096.218556] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3096.225419] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3096.228535] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3096.233697] ? woken_wake_function+0x30/0x30 [ 3096.236683] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3096.240765] kthread+0x1d1/0x200 [ 3096.244708] ? set_kthread_struct+0x70/0x70 [ 3096.247014] ret_from_fork+0x1f/0x30 [ 3108.055198] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 3108.057106] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 3108.058487] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3108.058566] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3108.058573] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3108.058576] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3108.058590] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3108.058594] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3108.058599] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3108.058601] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3108.058603] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3108.058605] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3108.058608] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3108.058612] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3108.058614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3108.058617] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3108.058625] Call Trace: [ 3108.058630] [ 3108.058636] ? show_regs.cold.9+0x22/0x2f [ 3108.058647] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3108.058653] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3108.058657] ? watchdog+0x40/0x40 [ 3108.058661] ? hrtimer_interrupt+0x128/0x2c0 [ 3108.058665] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3108.058672] ? apic_timer_interrupt+0xf/0x20 [ 3108.058677] [ 3108.058679] ? native_safe_halt+0x2e/0x50 [ 3108.058683] kvm_wait+0xcd/0xe0 [ 3108.058690] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3108.058700] do_raw_spin_lock+0x177/0x190 [ 3108.058704] _raw_spin_lock+0x21/0x30 [ 3108.058707] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3108.058984] ? do_raw_spin_unlock+0x75/0x190 [ 3108.058989] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3108.059366] ? do_wait_intr+0xf0/0xf0 [ 3108.059374] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3108.059580] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3108.059803] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3108.064801] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3108.064883] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3108.064890] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3108.064893] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3108.064906] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3108.064911] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3108.064915] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3108.064918] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3108.064920] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3108.064922] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3108.064924] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3108.064927] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3108.064930] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3108.064933] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3108.064941] Call Trace: [ 3108.064946] [ 3108.064951] ? show_regs.cold.9+0x22/0x2f [ 3108.064960] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3108.064966] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3108.064970] ? watchdog+0x40/0x40 [ 3108.064974] ? hrtimer_interrupt+0x128/0x2c0 [ 3108.064978] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3108.064984] ? apic_timer_interrupt+0xf/0x20 [ 3108.064989] [ 3108.064991] ? native_safe_halt+0x2e/0x50 [ 3108.064995] kvm_wait+0xcd/0xe0 [ 3108.094546] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3108.094667] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3108.094705] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3108.094736] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3108.094762] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3108.094845] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3108.102674] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3108.102697] do_raw_spin_lock+0x177/0x190 [ 3108.102702] _raw_spin_lock+0x21/0x30 [ 3108.102711] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3108.102983] obd_mod_free+0x46/0x140 [ptlrpc] [ 3108.103336] mdc_commit_open+0x91/0xb0 [mdc] [ 3108.103379] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3108.103583] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3108.103813] after_reply+0xe2c/0x1780 [ptlrpc] [ 3108.104007] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3108.104371] ? __next_timer_interrupt+0x160/0x160 [ 3108.104382] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3108.104584] ? do_wait_intr+0xf0/0xf0 [ 3108.104592] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3108.104800] kthread+0x1d1/0x200 [ 3108.104805] ? set_kthread_struct+0x70/0x70 [ 3108.104809] ret_from_fork+0x1f/0x30 [ 3108.406660] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3108.415876] lmv_intent_open+0x338/0x1150 [lmv] [ 3108.418612] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3108.422991] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3108.426813] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3108.429593] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3108.433212] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3108.438907] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3108.442623] ? do_raw_spin_unlock+0x75/0x190 [ 3108.445322] ? _raw_spin_unlock+0x12/0x30 [ 3108.449656] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3108.454961] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3108.456613] ll_atomic_open+0x36a/0xd70 [lustre] [ 3108.464017] lookup_open+0x356/0xb40 [ 3108.471366] do_last+0x313/0xfc0 [ 3108.473055] ? nd_jump_root+0xe5/0x160 [ 3108.474368] ? path_init+0x437/0x520 [ 3108.478243] path_openat+0xf7/0x500 [ 3108.480237] ? do_raw_spin_unlock+0x75/0x190 [ 3108.486001] do_filp_open+0x99/0x140 [ 3108.488226] ? getname_flags+0x6e/0x330 [ 3108.490572] ? __check_object_size+0xff/0x256 [ 3108.493263] ? do_raw_spin_unlock+0x75/0x190 [ 3108.494824] ? _raw_spin_unlock+0x12/0x30 [ 3108.497733] do_sys_openat2+0x2b4/0x410 [ 3108.499844] do_sys_open+0x73/0xa0 [ 3108.502215] __x64_sys_openat+0x24/0x30 [ 3108.504559] do_syscall_64+0xc1/0x440 [ 3108.506031] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3108.507668] RIP: 0033:0x7f53425a5332 [ 3108.508650] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3108.521979] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3108.534832] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3108.537450] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3108.539842] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3108.554561] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3108.559815] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3120.057377] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 3120.060353] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3120.126471] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3120.136315] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3120.143483] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3120.153589] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3120.166663] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3120.197491] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3120.210808] RAX: 000000000000762d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3120.226247] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3120.238568] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3120.249547] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3120.251823] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3120.272048] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3120.283142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3120.297519] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3120.303090] Call Trace: [ 3120.314240] [ 3120.317719] ? show_regs.cold.9+0x22/0x2f [ 3120.323508] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3120.328899] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3120.338719] ? watchdog+0x40/0x40 [ 3120.340246] ? hrtimer_interrupt+0x128/0x2c0 [ 3120.341742] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3120.354687] ? apic_timer_interrupt+0xf/0x20 [ 3120.359203] [ 3120.369574] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3120.379766] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3120.381750] ? newidle_balance.isra.93+0x207/0x580 [ 3120.394887] do_raw_spin_lock+0x177/0x190 [ 3120.401806] _raw_spin_lock+0x21/0x30 [ 3120.407141] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3120.414668] process_one_work+0x2c8/0x700 [ 3120.416141] worker_thread+0x296/0x6e0 [ 3120.425965] ? rescuer_thread+0x570/0x570 [ 3120.437913] kthread+0x1d1/0x200 [ 3120.439533] ? set_kthread_struct+0x70/0x70 [ 3120.445581] ret_from_fork+0x1f/0x30 [ 3124.058299] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 3124.074109] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3124.105529] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3124.126826] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3124.131285] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3124.135129] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3124.150839] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3124.155431] RAX: 0000000000006c09 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3124.158910] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3124.165522] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3124.185110] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3124.192779] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3124.205604] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3124.228775] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3124.236501] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3124.253134] Call Trace: [ 3124.258557] [ 3124.260059] ? show_regs.cold.9+0x22/0x2f [ 3124.266611] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3124.267704] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3124.270137] ? watchdog+0x40/0x40 [ 3124.271890] ? hrtimer_interrupt+0x128/0x2c0 [ 3124.274342] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3124.276884] ? apic_timer_interrupt+0xf/0x20 [ 3124.279200] [ 3124.282077] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3124.285711] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3124.288716] do_raw_spin_lock+0x177/0x190 [ 3124.291075] _raw_spin_lock+0x21/0x30 [ 3124.297903] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3124.305337] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3124.314054] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3124.317089] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3124.327887] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3124.344334] ? woken_wake_function+0x30/0x30 [ 3124.348867] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3124.357289] kthread+0x1d1/0x200 [ 3124.358509] ? set_kthread_struct+0x70/0x70 [ 3124.366364] ret_from_fork+0x1f/0x30 [ 3124.713685] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 487s! [ 3124.724303] Showing busy workqueues and worker pools: [ 3124.733422] workqueue events: flags=0x0 [ 3124.736934] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3124.743301] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3124.761566] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3124.774659] pending: cpu_latency_work [ptlrpc] [ 3124.781496] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3124.787694] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3124.794693] workqueue events_long: flags=0x0 [ 3124.796596] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3124.806570] pending: lu_objects_flush [obdclass] [ 3124.812341] workqueue events_unbound: flags=0x2 [ 3124.822568] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3124.833445] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3124.843630] workqueue events_power_efficient: flags=0x80 [ 3124.847266] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3124.861278] pending: check_lifetime, do_cache_clean [ 3124.867295] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3124.869593] pending: neigh_periodic_work, neigh_periodic_work [ 3124.886024] workqueue mm_percpu_wq: flags=0x8 [ 3124.889858] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3124.898453] pending: vmstat_update, lru_add_drain_per_cpu [ 3124.903709] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3124.916720] pending: vmstat_update, lru_add_drain_per_cpu [ 3124.923473] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3124.930338] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3124.938507] workqueue xprtiod: flags=0xa [ 3124.943154] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/256 refcnt=3 [ 3124.953158] pending: xprt_autoclose [ 3124.954526] workqueue ipv6_addrconf: flags=0x40008 [ 3124.966041] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3124.976099] pending: addrconf_verify_work [ 3124.977579] workqueue knbd0-recv: flags=0x1a [ 3124.984976] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3124.996674] in-flight: 651:recv_work [ 3125.000217] workqueue ptlrpc_pinger: flags=0x2 [ 3125.010335] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3125.023650] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3125.041082] pool 8: cpus=0-3 flags=0x4 nice=0 hung=487s workers=4 idle: 15979 31824 8 [ 3125.054449] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3115s workers=2 idle: 654 [ 3136.055911] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 3136.058062] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 3136.059847] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3136.059920] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3136.059927] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3136.059929] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3136.059945] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3136.059949] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3136.059953] RAX: 0000000000005b93 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3136.059956] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3136.059958] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3136.059960] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3136.059962] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3136.059966] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3136.059969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3136.059972] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3136.059981] Call Trace: [ 3136.059986] [ 3136.059992] ? show_regs.cold.9+0x22/0x2f [ 3136.060002] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3136.060007] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3136.060012] ? watchdog+0x40/0x40 [ 3136.060016] ? hrtimer_interrupt+0x128/0x2c0 [ 3136.060021] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3136.060029] ? apic_timer_interrupt+0xf/0x20 [ 3136.060034] [ 3136.060037] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3136.060044] do_raw_spin_lock+0x177/0x190 [ 3136.060467] _raw_spin_lock+0x21/0x30 [ 3136.060474] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3136.060764] ? do_raw_spin_unlock+0x75/0x190 [ 3136.060770] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3136.060966] ? do_wait_intr+0xf0/0xf0 [ 3136.060975] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3136.061381] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3136.061706] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3136.061920] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3136.062043] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3136.062080] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3136.062110] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3136.062242] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3136.062334] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3136.062644] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3136.062676] lmv_intent_open+0x338/0x1150 [lmv] [ 3136.062709] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3136.062802] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3136.062884] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3136.062907] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3136.062997] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3136.063232] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3136.063323] ? do_raw_spin_unlock+0x75/0x190 [ 3136.063331] ? _raw_spin_unlock+0x12/0x30 [ 3136.063338] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3136.063368] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3136.063389] ll_atomic_open+0x36a/0xd70 [lustre] [ 3136.063483] lookup_open+0x356/0xb40 [ 3136.063493] do_last+0x313/0xfc0 [ 3136.063497] ? nd_jump_root+0xe5/0x160 [ 3136.063502] ? path_init+0x437/0x520 [ 3136.063507] path_openat+0xf7/0x500 [ 3136.063512] ? do_raw_spin_unlock+0x75/0x190 [ 3136.063516] do_filp_open+0x99/0x140 [ 3136.063522] ? getname_flags+0x6e/0x330 [ 3136.063527] ? __check_object_size+0xff/0x256 [ 3136.063535] ? do_raw_spin_unlock+0x75/0x190 [ 3136.063538] ? _raw_spin_unlock+0x12/0x30 [ 3136.063543] do_sys_openat2+0x2b4/0x410 [ 3136.063548] do_sys_open+0x73/0xa0 [ 3136.063552] __x64_sys_openat+0x24/0x30 [ 3136.063555] do_syscall_64+0xc1/0x440 [ 3136.063578] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3136.063584] RIP: 0033:0x7f53425a5332 [ 3136.063591] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3136.063596] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3136.063601] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3136.063603] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3136.063605] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3136.063607] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3136.063610] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3136.221819] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3136.232550] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3136.235376] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3136.237388] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3136.238704] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3136.243059] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3136.244991] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3136.246913] RDX: 0000000000002543 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3136.248876] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3136.251608] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3136.254472] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3136.256377] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3136.258410] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3136.259925] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3136.261772] Call Trace: [ 3136.262396] [ 3136.262870] ? show_regs.cold.9+0x22/0x2f [ 3136.263874] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3136.265070] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3136.266278] ? watchdog+0x40/0x40 [ 3136.267107] ? hrtimer_interrupt+0x128/0x2c0 [ 3136.268140] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3136.269211] ? apic_timer_interrupt+0xf/0x20 [ 3136.270202] [ 3136.270670] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3136.271837] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3136.273206] do_raw_spin_lock+0x177/0x190 [ 3136.274226] _raw_spin_lock+0x21/0x30 [ 3136.275181] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3136.276497] obd_mod_free+0x46/0x140 [ptlrpc] [ 3136.277703] mdc_commit_open+0x91/0xb0 [mdc] [ 3136.278716] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3136.280024] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3136.281402] after_reply+0xe2c/0x1780 [ptlrpc] [ 3136.282528] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3136.283870] ? __next_timer_interrupt+0x160/0x160 [ 3136.284970] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3136.286200] ? do_wait_intr+0xf0/0xf0 [ 3136.287027] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3136.288552] kthread+0x1d1/0x200 [ 3136.289451] ? set_kthread_struct+0x70/0x70 [ 3136.290465] ret_from_fork+0x1f/0x30 [ 3152.054880] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 3152.063011] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) [ 3152.063154] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 3152.063160] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 [ 3152.072027] lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3152.072162] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3152.072169] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3152.072175] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3152.072898] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3152.072931] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3152.072935] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3152.072939] RAX: 0000000000007d07 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3152.072941] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3152.072943] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3152.072944] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3152.072946] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3152.072950] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3152.072952] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3152.072954] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3152.072966] Call Trace: [ 3152.072981] [ 3152.072988] ? show_regs.cold.9+0x22/0x2f [ 3152.073004] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3152.073012] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3152.073017] ? watchdog+0x40/0x40 [ 3152.073022] ? hrtimer_interrupt+0x128/0x2c0 [ 3152.073026] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3152.073035] ? apic_timer_interrupt+0xf/0x20 [ 3152.073039] [ 3152.073042] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3152.073047] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3152.073051] ? newidle_balance.isra.93+0x207/0x580 [ 3152.073057] do_raw_spin_lock+0x177/0x190 [ 3152.073062] _raw_spin_lock+0x21/0x30 [ 3152.073066] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3152.074151] process_one_work+0x2c8/0x700 [ 3152.074161] worker_thread+0x296/0x6e0 [ 3152.074166] ? rescuer_thread+0x570/0x570 [ 3152.074170] kthread+0x1d1/0x200 [ 3152.074175] ? set_kthread_struct+0x70/0x70 [ 3152.074179] ret_from_fork+0x1f/0x30 [ 3152.359290] auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3152.373686] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3152.378220] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3152.386661] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3152.393076] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3152.422897] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3152.430539] RAX: 0000000000007d8b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3152.441928] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3152.443731] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3152.451999] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3152.467643] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3152.473802] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3152.479304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3152.484544] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3152.489879] Call Trace: [ 3152.492788] [ 3152.493765] ? show_regs.cold.9+0x22/0x2f [ 3152.495759] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3152.502772] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3152.506709] ? watchdog+0x40/0x40 [ 3152.511875] ? hrtimer_interrupt+0x128/0x2c0 [ 3152.518063] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3152.523001] ? apic_timer_interrupt+0xf/0x20 [ 3152.531018] [ 3152.532449] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3152.535615] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3152.540657] do_raw_spin_lock+0x177/0x190 [ 3152.542501] _raw_spin_lock+0x21/0x30 [ 3152.548163] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3152.556512] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3152.570822] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3152.577689] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3152.581363] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3152.585007] ? woken_wake_function+0x30/0x30 [ 3152.589062] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3152.592872] kthread+0x1d1/0x200 [ 3152.595361] ? set_kthread_struct+0x70/0x70 [ 3152.601441] ret_from_fork+0x1f/0x30 [ 3155.426146] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 518s! [ 3155.436037] Showing busy workqueues and worker pools: [ 3155.445453] workqueue events: flags=0x0 [ 3155.446869] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3155.453465] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3155.470462] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3155.480431] pending: cpu_latency_work [ptlrpc] [ 3155.493070] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3155.510659] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3155.527431] workqueue events_long: flags=0x0 [ 3155.532398] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3155.543745] pending: lu_objects_flush [obdclass] [ 3155.549250] workqueue events_unbound: flags=0x2 [ 3155.550886] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3155.569669] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3155.580928] workqueue events_power_efficient: flags=0x80 [ 3155.586829] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3155.600242] pending: check_lifetime, do_cache_clean [ 3155.602143] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3155.625069] pending: neigh_periodic_work, neigh_periodic_work [ 3155.638324] workqueue mm_percpu_wq: flags=0x8 [ 3155.641764] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3155.649664] pending: vmstat_update, lru_add_drain_per_cpu [ 3155.656053] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3155.667967] pending: vmstat_update, lru_add_drain_per_cpu [ 3155.680664] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3155.683085] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3155.698120] workqueue xprtiod: flags=0xa [ 3155.703942] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3155.711426] pending: xprt_autoclose, xs_error_handle [ 3155.720079] workqueue ipv6_addrconf: flags=0x40008 [ 3155.734693] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3155.746104] pending: addrconf_verify_work [ 3155.753311] workqueue knbd0-recv: flags=0x1a [ 3155.756032] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3155.767694] in-flight: 651:recv_work [ 3155.772588] workqueue ptlrpc_pinger: flags=0x2 [ 3155.773948] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3155.779598] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3155.782870] pool 8: cpus=0-3 flags=0x4 nice=0 hung=518s workers=4 idle: 15979 31824 8 [ 3155.791923] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3146s workers=2 idle: 654 [ 3164.057060] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 3164.061677] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 3164.064661] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3164.064817] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3164.064824] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3164.064827] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3164.064915] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3164.064920] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3164.064925] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3164.064927] RDX: 00000000000010db RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3164.064930] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3164.064932] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3164.064934] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3164.064937] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3164.064939] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3164.064941] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3164.064949] Call Trace: [ 3164.064964] [ 3164.064971] ? show_regs.cold.9+0x22/0x2f [ 3164.064981] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3164.064999] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3164.065004] ? watchdog+0x40/0x40 [ 3164.065008] ? hrtimer_interrupt+0x128/0x2c0 [ 3164.065013] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3164.065025] ? apic_timer_interrupt+0xf/0x20 [ 3164.065029] [ 3164.065032] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3164.065037] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3164.065040] do_raw_spin_lock+0x177/0x190 [ 3164.065045] _raw_spin_lock+0x21/0x30 [ 3164.065050] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3164.065367] obd_mod_free+0x46/0x140 [ptlrpc] [ 3164.065576] mdc_commit_open+0x91/0xb0 [mdc] [ 3164.065618] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3164.065828] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3164.066029] after_reply+0xe2c/0x1780 [ptlrpc] [ 3164.066245] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3164.067134] ? __next_timer_interrupt+0x160/0x160 [ 3164.067145] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3164.067361] ? do_wait_intr+0xf0/0xf0 [ 3164.067370] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3164.067580] kthread+0x1d1/0x200 [ 3164.067586] ? set_kthread_struct+0x70/0x70 [ 3164.067590] ret_from_fork+0x1f/0x30 [ 3164.349441] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3164.392383] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3164.396569] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3164.401179] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3164.403935] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3164.411342] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3164.413536] RAX: 0000000000007d8c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3164.416163] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3164.418828] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3164.422330] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3164.425290] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3164.428025] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3164.431230] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3164.433451] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3164.440205] Call Trace: [ 3164.441863] [ 3164.443008] ? show_regs.cold.9+0x22/0x2f [ 3164.448181] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3164.452282] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3164.454838] ? watchdog+0x40/0x40 [ 3164.456214] ? hrtimer_interrupt+0x128/0x2c0 [ 3164.461463] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3164.465894] ? apic_timer_interrupt+0xf/0x20 [ 3164.468655] [ 3164.469346] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3164.475624] do_raw_spin_lock+0x177/0x190 [ 3164.480761] _raw_spin_lock+0x21/0x30 [ 3164.485632] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3164.488239] ? do_raw_spin_unlock+0x75/0x190 [ 3164.493134] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3164.495943] ? do_wait_intr+0xf0/0xf0 [ 3164.497971] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3164.500706] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3164.506238] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3164.509592] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3164.514901] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3164.517436] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3164.522940] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3164.526994] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3164.530329] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3164.536895] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3164.540593] lmv_intent_open+0x338/0x1150 [lmv] [ 3164.543408] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3164.552201] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3164.556578] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3164.558970] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3164.562055] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3164.565955] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3164.570935] ? do_raw_spin_unlock+0x75/0x190 [ 3164.573001] ? _raw_spin_unlock+0x12/0x30 [ 3164.576001] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3164.579390] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3164.581812] ll_atomic_open+0x36a/0xd70 [lustre] [ 3164.584403] lookup_open+0x356/0xb40 [ 3164.585984] do_last+0x313/0xfc0 [ 3164.587394] ? nd_jump_root+0xe5/0x160 [ 3164.589203] ? path_init+0x437/0x520 [ 3164.591246] path_openat+0xf7/0x500 [ 3164.592761] ? do_raw_spin_unlock+0x75/0x190 [ 3164.594599] do_filp_open+0x99/0x140 [ 3164.600322] ? getname_flags+0x6e/0x330 [ 3164.604336] ? __check_object_size+0xff/0x256 [ 3164.606794] ? do_raw_spin_unlock+0x75/0x190 [ 3164.609034] ? _raw_spin_unlock+0x12/0x30 [ 3164.611023] do_sys_openat2+0x2b4/0x410 [ 3164.612920] do_sys_open+0x73/0xa0 [ 3164.614378] __x64_sys_openat+0x24/0x30 [ 3164.615865] do_syscall_64+0xc1/0x440 [ 3164.617610] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3164.619792] RIP: 0033:0x7f53425a5332 [ 3164.621565] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3164.641431] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3164.645042] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3164.648691] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3164.651874] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3164.658592] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3164.661908] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3180.052138] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 3180.060066] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 3180.068506] Modules linked in: lustre(O) [ 3180.071571] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3180.072011] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3180.072018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3180.072021] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3180.072035] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3180.072039] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3180.072043] RAX: 00000000000041a1 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3180.072045] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3180.072047] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3180.072049] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3180.072051] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3180.072055] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3180.072057] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3180.072060] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3180.072068] Call Trace: [ 3180.072073] [ 3180.072078] ? show_regs.cold.9+0x22/0x2f [ 3180.072088] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3180.072094] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3180.072098] ? watchdog+0x40/0x40 [ 3180.072102] ? hrtimer_interrupt+0x128/0x2c0 [ 3180.072106] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3180.072114] ? apic_timer_interrupt+0xf/0x20 [ 3180.072119] [ 3180.072121] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3180.072126] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3180.072132] do_raw_spin_lock+0x177/0x190 [ 3180.072136] _raw_spin_lock+0x21/0x30 [ 3180.072140] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3180.072417] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3180.072618] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3180.073153] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3180.073369] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3180.073584] ? woken_wake_function+0x30/0x30 [ 3180.073593] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3180.073904] kthread+0x1d1/0x200 [ 3180.073910] ? set_kthread_struct+0x70/0x70 [ 3180.073914] ret_from_fork+0x1f/0x30 [ 3180.366992] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3180.422271] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3180.437189] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3180.447670] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3180.455256] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3180.461898] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3180.490310] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3180.496575] RAX: 0000000000006785 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3180.511214] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3180.523448] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3180.533084] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3180.540042] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3180.544761] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3180.553620] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3180.565061] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3180.567545] Call Trace: [ 3180.574227] [ 3180.574999] ? show_regs.cold.9+0x22/0x2f [ 3180.576166] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3180.578919] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3180.583175] ? watchdog+0x40/0x40 [ 3180.586611] ? hrtimer_interrupt+0x128/0x2c0 [ 3180.591104] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3180.596115] ? apic_timer_interrupt+0xf/0x20 [ 3180.599161] [ 3180.600656] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3180.606274] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3180.612372] ? newidle_balance.isra.93+0x207/0x580 [ 3180.617563] do_raw_spin_lock+0x177/0x190 [ 3180.626309] _raw_spin_lock+0x21/0x30 [ 3180.633531] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3180.639722] process_one_work+0x2c8/0x700 [ 3180.649732] worker_thread+0x296/0x6e0 [ 3180.657614] ? rescuer_thread+0x570/0x570 [ 3180.665570] kthread+0x1d1/0x200 [ 3180.666730] ? set_kthread_struct+0x70/0x70 [ 3180.677658] ret_from_fork+0x1f/0x30 [ 3186.146327] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 549s! [ 3186.149012] Showing busy workqueues and worker pools: [ 3186.151026] workqueue events: flags=0x0 [ 3186.152418] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3186.157028] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3186.164435] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3186.167225] pending: cpu_latency_work [ptlrpc] [ 3186.169397] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3186.174172] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3186.175984] workqueue events_long: flags=0x0 [ 3186.177574] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3186.181822] pending: lu_objects_flush [obdclass] [ 3186.184426] workqueue events_unbound: flags=0x2 [ 3186.188027] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3186.190707] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3186.192771] workqueue events_power_efficient: flags=0x80 [ 3186.194941] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3186.199405] pending: check_lifetime, do_cache_clean [ 3186.201146] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3186.203587] pending: neigh_periodic_work, neigh_periodic_work [ 3186.205787] workqueue mm_percpu_wq: flags=0x8 [ 3186.207459] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3186.210432] pending: vmstat_update, lru_add_drain_per_cpu [ 3186.212255] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3186.215157] pending: vmstat_update, lru_add_drain_per_cpu [ 3186.223051] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3186.225425] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3186.234278] workqueue xprtiod: flags=0xa [ 3186.236566] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3186.245404] pending: xprt_autoclose, xs_error_handle [ 3186.250143] workqueue ipv6_addrconf: flags=0x40008 [ 3186.254844] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3186.261140] pending: addrconf_verify_work [ 3186.263031] workqueue knbd0-recv: flags=0x1a [ 3186.267057] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3186.276352] in-flight: 651:recv_work [ 3186.278678] workqueue ptlrpc_pinger: flags=0x2 [ 3186.284870] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3186.290619] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3186.298152] pool 8: cpus=0-3 flags=0x4 nice=0 hung=549s workers=4 idle: 15979 31824 8 [ 3186.303605] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3176s workers=2 idle: 654 [ 3192.056088] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 3192.057014] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 3192.069705] Modules linked in: lustre(O) osp(O) [ 3192.075374] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3192.077646] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3192.077654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3192.077657] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3192.077671] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3192.077675] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3192.077679] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3192.077681] RDX: 0000000000000e18 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3192.077683] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3192.077685] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3192.077688] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3192.077690] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3192.077693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3192.077695] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3192.078321] Call Trace: [ 3192.078331] [ 3192.078336] ? show_regs.cold.9+0x22/0x2f [ 3192.078346] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3192.078354] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3192.078358] ? watchdog+0x40/0x40 [ 3192.078362] ? hrtimer_interrupt+0x128/0x2c0 [ 3192.078366] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3192.078376] ? apic_timer_interrupt+0xf/0x20 [ 3192.078382] [ 3192.078385] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3192.078390] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3192.078394] do_raw_spin_lock+0x177/0x190 [ 3192.078398] _raw_spin_lock+0x21/0x30 [ 3192.078405] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3192.081789] obd_mod_free+0x46/0x140 [ptlrpc] [ 3192.088334] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3192.089901] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3192.089910] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3192.089913] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3192.089926] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3192.089930] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3192.089935] RAX: 0000000000007b99 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3192.089938] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3192.089940] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3192.089942] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3192.089945] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3192.089948] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3192.089951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3192.089953] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3192.089962] Call Trace: [ 3192.089968] [ 3192.089974] ? show_regs.cold.9+0x22/0x2f [ 3192.089983] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3192.089989] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3192.089993] ? watchdog+0x40/0x40 [ 3192.089997] ? hrtimer_interrupt+0x128/0x2c0 [ 3192.090002] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3192.090012] ? apic_timer_interrupt+0xf/0x20 [ 3192.090017] [ 3192.090020] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3192.090029] do_raw_spin_lock+0x177/0x190 [ 3192.090034] _raw_spin_lock+0x21/0x30 [ 3192.090040] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3192.091700] ? do_raw_spin_unlock+0x75/0x190 [ 3192.091707] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3192.092125] ? do_wait_intr+0xf0/0xf0 [ 3192.092135] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3192.125652] mdc_commit_open+0x91/0xb0 [mdc] [ 3192.131940] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3192.135873] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3192.138959] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3192.151011] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3192.157327] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3192.162454] after_reply+0xe2c/0x1780 [ptlrpc] [ 3192.167717] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3192.167768] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3192.167810] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3192.167842] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3192.172500] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3192.172849] ? __next_timer_interrupt+0x160/0x160 [ 3192.172860] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3192.173360] ? do_wait_intr+0xf0/0xf0 [ 3192.173370] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3192.173569] kthread+0x1d1/0x200 [ 3192.173646] ? set_kthread_struct+0x70/0x70 [ 3192.173652] ret_from_fork+0x1f/0x30 [ 3192.350679] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3192.353137] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3192.355721] lmv_intent_open+0x338/0x1150 [lmv] [ 3192.357678] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3192.359634] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3192.361641] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3192.363631] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3192.365867] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3192.368038] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3192.369858] ? do_raw_spin_unlock+0x75/0x190 [ 3192.371248] ? _raw_spin_unlock+0x12/0x30 [ 3192.372879] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3192.374738] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3192.376782] ll_atomic_open+0x36a/0xd70 [lustre] [ 3192.378323] lookup_open+0x356/0xb40 [ 3192.379535] do_last+0x313/0xfc0 [ 3192.380743] ? nd_jump_root+0xe5/0x160 [ 3192.382059] ? path_init+0x437/0x520 [ 3192.383544] path_openat+0xf7/0x500 [ 3192.385066] ? do_raw_spin_unlock+0x75/0x190 [ 3192.386892] do_filp_open+0x99/0x140 [ 3192.388517] ? getname_flags+0x6e/0x330 [ 3192.390023] ? __check_object_size+0xff/0x256 [ 3192.391712] ? do_raw_spin_unlock+0x75/0x190 [ 3192.393168] ? _raw_spin_unlock+0x12/0x30 [ 3192.395128] do_sys_openat2+0x2b4/0x410 [ 3192.397009] do_sys_open+0x73/0xa0 [ 3192.398088] __x64_sys_openat+0x24/0x30 [ 3192.401078] do_syscall_64+0xc1/0x440 [ 3192.402134] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3192.404780] RIP: 0033:0x7f53425a5332 [ 3192.406553] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3192.414776] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3192.420504] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3192.424170] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3192.427779] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3192.431428] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3192.434196] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3208.058924] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 3208.064109] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3208.095090] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3208.101550] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3208.107133] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3208.111284] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3208.120655] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3208.124777] RAX: 0000000000000428 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3208.132180] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3208.139686] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3208.147107] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3208.155495] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3208.162987] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3208.168191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3208.171763] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3208.179237] Call Trace: [ 3208.180791] [ 3208.182147] ? show_regs.cold.9+0x22/0x2f [ 3208.185224] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3208.187109] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3208.189252] ? watchdog+0x40/0x40 [ 3208.191440] ? hrtimer_interrupt+0x128/0x2c0 [ 3208.199288] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3208.201956] ? apic_timer_interrupt+0xf/0x20 [ 3208.204377] [ 3208.205257] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3208.209956] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3208.212925] do_raw_spin_lock+0x177/0x190 [ 3208.216489] _raw_spin_lock+0x21/0x30 [ 3208.220410] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3208.223673] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3208.226079] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3208.229227] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3208.233770] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3208.237066] ? woken_wake_function+0x30/0x30 [ 3208.241688] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3208.244657] kthread+0x1d1/0x200 [ 3208.246087] ? set_kthread_struct+0x70/0x70 [ 3208.247977] ret_from_fork+0x1f/0x30 [ 3212.052030] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 3212.058358] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3212.113410] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3212.133884] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3212.152403] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3212.162989] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3212.173883] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3212.208918] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3212.223412] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3212.230717] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3212.232422] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3212.240400] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3212.250688] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3212.253175] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3212.286841] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3212.299073] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3212.313052] Call Trace: [ 3212.314930] [ 3212.317146] ? show_regs.cold.9+0x22/0x2f [ 3212.321152] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3212.325836] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3212.338433] ? watchdog+0x40/0x40 [ 3212.339782] ? hrtimer_interrupt+0x128/0x2c0 [ 3212.343978] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3212.351393] ? apic_timer_interrupt+0xf/0x20 [ 3212.352789] [ 3212.356048] ? native_safe_halt+0x2e/0x50 [ 3212.358962] kvm_wait+0xcd/0xe0 [ 3212.365152] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3212.370135] ? newidle_balance.isra.93+0x207/0x580 [ 3212.390299] do_raw_spin_lock+0x177/0x190 [ 3212.396158] _raw_spin_lock+0x21/0x30 [ 3212.402985] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3212.404990] process_one_work+0x2c8/0x700 [ 3212.412890] worker_thread+0x296/0x6e0 [ 3212.415476] ? rescuer_thread+0x570/0x570 [ 3212.423946] kthread+0x1d1/0x200 [ 3212.429563] ? set_kthread_struct+0x70/0x70 [ 3212.435888] ret_from_fork+0x1f/0x30 [ 3216.356511] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 579s! [ 3216.365076] Showing busy workqueues and worker pools: [ 3216.366862] workqueue events: flags=0x0 [ 3216.376617] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3216.385852] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3216.399187] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3216.406474] pending: cpu_latency_work [ptlrpc] [ 3216.412546] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3216.420742] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3216.429269] workqueue events_long: flags=0x0 [ 3216.434309] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3216.437009] pending: lu_objects_flush [obdclass] [ 3216.448353] workqueue events_unbound: flags=0x2 [ 3216.458787] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3216.461374] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3216.463613] workqueue events_power_efficient: flags=0x80 [ 3216.465894] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3216.470697] pending: check_lifetime, do_cache_clean [ 3216.473312] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3216.477109] pending: neigh_periodic_work, neigh_periodic_work [ 3216.480528] workqueue mm_percpu_wq: flags=0x8 [ 3216.482563] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3216.487132] pending: vmstat_update, lru_add_drain_per_cpu [ 3216.489886] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3216.493913] pending: vmstat_update, lru_add_drain_per_cpu [ 3216.497774] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3216.499706] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3216.508899] workqueue xprtiod: flags=0xa [ 3216.510550] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3216.527923] pending: xprt_autoclose, xs_error_handle [ 3216.530666] workqueue ipv6_addrconf: flags=0x40008 [ 3216.532335] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3216.540045] pending: addrconf_verify_work [ 3216.545798] workqueue knbd0-recv: flags=0x1a [ 3216.550059] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3216.554602] in-flight: 651:recv_work [ 3216.559356] workqueue ptlrpc_pinger: flags=0x2 [ 3216.562076] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3216.572827] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3216.585249] pool 8: cpus=0-3 flags=0x4 nice=0 hung=579s workers=4 idle: 15979 31824 8 [ 3216.593109] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3207s workers=2 idle: 654 [ 3220.058172] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 3220.058430] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 3220.064534] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3220.064609] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3220.064618] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3220.064621] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3220.064637] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3220.064642] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3220.064647] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3220.064649] RDX: 0000000000005b73 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3220.064651] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3220.064653] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3220.064655] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3220.064658] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3220.064661] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3220.064663] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3220.064672] Call Trace: [ 3220.064678] [ 3220.064684] ? show_regs.cold.9+0x22/0x2f [ 3220.064693] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3220.064699] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3220.064703] ? watchdog+0x40/0x40 [ 3220.064707] ? hrtimer_interrupt+0x128/0x2c0 [ 3220.064711] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3220.064718] ? apic_timer_interrupt+0xf/0x20 [ 3220.064722] [ 3220.064724] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3220.064729] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3220.064733] do_raw_spin_lock+0x177/0x190 [ 3220.064737] _raw_spin_lock+0x21/0x30 [ 3220.064743] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3220.065022] obd_mod_free+0x46/0x140 [ptlrpc] [ 3220.065232] mdc_commit_open+0x91/0xb0 [mdc] [ 3220.065270] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3220.069218] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3220.069297] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3220.069304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3220.069307] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3220.069322] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3220.069327] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3220.069333] RAX: 0000000000004b0d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3220.069335] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3220.069337] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3220.069339] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3220.069342] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3220.069399] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3220.069402] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3220.069404] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3220.069413] Call Trace: [ 3220.069418] [ 3220.069423] ? show_regs.cold.9+0x22/0x2f [ 3220.069432] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3220.069439] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3220.069443] ? watchdog+0x40/0x40 [ 3220.069447] ? hrtimer_interrupt+0x128/0x2c0 [ 3220.069450] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3220.069457] ? apic_timer_interrupt+0xf/0x20 [ 3220.069461] [ 3220.069463] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3220.069469] do_raw_spin_lock+0x177/0x190 [ 3220.069473] _raw_spin_lock+0x21/0x30 [ 3220.069478] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3220.069746] ? do_raw_spin_unlock+0x75/0x190 [ 3220.069751] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3220.069941] ? do_wait_intr+0xf0/0xf0 [ 3220.069950] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3220.070233] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3220.070966] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3220.071253] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3220.071383] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3220.071423] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3220.071451] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3220.071478] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3220.071560] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3220.071776] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3220.071805] lmv_intent_open+0x338/0x1150 [lmv] [ 3220.071841] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3220.071925] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3220.072004] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3220.072028] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3220.115824] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3220.116829] after_reply+0xe2c/0x1780 [ptlrpc] [ 3220.117056] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3220.117892] ? __next_timer_interrupt+0x160/0x160 [ 3220.117904] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3220.119779] ? do_wait_intr+0xf0/0xf0 [ 3220.119790] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3220.120070] kthread+0x1d1/0x200 [ 3220.120076] ? set_kthread_struct+0x70/0x70 [ 3220.120079] ret_from_fork+0x1f/0x30 [ 3220.551170] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3220.553710] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3220.555856] ? do_raw_spin_unlock+0x75/0x190 [ 3220.557813] ? _raw_spin_unlock+0x12/0x30 [ 3220.559017] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3220.561448] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3220.563859] ll_atomic_open+0x36a/0xd70 [lustre] [ 3220.566704] lookup_open+0x356/0xb40 [ 3220.568219] do_last+0x313/0xfc0 [ 3220.569280] ? nd_jump_root+0xe5/0x160 [ 3220.570351] ? path_init+0x437/0x520 [ 3220.571704] path_openat+0xf7/0x500 [ 3220.573815] ? do_raw_spin_unlock+0x75/0x190 [ 3220.576583] do_filp_open+0x99/0x140 [ 3220.578173] ? getname_flags+0x6e/0x330 [ 3220.579831] ? __check_object_size+0xff/0x256 [ 3220.581838] ? do_raw_spin_unlock+0x75/0x190 [ 3220.583790] ? _raw_spin_unlock+0x12/0x30 [ 3220.585703] do_sys_openat2+0x2b4/0x410 [ 3220.587477] do_sys_open+0x73/0xa0 [ 3220.590740] __x64_sys_openat+0x24/0x30 [ 3220.597632] do_syscall_64+0xc1/0x440 [ 3220.600801] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3220.609475] RIP: 0033:0x7f53425a5332 [ 3220.614336] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3220.630827] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3220.635340] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3220.656784] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3220.664645] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3220.668686] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3220.675291] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3234.454532] rcu: INFO: rcu_sched self-detected stall on CPU [ 3234.464992] rcu: 2-....: (610169 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 3234.470277] (t=1140028 jiffies g=274185 q=63886) [ 3234.476554] rcu: rcu_sched kthread starved for 598394 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 3234.481175] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3234.486348] rcu: RCU grace-period kthread stack dump: [ 3234.489821] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3234.503532] Call Trace: [ 3234.504511] __schedule+0x351/0xcb0 [ 3234.507368] schedule+0xc0/0x180 [ 3234.511528] schedule_timeout+0xb4/0x190 [ 3234.514505] ? do_raw_spin_unlock+0x75/0x190 [ 3234.521454] ? __next_timer_interrupt+0x160/0x160 [ 3234.545230] ? prepare_to_swait_event+0xf3/0x140 [ 3234.546994] rcu_gp_kthread+0x8dd/0x15b0 [ 3234.548365] ? force_qs_rnp+0x2c0/0x2c0 [ 3234.565407] kthread+0x1d1/0x200 [ 3234.566607] ? set_kthread_struct+0x70/0x70 [ 3234.572099] ret_from_fork+0x1f/0x30 [ 3234.579207] rcu: Stack dump where RCU GP kthread last ran: [ 3234.583604] Sending NMI from CPU 2 to CPUs 1: [ 3234.589080] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 3234.591889] Sending NMI from CPU 2 to CPUs 1: [ 3234.602816] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 3234.605729] NMI backtrace for cpu 2 [ 3234.609738] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3234.622182] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3234.627115] Call Trace: [ 3234.628427] [ 3234.629417] ? dump_stack+0xbb/0x10e [ 3234.635976] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3234.638279] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3234.643087] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3234.648776] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3234.651028] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3234.655534] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3234.660141] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3234.664568] ? update_process_times+0x6d/0xc0 [ 3234.666366] ? tick_sched_handle.isra.18+0x27/0x90 [ 3234.671966] ? tick_sched_timer+0x53/0xb0 [ 3234.673662] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3234.678200] ? tick_sched_handle.isra.18+0x90/0x90 [ 3234.680544] ? hrtimer_interrupt+0x128/0x2c0 [ 3234.682638] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3234.684037] ? apic_timer_interrupt+0xf/0x20 [ 3234.685264] [ 3234.687470] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3234.688808] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3234.695324] ? do_raw_spin_lock+0x177/0x190 [ 3234.699128] ? _raw_spin_lock+0x21/0x30 [ 3234.702757] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3234.711189] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3234.716088] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3234.717686] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3234.730340] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3234.733785] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3234.740492] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3234.742561] ? __next_timer_interrupt+0x160/0x160 [ 3234.752660] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3234.754984] ? do_wait_intr+0xf0/0xf0 [ 3234.756945] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3234.762953] ? kthread+0x1d1/0x200 [ 3234.765277] ? set_kthread_struct+0x70/0x70 [ 3234.767155] ? ret_from_fork+0x1f/0x30 [ 3236.058595] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 3236.061302] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3236.098379] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3236.113940] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3236.122191] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3236.126227] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3236.142544] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3236.153057] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3236.159636] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3236.170335] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3236.185072] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3236.191714] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3236.197001] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3236.203521] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3236.207032] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3236.214769] Call Trace: [ 3236.219226] [ 3236.220393] ? show_regs.cold.9+0x22/0x2f [ 3236.225735] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3236.228797] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3236.231909] ? watchdog+0x40/0x40 [ 3236.233731] ? hrtimer_interrupt+0x128/0x2c0 [ 3236.238592] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3236.241801] ? apic_timer_interrupt+0xf/0x20 [ 3236.247719] [ 3236.249769] ? native_safe_halt+0x2e/0x50 [ 3236.255989] kvm_wait+0xcd/0xe0 [ 3236.261427] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3236.267950] do_raw_spin_lock+0x177/0x190 [ 3236.269440] _raw_spin_lock+0x21/0x30 [ 3236.270758] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3236.275942] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3236.283578] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3236.291041] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3236.295717] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3236.303215] ? woken_wake_function+0x30/0x30 [ 3236.306085] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3236.312680] kthread+0x1d1/0x200 [ 3236.318088] ? set_kthread_struct+0x70/0x70 [ 3236.325202] ret_from_fork+0x1f/0x30 [ 3244.052717] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 3244.066050] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3244.103550] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3244.109889] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3244.113243] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3244.115936] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3244.118858] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3244.125899] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3244.129333] RAX: 0000000000007604 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3244.132370] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3244.135465] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3244.138776] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3244.143939] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3244.147516] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3244.150875] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3244.153858] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3244.157329] Call Trace: [ 3244.158184] [ 3244.159153] ? show_regs.cold.9+0x22/0x2f [ 3244.160650] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3244.162850] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3244.164752] ? watchdog+0x40/0x40 [ 3244.166115] ? hrtimer_interrupt+0x128/0x2c0 [ 3244.167396] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3244.169207] ? apic_timer_interrupt+0xf/0x20 [ 3244.170633] [ 3244.171622] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3244.173779] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3244.176427] ? newidle_balance.isra.93+0x207/0x580 [ 3244.178645] do_raw_spin_lock+0x177/0x190 [ 3244.180452] _raw_spin_lock+0x21/0x30 [ 3244.181838] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3244.184589] process_one_work+0x2c8/0x700 [ 3244.186428] worker_thread+0x296/0x6e0 [ 3244.191118] ? rescuer_thread+0x570/0x570 [ 3244.195827] kthread+0x1d1/0x200 [ 3244.196944] ? set_kthread_struct+0x70/0x70 [ 3244.198304] ret_from_fork+0x1f/0x30 [ 3247.073928] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 609s! [ 3247.079344] Showing busy workqueues and worker pools: [ 3247.084190] workqueue events: flags=0x0 [ 3247.086207] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3247.090206] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3247.098798] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3247.109103] pending: cpu_latency_work [ptlrpc] [ 3247.113096] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3247.125952] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3247.132557] workqueue events_long: flags=0x0 [ 3247.134874] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3247.138995] pending: lu_objects_flush [obdclass] [ 3247.142484] workqueue events_unbound: flags=0x2 [ 3247.145421] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3247.148724] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3247.151269] workqueue events_power_efficient: flags=0x80 [ 3247.156108] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3247.160479] pending: check_lifetime, do_cache_clean [ 3247.163763] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3247.168817] pending: neigh_periodic_work, neigh_periodic_work [ 3247.173582] workqueue mm_percpu_wq: flags=0x8 [ 3247.175469] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3247.178974] pending: vmstat_update, lru_add_drain_per_cpu [ 3247.182920] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3247.187004] pending: vmstat_update, lru_add_drain_per_cpu [ 3247.190653] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3247.194811] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3247.198848] workqueue xprtiod: flags=0xa [ 3247.203572] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3247.207810] pending: xprt_autoclose, xs_error_handle [ 3247.216057] workqueue ipv6_addrconf: flags=0x40008 [ 3247.221710] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3247.225284] pending: addrconf_verify_work [ 3247.227872] workqueue knbd0-recv: flags=0x1a [ 3247.230872] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3247.235088] in-flight: 651:recv_work [ 3247.237138] workqueue ptlrpc_pinger: flags=0x2 [ 3247.239882] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3247.243522] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3247.247595] pool 8: cpus=0-3 flags=0x4 nice=0 hung=610s workers=4 idle: 15979 31824 8 [ 3247.251807] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3237s workers=2 idle: 654 [ 3260.055066] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 3260.057496] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 3260.061922] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3260.061991] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3260.061998] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3260.062001] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3260.062013] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3260.062017] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3260.062022] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3260.062024] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3260.062026] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3260.062028] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3260.062031] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3260.062035] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3260.062039] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3260.062041] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3260.062050] Call Trace: [ 3260.062056] [ 3260.062061] ? show_regs.cold.9+0x22/0x2f [ 3260.062072] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3260.062078] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3260.062083] ? watchdog+0x40/0x40 [ 3260.062087] ? hrtimer_interrupt+0x128/0x2c0 [ 3260.062091] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3260.062097] ? apic_timer_interrupt+0xf/0x20 [ 3260.062101] [ 3260.062103] ? native_safe_halt+0x2e/0x50 [ 3260.062107] kvm_wait+0xcd/0xe0 [ 3260.062114] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3260.062122] do_raw_spin_lock+0x177/0x190 [ 3260.062126] _raw_spin_lock+0x21/0x30 [ 3260.062131] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3260.062425] ? do_raw_spin_unlock+0x75/0x190 [ 3260.062431] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3260.062624] ? do_wait_intr+0xf0/0xf0 [ 3260.062633] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3260.062825] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3260.063047] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3260.063275] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3260.063387] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3260.063437] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3260.063468] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3260.063496] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3260.063586] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3260.063811] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3260.063840] lmv_intent_open+0x338/0x1150 [lmv] [ 3260.063867] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3260.063953] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3260.064033] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3260.064055] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3260.064137] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3260.064223] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3260.064305] ? do_raw_spin_unlock+0x75/0x190 [ 3260.064312] ? _raw_spin_unlock+0x12/0x30 [ 3260.064319] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3260.064343] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3260.064364] ll_atomic_open+0x36a/0xd70 [lustre] [ 3260.064462] lookup_open+0x356/0xb40 [ 3260.064470] do_last+0x313/0xfc0 [ 3260.064475] ? nd_jump_root+0xe5/0x160 [ 3260.064479] ? path_init+0x437/0x520 [ 3260.064484] path_openat+0xf7/0x500 [ 3260.064489] ? do_raw_spin_unlock+0x75/0x190 [ 3260.064494] do_filp_open+0x99/0x140 [ 3260.064500] ? getname_flags+0x6e/0x330 [ 3260.064505] ? __check_object_size+0xff/0x256 [ 3260.064511] ? do_raw_spin_unlock+0x75/0x190 [ 3260.064515] ? _raw_spin_unlock+0x12/0x30 [ 3260.064520] do_sys_openat2+0x2b4/0x410 [ 3260.064524] do_sys_open+0x73/0xa0 [ 3260.064528] __x64_sys_openat+0x24/0x30 [ 3260.064531] do_syscall_64+0xc1/0x440 [ 3260.064537] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3260.064543] RIP: 0033:0x7f53425a5332 [ 3260.064551] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3260.064554] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3260.064559] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3260.064561] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3260.064563] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3260.064565] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3260.064567] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3260.312868] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3260.332287] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3260.337782] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3260.343751] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3260.346067] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3260.353057] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3260.358285] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3260.361348] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3260.363794] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3260.365741] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3260.368564] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3260.371800] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3260.376643] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3260.379559] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3260.385352] Call Trace: [ 3260.386560] [ 3260.388256] ? show_regs.cold.9+0x22/0x2f [ 3260.390783] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3260.394175] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3260.396704] ? watchdog+0x40/0x40 [ 3260.398328] ? hrtimer_interrupt+0x128/0x2c0 [ 3260.401085] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3260.404265] ? apic_timer_interrupt+0xf/0x20 [ 3260.405410] [ 3260.406405] ? native_safe_halt+0x2e/0x50 [ 3260.408600] kvm_wait+0xcd/0xe0 [ 3260.411648] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3260.414031] do_raw_spin_lock+0x177/0x190 [ 3260.415082] _raw_spin_lock+0x21/0x30 [ 3260.416623] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3260.418369] obd_mod_free+0x46/0x140 [ptlrpc] [ 3260.419725] mdc_commit_open+0x91/0xb0 [mdc] [ 3260.424150] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3260.426368] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3260.428255] after_reply+0xe2c/0x1780 [ptlrpc] [ 3260.429751] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3260.431586] ? __next_timer_interrupt+0x160/0x160 [ 3260.434243] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3260.437344] ? do_wait_intr+0xf0/0xf0 [ 3260.440109] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3260.443037] kthread+0x1d1/0x200 [ 3260.444635] ? set_kthread_struct+0x70/0x70 [ 3260.446383] ret_from_fork+0x1f/0x30 [ 3264.062376] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 3264.071004] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3264.105441] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3264.114223] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3264.119083] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3264.121193] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3264.135961] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3264.142542] RAX: 0000000000007e91 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3264.145893] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3264.149678] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3264.156082] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3264.159773] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3264.163982] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3264.166845] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3264.169208] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3264.172017] Call Trace: [ 3264.175006] [ 3264.175768] ? show_regs.cold.9+0x22/0x2f [ 3264.178897] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3264.181556] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3264.183167] ? watchdog+0x40/0x40 [ 3264.184991] ? hrtimer_interrupt+0x128/0x2c0 [ 3264.186732] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3264.189013] ? apic_timer_interrupt+0xf/0x20 [ 3264.191364] [ 3264.192980] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3264.200105] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3264.203704] do_raw_spin_lock+0x177/0x190 [ 3264.206621] _raw_spin_lock+0x21/0x30 [ 3264.211135] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3264.218346] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3264.221922] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3264.226011] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3264.228817] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3264.232599] ? woken_wake_function+0x30/0x30 [ 3264.235549] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3264.239688] kthread+0x1d1/0x200 [ 3264.241194] ? set_kthread_struct+0x70/0x70 [ 3264.248883] ret_from_fork+0x1f/0x30 [ 3272.053602] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 3272.061153] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3272.082515] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3272.088854] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3272.094554] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3272.097877] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3272.103054] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3272.110842] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3272.118912] RAX: 0000000000005987 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3272.126963] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3272.128873] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3272.135612] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3272.138411] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3272.141429] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3272.144839] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3272.147615] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3272.152201] Call Trace: [ 3272.153021] [ 3272.155406] ? show_regs.cold.9+0x22/0x2f [ 3272.157371] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3272.158874] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3272.160449] ? watchdog+0x40/0x40 [ 3272.161441] ? hrtimer_interrupt+0x128/0x2c0 [ 3272.162539] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3272.163847] ? apic_timer_interrupt+0xf/0x20 [ 3272.165714] [ 3272.166458] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3272.169986] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3272.174677] ? newidle_balance.isra.93+0x207/0x580 [ 3272.177519] do_raw_spin_lock+0x177/0x190 [ 3272.179503] _raw_spin_lock+0x21/0x30 [ 3272.181290] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3272.184014] process_one_work+0x2c8/0x700 [ 3272.186092] worker_thread+0x296/0x6e0 [ 3272.188937] ? rescuer_thread+0x570/0x570 [ 3272.191324] kthread+0x1d1/0x200 [ 3272.192726] ? set_kthread_struct+0x70/0x70 [ 3272.194833] ret_from_fork+0x1f/0x30 [ 3277.281269] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 640s! [ 3277.283183] Showing busy workqueues and worker pools: [ 3277.296071] workqueue events: flags=0x0 [ 3277.300752] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3277.307397] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3277.318565] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3277.323679] pending: cpu_latency_work [ptlrpc] [ 3277.329019] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3277.335414] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3277.337591] workqueue events_long: flags=0x0 [ 3277.340788] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3277.346200] pending: lu_objects_flush [obdclass] [ 3277.350059] workqueue events_unbound: flags=0x2 [ 3277.354810] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3277.362296] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3277.367020] workqueue events_power_efficient: flags=0x80 [ 3277.373671] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3277.377322] pending: check_lifetime, do_cache_clean [ 3277.380813] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3277.389772] pending: neigh_periodic_work, neigh_periodic_work [ 3277.398151] workqueue mm_percpu_wq: flags=0x8 [ 3277.401473] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3277.412525] pending: vmstat_update, lru_add_drain_per_cpu [ 3277.418545] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3277.422468] pending: vmstat_update, lru_add_drain_per_cpu [ 3277.428866] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3277.439931] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3277.446772] workqueue xprtiod: flags=0xa [ 3277.454554] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3277.464818] pending: xprt_autoclose, xs_error_handle [ 3277.469840] workqueue ipv6_addrconf: flags=0x40008 [ 3277.474788] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3277.479231] pending: addrconf_verify_work [ 3277.480904] workqueue knbd0-recv: flags=0x1a [ 3277.488944] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3277.493740] in-flight: 651:recv_work [ 3277.498329] workqueue ptlrpc_pinger: flags=0x2 [ 3277.501697] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3277.505309] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3277.508992] pool 8: cpus=0-3 flags=0x4 nice=0 hung=640s workers=4 idle: 15979 31824 8 [ 3277.513350] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3267s workers=2 idle: 654 [ 3288.057231] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 3288.057228] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 3288.057245] Modules linked in: lustre(O) [ 3288.059801] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3288.059873] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3288.059880] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3288.059883] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3288.059897] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3288.059901] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3288.059916] RAX: 0000000000001708 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3288.059919] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3288.059921] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3288.059923] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3288.059925] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3288.059927] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3288.059930] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3288.059932] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3288.059939] Call Trace: [ 3288.059945] [ 3288.059950] ? show_regs.cold.9+0x22/0x2f [ 3288.059961] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3288.059967] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3288.059970] ? watchdog+0x40/0x40 [ 3288.059974] ? hrtimer_interrupt+0x128/0x2c0 [ 3288.059978] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3288.059986] ? apic_timer_interrupt+0xf/0x20 [ 3288.059990] [ 3288.059993] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3288.060000] do_raw_spin_lock+0x177/0x190 [ 3288.060005] _raw_spin_lock+0x21/0x30 [ 3288.060011] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3288.067347] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3288.067422] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3288.067429] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3288.067433] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3288.067448] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3288.067453] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3288.067458] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3288.067460] RDX: 0000000000007948 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3288.067463] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3288.067465] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3288.067467] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3288.067471] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3288.067474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3288.067476] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3288.067484] Call Trace: [ 3288.067491] [ 3288.067497] ? show_regs.cold.9+0x22/0x2f [ 3288.067507] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3288.067513] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3288.067518] ? watchdog+0x40/0x40 [ 3288.067522] ? hrtimer_interrupt+0x128/0x2c0 [ 3288.067526] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3288.067535] ? apic_timer_interrupt+0xf/0x20 [ 3288.067539] [ 3288.067543] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3288.067547] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3288.067551] do_raw_spin_lock+0x177/0x190 [ 3288.067556] _raw_spin_lock+0x21/0x30 [ 3288.067561] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3288.128149] ? do_raw_spin_unlock+0x75/0x190 [ 3288.128168] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3288.128429] ? do_wait_intr+0xf0/0xf0 [ 3288.128438] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3288.128628] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3288.128838] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3288.130189] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3288.130296] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3288.137694] obd_mod_free+0x46/0x140 [ptlrpc] [ 3288.138006] mdc_commit_open+0x91/0xb0 [mdc] [ 3288.138046] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3288.138263] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3288.138764] after_reply+0xe2c/0x1780 [ptlrpc] [ 3288.139042] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3288.146566] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3288.146619] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3288.146646] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3288.146770] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3288.155072] ? __next_timer_interrupt+0x160/0x160 [ 3288.155093] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3288.155554] ? do_wait_intr+0xf0/0xf0 [ 3288.155564] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3288.155768] kthread+0x1d1/0x200 [ 3288.155775] ? set_kthread_struct+0x70/0x70 [ 3288.155779] ret_from_fork+0x1f/0x30 [ 3288.657753] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3288.660673] lmv_intent_open+0x338/0x1150 [lmv] [ 3288.662886] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3288.665604] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3288.669610] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3288.672205] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3288.678466] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3288.680351] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3288.684540] ? do_raw_spin_unlock+0x75/0x190 [ 3288.690531] ? _raw_spin_unlock+0x12/0x30 [ 3288.693219] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3288.697643] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3288.699347] ll_atomic_open+0x36a/0xd70 [lustre] [ 3288.703843] lookup_open+0x356/0xb40 [ 3288.707113] do_last+0x313/0xfc0 [ 3288.711382] ? nd_jump_root+0xe5/0x160 [ 3288.715819] ? path_init+0x437/0x520 [ 3288.718557] path_openat+0xf7/0x500 [ 3288.721738] ? do_raw_spin_unlock+0x75/0x190 [ 3288.724711] do_filp_open+0x99/0x140 [ 3288.728430] ? getname_flags+0x6e/0x330 [ 3288.729887] ? __check_object_size+0xff/0x256 [ 3288.732156] ? do_raw_spin_unlock+0x75/0x190 [ 3288.737137] ? _raw_spin_unlock+0x12/0x30 [ 3288.742170] do_sys_openat2+0x2b4/0x410 [ 3288.744767] do_sys_open+0x73/0xa0 [ 3288.749059] __x64_sys_openat+0x24/0x30 [ 3288.750348] do_syscall_64+0xc1/0x440 [ 3288.751619] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3288.753535] RIP: 0033:0x7f53425a5332 [ 3288.754755] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3288.777012] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3288.783267] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3288.792308] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3288.796213] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3288.799196] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3288.806915] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3292.061494] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 3292.067275] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3292.121286] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3292.134385] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3292.150364] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3292.165001] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3292.185144] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3292.194912] RAX: 0000000000006f9c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3292.200373] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3292.208931] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3292.216004] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3292.222898] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3292.231394] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3292.240751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3292.247611] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3292.254104] Call Trace: [ 3292.257107] [ 3292.260063] ? show_regs.cold.9+0x22/0x2f [ 3292.263273] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3292.269620] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3292.271822] ? watchdog+0x40/0x40 [ 3292.278034] ? hrtimer_interrupt+0x128/0x2c0 [ 3292.280015] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3292.286174] ? apic_timer_interrupt+0xf/0x20 [ 3292.291145] [ 3292.293817] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3292.300048] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3292.305640] do_raw_spin_lock+0x177/0x190 [ 3292.309669] _raw_spin_lock+0x21/0x30 [ 3292.311101] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3292.318618] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3292.326041] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3292.333644] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3292.341377] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3292.347944] ? woken_wake_function+0x30/0x30 [ 3292.356494] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3292.364098] kthread+0x1d1/0x200 [ 3292.369679] ? set_kthread_struct+0x70/0x70 [ 3292.375029] ret_from_fork+0x1f/0x30 [ 3304.052163] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 3304.059133] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3304.085685] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3304.091280] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3304.101211] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3304.108857] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3304.114618] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3304.127187] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3304.131515] RAX: 0000000000007f8f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3304.138333] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3304.140252] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3304.148823] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3304.152453] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3304.158881] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3304.168406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3304.173281] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3304.178140] Call Trace: [ 3304.179340] [ 3304.181851] ? show_regs.cold.9+0x22/0x2f [ 3304.184585] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3304.188312] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3304.193250] ? watchdog+0x40/0x40 [ 3304.195092] ? hrtimer_interrupt+0x128/0x2c0 [ 3304.197725] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3304.200502] ? apic_timer_interrupt+0xf/0x20 [ 3304.207280] [ 3304.208048] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3304.209963] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3304.212279] ? newidle_balance.isra.93+0x207/0x580 [ 3304.213964] do_raw_spin_lock+0x177/0x190 [ 3304.214963] _raw_spin_lock+0x21/0x30 [ 3304.216762] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3304.219088] process_one_work+0x2c8/0x700 [ 3304.220859] worker_thread+0x296/0x6e0 [ 3304.222889] ? rescuer_thread+0x570/0x570 [ 3304.224299] kthread+0x1d1/0x200 [ 3304.225587] ? set_kthread_struct+0x70/0x70 [ 3304.227261] ret_from_fork+0x1f/0x30 [ 3308.001148] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 670s! [ 3308.005109] Showing busy workqueues and worker pools: [ 3308.006448] workqueue events: flags=0x0 [ 3308.007403] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3308.012027] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3308.021872] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3308.024240] pending: cpu_latency_work [ptlrpc] [ 3308.029080] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3308.033621] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3308.038449] workqueue events_long: flags=0x0 [ 3308.045636] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3308.050926] pending: lu_objects_flush [obdclass] [ 3308.055209] workqueue events_unbound: flags=0x2 [ 3308.058920] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3308.063702] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3308.067870] workqueue events_power_efficient: flags=0x80 [ 3308.072305] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3308.080447] pending: check_lifetime, do_cache_clean [ 3308.086419] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3308.092544] pending: neigh_periodic_work, neigh_periodic_work [ 3308.100173] workqueue mm_percpu_wq: flags=0x8 [ 3308.104970] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3308.109673] pending: vmstat_update, lru_add_drain_per_cpu [ 3308.117855] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3308.122923] pending: vmstat_update, lru_add_drain_per_cpu [ 3308.126640] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3308.131358] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3308.137863] workqueue xprtiod: flags=0xa [ 3308.140585] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3308.146189] pending: xprt_autoclose, xs_error_handle [ 3308.149944] workqueue ipv6_addrconf: flags=0x40008 [ 3308.153449] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3308.157521] pending: addrconf_verify_work [ 3308.159205] workqueue knbd0-recv: flags=0x1a [ 3308.163060] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3308.168909] in-flight: 651:recv_work [ 3308.173917] workqueue ptlrpc_pinger: flags=0x2 [ 3308.178366] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3308.182895] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3308.188105] pool 8: cpus=0-3 flags=0x4 nice=0 hung=671s workers=4 idle: 15979 31824 8 [ 3308.193114] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3298s workers=2 idle: 654 [ 3316.055824] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 3316.060729] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 3316.061842] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl [ 3316.066677] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3316.066749] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3316.066756] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3316.066759] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3316.066773] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3316.066777] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3316.066781] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3316.066783] RDX: 00000000000053d0 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3316.066786] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3316.066787] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3316.066790] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3316.066793] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3316.066796] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3316.066798] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3316.066806] Call Trace: [ 3316.066812] [ 3316.066818] ? show_regs.cold.9+0x22/0x2f [ 3316.066827] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3316.066833] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3316.066837] ? watchdog+0x40/0x40 [ 3316.066842] ? hrtimer_interrupt+0x128/0x2c0 [ 3316.066846] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3316.066854] ? apic_timer_interrupt+0xf/0x20 [ 3316.066859] [ 3316.066862] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3316.066867] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3316.066872] do_raw_spin_lock+0x177/0x190 [ 3316.066876] _raw_spin_lock+0x21/0x30 [ 3316.066883] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3316.067156] obd_mod_free+0x46/0x140 [ptlrpc] [ 3316.067767] mdc_commit_open+0x91/0xb0 [mdc] [ 3316.067802] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3316.067998] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3316.084791] i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3316.084828] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3316.084833] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3316.084836] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3316.084849] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3316.084853] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3316.084858] RAX: 000000000000482c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3316.084860] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3316.084863] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3316.084865] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3316.084867] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3316.084870] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3316.084873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3316.084876] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3316.084883] Call Trace: [ 3316.084888] [ 3316.084892] ? show_regs.cold.9+0x22/0x2f [ 3316.084901] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3316.084907] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3316.084911] ? watchdog+0x40/0x40 [ 3316.084915] ? hrtimer_interrupt+0x128/0x2c0 [ 3316.084919] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3316.084926] ? apic_timer_interrupt+0xf/0x20 [ 3316.084930] [ 3316.084933] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3316.084941] do_raw_spin_lock+0x177/0x190 [ 3316.084945] _raw_spin_lock+0x21/0x30 [ 3316.084949] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3316.085325] ? do_raw_spin_unlock+0x75/0x190 [ 3316.085331] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3316.085705] ? do_wait_intr+0xf0/0xf0 [ 3316.085713] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3316.121055] after_reply+0xe2c/0x1780 [ptlrpc] [ 3316.121588] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3316.121780] ? __next_timer_interrupt+0x160/0x160 [ 3316.121789] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3316.122221] ? do_wait_intr+0xf0/0xf0 [ 3316.122233] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3316.123687] kthread+0x1d1/0x200 [ 3316.123694] ? set_kthread_struct+0x70/0x70 [ 3316.123698] ret_from_fork+0x1f/0x30 [ 3316.543191] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3316.552398] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3316.557908] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3316.560920] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3316.568712] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3316.577943] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3316.586357] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3316.596832] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3316.604155] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3316.613299] lmv_intent_open+0x338/0x1150 [lmv] [ 3316.622533] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3316.627116] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3316.635958] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3316.641519] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3316.644615] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3316.649224] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3316.653202] ? do_raw_spin_unlock+0x75/0x190 [ 3316.660165] ? _raw_spin_unlock+0x12/0x30 [ 3316.663546] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3316.669893] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3316.678707] ll_atomic_open+0x36a/0xd70 [lustre] [ 3316.685481] lookup_open+0x356/0xb40 [ 3316.686797] do_last+0x313/0xfc0 [ 3316.689722] ? nd_jump_root+0xe5/0x160 [ 3316.692698] ? path_init+0x437/0x520 [ 3316.694512] path_openat+0xf7/0x500 [ 3316.698049] ? do_raw_spin_unlock+0x75/0x190 [ 3316.702209] do_filp_open+0x99/0x140 [ 3316.705219] ? getname_flags+0x6e/0x330 [ 3316.708029] ? __check_object_size+0xff/0x256 [ 3316.711744] ? do_raw_spin_unlock+0x75/0x190 [ 3316.716044] ? _raw_spin_unlock+0x12/0x30 [ 3316.718717] do_sys_openat2+0x2b4/0x410 [ 3316.722914] do_sys_open+0x73/0xa0 [ 3316.729870] __x64_sys_openat+0x24/0x30 [ 3316.735096] do_syscall_64+0xc1/0x440 [ 3316.741991] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3316.743867] RIP: 0033:0x7f53425a5332 [ 3316.749550] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3316.764197] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3316.768764] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3316.772734] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3316.776642] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3316.779316] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3316.782744] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3320.058007] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 3320.060720] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3320.077677] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3320.080847] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3320.084040] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3320.086313] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3320.092717] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3320.095479] RAX: 00000000000051d9 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3320.099843] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3320.102123] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3320.104654] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3320.106879] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3320.109320] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3320.112203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3320.114034] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3320.118522] Call Trace: [ 3320.119486] [ 3320.120201] ? show_regs.cold.9+0x22/0x2f [ 3320.121727] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3320.123601] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3320.125216] ? watchdog+0x40/0x40 [ 3320.126778] ? hrtimer_interrupt+0x128/0x2c0 [ 3320.128648] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3320.130276] ? apic_timer_interrupt+0xf/0x20 [ 3320.131794] [ 3320.133616] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3320.137602] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3320.139420] do_raw_spin_lock+0x177/0x190 [ 3320.140657] _raw_spin_lock+0x21/0x30 [ 3320.144575] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3320.146987] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3320.149005] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3320.151668] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3320.153690] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3320.156257] ? woken_wake_function+0x30/0x30 [ 3320.158112] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3320.160487] kthread+0x1d1/0x200 [ 3320.161769] ? set_kthread_struct+0x70/0x70 [ 3320.163401] ret_from_fork+0x1f/0x30 [ 3336.052045] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 3336.056181] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3336.109425] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3336.128416] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3336.137870] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3336.143824] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3336.147218] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3336.158504] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3336.167619] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3336.172876] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3336.176925] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3336.183871] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3336.188769] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3336.191735] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3336.195665] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3336.204874] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3336.216726] Call Trace: [ 3336.218593] [ 3336.221958] ? show_regs.cold.9+0x22/0x2f [ 3336.226982] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3336.231117] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3336.235578] ? watchdog+0x40/0x40 [ 3336.242408] ? hrtimer_interrupt+0x128/0x2c0 [ 3336.246899] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3336.251957] ? apic_timer_interrupt+0xf/0x20 [ 3336.257618] [ 3336.259113] ? native_safe_halt+0x2e/0x50 [ 3336.264787] kvm_wait+0xcd/0xe0 [ 3336.267908] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3336.273111] ? newidle_balance.isra.93+0x207/0x580 [ 3336.277585] do_raw_spin_lock+0x177/0x190 [ 3336.279809] _raw_spin_lock+0x21/0x30 [ 3336.282647] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3336.289641] process_one_work+0x2c8/0x700 [ 3336.291095] worker_thread+0x296/0x6e0 [ 3336.292286] ? rescuer_thread+0x570/0x570 [ 3336.302726] kthread+0x1d1/0x200 [ 3336.307756] ? set_kthread_struct+0x70/0x70 [ 3336.314841] ret_from_fork+0x1f/0x30 [ 3338.212119] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 701s! [ 3338.228738] Showing busy workqueues and worker pools: [ 3338.235505] workqueue events: flags=0x0 [ 3338.239750] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3338.247200] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3338.258671] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3338.263090] pending: cpu_latency_work [ptlrpc] [ 3338.265938] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3338.269581] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3338.276996] workqueue events_long: flags=0x0 [ 3338.278879] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3338.281538] pending: lu_objects_flush [obdclass] [ 3338.283769] workqueue events_unbound: flags=0x2 [ 3338.287898] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3338.291077] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3338.293471] workqueue events_power_efficient: flags=0x80 [ 3338.295709] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3338.301846] pending: check_lifetime, do_cache_clean [ 3338.305598] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3338.313406] pending: neigh_periodic_work, neigh_periodic_work [ 3338.316918] workqueue mm_percpu_wq: flags=0x8 [ 3338.318832] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3338.323226] pending: vmstat_update, lru_add_drain_per_cpu [ 3338.325657] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3338.328442] pending: vmstat_update, lru_add_drain_per_cpu [ 3338.333139] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3338.336870] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3338.340141] workqueue xprtiod: flags=0xa [ 3338.345137] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3338.348842] pending: xprt_autoclose, xs_error_handle [ 3338.351488] workqueue ipv6_addrconf: flags=0x40008 [ 3338.356956] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3338.364531] pending: addrconf_verify_work [ 3338.367851] workqueue knbd0-recv: flags=0x1a [ 3338.369947] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3338.372600] in-flight: 651:recv_work [ 3338.380916] workqueue ptlrpc_pinger: flags=0x2 [ 3338.389707] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3338.392983] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3338.398933] pool 8: cpus=0-3 flags=0x4 nice=0 hung=701s workers=4 idle: 15979 31824 8 [ 3338.411970] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3328s workers=2 idle: 654 [ 3344.055008] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 3344.061374] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 3344.061391] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3344.061470] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3344.061476] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3344.061479] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3344.061494] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3344.061499] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3344.061504] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3344.061506] RDX: 000000000000088c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3344.061508] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3344.061511] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3344.061513] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3344.061516] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3344.061520] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3344.061523] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3344.061532] Call Trace: [ 3344.061539] [ 3344.061545] ? show_regs.cold.9+0x22/0x2f [ 3344.061555] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3344.061562] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3344.061566] ? watchdog+0x40/0x40 [ 3344.061570] ? hrtimer_interrupt+0x128/0x2c0 [ 3344.061573] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3344.061580] ? apic_timer_interrupt+0xf/0x20 [ 3344.061585] [ 3344.061588] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3344.061593] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3344.061599] do_raw_spin_lock+0x177/0x190 [ 3344.061605] _raw_spin_lock+0x21/0x30 [ 3344.061611] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3344.066243] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3344.066327] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3344.066336] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3344.066338] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3344.066352] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3344.066357] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3344.066361] RAX: 0000000000007040 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3344.066364] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3344.066366] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3344.066368] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3344.066370] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3344.066374] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3344.066376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3344.066379] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3344.066387] Call Trace: [ 3344.066393] [ 3344.066399] ? show_regs.cold.9+0x22/0x2f [ 3344.066409] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3344.066416] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3344.066420] ? watchdog+0x40/0x40 [ 3344.066423] ? hrtimer_interrupt+0x128/0x2c0 [ 3344.066427] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3344.066436] ? apic_timer_interrupt+0xf/0x20 [ 3344.066441] [ 3344.066444] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3344.066450] do_raw_spin_lock+0x177/0x190 [ 3344.066456] _raw_spin_lock+0x21/0x30 [ 3344.066461] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3344.067448] ? do_raw_spin_unlock+0x75/0x190 [ 3344.067455] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3344.069451] ? do_wait_intr+0xf0/0xf0 [ 3344.069463] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3344.069781] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3344.079151] obd_mod_free+0x46/0x140 [ptlrpc] [ 3344.079466] mdc_commit_open+0x91/0xb0 [mdc] [ 3344.079502] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3344.079832] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3344.080091] after_reply+0xe2c/0x1780 [ptlrpc] [ 3344.080363] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3344.080636] ? __next_timer_interrupt+0x160/0x160 [ 3344.080646] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3344.080853] ? do_wait_intr+0xf0/0xf0 [ 3344.080861] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3344.081131] kthread+0x1d1/0x200 [ 3344.081137] ? set_kthread_struct+0x70/0x70 [ 3344.081142] ret_from_fork+0x1f/0x30 [ 3344.303827] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3344.306269] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3344.308595] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3344.311398] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3344.313537] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3344.315764] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3344.318542] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3344.321400] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3344.324057] lmv_intent_open+0x338/0x1150 [lmv] [ 3344.326181] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3344.328226] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3344.330183] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3344.331458] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3344.333593] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3344.336270] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3344.338437] ? do_raw_spin_unlock+0x75/0x190 [ 3344.340122] ? _raw_spin_unlock+0x12/0x30 [ 3344.342076] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3344.344101] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3344.346371] ll_atomic_open+0x36a/0xd70 [lustre] [ 3344.348622] lookup_open+0x356/0xb40 [ 3344.350279] do_last+0x313/0xfc0 [ 3344.351687] ? nd_jump_root+0xe5/0x160 [ 3344.353246] ? path_init+0x437/0x520 [ 3344.354727] path_openat+0xf7/0x500 [ 3344.356269] ? do_raw_spin_unlock+0x75/0x190 [ 3344.357776] do_filp_open+0x99/0x140 [ 3344.359550] ? getname_flags+0x6e/0x330 [ 3344.362145] ? __check_object_size+0xff/0x256 [ 3344.364276] ? do_raw_spin_unlock+0x75/0x190 [ 3344.366207] ? _raw_spin_unlock+0x12/0x30 [ 3344.367584] do_sys_openat2+0x2b4/0x410 [ 3344.369281] do_sys_open+0x73/0xa0 [ 3344.370757] __x64_sys_openat+0x24/0x30 [ 3344.372372] do_syscall_64+0xc1/0x440 [ 3344.373992] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3344.376154] RIP: 0033:0x7f53425a5332 [ 3344.377499] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3344.386605] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3344.390106] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3344.393484] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3344.396979] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3344.400439] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3344.402602] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3348.059436] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 3348.064448] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3348.111793] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3348.120183] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3348.126829] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3348.134791] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3348.158725] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3348.170458] RAX: 0000000000004df8 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3348.174002] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3348.177838] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3348.181572] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3348.185766] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3348.191580] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3348.200649] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3348.204697] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3348.209498] Call Trace: [ 3348.210541] [ 3348.211420] ? show_regs.cold.9+0x22/0x2f [ 3348.212965] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3348.216008] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3348.218203] ? watchdog+0x40/0x40 [ 3348.219815] ? hrtimer_interrupt+0x128/0x2c0 [ 3348.221667] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3348.223577] ? apic_timer_interrupt+0xf/0x20 [ 3348.225529] [ 3348.226452] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3348.229296] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3348.231982] do_raw_spin_lock+0x177/0x190 [ 3348.234668] _raw_spin_lock+0x21/0x30 [ 3348.235927] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3348.239445] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3348.249407] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3348.258002] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3348.266214] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3348.273452] ? woken_wake_function+0x30/0x30 [ 3348.275996] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3348.279938] kthread+0x1d1/0x200 [ 3348.282827] ? set_kthread_struct+0x70/0x70 [ 3348.285724] ret_from_fork+0x1f/0x30 [ 3364.052012] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 3364.064763] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3364.090860] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3364.095446] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3364.098356] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3364.101021] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3364.107229] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3364.121249] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3364.124186] RAX: 0000000000002f10 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3364.126492] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3364.131510] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3364.138836] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3364.144166] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3364.148366] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3364.152456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3364.155310] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3364.157935] Call Trace: [ 3364.158923] [ 3364.159781] ? show_regs.cold.9+0x22/0x2f [ 3364.165612] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3364.167246] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3364.171069] ? watchdog+0x40/0x40 [ 3364.173941] ? hrtimer_interrupt+0x128/0x2c0 [ 3364.177032] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3364.179853] ? apic_timer_interrupt+0xf/0x20 [ 3364.183340] [ 3364.183972] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3364.186487] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3364.190116] ? newidle_balance.isra.93+0x207/0x580 [ 3364.194544] do_raw_spin_lock+0x177/0x190 [ 3364.197599] _raw_spin_lock+0x21/0x30 [ 3364.198985] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3364.200979] process_one_work+0x2c8/0x700 [ 3364.203781] worker_thread+0x296/0x6e0 [ 3364.206087] ? rescuer_thread+0x570/0x570 [ 3364.208067] kthread+0x1d1/0x200 [ 3364.211144] ? set_kthread_struct+0x70/0x70 [ 3364.214715] ret_from_fork+0x1f/0x30 [ 3368.931337] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 731s! [ 3368.940870] Showing busy workqueues and worker pools: [ 3368.946756] workqueue events: flags=0x0 [ 3368.952757] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3368.968304] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3368.988812] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3368.997819] pending: cpu_latency_work [ptlrpc] [ 3369.011029] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3369.015267] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3369.019161] workqueue events_long: flags=0x0 [ 3369.021248] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3369.026045] pending: lu_objects_flush [obdclass] [ 3369.029073] workqueue events_unbound: flags=0x2 [ 3369.031206] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3369.034676] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3369.036515] workqueue events_power_efficient: flags=0x80 [ 3369.038968] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3369.049101] pending: check_lifetime, do_cache_clean [ 3369.052073] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3369.063844] pending: neigh_periodic_work, neigh_periodic_work [ 3369.071866] workqueue mm_percpu_wq: flags=0x8 [ 3369.078434] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3369.089552] pending: vmstat_update, lru_add_drain_per_cpu [ 3369.093720] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3369.103291] pending: vmstat_update, lru_add_drain_per_cpu [ 3369.107427] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3369.114619] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3369.121349] workqueue xprtiod: flags=0xa [ 3369.126269] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3369.137274] pending: xprt_autoclose, xs_error_handle [ 3369.144774] workqueue ipv6_addrconf: flags=0x40008 [ 3369.149594] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3369.157763] pending: addrconf_verify_work [ 3369.165723] workqueue knbd0-recv: flags=0x1a [ 3369.172641] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3369.183637] in-flight: 651:recv_work [ 3369.187862] workqueue ptlrpc_pinger: flags=0x2 [ 3369.192415] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3369.204785] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3369.215125] pool 8: cpus=0-3 flags=0x4 nice=0 hung=732s workers=4 idle: 15979 31824 8 [ 3369.222560] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3359s workers=2 idle: 654 [ 3372.056230] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 3372.063095] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 3372.068907] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3372.068990] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3372.068997] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3372.069000] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3372.069014] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3372.069018] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3372.069023] RAX: 0000000000006183 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3372.069025] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3372.069028] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3372.069030] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3372.069032] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3372.069035] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3372.069037] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3372.069039] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3372.069048] Call Trace: [ 3372.069053] [ 3372.069059] ? show_regs.cold.9+0x22/0x2f [ 3372.069069] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3372.069074] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3372.069079] ? watchdog+0x40/0x40 [ 3372.069083] ? hrtimer_interrupt+0x128/0x2c0 [ 3372.069087] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3372.069094] ? apic_timer_interrupt+0xf/0x20 [ 3372.069099] [ 3372.069102] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3372.069108] do_raw_spin_lock+0x177/0x190 [ 3372.069113] _raw_spin_lock+0x21/0x30 [ 3372.069117] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3372.069491] ? do_raw_spin_unlock+0x75/0x190 [ 3372.069502] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3372.069705] ? do_wait_intr+0xf0/0xf0 [ 3372.069714] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3372.069919] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3372.070527] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3372.072055] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3372.072171] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3372.072211] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3372.072243] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3372.072272] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3372.072356] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3372.072637] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3372.072663] lmv_intent_open+0x338/0x1150 [lmv] [ 3372.072697] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3372.072782] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3372.072874] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3372.072898] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3372.081169] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3372.081245] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3372.081252] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3372.081255] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3372.081268] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3372.081272] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3372.081277] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3372.081279] RDX: 0000000000002939 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3372.081281] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3372.081283] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3372.081286] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3372.081288] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3372.081291] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3372.081294] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3372.081301] Call Trace: [ 3372.081307] [ 3372.081313] ? show_regs.cold.9+0x22/0x2f [ 3372.081323] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3372.081329] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3372.081333] ? watchdog+0x40/0x40 [ 3372.081337] ? hrtimer_interrupt+0x128/0x2c0 [ 3372.081341] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3372.081349] ? apic_timer_interrupt+0xf/0x20 [ 3372.081353] [ 3372.081356] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3372.081360] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3372.081364] do_raw_spin_lock+0x177/0x190 [ 3372.081368] _raw_spin_lock+0x21/0x30 [ 3372.081374] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3372.082106] obd_mod_free+0x46/0x140 [ptlrpc] [ 3372.082319] mdc_commit_open+0x91/0xb0 [mdc] [ 3372.082359] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3372.082758] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3372.111609] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3372.111764] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3372.111850] ? do_raw_spin_unlock+0x75/0x190 [ 3372.112055] ? _raw_spin_unlock+0x12/0x30 [ 3372.112064] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3372.112095] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3372.112118] ll_atomic_open+0x36a/0xd70 [lustre] [ 3372.112207] lookup_open+0x356/0xb40 [ 3372.112216] do_last+0x313/0xfc0 [ 3372.112221] ? nd_jump_root+0xe5/0x160 [ 3372.112225] ? path_init+0x437/0x520 [ 3372.112230] path_openat+0xf7/0x500 [ 3372.112235] ? do_raw_spin_unlock+0x75/0x190 [ 3372.112240] do_filp_open+0x99/0x140 [ 3372.112246] ? getname_flags+0x6e/0x330 [ 3372.112250] ? __check_object_size+0xff/0x256 [ 3372.112257] ? do_raw_spin_unlock+0x75/0x190 [ 3372.112261] ? _raw_spin_unlock+0x12/0x30 [ 3372.112265] do_sys_openat2+0x2b4/0x410 [ 3372.112269] do_sys_open+0x73/0xa0 [ 3372.112273] __x64_sys_openat+0x24/0x30 [ 3372.112277] do_syscall_64+0xc1/0x440 [ 3372.112285] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3372.112292] RIP: 0033:0x7f53425a5332 [ 3372.112301] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3372.112306] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3372.112311] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3372.112313] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3372.112316] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3372.112318] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3372.112321] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3372.522201] after_reply+0xe2c/0x1780 [ptlrpc] [ 3372.524407] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3372.526643] ? __next_timer_interrupt+0x160/0x160 [ 3372.528658] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3372.530207] ? do_wait_intr+0xf0/0xf0 [ 3372.531013] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3372.532931] kthread+0x1d1/0x200 [ 3372.534357] ? set_kthread_struct+0x70/0x70 [ 3372.536208] ret_from_fork+0x1f/0x30 [ 3376.058563] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 3376.070339] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3376.124718] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3376.132928] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3376.149540] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3376.159628] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3376.181812] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3376.187343] RAX: 0000000000007c92 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3376.199634] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3376.213697] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3376.230887] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3376.245263] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3376.255013] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3376.272887] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3376.285231] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3376.290485] Call Trace: [ 3376.291241] [ 3376.291915] ? show_regs.cold.9+0x22/0x2f [ 3376.299938] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3376.314277] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3376.320409] ? watchdog+0x40/0x40 [ 3376.324585] ? hrtimer_interrupt+0x128/0x2c0 [ 3376.330146] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3376.339715] ? apic_timer_interrupt+0xf/0x20 [ 3376.350589] [ 3376.353948] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3376.360736] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3376.364034] do_raw_spin_lock+0x177/0x190 [ 3376.367598] _raw_spin_lock+0x21/0x30 [ 3376.371014] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3376.381294] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3376.390319] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3376.407627] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3376.423130] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3376.437027] ? woken_wake_function+0x30/0x30 [ 3376.441586] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3376.469687] kthread+0x1d1/0x200 [ 3376.478583] ? set_kthread_struct+0x70/0x70 [ 3376.485098] ret_from_fork+0x1f/0x30 [ 3396.052049] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 3396.057246] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3396.083347] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3396.091081] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3396.095564] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3396.099866] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3396.111485] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3396.147786] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3396.164472] RAX: 0000000000007f96 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3396.174783] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3396.176803] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3396.178831] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3396.180377] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3396.182229] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3396.184275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3396.185764] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3396.187498] Call Trace: [ 3396.188635] [ 3396.189488] ? show_regs.cold.9+0x22/0x2f [ 3396.190875] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3396.192213] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3396.193363] ? watchdog+0x40/0x40 [ 3396.194184] ? hrtimer_interrupt+0x128/0x2c0 [ 3396.195469] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3396.196897] ? apic_timer_interrupt+0xf/0x20 [ 3396.198343] [ 3396.199411] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3396.200602] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3396.202287] ? newidle_balance.isra.93+0x207/0x580 [ 3396.203607] do_raw_spin_lock+0x177/0x190 [ 3396.204704] _raw_spin_lock+0x21/0x30 [ 3396.205658] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3396.207620] process_one_work+0x2c8/0x700 [ 3396.208957] worker_thread+0x296/0x6e0 [ 3396.210305] ? rescuer_thread+0x570/0x570 [ 3396.211593] kthread+0x1d1/0x200 [ 3396.212810] ? set_kthread_struct+0x70/0x70 [ 3396.214286] ret_from_fork+0x1f/0x30 [ 3399.650093] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 762s! [ 3399.655394] Showing busy workqueues and worker pools: [ 3399.663792] workqueue events: flags=0x0 [ 3399.667382] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3399.677141] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3399.694346] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3399.701362] pending: cpu_latency_work [ptlrpc] [ 3399.710453] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3399.715422] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3399.719034] workqueue events_long: flags=0x0 [ 3399.721023] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3399.724861] pending: lu_objects_flush [obdclass] [ 3399.727506] workqueue events_unbound: flags=0x2 [ 3399.730138] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3399.735728] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3399.739209] workqueue events_power_efficient: flags=0x80 [ 3399.744602] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3399.749459] pending: check_lifetime, do_cache_clean [ 3399.751212] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3399.754754] pending: neigh_periodic_work, neigh_periodic_work [ 3399.760607] workqueue mm_percpu_wq: flags=0x8 [ 3399.762034] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3399.768765] pending: vmstat_update, lru_add_drain_per_cpu [ 3399.770825] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3399.788772] pending: vmstat_update, lru_add_drain_per_cpu [ 3399.798775] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3399.810574] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3399.821522] workqueue xprtiod: flags=0xa [ 3399.826879] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3399.838129] pending: xprt_autoclose, xs_error_handle [ 3399.853819] workqueue ipv6_addrconf: flags=0x40008 [ 3399.855357] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3399.859394] pending: addrconf_verify_work [ 3399.862065] workqueue knbd0-recv: flags=0x1a [ 3399.864376] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3399.870720] in-flight: 651:recv_work [ 3399.875123] workqueue ptlrpc_pinger: flags=0x2 [ 3399.877705] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3399.885650] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3399.895578] pool 8: cpus=0-3 flags=0x4 nice=0 hung=762s workers=4 idle: 15979 31824 8 [ 3399.912927] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3390s workers=2 idle: 654 [ 3400.057022] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 3400.058015] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 3400.061306] Modules linked in: [ 3400.064197] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3400.064273] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3400.064281] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3400.064284] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3400.064299] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3400.064303] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3400.064307] RAX: 0000000000007e0a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3400.064309] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3400.064312] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3400.064313] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3400.064315] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3400.064318] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3400.064321] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3400.064322] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3400.064330] Call Trace: [ 3400.064336] [ 3400.064342] ? show_regs.cold.9+0x22/0x2f [ 3400.064351] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3400.064356] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3400.064360] ? watchdog+0x40/0x40 [ 3400.064364] ? hrtimer_interrupt+0x128/0x2c0 [ 3400.064368] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3400.064376] ? apic_timer_interrupt+0xf/0x20 [ 3400.064380] [ 3400.064383] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3400.064389] do_raw_spin_lock+0x177/0x190 [ 3400.064394] _raw_spin_lock+0x21/0x30 [ 3400.064399] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3400.096367] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3400.096448] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3400.101449] ? do_raw_spin_unlock+0x75/0x190 [ 3400.106799] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3400.106811] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3400.106827] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3400.106831] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3400.106837] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3400.106840] RDX: 0000000000007e01 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3400.106842] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3400.106845] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3400.106847] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3400.106851] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3400.106854] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3400.106856] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3400.106865] Call Trace: [ 3400.106871] [ 3400.106877] ? show_regs.cold.9+0x22/0x2f [ 3400.106887] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3400.106894] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3400.106899] ? watchdog+0x40/0x40 [ 3400.106903] ? hrtimer_interrupt+0x128/0x2c0 [ 3400.106907] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3400.106916] ? apic_timer_interrupt+0xf/0x20 [ 3400.106921] [ 3400.106924] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3400.106928] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3400.106933] do_raw_spin_lock+0x177/0x190 [ 3400.106937] _raw_spin_lock+0x21/0x30 [ 3400.107326] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3400.107643] obd_mod_free+0x46/0x140 [ptlrpc] [ 3400.107846] mdc_commit_open+0x91/0xb0 [mdc] [ 3400.107892] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3400.108477] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3400.108681] after_reply+0xe2c/0x1780 [ptlrpc] [ 3400.108883] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3400.109560] ? __next_timer_interrupt+0x160/0x160 [ 3400.109571] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3400.109766] ? do_wait_intr+0xf0/0xf0 [ 3400.109776] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3400.110041] kthread+0x1d1/0x200 [ 3400.110047] ? set_kthread_struct+0x70/0x70 [ 3400.110052] ret_from_fork+0x1f/0x30 [ 3400.303046] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3400.305684] ? do_wait_intr+0xf0/0xf0 [ 3400.307420] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3400.310303] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3400.312374] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3400.315150] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3400.319037] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3400.322921] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3400.327153] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3400.329729] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3400.333009] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3400.336460] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3400.339409] lmv_intent_open+0x338/0x1150 [lmv] [ 3400.341636] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3400.344377] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3400.347237] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3400.349336] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3400.351485] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3400.353845] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3400.355473] ? do_raw_spin_unlock+0x75/0x190 [ 3400.358382] ? _raw_spin_unlock+0x12/0x30 [ 3400.359687] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3400.361736] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3400.363643] ll_atomic_open+0x36a/0xd70 [lustre] [ 3400.365486] lookup_open+0x356/0xb40 [ 3400.367089] do_last+0x313/0xfc0 [ 3400.368566] ? nd_jump_root+0xe5/0x160 [ 3400.370113] ? path_init+0x437/0x520 [ 3400.371489] path_openat+0xf7/0x500 [ 3400.373199] ? do_raw_spin_unlock+0x75/0x190 [ 3400.374732] do_filp_open+0x99/0x140 [ 3400.376193] ? getname_flags+0x6e/0x330 [ 3400.377493] ? __check_object_size+0xff/0x256 [ 3400.379135] ? do_raw_spin_unlock+0x75/0x190 [ 3400.380619] ? _raw_spin_unlock+0x12/0x30 [ 3400.382209] do_sys_openat2+0x2b4/0x410 [ 3400.384072] do_sys_open+0x73/0xa0 [ 3400.385341] __x64_sys_openat+0x24/0x30 [ 3400.387182] do_syscall_64+0xc1/0x440 [ 3400.388582] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3400.391080] RIP: 0033:0x7f53425a5332 [ 3400.392328] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3400.400004] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3400.403047] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3400.405728] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3400.408300] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3400.410398] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3400.412266] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3404.058052] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 3404.061842] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3404.138713] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3404.154586] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3404.163345] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3404.173889] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3404.190940] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3404.197070] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3404.205151] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3404.215340] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3404.221283] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3404.230016] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3404.236681] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3404.249301] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3404.255239] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3404.263162] Call Trace: [ 3404.265343] [ 3404.269551] ? show_regs.cold.9+0x22/0x2f [ 3404.271018] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3404.277989] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3404.287054] ? watchdog+0x40/0x40 [ 3404.289312] ? hrtimer_interrupt+0x128/0x2c0 [ 3404.295550] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3404.299130] ? apic_timer_interrupt+0xf/0x20 [ 3404.304407] [ 3404.306021] ? native_safe_halt+0x2e/0x50 [ 3404.308435] kvm_wait+0xcd/0xe0 [ 3404.313207] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3404.317158] do_raw_spin_lock+0x177/0x190 [ 3404.319857] _raw_spin_lock+0x21/0x30 [ 3404.321191] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3404.325195] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3404.329000] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3404.332158] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3404.335062] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3404.338144] ? woken_wake_function+0x30/0x30 [ 3404.340076] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3404.343284] kthread+0x1d1/0x200 [ 3404.345107] ? set_kthread_struct+0x70/0x70 [ 3404.347125] ret_from_fork+0x1f/0x30 [ 3414.457095] rcu: INFO: rcu_sched self-detected stall on CPU [ 3414.459865] rcu: 2-....: (706115 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 3414.470139] (t=1320031 jiffies g=274185 q=63952) [ 3414.472533] rcu: rcu_sched kthread starved for 778397 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 3414.479524] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3414.485102] rcu: RCU grace-period kthread stack dump: [ 3414.487179] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3414.493313] Call Trace: [ 3414.494723] __schedule+0x351/0xcb0 [ 3414.496995] schedule+0xc0/0x180 [ 3414.499066] schedule_timeout+0xb4/0x190 [ 3414.500844] ? do_raw_spin_unlock+0x75/0x190 [ 3414.504013] ? __next_timer_interrupt+0x160/0x160 [ 3414.506317] ? prepare_to_swait_event+0xf3/0x140 [ 3414.511192] rcu_gp_kthread+0x8dd/0x15b0 [ 3414.514290] ? force_qs_rnp+0x2c0/0x2c0 [ 3414.518968] kthread+0x1d1/0x200 [ 3414.520544] ? set_kthread_struct+0x70/0x70 [ 3414.523463] ret_from_fork+0x1f/0x30 [ 3414.525215] rcu: Stack dump where RCU GP kthread last ran: [ 3414.529811] Sending NMI from CPU 2 to CPUs 3: [ 3414.532247] NMI backtrace for cpu 3 [ 3414.532253] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3414.532255] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3414.532256] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30a/0x6d0 [ 3414.532258] Code: 03 00 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 [ 3414.532259] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 [ 3414.532262] RAX: 0000000000007c2c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3414.532263] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3414.532265] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3414.532266] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3414.532267] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3414.532268] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3414.532269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3414.532270] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3414.532272] Call Trace: [ 3414.532273] [ 3414.532274] ? show_regs.cold.9+0x22/0x2f [ 3414.532275] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3414.532276] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3414.532277] ? nmi_handle+0xbd/0x210 [ 3414.532278] ? default_do_nmi+0x4b/0x1b0 [ 3414.532279] ? do_nmi+0x280/0x380 [ 3414.532279] ? end_repeat_nmi+0x16/0x1f [ 3414.532280] ? __pv_queued_spin_lock_slowpath+0x30a/0x6d0 [ 3414.532281] ? __pv_queued_spin_lock_slowpath+0x30a/0x6d0 [ 3414.532282] ? __pv_queued_spin_lock_slowpath+0x30a/0x6d0 [ 3414.532283] [ 3414.532284] do_raw_spin_lock+0x177/0x190 [ 3414.532285] _raw_spin_lock+0x21/0x30 [ 3414.532286] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3414.532287] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3414.532288] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3414.532289] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3414.532290] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3414.532291] ? woken_wake_function+0x30/0x30 [ 3414.532292] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3414.532293] kthread+0x1d1/0x200 [ 3414.532294] ? set_kthread_struct+0x70/0x70 [ 3414.532295] ret_from_fork+0x1f/0x30 [ 3414.532506] Sending NMI from CPU 2 to CPUs 1: [ 3414.630371] NMI backtrace for cpu 1 [ 3414.630377] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3414.630379] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3414.630381] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3414.630382] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3414.630384] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [ 3414.630387] RAX: 0000000000006507 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3414.630389] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3414.630390] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3414.630392] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3414.630393] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3414.630394] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3414.630396] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3414.630397] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3414.630398] Call Trace: [ 3414.630399] [ 3414.630404] ? show_regs.cold.9+0x22/0x2f [ 3414.630405] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3414.630407] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3414.630408] ? nmi_handle+0xbd/0x210 [ 3414.630409] ? default_do_nmi+0x4b/0x1b0 [ 3414.630410] ? do_nmi+0x280/0x380 [ 3414.630411] ? end_repeat_nmi+0x16/0x1f [ 3414.630412] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3414.630414] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3414.630415] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3414.630416] [ 3414.630417] do_raw_spin_lock+0x177/0x190 [ 3414.630418] _raw_spin_lock+0x21/0x30 [ 3414.630419] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3414.630420] ? do_raw_spin_unlock+0x75/0x190 [ 3414.630421] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3414.630423] ? do_wait_intr+0xf0/0xf0 [ 3414.630424] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3414.630425] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3414.630426] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3414.630427] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3414.630428] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3414.630430] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3414.630431] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3414.630432] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3414.630434] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3414.630435] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3414.630436] lmv_intent_open+0x338/0x1150 [lmv] [ 3414.630437] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3414.630438] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3414.630439] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3414.630440] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3414.630441] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3414.630443] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3414.630443] ? do_raw_spin_unlock+0x75/0x190 [ 3414.630444] ? _raw_spin_unlock+0x12/0x30 [ 3414.630446] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3414.630447] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3414.630448] ll_atomic_open+0x36a/0xd70 [lustre] [ 3414.630453] lookup_open+0x356/0xb40 [ 3414.630454] do_last+0x313/0xfc0 [ 3414.630455] ? nd_jump_root+0xe5/0x160 [ 3414.630456] ? path_init+0x437/0x520 [ 3414.630457] path_openat+0xf7/0x500 [ 3414.630458] ? do_raw_spin_unlock+0x75/0x190 [ 3414.630459] do_filp_open+0x99/0x140 [ 3414.630460] ? getname_flags+0x6e/0x330 [ 3414.630461] ? __check_object_size+0xff/0x256 [ 3414.630462] ? do_raw_spin_unlock+0x75/0x190 [ 3414.630463] ? _raw_spin_unlock+0x12/0x30 [ 3414.630464] do_sys_openat2+0x2b4/0x410 [ 3414.630465] do_sys_open+0x73/0xa0 [ 3414.630466] __x64_sys_openat+0x24/0x30 [ 3414.630467] do_syscall_64+0xc1/0x440 [ 3414.630468] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3414.630469] RIP: 0033:0x7f53425a5332 [ 3414.630470] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3414.630472] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3414.630475] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3414.630476] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3414.630477] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3414.630482] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3414.630483] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3414.631477] NMI backtrace for cpu 2 [ 3414.805410] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3414.810008] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3414.813642] Call Trace: [ 3414.814661] [ 3414.815643] ? dump_stack+0xbb/0x10e [ 3414.817057] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3414.819660] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3414.821142] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3414.823030] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3414.825264] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3414.827451] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3414.829559] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3414.831354] ? update_process_times+0x6d/0xc0 [ 3414.832935] ? tick_sched_handle.isra.18+0x27/0x90 [ 3414.834624] ? tick_sched_timer+0x53/0xb0 [ 3414.835943] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3414.837173] ? tick_sched_handle.isra.18+0x90/0x90 [ 3414.838386] ? hrtimer_interrupt+0x128/0x2c0 [ 3414.840312] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3414.842065] ? apic_timer_interrupt+0xf/0x20 [ 3414.843598] [ 3414.844435] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3414.846767] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3414.848904] ? do_raw_spin_lock+0x177/0x190 [ 3414.850817] ? _raw_spin_lock+0x21/0x30 [ 3414.852278] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3414.854833] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3414.856891] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3414.858919] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3414.861291] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3414.863816] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3414.865919] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3414.868045] ? __next_timer_interrupt+0x160/0x160 [ 3414.870240] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3414.872159] ? do_wait_intr+0xf0/0xf0 [ 3414.873621] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3414.876148] ? kthread+0x1d1/0x200 [ 3414.877519] ? set_kthread_struct+0x70/0x70 [ 3414.879492] ? ret_from_fork+0x1f/0x30 [ 3424.052009] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 3424.059209] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3424.085044] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3424.091339] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3424.098886] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3424.104359] RIP: 0010:__pv_queued_spin_lock_slowpath+0x318/0x6d0 [ 3424.111021] Code: b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 <85> d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 eb e0 48 83 05 06 [ 3424.125277] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3424.131409] RAX: 000000000000710b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3424.140103] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3424.144373] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3424.151208] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3424.158747] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3424.165274] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3424.176340] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3424.183982] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3424.192313] Call Trace: [ 3424.194202] [ 3424.195032] ? show_regs.cold.9+0x22/0x2f [ 3424.199313] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3424.207769] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3424.209097] ? watchdog+0x40/0x40 [ 3424.215992] ? hrtimer_interrupt+0x128/0x2c0 [ 3424.218737] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3424.226022] ? apic_timer_interrupt+0xf/0x20 [ 3424.228046] [ 3424.229568] ? __pv_queued_spin_lock_slowpath+0x318/0x6d0 [ 3424.232203] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3424.235121] ? newidle_balance.isra.93+0x207/0x580 [ 3424.238433] do_raw_spin_lock+0x177/0x190 [ 3424.241386] _raw_spin_lock+0x21/0x30 [ 3424.244516] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3424.248515] process_one_work+0x2c8/0x700 [ 3424.253990] worker_thread+0x296/0x6e0 [ 3424.258911] ? rescuer_thread+0x570/0x570 [ 3424.261192] kthread+0x1d1/0x200 [ 3424.263099] ? set_kthread_struct+0x70/0x70 [ 3424.266089] ret_from_fork+0x1f/0x30 [ 3430.367635] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 793s! [ 3430.371051] Showing busy workqueues and worker pools: [ 3430.373080] workqueue events: flags=0x0 [ 3430.378084] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3430.381132] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3430.392991] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3430.399485] pending: cpu_latency_work [ptlrpc] [ 3430.401525] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3430.404525] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3430.409752] workqueue events_long: flags=0x0 [ 3430.411738] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3430.417492] pending: lu_objects_flush [obdclass] [ 3430.420825] workqueue events_unbound: flags=0x2 [ 3430.431477] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3430.443791] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3430.449954] workqueue events_power_efficient: flags=0x80 [ 3430.454675] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3430.460116] pending: check_lifetime, do_cache_clean [ 3430.462797] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3430.475232] pending: neigh_periodic_work, neigh_periodic_work [ 3430.478739] workqueue mm_percpu_wq: flags=0x8 [ 3430.482603] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3430.485019] pending: vmstat_update, lru_add_drain_per_cpu [ 3430.492168] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3430.495614] pending: vmstat_update, lru_add_drain_per_cpu [ 3430.498206] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3430.505720] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3430.522314] workqueue xprtiod: flags=0xa [ 3430.526036] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3430.530707] pending: xprt_autoclose, xs_error_handle [ 3430.539396] workqueue ipv6_addrconf: flags=0x40008 [ 3430.543401] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3430.556562] pending: addrconf_verify_work [ 3430.561691] workqueue knbd0-recv: flags=0x1a [ 3430.565337] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3430.569002] in-flight: 651:recv_work [ 3430.571821] workqueue ptlrpc_pinger: flags=0x2 [ 3430.574833] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3430.583833] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3430.586929] pool 8: cpus=0-3 flags=0x4 nice=0 hung=793s workers=4 idle: 15979 31824 8 [ 3430.590457] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3421s workers=2 idle: 654 [ 3440.055143] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 3440.057353] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 3440.067758] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 3440.067774] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3440.067852] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3440.067857] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3440.067860] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3440.067873] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3440.067878] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3440.067881] RAX: 0000000000001f9b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3440.067884] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3440.067886] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3440.067888] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3440.067890] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3440.067893] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3440.067895] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3440.067897] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3440.067905] Call Trace: [ 3440.067910] [ 3440.067916] ? show_regs.cold.9+0x22/0x2f [ 3440.067926] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3440.067934] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3440.067938] ? watchdog+0x40/0x40 [ 3440.067941] ? hrtimer_interrupt+0x128/0x2c0 [ 3440.067946] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3440.067953] ? apic_timer_interrupt+0xf/0x20 [ 3440.067958] [ 3440.067961] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3440.067966] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3440.067973] do_raw_spin_lock+0x177/0x190 [ 3440.067978] _raw_spin_lock+0x21/0x30 [ 3440.067983] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3440.068960] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3440.069023] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3440.069029] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3440.069031] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3440.069040] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3440.069044] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3440.069048] RAX: 0000000000005b8f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3440.069050] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3440.069052] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3440.069054] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3440.069055] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3440.069058] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3440.069060] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3440.069062] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3440.072407] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3440.072547] Call Trace: [ 3440.072553] [ 3440.072556] ? show_regs.cold.9+0x22/0x2f [ 3440.072567] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3440.072575] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3440.072580] ? watchdog+0x40/0x40 [ 3440.072584] ? hrtimer_interrupt+0x128/0x2c0 [ 3440.072588] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3440.072596] ? apic_timer_interrupt+0xf/0x20 [ 3440.072600] [ 3440.072603] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3440.072610] do_raw_spin_lock+0x177/0x190 [ 3440.072627] _raw_spin_lock+0x21/0x30 [ 3440.072634] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3440.072856] ? do_raw_spin_unlock+0x75/0x190 [ 3440.072861] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3440.073056] ? do_wait_intr+0xf0/0xf0 [ 3440.073065] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3440.073298] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3440.073521] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3440.073746] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3440.074881] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3440.075002] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3440.075140] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3440.075195] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3440.075519] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3440.075891] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3440.075924] lmv_intent_open+0x338/0x1150 [lmv] [ 3440.075954] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3440.076057] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3440.074989] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3440.077336] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3440.077560] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3440.077789] ? woken_wake_function+0x30/0x30 [ 3440.077800] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3440.078008] kthread+0x1d1/0x200 [ 3440.078015] ? set_kthread_struct+0x70/0x70 [ 3440.078019] ret_from_fork+0x1f/0x30 [ 3440.086503] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3440.086581] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3440.086590] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3440.086592] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3440.086607] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3440.086611] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3440.086616] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3440.086619] RDX: 0000000000001a00 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3440.086621] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3440.086623] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3440.086626] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3440.086629] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3440.086632] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3440.086635] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3440.086645] Call Trace: [ 3440.086650] [ 3440.086656] ? show_regs.cold.9+0x22/0x2f [ 3440.086666] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3440.086673] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3440.086677] ? watchdog+0x40/0x40 [ 3440.086681] ? hrtimer_interrupt+0x128/0x2c0 [ 3440.086685] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3440.086694] ? apic_timer_interrupt+0xf/0x20 [ 3440.086698] [ 3440.086701] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3440.086707] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3440.086712] do_raw_spin_lock+0x177/0x190 [ 3440.086716] _raw_spin_lock+0x21/0x30 [ 3440.086723] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3440.087014] obd_mod_free+0x46/0x140 [ptlrpc] [ 3440.087230] mdc_commit_open+0x91/0xb0 [mdc] [ 3440.087281] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3440.087489] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3440.087682] after_reply+0xe2c/0x1780 [ptlrpc] [ 3440.087878] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3440.088068] ? __next_timer_interrupt+0x160/0x160 [ 3440.088077] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3440.088305] ? do_wait_intr+0xf0/0xf0 [ 3440.088314] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3440.088514] kthread+0x1d1/0x200 [ 3440.088520] ? set_kthread_struct+0x70/0x70 [ 3440.088523] ret_from_fork+0x1f/0x30 [ 3440.818508] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3440.820038] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3440.823834] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3440.826132] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3440.828398] ? do_raw_spin_unlock+0x75/0x190 [ 3440.831007] ? _raw_spin_unlock+0x12/0x30 [ 3440.834461] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3440.836043] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3440.839676] ll_atomic_open+0x36a/0xd70 [lustre] [ 3440.841712] lookup_open+0x356/0xb40 [ 3440.843624] do_last+0x313/0xfc0 [ 3440.846063] ? nd_jump_root+0xe5/0x160 [ 3440.850644] ? path_init+0x437/0x520 [ 3440.856354] path_openat+0xf7/0x500 [ 3440.858433] ? do_raw_spin_unlock+0x75/0x190 [ 3440.860723] do_filp_open+0x99/0x140 [ 3440.862675] ? getname_flags+0x6e/0x330 [ 3440.865554] ? __check_object_size+0xff/0x256 [ 3440.869516] ? do_raw_spin_unlock+0x75/0x190 [ 3440.872793] ? _raw_spin_unlock+0x12/0x30 [ 3440.875909] do_sys_openat2+0x2b4/0x410 [ 3440.878916] do_sys_open+0x73/0xa0 [ 3440.882679] __x64_sys_openat+0x24/0x30 [ 3440.884109] do_syscall_64+0xc1/0x440 [ 3440.888152] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3440.894896] RIP: 0033:0x7f53425a5332 [ 3440.898336] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3440.924483] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3440.932850] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3440.938016] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3440.942061] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3440.946749] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3440.952808] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3456.052010] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 3456.075595] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3456.139514] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3456.149021] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3456.160666] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3456.163210] RIP: 0010:__pv_queued_spin_lock_slowpath+0x320/0x6d0 [ 3456.182174] Code: 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 <48> 83 05 18 d4 dc 02 01 eb e0 48 83 05 06 d4 dc 02 01 48 83 05 be [ 3456.204846] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3456.218781] RAX: 000000000000414d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3456.242174] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3456.244771] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3456.257428] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3456.265399] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3456.274199] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3456.280612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3456.283939] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3456.303950] Call Trace: [ 3456.305822] [ 3456.308525] ? show_regs.cold.9+0x22/0x2f [ 3456.313151] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3456.328949] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3456.330462] ? watchdog+0x40/0x40 [ 3456.333790] ? hrtimer_interrupt+0x128/0x2c0 [ 3456.336447] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3456.344308] ? apic_timer_interrupt+0xf/0x20 [ 3456.347744] [ 3456.349498] ? __pv_queued_spin_lock_slowpath+0x320/0x6d0 [ 3456.358530] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3456.366919] ? newidle_balance.isra.93+0x207/0x580 [ 3456.376374] do_raw_spin_lock+0x177/0x190 [ 3456.385905] _raw_spin_lock+0x21/0x30 [ 3456.389491] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3456.398108] process_one_work+0x2c8/0x700 [ 3456.414114] worker_thread+0x296/0x6e0 [ 3456.419420] ? rescuer_thread+0x570/0x570 [ 3456.431872] kthread+0x1d1/0x200 [ 3456.432905] ? set_kthread_struct+0x70/0x70 [ 3456.442424] ret_from_fork+0x1f/0x30 [ 3461.088078] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 823s! [ 3461.092814] Showing busy workqueues and worker pools: [ 3461.099945] workqueue events: flags=0x0 [ 3461.103124] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3461.107133] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3461.111244] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3461.123856] pending: cpu_latency_work [ptlrpc] [ 3461.130974] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3461.134477] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3461.143417] workqueue events_long: flags=0x0 [ 3461.145619] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3461.150860] pending: lu_objects_flush [obdclass] [ 3461.152581] workqueue events_unbound: flags=0x2 [ 3461.155524] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3461.160393] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3461.171679] workqueue events_power_efficient: flags=0x80 [ 3461.174155] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3461.177678] pending: check_lifetime, do_cache_clean [ 3461.183007] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3461.186895] pending: neigh_periodic_work, neigh_periodic_work [ 3461.193738] workqueue mm_percpu_wq: flags=0x8 [ 3461.197783] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3461.203156] pending: vmstat_update, lru_add_drain_per_cpu [ 3461.205708] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3461.215152] pending: vmstat_update, lru_add_drain_per_cpu [ 3461.225012] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3461.231807] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3461.240986] workqueue xprtiod: flags=0xa [ 3461.243180] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3461.245444] pending: xprt_autoclose, xs_error_handle [ 3461.248980] workqueue ipv6_addrconf: flags=0x40008 [ 3461.252185] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3461.259562] pending: addrconf_verify_work [ 3461.264805] workqueue knbd0-recv: flags=0x1a [ 3461.266218] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3461.269155] in-flight: 651:recv_work [ 3461.270246] workqueue ptlrpc_pinger: flags=0x2 [ 3461.272012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3461.273891] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3461.290473] pool 8: cpus=0-3 flags=0x4 nice=0 hung=824s workers=4 idle: 15979 31824 8 [ 3461.303406] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3451s workers=2 idle: 654 [ 3468.056509] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 3468.057004] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 3468.061008] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 3468.061021] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3468.061089] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3468.061094] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3468.061097] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3468.061111] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3468.061115] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3468.061119] RAX: 0000000000007b0c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3468.061121] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3468.061124] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3468.061125] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3468.061127] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3468.061130] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3468.061133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3468.061136] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3468.061145] Call Trace: [ 3468.061151] [ 3468.061156] ? show_regs.cold.9+0x22/0x2f [ 3468.061167] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3468.061174] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3468.061178] ? watchdog+0x40/0x40 [ 3468.061182] ? hrtimer_interrupt+0x128/0x2c0 [ 3468.061186] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3468.061194] ? apic_timer_interrupt+0xf/0x20 [ 3468.061198] [ 3468.061201] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3468.061206] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3468.061211] do_raw_spin_lock+0x177/0x190 [ 3468.061216] _raw_spin_lock+0x21/0x30 [ 3468.061221] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3468.061489] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3468.064037] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3468.064110] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3468.064118] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3468.064121] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3468.064134] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3468.064138] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3468.064142] RAX: 0000000000002283 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3468.064145] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3468.064147] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3468.064149] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3468.064152] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3468.064155] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3468.064157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3468.064160] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3468.064168] Call Trace: [ 3468.064173] [ 3468.064179] ? show_regs.cold.9+0x22/0x2f [ 3468.064188] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3468.064195] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3468.064199] ? watchdog+0x40/0x40 [ 3468.064203] ? hrtimer_interrupt+0x128/0x2c0 [ 3468.064208] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3468.064216] ? apic_timer_interrupt+0xf/0x20 [ 3468.064220] [ 3468.064223] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3468.064230] do_raw_spin_lock+0x177/0x190 [ 3468.064235] _raw_spin_lock+0x21/0x30 [ 3468.064240] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3468.064504] ? do_raw_spin_unlock+0x75/0x190 [ 3468.064510] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3468.064699] ? do_wait_intr+0xf0/0xf0 [ 3468.061682] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3468.064707] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3468.068766] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3468.068982] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3468.069095] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3468.069163] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3468.069169] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3468.069172] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3468.069184] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3468.069201] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3468.069206] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3468.069209] RDX: 0000000000005430 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3468.069195] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3468.069318] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3468.069321] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3468.069323] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3468.069326] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3468.069329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3468.069310] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3468.069332] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3468.069342] Call Trace: [ 3468.069347] [ 3468.069353] ? show_regs.cold.9+0x22/0x2f [ 3468.069350] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3468.069365] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3468.069373] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3468.069377] ? watchdog+0x40/0x40 [ 3468.069387] ? hrtimer_interrupt+0x128/0x2c0 [ 3468.069392] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3468.069400] ? apic_timer_interrupt+0xf/0x20 [ 3468.069389] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3468.069408] [ 3468.069410] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3468.069415] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3468.069431] do_raw_spin_lock+0x177/0x190 [ 3468.069439] _raw_spin_lock+0x21/0x30 [ 3468.069445] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3468.069431] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3468.070077] obd_mod_free+0x46/0x140 [ptlrpc] [ 3468.071481] mdc_commit_open+0x91/0xb0 [mdc] [ 3468.071520] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3468.071724] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3468.071930] after_reply+0xe2c/0x1780 [ptlrpc] [ 3468.070081] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3468.073054] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3468.073098] lmv_intent_open+0x338/0x1150 [lmv] [ 3468.073133] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3468.073247] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3468.072122] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3468.073511] ? __next_timer_interrupt+0x160/0x160 [ 3468.073522] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3468.076652] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3468.076712] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3468.076848] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3468.076977] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3468.077075] ? do_raw_spin_unlock+0x75/0x190 [ 3468.077082] ? _raw_spin_unlock+0x12/0x30 [ 3468.077090] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3468.077184] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3468.077205] ll_atomic_open+0x36a/0xd70 [lustre] [ 3468.077290] lookup_open+0x356/0xb40 [ 3468.077300] do_last+0x313/0xfc0 [ 3468.077306] ? nd_jump_root+0xe5/0x160 [ 3468.077311] ? path_init+0x437/0x520 [ 3468.077316] path_openat+0xf7/0x500 [ 3468.077320] ? do_raw_spin_unlock+0x75/0x190 [ 3468.077325] do_filp_open+0x99/0x140 [ 3468.077331] ? getname_flags+0x6e/0x330 [ 3468.077336] ? __check_object_size+0xff/0x256 [ 3468.077351] ? do_raw_spin_unlock+0x75/0x190 [ 3468.077356] ? _raw_spin_unlock+0x12/0x30 [ 3468.077361] do_sys_openat2+0x2b4/0x410 [ 3468.077365] do_sys_open+0x73/0xa0 [ 3468.077502] __x64_sys_openat+0x24/0x30 [ 3468.077506] do_syscall_64+0xc1/0x440 [ 3468.077512] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3468.077519] RIP: 0033:0x7f53425a5332 [ 3468.077527] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3468.077531] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3468.077537] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3468.077539] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3468.077541] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3468.077543] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3468.077545] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3468.089610] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3468.117837] ? do_wait_intr+0xf0/0xf0 [ 3468.117860] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3468.126743] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3468.131454] kthread+0x1d1/0x200 [ 3468.131472] ? set_kthread_struct+0x70/0x70 [ 3468.131477] ret_from_fork+0x1f/0x30 [ 3469.072709] ? woken_wake_function+0x30/0x30 [ 3469.075313] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3469.089131] kthread+0x1d1/0x200 [ 3469.093484] ? set_kthread_struct+0x70/0x70 [ 3469.100014] ret_from_fork+0x1f/0x30 [ 3488.052476] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 3488.061087] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3488.104312] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3488.120729] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3488.129855] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3488.137776] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3488.145499] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3488.160329] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3488.166811] RAX: 0000000000007e3b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3488.173712] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3488.180617] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3488.191246] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3488.199338] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3488.203306] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3488.210796] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3488.214023] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3488.219157] Call Trace: [ 3488.220755] [ 3488.222233] ? show_regs.cold.9+0x22/0x2f [ 3488.225198] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3488.227868] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3488.230166] ? watchdog+0x40/0x40 [ 3488.232158] ? hrtimer_interrupt+0x128/0x2c0 [ 3488.234300] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3488.236823] ? apic_timer_interrupt+0xf/0x20 [ 3488.239160] [ 3488.240472] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3488.243195] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3488.247044] ? newidle_balance.isra.93+0x207/0x580 [ 3488.251770] do_raw_spin_lock+0x177/0x190 [ 3488.253456] _raw_spin_lock+0x21/0x30 [ 3488.256706] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3488.262903] process_one_work+0x2c8/0x700 [ 3488.267690] worker_thread+0x296/0x6e0 [ 3488.271657] ? rescuer_thread+0x570/0x570 [ 3488.273169] kthread+0x1d1/0x200 [ 3488.274387] ? set_kthread_struct+0x70/0x70 [ 3488.279473] ret_from_fork+0x1f/0x30 [ 3491.814858] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 854s! [ 3491.820645] Showing busy workqueues and worker pools: [ 3491.823607] workqueue events: flags=0x0 [ 3491.826995] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3491.830667] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3491.837516] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3491.839649] pending: cpu_latency_work [ptlrpc] [ 3491.843778] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3491.846866] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3491.853297] workqueue events_long: flags=0x0 [ 3491.855378] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3491.859322] pending: lu_objects_flush [obdclass] [ 3491.862019] workqueue events_unbound: flags=0x2 [ 3491.864226] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3491.868692] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3491.871691] workqueue events_power_efficient: flags=0x80 [ 3491.876109] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3491.881002] pending: check_lifetime, do_cache_clean [ 3491.883047] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3491.887669] pending: neigh_periodic_work, neigh_periodic_work [ 3491.890416] workqueue mm_percpu_wq: flags=0x8 [ 3491.892510] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3491.895414] pending: vmstat_update, lru_add_drain_per_cpu [ 3491.902171] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3491.905045] pending: vmstat_update, lru_add_drain_per_cpu [ 3491.907651] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3491.910060] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3491.913057] workqueue xprtiod: flags=0xa [ 3491.914497] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3491.917223] pending: xprt_autoclose, xs_error_handle [ 3491.919121] workqueue ipv6_addrconf: flags=0x40008 [ 3491.920977] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3491.925821] pending: addrconf_verify_work [ 3491.927673] workqueue knbd0-recv: flags=0x1a [ 3491.929786] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3491.935179] in-flight: 651:recv_work [ 3491.936941] workqueue ptlrpc_pinger: flags=0x2 [ 3491.938462] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3491.941103] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3491.943417] pool 8: cpus=0-3 flags=0x4 nice=0 hung=854s workers=4 idle: 15979 31824 8 [ 3491.946213] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3482s workers=2 idle: 654 [ 3496.056662] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 3496.058825] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 3496.059507] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3496.059576] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3496.059582] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3496.059586] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3496.059600] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3496.059604] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3496.059608] RAX: 0000000000007b86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3496.059610] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3496.059612] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3496.059614] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3496.059616] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3496.059620] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3496.059623] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3496.059626] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3496.059636] Call Trace: [ 3496.059641] [ 3496.059646] ? show_regs.cold.9+0x22/0x2f [ 3496.059656] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3496.059662] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3496.059667] ? watchdog+0x40/0x40 [ 3496.059670] ? hrtimer_interrupt+0x128/0x2c0 [ 3496.059674] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3496.059682] ? apic_timer_interrupt+0xf/0x20 [ 3496.059686] [ 3496.059689] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3496.059696] do_raw_spin_lock+0x177/0x190 [ 3496.059701] _raw_spin_lock+0x21/0x30 [ 3496.059706] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3496.059970] ? do_raw_spin_unlock+0x75/0x190 [ 3496.059976] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3496.062536] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 3496.062549] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3496.062618] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3496.062623] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3496.062626] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3496.062640] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3496.062645] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3496.062649] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3496.062652] RDX: 00000000000030b9 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3496.062654] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3496.062656] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3496.062659] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3496.062662] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3496.062665] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3496.062667] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3496.062678] Call Trace: [ 3496.062683] [ 3496.062688] ? show_regs.cold.9+0x22/0x2f [ 3496.062697] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3496.062703] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3496.062707] ? watchdog+0x40/0x40 [ 3496.062712] ? hrtimer_interrupt+0x128/0x2c0 [ 3496.062716] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3496.062724] ? apic_timer_interrupt+0xf/0x20 [ 3496.062729] [ 3496.062732] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3496.062736] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3496.062742] do_raw_spin_lock+0x177/0x190 [ 3496.062746] _raw_spin_lock+0x21/0x30 [ 3496.062753] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3496.063051] obd_mod_free+0x46/0x140 [ptlrpc] [ 3496.063257] mdc_commit_open+0x91/0xb0 [mdc] [ 3496.063308] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3496.063503] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3496.063712] after_reply+0xe2c/0x1780 [ptlrpc] [ 3496.063932] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3496.064146] ? __next_timer_interrupt+0x160/0x160 [ 3496.064156] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3496.064375] ? do_wait_intr+0xf0/0xf0 [ 3496.064384] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3496.064587] kthread+0x1d1/0x200 [ 3496.064593] ? set_kthread_struct+0x70/0x70 [ 3496.064596] ret_from_fork+0x1f/0x30 [ 3496.065184] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3496.065443] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3496.065450] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3496.065453] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3496.065463] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3496.065467] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3496.065471] RAX: 0000000000007885 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3496.065473] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3496.065476] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3496.065478] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3496.065480] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3496.065483] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3496.065486] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3496.065488] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3496.065496] Call Trace: [ 3496.065500] [ 3496.065504] ? show_regs.cold.9+0x22/0x2f [ 3496.065514] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3496.065519] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3496.065522] ? watchdog+0x40/0x40 [ 3496.065526] ? hrtimer_interrupt+0x128/0x2c0 [ 3496.065530] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3496.065537] ? apic_timer_interrupt+0xf/0x20 [ 3496.065541] [ 3496.065544] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3496.065549] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3496.065555] do_raw_spin_lock+0x177/0x190 [ 3496.065559] _raw_spin_lock+0x21/0x30 [ 3496.065563] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3496.065862] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3496.066060] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3496.066486] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3496.066693] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3496.069480] ? woken_wake_function+0x30/0x30 [ 3496.069490] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3496.069867] kthread+0x1d1/0x200 [ 3496.069874] ? set_kthread_struct+0x70/0x70 [ 3496.069878] ret_from_fork+0x1f/0x30 [ 3496.618370] ? do_wait_intr+0xf0/0xf0 [ 3496.620117] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3496.622930] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3496.625446] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3496.629376] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3496.632087] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3496.634637] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3496.637212] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3496.639156] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3496.643055] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3496.646645] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3496.649940] lmv_intent_open+0x338/0x1150 [lmv] [ 3496.652116] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3496.654899] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3496.657041] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3496.658711] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3496.660750] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3496.663960] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3496.666959] ? do_raw_spin_unlock+0x75/0x190 [ 3496.669043] ? _raw_spin_unlock+0x12/0x30 [ 3496.671607] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3496.674568] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3496.677842] ll_atomic_open+0x36a/0xd70 [lustre] [ 3496.680684] lookup_open+0x356/0xb40 [ 3496.682395] do_last+0x313/0xfc0 [ 3496.684044] ? nd_jump_root+0xe5/0x160 [ 3496.685744] ? path_init+0x437/0x520 [ 3496.686662] path_openat+0xf7/0x500 [ 3496.687736] ? do_raw_spin_unlock+0x75/0x190 [ 3496.689603] do_filp_open+0x99/0x140 [ 3496.691527] ? getname_flags+0x6e/0x330 [ 3496.693129] ? __check_object_size+0xff/0x256 [ 3496.695092] ? do_raw_spin_unlock+0x75/0x190 [ 3496.696680] ? _raw_spin_unlock+0x12/0x30 [ 3496.698337] do_sys_openat2+0x2b4/0x410 [ 3496.699843] do_sys_open+0x73/0xa0 [ 3496.701330] __x64_sys_openat+0x24/0x30 [ 3496.702820] do_syscall_64+0xc1/0x440 [ 3496.704438] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3496.706825] RIP: 0033:0x7f53425a5332 [ 3496.708107] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3496.715092] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3496.718434] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3496.720746] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3496.722941] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3496.725765] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3496.728546] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3516.054978] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 3516.060866] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3516.123548] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3516.137329] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3516.145857] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3516.160052] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3516.168339] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3516.199158] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3516.204165] RAX: 0000000000007d8a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3516.210148] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3516.219779] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3516.228616] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3516.240361] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3516.242662] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3516.251046] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3516.255989] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3516.259237] Call Trace: [ 3516.260151] [ 3516.260800] ? show_regs.cold.9+0x22/0x2f [ 3516.262498] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3516.264312] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3516.266154] ? watchdog+0x40/0x40 [ 3516.267393] ? hrtimer_interrupt+0x128/0x2c0 [ 3516.269146] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3516.278169] ? apic_timer_interrupt+0xf/0x20 [ 3516.279646] [ 3516.280430] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3516.285561] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3516.291798] ? newidle_balance.isra.93+0x207/0x580 [ 3516.297870] do_raw_spin_lock+0x177/0x190 [ 3516.301671] _raw_spin_lock+0x21/0x30 [ 3516.302777] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3516.305219] process_one_work+0x2c8/0x700 [ 3516.306568] worker_thread+0x296/0x6e0 [ 3516.307854] ? rescuer_thread+0x570/0x570 [ 3516.309991] kthread+0x1d1/0x200 [ 3516.311049] ? set_kthread_struct+0x70/0x70 [ 3516.312397] ret_from_fork+0x1f/0x30 [ 3522.016058] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 884s! [ 3522.019242] Showing busy workqueues and worker pools: [ 3522.022214] workqueue events: flags=0x0 [ 3522.024297] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3522.028985] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3522.035140] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3522.038788] pending: cpu_latency_work [ptlrpc] [ 3522.041209] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3522.044994] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3522.048612] workqueue events_long: flags=0x0 [ 3522.050689] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3522.054057] pending: lu_objects_flush [obdclass] [ 3522.057016] workqueue events_unbound: flags=0x2 [ 3522.059174] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3522.062544] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3522.066188] workqueue events_power_efficient: flags=0x80 [ 3522.068601] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3522.071760] pending: check_lifetime, do_cache_clean [ 3522.074254] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3522.077528] pending: neigh_periodic_work, neigh_periodic_work [ 3522.083869] workqueue mm_percpu_wq: flags=0x8 [ 3522.089699] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3522.096654] pending: vmstat_update, lru_add_drain_per_cpu [ 3522.100246] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3522.108168] pending: vmstat_update, lru_add_drain_per_cpu [ 3522.114169] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3522.120771] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3522.126944] workqueue xprtiod: flags=0xa [ 3522.129240] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3522.133079] pending: xprt_autoclose, xs_error_handle [ 3522.142187] workqueue ipv6_addrconf: flags=0x40008 [ 3522.145105] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3522.148576] pending: addrconf_verify_work [ 3522.150922] workqueue knbd0-recv: flags=0x1a [ 3522.153051] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3522.156557] in-flight: 651:recv_work [ 3522.162136] workqueue ptlrpc_pinger: flags=0x2 [ 3522.167603] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3522.171527] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3522.174986] pool 8: cpus=0-3 flags=0x4 nice=0 hung=885s workers=4 idle: 15979 31824 8 [ 3522.179068] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3512s workers=2 idle: 654 [ 3524.056765] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 3524.057066] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 3524.059198] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 3524.059209] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3524.059284] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3524.059289] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3524.059291] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3524.059304] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3524.059308] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3524.059313] RAX: 0000000000007b13 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3524.059315] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3524.059318] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3524.059320] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3524.059322] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3524.059326] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3524.059329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3524.059331] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3524.059571] Call Trace: [ 3524.059579] [ 3524.059585] ? show_regs.cold.9+0x22/0x2f [ 3524.059596] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3524.059603] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3524.059608] ? watchdog+0x40/0x40 [ 3524.059612] ? hrtimer_interrupt+0x128/0x2c0 [ 3524.059617] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3524.059626] ? apic_timer_interrupt+0xf/0x20 [ 3524.059631] [ 3524.059633] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3524.059638] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3524.059644] do_raw_spin_lock+0x177/0x190 [ 3524.059649] _raw_spin_lock+0x21/0x30 [ 3524.059654] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3524.059956] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3524.060157] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3524.060691] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3524.060908] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3524.061121] ? woken_wake_function+0x30/0x30 [ 3524.061131] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3524.061700] kthread+0x1d1/0x200 [ 3524.061707] ? set_kthread_struct+0x70/0x70 [ 3524.061710] ret_from_fork+0x1f/0x30 [ 3524.065607] Modules linked in: [ 3524.072026] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3524.072104] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3524.072111] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3524.072114] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3524.072129] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3524.072132] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3524.072137] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3524.072139] RDX: 00000000000022a8 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3524.072141] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3524.072143] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3524.072145] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3524.072148] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3524.072151] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3524.072153] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3524.072163] Call Trace: [ 3524.072168] [ 3524.072174] ? show_regs.cold.9+0x22/0x2f [ 3524.072183] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3524.072189] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3524.072194] ? watchdog+0x40/0x40 [ 3524.072198] ? hrtimer_interrupt+0x128/0x2c0 [ 3524.072202] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3524.072210] ? apic_timer_interrupt+0xf/0x20 [ 3524.072215] [ 3524.072218] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3524.072223] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3524.072227] do_raw_spin_lock+0x177/0x190 [ 3524.072232] _raw_spin_lock+0x21/0x30 [ 3524.072254] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3524.072682] obd_mod_free+0x46/0x140 [ptlrpc] [ 3524.072899] mdc_commit_open+0x91/0xb0 [mdc] [ 3524.072936] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3524.073133] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3524.080954] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3524.081034] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3524.081041] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3524.081044] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3524.081059] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3524.081064] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3524.081068] RAX: 0000000000007e95 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3524.081070] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3524.081072] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3524.081075] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3524.081077] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3524.081080] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3524.081083] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3524.081085] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3524.081094] Call Trace: [ 3524.081099] [ 3524.081105] ? show_regs.cold.9+0x22/0x2f [ 3524.081117] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3524.081124] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3524.081128] ? watchdog+0x40/0x40 [ 3524.081133] ? hrtimer_interrupt+0x128/0x2c0 [ 3524.081137] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3524.081145] ? apic_timer_interrupt+0xf/0x20 [ 3524.081149] [ 3524.081152] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3524.118431] after_reply+0xe2c/0x1780 [ptlrpc] [ 3524.120460] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3524.134306] do_raw_spin_lock+0x177/0x190 [ 3524.139356] ? __next_timer_interrupt+0x160/0x160 [ 3524.146695] _raw_spin_lock+0x21/0x30 [ 3524.157710] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3524.158195] ? do_wait_intr+0xf0/0xf0 [ 3524.158208] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3524.162053] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3524.162318] ? do_raw_spin_unlock+0x75/0x190 [ 3524.162327] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3524.162521] ? do_wait_intr+0xf0/0xf0 [ 3524.162531] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3524.162746] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3524.163148] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3524.163362] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3524.163475] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3524.163517] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3524.163544] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3524.163577] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3524.163668] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3524.163909] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3524.163939] lmv_intent_open+0x338/0x1150 [lmv] [ 3524.163974] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3524.164061] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3524.164145] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3524.164168] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3524.164255] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3524.164344] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3524.164426] ? do_raw_spin_unlock+0x75/0x190 [ 3524.164432] ? _raw_spin_unlock+0x12/0x30 [ 3524.164438] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3524.164464] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3524.164484] ll_atomic_open+0x36a/0xd70 [lustre] [ 3524.165026] lookup_open+0x356/0xb40 [ 3524.165034] do_last+0x313/0xfc0 [ 3524.165039] ? nd_jump_root+0xe5/0x160 [ 3524.165042] ? path_init+0x437/0x520 [ 3524.165046] path_openat+0xf7/0x500 [ 3524.165050] ? do_raw_spin_unlock+0x75/0x190 [ 3524.165055] do_filp_open+0x99/0x140 [ 3524.165061] ? getname_flags+0x6e/0x330 [ 3524.165066] ? __check_object_size+0xff/0x256 [ 3524.165074] ? do_raw_spin_unlock+0x75/0x190 [ 3524.165076] ? _raw_spin_unlock+0x12/0x30 [ 3524.165080] do_sys_openat2+0x2b4/0x410 [ 3524.165084] do_sys_open+0x73/0xa0 [ 3524.165088] __x64_sys_openat+0x24/0x30 [ 3524.165091] do_syscall_64+0xc1/0x440 [ 3524.165099] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3524.165105] RIP: 0033:0x7f53425a5332 [ 3524.165112] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3524.165116] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3524.165121] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3524.165123] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3524.165125] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3524.165126] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3524.165128] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3524.498151] kthread+0x1d1/0x200 [ 3524.499353] ? set_kthread_struct+0x70/0x70 [ 3524.500576] ret_from_fork+0x1f/0x30 [ 3548.052985] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 3548.064009] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3548.150688] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3548.171341] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3548.192907] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3548.210543] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3548.220138] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3548.246340] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3548.278944] RAX: 0000000000002b9a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3548.281017] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3548.295591] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3548.311726] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3548.318815] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3548.326224] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3548.337620] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3548.343919] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3548.355137] Call Trace: [ 3548.357066] [ 3548.358952] ? show_regs.cold.9+0x22/0x2f [ 3548.361228] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3548.371225] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3548.379503] ? watchdog+0x40/0x40 [ 3548.381216] ? hrtimer_interrupt+0x128/0x2c0 [ 3548.386649] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3548.391718] ? apic_timer_interrupt+0xf/0x20 [ 3548.395171] [ 3548.398999] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3548.411172] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3548.421452] ? newidle_balance.isra.93+0x207/0x580 [ 3548.429751] do_raw_spin_lock+0x177/0x190 [ 3548.434296] _raw_spin_lock+0x21/0x30 [ 3548.436255] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3548.443230] process_one_work+0x2c8/0x700 [ 3548.446289] worker_thread+0x296/0x6e0 [ 3548.454660] ? rescuer_thread+0x570/0x570 [ 3548.465452] kthread+0x1d1/0x200 [ 3548.467432] ? set_kthread_struct+0x70/0x70 [ 3548.472114] ret_from_fork+0x1f/0x30 [ 3552.056880] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 3552.060313] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 3552.060750] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 3552.060756] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3552.060825] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3552.060831] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3552.060834] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3552.060847] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3552.060851] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3552.060855] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3552.060857] RDX: 000000000000508d RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3552.060859] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3552.060861] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3552.060862] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3552.060865] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3552.060867] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3552.060869] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3552.060878] Call Trace: [ 3552.060883] [ 3552.060889] ? show_regs.cold.9+0x22/0x2f [ 3552.060898] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3552.060904] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3552.060908] ? watchdog+0x40/0x40 [ 3552.060913] ? hrtimer_interrupt+0x128/0x2c0 [ 3552.060916] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3552.060924] ? apic_timer_interrupt+0xf/0x20 [ 3552.060928] [ 3552.060931] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3552.060936] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3552.060941] do_raw_spin_lock+0x177/0x190 [ 3552.060946] _raw_spin_lock+0x21/0x30 [ 3552.060952] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3552.061215] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 [ 3552.061220] obd_mod_free+0x46/0x140 [ptlrpc] [ 3552.061941] mdc_commit_open+0x91/0xb0 [mdc] [ 3552.061978] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3552.062174] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3552.062489] sha512_generic [ 3552.062497] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3552.062503] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3552.062506] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3552.062516] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3552.062521] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3552.062526] RAX: 0000000000002f8c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3552.062528] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3552.062531] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3552.062533] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3552.062536] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3552.062538] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3552.062541] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3552.062544] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3552.062552] Call Trace: [ 3552.062556] [ 3552.062559] ? show_regs.cold.9+0x22/0x2f [ 3552.062568] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3552.062574] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3552.062578] ? watchdog+0x40/0x40 [ 3552.062582] ? hrtimer_interrupt+0x128/0x2c0 [ 3552.062586] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3552.063017] ? apic_timer_interrupt+0xf/0x20 [ 3552.063023] [ 3552.063027] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3552.063034] do_raw_spin_lock+0x177/0x190 [ 3552.063039] _raw_spin_lock+0x21/0x30 [ 3552.063044] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3552.062379] after_reply+0xe2c/0x1780 [ptlrpc] [ 3552.065839] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3552.066051] ? __next_timer_interrupt+0x160/0x160 [ 3552.066061] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3552.066260] ? do_wait_intr+0xf0/0xf0 [ 3552.066270] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3552.067988] kthread+0x1d1/0x200 [ 3552.067996] ? set_kthread_struct+0x70/0x70 [ 3552.068000] ret_from_fork+0x1f/0x30 [ 3552.075234] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver [ 3552.086391] ? do_raw_spin_unlock+0x75/0x190 [ 3552.086412] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3552.086685] ? do_wait_intr+0xf0/0xf0 [ 3552.086694] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3552.086901] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3552.087123] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3552.091157] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3552.091353] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3552.091394] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3552.091423] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3552.091450] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3552.091535] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3552.092478] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3552.092511] lmv_intent_open+0x338/0x1150 [lmv] [ 3552.092541] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3552.093164] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3552.093248] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3552.093451] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3552.093542] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3552.094109] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3552.094194] ? do_raw_spin_unlock+0x75/0x190 [ 3552.094202] ? _raw_spin_unlock+0x12/0x30 [ 3552.094209] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3552.094234] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3552.094254] ll_atomic_open+0x36a/0xd70 [lustre] [ 3552.095425] lookup_open+0x356/0xb40 [ 3552.095434] do_last+0x313/0xfc0 [ 3552.095440] ? nd_jump_root+0xe5/0x160 [ 3552.095445] ? path_init+0x437/0x520 [ 3552.095449] path_openat+0xf7/0x500 [ 3552.095454] ? do_raw_spin_unlock+0x75/0x190 [ 3552.095459] do_filp_open+0x99/0x140 [ 3552.095465] ? getname_flags+0x6e/0x330 [ 3552.095469] ? __check_object_size+0xff/0x256 [ 3552.095476] ? do_raw_spin_unlock+0x75/0x190 [ 3552.095479] ? _raw_spin_unlock+0x12/0x30 [ 3552.095484] do_sys_openat2+0x2b4/0x410 [ 3552.095488] do_sys_open+0x73/0xa0 [ 3552.095492] __x64_sys_openat+0x24/0x30 [ 3552.095496] do_syscall_64+0xc1/0x440 [ 3552.095501] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3552.095507] RIP: 0033:0x7f53425a5332 [ 3552.095516] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3552.095520] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3552.095525] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3552.095527] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3552.095529] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3552.095531] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3552.095533] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3552.241775] BUG: workqueue lockup - pool [ 3552.245090] intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3552.245127] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3552.245133] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3552.245136] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3552.245149] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3552.245153] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3552.245157] RAX: 0000000000007689 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3552.245159] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3552.245162] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3552.245163] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3552.245165] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3552.245168] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3552.245171] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3552.245173] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3552.245182] Call Trace: [ 3552.245187] [ 3552.245194] ? show_regs.cold.9+0x22/0x2f [ 3552.245203] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3552.245208] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3552.245212] ? watchdog+0x40/0x40 [ 3552.245216] ? hrtimer_interrupt+0x128/0x2c0 [ 3552.245220] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3552.245228] ? apic_timer_interrupt+0xf/0x20 [ 3552.245232] [ 3552.245235] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3552.245240] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3552.245247] do_raw_spin_lock+0x177/0x190 [ 3552.245251] _raw_spin_lock+0x21/0x30 [ 3552.245256] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3552.245527] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3552.245721] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3552.245933] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3552.246141] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3552.269852] cpus=0-3 flags=0x4 nice=0 stuck for 915s! [ 3552.269885] Showing busy workqueues and worker pools: [ 3552.269887] workqueue events: flags=0x0 [ 3552.269890] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3552.269898] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3552.270360] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3552.270370] pending: cpu_latency_work [ptlrpc] [ 3552.286282] ? woken_wake_function+0x30/0x30 [ 3552.299264] [ 3552.299655] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3552.299670] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3552.299732] workqueue events_long: flags=0x0 [ 3552.299735] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3552.299741] pending: lu_objects_flush [obdclass] [ 3552.301743] workqueue events_unbound: flags=0x2 [ 3552.301748] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3552.301758] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3552.301783] workqueue events_power_efficient: flags=0x80 [ 3552.301787] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3552.301794] pending: check_lifetime, do_cache_clean [ 3552.301806] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3552.301814] pending: neigh_periodic_work, neigh_periodic_work [ 3552.301826] workqueue mm_percpu_wq: flags=0x8 [ 3552.301829] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3552.301835] pending: vmstat_update, lru_add_drain_per_cpu [ 3552.301847] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3552.301853] pending: vmstat_update, lru_add_drain_per_cpu [ 3552.301860] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3552.301867] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3552.301887] workqueue xprtiod: flags=0xa [ 3552.301889] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3552.301895] pending: xprt_autoclose, xs_error_handle [ 3552.301909] workqueue ipv6_addrconf: flags=0x40008 [ 3552.301912] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3552.301920] pending: addrconf_verify_work [ 3552.301932] workqueue knbd0-recv: flags=0x1a [ 3552.301935] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3552.301941] in-flight: 651:recv_work [ 3552.301951] workqueue ptlrpc_pinger: flags=0x2 [ 3552.301954] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3552.301958] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3552.302300] pool 8: cpus=0-3 flags=0x4 nice=0 hung=915s workers=4 idle: 15979 31824 8 [ 3552.302313] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3542s workers=2 idle: 654 [ 3553.797967] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3553.819470] kthread+0x1d1/0x200 [ 3553.826152] ? set_kthread_struct+0x70/0x70 [ 3553.834709] ret_from_fork+0x1f/0x30 [ 3580.054565] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 3580.055648] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 3580.059096] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 3580.059110] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [ 3580.059121] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 3580.059125] mdt(O) [ 3580.059131] Modules linked in: [ 3580.059131] mdd(O) [ 3580.059136] lustre(O) [ 3580.059137] mgs(O) [ 3580.059142] osp(O) [ 3580.059143] lquota(O) [ 3580.059148] ofd(O) [ 3580.059149] lfsck(O) [ 3580.059154] lod(O) [ 3580.059155] obdecho(O) [ 3580.059159] mdt(O) [ 3580.059160] mgc(O) [ 3580.059165] mdd(O) [ 3580.059167] mdc(O) [ 3580.059170] mgs(O) [ 3580.059172] lov(O) [ 3580.059175] lquota(O) [ 3580.059177] osc(O) [ 3580.059179] lfsck(O) [ 3580.059182] lmv(O) [ 3580.059184] obdecho(O) [ 3580.059187] ec(O) [ 3580.059189] mgc(O) [ 3580.059193] fid(O) [ 3580.059195] mdc(O) [ 3580.059199] fld(O) [ 3580.059200] lov(O) [ 3580.059204] ptlrpc_gss(O) [ 3580.059205] osc(O) [ 3580.059210] ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3580.059252] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3580.059711] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3580.059757] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3580.059763] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3580.059766] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3580.059779] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3580.059784] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3580.059789] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3580.059791] RDX: 00000000000065d9 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3580.059793] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3580.059795] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3580.059808] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3580.059811] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3580.059814] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3580.059817] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3580.059826] Call Trace: [ 3580.059831] [ 3580.059837] ? show_regs.cold.9+0x22/0x2f [ 3580.059847] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3580.059854] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3580.059858] ? watchdog+0x40/0x40 [ 3580.059863] ? hrtimer_interrupt+0x128/0x2c0 [ 3580.059868] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3580.059876] ? apic_timer_interrupt+0xf/0x20 [ 3580.059881] [ 3580.059884] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3580.059888] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3580.059893] do_raw_spin_lock+0x177/0x190 [ 3580.059897] _raw_spin_lock+0x21/0x30 [ 3580.060571] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3580.060575] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3580.060584] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3580.060588] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3580.059904] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3580.060596] RAX: 0000000000006251 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3580.060599] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3580.060601] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3580.060603] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3580.060606] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3580.060608] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3580.060611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3580.060613] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3580.060619] Call Trace: [ 3580.060623] [ 3580.060626] ? show_regs.cold.9+0x22/0x2f [ 3580.060633] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3580.060639] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3580.060643] ? watchdog+0x40/0x40 [ 3580.060648] ? hrtimer_interrupt+0x128/0x2c0 [ 3580.060652] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3580.060658] ? apic_timer_interrupt+0xf/0x20 [ 3580.060663] [ 3580.060666] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3580.060671] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3580.060677] do_raw_spin_lock+0x177/0x190 [ 3580.060682] _raw_spin_lock+0x21/0x30 [ 3580.060890] obd_mod_free+0x46/0x140 [ptlrpc] [ 3580.060685] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3580.061679] mdc_commit_open+0x91/0xb0 [mdc] [ 3580.061853] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3580.062105] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3580.062300] after_reply+0xe2c/0x1780 [ptlrpc] [ 3580.062496] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3580.062688] ? __next_timer_interrupt+0x160/0x160 [ 3580.062698] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3580.062908] ? do_wait_intr+0xf0/0xf0 [ 3580.062917] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3580.063166] kthread+0x1d1/0x200 [ 3580.063172] ? set_kthread_struct+0x70/0x70 [ 3580.063176] ret_from_fork+0x1f/0x30 [ 3580.063151] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3580.063541] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3580.063759] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3580.064144] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3580.064681] ? woken_wake_function+0x30/0x30 [ 3580.064689] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3580.065101] kthread+0x1d1/0x200 [ 3580.065108] ? set_kthread_struct+0x70/0x70 [ 3580.065112] ret_from_fork+0x1f/0x30 [ 3580.066010] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3580.066077] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3580.066084] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3580.066090] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3580.066361] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3580.066371] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3580.066375] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3580.066380] RAX: 0000000000007584 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3580.066382] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3580.066384] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3580.066386] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3580.066388] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3580.066391] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3580.066393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3580.066395] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3580.066404] Call Trace: [ 3580.066409] [ 3580.066412] ? show_regs.cold.9+0x22/0x2f [ 3580.066422] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3580.066428] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3580.066453] ? watchdog+0x40/0x40 [ 3580.066458] ? hrtimer_interrupt+0x128/0x2c0 [ 3580.066462] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3580.066469] ? apic_timer_interrupt+0xf/0x20 [ 3580.066473] [ 3580.066475] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3580.066479] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3580.066483] ? newidle_balance.isra.93+0x207/0x580 [ 3580.066489] do_raw_spin_lock+0x177/0x190 [ 3580.066493] _raw_spin_lock+0x21/0x30 [ 3580.066497] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3580.066710] process_one_work+0x2c8/0x700 [ 3580.066717] worker_thread+0x296/0x6e0 [ 3580.066721] ? rescuer_thread+0x570/0x570 [ 3580.066724] kthread+0x1d1/0x200 [ 3580.066728] ? set_kthread_struct+0x70/0x70 [ 3580.066731] ret_from_fork+0x1f/0x30 [ 3580.806947] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3580.833292] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3580.844289] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3580.856389] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3580.858358] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3580.883430] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3580.895067] RAX: 0000000000004b84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3580.899604] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3580.902730] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3580.905671] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3580.912685] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3580.924142] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3580.926760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3580.935375] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3580.945291] Call Trace: [ 3580.946220] [ 3580.951246] ? show_regs.cold.9+0x22/0x2f [ 3580.954175] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3580.955540] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3580.960003] ? watchdog+0x40/0x40 [ 3580.961436] ? hrtimer_interrupt+0x128/0x2c0 [ 3580.965545] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3580.971600] ? apic_timer_interrupt+0xf/0x20 [ 3580.977299] [ 3580.980054] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3580.982260] do_raw_spin_lock+0x177/0x190 [ 3580.984665] _raw_spin_lock+0x21/0x30 [ 3580.986683] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3580.989523] ? do_raw_spin_unlock+0x75/0x190 [ 3580.991209] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3580.993039] ? do_wait_intr+0xf0/0xf0 [ 3580.995885] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3580.997928] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3581.005227] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3581.008222] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3581.013324] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3581.018119] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3581.020210] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3581.027159] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3581.030421] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3581.034406] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3581.038682] lmv_intent_open+0x338/0x1150 [lmv] [ 3581.042375] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3581.045454] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3581.050300] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3581.053932] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3581.055666] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3581.058905] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3581.061310] ? do_raw_spin_unlock+0x75/0x190 [ 3581.067240] ? _raw_spin_unlock+0x12/0x30 [ 3581.069888] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3581.075550] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3581.081322] ll_atomic_open+0x36a/0xd70 [lustre] [ 3581.085011] lookup_open+0x356/0xb40 [ 3581.087015] do_last+0x313/0xfc0 [ 3581.088512] ? nd_jump_root+0xe5/0x160 [ 3581.089955] ? path_init+0x437/0x520 [ 3581.092171] path_openat+0xf7/0x500 [ 3581.093426] ? do_raw_spin_unlock+0x75/0x190 [ 3581.095218] do_filp_open+0x99/0x140 [ 3581.096684] ? getname_flags+0x6e/0x330 [ 3581.098959] ? __check_object_size+0xff/0x256 [ 3581.100907] ? do_raw_spin_unlock+0x75/0x190 [ 3581.102973] ? _raw_spin_unlock+0x12/0x30 [ 3581.104609] do_sys_openat2+0x2b4/0x410 [ 3581.106722] do_sys_open+0x73/0xa0 [ 3581.110028] __x64_sys_openat+0x24/0x30 [ 3581.111396] do_syscall_64+0xc1/0x440 [ 3581.114031] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3581.118542] RIP: 0033:0x7f53425a5332 [ 3581.122117] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3581.138539] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3581.142348] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3581.146075] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3581.149719] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3581.152972] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3581.156221] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3582.434808] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 945s! [ 3582.442613] Showing busy workqueues and worker pools: [ 3582.454571] workqueue events: flags=0x0 [ 3582.463096] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3582.472099] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3582.487972] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3582.490320] pending: cpu_latency_work [ptlrpc] [ 3582.501451] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3582.513356] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3582.523131] workqueue events_long: flags=0x0 [ 3582.533132] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3582.547018] pending: lu_objects_flush [obdclass] [ 3582.555544] workqueue events_unbound: flags=0x2 [ 3582.562672] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3582.572141] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3582.574239] workqueue events_power_efficient: flags=0x80 [ 3582.588653] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3582.612486] pending: check_lifetime, do_cache_clean [ 3582.624600] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3582.630809] pending: neigh_periodic_work, neigh_periodic_work [ 3582.634494] workqueue mm_percpu_wq: flags=0x8 [ 3582.637970] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3582.643274] pending: vmstat_update, lru_add_drain_per_cpu [ 3582.648354] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3582.654218] pending: vmstat_update, lru_add_drain_per_cpu [ 3582.658555] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3582.668440] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3582.682337] workqueue xprtiod: flags=0xa [ 3582.683742] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3582.694396] pending: xprt_autoclose, xs_error_handle [ 3582.703206] workqueue ipv6_addrconf: flags=0x40008 [ 3582.707628] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3582.717302] pending: addrconf_verify_work [ 3582.719250] workqueue knbd0-recv: flags=0x1a [ 3582.722135] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3582.726489] in-flight: 651:recv_work [ 3582.730287] workqueue ptlrpc_pinger: flags=0x2 [ 3582.733953] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3582.736084] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3582.741616] pool 8: cpus=0-3 flags=0x4 nice=0 hung=945s workers=4 idle: 15979 31824 8 [ 3582.748207] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3573s workers=2 idle: 654 [ 3594.462406] rcu: INFO: rcu_sched self-detected stall on CPU [ 3594.467999] rcu: 2-....: (798341 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 3594.473345] (t=1500036 jiffies g=274185 q=64016) [ 3594.476536] rcu: rcu_sched kthread starved for 958402 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 3594.482574] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3594.486872] rcu: RCU grace-period kthread stack dump: [ 3594.491881] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3594.503819] Call Trace: [ 3594.504947] __schedule+0x351/0xcb0 [ 3594.507345] schedule+0xc0/0x180 [ 3594.508239] schedule_timeout+0xb4/0x190 [ 3594.509884] ? do_raw_spin_unlock+0x75/0x190 [ 3594.515019] ? __next_timer_interrupt+0x160/0x160 [ 3594.520844] ? prepare_to_swait_event+0xf3/0x140 [ 3594.526004] rcu_gp_kthread+0x8dd/0x15b0 [ 3594.527897] ? force_qs_rnp+0x2c0/0x2c0 [ 3594.531439] kthread+0x1d1/0x200 [ 3594.532607] ? set_kthread_struct+0x70/0x70 [ 3594.540787] ret_from_fork+0x1f/0x30 [ 3594.542377] rcu: Stack dump where RCU GP kthread last ran: [ 3594.544082] Sending NMI from CPU 2 to CPUs 3: [ 3594.548938] NMI backtrace for cpu 3 [ 3594.548944] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3594.548946] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3594.548948] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3594.548949] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3594.548952] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 [ 3594.548954] RAX: 0000000000000c56 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3594.548956] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3594.548957] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3594.548958] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3594.548960] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3594.548961] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3594.548962] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3594.548963] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3594.548965] Call Trace: [ 3594.548966] [ 3594.548966] ? show_regs.cold.9+0x22/0x2f [ 3594.548968] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3594.548969] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3594.548970] ? nmi_handle+0xbd/0x210 [ 3594.548971] ? default_do_nmi+0x4b/0x1b0 [ 3594.548972] ? do_nmi+0x280/0x380 [ 3594.548973] ? end_repeat_nmi+0x16/0x1f [ 3594.548974] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3594.548975] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3594.548976] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3594.548977] [ 3594.548978] do_raw_spin_lock+0x177/0x190 [ 3594.548979] _raw_spin_lock+0x21/0x30 [ 3594.548980] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3594.548981] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3594.548982] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3594.548983] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3594.548984] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3594.548985] ? woken_wake_function+0x30/0x30 [ 3594.548986] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3594.548987] kthread+0x1d1/0x200 [ 3594.548988] ? set_kthread_struct+0x70/0x70 [ 3594.548989] ret_from_fork+0x1f/0x30 [ 3594.549071] Sending NMI from CPU 2 to CPUs 1: [ 3594.748490] NMI backtrace for cpu 1 [ 3594.748498] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3594.748500] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3594.748502] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3594.748503] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3594.748506] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [ 3594.748509] RAX: 0000000000000459 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3594.748510] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3594.748512] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3594.748513] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3594.748515] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3594.748516] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3594.748517] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3594.748519] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3594.748520] Call Trace: [ 3594.748520] [ 3594.748522] ? show_regs.cold.9+0x22/0x2f [ 3594.748523] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3594.748524] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3594.748525] ? nmi_handle+0xbd/0x210 [ 3594.748526] ? default_do_nmi+0x4b/0x1b0 [ 3594.748527] ? do_nmi+0x280/0x380 [ 3594.748528] ? end_repeat_nmi+0x16/0x1f [ 3594.748529] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3594.748530] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3594.748531] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3594.748532] [ 3594.748533] do_raw_spin_lock+0x177/0x190 [ 3594.748534] _raw_spin_lock+0x21/0x30 [ 3594.748535] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3594.748536] ? do_raw_spin_unlock+0x75/0x190 [ 3594.748537] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3594.748538] ? do_wait_intr+0xf0/0xf0 [ 3594.748539] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3594.748550] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3594.748551] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3594.748552] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3594.748554] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3594.748555] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3594.748556] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3594.748557] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3594.748558] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3594.748559] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3594.748560] lmv_intent_open+0x338/0x1150 [lmv] [ 3594.748561] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3594.748562] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3594.748563] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3594.748564] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3594.748564] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3594.748565] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3594.748566] ? do_raw_spin_unlock+0x75/0x190 [ 3594.748568] ? _raw_spin_unlock+0x12/0x30 [ 3594.748568] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3594.748570] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3594.748571] ll_atomic_open+0x36a/0xd70 [lustre] [ 3594.748571] lookup_open+0x356/0xb40 [ 3594.748572] do_last+0x313/0xfc0 [ 3594.748573] ? nd_jump_root+0xe5/0x160 [ 3594.748575] ? path_init+0x437/0x520 [ 3594.748576] path_openat+0xf7/0x500 [ 3594.748577] ? do_raw_spin_unlock+0x75/0x190 [ 3594.748577] do_filp_open+0x99/0x140 [ 3594.748578] ? getname_flags+0x6e/0x330 [ 3594.749001] ? __check_object_size+0xff/0x256 [ 3594.749003] ? do_raw_spin_unlock+0x75/0x190 [ 3594.749004] ? _raw_spin_unlock+0x12/0x30 [ 3594.749005] do_sys_openat2+0x2b4/0x410 [ 3594.749006] do_sys_open+0x73/0xa0 [ 3594.749007] __x64_sys_openat+0x24/0x30 [ 3594.749008] do_syscall_64+0xc1/0x440 [ 3594.749009] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3594.749010] RIP: 0033:0x7f53425a5332 [ 3594.749012] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3594.749014] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3594.749017] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3594.749018] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3594.749020] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3594.749021] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3594.749023] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3594.750135] NMI backtrace for cpu 2 [ 3595.117101] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3595.128364] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3595.136808] Call Trace: [ 3595.137766] [ 3595.138491] ? dump_stack+0xbb/0x10e [ 3595.141157] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3595.148628] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3595.152375] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3595.154850] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3595.157504] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3595.160887] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3595.163044] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3595.165487] ? update_process_times+0x6d/0xc0 [ 3595.168043] ? tick_sched_handle.isra.18+0x27/0x90 [ 3595.174742] ? tick_sched_timer+0x53/0xb0 [ 3595.177293] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3595.179644] ? tick_sched_handle.isra.18+0x90/0x90 [ 3595.182597] ? hrtimer_interrupt+0x128/0x2c0 [ 3595.185840] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3595.190181] ? apic_timer_interrupt+0xf/0x20 [ 3595.192834] [ 3595.195461] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3595.198811] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3595.208033] ? do_raw_spin_lock+0x177/0x190 [ 3595.211706] ? _raw_spin_lock+0x21/0x30 [ 3595.213962] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3595.217657] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3595.222144] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3595.224656] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3595.230494] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3595.235848] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3595.238803] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3595.242904] ? __next_timer_interrupt+0x160/0x160 [ 3595.246747] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3595.250103] ? do_wait_intr+0xf0/0xf0 [ 3595.251863] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3595.256776] ? kthread+0x1d1/0x200 [ 3595.259899] ? set_kthread_struct+0x70/0x70 [ 3595.264054] ? ret_from_fork+0x1f/0x30 [ 3608.053980] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 3608.066789] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3608.111204] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3608.124054] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3608.133913] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3608.136305] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3608.147916] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3608.167602] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3608.176570] RAX: 0000000000007986 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3608.183692] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3608.190813] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3608.194292] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3608.204733] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3608.209001] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3608.216817] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3608.220989] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3608.226768] Call Trace: [ 3608.228636] [ 3608.229717] ? show_regs.cold.9+0x22/0x2f [ 3608.233609] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3608.236429] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3608.238957] ? watchdog+0x40/0x40 [ 3608.240101] ? hrtimer_interrupt+0x128/0x2c0 [ 3608.243131] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3608.246574] ? apic_timer_interrupt+0xf/0x20 [ 3608.248933] [ 3608.252424] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3608.259717] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3608.263642] ? newidle_balance.isra.93+0x207/0x580 [ 3608.267929] do_raw_spin_lock+0x177/0x190 [ 3608.271839] _raw_spin_lock+0x21/0x30 [ 3608.273823] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3608.276682] process_one_work+0x2c8/0x700 [ 3608.281078] worker_thread+0x296/0x6e0 [ 3608.283269] ? rescuer_thread+0x570/0x570 [ 3608.287048] kthread+0x1d1/0x200 [ 3608.289225] ? set_kthread_struct+0x70/0x70 [ 3608.294832] ret_from_fork+0x1f/0x30 [ 3613.158367] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 976s! [ 3613.165405] Showing busy workqueues and worker pools: [ 3613.171094] workqueue events: flags=0x0 [ 3613.172867] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3613.179320] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3613.192988] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3613.202910] pending: cpu_latency_work [ptlrpc] [ 3613.206924] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3613.213786] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3613.221019] workqueue events_long: flags=0x0 [ 3613.225449] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3613.232462] pending: lu_objects_flush [obdclass] [ 3613.238128] workqueue events_unbound: flags=0x2 [ 3613.249211] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3613.256011] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3613.260885] workqueue events_power_efficient: flags=0x80 [ 3613.271145] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3613.277327] pending: check_lifetime, do_cache_clean [ 3613.283401] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3613.288898] pending: neigh_periodic_work, neigh_periodic_work [ 3613.291911] workqueue mm_percpu_wq: flags=0x8 [ 3613.293786] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3613.296842] pending: vmstat_update, lru_add_drain_per_cpu [ 3613.299671] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3613.302357] pending: vmstat_update, lru_add_drain_per_cpu [ 3613.305164] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3613.317719] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3613.330040] workqueue xprtiod: flags=0xa [ 3613.336706] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3613.345420] pending: xprt_autoclose, xs_error_handle [ 3613.354309] workqueue ipv6_addrconf: flags=0x40008 [ 3613.360989] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3613.369280] pending: addrconf_verify_work [ 3613.374072] workqueue knbd0-recv: flags=0x1a [ 3613.379591] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3613.386440] in-flight: 651:recv_work [ 3613.390729] workqueue ptlrpc_pinger: flags=0x2 [ 3613.393624] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3613.400262] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3613.411550] pool 8: cpus=0-3 flags=0x4 nice=0 hung=976s workers=4 idle: 15979 31824 8 [ 3613.422249] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3603s workers=2 idle: 654 [ 3620.057007] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 3620.057140] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 3620.058023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 3620.058030] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3620.058129] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3620.058134] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3620.058137] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3620.058150] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3620.058155] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3620.058160] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3620.058163] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3620.058165] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3620.058168] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3620.058171] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3620.058174] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3620.058177] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3620.058180] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3620.058197] Call Trace: [ 3620.058203] [ 3620.058209] ? show_regs.cold.9+0x22/0x2f [ 3620.058219] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3620.058226] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3620.058231] ? watchdog+0x40/0x40 [ 3620.058236] ? hrtimer_interrupt+0x128/0x2c0 [ 3620.058240] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3620.058247] ? apic_timer_interrupt+0xf/0x20 [ 3620.058252] [ 3620.058255] ? native_safe_halt+0x2e/0x50 [ 3620.058259] kvm_wait+0xcd/0xe0 [ 3620.058266] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3620.058274] do_raw_spin_lock+0x177/0x190 [ 3620.058279] _raw_spin_lock+0x21/0x30 [ 3620.058284] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3620.058624] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3620.058841] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3620.059078] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3620.059326] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3620.060158] ? woken_wake_function+0x30/0x30 [ 3620.060168] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3620.060960] kthread+0x1d1/0x200 [ 3620.060968] ? set_kthread_struct+0x70/0x70 [ 3620.060972] ret_from_fork+0x1f/0x30 [ 3620.061596] Modules linked in: lustre(O) [ 3620.067801] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3620.067880] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3620.067887] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3620.067889] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3620.067903] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3620.067907] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3620.067912] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3620.067914] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3620.067916] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3620.067918] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3620.067920] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3620.067922] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3620.067925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3620.067927] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3620.067937] Call Trace: [ 3620.067943] [ 3620.067949] ? show_regs.cold.9+0x22/0x2f [ 3620.067958] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3620.067964] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3620.067967] ? watchdog+0x40/0x40 [ 3620.067971] ? hrtimer_interrupt+0x128/0x2c0 [ 3620.067974] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3620.067981] ? apic_timer_interrupt+0xf/0x20 [ 3620.067985] [ 3620.067988] ? native_safe_halt+0x2e/0x50 [ 3620.067992] kvm_wait+0xcd/0xe0 [ 3620.067999] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3620.068008] do_raw_spin_lock+0x177/0x190 [ 3620.076013] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3620.076287] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3620.076295] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3620.076297] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3620.076311] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3620.076315] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3620.076319] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3620.076322] RDX: 000000000000562a RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3620.076324] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3620.076326] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3620.076329] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3620.076332] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3620.076335] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3620.076338] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3620.076346] Call Trace: [ 3620.076351] [ 3620.076357] ? show_regs.cold.9+0x22/0x2f [ 3620.076366] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3620.076373] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3620.076378] ? watchdog+0x40/0x40 [ 3620.076382] ? hrtimer_interrupt+0x128/0x2c0 [ 3620.076387] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3620.076394] ? apic_timer_interrupt+0xf/0x20 [ 3620.076398] [ 3620.076401] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3620.076405] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3620.076410] do_raw_spin_lock+0x177/0x190 [ 3620.076414] _raw_spin_lock+0x21/0x30 [ 3620.076421] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3620.076691] obd_mod_free+0x46/0x140 [ptlrpc] [ 3620.078490] mdc_commit_open+0x91/0xb0 [mdc] [ 3620.078535] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3620.078855] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3620.079286] after_reply+0xe2c/0x1780 [ptlrpc] [ 3620.079592] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3620.079866] ? __next_timer_interrupt+0x160/0x160 [ 3620.079876] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3620.080177] ? do_wait_intr+0xf0/0xf0 [ 3620.080187] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3620.080478] kthread+0x1d1/0x200 [ 3620.080485] ? set_kthread_struct+0x70/0x70 [ 3620.080488] ret_from_fork+0x1f/0x30 [ 3620.594285] _raw_spin_lock+0x21/0x30 [ 3620.598956] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3620.607848] ? do_raw_spin_unlock+0x75/0x190 [ 3620.611755] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3620.615739] ? do_wait_intr+0xf0/0xf0 [ 3620.617086] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3620.620761] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3620.623579] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3620.626909] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3620.632002] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3620.636607] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3620.642941] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3620.645951] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3620.649450] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3620.655433] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3620.657953] lmv_intent_open+0x338/0x1150 [lmv] [ 3620.665505] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3620.669378] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3620.673881] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3620.676185] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3620.677771] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3620.687700] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3620.691046] ? do_raw_spin_unlock+0x75/0x190 [ 3620.693163] ? _raw_spin_unlock+0x12/0x30 [ 3620.699427] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3620.702487] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3620.705272] ll_atomic_open+0x36a/0xd70 [lustre] [ 3620.707558] lookup_open+0x356/0xb40 [ 3620.709358] do_last+0x313/0xfc0 [ 3620.710438] ? nd_jump_root+0xe5/0x160 [ 3620.712461] ? path_init+0x437/0x520 [ 3620.714247] path_openat+0xf7/0x500 [ 3620.717718] ? do_raw_spin_unlock+0x75/0x190 [ 3620.721811] do_filp_open+0x99/0x140 [ 3620.724578] ? getname_flags+0x6e/0x330 [ 3620.726321] ? __check_object_size+0xff/0x256 [ 3620.729461] ? do_raw_spin_unlock+0x75/0x190 [ 3620.731792] ? _raw_spin_unlock+0x12/0x30 [ 3620.733942] do_sys_openat2+0x2b4/0x410 [ 3620.736630] do_sys_open+0x73/0xa0 [ 3620.747103] __x64_sys_openat+0x24/0x30 [ 3620.748666] do_syscall_64+0xc1/0x440 [ 3620.750912] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3620.752901] RIP: 0033:0x7f53425a5332 [ 3620.754948] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3620.766478] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3620.772641] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3620.775859] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3620.783401] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3620.786603] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3620.789950] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3640.052804] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 3640.058958] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3640.104715] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3640.114724] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3640.122545] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3640.126529] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3640.132733] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3640.151326] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3640.155221] RAX: 0000000000004507 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3640.158334] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3640.162202] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3640.166584] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3640.170559] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3640.178040] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3640.185091] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3640.187437] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3640.192686] Call Trace: [ 3640.194629] [ 3640.196420] ? show_regs.cold.9+0x22/0x2f [ 3640.199887] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3640.202533] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3640.204149] ? watchdog+0x40/0x40 [ 3640.210500] ? hrtimer_interrupt+0x128/0x2c0 [ 3640.214323] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3640.218484] ? apic_timer_interrupt+0xf/0x20 [ 3640.221225] [ 3640.223631] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3640.228709] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3640.234325] ? newidle_balance.isra.93+0x207/0x580 [ 3640.238726] do_raw_spin_lock+0x177/0x190 [ 3640.244878] _raw_spin_lock+0x21/0x30 [ 3640.247464] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3640.251793] process_one_work+0x2c8/0x700 [ 3640.253684] worker_thread+0x296/0x6e0 [ 3640.255094] ? rescuer_thread+0x570/0x570 [ 3640.256892] kthread+0x1d1/0x200 [ 3640.258160] ? set_kthread_struct+0x70/0x70 [ 3640.259985] ret_from_fork+0x1f/0x30 [ 3643.874283] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1006s! [ 3643.878534] Showing busy workqueues and worker pools: [ 3643.881422] workqueue events: flags=0x0 [ 3643.884519] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3643.887666] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3643.895475] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3643.897776] pending: cpu_latency_work [ptlrpc] [ 3643.899409] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3643.903942] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3643.906933] workqueue events_long: flags=0x0 [ 3643.908502] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3643.913103] pending: lu_objects_flush [obdclass] [ 3643.919562] workqueue events_unbound: flags=0x2 [ 3643.921272] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3643.924900] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3643.927419] workqueue events_power_efficient: flags=0x80 [ 3643.930651] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3643.939596] pending: check_lifetime, do_cache_clean [ 3643.941880] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3643.946514] pending: neigh_periodic_work, neigh_periodic_work [ 3643.949825] workqueue mm_percpu_wq: flags=0x8 [ 3643.952597] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3643.955591] pending: vmstat_update, lru_add_drain_per_cpu [ 3643.958120] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3643.961037] pending: vmstat_update, lru_add_drain_per_cpu [ 3643.964036] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3643.967634] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3643.970730] workqueue xprtiod: flags=0xa [ 3643.979622] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3643.985111] pending: xprt_autoclose, xs_error_handle [ 3643.988080] workqueue ipv6_addrconf: flags=0x40008 [ 3643.990045] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3643.992433] pending: addrconf_verify_work [ 3643.994329] workqueue knbd0-recv: flags=0x1a [ 3643.998181] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3644.000502] in-flight: 651:recv_work [ 3644.006644] workqueue ptlrpc_pinger: flags=0x2 [ 3644.008619] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3644.020033] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3644.027883] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1006s workers=4 idle: 15979 31824 8 [ 3644.036374] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3634s workers=2 idle: 654 [ 3648.056174] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 3648.057057] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 3648.060353] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 3648.060366] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3648.060468] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3648.060474] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3648.060478] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3648.060500] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3648.060505] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3648.060510] RAX: 0000000000007383 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3648.060513] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3648.060515] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3648.060517] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3648.060520] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3648.060523] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3648.060526] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3648.060528] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3648.060536] Call Trace: [ 3648.060541] [ 3648.060547] ? show_regs.cold.9+0x22/0x2f [ 3648.060557] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3648.060563] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3648.060568] ? watchdog+0x40/0x40 [ 3648.060572] ? hrtimer_interrupt+0x128/0x2c0 [ 3648.060577] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3648.060663] ? apic_timer_interrupt+0xf/0x20 [ 3648.060668] [ 3648.060671] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3648.060676] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3648.060683] do_raw_spin_lock+0x177/0x190 [ 3648.060688] _raw_spin_lock+0x21/0x30 [ 3648.061190] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [ 3648.060693] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3648.061473] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3648.061700] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3648.062051] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3648.062419] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3648.062662] ? woken_wake_function+0x30/0x30 [ 3648.062672] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3648.063016] kthread+0x1d1/0x200 [ 3648.063022] ? set_kthread_struct+0x70/0x70 [ 3648.063027] ret_from_fork+0x1f/0x30 [ 3648.065135] Modules linked in: [ 3648.068520] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix [ 3648.119061] lustre(O) [ 3648.122904] serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3648.122926] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3648.122933] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3648.122936] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3648.122949] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3648.122954] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3648.122959] RAX: 0000000000004935 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3648.122961] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3648.122963] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3648.122965] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3648.122967] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3648.122970] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3648.122973] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3648.122975] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3648.122983] Call Trace: [ 3648.122989] [ 3648.122995] ? show_regs.cold.9+0x22/0x2f [ 3648.123004] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3648.123010] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3648.123014] ? watchdog+0x40/0x40 [ 3648.123018] ? hrtimer_interrupt+0x128/0x2c0 [ 3648.123021] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3648.123029] ? apic_timer_interrupt+0xf/0x20 [ 3648.123033] [ 3648.123036] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3648.123043] do_raw_spin_lock+0x177/0x190 [ 3648.123047] _raw_spin_lock+0x21/0x30 [ 3648.123051] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3648.123326] ? do_raw_spin_unlock+0x75/0x190 [ 3648.123332] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3648.123546] ? do_wait_intr+0xf0/0xf0 [ 3648.123556] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3648.123936] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3648.124208] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3648.124415] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3648.124590] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3648.124695] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3648.124725] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3648.124906] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3648.124993] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3648.125213] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3648.125240] lmv_intent_open+0x338/0x1150 [lmv] [ 3648.125270] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3648.125402] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3648.125487] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3648.125527] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3648.125740] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3648.125901] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3648.125987] ? do_raw_spin_unlock+0x75/0x190 [ 3648.125995] ? _raw_spin_unlock+0x12/0x30 [ 3648.126000] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3648.126026] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3648.126047] ll_atomic_open+0x36a/0xd70 [lustre] [ 3648.126137] lookup_open+0x356/0xb40 [ 3648.126145] do_last+0x313/0xfc0 [ 3648.126150] ? nd_jump_root+0xe5/0x160 [ 3648.126154] ? path_init+0x437/0x520 [ 3648.126159] path_openat+0xf7/0x500 [ 3648.126164] ? do_raw_spin_unlock+0x75/0x190 [ 3648.126168] do_filp_open+0x99/0x140 [ 3648.126175] ? getname_flags+0x6e/0x330 [ 3648.126180] ? __check_object_size+0xff/0x256 [ 3648.126187] ? do_raw_spin_unlock+0x75/0x190 [ 3648.126191] ? _raw_spin_unlock+0x12/0x30 [ 3648.126195] do_sys_openat2+0x2b4/0x410 [ 3648.126201] do_sys_open+0x73/0xa0 [ 3648.126206] __x64_sys_openat+0x24/0x30 [ 3648.126209] do_syscall_64+0xc1/0x440 [ 3648.126216] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3648.126222] RIP: 0033:0x7f53425a5332 [ 3648.126230] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3648.126234] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3648.126239] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3648.126242] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3648.126244] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3648.126247] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3648.126249] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3648.540687] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3648.575130] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3648.583696] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3648.591479] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3648.598501] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3648.619555] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3648.624141] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3648.628084] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3648.632313] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3648.637240] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3648.643210] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3648.650820] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3648.658427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3648.663231] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3648.669820] Call Trace: [ 3648.672881] [ 3648.674192] ? show_regs.cold.9+0x22/0x2f [ 3648.677226] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3648.680224] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3648.681769] ? watchdog+0x40/0x40 [ 3648.683235] ? hrtimer_interrupt+0x128/0x2c0 [ 3648.685118] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3648.687201] ? apic_timer_interrupt+0xf/0x20 [ 3648.690056] [ 3648.691032] ? native_safe_halt+0x2e/0x50 [ 3648.692835] kvm_wait+0xcd/0xe0 [ 3648.695214] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3648.697329] do_raw_spin_lock+0x177/0x190 [ 3648.699018] _raw_spin_lock+0x21/0x30 [ 3648.700452] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3648.704941] obd_mod_free+0x46/0x140 [ptlrpc] [ 3648.707098] mdc_commit_open+0x91/0xb0 [mdc] [ 3648.710545] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3648.718580] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3648.721373] after_reply+0xe2c/0x1780 [ptlrpc] [ 3648.726350] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3648.730990] ? __next_timer_interrupt+0x160/0x160 [ 3648.737856] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3648.743351] ? do_wait_intr+0xf0/0xf0 [ 3648.746325] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3648.749300] kthread+0x1d1/0x200 [ 3648.751283] ? set_kthread_struct+0x70/0x70 [ 3648.753577] ret_from_fork+0x1f/0x30 [ 3668.055986] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 3668.060187] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3668.096595] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3668.103354] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3668.107573] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3668.110045] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3668.118649] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3668.136181] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3668.140319] RAX: 0000000000000292 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3668.146104] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3668.149141] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3668.156769] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3668.159625] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3668.162632] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3668.168111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3668.172788] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3668.176630] Call Trace: [ 3668.178709] [ 3668.180246] ? show_regs.cold.9+0x22/0x2f [ 3668.182684] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3668.184747] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3668.188600] ? watchdog+0x40/0x40 [ 3668.189761] ? hrtimer_interrupt+0x128/0x2c0 [ 3668.190964] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3668.194565] ? apic_timer_interrupt+0xf/0x20 [ 3668.197992] [ 3668.198994] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3668.203777] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3668.206040] ? newidle_balance.isra.93+0x207/0x580 [ 3668.208512] do_raw_spin_lock+0x177/0x190 [ 3668.209905] _raw_spin_lock+0x21/0x30 [ 3668.211613] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3668.213639] process_one_work+0x2c8/0x700 [ 3668.215687] worker_thread+0x296/0x6e0 [ 3668.218690] ? rescuer_thread+0x570/0x570 [ 3668.222815] kthread+0x1d1/0x200 [ 3668.224103] ? set_kthread_struct+0x70/0x70 [ 3668.229084] ret_from_fork+0x1f/0x30 [ 3674.083282] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1036s! [ 3674.089365] Showing busy workqueues and worker pools: [ 3674.091831] workqueue events: flags=0x0 [ 3674.093086] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3674.098502] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3674.104531] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3674.107229] pending: cpu_latency_work [ptlrpc] [ 3674.110031] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3674.112508] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3674.115104] workqueue events_long: flags=0x0 [ 3674.117362] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3674.120647] pending: lu_objects_flush [obdclass] [ 3674.122654] workqueue events_unbound: flags=0x2 [ 3674.124502] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3674.126849] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3674.129738] workqueue events_power_efficient: flags=0x80 [ 3674.132793] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3674.135701] pending: check_lifetime, do_cache_clean [ 3674.138066] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3674.140827] pending: neigh_periodic_work, neigh_periodic_work [ 3674.143851] workqueue mm_percpu_wq: flags=0x8 [ 3674.146365] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3674.149629] pending: vmstat_update, lru_add_drain_per_cpu [ 3674.152597] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3674.155413] pending: vmstat_update, lru_add_drain_per_cpu [ 3674.158385] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3674.161200] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3674.168639] workqueue xprtiod: flags=0xa [ 3674.171684] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3674.177622] pending: xprt_autoclose, xs_error_handle [ 3674.182061] workqueue ipv6_addrconf: flags=0x40008 [ 3674.184301] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3674.187755] pending: addrconf_verify_work [ 3674.190452] workqueue knbd0-recv: flags=0x1a [ 3674.192442] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3674.195730] in-flight: 651:recv_work [ 3674.197752] workqueue ptlrpc_pinger: flags=0x2 [ 3674.200264] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3674.203252] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3674.207174] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1037s workers=4 idle: 15979 31824 8 [ 3674.212788] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3664s workers=2 idle: 654 [ 3676.057016] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 3676.057793] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 3676.058885] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 3676.058897] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3676.058967] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3676.058973] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3676.058975] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3676.058995] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3676.059000] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3676.059005] RAX: 0000000000005c09 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3676.059008] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3676.059009] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3676.059011] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3676.059014] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3676.059017] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3676.059020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3676.059021] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3676.059030] Call Trace: [ 3676.059037] [ 3676.059042] ? show_regs.cold.9+0x22/0x2f [ 3676.059052] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3676.059058] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3676.059062] ? watchdog+0x40/0x40 [ 3676.059066] ? hrtimer_interrupt+0x128/0x2c0 [ 3676.059070] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3676.059079] ? apic_timer_interrupt+0xf/0x20 [ 3676.059084] [ 3676.059087] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3676.059091] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3676.059097] do_raw_spin_lock+0x177/0x190 [ 3676.059103] _raw_spin_lock+0x21/0x30 [ 3676.059108] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3676.061448] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3676.061645] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3676.061877] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3676.062103] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3676.062345] ? woken_wake_function+0x30/0x30 [ 3676.062354] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3676.062571] kthread+0x1d1/0x200 [ 3676.062577] ? set_kthread_struct+0x70/0x70 [ 3676.062581] ret_from_fork+0x1f/0x30 [ 3676.064313] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3676.064389] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3676.064397] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3676.064400] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3676.064413] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3676.064417] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3676.064422] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3676.064424] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3676.064427] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3676.064429] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3676.064432] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3676.064435] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3676.064438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3676.064441] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3676.064463] Call Trace: [ 3676.064469] [ 3676.064473] ? show_regs.cold.9+0x22/0x2f [ 3676.064485] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3676.064492] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3676.064497] ? watchdog+0x40/0x40 [ 3676.064502] ? hrtimer_interrupt+0x128/0x2c0 [ 3676.064506] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3676.064513] ? apic_timer_interrupt+0xf/0x20 [ 3676.064518] [ 3676.064521] ? native_safe_halt+0x2e/0x50 [ 3676.064526] kvm_wait+0xcd/0xe0 [ 3676.064533] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3676.064540] do_raw_spin_lock+0x177/0x190 [ 3676.064545] _raw_spin_lock+0x21/0x30 [ 3676.064550] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3676.064843] obd_mod_free+0x46/0x140 [ptlrpc] [ 3676.065053] mdc_commit_open+0x91/0xb0 [mdc] [ 3676.065091] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3676.065312] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3676.065509] after_reply+0xe2c/0x1780 [ptlrpc] [ 3676.065710] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3676.065899] ? __next_timer_interrupt+0x160/0x160 [ 3676.065909] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3676.066109] ? do_wait_intr+0xf0/0xf0 [ 3676.068888] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3676.075886] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3676.075960] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3676.075968] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3676.075971] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3676.075986] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3676.075990] RSP: 0018:ffffbb5381503500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 3676.075995] RAX: 0000000000002d0e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3676.075997] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3676.076011] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3676.076013] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3676.076015] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3676.076018] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3676.076021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3676.076023] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3676.076031] Call Trace: [ 3676.076037] [ 3676.076043] ? show_regs.cold.9+0x22/0x2f [ 3676.076055] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3676.076061] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3676.076070] ? watchdog+0x40/0x40 [ 3676.076075] ? hrtimer_interrupt+0x128/0x2c0 [ 3676.076079] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3676.076087] ? apic_timer_interrupt+0xf/0x20 [ 3676.076092] [ 3676.076095] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3676.076108] do_raw_spin_lock+0x177/0x190 [ 3676.076114] _raw_spin_lock+0x21/0x30 [ 3676.076129] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3676.076415] ? do_raw_spin_unlock+0x75/0x190 [ 3676.077102] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3676.077342] ? do_wait_intr+0xf0/0xf0 [ 3676.077352] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3676.077563] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3676.077795] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3676.077996] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3676.078130] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3676.078173] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3676.078202] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3676.078230] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3676.078314] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3676.078522] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3676.078551] lmv_intent_open+0x338/0x1150 [lmv] [ 3676.078583] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3676.078667] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3676.078749] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3676.078772] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3676.078854] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3676.078938] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3676.079023] ? do_raw_spin_unlock+0x75/0x190 [ 3676.079030] ? _raw_spin_unlock+0x12/0x30 [ 3676.079037] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3676.079065] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3676.079086] ll_atomic_open+0x36a/0xd70 [lustre] [ 3676.079192] lookup_open+0x356/0xb40 [ 3676.079202] do_last+0x313/0xfc0 [ 3676.079207] ? nd_jump_root+0xe5/0x160 [ 3676.079211] ? path_init+0x437/0x520 [ 3676.079216] path_openat+0xf7/0x500 [ 3676.079220] ? do_raw_spin_unlock+0x75/0x190 [ 3676.079225] do_filp_open+0x99/0x140 [ 3676.079231] ? getname_flags+0x6e/0x330 [ 3676.079236] ? __check_object_size+0xff/0x256 [ 3676.079245] ? do_raw_spin_unlock+0x75/0x190 [ 3676.079249] ? _raw_spin_unlock+0x12/0x30 [ 3676.079253] do_sys_openat2+0x2b4/0x410 [ 3676.079258] do_sys_open+0x73/0xa0 [ 3676.079262] __x64_sys_openat+0x24/0x30 [ 3676.079265] do_syscall_64+0xc1/0x440 [ 3676.079272] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3676.079278] RIP: 0033:0x7f53425a5332 [ 3676.079316] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3676.079322] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3676.079327] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3676.079330] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3676.079332] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3676.079333] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3676.079336] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3676.827584] kthread+0x1d1/0x200 [ 3676.829226] ? set_kthread_struct+0x70/0x70 [ 3676.831564] ret_from_fork+0x1f/0x30 [ 3700.052865] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 3700.059112] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3700.111145] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3700.128376] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3700.136591] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3700.145350] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3700.152453] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3700.172281] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3700.177486] RAX: 0000000000007483 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3700.181855] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3700.187552] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3700.194984] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3700.201049] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3700.206644] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3700.210607] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3700.225167] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3700.232969] Call Trace: [ 3700.234017] [ 3700.235014] ? show_regs.cold.9+0x22/0x2f [ 3700.240434] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3700.245716] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3700.251410] ? watchdog+0x40/0x40 [ 3700.254488] ? hrtimer_interrupt+0x128/0x2c0 [ 3700.260451] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3700.266109] ? apic_timer_interrupt+0xf/0x20 [ 3700.272469] [ 3700.273330] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3700.277278] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3700.283182] ? newidle_balance.isra.93+0x207/0x580 [ 3700.289707] do_raw_spin_lock+0x177/0x190 [ 3700.294890] _raw_spin_lock+0x21/0x30 [ 3700.307347] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3700.313516] process_one_work+0x2c8/0x700 [ 3700.319433] worker_thread+0x296/0x6e0 [ 3700.323418] ? rescuer_thread+0x570/0x570 [ 3700.332934] kthread+0x1d1/0x200 [ 3700.334506] ? set_kthread_struct+0x70/0x70 [ 3700.340407] ret_from_fork+0x1f/0x30 [ 3704.055070] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 3704.057872] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3704.061008] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 3704.061908] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 3704.061917] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3704.061981] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3704.061987] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3704.061989] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3704.062002] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3704.062007] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3704.062011] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3704.062014] RDX: 00000000000070b0 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3704.062016] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3704.062019] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3704.062021] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3704.062024] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3704.062027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3704.062029] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3704.062038] Call Trace: [ 3704.062044] [ 3704.062050] ? show_regs.cold.9+0x22/0x2f [ 3704.062059] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3704.062066] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3704.062071] ? watchdog+0x40/0x40 [ 3704.062076] ? hrtimer_interrupt+0x128/0x2c0 [ 3704.062080] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3704.062089] ? apic_timer_interrupt+0xf/0x20 [ 3704.062094] [ 3704.062097] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3704.062102] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3704.062107] do_raw_spin_lock+0x177/0x190 [ 3704.062111] _raw_spin_lock+0x21/0x30 [ 3704.062117] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3704.062382] obd_mod_free+0x46/0x140 [ptlrpc] [ 3704.062599] mdc_commit_open+0x91/0xb0 [mdc] [ 3704.062644] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3704.064009] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3704.064226] after_reply+0xe2c/0x1780 [ptlrpc] [ 3704.064442] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3704.067930] ? __next_timer_interrupt+0x160/0x160 [ 3704.067941] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3704.068159] ? do_wait_intr+0xf0/0xf0 [ 3704.068169] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3704.068412] kthread+0x1d1/0x200 [ 3704.068418] ? set_kthread_struct+0x70/0x70 [ 3704.068422] ret_from_fork+0x1f/0x30 [ 3704.089167] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3704.089181] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3704.089184] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3704.089198] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3704.089203] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3704.089209] RAX: 000000000000064f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3704.089211] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3704.089213] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3704.089215] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3704.089218] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3704.089221] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3704.089224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3704.089227] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3704.089236] Call Trace: [ 3704.089241] [ 3704.089247] ? show_regs.cold.9+0x22/0x2f [ 3704.089256] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3704.089262] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3704.089266] ? watchdog+0x40/0x40 [ 3704.089270] ? hrtimer_interrupt+0x128/0x2c0 [ 3704.089274] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3704.089282] ? apic_timer_interrupt+0xf/0x20 [ 3704.089286] [ 3704.089288] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3704.089297] do_raw_spin_lock+0x177/0x190 [ 3704.089301] _raw_spin_lock+0x21/0x30 [ 3704.089305] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3704.089571] ? do_raw_spin_unlock+0x75/0x190 [ 3704.089576] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3704.089834] ? do_wait_intr+0xf0/0xf0 [ 3704.089843] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3704.090129] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3704.090533] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3704.090805] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3704.094398] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3704.094482] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3704.094489] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3704.094491] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3704.094505] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3704.094510] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3704.094514] RAX: 0000000000003f12 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3704.094517] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3704.094519] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3704.094521] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3704.094523] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3704.094526] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3704.094528] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3704.094531] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3704.094538] Call Trace: [ 3704.094544] [ 3704.094549] ? show_regs.cold.9+0x22/0x2f [ 3704.094558] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3704.094565] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3704.094569] ? watchdog+0x40/0x40 [ 3704.094573] ? hrtimer_interrupt+0x128/0x2c0 [ 3704.094577] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3704.094585] ? apic_timer_interrupt+0xf/0x20 [ 3704.094589] [ 3704.094592] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3704.094597] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3704.094602] do_raw_spin_lock+0x177/0x190 [ 3704.094608] _raw_spin_lock+0x21/0x30 [ 3704.094613] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3704.095063] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3704.095292] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3704.095549] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3704.096388] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3704.096603] ? woken_wake_function+0x30/0x30 [ 3704.096612] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3704.098815] kthread+0x1d1/0x200 [ 3704.098824] ? set_kthread_struct+0x70/0x70 [ 3704.098829] ret_from_fork+0x1f/0x30 [ 3704.287044] BUG: workqueue lockup - pool [ 3704.291500] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3704.291562] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3704.291592] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3704.291623] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3704.291819] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3704.300818] cpus=0-3 flags=0x4 nice=0 stuck for 1067s! [ 3704.300838] Showing busy workqueues and worker pools: [ 3704.300841] workqueue events: flags=0x0 [ 3704.300844] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3704.300853] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3704.301752] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3704.301766] pending: cpu_latency_work [ptlrpc] [ 3704.302102] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3704.302111] pending: vmstat_shepherd, osc_grant_work_handler [osc] [ 3704.302168] workqueue events_long: flags=0x0 [ 3704.302171] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3704.302178] pending: lu_objects_flush [obdclass] [ 3704.302460] workqueue events_unbound: flags=0x2 [ 3704.302463] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3704.302471] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3704.302488] workqueue events_power_efficient: flags=0x80 [ 3704.302490] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3704.302497] pending: check_lifetime, do_cache_clean [ 3704.302509] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3704.302517] pending: neigh_periodic_work, neigh_periodic_work [ 3704.302529] workqueue mm_percpu_wq: flags=0x8 [ 3704.302532] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3704.302538] pending: vmstat_update, lru_add_drain_per_cpu [ 3704.302604] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3704.302611] pending: vmstat_update, lru_add_drain_per_cpu [ 3704.302619] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3704.302634] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3704.302655] workqueue xprtiod: flags=0xa [ 3704.302657] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3704.302663] pending: xprt_autoclose, xs_error_handle [ 3704.302677] workqueue ipv6_addrconf: flags=0x40008 [ 3704.302680] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3704.302687] pending: addrconf_verify_work [ 3704.302698] workqueue knbd0-recv: flags=0x1a [ 3704.302700] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3704.302706] in-flight: 651:recv_work [ 3704.302715] workqueue ptlrpc_pinger: flags=0x2 [ 3704.302717] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3704.302722] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3704.302958] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1067s workers=4 idle: 15979 31824 8 [ 3704.302971] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3694s workers=2 idle: 654 [ 3705.054867] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3705.056634] lmv_intent_open+0x338/0x1150 [lmv] [ 3705.063472] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3705.067411] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3705.072439] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3705.076149] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3705.078455] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3705.082502] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3705.085244] ? do_raw_spin_unlock+0x75/0x190 [ 3705.087624] ? _raw_spin_unlock+0x12/0x30 [ 3705.095364] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3705.108047] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3705.109819] ll_atomic_open+0x36a/0xd70 [lustre] [ 3705.115641] lookup_open+0x356/0xb40 [ 3705.116809] do_last+0x313/0xfc0 [ 3705.123866] ? nd_jump_root+0xe5/0x160 [ 3705.129312] ? path_init+0x437/0x520 [ 3705.133062] path_openat+0xf7/0x500 [ 3705.137501] ? do_raw_spin_unlock+0x75/0x190 [ 3705.147058] do_filp_open+0x99/0x140 [ 3705.148536] ? getname_flags+0x6e/0x330 [ 3705.153553] ? __check_object_size+0xff/0x256 [ 3705.163679] ? do_raw_spin_unlock+0x75/0x190 [ 3705.167454] ? _raw_spin_unlock+0x12/0x30 [ 3705.170761] do_sys_openat2+0x2b4/0x410 [ 3705.173034] do_sys_open+0x73/0xa0 [ 3705.174934] __x64_sys_openat+0x24/0x30 [ 3705.177735] do_syscall_64+0xc1/0x440 [ 3705.178818] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3705.189774] RIP: 0033:0x7f53425a5332 [ 3705.194767] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3705.221445] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3705.234633] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3705.239309] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3705.253249] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3705.256563] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3705.264481] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3732.052010] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 3732.056731] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3732.058200] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 3732.058269] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 3732.058275] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) [ 3732.058286] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 3732.058288] lfsck(O) [ 3732.058290] Modules linked in: [ 3732.058290] obdecho(O) [ 3732.058291] lustre(O) [ 3732.058292] mgc(O) [ 3732.058293] osp(O) [ 3732.058294] mdc(O) [ 3732.058294] ofd(O) lod(O) [ 3732.058296] lov(O) [ 3732.058297] mdt(O) [ 3732.058297] osc(O) [ 3732.058298] mdd(O) [ 3732.058299] lmv(O) [ 3732.058299] mgs(O) lquota(O) [ 3732.058300] ec(O) [ 3732.058301] lfsck(O) [ 3732.058302] fid(O) [ 3732.058303] obdecho(O) [ 3732.058303] fld(O) [ 3732.058304] mgc(O) [ 3732.058305] ptlrpc_gss(O) [ 3732.058306] mdc(O) lov(O) [ 3732.058307] ptlrpc(O) [ 3732.058308] osc(O) [ 3732.058308] obdclass(O) [ 3732.058309] lmv(O) ec(O) [ 3732.058311] ksocklnd(O) [ 3732.058312] fid(O) [ 3732.058312] lnet(O) [ 3732.058313] fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod [ 3732.058314] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3732.058417] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3732.058420] sha512_ssse3 sha512_generic [ 3732.058423] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3732.058424] [ 3732.058425] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3732.058429] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3732.058430] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3732.058426] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3732.058442] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3732.058442] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3732.058448] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 [ 3732.058447] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3732.058450] ORIG_RAX: ffffffffffffff13 [ 3732.058452] RAX: 0000000000005a13 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3732.058452] RAX: 0000000000003a88 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3732.058455] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3732.058455] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3732.058457] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3732.058457] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3732.058460] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3732.058460] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3732.058463] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3732.058463] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3732.058466] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3732.058466] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3732.058469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3732.058469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3732.058472] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3732.058472] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3732.058480] Call Trace: [ 3732.058480] Call Trace: [ 3732.058485] [ 3732.058485] [ 3732.058491] ? show_regs.cold.9+0x22/0x2f [ 3732.058491] ? show_regs.cold.9+0x22/0x2f [ 3732.058499] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3732.058499] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3732.058505] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3732.058505] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3732.058510] ? watchdog+0x40/0x40 [ 3732.058510] ? watchdog+0x40/0x40 [ 3732.058515] ? hrtimer_interrupt+0x128/0x2c0 [ 3732.058515] ? hrtimer_interrupt+0x128/0x2c0 [ 3732.058519] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3732.058519] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3732.058527] ? apic_timer_interrupt+0xf/0x20 [ 3732.058527] ? apic_timer_interrupt+0xf/0x20 [ 3732.058531] [ 3732.058532] [ 3732.058535] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3732.058535] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3732.058540] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3732.058542] do_raw_spin_lock+0x177/0x190 [ 3732.058545] do_raw_spin_lock+0x177/0x190 [ 3732.058547] _raw_spin_lock+0x21/0x30 [ 3732.058549] _raw_spin_lock+0x21/0x30 [ 3732.058553] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3732.058551] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3732.059188] ? do_raw_spin_unlock+0x75/0x190 [ 3732.059194] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3732.060290] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3732.061441] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3732.062181] ? do_wait_intr+0xf0/0xf0 [ 3732.062194] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3732.062727] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3732.062943] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3732.063169] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3732.061774] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3732.063300] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3732.064228] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3732.064178] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3732.064306] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3732.064453] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3732.064646] ? woken_wake_function+0x30/0x30 [ 3732.064655] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3732.064695] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3732.068550] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3732.068604] lmv_intent_open+0x338/0x1150 [lmv] [ 3732.068644] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3732.068786] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3732.068880] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3732.068907] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3732.069010] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3732.069106] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3732.069218] ? do_raw_spin_unlock+0x75/0x190 [ 3732.069228] ? _raw_spin_unlock+0x12/0x30 [ 3732.069237] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3732.069271] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3732.069292] ll_atomic_open+0x36a/0xd70 [lustre] [ 3732.069449] lookup_open+0x356/0xb40 [ 3732.069460] do_last+0x313/0xfc0 [ 3732.069465] ? nd_jump_root+0xe5/0x160 [ 3732.069470] ? path_init+0x437/0x520 [ 3732.069476] path_openat+0xf7/0x500 [ 3732.069481] ? do_raw_spin_unlock+0x75/0x190 [ 3732.069486] do_filp_open+0x99/0x140 [ 3732.069492] ? getname_flags+0x6e/0x330 [ 3732.069497] ? __check_object_size+0xff/0x256 [ 3732.069506] ? do_raw_spin_unlock+0x75/0x190 [ 3732.069510] ? _raw_spin_unlock+0x12/0x30 [ 3732.069515] do_sys_openat2+0x2b4/0x410 [ 3732.069520] do_sys_open+0x73/0xa0 [ 3732.069523] __x64_sys_openat+0x24/0x30 [ 3732.069526] do_syscall_64+0xc1/0x440 [ 3732.069535] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3732.069542] RIP: 0033:0x7f53425a5332 [ 3732.069551] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3732.069554] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3732.069560] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3732.069562] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3732.069564] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3732.069567] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3732.069569] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3732.069759] kthread+0x1d1/0x200 [ 3732.069766] ? set_kthread_struct+0x70/0x70 [ 3732.069769] ret_from_fork+0x1f/0x30 [ 3732.084567] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3732.087838] Modules linked in: [ 3732.090344] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3732.098390] lustre(O) [ 3732.100899] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3732.101240] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3732.101251] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3732.101256] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3732.101260] RAX: 000000000000171e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3732.101263] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3732.101265] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3732.101267] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3732.101270] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3732.101273] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3732.101276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3732.101278] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3732.101288] Call Trace: [ 3732.101294] [ 3732.101300] ? show_regs.cold.9+0x22/0x2f [ 3732.101309] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3732.101315] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3732.101318] ? watchdog+0x40/0x40 [ 3732.101323] ? hrtimer_interrupt+0x128/0x2c0 [ 3732.101327] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3732.101335] ? apic_timer_interrupt+0xf/0x20 [ 3732.101339] [ 3732.101341] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3732.101346] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3732.101349] ? newidle_balance.isra.93+0x207/0x580 [ 3732.101354] do_raw_spin_lock+0x177/0x190 [ 3732.101359] _raw_spin_lock+0x21/0x30 [ 3732.101363] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3732.101656] process_one_work+0x2c8/0x700 [ 3732.101664] worker_thread+0x296/0x6e0 [ 3732.101669] ? rescuer_thread+0x570/0x570 [ 3732.101672] kthread+0x1d1/0x200 [ 3732.101676] ? set_kthread_struct+0x70/0x70 [ 3732.101680] ret_from_fork+0x1f/0x30 [ 3732.906331] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3732.939915] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3732.959435] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3732.969184] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3732.973888] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3732.993591] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3732.996917] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3733.006040] RDX: 0000000000005dce RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3733.016086] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3733.023625] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3733.030186] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3733.039787] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3733.045245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3733.047966] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3733.055868] Call Trace: [ 3733.062792] [ 3733.063544] ? show_regs.cold.9+0x22/0x2f [ 3733.064974] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3733.068221] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3733.069859] ? watchdog+0x40/0x40 [ 3733.071609] ? hrtimer_interrupt+0x128/0x2c0 [ 3733.075985] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3733.079095] ? apic_timer_interrupt+0xf/0x20 [ 3733.080886] [ 3733.081689] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3733.086754] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3733.091877] do_raw_spin_lock+0x177/0x190 [ 3733.096253] _raw_spin_lock+0x21/0x30 [ 3733.100674] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3733.113087] obd_mod_free+0x46/0x140 [ptlrpc] [ 3733.119385] mdc_commit_open+0x91/0xb0 [mdc] [ 3733.123192] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3733.125913] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3733.129689] after_reply+0xe2c/0x1780 [ptlrpc] [ 3733.134809] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3733.142247] ? __next_timer_interrupt+0x160/0x160 [ 3733.148617] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3733.150923] ? do_wait_intr+0xf0/0xf0 [ 3733.155901] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3733.163469] kthread+0x1d1/0x200 [ 3733.164808] ? set_kthread_struct+0x70/0x70 [ 3733.170779] ret_from_fork+0x1f/0x30 [ 3734.500292] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1097s! [ 3734.512041] Showing busy workqueues and worker pools: [ 3734.515092] workqueue events: flags=0x0 [ 3734.519644] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3734.523525] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3734.540548] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3734.549751] pending: cpu_latency_work [ptlrpc] [ 3734.559589] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3734.566355] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 3734.577260] workqueue events_long: flags=0x0 [ 3734.580492] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3734.584390] pending: lu_objects_flush [obdclass] [ 3734.587349] workqueue events_unbound: flags=0x2 [ 3734.590143] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3734.593709] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3734.596006] workqueue events_power_efficient: flags=0x80 [ 3734.598231] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3734.602679] pending: check_lifetime, do_cache_clean [ 3734.608970] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3734.613988] pending: neigh_periodic_work, neigh_periodic_work [ 3734.621938] workqueue mm_percpu_wq: flags=0x8 [ 3734.624608] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3734.629294] pending: vmstat_update, lru_add_drain_per_cpu [ 3734.636373] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3734.645301] pending: vmstat_update, lru_add_drain_per_cpu [ 3734.652294] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3734.658340] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3734.663079] workqueue xprtiod: flags=0xa [ 3734.669287] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3734.674761] pending: xprt_autoclose, xs_error_handle [ 3734.682811] workqueue ipv6_addrconf: flags=0x40008 [ 3734.686358] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3734.693005] pending: addrconf_verify_work [ 3734.701411] workqueue knbd0-recv: flags=0x1a [ 3734.708972] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3734.714307] in-flight: 651:recv_work [ 3734.717760] workqueue ptlrpc_pinger: flags=0x2 [ 3734.720194] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3734.722961] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3734.731070] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1097s workers=4 idle: 15979 31824 8 [ 3734.741666] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3725s workers=2 idle: 654 [ 3760.052433] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 3760.055062] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 3760.057050] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 3760.057062] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3760.057188] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3760.057195] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3760.057197] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3760.057213] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3760.057218] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3760.057222] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3760.057225] RDX: 0000000000003f56 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3760.057228] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3760.057230] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3760.057232] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3760.057236] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3760.057239] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3760.057241] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3760.057250] Call Trace: [ 3760.057256] [ 3760.057262] ? show_regs.cold.9+0x22/0x2f [ 3760.057272] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3760.057279] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3760.057297] ? watchdog+0x40/0x40 [ 3760.057302] ? hrtimer_interrupt+0x128/0x2c0 [ 3760.057306] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3760.057314] ? apic_timer_interrupt+0xf/0x20 [ 3760.057319] [ 3760.057323] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3760.057327] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3760.057333] do_raw_spin_lock+0x177/0x190 [ 3760.057338] _raw_spin_lock+0x21/0x30 [ 3760.057344] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3760.057667] obd_mod_free+0x46/0x140 [ptlrpc] [ 3760.057949] mdc_commit_open+0x91/0xb0 [mdc] [ 3760.058003] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3760.058409] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3760.058632] after_reply+0xe2c/0x1780 [ptlrpc] [ 3760.058905] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3760.059173] ? __next_timer_interrupt+0x160/0x160 [ 3760.059187] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3760.059471] ? do_wait_intr+0xf0/0xf0 [ 3760.059482] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3760.059551] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3760.059557] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3760.059480] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3760.059749] kthread+0x1d1/0x200 [ 3760.059755] ? set_kthread_struct+0x70/0x70 [ 3760.059759] ret_from_fork+0x1f/0x30 [ 3760.059854] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 3760.059859] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3760.059912] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3760.059916] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3760.059918] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3760.059928] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3760.059932] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3760.059937] RAX: 00000000000055de RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3760.059939] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3760.059941] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3760.059943] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3760.059945] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3760.059948] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3760.060076] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3760.060080] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3760.060089] Call Trace: [ 3760.060093] [ 3760.060096] ? show_regs.cold.9+0x22/0x2f [ 3760.060106] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3760.060112] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3760.060115] ? watchdog+0x40/0x40 [ 3760.060120] ? hrtimer_interrupt+0x128/0x2c0 [ 3760.060124] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3760.060131] ? apic_timer_interrupt+0xf/0x20 [ 3760.060136] [ 3760.060139] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3760.060143] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3760.060149] do_raw_spin_lock+0x177/0x190 [ 3760.060154] _raw_spin_lock+0x21/0x30 [ 3760.060159] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3760.060465] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3760.060688] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3760.061394] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3760.061632] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3760.059562] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3760.062191] ? woken_wake_function+0x30/0x30 [ 3760.062194] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3760.062203] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3760.062207] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3760.062211] RAX: 0000000000006f0d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3760.062213] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3760.062216] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3760.062218] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3760.062220] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3760.062223] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3760.062225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3760.062228] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3760.062237] Call Trace: [ 3760.062241] [ 3760.062243] ? show_regs.cold.9+0x22/0x2f [ 3760.062251] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3760.062257] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3760.062260] ? watchdog+0x40/0x40 [ 3760.062264] ? hrtimer_interrupt+0x128/0x2c0 [ 3760.062268] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3760.062275] ? apic_timer_interrupt+0xf/0x20 [ 3760.062279] [ 3760.062281] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3760.062286] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3760.062291] ? newidle_balance.isra.93+0x207/0x580 [ 3760.062296] do_raw_spin_lock+0x177/0x190 [ 3760.062300] _raw_spin_lock+0x21/0x30 [ 3760.062200] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3760.064444] kthread+0x1d1/0x200 [ 3760.062305] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3760.065085] process_one_work+0x2c8/0x700 [ 3760.065094] worker_thread+0x296/0x6e0 [ 3760.065098] ? rescuer_thread+0x570/0x570 [ 3760.065102] kthread+0x1d1/0x200 [ 3760.065106] ? set_kthread_struct+0x70/0x70 [ 3760.065109] ret_from_fork+0x1f/0x30 [ 3760.070225] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3760.070319] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3760.070326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3760.070328] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3760.070342] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3760.070346] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3760.070351] RAX: 0000000000007388 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3760.070353] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3760.070356] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3760.070358] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3760.070361] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3760.070364] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3760.070367] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3760.070370] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3760.070378] Call Trace: [ 3760.070384] [ 3760.070390] ? show_regs.cold.9+0x22/0x2f [ 3760.070400] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3760.070407] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3760.070411] ? watchdog+0x40/0x40 [ 3760.070415] ? hrtimer_interrupt+0x128/0x2c0 [ 3760.070419] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3760.070428] ? apic_timer_interrupt+0xf/0x20 [ 3760.070432] [ 3760.070435] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3760.070442] do_raw_spin_lock+0x177/0x190 [ 3760.070446] _raw_spin_lock+0x21/0x30 [ 3760.070452] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3760.070752] ? do_raw_spin_unlock+0x75/0x190 [ 3760.070758] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3760.071171] ? do_wait_intr+0xf0/0xf0 [ 3760.071180] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3760.071445] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3760.071686] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3760.072038] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3760.072178] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3760.072222] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3760.076006] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3760.076047] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3760.076173] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3760.076902] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3760.076936] lmv_intent_open+0x338/0x1150 [lmv] [ 3760.081017] ? set_kthread_struct+0x70/0x70 [ 3760.081035] ret_from_fork+0x1f/0x30 [ 3760.975514] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3760.978255] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3760.980941] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3760.982768] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3760.984929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3760.988624] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3760.991733] ? do_raw_spin_unlock+0x75/0x190 [ 3760.993983] ? _raw_spin_unlock+0x12/0x30 [ 3760.997027] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3761.000342] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3761.003399] ll_atomic_open+0x36a/0xd70 [lustre] [ 3761.006414] lookup_open+0x356/0xb40 [ 3761.008826] do_last+0x313/0xfc0 [ 3761.011057] ? nd_jump_root+0xe5/0x160 [ 3761.013682] ? path_init+0x437/0x520 [ 3761.016509] path_openat+0xf7/0x500 [ 3761.018777] ? do_raw_spin_unlock+0x75/0x190 [ 3761.022510] do_filp_open+0x99/0x140 [ 3761.024127] ? getname_flags+0x6e/0x330 [ 3761.025961] ? __check_object_size+0xff/0x256 [ 3761.028995] ? do_raw_spin_unlock+0x75/0x190 [ 3761.033275] ? _raw_spin_unlock+0x12/0x30 [ 3761.037144] do_sys_openat2+0x2b4/0x410 [ 3761.039923] do_sys_open+0x73/0xa0 [ 3761.041414] __x64_sys_openat+0x24/0x30 [ 3761.043734] do_syscall_64+0xc1/0x440 [ 3761.044964] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3761.051361] RIP: 0033:0x7f53425a5332 [ 3761.058011] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3761.075261] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3761.082032] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3761.085831] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3761.092917] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3761.102488] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3761.117846] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3765.215046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1128s! [ 3765.226870] Showing busy workqueues and worker pools: [ 3765.228288] workqueue events: flags=0x0 [ 3765.235000] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3765.244038] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3765.256507] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3765.262888] pending: cpu_latency_work [ptlrpc] [ 3765.268468] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3765.274212] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 3765.289277] workqueue events_long: flags=0x0 [ 3765.291062] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3765.293269] pending: lu_objects_flush [obdclass] [ 3765.296091] workqueue events_unbound: flags=0x2 [ 3765.300662] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3765.303160] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3765.305414] workqueue events_power_efficient: flags=0x80 [ 3765.307439] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3765.309769] pending: check_lifetime, do_cache_clean [ 3765.311915] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3765.314308] pending: neigh_periodic_work, neigh_periodic_work [ 3765.316365] workqueue mm_percpu_wq: flags=0x8 [ 3765.317837] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3765.320490] pending: vmstat_update, lru_add_drain_per_cpu [ 3765.322523] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3765.325971] pending: vmstat_update, lru_add_drain_per_cpu [ 3765.328368] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3765.330633] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3765.333019] workqueue xprtiod: flags=0xa [ 3765.334503] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3765.336759] pending: xprt_autoclose, xs_error_handle [ 3765.338759] workqueue ipv6_addrconf: flags=0x40008 [ 3765.340836] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3765.343445] pending: addrconf_verify_work [ 3765.347383] workqueue knbd0-recv: flags=0x1a [ 3765.351363] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3765.356339] in-flight: 651:recv_work [ 3765.358944] workqueue ptlrpc_pinger: flags=0x2 [ 3765.362392] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3765.368230] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3765.372790] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1128s workers=4 idle: 15979 31824 8 [ 3765.376344] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3755s workers=2 idle: 654 [ 3774.472178] rcu: INFO: rcu_sched self-detected stall on CPU [ 3774.476194] rcu: 2-....: (896894 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 3774.482392] (t=1680046 jiffies g=274185 q=64047) [ 3774.483944] rcu: rcu_sched kthread starved for 1138412 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 3774.494193] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3774.498919] rcu: RCU grace-period kthread stack dump: [ 3774.501330] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3774.515470] Call Trace: [ 3774.519077] __schedule+0x351/0xcb0 [ 3774.524093] schedule+0xc0/0x180 [ 3774.525357] schedule_timeout+0xb4/0x190 [ 3774.532752] ? do_raw_spin_unlock+0x75/0x190 [ 3774.534529] ? __next_timer_interrupt+0x160/0x160 [ 3774.538549] ? prepare_to_swait_event+0xf3/0x140 [ 3774.545181] rcu_gp_kthread+0x8dd/0x15b0 [ 3774.547627] ? force_qs_rnp+0x2c0/0x2c0 [ 3774.548941] kthread+0x1d1/0x200 [ 3774.553097] ? set_kthread_struct+0x70/0x70 [ 3774.554503] ret_from_fork+0x1f/0x30 [ 3774.556360] rcu: Stack dump where RCU GP kthread last ran: [ 3774.562208] NMI backtrace for cpu 2 [ 3774.563669] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3774.567828] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3774.571978] Call Trace: [ 3774.572656] [ 3774.573215] ? dump_stack+0xbb/0x10e [ 3774.574325] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3774.575807] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3774.576921] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3774.578556] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3774.580563] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3774.584381] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 3774.585748] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 3774.587983] ? update_process_times+0x6d/0xc0 [ 3774.590428] ? tick_sched_handle.isra.18+0x27/0x90 [ 3774.591868] ? tick_sched_timer+0x53/0xb0 [ 3774.594474] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3774.596896] ? tick_sched_handle.isra.18+0x90/0x90 [ 3774.599787] ? hrtimer_interrupt+0x128/0x2c0 [ 3774.601187] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3774.602548] ? apic_timer_interrupt+0xf/0x20 [ 3774.605434] [ 3774.605886] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3774.607290] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3774.609171] ? do_raw_spin_lock+0x177/0x190 [ 3774.610271] ? _raw_spin_lock+0x21/0x30 [ 3774.612909] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3774.615420] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3774.617763] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3774.619986] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3774.622079] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3774.624381] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3774.626373] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3774.628666] ? __next_timer_interrupt+0x160/0x160 [ 3774.630477] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3774.632226] ? do_wait_intr+0xf0/0xf0 [ 3774.633497] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3774.635265] ? kthread+0x1d1/0x200 [ 3774.636652] ? set_kthread_struct+0x70/0x70 [ 3774.638080] ? ret_from_fork+0x1f/0x30 [ 3774.641900] Sending NMI from CPU 2 to CPUs 1: [ 3774.653691] NMI backtrace for cpu 1 [ 3774.653697] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3774.653700] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3774.653702] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3774.653704] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3774.653706] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [ 3774.653709] RAX: 0000000000007e87 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3774.653711] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3774.653712] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3774.653714] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3774.653716] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3774.653718] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3774.653719] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3774.653720] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3774.653722] Call Trace: [ 3774.653723] [ 3774.653723] ? show_regs.cold.9+0x22/0x2f [ 3774.653725] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3774.653726] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3774.653727] ? nmi_handle+0xbd/0x210 [ 3774.653728] ? default_do_nmi+0x4b/0x1b0 [ 3774.653729] ? do_nmi+0x280/0x380 [ 3774.653731] ? end_repeat_nmi+0x16/0x1f [ 3774.653732] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3774.653733] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3774.653734] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3774.653736] [ 3774.653737] do_raw_spin_lock+0x177/0x190 [ 3774.653738] _raw_spin_lock+0x21/0x30 [ 3774.653739] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3774.653740] ? do_raw_spin_unlock+0x75/0x190 [ 3774.653742] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3774.653743] ? do_wait_intr+0xf0/0xf0 [ 3774.653744] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3774.653745] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3774.653747] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3774.653748] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3774.653749] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3774.653750] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3774.653751] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3774.653752] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3774.653753] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3774.653755] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3774.653756] lmv_intent_open+0x338/0x1150 [lmv] [ 3774.653757] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3774.653758] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3774.653759] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3774.653760] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3774.653761] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3774.653763] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3774.653764] ? do_raw_spin_unlock+0x75/0x190 [ 3774.653765] ? _raw_spin_unlock+0x12/0x30 [ 3774.653766] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3774.653767] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3774.653769] ll_atomic_open+0x36a/0xd70 [lustre] [ 3774.653770] lookup_open+0x356/0xb40 [ 3774.653771] do_last+0x313/0xfc0 [ 3774.653772] ? nd_jump_root+0xe5/0x160 [ 3774.653773] ? path_init+0x437/0x520 [ 3774.653774] path_openat+0xf7/0x500 [ 3774.653775] ? do_raw_spin_unlock+0x75/0x190 [ 3774.653777] do_filp_open+0x99/0x140 [ 3774.653778] ? getname_flags+0x6e/0x330 [ 3774.653779] ? __check_object_size+0xff/0x256 [ 3774.653780] ? do_raw_spin_unlock+0x75/0x190 [ 3774.653781] ? _raw_spin_unlock+0x12/0x30 [ 3774.653782] do_sys_openat2+0x2b4/0x410 [ 3774.653784] do_sys_open+0x73/0xa0 [ 3774.653785] __x64_sys_openat+0x24/0x30 [ 3774.653785] do_syscall_64+0xc1/0x440 [ 3774.653786] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3774.653788] RIP: 0033:0x7f53425a5332 [ 3774.653789] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3774.653791] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3774.653794] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3774.653796] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3774.653797] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3774.653799] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3774.653801] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3774.656650] NMI backtrace for cpu 2 [ 3775.019868] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3775.029337] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3775.036210] Call Trace: [ 3775.037421] [ 3775.038220] ? dump_stack+0xbb/0x10e [ 3775.041627] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3775.043354] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3775.051133] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3775.054026] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3775.057928] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3775.061142] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3775.063519] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3775.066526] ? update_process_times+0x6d/0xc0 [ 3775.068773] ? tick_sched_handle.isra.18+0x27/0x90 [ 3775.071005] ? tick_sched_timer+0x53/0xb0 [ 3775.074176] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3775.076073] ? tick_sched_handle.isra.18+0x90/0x90 [ 3775.083994] ? hrtimer_interrupt+0x128/0x2c0 [ 3775.087063] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3775.091180] ? apic_timer_interrupt+0xf/0x20 [ 3775.093919] [ 3775.095037] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3775.098054] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3775.100009] ? do_raw_spin_lock+0x177/0x190 [ 3775.104068] ? _raw_spin_lock+0x21/0x30 [ 3775.105916] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3775.107912] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3775.110661] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3775.112525] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3775.115469] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3775.118192] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3775.124157] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3775.128780] ? __next_timer_interrupt+0x160/0x160 [ 3775.131929] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3775.143036] ? do_wait_intr+0xf0/0xf0 [ 3775.144419] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3775.146639] ? kthread+0x1d1/0x200 [ 3775.155156] ? set_kthread_struct+0x70/0x70 [ 3775.156508] ? ret_from_fork+0x1f/0x30 [ 3788.058009] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 3788.061113] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3788.101787] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3788.109022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3788.111963] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3788.116170] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3788.131849] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3788.137601] RAX: 0000000000005d66 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3788.140797] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3788.147469] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3788.153677] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3788.157173] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3788.161603] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3788.167048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3788.169905] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3788.175429] Call Trace: [ 3788.177399] [ 3788.178621] ? show_regs.cold.9+0x22/0x2f [ 3788.181434] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3788.184507] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3788.187633] ? watchdog+0x40/0x40 [ 3788.189303] ? hrtimer_interrupt+0x128/0x2c0 [ 3788.191772] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3788.196346] ? apic_timer_interrupt+0xf/0x20 [ 3788.202564] [ 3788.207664] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3788.214822] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3788.216662] do_raw_spin_lock+0x177/0x190 [ 3788.219760] _raw_spin_lock+0x21/0x30 [ 3788.222594] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3788.227871] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3788.232041] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3788.235587] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3788.239094] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3788.242220] ? woken_wake_function+0x30/0x30 [ 3788.249913] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3788.257055] kthread+0x1d1/0x200 [ 3788.259419] ? set_kthread_struct+0x70/0x70 [ 3788.263825] ret_from_fork+0x1f/0x30 [ 3792.052025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 3792.057999] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3792.091118] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3792.099587] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3792.102901] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3792.106310] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3792.109959] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3792.125561] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3792.129971] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3792.132764] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3792.136607] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3792.142998] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3792.146810] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3792.153335] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3792.158754] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3792.160785] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3792.164516] Call Trace: [ 3792.165859] [ 3792.166739] ? show_regs.cold.9+0x22/0x2f [ 3792.169325] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3792.170976] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3792.173906] ? watchdog+0x40/0x40 [ 3792.177017] ? hrtimer_interrupt+0x128/0x2c0 [ 3792.178688] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3792.180025] ? apic_timer_interrupt+0xf/0x20 [ 3792.181962] [ 3792.182823] ? native_safe_halt+0x2e/0x50 [ 3792.186912] kvm_wait+0xcd/0xe0 [ 3792.188649] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3792.190953] ? newidle_balance.isra.93+0x207/0x580 [ 3792.195998] do_raw_spin_lock+0x177/0x190 [ 3792.198840] _raw_spin_lock+0x21/0x30 [ 3792.200028] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3792.207366] process_one_work+0x2c8/0x700 [ 3792.210275] worker_thread+0x296/0x6e0 [ 3792.211741] ? rescuer_thread+0x570/0x570 [ 3792.214217] kthread+0x1d1/0x200 [ 3792.216378] ? set_kthread_struct+0x70/0x70 [ 3792.217866] ret_from_fork+0x1f/0x30 [ 3795.429986] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1158s! [ 3795.438716] Showing busy workqueues and worker pools: [ 3795.448135] workqueue events: flags=0x0 [ 3795.454274] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3795.458675] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3795.467771] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3795.471775] pending: cpu_latency_work [ptlrpc] [ 3795.474211] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3795.477591] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 3795.482934] workqueue events_long: flags=0x0 [ 3795.484922] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3795.488623] pending: lu_objects_flush [obdclass] [ 3795.491621] workqueue events_unbound: flags=0x2 [ 3795.493874] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3795.496587] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3795.499420] workqueue events_power_efficient: flags=0x80 [ 3795.501907] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3795.504945] pending: check_lifetime, do_cache_clean [ 3795.507285] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3795.510788] pending: neigh_periodic_work, neigh_periodic_work [ 3795.513778] workqueue mm_percpu_wq: flags=0x8 [ 3795.516166] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3795.519379] pending: vmstat_update, lru_add_drain_per_cpu [ 3795.521925] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3795.524865] pending: vmstat_update, lru_add_drain_per_cpu [ 3795.531961] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3795.537876] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3795.544251] workqueue xprtiod: flags=0xa [ 3795.548132] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3795.554762] pending: xprt_autoclose, xs_error_handle [ 3795.556690] workqueue ipv6_addrconf: flags=0x40008 [ 3795.563293] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3795.568138] pending: addrconf_verify_work [ 3795.571233] workqueue knbd0-recv: flags=0x1a [ 3795.575427] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3795.581214] in-flight: 651:recv_work [ 3795.583599] workqueue ptlrpc_pinger: flags=0x2 [ 3795.587359] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3795.601502] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3795.604683] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1158s workers=4 idle: 15979 31824 8 [ 3795.613265] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3786s workers=2 idle: 654 [ 3800.055765] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 3800.057400] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 3800.063852] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3800.063942] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3800.064348] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3800.064352] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3800.064368] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3800.064373] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3800.064378] RAX: 0000000000001245 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3800.064381] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3800.064383] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3800.064385] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3800.064388] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3800.064392] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3800.064395] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3800.064397] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3800.064407] Call Trace: [ 3800.064413] [ 3800.064419] ? show_regs.cold.9+0x22/0x2f [ 3800.064429] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3800.064436] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3800.064441] ? watchdog+0x40/0x40 [ 3800.064445] ? hrtimer_interrupt+0x128/0x2c0 [ 3800.064449] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3800.064458] ? apic_timer_interrupt+0xf/0x20 [ 3800.064463] [ 3800.064466] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3800.064474] do_raw_spin_lock+0x177/0x190 [ 3800.064479] _raw_spin_lock+0x21/0x30 [ 3800.064484] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3800.067263] ? do_raw_spin_unlock+0x75/0x190 [ 3800.067273] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3800.069692] ? do_wait_intr+0xf0/0xf0 [ 3800.069713] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3800.071840] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3800.079547] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3800.079626] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3800.079634] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3800.079637] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3800.079650] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3800.079655] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3800.079659] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3800.079662] RDX: 0000000000001563 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3800.079665] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3800.079667] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3800.079670] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3800.079673] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3800.079675] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3800.079677] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3800.079686] Call Trace: [ 3800.079691] [ 3800.079699] ? show_regs.cold.9+0x22/0x2f [ 3800.079708] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3800.079714] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3800.079718] ? watchdog+0x40/0x40 [ 3800.079722] ? hrtimer_interrupt+0x128/0x2c0 [ 3800.079726] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3800.079735] ? apic_timer_interrupt+0xf/0x20 [ 3800.079739] [ 3800.079742] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3800.079746] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3800.079750] do_raw_spin_lock+0x177/0x190 [ 3800.079755] _raw_spin_lock+0x21/0x30 [ 3800.079760] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3800.080066] obd_mod_free+0x46/0x140 [ptlrpc] [ 3800.080270] mdc_commit_open+0x91/0xb0 [mdc] [ 3800.080307] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3800.080496] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3800.080694] after_reply+0xe2c/0x1780 [ptlrpc] [ 3800.123628] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3800.128749] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3800.137939] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3800.138102] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3800.138139] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3800.138166] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3800.138195] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3800.138358] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3800.138887] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3800.138918] lmv_intent_open+0x338/0x1150 [lmv] [ 3800.138964] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3800.139402] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3800.139710] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3800.139736] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3800.139840] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3800.139936] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3800.140722] ? do_raw_spin_unlock+0x75/0x190 [ 3800.140731] ? _raw_spin_unlock+0x12/0x30 [ 3800.140737] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3800.140767] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3800.140787] ll_atomic_open+0x36a/0xd70 [lustre] [ 3800.140875] lookup_open+0x356/0xb40 [ 3800.140885] do_last+0x313/0xfc0 [ 3800.140891] ? nd_jump_root+0xe5/0x160 [ 3800.140896] ? path_init+0x437/0x520 [ 3800.140900] path_openat+0xf7/0x500 [ 3800.140905] ? do_raw_spin_unlock+0x75/0x190 [ 3800.140910] do_filp_open+0x99/0x140 [ 3800.140916] ? getname_flags+0x6e/0x330 [ 3800.140921] ? __check_object_size+0xff/0x256 [ 3800.140928] ? do_raw_spin_unlock+0x75/0x190 [ 3800.140931] ? _raw_spin_unlock+0x12/0x30 [ 3800.140936] do_sys_openat2+0x2b4/0x410 [ 3800.140941] do_sys_open+0x73/0xa0 [ 3800.140945] __x64_sys_openat+0x24/0x30 [ 3800.142686] do_syscall_64+0xc1/0x440 [ 3800.142701] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3800.142707] RIP: 0033:0x7f53425a5332 [ 3800.142716] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3800.142721] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3800.142726] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3800.142729] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3800.142731] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3800.142733] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3800.142735] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3800.621527] ? __next_timer_interrupt+0x160/0x160 [ 3800.623834] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3800.626549] ? do_wait_intr+0xf0/0xf0 [ 3800.628646] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3800.631352] kthread+0x1d1/0x200 [ 3800.633129] ? set_kthread_struct+0x70/0x70 [ 3800.635255] ret_from_fork+0x1f/0x30 [ 3816.058680] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 3816.063975] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3816.077343] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3816.080419] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3816.082879] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3816.084539] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3816.089587] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3816.092122] RAX: 0000000000006850 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3816.094096] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3816.096107] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3816.098562] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3816.104138] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3816.105819] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3816.108530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3816.110355] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3816.112623] Call Trace: [ 3816.113386] [ 3816.113986] ? show_regs.cold.9+0x22/0x2f [ 3816.115175] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3816.116620] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3816.117964] ? watchdog+0x40/0x40 [ 3816.119153] ? hrtimer_interrupt+0x128/0x2c0 [ 3816.120299] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3816.121885] ? apic_timer_interrupt+0xf/0x20 [ 3816.123324] [ 3816.124166] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3816.125640] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3816.127238] do_raw_spin_lock+0x177/0x190 [ 3816.128415] _raw_spin_lock+0x21/0x30 [ 3816.129477] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3816.131538] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3816.133472] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3816.135324] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3816.137222] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3816.138867] ? woken_wake_function+0x30/0x30 [ 3816.140174] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3816.142220] kthread+0x1d1/0x200 [ 3816.143449] ? set_kthread_struct+0x70/0x70 [ 3816.144660] ret_from_fork+0x1f/0x30 [ 3820.054086] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 3820.062599] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3820.111053] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3820.123140] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3820.132866] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3820.146171] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3820.151117] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3820.164552] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3820.170364] RAX: 000000000000398b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3820.172796] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3820.178116] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3820.185548] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3820.190920] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3820.194401] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3820.199854] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3820.201299] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3820.206435] Call Trace: [ 3820.208163] [ 3820.208675] ? show_regs.cold.9+0x22/0x2f [ 3820.210171] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3820.216359] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3820.217741] ? watchdog+0x40/0x40 [ 3820.220972] ? hrtimer_interrupt+0x128/0x2c0 [ 3820.225423] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3820.229099] ? apic_timer_interrupt+0xf/0x20 [ 3820.235811] [ 3820.236430] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3820.239049] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3820.249873] ? newidle_balance.isra.93+0x207/0x580 [ 3820.256440] do_raw_spin_lock+0x177/0x190 [ 3820.261722] _raw_spin_lock+0x21/0x30 [ 3820.263142] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3820.265742] process_one_work+0x2c8/0x700 [ 3820.268322] worker_thread+0x296/0x6e0 [ 3820.269970] ? rescuer_thread+0x570/0x570 [ 3820.275421] kthread+0x1d1/0x200 [ 3820.277131] ? set_kthread_struct+0x70/0x70 [ 3820.278724] ret_from_fork+0x1f/0x30 [ 3825.632180] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1188s! [ 3825.638432] Showing busy workqueues and worker pools: [ 3825.644823] workqueue events: flags=0x0 [ 3825.647347] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3825.652789] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3825.658973] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3825.662384] pending: cpu_latency_work [ptlrpc] [ 3825.665536] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3825.669718] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 3825.675029] workqueue events_long: flags=0x0 [ 3825.676701] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3825.679410] pending: lu_objects_flush [obdclass] [ 3825.681444] workqueue events_unbound: flags=0x2 [ 3825.682994] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3825.685560] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3825.688818] workqueue events_power_efficient: flags=0x80 [ 3825.693697] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3825.696273] pending: check_lifetime, do_cache_clean [ 3825.699696] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3825.704160] pending: neigh_periodic_work, neigh_periodic_work [ 3825.706880] workqueue mm_percpu_wq: flags=0x8 [ 3825.708339] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3825.712060] pending: vmstat_update, lru_add_drain_per_cpu [ 3825.714360] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3825.717453] pending: vmstat_update, lru_add_drain_per_cpu [ 3825.719719] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3825.722681] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3825.728561] workqueue xprtiod: flags=0xa [ 3825.730915] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3825.734433] pending: xprt_autoclose, xs_error_handle [ 3825.738153] workqueue ipv6_addrconf: flags=0x40008 [ 3825.739950] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3825.748182] pending: addrconf_verify_work [ 3825.750015] workqueue knbd0-recv: flags=0x1a [ 3825.752685] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3825.756444] in-flight: 651:recv_work [ 3825.758258] workqueue ptlrpc_pinger: flags=0x2 [ 3825.766690] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3825.769131] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3825.773094] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1188s workers=4 idle: 15979 31824 8 [ 3825.776363] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3816s workers=2 idle: 654 [ 3828.055355] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 3828.058663] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3828.059262] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 3828.078715] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3828.078735] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3828.078738] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3828.078760] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3828.078765] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3828.078773] RAX: 0000000000004c1f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3828.078776] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3828.078778] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3828.078781] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3828.078783] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3828.078788] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3828.078791] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3828.078793] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3828.078803] Call Trace: [ 3828.078810] [ 3828.078816] ? show_regs.cold.9+0x22/0x2f [ 3828.078828] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3828.078837] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3828.078841] ? watchdog+0x40/0x40 [ 3828.078847] ? hrtimer_interrupt+0x128/0x2c0 [ 3828.078851] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3828.078861] ? apic_timer_interrupt+0xf/0x20 [ 3828.078866] [ 3828.078870] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3828.078881] do_raw_spin_lock+0x177/0x190 [ 3828.078887] _raw_spin_lock+0x21/0x30 [ 3828.078893] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3828.079520] ? do_raw_spin_unlock+0x75/0x190 [ 3828.079526] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3828.079729] ? do_wait_intr+0xf0/0xf0 [ 3828.079739] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3828.080514] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3828.080734] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3828.080949] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3828.081068] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3828.081109] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3828.081138] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3828.081165] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3828.081263] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3828.081625] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3828.081655] lmv_intent_open+0x338/0x1150 [lmv] [ 3828.081688] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3828.081776] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3828.081861] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3828.081884] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3828.081970] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3828.082054] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3828.082137] ? do_raw_spin_unlock+0x75/0x190 [ 3828.082145] ? _raw_spin_unlock+0x12/0x30 [ 3828.082151] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3828.082179] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3828.082213] ll_atomic_open+0x36a/0xd70 [lustre] [ 3828.082418] lookup_open+0x356/0xb40 [ 3828.082428] do_last+0x313/0xfc0 [ 3828.082433] ? nd_jump_root+0xe5/0x160 [ 3828.082437] ? path_init+0x437/0x520 [ 3828.082442] path_openat+0xf7/0x500 [ 3828.082448] ? do_raw_spin_unlock+0x75/0x190 [ 3828.082453] do_filp_open+0x99/0x140 [ 3828.082459] ? getname_flags+0x6e/0x330 [ 3828.082481] ? __check_object_size+0xff/0x256 [ 3828.082488] ? do_raw_spin_unlock+0x75/0x190 [ 3828.082492] ? _raw_spin_unlock+0x12/0x30 [ 3828.082497] do_sys_openat2+0x2b4/0x410 [ 3828.082502] do_sys_open+0x73/0xa0 [ 3828.082507] __x64_sys_openat+0x24/0x30 [ 3828.082510] do_syscall_64+0xc1/0x440 [ 3828.082518] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3828.082525] RIP: 0033:0x7f53425a5332 [ 3828.082532] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3828.082536] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3828.082542] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3828.082545] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3828.082547] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3828.082549] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3828.082552] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3828.260166] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3828.281055] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3828.286312] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3828.289632] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3828.291733] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3828.298847] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3828.302356] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3828.311921] RDX: 0000000000005e67 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3828.319451] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3828.322376] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3828.324980] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3828.327799] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3828.330831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3828.335374] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3828.338864] Call Trace: [ 3828.339899] [ 3828.340784] ? show_regs.cold.9+0x22/0x2f [ 3828.345835] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3828.348035] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3828.350013] ? watchdog+0x40/0x40 [ 3828.351528] ? hrtimer_interrupt+0x128/0x2c0 [ 3828.353616] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3828.355845] ? apic_timer_interrupt+0xf/0x20 [ 3828.357734] [ 3828.358775] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3828.361753] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3828.364204] do_raw_spin_lock+0x177/0x190 [ 3828.365990] _raw_spin_lock+0x21/0x30 [ 3828.367991] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3828.371619] obd_mod_free+0x46/0x140 [ptlrpc] [ 3828.374570] mdc_commit_open+0x91/0xb0 [mdc] [ 3828.379509] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3828.383862] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3828.387512] after_reply+0xe2c/0x1780 [ptlrpc] [ 3828.389933] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3828.392717] ? __next_timer_interrupt+0x160/0x160 [ 3828.394826] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3828.397230] ? do_wait_intr+0xf0/0xf0 [ 3828.398873] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3828.401649] kthread+0x1d1/0x200 [ 3828.403015] ? set_kthread_struct+0x70/0x70 [ 3828.404931] ret_from_fork+0x1f/0x30 [ 3844.060232] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 3844.065787] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3844.084365] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3844.087041] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3844.090283] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3844.094161] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3844.112019] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3844.114811] RAX: 0000000000007a58 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3844.123385] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3844.128010] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3844.131941] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3844.135036] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3844.138064] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3844.142081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3844.145358] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3844.147033] Call Trace: [ 3844.148453] [ 3844.149953] ? show_regs.cold.9+0x22/0x2f [ 3844.151617] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3844.155006] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3844.157113] ? watchdog+0x40/0x40 [ 3844.160265] ? hrtimer_interrupt+0x128/0x2c0 [ 3844.161724] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3844.172914] ? apic_timer_interrupt+0xf/0x20 [ 3844.174379] [ 3844.175086] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3844.187401] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3844.193998] do_raw_spin_lock+0x177/0x190 [ 3844.200604] _raw_spin_lock+0x21/0x30 [ 3844.204531] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3844.206703] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3844.211929] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3844.215785] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3844.217828] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3844.225927] ? woken_wake_function+0x30/0x30 [ 3844.231040] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3844.235078] kthread+0x1d1/0x200 [ 3844.238124] ? set_kthread_struct+0x70/0x70 [ 3844.241292] ret_from_fork+0x1f/0x30 [ 3852.055917] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 3852.066917] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3852.131358] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3852.146838] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3852.153824] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3852.165230] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3852.170189] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3852.191969] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3852.195254] RAX: 0000000000007506 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3852.201317] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3852.203298] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3852.205462] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3852.214126] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3852.217549] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3852.219977] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3852.223111] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3852.224717] Call Trace: [ 3852.225299] [ 3852.225802] ? show_regs.cold.9+0x22/0x2f [ 3852.236256] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3852.238837] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3852.243291] ? watchdog+0x40/0x40 [ 3852.244184] ? hrtimer_interrupt+0x128/0x2c0 [ 3852.246788] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3852.248618] ? apic_timer_interrupt+0xf/0x20 [ 3852.250549] [ 3852.251201] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3852.252823] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3852.254631] ? newidle_balance.isra.93+0x207/0x580 [ 3852.257999] do_raw_spin_lock+0x177/0x190 [ 3852.259699] _raw_spin_lock+0x21/0x30 [ 3852.260564] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3852.263850] process_one_work+0x2c8/0x700 [ 3852.264947] worker_thread+0x296/0x6e0 [ 3852.266305] ? rescuer_thread+0x570/0x570 [ 3852.267394] kthread+0x1d1/0x200 [ 3852.268429] ? set_kthread_struct+0x70/0x70 [ 3852.269563] ret_from_fork+0x1f/0x30 [ 3855.839480] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1218s! [ 3855.843864] Showing busy workqueues and worker pools: [ 3855.846482] workqueue events: flags=0x0 [ 3855.847743] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3855.852085] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3855.857865] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3855.860856] pending: cpu_latency_work [ptlrpc] [ 3855.863461] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3855.867193] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 3855.874432] workqueue events_long: flags=0x0 [ 3855.877563] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3855.884937] pending: lu_objects_flush [obdclass] [ 3855.892063] workqueue events_unbound: flags=0x2 [ 3855.898196] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3855.905940] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3855.912607] workqueue events_power_efficient: flags=0x80 [ 3855.918860] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3855.920682] pending: check_lifetime, do_cache_clean [ 3855.923767] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3855.927499] pending: neigh_periodic_work, neigh_periodic_work [ 3855.934157] workqueue mm_percpu_wq: flags=0x8 [ 3855.937214] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3855.946080] pending: vmstat_update, lru_add_drain_per_cpu [ 3855.949921] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3855.954048] pending: vmstat_update, lru_add_drain_per_cpu [ 3855.959401] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3855.963852] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3855.967273] workqueue xprtiod: flags=0xa [ 3855.969378] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3855.971712] pending: xprt_autoclose, xs_error_handle [ 3855.973395] workqueue ipv6_addrconf: flags=0x40008 [ 3855.979324] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3855.989630] pending: addrconf_verify_work [ 3855.992219] workqueue knbd0-recv: flags=0x1a [ 3855.994889] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3856.005142] in-flight: 651:recv_work [ 3856.009080] workqueue ptlrpc_pinger: flags=0x2 [ 3856.012465] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3856.015635] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3856.024057] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1218s workers=4 idle: 15979 31824 8 [ 3856.031338] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3846s workers=2 idle: 654 [ 3856.057014] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 3856.057024] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 3856.062714] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3856.062799] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3856.062807] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3856.062810] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3856.062824] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3856.062829] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3856.062834] RAX: 0000000000007e0a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3856.062836] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3856.062838] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3856.062840] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3856.062842] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3856.062846] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3856.062848] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3856.062851] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3856.062861] Call Trace: [ 3856.062868] [ 3856.062874] ? show_regs.cold.9+0x22/0x2f [ 3856.062884] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3856.062892] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3856.062896] ? watchdog+0x40/0x40 [ 3856.062900] ? hrtimer_interrupt+0x128/0x2c0 [ 3856.062903] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3856.062911] ? apic_timer_interrupt+0xf/0x20 [ 3856.062915] [ 3856.062918] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3856.062925] do_raw_spin_lock+0x177/0x190 [ 3856.062929] _raw_spin_lock+0x21/0x30 [ 3856.062934] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3856.063508] ? do_raw_spin_unlock+0x75/0x190 [ 3856.068503] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3856.068587] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3856.068594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3856.068597] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3856.068611] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3856.068615] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3856.068620] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3856.068623] RDX: 00000000000050c3 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3856.068625] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3856.068627] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3856.068629] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3856.068632] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3856.068635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3856.068637] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3856.068646] Call Trace: [ 3856.089658] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3856.103430] [ 3856.110882] ? do_wait_intr+0xf0/0xf0 [ 3856.110902] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3856.111317] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3856.112358] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3856.113229] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3856.113359] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3856.113402] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3856.113434] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3856.113462] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3856.113751] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3856.114095] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3856.114127] lmv_intent_open+0x338/0x1150 [lmv] [ 3856.114159] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3856.114311] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3856.114405] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3856.114430] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3856.114660] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3856.118903] ? show_regs.cold.9+0x22/0x2f [ 3856.118927] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3856.118933] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3856.118937] ? watchdog+0x40/0x40 [ 3856.118941] ? hrtimer_interrupt+0x128/0x2c0 [ 3856.118945] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3856.118954] ? apic_timer_interrupt+0xf/0x20 [ 3856.118958] [ 3856.118963] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3856.118970] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3856.118988] do_raw_spin_lock+0x177/0x190 [ 3856.118994] _raw_spin_lock+0x21/0x30 [ 3856.119001] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3856.119424] obd_mod_free+0x46/0x140 [ptlrpc] [ 3856.130117] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3856.130383] ? do_raw_spin_unlock+0x75/0x190 [ 3856.130392] ? _raw_spin_unlock+0x12/0x30 [ 3856.130400] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3856.130431] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3856.130452] ll_atomic_open+0x36a/0xd70 [lustre] [ 3856.130669] lookup_open+0x356/0xb40 [ 3856.130679] do_last+0x313/0xfc0 [ 3856.130684] ? nd_jump_root+0xe5/0x160 [ 3856.130689] ? path_init+0x437/0x520 [ 3856.130694] path_openat+0xf7/0x500 [ 3856.130699] ? do_raw_spin_unlock+0x75/0x190 [ 3856.130704] do_filp_open+0x99/0x140 [ 3856.130711] ? getname_flags+0x6e/0x330 [ 3856.130716] ? __check_object_size+0xff/0x256 [ 3856.130723] ? do_raw_spin_unlock+0x75/0x190 [ 3856.130727] ? _raw_spin_unlock+0x12/0x30 [ 3856.130731] do_sys_openat2+0x2b4/0x410 [ 3856.130736] do_sys_open+0x73/0xa0 [ 3856.130740] __x64_sys_openat+0x24/0x30 [ 3856.130744] do_syscall_64+0xc1/0x440 [ 3856.130751] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3856.130758] RIP: 0033:0x7f53425a5332 [ 3856.130767] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3856.130771] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3856.130776] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3856.130778] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3856.130781] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3856.130783] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3856.130785] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3856.615796] mdc_commit_open+0x91/0xb0 [mdc] [ 3856.617651] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3856.619413] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3856.627274] after_reply+0xe2c/0x1780 [ptlrpc] [ 3856.632166] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3856.637679] ? __next_timer_interrupt+0x160/0x160 [ 3856.640659] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3856.643187] ? do_wait_intr+0xf0/0xf0 [ 3856.645136] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3856.647484] kthread+0x1d1/0x200 [ 3856.649093] ? set_kthread_struct+0x70/0x70 [ 3856.650458] ret_from_fork+0x1f/0x30 [ 3872.059496] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 3872.064439] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3872.091067] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3872.099656] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3872.103497] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3872.107168] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3872.114791] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3872.119108] RAX: 000000000000771b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3872.124125] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3872.125993] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3872.131369] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3872.134860] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3872.138100] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3872.141971] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3872.143991] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3872.146243] Call Trace: [ 3872.147490] [ 3872.148468] ? show_regs.cold.9+0x22/0x2f [ 3872.150097] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3872.152418] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3872.154363] ? watchdog+0x40/0x40 [ 3872.155873] ? hrtimer_interrupt+0x128/0x2c0 [ 3872.157614] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3872.159899] ? apic_timer_interrupt+0xf/0x20 [ 3872.161809] [ 3872.162579] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3872.164841] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3872.167097] do_raw_spin_lock+0x177/0x190 [ 3872.168907] _raw_spin_lock+0x21/0x30 [ 3872.170491] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3872.173439] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3872.175963] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3872.178350] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3872.180955] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3872.183610] ? woken_wake_function+0x30/0x30 [ 3872.185366] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3872.188385] kthread+0x1d1/0x200 [ 3872.189767] ? set_kthread_struct+0x70/0x70 [ 3872.191606] ret_from_fork+0x1f/0x30 [ 3880.052226] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 3880.056975] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3880.118267] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3880.128206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3880.152696] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3880.169688] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3880.180456] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3880.196668] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3880.202630] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3880.208332] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3880.214142] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3880.222083] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3880.225770] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3880.233621] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3880.246205] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3880.254136] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3880.259966] Call Trace: [ 3880.261944] [ 3880.263459] ? show_regs.cold.9+0x22/0x2f [ 3880.266996] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3880.276142] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3880.282644] ? watchdog+0x40/0x40 [ 3880.284475] ? hrtimer_interrupt+0x128/0x2c0 [ 3880.287229] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3880.291700] ? apic_timer_interrupt+0xf/0x20 [ 3880.296636] [ 3880.300172] ? native_safe_halt+0x2e/0x50 [ 3880.311951] kvm_wait+0xcd/0xe0 [ 3880.313049] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3880.321945] ? newidle_balance.isra.93+0x207/0x580 [ 3880.325071] do_raw_spin_lock+0x177/0x190 [ 3880.328292] _raw_spin_lock+0x21/0x30 [ 3880.332881] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3880.337800] process_one_work+0x2c8/0x700 [ 3880.339987] worker_thread+0x296/0x6e0 [ 3880.342096] ? rescuer_thread+0x570/0x570 [ 3880.347121] kthread+0x1d1/0x200 [ 3880.349826] ? set_kthread_struct+0x70/0x70 [ 3880.352650] ret_from_fork+0x1f/0x30 [ 3884.056601] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 3884.057653] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 3884.063308] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3884.063398] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3884.065091] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3884.065099] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3884.065114] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3884.065118] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3884.065124] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3884.065126] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3884.065127] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3884.065129] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3884.065131] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3884.065134] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3884.065137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3884.065139] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3884.065149] Call Trace: [ 3884.065156] [ 3884.065162] ? show_regs.cold.9+0x22/0x2f [ 3884.065172] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3884.065178] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3884.065183] ? watchdog+0x40/0x40 [ 3884.065188] ? hrtimer_interrupt+0x128/0x2c0 [ 3884.065192] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3884.065199] ? apic_timer_interrupt+0xf/0x20 [ 3884.065204] [ 3884.065207] ? native_safe_halt+0x2e/0x50 [ 3884.065211] kvm_wait+0xcd/0xe0 [ 3884.065220] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3884.065228] do_raw_spin_lock+0x177/0x190 [ 3884.065233] _raw_spin_lock+0x21/0x30 [ 3884.065238] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3884.069248] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3884.069333] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3884.069342] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3884.069346] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3884.069361] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3884.069365] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3884.069371] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3884.069374] RDX: 0000000000002137 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3884.069377] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3884.069379] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3884.069381] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3884.069385] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3884.069388] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3884.069390] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3884.069399] Call Trace: [ 3884.069406] [ 3884.069414] ? show_regs.cold.9+0x22/0x2f [ 3884.069424] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3884.069431] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3884.069436] ? watchdog+0x40/0x40 [ 3884.069441] ? hrtimer_interrupt+0x128/0x2c0 [ 3884.069446] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3884.069455] ? apic_timer_interrupt+0xf/0x20 [ 3884.069460] [ 3884.069464] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3884.069469] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3884.069474] do_raw_spin_lock+0x177/0x190 [ 3884.069480] _raw_spin_lock+0x21/0x30 [ 3884.069499] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3884.114469] ? do_raw_spin_unlock+0x75/0x190 [ 3884.119925] obd_mod_free+0x46/0x140 [ptlrpc] [ 3884.128819] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3884.131140] mdc_commit_open+0x91/0xb0 [mdc] [ 3884.131188] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3884.146633] ? do_wait_intr+0xf0/0xf0 [ 3884.149959] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3884.157081] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3884.160703] after_reply+0xe2c/0x1780 [ptlrpc] [ 3884.160984] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3884.161371] ? __next_timer_interrupt+0x160/0x160 [ 3884.161381] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3884.161605] ? do_wait_intr+0xf0/0xf0 [ 3884.161614] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3884.161819] kthread+0x1d1/0x200 [ 3884.161825] ? set_kthread_struct+0x70/0x70 [ 3884.161829] ret_from_fork+0x1f/0x30 [ 3884.434230] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3884.437878] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3884.443328] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3884.445577] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3884.449206] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3884.456584] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3884.462862] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3884.466402] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3884.472460] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3884.476210] lmv_intent_open+0x338/0x1150 [lmv] [ 3884.480649] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3884.485432] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3884.488978] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3884.493843] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3884.498175] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3884.505091] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3884.507787] ? do_raw_spin_unlock+0x75/0x190 [ 3884.511318] ? _raw_spin_unlock+0x12/0x30 [ 3884.515604] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3884.518424] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3884.521253] ll_atomic_open+0x36a/0xd70 [lustre] [ 3884.523956] lookup_open+0x356/0xb40 [ 3884.525856] do_last+0x313/0xfc0 [ 3884.527471] ? nd_jump_root+0xe5/0x160 [ 3884.529696] ? path_init+0x437/0x520 [ 3884.531608] path_openat+0xf7/0x500 [ 3884.536103] ? do_raw_spin_unlock+0x75/0x190 [ 3884.538954] do_filp_open+0x99/0x140 [ 3884.542227] ? getname_flags+0x6e/0x330 [ 3884.544823] ? __check_object_size+0xff/0x256 [ 3884.547137] ? do_raw_spin_unlock+0x75/0x190 [ 3884.549306] ? _raw_spin_unlock+0x12/0x30 [ 3884.552072] do_sys_openat2+0x2b4/0x410 [ 3884.557191] do_sys_open+0x73/0xa0 [ 3884.567952] __x64_sys_openat+0x24/0x30 [ 3884.572275] do_syscall_64+0xc1/0x440 [ 3884.580026] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3884.587915] RIP: 0033:0x7f53425a5332 [ 3884.591314] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3884.617147] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3884.625504] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3884.629949] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3884.633730] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3884.637104] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3884.641304] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3886.047162] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1248s! [ 3886.058354] Showing busy workqueues and worker pools: [ 3886.064975] workqueue events: flags=0x0 [ 3886.070054] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3886.076306] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3886.082849] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3886.086290] pending: cpu_latency_work [ptlrpc] [ 3886.089381] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3886.092894] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 3886.103618] workqueue events_long: flags=0x0 [ 3886.107210] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3886.110610] pending: lu_objects_flush [obdclass] [ 3886.113608] workqueue events_unbound: flags=0x2 [ 3886.115402] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3886.119271] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3886.123073] workqueue events_power_efficient: flags=0x80 [ 3886.126936] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3886.131304] pending: check_lifetime, do_cache_clean [ 3886.134808] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3886.139680] pending: neigh_periodic_work, neigh_periodic_work [ 3886.144090] workqueue mm_percpu_wq: flags=0x8 [ 3886.148838] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3886.153279] pending: vmstat_update, lru_add_drain_per_cpu [ 3886.157527] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3886.164399] pending: vmstat_update, lru_add_drain_per_cpu [ 3886.170256] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3886.177615] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3886.185559] workqueue xprtiod: flags=0xa [ 3886.190590] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3886.197371] pending: xprt_autoclose, xs_error_handle [ 3886.201053] workqueue ipv6_addrconf: flags=0x40008 [ 3886.204515] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3886.210271] pending: addrconf_verify_work [ 3886.213238] workqueue knbd0-recv: flags=0x1a [ 3886.218707] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3886.223084] in-flight: 651:recv_work [ 3886.226865] workqueue ptlrpc_pinger: flags=0x2 [ 3886.229770] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3886.235348] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3886.244241] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1249s workers=4 idle: 15979 31824 8 [ 3886.249562] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3876s workers=2 idle: 654 [ 3900.059106] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 3900.066479] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3900.118981] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3900.127794] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3900.133718] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3900.140380] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3900.160758] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3900.168773] RAX: 0000000000007d0f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3900.178283] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3900.185029] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3900.196607] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3900.201382] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3900.215481] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3900.223084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3900.229930] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3900.236334] Call Trace: [ 3900.237237] [ 3900.237978] ? show_regs.cold.9+0x22/0x2f [ 3900.248257] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3900.251337] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3900.255059] ? watchdog+0x40/0x40 [ 3900.259033] ? hrtimer_interrupt+0x128/0x2c0 [ 3900.261517] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3900.268050] ? apic_timer_interrupt+0xf/0x20 [ 3900.276539] [ 3900.280009] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3900.286648] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3900.293467] do_raw_spin_lock+0x177/0x190 [ 3900.303073] _raw_spin_lock+0x21/0x30 [ 3900.310306] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3900.317042] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3900.323317] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3900.327877] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3900.332187] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3900.340109] ? woken_wake_function+0x30/0x30 [ 3900.343130] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3900.357238] kthread+0x1d1/0x200 [ 3900.359400] ? set_kthread_struct+0x70/0x70 [ 3900.360889] ret_from_fork+0x1f/0x30 [ 3912.052144] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 3912.055087] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 3912.060057] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3912.060384] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 3912.060881] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3912.060956] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3912.060964] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3912.060967] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3912.060980] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3912.060984] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3912.060989] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3912.060991] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3912.061005] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3912.061008] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3912.061010] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3912.061013] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3912.061016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3912.061019] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3912.061028] Call Trace: [ 3912.061034] [ 3912.061040] ? show_regs.cold.9+0x22/0x2f [ 3912.061051] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3912.061059] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3912.061063] ? watchdog+0x40/0x40 [ 3912.061068] ? hrtimer_interrupt+0x128/0x2c0 [ 3912.061072] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3912.061080] ? apic_timer_interrupt+0xf/0x20 [ 3912.061085] [ 3912.061088] ? native_safe_halt+0x2e/0x50 [ 3912.061092] kvm_wait+0xcd/0xe0 [ 3912.061100] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3912.061109] do_raw_spin_lock+0x177/0x190 [ 3912.061114] _raw_spin_lock+0x21/0x30 [ 3912.061119] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3912.061436] ? do_raw_spin_unlock+0x75/0x190 [ 3912.061442] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3912.062941] ? do_wait_intr+0xf0/0xf0 [ 3912.062952] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3912.063214] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3912.063443] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3912.066522] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3912.066939] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3912.066984] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3912.067028] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3912.067058] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3912.067460] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3912.067775] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3912.067822] lmv_intent_open+0x338/0x1150 [lmv] [ 3912.067857] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3912.067950] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3912.068046] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3912.068070] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3912.068186] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3912.068270] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3912.068349] ? do_raw_spin_unlock+0x75/0x190 [ 3912.068357] ? _raw_spin_unlock+0x12/0x30 [ 3912.068364] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3912.068395] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3912.068413] ll_atomic_open+0x36a/0xd70 [lustre] [ 3912.068499] lookup_open+0x356/0xb40 [ 3912.068508] do_last+0x313/0xfc0 [ 3912.068514] ? nd_jump_root+0xe5/0x160 [ 3912.068518] ? path_init+0x437/0x520 [ 3912.068523] path_openat+0xf7/0x500 [ 3912.068528] ? do_raw_spin_unlock+0x75/0x190 [ 3912.068532] do_filp_open+0x99/0x140 [ 3912.068537] ? getname_flags+0x6e/0x330 [ 3912.068542] ? __check_object_size+0xff/0x256 [ 3912.068549] ? do_raw_spin_unlock+0x75/0x190 [ 3912.068553] ? _raw_spin_unlock+0x12/0x30 [ 3912.068557] do_sys_openat2+0x2b4/0x410 [ 3912.068561] do_sys_open+0x73/0xa0 [ 3912.068565] __x64_sys_openat+0x24/0x30 [ 3912.068569] do_syscall_64+0xc1/0x440 [ 3912.068575] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3912.068582] RIP: 0033:0x7f53425a5332 [ 3912.068589] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3912.068593] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3912.068598] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3912.068600] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3912.068602] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3912.068605] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3912.068607] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3912.085559] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3912.085572] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3912.085579] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3912.085849] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3912.085859] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3912.085863] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3912.085869] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3912.085871] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3912.085873] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3912.085875] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3912.085878] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3912.085882] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3912.085884] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3912.085886] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3912.085895] Call Trace: [ 3912.085900] [ 3912.085906] ? show_regs.cold.9+0x22/0x2f [ 3912.085917] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3912.085923] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3912.085928] ? watchdog+0x40/0x40 [ 3912.085932] ? hrtimer_interrupt+0x128/0x2c0 [ 3912.085936] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3912.085942] ? apic_timer_interrupt+0xf/0x20 [ 3912.085947] [ 3912.085950] ? native_safe_halt+0x2e/0x50 [ 3912.085953] kvm_wait+0xcd/0xe0 [ 3912.085961] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3912.085967] ? newidle_balance.isra.93+0x207/0x580 [ 3912.085973] do_raw_spin_lock+0x177/0x190 [ 3912.085977] _raw_spin_lock+0x21/0x30 [ 3912.085981] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3912.086380] process_one_work+0x2c8/0x700 [ 3912.086388] worker_thread+0x296/0x6e0 [ 3912.086392] ? rescuer_thread+0x570/0x570 [ 3912.086395] kthread+0x1d1/0x200 [ 3912.086399] ? set_kthread_struct+0x70/0x70 [ 3912.086402] ret_from_fork+0x1f/0x30 [ 3912.504964] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3912.520503] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3912.524488] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3912.527801] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3912.530094] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3912.536870] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3912.539621] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3912.542299] RDX: 0000000000004ebc RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3912.544770] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3912.547321] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3912.549909] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3912.552616] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3912.555571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3912.557593] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3912.560211] Call Trace: [ 3912.561126] [ 3912.561857] ? show_regs.cold.9+0x22/0x2f [ 3912.563421] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3912.565261] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3912.566868] ? watchdog+0x40/0x40 [ 3912.568106] ? hrtimer_interrupt+0x128/0x2c0 [ 3912.569611] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3912.571308] ? apic_timer_interrupt+0xf/0x20 [ 3912.572827] [ 3912.573653] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3912.575637] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3912.577591] do_raw_spin_lock+0x177/0x190 [ 3912.579049] _raw_spin_lock+0x21/0x30 [ 3912.580361] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3912.582400] obd_mod_free+0x46/0x140 [ptlrpc] [ 3912.584310] mdc_commit_open+0x91/0xb0 [mdc] [ 3912.585956] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3912.588138] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3912.590360] after_reply+0xe2c/0x1780 [ptlrpc] [ 3912.592271] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3912.594233] ? __next_timer_interrupt+0x160/0x160 [ 3912.595839] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3912.597597] ? do_wait_intr+0xf0/0xf0 [ 3912.599305] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3912.603383] kthread+0x1d1/0x200 [ 3912.604689] ? set_kthread_struct+0x70/0x70 [ 3912.608479] ret_from_fork+0x1f/0x30 [ 3916.263748] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1279s! [ 3916.267809] Showing busy workqueues and worker pools: [ 3916.270321] workqueue events: flags=0x0 [ 3916.272535] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3916.280399] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3916.290135] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3916.298248] pending: cpu_latency_work [ptlrpc] [ 3916.308751] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3916.313693] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 3916.323099] workqueue events_long: flags=0x0 [ 3916.326502] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3916.332550] pending: lu_objects_flush [obdclass] [ 3916.335658] workqueue events_unbound: flags=0x2 [ 3916.338728] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3916.344349] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3916.348782] workqueue events_power_efficient: flags=0x80 [ 3916.354462] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3916.359701] pending: check_lifetime, do_cache_clean [ 3916.362515] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3916.367149] pending: neigh_periodic_work, neigh_periodic_work [ 3916.371761] workqueue mm_percpu_wq: flags=0x8 [ 3916.375812] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3916.380962] pending: vmstat_update, lru_add_drain_per_cpu [ 3916.385329] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3916.389094] pending: vmstat_update, lru_add_drain_per_cpu [ 3916.390973] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3916.394252] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3916.396824] workqueue xprtiod: flags=0xa [ 3916.398370] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3916.404369] pending: xprt_autoclose, xs_error_handle [ 3916.408521] workqueue ipv6_addrconf: flags=0x40008 [ 3916.416666] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3916.423291] pending: addrconf_verify_work [ 3916.426620] workqueue knbd0-recv: flags=0x1a [ 3916.430222] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3916.437834] in-flight: 651:recv_work [ 3916.447512] workqueue ptlrpc_pinger: flags=0x2 [ 3916.451588] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3916.454055] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3916.460418] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1279s workers=4 idle: 15979 31824 8 [ 3916.466793] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3906s workers=2 idle: 654 [ 3928.058863] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 3928.064903] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3928.115707] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3928.132959] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3928.136373] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3928.138544] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3928.157093] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3928.174133] RAX: 0000000000007f17 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3928.188278] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3928.194188] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3928.205246] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3928.217563] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3928.225498] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3928.231236] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3928.249656] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3928.259358] Call Trace: [ 3928.260262] [ 3928.264134] ? show_regs.cold.9+0x22/0x2f [ 3928.266238] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3928.276066] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3928.286463] ? watchdog+0x40/0x40 [ 3928.290923] ? hrtimer_interrupt+0x128/0x2c0 [ 3928.300091] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3928.309158] ? apic_timer_interrupt+0xf/0x20 [ 3928.316899] [ 3928.318441] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3928.321947] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3928.333231] do_raw_spin_lock+0x177/0x190 [ 3928.334661] _raw_spin_lock+0x21/0x30 [ 3928.335988] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3928.346722] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3928.350852] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3928.355595] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3928.360638] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3928.368991] ? woken_wake_function+0x30/0x30 [ 3928.371906] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3928.381634] kthread+0x1d1/0x200 [ 3928.386671] ? set_kthread_struct+0x70/0x70 [ 3928.394374] ret_from_fork+0x1f/0x30 [ 3940.055342] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 3940.057053] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 3940.068328] Modules linked in: lustre(O) [ 3940.072167] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr [ 3940.082790] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod [ 3940.106994] intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3940.107037] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3940.107045] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3940.107048] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3940.107061] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3940.107065] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3940.107070] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3940.107072] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3940.107074] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3940.107076] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3940.107079] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3940.107082] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3940.107185] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3940.107188] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3940.107196] Call Trace: [ 3940.107202] [ 3940.107208] ? show_regs.cold.9+0x22/0x2f [ 3940.107219] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3940.107225] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3940.107229] ? watchdog+0x40/0x40 [ 3940.107233] ? hrtimer_interrupt+0x128/0x2c0 [ 3940.107237] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3940.107245] ? apic_timer_interrupt+0xf/0x20 [ 3940.107249] [ 3940.107252] ? native_safe_halt+0x2e/0x50 [ 3940.107256] kvm_wait+0xcd/0xe0 [ 3940.107265] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3940.107273] do_raw_spin_lock+0x177/0x190 [ 3940.107278] _raw_spin_lock+0x21/0x30 [ 3940.107284] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3940.108094] obd_mod_free+0x46/0x140 [ptlrpc] [ 3940.108307] mdc_commit_open+0x91/0xb0 [mdc] [ 3940.108349] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3940.108681] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3940.109106] after_reply+0xe2c/0x1780 [ptlrpc] [ 3940.109310] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3940.109795] ? __next_timer_interrupt+0x160/0x160 [ 3940.109911] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3940.110169] ? do_wait_intr+0xf0/0xf0 [ 3940.110179] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3940.110369] kthread+0x1d1/0x200 [ 3940.110375] ? set_kthread_struct+0x70/0x70 [ 3940.110378] ret_from_fork+0x1f/0x30 [ 3940.450106] sha512_ssse3 sha512_generic [ 3940.452963] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3940.466219] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3940.480921] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3940.482539] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3940.511240] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 3940.524446] RAX: 00000000000078e9 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3940.533564] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3940.540145] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3940.544618] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3940.551286] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3940.556161] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3940.564855] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3940.574330] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3940.582462] Call Trace: [ 3940.583718] [ 3940.584352] ? show_regs.cold.9+0x22/0x2f [ 3940.597011] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3940.600353] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3940.613712] ? watchdog+0x40/0x40 [ 3940.614867] ? hrtimer_interrupt+0x128/0x2c0 [ 3940.616127] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3940.624414] ? apic_timer_interrupt+0xf/0x20 [ 3940.630508] [ 3940.631436] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3940.632926] do_raw_spin_lock+0x177/0x190 [ 3940.634069] _raw_spin_lock+0x21/0x30 [ 3940.642940] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3940.644778] ? do_raw_spin_unlock+0x75/0x190 [ 3940.649164] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3940.651249] ? do_wait_intr+0xf0/0xf0 [ 3940.656607] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3940.660059] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3940.665602] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3940.672363] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3940.682041] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3940.693415] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3940.709524] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3940.712365] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3940.721839] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3940.731397] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3940.741993] lmv_intent_open+0x338/0x1150 [lmv] [ 3940.746127] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3940.751510] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3940.757620] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3940.760398] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3940.764451] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3940.769589] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3940.773156] ? do_raw_spin_unlock+0x75/0x190 [ 3940.775769] ? _raw_spin_unlock+0x12/0x30 [ 3940.777222] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3940.779111] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3940.785060] ll_atomic_open+0x36a/0xd70 [lustre] [ 3940.787413] lookup_open+0x356/0xb40 [ 3940.790815] do_last+0x313/0xfc0 [ 3940.792856] ? nd_jump_root+0xe5/0x160 [ 3940.794966] ? path_init+0x437/0x520 [ 3940.798015] path_openat+0xf7/0x500 [ 3940.799044] ? do_raw_spin_unlock+0x75/0x190 [ 3940.800103] do_filp_open+0x99/0x140 [ 3940.804304] ? getname_flags+0x6e/0x330 [ 3940.806477] ? __check_object_size+0xff/0x256 [ 3940.810225] ? do_raw_spin_unlock+0x75/0x190 [ 3940.813843] ? _raw_spin_unlock+0x12/0x30 [ 3940.816064] do_sys_openat2+0x2b4/0x410 [ 3940.820006] do_sys_open+0x73/0xa0 [ 3940.821156] __x64_sys_openat+0x24/0x30 [ 3940.826478] do_syscall_64+0xc1/0x440 [ 3940.827502] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3940.830184] RIP: 0033:0x7f53425a5332 [ 3940.831318] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3940.844162] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3940.853099] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3940.861167] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3940.868283] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3940.873199] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3940.889382] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3944.052167] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 3944.058912] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3944.106220] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3944.121381] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3944.132060] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3944.141409] RIP: 0010:native_safe_halt+0x2e/0x50 [ 3944.145799] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 3944.175500] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3944.187349] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3944.200268] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3944.208643] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3944.210870] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3944.220273] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3944.226757] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3944.231788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3944.235690] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3944.243265] Call Trace: [ 3944.244576] [ 3944.245276] ? show_regs.cold.9+0x22/0x2f [ 3944.248942] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3944.250647] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3944.253838] ? watchdog+0x40/0x40 [ 3944.256195] ? hrtimer_interrupt+0x128/0x2c0 [ 3944.258656] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3944.262226] ? apic_timer_interrupt+0xf/0x20 [ 3944.264848] [ 3944.266850] ? native_safe_halt+0x2e/0x50 [ 3944.272368] kvm_wait+0xcd/0xe0 [ 3944.276215] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3944.280411] ? newidle_balance.isra.93+0x207/0x580 [ 3944.282365] do_raw_spin_lock+0x177/0x190 [ 3944.287271] _raw_spin_lock+0x21/0x30 [ 3944.291587] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3944.299286] process_one_work+0x2c8/0x700 [ 3944.303349] worker_thread+0x296/0x6e0 [ 3944.305138] ? rescuer_thread+0x570/0x570 [ 3944.307689] kthread+0x1d1/0x200 [ 3944.309422] ? set_kthread_struct+0x70/0x70 [ 3944.314713] ret_from_fork+0x1f/0x30 [ 3946.978679] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1309s! [ 3946.983078] Showing busy workqueues and worker pools: [ 3946.985994] workqueue events: flags=0x0 [ 3946.991672] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3946.998679] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3947.008204] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3947.013080] pending: cpu_latency_work [ptlrpc] [ 3947.015190] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3947.027141] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 3947.038510] workqueue events_long: flags=0x0 [ 3947.042953] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3947.055269] pending: lu_objects_flush [obdclass] [ 3947.069525] workqueue events_unbound: flags=0x2 [ 3947.084534] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3947.095218] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3947.102791] workqueue events_power_efficient: flags=0x80 [ 3947.110045] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3947.114427] pending: check_lifetime, do_cache_clean [ 3947.119207] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3947.131781] pending: neigh_periodic_work, neigh_periodic_work [ 3947.137475] workqueue mm_percpu_wq: flags=0x8 [ 3947.140112] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3947.147090] pending: vmstat_update, lru_add_drain_per_cpu [ 3947.151124] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3947.155501] pending: vmstat_update, lru_add_drain_per_cpu [ 3947.158557] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3947.162156] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3947.165908] workqueue xprtiod: flags=0xa [ 3947.168153] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3947.172990] pending: xprt_autoclose, xs_error_handle [ 3947.175368] workqueue ipv6_addrconf: flags=0x40008 [ 3947.186882] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3947.193567] pending: addrconf_verify_work [ 3947.196928] workqueue knbd0-recv: flags=0x1a [ 3947.199061] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3947.202287] in-flight: 651:recv_work [ 3947.204487] workqueue ptlrpc_pinger: flags=0x2 [ 3947.206946] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3947.213574] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3947.216246] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1310s workers=4 idle: 15979 31824 8 [ 3947.225215] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3937s workers=2 idle: 654 [ 3954.476381] rcu: INFO: rcu_sched self-detected stall on CPU [ 3954.484752] rcu: 2-....: (996582 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 3954.494221] (t=1860050 jiffies g=274185 q=64077) [ 3954.499081] rcu: rcu_sched kthread starved for 1318416 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 3954.504038] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 3954.508804] rcu: RCU grace-period kthread stack dump: [ 3954.511756] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 3954.517159] Call Trace: [ 3954.518707] __schedule+0x351/0xcb0 [ 3954.520356] schedule+0xc0/0x180 [ 3954.521741] schedule_timeout+0xb4/0x190 [ 3954.527143] ? do_raw_spin_unlock+0x75/0x190 [ 3954.528842] ? __next_timer_interrupt+0x160/0x160 [ 3954.536182] ? prepare_to_swait_event+0xf3/0x140 [ 3954.542788] rcu_gp_kthread+0x8dd/0x15b0 [ 3954.544261] ? force_qs_rnp+0x2c0/0x2c0 [ 3954.547748] kthread+0x1d1/0x200 [ 3954.549426] ? set_kthread_struct+0x70/0x70 [ 3954.552536] ret_from_fork+0x1f/0x30 [ 3954.555137] rcu: Stack dump where RCU GP kthread last ran: [ 3954.561133] Sending NMI from CPU 2 to CPUs 1: [ 3954.567705] NMI backtrace for cpu 1 [ 3954.567715] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3954.567718] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3954.567719] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3954.567721] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3954.567725] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [ 3954.567728] RAX: 0000000000003384 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3954.567730] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3954.567731] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3954.567733] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3954.567735] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3954.567736] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3954.567738] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3954.567740] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3954.567741] Call Trace: [ 3954.567742] [ 3954.567743] ? show_regs.cold.9+0x22/0x2f [ 3954.567745] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3954.567746] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3954.567747] ? nmi_handle+0xbd/0x210 [ 3954.567749] ? default_do_nmi+0x4b/0x1b0 [ 3954.567750] ? do_nmi+0x280/0x380 [ 3954.567751] ? end_repeat_nmi+0x16/0x1f [ 3954.567752] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3954.567753] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3954.567754] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3954.567755] [ 3954.567756] do_raw_spin_lock+0x177/0x190 [ 3954.567757] _raw_spin_lock+0x21/0x30 [ 3954.567758] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3954.567759] ? do_raw_spin_unlock+0x75/0x190 [ 3954.567760] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3954.567761] ? do_wait_intr+0xf0/0xf0 [ 3954.567762] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3954.567763] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3954.567764] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3954.567765] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3954.567766] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3954.567767] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3954.567768] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3954.567769] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3954.567770] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3954.567771] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3954.567772] lmv_intent_open+0x338/0x1150 [lmv] [ 3954.567773] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3954.567774] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3954.567775] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3954.567776] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3954.567777] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3954.567777] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3954.567778] ? do_raw_spin_unlock+0x75/0x190 [ 3954.567779] ? _raw_spin_unlock+0x12/0x30 [ 3954.567780] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3954.567781] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3954.567782] ll_atomic_open+0x36a/0xd70 [lustre] [ 3954.567783] lookup_open+0x356/0xb40 [ 3954.567784] do_last+0x313/0xfc0 [ 3954.567784] ? nd_jump_root+0xe5/0x160 [ 3954.567786] ? path_init+0x437/0x520 [ 3954.567787] path_openat+0xf7/0x500 [ 3954.567788] ? do_raw_spin_unlock+0x75/0x190 [ 3954.567789] do_filp_open+0x99/0x140 [ 3954.567790] ? getname_flags+0x6e/0x330 [ 3954.567791] ? __check_object_size+0xff/0x256 [ 3954.567792] ? do_raw_spin_unlock+0x75/0x190 [ 3954.567793] ? _raw_spin_unlock+0x12/0x30 [ 3954.567794] do_sys_openat2+0x2b4/0x410 [ 3954.567795] do_sys_open+0x73/0xa0 [ 3954.567796] __x64_sys_openat+0x24/0x30 [ 3954.567797] do_syscall_64+0xc1/0x440 [ 3954.567798] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3954.567799] RIP: 0033:0x7f53425a5332 [ 3954.567800] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3954.567802] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3954.567804] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3954.567805] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3954.567806] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3954.567807] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3954.567809] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3954.569313] Sending NMI from CPU 2 to CPUs 1: [ 3954.793294] NMI backtrace for cpu 1 [ 3954.793300] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3954.793303] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3954.793304] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3954.793306] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3954.793308] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [ 3954.793310] RAX: 0000000000002b84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3954.793311] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3954.793313] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3954.793314] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3954.793315] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3954.793316] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3954.793318] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3954.793319] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3954.793321] Call Trace: [ 3954.793322] [ 3954.793323] ? show_regs.cold.9+0x22/0x2f [ 3954.793324] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 3954.793325] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 3954.793326] ? nmi_handle+0xbd/0x210 [ 3954.793328] ? default_do_nmi+0x4b/0x1b0 [ 3954.793329] ? do_nmi+0x280/0x380 [ 3954.793330] ? end_repeat_nmi+0x16/0x1f [ 3954.793331] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3954.793332] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3954.793334] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3954.793335] [ 3954.793336] do_raw_spin_lock+0x177/0x190 [ 3954.793337] _raw_spin_lock+0x21/0x30 [ 3954.793338] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3954.793339] ? do_raw_spin_unlock+0x75/0x190 [ 3954.793340] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3954.793341] ? do_wait_intr+0xf0/0xf0 [ 3954.793342] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3954.793343] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3954.793344] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3954.793346] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3954.793347] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3954.793348] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3954.793350] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3954.793351] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3954.793352] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3954.793354] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3954.793355] lmv_intent_open+0x338/0x1150 [lmv] [ 3954.793356] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3954.793357] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3954.793358] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3954.793359] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3954.793361] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3954.793362] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3954.793363] ? do_raw_spin_unlock+0x75/0x190 [ 3954.793364] ? _raw_spin_unlock+0x12/0x30 [ 3954.793365] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3954.793366] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3954.793367] ll_atomic_open+0x36a/0xd70 [lustre] [ 3954.793368] lookup_open+0x356/0xb40 [ 3954.793369] do_last+0x313/0xfc0 [ 3954.793370] ? nd_jump_root+0xe5/0x160 [ 3954.793371] ? path_init+0x437/0x520 [ 3954.793372] path_openat+0xf7/0x500 [ 3954.793373] ? do_raw_spin_unlock+0x75/0x190 [ 3954.793374] do_filp_open+0x99/0x140 [ 3954.793375] ? getname_flags+0x6e/0x330 [ 3954.793376] ? __check_object_size+0xff/0x256 [ 3954.793378] ? do_raw_spin_unlock+0x75/0x190 [ 3954.793379] ? _raw_spin_unlock+0x12/0x30 [ 3954.793380] do_sys_openat2+0x2b4/0x410 [ 3954.793381] do_sys_open+0x73/0xa0 [ 3954.793382] __x64_sys_openat+0x24/0x30 [ 3954.793383] do_syscall_64+0xc1/0x440 [ 3954.793384] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3954.793385] RIP: 0033:0x7f53425a5332 [ 3954.793387] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3954.793389] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3954.793392] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3954.793393] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3954.793394] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3954.793396] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3954.793397] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3954.796194] NMI backtrace for cpu 2 [ 3955.036570] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3955.048945] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3955.054994] Call Trace: [ 3955.056319] [ 3955.057646] ? dump_stack+0xbb/0x10e [ 3955.062064] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 3955.067733] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 3955.073785] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 3955.078153] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 3955.081795] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 3955.084737] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 3955.087750] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 3955.090725] ? update_process_times+0x6d/0xc0 [ 3955.094311] ? tick_sched_handle.isra.18+0x27/0x90 [ 3955.100539] ? tick_sched_timer+0x53/0xb0 [ 3955.104196] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3955.105709] ? tick_sched_handle.isra.18+0x90/0x90 [ 3955.112264] ? hrtimer_interrupt+0x128/0x2c0 [ 3955.114759] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3955.116979] ? apic_timer_interrupt+0xf/0x20 [ 3955.122066] [ 3955.122886] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3955.125962] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3955.128700] ? do_raw_spin_lock+0x177/0x190 [ 3955.129787] ? _raw_spin_lock+0x21/0x30 [ 3955.131786] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3955.133813] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 3955.136317] ? mdc_commit_open+0x91/0xb0 [mdc] [ 3955.137500] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3955.140941] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3955.149116] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 3955.152160] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3955.159917] ? __next_timer_interrupt+0x160/0x160 [ 3955.167875] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3955.170578] ? do_wait_intr+0xf0/0xf0 [ 3955.176524] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3955.181065] ? kthread+0x1d1/0x200 [ 3955.182699] ? set_kthread_struct+0x70/0x70 [ 3955.185096] ? ret_from_fork+0x1f/0x30 [ 3956.059906] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 3956.072670] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3956.117858] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3956.128049] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3956.134964] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3956.142932] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3956.163196] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3956.168796] RAX: 000000000000328d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3956.177972] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3956.186417] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3956.194655] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3956.203079] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3956.212839] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3956.217758] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3956.224961] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3956.235638] Call Trace: [ 3956.237260] [ 3956.237995] ? show_regs.cold.9+0x22/0x2f [ 3956.239503] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3956.241933] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3956.247635] ? watchdog+0x40/0x40 [ 3956.248925] ? hrtimer_interrupt+0x128/0x2c0 [ 3956.250244] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3956.255012] ? apic_timer_interrupt+0xf/0x20 [ 3956.262842] [ 3956.265975] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3956.271801] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3956.277890] do_raw_spin_lock+0x177/0x190 [ 3956.283809] _raw_spin_lock+0x21/0x30 [ 3956.287379] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3956.295684] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3956.298073] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3956.303780] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3956.309388] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3956.314921] ? woken_wake_function+0x30/0x30 [ 3956.317466] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3956.323896] kthread+0x1d1/0x200 [ 3956.327214] ? set_kthread_struct+0x70/0x70 [ 3956.333963] ret_from_fork+0x1f/0x30 [ 3972.052454] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 3972.059495] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3972.095518] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3972.111580] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3972.117438] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 3972.120600] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3972.123432] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3972.133417] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3972.139316] RAX: 0000000000007f1f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3972.146940] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 3972.154994] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 3972.167595] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 3972.176656] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 3972.182649] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 3972.188394] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3972.190999] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 3972.195910] Call Trace: [ 3972.198240] [ 3972.199458] ? show_regs.cold.9+0x22/0x2f [ 3972.203993] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3972.205693] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3972.213330] ? watchdog+0x40/0x40 [ 3972.214771] ? hrtimer_interrupt+0x128/0x2c0 [ 3972.217602] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3972.220408] ? apic_timer_interrupt+0xf/0x20 [ 3972.222475] [ 3972.223538] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3972.226209] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3972.229336] ? newidle_balance.isra.93+0x207/0x580 [ 3972.232788] do_raw_spin_lock+0x177/0x190 [ 3972.238357] _raw_spin_lock+0x21/0x30 [ 3972.246286] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 3972.248141] process_one_work+0x2c8/0x700 [ 3972.253378] worker_thread+0x296/0x6e0 [ 3972.258796] ? rescuer_thread+0x570/0x570 [ 3972.265214] kthread+0x1d1/0x200 [ 3972.267885] ? set_kthread_struct+0x70/0x70 [ 3972.272649] ret_from_fork+0x1f/0x30 [ 3977.695165] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1340s! [ 3977.699357] Showing busy workqueues and worker pools: [ 3977.702563] workqueue events: flags=0x0 [ 3977.704633] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 3977.707554] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 3977.712629] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3977.715261] pending: cpu_latency_work [ptlrpc] [ 3977.716826] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 3977.721786] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 3977.727131] workqueue events_long: flags=0x0 [ 3977.728677] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 3977.731606] pending: lu_objects_flush [obdclass] [ 3977.734666] workqueue events_unbound: flags=0x2 [ 3977.736324] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 3977.737802] pending: flush_memcg_stats_dwork, idle_cull_fn [ 3977.744473] workqueue events_power_efficient: flags=0x80 [ 3977.746817] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3977.749577] pending: check_lifetime, do_cache_clean [ 3977.751781] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3977.754764] pending: neigh_periodic_work, neigh_periodic_work [ 3977.760444] workqueue mm_percpu_wq: flags=0x8 [ 3977.762879] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3977.766897] pending: vmstat_update, lru_add_drain_per_cpu [ 3977.769748] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 3977.772762] pending: vmstat_update, lru_add_drain_per_cpu [ 3977.774891] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 3977.777702] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 3977.779896] workqueue xprtiod: flags=0xa [ 3977.781534] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 3977.783363] pending: xprt_autoclose, xs_error_handle [ 3977.784887] workqueue ipv6_addrconf: flags=0x40008 [ 3977.786174] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 3977.787828] pending: addrconf_verify_work [ 3977.789073] workqueue knbd0-recv: flags=0x1a [ 3977.790143] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 3977.791762] in-flight: 651:recv_work [ 3977.793129] workqueue ptlrpc_pinger: flags=0x2 [ 3977.794188] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 3977.795797] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 3977.797634] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1340s workers=4 idle: 15979 31824 8 [ 3977.800974] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3968s workers=2 idle: 654 [ 3980.055349] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 3980.058050] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 3980.065526] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3980.065605] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3980.065615] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3980.065617] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3980.065634] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3980.065638] RSP: 0018:ffffbb5381503500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 3980.065641] RAX: 0000000000006d0d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3980.065644] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 3980.065646] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3980.065647] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 3980.065649] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 3980.065653] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 3980.065655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3980.065658] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 3980.065667] Call Trace: [ 3980.065673] [ 3980.065679] ? show_regs.cold.9+0x22/0x2f [ 3980.065690] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3980.065698] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3980.065702] ? watchdog+0x40/0x40 [ 3980.065706] ? hrtimer_interrupt+0x128/0x2c0 [ 3980.065710] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3980.065720] ? apic_timer_interrupt+0xf/0x20 [ 3980.065724] [ 3980.065726] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3980.065733] do_raw_spin_lock+0x177/0x190 [ 3980.065738] _raw_spin_lock+0x21/0x30 [ 3980.065743] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 3980.071252] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3980.071334] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3980.071342] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3980.071345] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3980.071360] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 3980.071365] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3980.071370] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 3980.105693] ? do_raw_spin_unlock+0x75/0x190 [ 3980.105714] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 3980.107042] ? do_wait_intr+0xf0/0xf0 [ 3980.107053] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 3980.107266] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 3980.107633] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3980.107874] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3980.108239] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3980.119363] RDX: 00000000000063ab RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 3980.119379] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 3980.119381] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 3980.119384] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 3980.119388] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 3980.119391] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3980.119394] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 3980.119404] Call Trace: [ 3980.119412] [ 3980.119418] ? show_regs.cold.9+0x22/0x2f [ 3980.119439] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3980.119446] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3980.119451] ? watchdog+0x40/0x40 [ 3980.119456] ? hrtimer_interrupt+0x128/0x2c0 [ 3980.119460] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3980.119470] ? apic_timer_interrupt+0xf/0x20 [ 3980.119474] [ 3980.119477] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 3980.119485] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 3980.119490] do_raw_spin_lock+0x177/0x190 [ 3980.119494] _raw_spin_lock+0x21/0x30 [ 3980.119502] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 3980.128410] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 3980.128482] mdc_intent_lock+0x353/0x7e0 [mdc] [ 3980.128515] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3980.128672] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 3980.129031] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 3980.129061] lmv_intent_open+0x338/0x1150 [lmv] [ 3980.129097] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3980.129191] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3980.129271] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 3980.129435] ll_intent_lock+0xc5/0xaa0 [lustre] [ 3980.129518] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 3980.129601] ll_lookup_it+0xe0e/0x2270 [lustre] [ 3980.129685] ? do_raw_spin_unlock+0x75/0x190 [ 3980.129693] ? _raw_spin_unlock+0x12/0x30 [ 3980.129701] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 3980.129731] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 3980.130098] ll_atomic_open+0x36a/0xd70 [lustre] [ 3980.130188] lookup_open+0x356/0xb40 [ 3980.130198] do_last+0x313/0xfc0 [ 3980.130203] ? nd_jump_root+0xe5/0x160 [ 3980.130208] ? path_init+0x437/0x520 [ 3980.130213] path_openat+0xf7/0x500 [ 3980.130219] ? do_raw_spin_unlock+0x75/0x190 [ 3980.130224] do_filp_open+0x99/0x140 [ 3980.130231] ? getname_flags+0x6e/0x330 [ 3980.130236] ? __check_object_size+0xff/0x256 [ 3980.130244] ? do_raw_spin_unlock+0x75/0x190 [ 3980.130247] ? _raw_spin_unlock+0x12/0x30 [ 3980.130251] do_sys_openat2+0x2b4/0x410 [ 3980.130255] do_sys_open+0x73/0xa0 [ 3980.130259] __x64_sys_openat+0x24/0x30 [ 3980.130263] do_syscall_64+0xc1/0x440 [ 3980.130270] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 3980.130277] RIP: 0033:0x7f53425a5332 [ 3980.130285] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 3980.130365] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 3980.130373] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 3980.130375] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 3980.130377] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 3980.130379] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 3980.130382] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 3980.521159] obd_mod_free+0x46/0x140 [ptlrpc] [ 3980.522735] mdc_commit_open+0x91/0xb0 [mdc] [ 3980.523865] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 3980.525633] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 3980.527719] after_reply+0xe2c/0x1780 [ptlrpc] [ 3980.529868] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 3980.531754] ? __next_timer_interrupt+0x160/0x160 [ 3980.534042] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 3980.535685] ? do_wait_intr+0xf0/0xf0 [ 3980.536926] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 3980.538406] kthread+0x1d1/0x200 [ 3980.539380] ? set_kthread_struct+0x70/0x70 [ 3980.541081] ret_from_fork+0x1f/0x30 [ 3984.059086] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 3984.065285] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 3984.112601] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 3984.124708] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 3984.133097] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3984.140327] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 3984.159871] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 3984.172000] RAX: 000000000000750e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 3984.179647] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 3984.184978] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 3984.189490] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 3984.193413] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 3984.197834] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 3984.203823] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3984.205800] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 3984.212500] Call Trace: [ 3984.213527] [ 3984.214201] ? show_regs.cold.9+0x22/0x2f [ 3984.215987] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 3984.217568] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 3984.220499] ? watchdog+0x40/0x40 [ 3984.222516] ? hrtimer_interrupt+0x128/0x2c0 [ 3984.224616] ? smp_apic_timer_interrupt+0xa6/0x270 [ 3984.237186] ? apic_timer_interrupt+0xf/0x20 [ 3984.242395] [ 3984.245054] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 3984.253818] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 3984.258629] do_raw_spin_lock+0x177/0x190 [ 3984.263205] _raw_spin_lock+0x21/0x30 [ 3984.268152] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 3984.280857] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 3984.282354] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 3984.292903] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 3984.303065] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 3984.310151] ? woken_wake_function+0x30/0x30 [ 3984.313299] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 3984.321599] kthread+0x1d1/0x200 [ 3984.327776] ? set_kthread_struct+0x70/0x70 [ 3984.332237] ret_from_fork+0x1f/0x30 [ 4004.057083] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 4004.066856] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4004.115882] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4004.134367] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4004.141197] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4004.147682] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4004.150374] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4004.167219] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4004.172721] RAX: 0000000000005f11 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4004.178966] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4004.185171] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4004.199017] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4004.207645] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4004.216203] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4004.232771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4004.243605] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4004.259801] Call Trace: [ 4004.260654] [ 4004.261356] ? show_regs.cold.9+0x22/0x2f [ 4004.269786] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4004.285963] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4004.287712] ? watchdog+0x40/0x40 [ 4004.288903] ? hrtimer_interrupt+0x128/0x2c0 [ 4004.299902] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4004.303601] ? apic_timer_interrupt+0xf/0x20 [ 4004.308950] [ 4004.312596] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4004.315524] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4004.320169] ? newidle_balance.isra.93+0x207/0x580 [ 4004.323314] do_raw_spin_lock+0x177/0x190 [ 4004.326397] _raw_spin_lock+0x21/0x30 [ 4004.327724] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4004.331310] process_one_work+0x2c8/0x700 [ 4004.332821] worker_thread+0x296/0x6e0 [ 4004.338865] ? rescuer_thread+0x570/0x570 [ 4004.341547] kthread+0x1d1/0x200 [ 4004.343522] ? set_kthread_struct+0x70/0x70 [ 4004.346279] ret_from_fork+0x1f/0x30 [ 4007.904595] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1370s! [ 4007.908330] Showing busy workqueues and worker pools: [ 4007.912746] workqueue events: flags=0x0 [ 4007.914567] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4007.920570] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4007.926682] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4007.931536] pending: cpu_latency_work [ptlrpc] [ 4007.933974] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4007.937889] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4007.942546] workqueue events_long: flags=0x0 [ 4007.944685] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4007.947950] pending: lu_objects_flush [obdclass] [ 4007.952148] workqueue events_unbound: flags=0x2 [ 4007.953687] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4007.959273] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4007.962020] workqueue events_power_efficient: flags=0x80 [ 4007.964539] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4007.967671] pending: check_lifetime, do_cache_clean [ 4007.970158] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4007.975599] pending: neigh_periodic_work, neigh_periodic_work [ 4007.978964] workqueue mm_percpu_wq: flags=0x8 [ 4007.981199] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4007.984587] pending: vmstat_update, lru_add_drain_per_cpu [ 4007.987565] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4007.990059] pending: vmstat_update, lru_add_drain_per_cpu [ 4007.992247] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4007.994767] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4007.997189] workqueue xprtiod: flags=0xa [ 4007.998582] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4008.000922] pending: xprt_autoclose, xs_error_handle [ 4008.002979] workqueue ipv6_addrconf: flags=0x40008 [ 4008.004854] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4008.007584] pending: addrconf_verify_work [ 4008.009379] workqueue knbd0-recv: flags=0x1a [ 4008.011055] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4008.013640] in-flight: 651:recv_work [ 4008.015274] workqueue ptlrpc_pinger: flags=0x2 [ 4008.017095] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4008.019579] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4008.022091] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1370s workers=4 idle: 15979 31824 8 [ 4008.025062] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=3998s workers=2 idle: 654 [ 4008.056086] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 4008.063184] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4008.066095] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 4008.097801] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4008.097812] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4008.097815] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4008.097826] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4008.097829] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4008.097833] RAX: 0000000000007285 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4008.097835] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4008.097837] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4008.097838] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4008.097840] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4008.097842] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4008.097844] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4008.097845] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4008.097851] Call Trace: [ 4008.097855] [ 4008.097860] ? show_regs.cold.9+0x22/0x2f [ 4008.097867] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4008.097872] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4008.097875] ? watchdog+0x40/0x40 [ 4008.097878] ? hrtimer_interrupt+0x128/0x2c0 [ 4008.097880] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4008.097886] ? apic_timer_interrupt+0xf/0x20 [ 4008.097889] [ 4008.097891] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4008.097897] do_raw_spin_lock+0x177/0x190 [ 4008.097900] _raw_spin_lock+0x21/0x30 [ 4008.097903] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4008.105975] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4008.106046] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4008.106053] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4008.106056] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4008.106071] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4008.106074] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4008.106078] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4008.106081] RDX: 000000000000476a RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4008.106083] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4008.106085] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4008.106087] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4008.106090] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4008.106093] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4008.106095] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4008.106104] Call Trace: [ 4008.106109] [ 4008.106114] ? show_regs.cold.9+0x22/0x2f [ 4008.106124] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4008.106130] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4008.106134] ? watchdog+0x40/0x40 [ 4008.106138] ? hrtimer_interrupt+0x128/0x2c0 [ 4008.106142] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4008.106150] ? apic_timer_interrupt+0xf/0x20 [ 4008.106155] [ 4008.106157] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4008.106162] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4008.106167] do_raw_spin_lock+0x177/0x190 [ 4008.106171] _raw_spin_lock+0x21/0x30 [ 4008.106247] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4008.115571] ? do_raw_spin_unlock+0x75/0x190 [ 4008.115594] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4008.115912] ? do_wait_intr+0xf0/0xf0 [ 4008.115921] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4008.116133] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4008.116385] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4008.116753] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4008.116849] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4008.116878] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4008.116896] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4008.116913] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4008.116968] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4008.117441] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4008.117469] lmv_intent_open+0x338/0x1150 [lmv] [ 4008.117493] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4008.117577] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4008.117730] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4008.117752] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4008.117839] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4008.117895] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4008.117950] ? do_raw_spin_unlock+0x75/0x190 [ 4008.117954] ? _raw_spin_unlock+0x12/0x30 [ 4008.117960] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4008.117979] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4008.117992] ll_atomic_open+0x36a/0xd70 [lustre] [ 4008.118078] lookup_open+0x356/0xb40 [ 4008.118086] do_last+0x313/0xfc0 [ 4008.118093] ? nd_jump_root+0xe5/0x160 [ 4008.118098] ? path_init+0x437/0x520 [ 4008.118102] path_openat+0xf7/0x500 [ 4008.118107] ? do_raw_spin_unlock+0x75/0x190 [ 4008.118111] do_filp_open+0x99/0x140 [ 4008.118118] ? getname_flags+0x6e/0x330 [ 4008.118123] ? __check_object_size+0xff/0x256 [ 4008.118129] ? do_raw_spin_unlock+0x75/0x190 [ 4008.118132] ? _raw_spin_unlock+0x12/0x30 [ 4008.118136] do_sys_openat2+0x2b4/0x410 [ 4008.118140] do_sys_open+0x73/0xa0 [ 4008.118144] __x64_sys_openat+0x24/0x30 [ 4008.118148] do_syscall_64+0xc1/0x440 [ 4008.118155] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4008.118161] RIP: 0033:0x7f53425a5332 [ 4008.118169] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4008.118554] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4008.118561] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4008.118564] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4008.118566] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4008.118568] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4008.118570] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4008.460794] obd_mod_free+0x46/0x140 [ptlrpc] [ 4008.465399] mdc_commit_open+0x91/0xb0 [mdc] [ 4008.467294] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4008.472064] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4008.475462] after_reply+0xe2c/0x1780 [ptlrpc] [ 4008.480862] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4008.483530] ? __next_timer_interrupt+0x160/0x160 [ 4008.486206] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4008.490350] ? do_wait_intr+0xf0/0xf0 [ 4008.491248] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4008.494371] kthread+0x1d1/0x200 [ 4008.495849] ? set_kthread_struct+0x70/0x70 [ 4008.500418] ret_from_fork+0x1f/0x30 [ 4012.058085] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4012.061060] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4012.082391] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4012.087567] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4012.091528] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4012.093536] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4012.102070] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4012.105421] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4012.108642] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4012.111627] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4012.114728] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4012.120604] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4012.124395] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4012.128672] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4012.130499] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4012.133431] Call Trace: [ 4012.134482] [ 4012.135474] ? show_regs.cold.9+0x22/0x2f [ 4012.137551] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4012.139631] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4012.141764] ? watchdog+0x40/0x40 [ 4012.143382] ? hrtimer_interrupt+0x128/0x2c0 [ 4012.145212] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4012.147447] ? apic_timer_interrupt+0xf/0x20 [ 4012.149299] [ 4012.150068] ? native_safe_halt+0x2e/0x50 [ 4012.151656] kvm_wait+0xcd/0xe0 [ 4012.153183] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4012.155496] do_raw_spin_lock+0x177/0x190 [ 4012.161314] _raw_spin_lock+0x21/0x30 [ 4012.162957] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4012.166658] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4012.169467] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4012.172140] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4012.178878] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4012.181313] ? woken_wake_function+0x30/0x30 [ 4012.183087] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4012.185763] kthread+0x1d1/0x200 [ 4012.187044] ? set_kthread_struct+0x70/0x70 [ 4012.188820] ret_from_fork+0x1f/0x30 [ 4032.052582] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 4032.061072] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4032.096214] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4032.105402] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4032.114774] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4032.129257] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4032.135069] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4032.161394] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4032.166314] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4032.173252] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4032.179858] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4032.184692] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4032.191093] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4032.195337] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4032.201264] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4032.205154] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4032.211198] Call Trace: [ 4032.212101] [ 4032.215706] ? show_regs.cold.9+0x22/0x2f [ 4032.219945] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4032.223186] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4032.228303] ? watchdog+0x40/0x40 [ 4032.230205] ? hrtimer_interrupt+0x128/0x2c0 [ 4032.232584] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4032.237264] ? apic_timer_interrupt+0xf/0x20 [ 4032.239560] [ 4032.242736] ? native_safe_halt+0x2e/0x50 [ 4032.245655] kvm_wait+0xcd/0xe0 [ 4032.248103] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4032.250887] ? newidle_balance.isra.93+0x207/0x580 [ 4032.255300] do_raw_spin_lock+0x177/0x190 [ 4032.258050] _raw_spin_lock+0x21/0x30 [ 4032.265734] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4032.268571] process_one_work+0x2c8/0x700 [ 4032.270056] worker_thread+0x296/0x6e0 [ 4032.271450] ? rescuer_thread+0x570/0x570 [ 4032.272831] kthread+0x1d1/0x200 [ 4032.275255] ? set_kthread_struct+0x70/0x70 [ 4032.278271] ret_from_fork+0x1f/0x30 [ 4036.057288] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 4036.058021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 4036.060930] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4036.061019] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4036.061026] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4036.061030] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4036.061045] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4036.061050] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4036.061054] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4036.061056] RDX: 0000000000001b0e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4036.061058] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4036.061060] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4036.061062] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4036.061066] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4036.061069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4036.061071] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4036.061080] Call Trace: [ 4036.061086] [ 4036.061091] ? show_regs.cold.9+0x22/0x2f [ 4036.061101] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4036.061107] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4036.061111] ? watchdog+0x40/0x40 [ 4036.061116] ? hrtimer_interrupt+0x128/0x2c0 [ 4036.061120] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4036.061128] ? apic_timer_interrupt+0xf/0x20 [ 4036.061132] [ 4036.061134] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4036.061139] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4036.061143] do_raw_spin_lock+0x177/0x190 [ 4036.061148] _raw_spin_lock+0x21/0x30 [ 4036.061153] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4036.061551] obd_mod_free+0x46/0x140 [ptlrpc] [ 4036.061748] mdc_commit_open+0x91/0xb0 [mdc] [ 4036.061784] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4036.062196] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4036.067054] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4036.067129] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4036.067141] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4036.067143] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4036.067156] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4036.067160] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4036.067164] RAX: 0000000000007d0a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4036.067166] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4036.067168] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4036.067170] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4036.067172] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4036.067175] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4036.067178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4036.067180] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4036.067188] Call Trace: [ 4036.067194] [ 4036.067199] ? show_regs.cold.9+0x22/0x2f [ 4036.067208] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4036.067215] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4036.067219] ? watchdog+0x40/0x40 [ 4036.067222] ? hrtimer_interrupt+0x128/0x2c0 [ 4036.067226] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4036.067236] ? apic_timer_interrupt+0xf/0x20 [ 4036.067240] [ 4036.067242] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4036.067248] do_raw_spin_lock+0x177/0x190 [ 4036.067252] _raw_spin_lock+0x21/0x30 [ 4036.067257] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4036.067634] ? do_raw_spin_unlock+0x75/0x190 [ 4036.067639] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4036.067922] ? do_wait_intr+0xf0/0xf0 [ 4036.067930] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4036.068136] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4036.068350] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4036.068798] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4036.069299] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4036.069349] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4036.069377] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4036.069405] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4036.069491] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4036.069740] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4036.069767] lmv_intent_open+0x338/0x1150 [lmv] [ 4036.104407] after_reply+0xe2c/0x1780 [ptlrpc] [ 4036.104816] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4036.105091] ? __next_timer_interrupt+0x160/0x160 [ 4036.105103] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4036.105305] ? do_wait_intr+0xf0/0xf0 [ 4036.105315] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4036.105524] kthread+0x1d1/0x200 [ 4036.105531] ? set_kthread_struct+0x70/0x70 [ 4036.105536] ret_from_fork+0x1f/0x30 [ 4036.347338] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4036.349507] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4036.352283] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4036.354400] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4036.356401] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4036.358943] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4036.360717] ? do_raw_spin_unlock+0x75/0x190 [ 4036.362656] ? _raw_spin_unlock+0x12/0x30 [ 4036.364402] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4036.366789] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4036.368950] ll_atomic_open+0x36a/0xd70 [lustre] [ 4036.370906] lookup_open+0x356/0xb40 [ 4036.372386] do_last+0x313/0xfc0 [ 4036.374041] ? nd_jump_root+0xe5/0x160 [ 4036.376108] ? path_init+0x437/0x520 [ 4036.377518] path_openat+0xf7/0x500 [ 4036.379285] ? do_raw_spin_unlock+0x75/0x190 [ 4036.381042] do_filp_open+0x99/0x140 [ 4036.382471] ? getname_flags+0x6e/0x330 [ 4036.384213] ? __check_object_size+0xff/0x256 [ 4036.386142] ? do_raw_spin_unlock+0x75/0x190 [ 4036.388035] ? _raw_spin_unlock+0x12/0x30 [ 4036.389847] do_sys_openat2+0x2b4/0x410 [ 4036.391518] do_sys_open+0x73/0xa0 [ 4036.393116] __x64_sys_openat+0x24/0x30 [ 4036.395063] do_syscall_64+0xc1/0x440 [ 4036.396811] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4036.398977] RIP: 0033:0x7f53425a5332 [ 4036.400482] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4036.408402] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4036.411690] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4036.414688] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4036.417503] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4036.420804] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4036.423560] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4038.115855] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1401s! [ 4038.146354] Showing busy workqueues and worker pools: [ 4038.157586] workqueue events: flags=0x0 [ 4038.169836] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4038.182918] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4038.201445] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4038.217146] pending: cpu_latency_work [ptlrpc] [ 4038.223780] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4038.229412] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4038.239877] workqueue events_long: flags=0x0 [ 4038.242504] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4038.247144] pending: lu_objects_flush [obdclass] [ 4038.251867] workqueue events_unbound: flags=0x2 [ 4038.255382] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4038.260120] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4038.263916] workqueue events_power_efficient: flags=0x80 [ 4038.267178] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4038.275283] pending: check_lifetime, do_cache_clean [ 4038.287214] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4038.295932] pending: neigh_periodic_work, neigh_periodic_work [ 4038.302137] workqueue mm_percpu_wq: flags=0x8 [ 4038.308653] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4038.321474] pending: vmstat_update, lru_add_drain_per_cpu [ 4038.323503] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4038.340153] pending: vmstat_update, lru_add_drain_per_cpu [ 4038.358089] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4038.360984] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4038.363427] workqueue xprtiod: flags=0xa [ 4038.365249] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4038.367974] pending: xprt_autoclose, xs_error_handle [ 4038.369974] workqueue ipv6_addrconf: flags=0x40008 [ 4038.373074] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4038.382049] pending: addrconf_verify_work [ 4038.385178] workqueue knbd0-recv: flags=0x1a [ 4038.388443] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4038.397193] in-flight: 651:recv_work [ 4038.400267] workqueue ptlrpc_pinger: flags=0x2 [ 4038.402941] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4038.411281] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4038.414944] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1401s workers=4 idle: 15979 31824 8 [ 4038.422066] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4028s workers=2 idle: 654 [ 4040.061861] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4040.066490] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4040.101873] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4040.111313] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4040.114234] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4040.119574] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4040.134137] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4040.144202] RAX: 0000000000007e11 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4040.150724] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4040.158018] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4040.167629] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4040.177228] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4040.188172] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4040.198829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4040.213569] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4040.223730] Call Trace: [ 4040.225349] [ 4040.226530] ? show_regs.cold.9+0x22/0x2f [ 4040.231575] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4040.242962] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4040.255964] ? watchdog+0x40/0x40 [ 4040.261115] ? hrtimer_interrupt+0x128/0x2c0 [ 4040.272308] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4040.280317] ? apic_timer_interrupt+0xf/0x20 [ 4040.288600] [ 4040.292399] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4040.298359] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4040.303371] do_raw_spin_lock+0x177/0x190 [ 4040.306548] _raw_spin_lock+0x21/0x30 [ 4040.314423] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4040.326181] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4040.333151] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4040.340517] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4040.347903] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4040.354198] ? woken_wake_function+0x30/0x30 [ 4040.359244] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4040.364590] kthread+0x1d1/0x200 [ 4040.366641] ? set_kthread_struct+0x70/0x70 [ 4040.368705] ret_from_fork+0x1f/0x30 [ 4064.053390] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 4064.059648] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 4064.063619] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 4064.063634] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4064.063706] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4064.063711] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4064.063714] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4064.065691] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4064.065696] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4064.065702] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4064.065704] RDX: 0000000000006e3e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4064.065706] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4064.065708] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4064.065710] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4064.065713] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4064.065716] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4064.065907] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4064.065917] Call Trace: [ 4064.065922] [ 4064.065927] ? show_regs.cold.9+0x22/0x2f [ 4064.065936] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4064.065942] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4064.065945] ? watchdog+0x40/0x40 [ 4064.065949] ? hrtimer_interrupt+0x128/0x2c0 [ 4064.065952] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4064.065959] ? apic_timer_interrupt+0xf/0x20 [ 4064.065963] [ 4064.065966] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4064.065970] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4064.065974] do_raw_spin_lock+0x177/0x190 [ 4064.065978] _raw_spin_lock+0x21/0x30 [ 4064.066130] Modules linked in: lustre(O) [ 4064.065984] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4064.066143] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4064.066238] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4064.066247] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4064.066244] obd_mod_free+0x46/0x140 [ptlrpc] [ 4064.066256] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4064.066891] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4064.066900] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4064.066903] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4064.066908] RAX: 0000000000003108 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4064.066910] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4064.066912] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4064.066914] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4064.066917] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4064.066919] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4064.066923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4064.066925] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4064.066933] Call Trace: [ 4064.066936] [ 4064.066939] ? show_regs.cold.9+0x22/0x2f [ 4064.066947] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4064.066952] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4064.066956] ? watchdog+0x40/0x40 [ 4064.066966] ? hrtimer_interrupt+0x128/0x2c0 [ 4064.066970] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4064.066977] ? apic_timer_interrupt+0xf/0x20 [ 4064.066985] [ 4064.066988] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4064.066993] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4064.066996] ? newidle_balance.isra.93+0x207/0x580 [ 4064.067001] do_raw_spin_lock+0x177/0x190 [ 4064.067007] _raw_spin_lock+0x21/0x30 [ 4064.067011] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4064.067463] mdc_commit_open+0x91/0xb0 [mdc] [ 4064.067887] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4064.068101] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4064.068302] after_reply+0xe2c/0x1780 [ptlrpc] [ 4064.069421] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4064.069851] ? __next_timer_interrupt+0x160/0x160 [ 4064.069862] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4064.070074] ? do_wait_intr+0xf0/0xf0 [ 4064.070082] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4064.070275] kthread+0x1d1/0x200 [ 4064.070281] ? set_kthread_struct+0x70/0x70 [ 4064.070284] ret_from_fork+0x1f/0x30 [ 4064.087304] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [ 4064.094024] process_one_work+0x2c8/0x700 [ 4064.094049] worker_thread+0x296/0x6e0 [ 4064.094053] ? rescuer_thread+0x570/0x570 [ 4064.094058] kthread+0x1d1/0x200 [ 4064.094062] ? set_kthread_struct+0x70/0x70 [ 4064.094066] ret_from_fork+0x1f/0x30 [ 4064.633101] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4064.664121] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4064.674970] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4064.683714] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4064.691094] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4064.706690] RSP: 0018:ffffbb5381503500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4064.711264] RAX: 0000000000006fad RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4064.717143] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4064.724127] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4064.730216] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4064.735371] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4064.745184] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4064.753992] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4064.758131] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4064.761093] Call Trace: [ 4064.762556] [ 4064.763150] ? show_regs.cold.9+0x22/0x2f [ 4064.765182] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4064.767258] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4064.770096] ? watchdog+0x40/0x40 [ 4064.772164] ? hrtimer_interrupt+0x128/0x2c0 [ 4064.774472] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4064.786654] ? apic_timer_interrupt+0xf/0x20 [ 4064.791621] [ 4064.794109] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4064.798923] do_raw_spin_lock+0x177/0x190 [ 4064.800670] _raw_spin_lock+0x21/0x30 [ 4064.805295] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4064.815552] ? do_raw_spin_unlock+0x75/0x190 [ 4064.818405] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4064.822483] ? do_wait_intr+0xf0/0xf0 [ 4064.824287] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4064.829090] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4064.833310] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4064.838546] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4064.848424] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4064.858776] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4064.869438] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4064.875608] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4064.891865] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4064.906741] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4064.908732] lmv_intent_open+0x338/0x1150 [lmv] [ 4064.921209] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4064.929106] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4064.936522] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4064.939265] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4064.945378] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4064.949656] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4064.954819] ? do_raw_spin_unlock+0x75/0x190 [ 4064.956409] ? _raw_spin_unlock+0x12/0x30 [ 4064.961444] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4064.967546] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4064.975656] ll_atomic_open+0x36a/0xd70 [lustre] [ 4064.978876] lookup_open+0x356/0xb40 [ 4064.990258] do_last+0x313/0xfc0 [ 4064.996620] ? nd_jump_root+0xe5/0x160 [ 4064.999314] ? path_init+0x437/0x520 [ 4065.001684] path_openat+0xf7/0x500 [ 4065.003529] ? do_raw_spin_unlock+0x75/0x190 [ 4065.008842] do_filp_open+0x99/0x140 [ 4065.014092] ? getname_flags+0x6e/0x330 [ 4065.016685] ? __check_object_size+0xff/0x256 [ 4065.020529] ? do_raw_spin_unlock+0x75/0x190 [ 4065.022772] ? _raw_spin_unlock+0x12/0x30 [ 4065.024181] do_sys_openat2+0x2b4/0x410 [ 4065.030173] do_sys_open+0x73/0xa0 [ 4065.033486] __x64_sys_openat+0x24/0x30 [ 4065.037359] do_syscall_64+0xc1/0x440 [ 4065.045920] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4065.050162] RIP: 0033:0x7f53425a5332 [ 4065.056180] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4065.071262] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4065.074864] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4065.079112] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4065.082459] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4065.088464] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4065.092827] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4068.059213] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4068.067949] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4068.108999] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4068.116968] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4068.119858] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4068.126996] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4068.141900] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4068.144804] RAX: 0000000000007f3b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4068.153130] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4068.159568] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4068.165798] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4068.171507] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4068.176196] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4068.182882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4068.185932] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4068.190084] Call Trace: [ 4068.191609] [ 4068.194753] ? show_regs.cold.9+0x22/0x2f [ 4068.196981] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4068.198778] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4068.205782] ? watchdog+0x40/0x40 [ 4068.207875] ? hrtimer_interrupt+0x128/0x2c0 [ 4068.209705] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4068.214169] ? apic_timer_interrupt+0xf/0x20 [ 4068.216513] [ 4068.219500] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4068.222092] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4068.224514] do_raw_spin_lock+0x177/0x190 [ 4068.226384] _raw_spin_lock+0x21/0x30 [ 4068.227961] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4068.230991] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4068.234095] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4068.238812] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4068.242701] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4068.245175] ? woken_wake_function+0x30/0x30 [ 4068.247045] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4068.250090] kthread+0x1d1/0x200 [ 4068.251787] ? set_kthread_struct+0x70/0x70 [ 4068.256509] ret_from_fork+0x1f/0x30 [ 4068.846171] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1431s! [ 4068.854795] Showing busy workqueues and worker pools: [ 4068.856464] workqueue events: flags=0x0 [ 4068.859174] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4068.861614] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4068.869926] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4068.873060] pending: cpu_latency_work [ptlrpc] [ 4068.880177] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4068.888784] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4068.895107] workqueue events_long: flags=0x0 [ 4068.896892] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4068.904415] pending: lu_objects_flush [obdclass] [ 4068.908836] workqueue events_unbound: flags=0x2 [ 4068.910766] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4068.913480] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4068.920885] workqueue events_power_efficient: flags=0x80 [ 4068.927021] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4068.935857] pending: check_lifetime, do_cache_clean [ 4068.941579] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4068.945863] pending: neigh_periodic_work, neigh_periodic_work [ 4068.951367] workqueue mm_percpu_wq: flags=0x8 [ 4068.952961] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4068.959457] pending: vmstat_update, lru_add_drain_per_cpu [ 4068.964857] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4068.971418] pending: vmstat_update, lru_add_drain_per_cpu [ 4068.975982] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4068.980417] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4068.988458] workqueue xprtiod: flags=0xa [ 4068.991498] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4068.995281] pending: xprt_autoclose, xs_error_handle [ 4068.998124] workqueue ipv6_addrconf: flags=0x40008 [ 4069.003813] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4069.013303] pending: addrconf_verify_work [ 4069.030212] workqueue knbd0-recv: flags=0x1a [ 4069.041016] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4069.049826] in-flight: 651:recv_work [ 4069.060792] workqueue ptlrpc_pinger: flags=0x2 [ 4069.062330] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4069.079011] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4069.082600] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1431s workers=4 idle: 15979 31824 8 [ 4069.098806] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4059s workers=2 idle: 654 [ 4092.055084] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 4092.061249] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4092.070775] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 4092.120406] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4092.120421] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4092.120423] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4092.120486] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4092.120489] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4092.120495] RAX: 0000000000007e87 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4092.120497] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4092.120499] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4092.120501] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4092.120503] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4092.120507] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4092.120510] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4092.120513] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4092.120522] Call Trace: [ 4092.120540] [ 4092.120546] ? show_regs.cold.9+0x22/0x2f [ 4092.120556] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4092.120563] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4092.120568] ? watchdog+0x40/0x40 [ 4092.120573] ? hrtimer_interrupt+0x128/0x2c0 [ 4092.120577] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4092.120588] ? apic_timer_interrupt+0xf/0x20 [ 4092.120592] [ 4092.120595] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4092.120603] do_raw_spin_lock+0x177/0x190 [ 4092.120607] _raw_spin_lock+0x21/0x30 [ 4092.120612] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4092.120912] ? do_raw_spin_unlock+0x75/0x190 [ 4092.120919] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4092.121148] ? do_wait_intr+0xf0/0xf0 [ 4092.121157] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4092.121373] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4092.121606] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4092.126957] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [ 4092.132294] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4092.132417] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4092.132530] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4092.132563] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4092.132592] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4092.132674] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4092.133691] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4092.133725] lmv_intent_open+0x338/0x1150 [lmv] [ 4092.133753] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4092.139349] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4092.139408] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4092.139416] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4092.139419] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4092.139435] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4092.139438] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4092.139443] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4092.139445] RDX: 0000000000006a4e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4092.139447] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4092.139449] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4092.139452] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4092.139454] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4092.139529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4092.139532] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4092.139540] Call Trace: [ 4092.139545] [ 4092.139550] ? show_regs.cold.9+0x22/0x2f [ 4092.139558] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4092.139565] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4092.139568] ? watchdog+0x40/0x40 [ 4092.139572] ? hrtimer_interrupt+0x128/0x2c0 [ 4092.139576] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4092.139583] ? apic_timer_interrupt+0xf/0x20 [ 4092.139588] [ 4092.139591] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4092.139595] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4092.139599] do_raw_spin_lock+0x177/0x190 [ 4092.139663] _raw_spin_lock+0x21/0x30 [ 4092.139671] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4092.140222] obd_mod_free+0x46/0x140 [ptlrpc] [ 4092.140443] mdc_commit_open+0x91/0xb0 [mdc] [ 4092.140569] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4092.141210] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4092.141410] after_reply+0xe2c/0x1780 [ptlrpc] [ 4092.141750] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4092.142015] ? __next_timer_interrupt+0x160/0x160 [ 4092.142206] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4092.142435] ? do_wait_intr+0xf0/0xf0 [ 4092.142445] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4092.142805] kthread+0x1d1/0x200 [ 4092.142812] ? set_kthread_struct+0x70/0x70 [ 4092.142816] ret_from_fork+0x1f/0x30 [ 4092.630635] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4092.633976] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4092.637573] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4092.642459] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4092.647379] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4092.650298] ? do_raw_spin_unlock+0x75/0x190 [ 4092.654797] ? _raw_spin_unlock+0x12/0x30 [ 4092.660345] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4092.664302] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4092.670374] ll_atomic_open+0x36a/0xd70 [lustre] [ 4092.676374] lookup_open+0x356/0xb40 [ 4092.678496] do_last+0x313/0xfc0 [ 4092.680003] ? nd_jump_root+0xe5/0x160 [ 4092.682221] ? path_init+0x437/0x520 [ 4092.684359] path_openat+0xf7/0x500 [ 4092.686764] ? do_raw_spin_unlock+0x75/0x190 [ 4092.695720] do_filp_open+0x99/0x140 [ 4092.698825] ? getname_flags+0x6e/0x330 [ 4092.704291] ? __check_object_size+0xff/0x256 [ 4092.706445] ? do_raw_spin_unlock+0x75/0x190 [ 4092.710331] ? _raw_spin_unlock+0x12/0x30 [ 4092.716269] do_sys_openat2+0x2b4/0x410 [ 4092.720693] do_sys_open+0x73/0xa0 [ 4092.724747] __x64_sys_openat+0x24/0x30 [ 4092.728768] do_syscall_64+0xc1/0x440 [ 4092.735911] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4092.740796] RIP: 0033:0x7f53425a5332 [ 4092.751959] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4092.769379] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4092.774573] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4092.782621] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4092.788416] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4092.813532] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4092.825777] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4096.058293] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4096.059654] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 4096.061269] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4096.061487] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4096.061495] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4096.061498] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4096.061512] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4096.061516] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4096.061520] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4096.061523] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4096.061525] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4096.061527] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4096.061529] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4096.061533] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4096.061536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4096.061538] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4096.061546] Call Trace: [ 4096.061552] [ 4096.061557] ? show_regs.cold.9+0x22/0x2f [ 4096.061567] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4096.061574] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4096.061578] ? watchdog+0x40/0x40 [ 4096.061582] ? hrtimer_interrupt+0x128/0x2c0 [ 4096.061586] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4096.061592] ? apic_timer_interrupt+0xf/0x20 [ 4096.061598] [ 4096.061600] ? native_safe_halt+0x2e/0x50 [ 4096.061604] kvm_wait+0xcd/0xe0 [ 4096.061611] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4096.061621] do_raw_spin_lock+0x177/0x190 [ 4096.061625] _raw_spin_lock+0x21/0x30 [ 4096.061630] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4096.061979] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4096.066714] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4096.066954] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4096.066961] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4096.066969] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4096.067315] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4096.067328] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4096.067333] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4096.067338] RAX: 0000000000007806 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4096.067340] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4096.067343] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4096.067345] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4096.067348] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4096.067352] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4096.067355] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4096.067357] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4096.067367] Call Trace: [ 4096.067374] [ 4096.067380] ? show_regs.cold.9+0x22/0x2f [ 4096.067389] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4096.067396] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4096.067401] ? watchdog+0x40/0x40 [ 4096.067406] ? hrtimer_interrupt+0x128/0x2c0 [ 4096.067410] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4096.067419] ? apic_timer_interrupt+0xf/0x20 [ 4096.067423] [ 4096.067426] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4096.067431] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4096.067435] ? newidle_balance.isra.93+0x207/0x580 [ 4096.067442] do_raw_spin_lock+0x177/0x190 [ 4096.067447] _raw_spin_lock+0x21/0x30 [ 4096.067452] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4096.068673] process_one_work+0x2c8/0x700 [ 4096.068681] worker_thread+0x296/0x6e0 [ 4096.068685] ? rescuer_thread+0x570/0x570 [ 4096.068688] kthread+0x1d1/0x200 [ 4096.068692] ? set_kthread_struct+0x70/0x70 [ 4096.068696] ret_from_fork+0x1f/0x30 [ 4096.401184] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4096.406033] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4096.408190] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4096.409863] ? woken_wake_function+0x30/0x30 [ 4096.411405] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4096.414884] kthread+0x1d1/0x200 [ 4096.416138] ? set_kthread_struct+0x70/0x70 [ 4096.417831] ret_from_fork+0x1f/0x30 [ 4099.551063] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1462s! [ 4099.554038] Showing busy workqueues and worker pools: [ 4099.568979] workqueue events: flags=0x0 [ 4099.575671] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4099.581548] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4099.598344] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4099.606800] pending: cpu_latency_work [ptlrpc] [ 4099.614431] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4099.620116] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4099.637560] workqueue events_long: flags=0x0 [ 4099.639856] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4099.643078] pending: lu_objects_flush [obdclass] [ 4099.646795] workqueue events_unbound: flags=0x2 [ 4099.649709] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4099.653689] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4099.660828] workqueue events_power_efficient: flags=0x80 [ 4099.665063] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4099.669409] pending: check_lifetime, do_cache_clean [ 4099.672305] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4099.676161] pending: neigh_periodic_work, neigh_periodic_work [ 4099.682242] workqueue mm_percpu_wq: flags=0x8 [ 4099.696041] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4099.703463] pending: vmstat_update, lru_add_drain_per_cpu [ 4099.735501] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4099.738113] pending: vmstat_update, lru_add_drain_per_cpu [ 4099.762674] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4099.765682] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4099.770916] workqueue xprtiod: flags=0xa [ 4099.772197] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4099.777078] pending: xprt_autoclose, xs_error_handle [ 4099.780014] workqueue ipv6_addrconf: flags=0x40008 [ 4099.781503] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4099.783269] pending: addrconf_verify_work [ 4099.784755] workqueue knbd0-recv: flags=0x1a [ 4099.785944] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4099.789178] in-flight: 651:recv_work [ 4099.791308] workqueue ptlrpc_pinger: flags=0x2 [ 4099.793962] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4099.797262] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4099.801541] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1462s workers=4 idle: 15979 31824 8 [ 4099.809197] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4090s workers=2 idle: 654 [ 4120.057561] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 4120.061110] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 4120.064022] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4120.064102] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4120.064111] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4120.064114] RIP: 0010:__do_softirq+0xa2/0x4e8 [ 4120.064128] Code: f6 01 01 c7 44 24 10 0a 00 00 00 48 c7 c0 40 2a 03 00 65 66 c7 00 00 00 48 c7 c0 30 bb 93 93 48 83 00 01 fb 66 0f 1f 44 00 00 ff ff ff ff 48 83 05 b9 1f f6 01 01 48 c7 c3 00 61 60 92 41 0f [ 4120.067226] RSP: 0018:ffffbb5380d24f78 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4120.067232] RAX: ffffffff9393bb30 RBX: ffff98a70209df40 RCX: 0000000000000001 [ 4120.067235] RDX: 00000000000007e3 RSI: 000000000000001c RDI: ffffffff94801040 [ 4120.067237] RBP: 0000000000000000 R08: 000003bf465ae980 R09: 000003bf465ae980 [ 4120.067239] R10: 00000000000f4240 R11: 000003bf465ae980 R12: 0000000000000000 [ 4120.067241] R13: 0000000000000000 R14: 0000000000000080 R15: 0000000000000000 [ 4120.067244] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4120.067247] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4120.067249] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4120.067257] Call Trace: [ 4120.067262] [ 4120.067267] ? show_regs.cold.9+0x22/0x2f [ 4120.067278] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4120.067285] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4120.067288] ? watchdog+0x40/0x40 [ 4120.067292] ? hrtimer_interrupt+0x128/0x2c0 [ 4120.067297] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4120.085327] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) [ 4120.128583] ? apic_timer_interrupt+0xf/0x20 [ 4120.128603] ? apic_timer_interrupt+0xa/0x20 [ 4120.128609] ? __do_softirq+0xa2/0x4e8 [ 4120.128613] ? __do_softirq+0x73/0x4e8 [ 4120.128617] ? kvm_sched_clock_read+0x2c/0x50 [ 4120.128625] irq_exit_rcu+0x1c6/0x1e0 [ 4120.128632] irq_exit+0x12/0x30 [ 4120.128637] smp_apic_timer_interrupt+0xe0/0x270 [ 4120.128643] apic_timer_interrupt+0xf/0x20 [ 4120.128651] [ 4120.128653] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4120.128663] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4120.128667] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4120.128672] RAX: 0000000000000e1f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4120.128675] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4120.128677] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4120.128679] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4120.128682] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4120.128690] do_raw_spin_lock+0x177/0x190 [ 4120.128694] _raw_spin_lock+0x21/0x30 [ 4120.128700] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4120.128966] ? do_raw_spin_unlock+0x75/0x190 [ 4120.128972] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4120.129165] ? do_wait_intr+0xf0/0xf0 [ 4120.129174] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4120.133094] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4120.133334] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4120.134350] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4120.134651] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4120.134693] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4120.134724] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4120.134752] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4120.134860] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4120.135087] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4120.135116] lmv_intent_open+0x338/0x1150 [lmv] [ 4120.135627] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4120.135720] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4120.135806] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4120.135829] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4120.135916] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4120.135998] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4120.136079] ? do_raw_spin_unlock+0x75/0x190 [ 4120.136086] ? _raw_spin_unlock+0x12/0x30 [ 4120.136093] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4120.136121] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4120.136152] ll_atomic_open+0x36a/0xd70 [lustre] [ 4120.136533] lookup_open+0x356/0xb40 [ 4120.136542] do_last+0x313/0xfc0 [ 4120.136547] ? nd_jump_root+0xe5/0x160 [ 4120.136552] ? path_init+0x437/0x520 [ 4120.136556] path_openat+0xf7/0x500 [ 4120.136561] ? do_raw_spin_unlock+0x75/0x190 [ 4120.136565] do_filp_open+0x99/0x140 [ 4120.136572] ? getname_flags+0x6e/0x330 [ 4120.136576] ? __check_object_size+0xff/0x256 [ 4120.136581] ? do_raw_spin_unlock+0x75/0x190 [ 4120.136585] ? _raw_spin_unlock+0x12/0x30 [ 4120.136589] do_sys_openat2+0x2b4/0x410 [ 4120.136593] do_sys_open+0x73/0xa0 [ 4120.136597] __x64_sys_openat+0x24/0x30 [ 4120.136601] do_syscall_64+0xc1/0x440 [ 4120.136607] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4120.136612] RIP: 0033:0x7f53425a5332 [ 4120.136619] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4120.136623] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4120.136628] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4120.136630] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4120.136632] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4120.136634] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4120.136636] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4121.014204] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4121.050423] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4121.067287] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4121.080940] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4121.090311] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4121.115059] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4121.134216] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4121.148621] RDX: 0000000000001bd4 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4121.155032] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4121.168266] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4121.183983] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4121.193397] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4121.199230] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4121.201320] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4121.207469] Call Trace: [ 4121.208185] [ 4121.218976] ? show_regs.cold.9+0x22/0x2f [ 4121.224210] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4121.230040] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4121.232117] ? watchdog+0x40/0x40 [ 4121.234685] ? hrtimer_interrupt+0x128/0x2c0 [ 4121.239037] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4121.241977] ? apic_timer_interrupt+0xf/0x20 [ 4121.245061] [ 4121.247178] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4121.250923] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4121.255698] do_raw_spin_lock+0x177/0x190 [ 4121.258664] _raw_spin_lock+0x21/0x30 [ 4121.261892] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4121.264717] obd_mod_free+0x46/0x140 [ptlrpc] [ 4121.267552] mdc_commit_open+0x91/0xb0 [mdc] [ 4121.268961] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4121.275058] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4121.287891] after_reply+0xe2c/0x1780 [ptlrpc] [ 4121.298280] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4121.317006] ? __next_timer_interrupt+0x160/0x160 [ 4121.318965] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4121.338094] ? do_wait_intr+0xf0/0xf0 [ 4121.342257] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4121.348006] kthread+0x1d1/0x200 [ 4121.353167] ? set_kthread_struct+0x70/0x70 [ 4121.355877] ret_from_fork+0x1f/0x30 [ 4124.052184] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 4124.059124] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4124.059889] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4124.094541] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4124.094555] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4124.094562] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4124.094939] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4124.094949] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4124.094953] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4124.095726] RAX: 0000000000003389 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4124.095728] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4124.095731] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4124.095733] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4124.095736] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4124.095739] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4124.095742] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4124.095744] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4124.095752] Call Trace: [ 4124.095757] [ 4124.095762] ? show_regs.cold.9+0x22/0x2f [ 4124.095771] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4124.095778] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4124.095782] ? watchdog+0x40/0x40 [ 4124.095786] ? hrtimer_interrupt+0x128/0x2c0 [ 4124.095790] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4124.095797] ? apic_timer_interrupt+0xf/0x20 [ 4124.095801] [ 4124.095804] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4124.095811] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4124.095814] ? newidle_balance.isra.93+0x207/0x580 [ 4124.095820] do_raw_spin_lock+0x177/0x190 [ 4124.095825] _raw_spin_lock+0x21/0x30 [ 4124.095830] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4124.096236] process_one_work+0x2c8/0x700 [ 4124.096243] worker_thread+0x296/0x6e0 [ 4124.096247] ? rescuer_thread+0x570/0x570 [ 4124.096251] kthread+0x1d1/0x200 [ 4124.096255] ? set_kthread_struct+0x70/0x70 [ 4124.096259] ret_from_fork+0x1f/0x30 [ 4124.300021] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4124.332278] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4124.339086] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4124.350986] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4124.360220] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4124.376972] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4124.382048] RAX: 0000000000006e68 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4124.386202] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4124.392183] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4124.397615] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4124.401804] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4124.406454] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4124.411069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4124.414272] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4124.419095] Call Trace: [ 4124.420348] [ 4124.423345] ? show_regs.cold.9+0x22/0x2f [ 4124.427072] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4124.435814] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4124.439197] ? watchdog+0x40/0x40 [ 4124.445070] ? hrtimer_interrupt+0x128/0x2c0 [ 4124.458001] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4124.466860] ? apic_timer_interrupt+0xf/0x20 [ 4124.467969] [ 4124.472986] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4124.476258] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4124.481952] do_raw_spin_lock+0x177/0x190 [ 4124.484410] _raw_spin_lock+0x21/0x30 [ 4124.487229] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4124.492923] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4124.500275] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4124.506583] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4124.519138] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4124.522939] ? woken_wake_function+0x30/0x30 [ 4124.527240] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4124.535117] kthread+0x1d1/0x200 [ 4124.537661] ? set_kthread_struct+0x70/0x70 [ 4124.543094] ret_from_fork+0x1f/0x30 [ 4130.273156] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1493s! [ 4130.283659] Showing busy workqueues and worker pools: [ 4130.287848] workqueue events: flags=0x0 [ 4130.295064] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4130.305835] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4130.315616] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4130.323259] pending: cpu_latency_work [ptlrpc] [ 4130.326106] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4130.329173] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4130.341321] workqueue events_long: flags=0x0 [ 4130.345834] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4130.352159] pending: lu_objects_flush [obdclass] [ 4130.357310] workqueue events_unbound: flags=0x2 [ 4130.361028] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4130.367305] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4130.376480] workqueue events_power_efficient: flags=0x80 [ 4130.377972] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4130.385806] pending: check_lifetime, do_cache_clean [ 4130.393887] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4130.400778] pending: neigh_periodic_work, neigh_periodic_work [ 4130.406825] workqueue mm_percpu_wq: flags=0x8 [ 4130.414098] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4130.424945] pending: vmstat_update, lru_add_drain_per_cpu [ 4130.433027] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4130.441910] pending: vmstat_update, lru_add_drain_per_cpu [ 4130.443861] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4130.456815] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4130.464906] workqueue xprtiod: flags=0xa [ 4130.470446] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4130.479424] pending: xprt_autoclose, xs_error_handle [ 4130.489680] workqueue ipv6_addrconf: flags=0x40008 [ 4130.491716] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4130.495102] pending: addrconf_verify_work [ 4130.499720] workqueue knbd0-recv: flags=0x1a [ 4130.503140] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4130.509098] in-flight: 651:recv_work [ 4130.512134] workqueue ptlrpc_pinger: flags=0x2 [ 4130.517531] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4130.522960] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4130.527260] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1493s workers=4 idle: 15979 31824 8 [ 4130.533612] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4120s workers=2 idle: 654 [ 4134.479806] rcu: INFO: rcu_sched self-detected stall on CPU [ 4134.483274] rcu: 2-....: (1077411 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 4134.487311] (t=2040053 jiffies g=274185 q=64175) [ 4134.488764] rcu: rcu_sched kthread starved for 1498419 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 4134.492492] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4134.496732] rcu: RCU grace-period kthread stack dump: [ 4134.498762] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4134.502779] Call Trace: [ 4134.503839] __schedule+0x351/0xcb0 [ 4134.505126] schedule+0xc0/0x180 [ 4134.506718] schedule_timeout+0xb4/0x190 [ 4134.508304] ? do_raw_spin_unlock+0x75/0x190 [ 4134.510262] ? __next_timer_interrupt+0x160/0x160 [ 4134.512508] ? prepare_to_swait_event+0xf3/0x140 [ 4134.514501] rcu_gp_kthread+0x8dd/0x15b0 [ 4134.516267] ? force_qs_rnp+0x2c0/0x2c0 [ 4134.517975] kthread+0x1d1/0x200 [ 4134.519389] ? set_kthread_struct+0x70/0x70 [ 4134.521041] ret_from_fork+0x1f/0x30 [ 4134.524310] rcu: Stack dump where RCU GP kthread last ran: [ 4134.528764] Sending NMI from CPU 2 to CPUs 1: [ 4134.536427] NMI backtrace for cpu 1 [ 4134.536434] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4134.536435] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4134.536437] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4134.536438] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4134.536440] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [ 4134.536443] RAX: 000000000000322a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4134.536444] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4134.536445] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4134.536447] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4134.536448] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4134.536449] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4134.536451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4134.536452] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4134.536453] Call Trace: [ 4134.536454] [ 4134.536455] ? show_regs.cold.9+0x22/0x2f [ 4134.536456] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4134.536457] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4134.536458] ? nmi_handle+0xbd/0x210 [ 4134.536459] ? default_do_nmi+0x4b/0x1b0 [ 4134.536460] ? do_nmi+0x280/0x380 [ 4134.536461] ? end_repeat_nmi+0x16/0x1f [ 4134.536462] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4134.536463] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4134.536464] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4134.536465] [ 4134.536466] do_raw_spin_lock+0x177/0x190 [ 4134.536467] _raw_spin_lock+0x21/0x30 [ 4134.536468] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4134.536469] ? do_raw_spin_unlock+0x75/0x190 [ 4134.536470] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4134.536471] ? do_wait_intr+0xf0/0xf0 [ 4134.536472] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4134.536473] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4134.536474] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4134.536475] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4134.536476] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4134.536477] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4134.536478] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4134.536479] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4134.536480] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4134.536481] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4134.536482] lmv_intent_open+0x338/0x1150 [lmv] [ 4134.536483] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4134.536484] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4134.536485] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4134.536486] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4134.536487] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4134.536488] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4134.536489] ? do_raw_spin_unlock+0x75/0x190 [ 4134.536490] ? _raw_spin_unlock+0x12/0x30 [ 4134.536491] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4134.536492] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4134.536493] ll_atomic_open+0x36a/0xd70 [lustre] [ 4134.536494] lookup_open+0x356/0xb40 [ 4134.536495] do_last+0x313/0xfc0 [ 4134.536496] ? nd_jump_root+0xe5/0x160 [ 4134.536497] ? path_init+0x437/0x520 [ 4134.536498] path_openat+0xf7/0x500 [ 4134.536499] ? do_raw_spin_unlock+0x75/0x190 [ 4134.536500] do_filp_open+0x99/0x140 [ 4134.536501] ? getname_flags+0x6e/0x330 [ 4134.536502] ? __check_object_size+0xff/0x256 [ 4134.536503] ? do_raw_spin_unlock+0x75/0x190 [ 4134.536504] ? _raw_spin_unlock+0x12/0x30 [ 4134.536505] do_sys_openat2+0x2b4/0x410 [ 4134.536506] do_sys_open+0x73/0xa0 [ 4134.536508] __x64_sys_openat+0x24/0x30 [ 4134.536509] do_syscall_64+0xc1/0x440 [ 4134.536510] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4134.536512] RIP: 0033:0x7f53425a5332 [ 4134.536513] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4134.536516] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4134.536518] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4134.536519] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4134.536521] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4134.536522] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4134.536523] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4134.539462] Sending NMI from CPU 2 to CPUs 1: [ 4134.854921] NMI backtrace for cpu 1 [ 4134.854930] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4134.854932] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4134.854934] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4134.854936] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4134.854938] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [ 4134.854941] RAX: 0000000000006e15 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4134.854943] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4134.854944] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4134.854946] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4134.854947] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4134.854948] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4134.854950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4134.854951] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4134.854952] Call Trace: [ 4134.854953] [ 4134.854954] ? show_regs.cold.9+0x22/0x2f [ 4134.854955] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4134.854956] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4134.854958] ? nmi_handle+0xbd/0x210 [ 4134.854959] ? default_do_nmi+0x4b/0x1b0 [ 4134.854960] ? do_nmi+0x280/0x380 [ 4134.855210] ? end_repeat_nmi+0x16/0x1f [ 4134.855213] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4134.855214] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4134.855215] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4134.855216] [ 4134.855217] do_raw_spin_lock+0x177/0x190 [ 4134.855222] _raw_spin_lock+0x21/0x30 [ 4134.855223] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4134.855224] ? do_raw_spin_unlock+0x75/0x190 [ 4134.855225] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4134.855226] ? do_wait_intr+0xf0/0xf0 [ 4134.855227] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4134.855228] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4134.855229] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4134.855231] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4134.855232] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4134.855233] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4134.855234] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4134.855235] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4134.855236] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4134.855237] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4134.855238] lmv_intent_open+0x338/0x1150 [lmv] [ 4134.855309] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4134.855312] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4134.855313] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4134.855314] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4134.855315] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4134.855316] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4134.855318] ? do_raw_spin_unlock+0x75/0x190 [ 4134.855319] ? _raw_spin_unlock+0x12/0x30 [ 4134.855320] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4134.855321] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4134.855323] ll_atomic_open+0x36a/0xd70 [lustre] [ 4134.855324] lookup_open+0x356/0xb40 [ 4134.855325] do_last+0x313/0xfc0 [ 4134.855326] ? nd_jump_root+0xe5/0x160 [ 4134.855327] ? path_init+0x437/0x520 [ 4134.855328] path_openat+0xf7/0x500 [ 4134.855329] ? do_raw_spin_unlock+0x75/0x190 [ 4134.855329] do_filp_open+0x99/0x140 [ 4134.855331] ? getname_flags+0x6e/0x330 [ 4134.855332] ? __check_object_size+0xff/0x256 [ 4134.855333] ? do_raw_spin_unlock+0x75/0x190 [ 4134.855334] ? _raw_spin_unlock+0x12/0x30 [ 4134.855335] do_sys_openat2+0x2b4/0x410 [ 4134.855336] do_sys_open+0x73/0xa0 [ 4134.855338] __x64_sys_openat+0x24/0x30 [ 4134.855338] do_syscall_64+0xc1/0x440 [ 4134.855340] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4134.855341] RIP: 0033:0x7f53425a5332 [ 4134.855343] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4134.855346] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4134.855570] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4134.855573] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4134.855575] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4134.855576] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4134.855578] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4134.855630] NMI backtrace for cpu 2 [ 4135.183585] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4135.193402] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4135.203265] Call Trace: [ 4135.205038] [ 4135.206283] ? dump_stack+0xbb/0x10e [ 4135.209752] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4135.215159] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4135.219758] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4135.222731] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4135.228849] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4135.231815] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4135.234257] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4135.236957] ? update_process_times+0x6d/0xc0 [ 4135.239063] ? tick_sched_handle.isra.18+0x27/0x90 [ 4135.244754] ? tick_sched_timer+0x53/0xb0 [ 4135.247530] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4135.252237] ? tick_sched_handle.isra.18+0x90/0x90 [ 4135.256980] ? hrtimer_interrupt+0x128/0x2c0 [ 4135.259499] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4135.264774] ? apic_timer_interrupt+0xf/0x20 [ 4135.273392] [ 4135.274419] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4135.281700] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4135.286099] ? do_raw_spin_lock+0x177/0x190 [ 4135.289283] ? _raw_spin_lock+0x21/0x30 [ 4135.292086] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4135.297065] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4135.302613] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4135.309181] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4135.317740] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4135.324546] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4135.327937] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4135.335177] ? __next_timer_interrupt+0x160/0x160 [ 4135.340505] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4135.350546] ? do_wait_intr+0xf0/0xf0 [ 4135.351858] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4135.365084] ? kthread+0x1d1/0x200 [ 4135.370672] ? set_kthread_struct+0x70/0x70 [ 4135.379685] ? ret_from_fork+0x1f/0x30 [ 4152.061544] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4152.074639] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4152.127573] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4152.134046] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4152.143558] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4152.148953] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4152.161764] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4152.170559] RAX: 0000000000007e16 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4152.176544] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4152.181651] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4152.190397] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4152.203349] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4152.209857] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4152.213556] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4152.222037] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4152.233649] Call Trace: [ 4152.234528] [ 4152.235214] ? show_regs.cold.9+0x22/0x2f [ 4152.242303] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4152.247250] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4152.254106] ? watchdog+0x40/0x40 [ 4152.257697] ? hrtimer_interrupt+0x128/0x2c0 [ 4152.261327] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4152.264721] ? apic_timer_interrupt+0xf/0x20 [ 4152.267007] [ 4152.269187] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4152.273461] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4152.275282] do_raw_spin_lock+0x177/0x190 [ 4152.282592] _raw_spin_lock+0x21/0x30 [ 4152.287228] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4152.296116] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4152.299759] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4152.306067] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4152.316178] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4152.318742] ? woken_wake_function+0x30/0x30 [ 4152.323705] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4152.325968] kthread+0x1d1/0x200 [ 4152.331081] ? set_kthread_struct+0x70/0x70 [ 4152.334036] ret_from_fork+0x1f/0x30 [ 4156.052163] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 4156.060290] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4156.109247] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4156.115237] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4156.124572] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4156.131861] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4156.133414] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4156.161648] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4156.176598] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4156.184989] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4156.195697] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4156.208672] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4156.214874] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4156.234149] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4156.249651] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4156.259660] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4156.276093] Call Trace: [ 4156.276970] [ 4156.277665] ? show_regs.cold.9+0x22/0x2f [ 4156.293989] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4156.295688] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4156.313038] ? watchdog+0x40/0x40 [ 4156.314230] ? hrtimer_interrupt+0x128/0x2c0 [ 4156.328663] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4156.342418] ? apic_timer_interrupt+0xf/0x20 [ 4156.346018] [ 4156.348306] ? native_safe_halt+0x2e/0x50 [ 4156.350195] kvm_wait+0xcd/0xe0 [ 4156.351716] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4156.353670] ? newidle_balance.isra.93+0x207/0x580 [ 4156.357455] do_raw_spin_lock+0x177/0x190 [ 4156.361736] _raw_spin_lock+0x21/0x30 [ 4156.364226] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4156.367409] process_one_work+0x2c8/0x700 [ 4156.368848] worker_thread+0x296/0x6e0 [ 4156.370693] ? rescuer_thread+0x570/0x570 [ 4156.372093] kthread+0x1d1/0x200 [ 4156.373301] ? set_kthread_struct+0x70/0x70 [ 4156.374737] ret_from_fork+0x1f/0x30 [ 4160.060463] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 4160.063832] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 4160.070346] Modules linked in: [ 4160.074299] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4160.074392] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4160.074401] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4160.074404] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4160.074422] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4160.074426] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4160.074431] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4160.074434] RDX: 0000000000003841 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4160.074436] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4160.074439] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4160.074442] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4160.074447] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4160.074450] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4160.074452] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4160.074463] Call Trace: [ 4160.074472] [ 4160.074481] ? show_regs.cold.9+0x22/0x2f [ 4160.074494] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4160.074502] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4160.074506] ? watchdog+0x40/0x40 [ 4160.074510] ? hrtimer_interrupt+0x128/0x2c0 [ 4160.074514] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4160.074525] ? apic_timer_interrupt+0xf/0x20 [ 4160.074531] [ 4160.074534] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4160.074540] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4160.074545] do_raw_spin_lock+0x177/0x190 [ 4160.074550] _raw_spin_lock+0x21/0x30 [ 4160.074557] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4160.074926] obd_mod_free+0x46/0x140 [ptlrpc] [ 4160.130943] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4160.131103] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4160.131111] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4160.131113] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4160.131126] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4160.131130] RSP: 0018:ffffbb5381503500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4160.131135] RAX: 000000000000556d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4160.131137] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4160.131139] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4160.131141] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4160.131144] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4160.131147] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4160.131150] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4160.131152] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4160.131283] Call Trace: [ 4160.131289] [ 4160.131294] ? show_regs.cold.9+0x22/0x2f [ 4160.131302] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4160.131309] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4160.131312] ? watchdog+0x40/0x40 [ 4160.131316] ? hrtimer_interrupt+0x128/0x2c0 [ 4160.131320] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4160.131327] ? apic_timer_interrupt+0xf/0x20 [ 4160.131331] [ 4160.131334] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4160.131340] do_raw_spin_lock+0x177/0x190 [ 4160.131344] _raw_spin_lock+0x21/0x30 [ 4160.131349] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4160.131652] ? do_raw_spin_unlock+0x75/0x190 [ 4160.139390] mdc_commit_open+0x91/0xb0 [mdc] [ 4160.139446] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4160.139747] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4160.151807] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4160.155595] after_reply+0xe2c/0x1780 [ptlrpc] [ 4160.155965] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4160.156172] ? __next_timer_interrupt+0x160/0x160 [ 4160.156186] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4160.156396] ? do_wait_intr+0xf0/0xf0 [ 4160.156409] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4160.156625] kthread+0x1d1/0x200 [ 4160.156632] ? set_kthread_struct+0x70/0x70 [ 4160.156636] ret_from_fork+0x1f/0x30 [ 4160.417795] ? do_wait_intr+0xf0/0xf0 [ 4160.419121] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4160.421214] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4160.423841] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4160.426176] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4160.428076] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4160.430966] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4160.435278] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4160.440411] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4160.446380] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4160.455006] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4160.460009] lmv_intent_open+0x338/0x1150 [lmv] [ 4160.463092] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4160.468380] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4160.473417] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4160.475293] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4160.479380] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4160.483573] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4160.488938] ? do_raw_spin_unlock+0x75/0x190 [ 4160.494719] ? _raw_spin_unlock+0x12/0x30 [ 4160.495705] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4160.496968] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4160.499845] ll_atomic_open+0x36a/0xd70 [lustre] [ 4160.507298] lookup_open+0x356/0xb40 [ 4160.509097] do_last+0x313/0xfc0 [ 4160.511553] ? nd_jump_root+0xe5/0x160 [ 4160.513744] ? path_init+0x437/0x520 [ 4160.515360] path_openat+0xf7/0x500 [ 4160.516884] ? do_raw_spin_unlock+0x75/0x190 [ 4160.518826] do_filp_open+0x99/0x140 [ 4160.520338] ? getname_flags+0x6e/0x330 [ 4160.521938] ? __check_object_size+0xff/0x256 [ 4160.524343] ? do_raw_spin_unlock+0x75/0x190 [ 4160.525805] ? _raw_spin_unlock+0x12/0x30 [ 4160.527794] do_sys_openat2+0x2b4/0x410 [ 4160.530451] do_sys_open+0x73/0xa0 [ 4160.531872] __x64_sys_openat+0x24/0x30 [ 4160.533340] do_syscall_64+0xc1/0x440 [ 4160.534572] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4160.536338] RIP: 0033:0x7f53425a5332 [ 4160.537882] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4160.545940] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4160.548957] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4160.551503] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4160.553955] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4160.556483] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4160.559211] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4160.999691] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1523s! [ 4161.006940] Showing busy workqueues and worker pools: [ 4161.008154] workqueue events: flags=0x0 [ 4161.012308] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4161.016958] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4161.027090] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4161.028548] pending: cpu_latency_work [ptlrpc] [ 4161.039250] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4161.049256] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4161.070584] workqueue events_long: flags=0x0 [ 4161.071602] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4161.094257] pending: lu_objects_flush [obdclass] [ 4161.096925] workqueue events_unbound: flags=0x2 [ 4161.098532] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4161.110714] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4161.113520] workqueue events_power_efficient: flags=0x80 [ 4161.121135] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4161.129259] pending: check_lifetime, do_cache_clean [ 4161.133507] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4161.136902] pending: neigh_periodic_work, neigh_periodic_work [ 4161.140082] workqueue mm_percpu_wq: flags=0x8 [ 4161.148219] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4161.152387] pending: vmstat_update, lru_add_drain_per_cpu [ 4161.157656] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4161.162045] pending: vmstat_update, lru_add_drain_per_cpu [ 4161.164439] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4161.173120] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4161.175402] workqueue xprtiod: flags=0xa [ 4161.184609] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4161.189893] pending: xprt_autoclose, xs_error_handle [ 4161.201952] workqueue ipv6_addrconf: flags=0x40008 [ 4161.209022] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4161.212190] pending: addrconf_verify_work [ 4161.214154] workqueue knbd0-recv: flags=0x1a [ 4161.216389] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4161.218453] in-flight: 651:recv_work [ 4161.224187] workqueue ptlrpc_pinger: flags=0x2 [ 4161.228914] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4161.237260] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4161.254405] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1524s workers=4 idle: 15979 31824 8 [ 4161.263809] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4151s workers=2 idle: 654 [ 4180.061796] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4180.071794] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4180.117997] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4180.126197] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4180.131260] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4180.136539] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4180.154232] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4180.158853] RAX: 0000000000005d11 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4180.168963] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4180.182812] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4180.192656] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4180.201804] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4180.208298] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4180.221572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4180.224728] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4180.229908] Call Trace: [ 4180.230950] [ 4180.232940] ? show_regs.cold.9+0x22/0x2f [ 4180.235522] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4180.241610] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4180.247413] ? watchdog+0x40/0x40 [ 4180.249051] ? hrtimer_interrupt+0x128/0x2c0 [ 4180.251100] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4180.253211] ? apic_timer_interrupt+0xf/0x20 [ 4180.254984] [ 4180.255961] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4180.259820] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4180.262389] do_raw_spin_lock+0x177/0x190 [ 4180.266100] _raw_spin_lock+0x21/0x30 [ 4180.268622] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4180.273660] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4180.275807] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4180.279380] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4180.282754] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4180.285970] ? woken_wake_function+0x30/0x30 [ 4180.290053] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4180.293559] kthread+0x1d1/0x200 [ 4180.295083] ? set_kthread_struct+0x70/0x70 [ 4180.302710] ret_from_fork+0x1f/0x30 [ 4188.055659] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 4188.057413] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 4188.060502] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 4188.060516] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4188.060588] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4188.060594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4188.060597] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4188.060611] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4188.060615] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4188.060619] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4188.060621] RDX: 0000000000001b51 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4188.060624] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4188.060626] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4188.060628] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4188.060631] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4188.060635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4188.060637] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4188.060646] Call Trace: [ 4188.060653] [ 4188.060658] ? show_regs.cold.9+0x22/0x2f [ 4188.060668] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4188.060675] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4188.060679] ? watchdog+0x40/0x40 [ 4188.060684] ? hrtimer_interrupt+0x128/0x2c0 [ 4188.060689] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4188.060697] ? apic_timer_interrupt+0xf/0x20 [ 4188.060702] [ 4188.060706] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4188.060710] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4188.060715] do_raw_spin_lock+0x177/0x190 [ 4188.060720] _raw_spin_lock+0x21/0x30 [ 4188.060726] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4188.061445] obd_mod_free+0x46/0x140 [ptlrpc] [ 4188.061657] mdc_commit_open+0x91/0xb0 [mdc] [ 4188.061704] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4188.062052] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4188.062357] after_reply+0xe2c/0x1780 [ptlrpc] [ 4188.062635] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4188.063081] ? __next_timer_interrupt+0x160/0x160 [ 4188.063092] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4188.063673] ? do_wait_intr+0xf0/0xf0 [ 4188.063684] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4188.063955] kthread+0x1d1/0x200 [ 4188.063962] ? set_kthread_struct+0x70/0x70 [ 4188.063966] ret_from_fork+0x1f/0x30 [ 4188.066978] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4188.067067] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4188.067075] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4188.067079] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4188.067098] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4188.067102] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4188.067106] RAX: 0000000000007e25 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4188.067108] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4188.067111] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4188.067113] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4188.067117] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4188.067121] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4188.067123] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4188.067126] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4188.067134] Call Trace: [ 4188.067141] [ 4188.067148] ? show_regs.cold.9+0x22/0x2f [ 4188.067161] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4188.067235] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4188.067239] ? watchdog+0x40/0x40 [ 4188.067243] ? hrtimer_interrupt+0x128/0x2c0 [ 4188.067246] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4188.067255] ? apic_timer_interrupt+0xf/0x20 [ 4188.067259] [ 4188.067262] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4188.067268] do_raw_spin_lock+0x177/0x190 [ 4188.067273] _raw_spin_lock+0x21/0x30 [ 4188.067278] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4188.067566] ? do_raw_spin_unlock+0x75/0x190 [ 4188.067571] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4188.067793] ? do_wait_intr+0xf0/0xf0 [ 4188.067803] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4188.068566] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4188.068790] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4188.069014] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4188.069148] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4188.076010] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4188.078307] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4188.078317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4188.078326] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4188.078623] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4188.078633] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4188.078637] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4188.078642] RAX: 0000000000002184 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4188.078644] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4188.078646] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4188.078648] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4188.078650] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4188.078653] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4188.078655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4188.078658] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4188.078667] Call Trace: [ 4188.078673] [ 4188.078679] ? show_regs.cold.9+0x22/0x2f [ 4188.078689] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4188.078695] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4188.078700] ? watchdog+0x40/0x40 [ 4188.078703] ? hrtimer_interrupt+0x128/0x2c0 [ 4188.078707] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4188.078714] ? apic_timer_interrupt+0xf/0x20 [ 4188.078718] [ 4188.078721] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4188.078725] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4188.078728] ? newidle_balance.isra.93+0x207/0x580 [ 4188.078734] do_raw_spin_lock+0x177/0x190 [ 4188.078739] _raw_spin_lock+0x21/0x30 [ 4188.078744] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4188.078948] process_one_work+0x2c8/0x700 [ 4188.078955] worker_thread+0x296/0x6e0 [ 4188.078960] ? rescuer_thread+0x570/0x570 [ 4188.078964] kthread+0x1d1/0x200 [ 4188.078968] ? set_kthread_struct+0x70/0x70 [ 4188.078971] ret_from_fork+0x1f/0x30 [ 4188.564513] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4188.565843] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4188.567467] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4188.573305] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4188.577947] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4188.581835] lmv_intent_open+0x338/0x1150 [lmv] [ 4188.587051] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4188.588797] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4188.593667] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4188.596150] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4188.597937] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4188.601668] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4188.603528] ? do_raw_spin_unlock+0x75/0x190 [ 4188.604688] ? _raw_spin_unlock+0x12/0x30 [ 4188.606404] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4188.607873] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4188.609473] ll_atomic_open+0x36a/0xd70 [lustre] [ 4188.611739] lookup_open+0x356/0xb40 [ 4188.613366] do_last+0x313/0xfc0 [ 4188.619348] ? nd_jump_root+0xe5/0x160 [ 4188.622715] ? path_init+0x437/0x520 [ 4188.624892] path_openat+0xf7/0x500 [ 4188.629530] ? do_raw_spin_unlock+0x75/0x190 [ 4188.631388] do_filp_open+0x99/0x140 [ 4188.634139] ? getname_flags+0x6e/0x330 [ 4188.638522] ? __check_object_size+0xff/0x256 [ 4188.639646] ? do_raw_spin_unlock+0x75/0x190 [ 4188.642061] ? _raw_spin_unlock+0x12/0x30 [ 4188.646059] do_sys_openat2+0x2b4/0x410 [ 4188.647298] do_sys_open+0x73/0xa0 [ 4188.648142] __x64_sys_openat+0x24/0x30 [ 4188.652001] do_syscall_64+0xc1/0x440 [ 4188.654551] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4188.660089] RIP: 0033:0x7f53425a5332 [ 4188.662915] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4188.682805] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4188.691189] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4188.700794] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4188.703843] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4188.707657] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4188.714609] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4191.720679] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1554s! [ 4191.727773] Showing busy workqueues and worker pools: [ 4191.730264] workqueue events: flags=0x0 [ 4191.733056] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4191.737088] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4191.745825] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4191.751073] pending: cpu_latency_work [ptlrpc] [ 4191.756693] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4191.766024] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4191.775658] workqueue events_long: flags=0x0 [ 4191.781701] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4191.789953] pending: lu_objects_flush [obdclass] [ 4191.797250] workqueue events_unbound: flags=0x2 [ 4191.800927] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4191.808613] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4191.816770] workqueue events_power_efficient: flags=0x80 [ 4191.824362] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4191.832041] pending: check_lifetime, do_cache_clean [ 4191.837911] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4191.845377] pending: neigh_periodic_work, neigh_periodic_work [ 4191.851882] workqueue mm_percpu_wq: flags=0x8 [ 4191.856050] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4191.862302] pending: vmstat_update, lru_add_drain_per_cpu [ 4191.868804] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4191.876712] pending: vmstat_update, lru_add_drain_per_cpu [ 4191.880699] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4191.884045] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4191.886966] workqueue xprtiod: flags=0xa [ 4191.888962] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4191.892062] pending: xprt_autoclose, xs_error_handle [ 4191.895486] workqueue ipv6_addrconf: flags=0x40008 [ 4191.907084] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4191.915654] pending: addrconf_verify_work [ 4191.921567] workqueue knbd0-recv: flags=0x1a [ 4191.925805] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4191.933986] in-flight: 651:recv_work [ 4191.939821] workqueue ptlrpc_pinger: flags=0x2 [ 4191.948060] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4191.954672] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4191.961906] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1554s workers=4 idle: 15979 31824 8 [ 4191.968545] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4182s workers=2 idle: 654 [ 4208.058299] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 4208.068072] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4208.119998] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4208.135710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4208.142269] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4208.147039] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4208.163635] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4208.173772] RAX: 0000000000007c85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4208.180344] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4208.185981] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4208.194761] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4208.204941] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4208.216373] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4208.229048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4208.240138] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4208.247478] Call Trace: [ 4208.252691] [ 4208.255602] ? show_regs.cold.9+0x22/0x2f [ 4208.261776] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4208.264006] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4208.267829] ? watchdog+0x40/0x40 [ 4208.270204] ? hrtimer_interrupt+0x128/0x2c0 [ 4208.273848] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4208.275618] ? apic_timer_interrupt+0xf/0x20 [ 4208.288592] [ 4208.289359] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4208.294303] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4208.301898] do_raw_spin_lock+0x177/0x190 [ 4208.305707] _raw_spin_lock+0x21/0x30 [ 4208.311264] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4208.317734] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4208.322202] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4208.326619] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4208.330719] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4208.337479] ? woken_wake_function+0x30/0x30 [ 4208.340447] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4208.345644] kthread+0x1d1/0x200 [ 4208.349627] ? set_kthread_struct+0x70/0x70 [ 4208.350786] ret_from_fork+0x1f/0x30 [ 4216.053401] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 4216.057181] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 4216.057397] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 4216.057429] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4216.057499] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4216.057505] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4216.057508] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4216.057522] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4216.057527] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4216.057530] RAX: 000000000000768a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4216.057533] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4216.057535] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4216.057537] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4216.057539] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4216.057542] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4216.057545] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4216.057547] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4216.057556] Call Trace: [ 4216.057562] [ 4216.057574] ? show_regs.cold.9+0x22/0x2f [ 4216.057721] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4216.057729] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4216.057734] ? watchdog+0x40/0x40 [ 4216.057738] ? hrtimer_interrupt+0x128/0x2c0 [ 4216.057741] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4216.057749] ? apic_timer_interrupt+0xf/0x20 [ 4216.057754] [ 4216.057757] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4216.057764] do_raw_spin_lock+0x177/0x190 [ 4216.057769] _raw_spin_lock+0x21/0x30 [ 4216.057774] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4216.058145] ? do_raw_spin_unlock+0x75/0x190 [ 4216.058151] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4216.058397] ? do_wait_intr+0xf0/0xf0 [ 4216.058410] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4216.058627] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4216.058887] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4216.059239] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4216.059474] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4216.059513] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4216.059547] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4216.059583] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4216.059710] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4216.062942] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4216.062990] lmv_intent_open+0x338/0x1150 [lmv] [ 4216.063027] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4216.063151] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4216.063306] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4216.063332] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4216.063422] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4216.063511] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4216.063601] ? do_raw_spin_unlock+0x75/0x190 [ 4216.063609] ? _raw_spin_unlock+0x12/0x30 [ 4216.063617] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4216.063650] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4216.063670] ll_atomic_open+0x36a/0xd70 [lustre] [ 4216.065029] lookup_open+0x356/0xb40 [ 4216.065038] do_last+0x313/0xfc0 [ 4216.065043] ? nd_jump_root+0xe5/0x160 [ 4216.065047] ? path_init+0x437/0x520 [ 4216.065134] path_openat+0xf7/0x500 [ 4216.065140] ? do_raw_spin_unlock+0x75/0x190 [ 4216.065146] do_filp_open+0x99/0x140 [ 4216.065197] ? getname_flags+0x6e/0x330 [ 4216.065201] ? __check_object_size+0xff/0x256 [ 4216.065209] ? do_raw_spin_unlock+0x75/0x190 [ 4216.065212] ? _raw_spin_unlock+0x12/0x30 [ 4216.065217] do_sys_openat2+0x2b4/0x410 [ 4216.065220] do_sys_open+0x73/0xa0 [ 4216.065225] __x64_sys_openat+0x24/0x30 [ 4216.065228] do_syscall_64+0xc1/0x440 [ 4216.065235] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4216.065241] RIP: 0033:0x7f53425a5332 [ 4216.065259] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4216.065262] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4216.065268] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4216.065286] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4216.065288] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4216.065290] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4216.065292] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4216.067000] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4216.067090] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4216.067099] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4216.067107] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4216.067437] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4216.067447] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4216.067451] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4216.067455] RAX: 00000000000013aa RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4216.067458] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4216.067460] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4216.067462] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4216.067464] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4216.067468] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4216.067470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4216.067473] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4216.067482] Call Trace: [ 4216.067489] [ 4216.067494] ? show_regs.cold.9+0x22/0x2f [ 4216.067503] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4216.067509] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4216.067513] ? watchdog+0x40/0x40 [ 4216.067517] ? hrtimer_interrupt+0x128/0x2c0 [ 4216.067520] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4216.067528] ? apic_timer_interrupt+0xf/0x20 [ 4216.067533] [ 4216.067535] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4216.067539] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4216.067543] ? newidle_balance.isra.93+0x207/0x580 [ 4216.067548] do_raw_spin_lock+0x177/0x190 [ 4216.067553] _raw_spin_lock+0x21/0x30 [ 4216.067559] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4216.073330] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4216.073411] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4216.073419] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4216.073421] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4216.073436] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4216.073440] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4216.073445] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4216.073627] RDX: 0000000000007f84 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4216.073631] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4216.073633] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4216.073635] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4216.073638] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4216.073641] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4216.073643] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4216.073652] Call Trace: [ 4216.073658] [ 4216.073666] ? show_regs.cold.9+0x22/0x2f [ 4216.073676] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4216.073695] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4216.073700] ? watchdog+0x40/0x40 [ 4216.073703] ? hrtimer_interrupt+0x128/0x2c0 [ 4216.073707] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4216.073717] ? apic_timer_interrupt+0xf/0x20 [ 4216.073722] [ 4216.073725] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4216.073729] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4216.073734] do_raw_spin_lock+0x177/0x190 [ 4216.073739] _raw_spin_lock+0x21/0x30 [ 4216.073745] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4216.074040] obd_mod_free+0x46/0x140 [ptlrpc] [ 4216.074381] mdc_commit_open+0x91/0xb0 [mdc] [ 4216.074428] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4216.091505] process_one_work+0x2c8/0x700 [ 4216.091529] worker_thread+0x296/0x6e0 [ 4216.091534] ? rescuer_thread+0x570/0x570 [ 4216.091539] kthread+0x1d1/0x200 [ 4216.091543] ? set_kthread_struct+0x70/0x70 [ 4216.091547] ret_from_fork+0x1f/0x30 [ 4217.016308] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4217.018739] after_reply+0xe2c/0x1780 [ptlrpc] [ 4217.020766] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4217.023084] ? __next_timer_interrupt+0x160/0x160 [ 4217.025144] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4217.027159] ? do_wait_intr+0xf0/0xf0 [ 4217.028498] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4217.030499] kthread+0x1d1/0x200 [ 4217.031787] ? set_kthread_struct+0x70/0x70 [ 4217.033160] ret_from_fork+0x1f/0x30 [ 4222.433227] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1585s! [ 4222.444865] Showing busy workqueues and worker pools: [ 4222.446656] workqueue events: flags=0x0 [ 4222.464479] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4222.476479] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4222.503173] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4222.517560] pending: cpu_latency_work [ptlrpc] [ 4222.524194] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4222.534132] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4222.545842] workqueue events_long: flags=0x0 [ 4222.547517] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4222.558412] pending: lu_objects_flush [obdclass] [ 4222.571372] workqueue events_unbound: flags=0x2 [ 4222.572321] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4222.573714] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4222.576436] workqueue events_power_efficient: flags=0x80 [ 4222.578507] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4222.581484] pending: check_lifetime, do_cache_clean [ 4222.584018] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4222.586562] pending: neigh_periodic_work, neigh_periodic_work [ 4222.589064] workqueue mm_percpu_wq: flags=0x8 [ 4222.590736] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4222.592717] pending: vmstat_update, lru_add_drain_per_cpu [ 4222.594180] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4222.596433] pending: vmstat_update, lru_add_drain_per_cpu [ 4222.598630] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4222.601065] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4222.603476] workqueue xprtiod: flags=0xa [ 4222.605017] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4222.607350] pending: xprt_autoclose, xs_error_handle [ 4222.608956] workqueue ipv6_addrconf: flags=0x40008 [ 4222.610820] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4222.613058] pending: addrconf_verify_work [ 4222.614076] workqueue knbd0-recv: flags=0x1a [ 4222.614996] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4222.616783] in-flight: 651:recv_work [ 4222.618088] workqueue ptlrpc_pinger: flags=0x2 [ 4222.619258] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4222.620627] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4222.622119] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1585s workers=4 idle: 15979 31824 8 [ 4222.623828] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4213s workers=2 idle: 654 [ 4236.058193] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 4236.066455] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4236.109278] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4236.118177] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4236.123741] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4236.129288] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4236.144692] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4236.152217] RAX: 0000000000002acf RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4236.159566] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4236.162744] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4236.169711] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4236.172851] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4236.176697] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4236.180352] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4236.183179] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4236.186171] Call Trace: [ 4236.187185] [ 4236.187956] ? show_regs.cold.9+0x22/0x2f [ 4236.189731] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4236.191419] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4236.192900] ? watchdog+0x40/0x40 [ 4236.197231] ? hrtimer_interrupt+0x128/0x2c0 [ 4236.201039] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4236.205136] ? apic_timer_interrupt+0xf/0x20 [ 4236.206747] [ 4236.207815] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4236.210758] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4236.215419] do_raw_spin_lock+0x177/0x190 [ 4236.217106] _raw_spin_lock+0x21/0x30 [ 4236.218302] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4236.222135] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4236.226045] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4236.228285] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4236.231172] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4236.234149] ? woken_wake_function+0x30/0x30 [ 4236.235529] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4236.238451] kthread+0x1d1/0x200 [ 4236.239790] ? set_kthread_struct+0x70/0x70 [ 4236.242145] ret_from_fork+0x1f/0x30 [ 4244.055637] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 4244.058078] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 4244.061750] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4244.061838] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4244.061847] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4244.061851] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4244.061867] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4244.061872] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4244.061876] RAX: 0000000000006b8f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4244.061880] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4244.061882] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4244.061884] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4244.061887] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4244.061891] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4244.061893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4244.061896] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4244.061905] Call Trace: [ 4244.061911] [ 4244.061916] ? show_regs.cold.9+0x22/0x2f [ 4244.061926] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4244.061932] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4244.061936] ? watchdog+0x40/0x40 [ 4244.061940] ? hrtimer_interrupt+0x128/0x2c0 [ 4244.061945] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4244.061953] ? apic_timer_interrupt+0xf/0x20 [ 4244.061958] [ 4244.061961] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4244.061968] do_raw_spin_lock+0x177/0x190 [ 4244.061973] _raw_spin_lock+0x21/0x30 [ 4244.061978] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4244.063422] ? do_raw_spin_unlock+0x75/0x190 [ 4244.063433] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4244.072049] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4244.072139] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4244.072148] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4244.072151] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4244.072170] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4244.072175] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4244.072180] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4244.072183] RDX: 00000000000025fb RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4244.072185] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4244.072187] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4244.072428] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4244.072432] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4244.072435] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4244.072437] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4244.072445] Call Trace: [ 4244.072452] [ 4244.072459] ? show_regs.cold.9+0x22/0x2f [ 4244.072471] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4244.072477] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4244.072481] ? watchdog+0x40/0x40 [ 4244.072486] ? hrtimer_interrupt+0x128/0x2c0 [ 4244.072489] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4244.072499] ? apic_timer_interrupt+0xf/0x20 [ 4244.072503] [ 4244.072507] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4244.072512] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4244.072516] do_raw_spin_lock+0x177/0x190 [ 4244.072520] _raw_spin_lock+0x21/0x30 [ 4244.072528] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4244.072866] obd_mod_free+0x46/0x140 [ptlrpc] [ 4244.073061] mdc_commit_open+0x91/0xb0 [mdc] [ 4244.073110] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4244.073323] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4244.073520] after_reply+0xe2c/0x1780 [ptlrpc] [ 4244.073716] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4244.074300] ? __next_timer_interrupt+0x160/0x160 [ 4244.074312] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4244.074524] ? do_wait_intr+0xf0/0xf0 [ 4244.074533] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4244.074667] kthread+0x1d1/0x200 [ 4244.074671] ? set_kthread_struct+0x70/0x70 [ 4244.074673] ret_from_fork+0x1f/0x30 [ 4244.539770] ? do_wait_intr+0xf0/0xf0 [ 4244.541071] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4244.543304] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4244.545721] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4244.550564] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4244.552510] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4244.555106] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4244.556740] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4244.561081] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4244.565566] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4244.569864] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4244.574406] lmv_intent_open+0x338/0x1150 [lmv] [ 4244.580997] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4244.583925] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4244.587364] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4244.593425] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4244.597789] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4244.600907] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4244.604352] ? do_raw_spin_unlock+0x75/0x190 [ 4244.608567] ? _raw_spin_unlock+0x12/0x30 [ 4244.611752] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4244.613598] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4244.620057] ll_atomic_open+0x36a/0xd70 [lustre] [ 4244.623057] lookup_open+0x356/0xb40 [ 4244.627359] do_last+0x313/0xfc0 [ 4244.628627] ? nd_jump_root+0xe5/0x160 [ 4244.632027] ? path_init+0x437/0x520 [ 4244.637366] path_openat+0xf7/0x500 [ 4244.641951] ? do_raw_spin_unlock+0x75/0x190 [ 4244.646899] do_filp_open+0x99/0x140 [ 4244.648999] ? getname_flags+0x6e/0x330 [ 4244.650433] ? __check_object_size+0xff/0x256 [ 4244.663089] ? do_raw_spin_unlock+0x75/0x190 [ 4244.664589] ? _raw_spin_unlock+0x12/0x30 [ 4244.667561] do_sys_openat2+0x2b4/0x410 [ 4244.669103] do_sys_open+0x73/0xa0 [ 4244.670474] __x64_sys_openat+0x24/0x30 [ 4244.672286] do_syscall_64+0xc1/0x440 [ 4244.675279] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4244.679080] RIP: 0033:0x7f53425a5332 [ 4244.680208] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4244.687925] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4244.696561] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4244.700199] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4244.702726] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4244.707884] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4244.717110] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4248.053417] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 4248.061902] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4248.111937] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4248.143928] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4248.151027] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4248.171396] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4248.186021] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4248.208553] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4248.218902] RAX: 0000000000005cad RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4248.228791] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4248.238080] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4248.250989] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4248.257937] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4248.269885] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4248.286449] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4248.298306] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4248.308519] Call Trace: [ 4248.311648] [ 4248.313631] ? show_regs.cold.9+0x22/0x2f [ 4248.318849] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4248.320489] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4248.328483] ? watchdog+0x40/0x40 [ 4248.333037] ? hrtimer_interrupt+0x128/0x2c0 [ 4248.340191] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4248.351557] ? apic_timer_interrupt+0xf/0x20 [ 4248.357035] [ 4248.357858] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4248.372907] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4248.382672] ? newidle_balance.isra.93+0x207/0x580 [ 4248.386568] do_raw_spin_lock+0x177/0x190 [ 4248.393544] _raw_spin_lock+0x21/0x30 [ 4248.399618] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4248.406403] process_one_work+0x2c8/0x700 [ 4248.410472] worker_thread+0x296/0x6e0 [ 4248.419422] ? rescuer_thread+0x570/0x570 [ 4248.424206] kthread+0x1d1/0x200 [ 4248.430357] ? set_kthread_struct+0x70/0x70 [ 4248.436574] ret_from_fork+0x1f/0x30 [ 4252.647347] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1615s! [ 4252.657919] Showing busy workqueues and worker pools: [ 4252.667555] workqueue events: flags=0x0 [ 4252.675495] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4252.686673] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4252.699692] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4252.709525] pending: cpu_latency_work [ptlrpc] [ 4252.719560] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4252.725827] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4252.735991] workqueue events_long: flags=0x0 [ 4252.742310] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4252.753371] pending: lu_objects_flush [obdclass] [ 4252.757975] workqueue events_unbound: flags=0x2 [ 4252.764723] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4252.774383] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4252.780906] workqueue events_power_efficient: flags=0x80 [ 4252.783326] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4252.790010] pending: check_lifetime, do_cache_clean [ 4252.794896] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4252.801413] pending: neigh_periodic_work, neigh_periodic_work [ 4252.805625] workqueue mm_percpu_wq: flags=0x8 [ 4252.809195] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4252.817989] pending: vmstat_update, lru_add_drain_per_cpu [ 4252.823529] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4252.830191] pending: vmstat_update, lru_add_drain_per_cpu [ 4252.835170] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4252.840420] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4252.844644] workqueue xprtiod: flags=0xa [ 4252.853426] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4252.855680] pending: xprt_autoclose, xs_error_handle [ 4252.862667] workqueue ipv6_addrconf: flags=0x40008 [ 4252.865759] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4252.870906] pending: addrconf_verify_work [ 4252.873708] workqueue knbd0-recv: flags=0x1a [ 4252.877665] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4252.883325] in-flight: 651:recv_work [ 4252.888497] workqueue ptlrpc_pinger: flags=0x2 [ 4252.891774] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4252.896773] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4252.901028] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1615s workers=4 idle: 15979 31824 8 [ 4252.908729] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4243s workers=2 idle: 654 [ 4264.060237] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 4264.069088] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4264.127360] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4264.143654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4264.154811] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4264.159614] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4264.174022] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4264.181701] RAX: 000000000000090c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4264.187328] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4264.192536] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4264.198403] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4264.200886] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4264.209013] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4264.213552] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4264.215346] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4264.222170] Call Trace: [ 4264.223943] [ 4264.225353] ? show_regs.cold.9+0x22/0x2f [ 4264.229730] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4264.234746] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4264.239534] ? watchdog+0x40/0x40 [ 4264.241198] ? hrtimer_interrupt+0x128/0x2c0 [ 4264.244719] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4264.249175] ? apic_timer_interrupt+0xf/0x20 [ 4264.252319] [ 4264.253096] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4264.255184] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4264.261134] do_raw_spin_lock+0x177/0x190 [ 4264.263953] _raw_spin_lock+0x21/0x30 [ 4264.269905] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4264.282681] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4264.295978] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4264.302150] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4264.312808] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4264.325620] ? woken_wake_function+0x30/0x30 [ 4264.331099] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4264.335570] kthread+0x1d1/0x200 [ 4264.343162] ? set_kthread_struct+0x70/0x70 [ 4264.354735] ret_from_fork+0x1f/0x30 [ 4272.056924] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 4272.057683] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 4272.068336] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4272.068414] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4272.068422] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4272.068425] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4272.068439] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4272.068443] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4272.068448] RAX: 00000000000056b6 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4272.068450] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4272.068453] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4272.068455] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4272.068458] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4272.068461] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4272.068463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4272.068466] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4272.068474] Call Trace: [ 4272.068479] [ 4272.068486] ? show_regs.cold.9+0x22/0x2f [ 4272.068496] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4272.068502] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4272.068506] ? watchdog+0x40/0x40 [ 4272.068510] ? hrtimer_interrupt+0x128/0x2c0 [ 4272.068514] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4272.068523] ? apic_timer_interrupt+0xf/0x20 [ 4272.068528] [ 4272.068531] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4272.068538] do_raw_spin_lock+0x177/0x190 [ 4272.068542] _raw_spin_lock+0x21/0x30 [ 4272.068547] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4272.078009] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4272.078084] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4272.078091] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4272.078095] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4272.078108] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4272.078113] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4272.078117] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4272.078120] RDX: 0000000000004990 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4272.078122] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4272.078125] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4272.078127] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4272.078130] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4272.078133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4272.078135] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4272.078309] Call Trace: [ 4272.078316] [ 4272.124607] ? do_raw_spin_unlock+0x75/0x190 [ 4272.124627] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4272.125216] ? do_wait_intr+0xf0/0xf0 [ 4272.125226] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4272.145877] ? show_regs.cold.9+0x22/0x2f [ 4272.145903] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4272.145910] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4272.145915] ? watchdog+0x40/0x40 [ 4272.145919] ? hrtimer_interrupt+0x128/0x2c0 [ 4272.145924] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4272.145934] ? apic_timer_interrupt+0xf/0x20 [ 4272.145939] [ 4272.145944] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4272.145951] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4272.145957] do_raw_spin_lock+0x177/0x190 [ 4272.164124] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4272.164907] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4272.165936] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4272.166572] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4272.166634] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4272.166668] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4272.166783] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4272.166952] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4272.169807] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4272.169934] lmv_intent_open+0x338/0x1150 [lmv] [ 4272.169987] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4272.172551] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4272.181208] _raw_spin_lock+0x21/0x30 [ 4272.181236] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4272.183004] obd_mod_free+0x46/0x140 [ptlrpc] [ 4272.183317] mdc_commit_open+0x91/0xb0 [mdc] [ 4272.183363] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4272.183606] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4272.183942] after_reply+0xe2c/0x1780 [ptlrpc] [ 4272.184240] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4272.184600] ? __next_timer_interrupt+0x160/0x160 [ 4272.184617] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4272.185234] ? do_wait_intr+0xf0/0xf0 [ 4272.185244] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4272.185493] kthread+0x1d1/0x200 [ 4272.185499] ? set_kthread_struct+0x70/0x70 [ 4272.185507] ret_from_fork+0x1f/0x30 [ 4272.676063] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4272.679716] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4272.685411] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4272.697510] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4272.701125] ? do_raw_spin_unlock+0x75/0x190 [ 4272.708522] ? _raw_spin_unlock+0x12/0x30 [ 4272.712404] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4272.717699] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4272.725361] ll_atomic_open+0x36a/0xd70 [lustre] [ 4272.731625] lookup_open+0x356/0xb40 [ 4272.738005] do_last+0x313/0xfc0 [ 4272.740016] ? nd_jump_root+0xe5/0x160 [ 4272.744609] ? path_init+0x437/0x520 [ 4272.750249] path_openat+0xf7/0x500 [ 4272.756024] ? do_raw_spin_unlock+0x75/0x190 [ 4272.769228] do_filp_open+0x99/0x140 [ 4272.772531] ? getname_flags+0x6e/0x330 [ 4272.782574] ? __check_object_size+0xff/0x256 [ 4272.792501] ? do_raw_spin_unlock+0x75/0x190 [ 4272.798742] ? _raw_spin_unlock+0x12/0x30 [ 4272.803363] do_sys_openat2+0x2b4/0x410 [ 4272.807967] do_sys_open+0x73/0xa0 [ 4272.824425] __x64_sys_openat+0x24/0x30 [ 4272.830144] do_syscall_64+0xc1/0x440 [ 4272.834369] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4272.839565] RIP: 0033:0x7f53425a5332 [ 4272.845644] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4272.863199] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4272.866917] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4272.874027] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4272.875828] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4272.879313] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4272.885812] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4280.052068] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 4280.063374] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4280.095036] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4280.101453] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4280.106141] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4280.109732] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4280.113736] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4280.123726] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4280.127419] RAX: 0000000000003186 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4280.131140] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4280.134748] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4280.138196] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4280.141845] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4280.145510] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4280.149734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4280.152741] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4280.156358] Call Trace: [ 4280.166969] [ 4280.167733] ? show_regs.cold.9+0x22/0x2f [ 4280.170199] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4280.172664] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4280.178298] ? watchdog+0x40/0x40 [ 4280.181709] ? hrtimer_interrupt+0x128/0x2c0 [ 4280.186202] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4280.189196] ? apic_timer_interrupt+0xf/0x20 [ 4280.193393] [ 4280.194797] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4280.200050] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4280.203733] ? newidle_balance.isra.93+0x207/0x580 [ 4280.209719] do_raw_spin_lock+0x177/0x190 [ 4280.211399] _raw_spin_lock+0x21/0x30 [ 4280.215427] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4280.223138] process_one_work+0x2c8/0x700 [ 4280.228709] worker_thread+0x296/0x6e0 [ 4280.232338] ? rescuer_thread+0x570/0x570 [ 4280.234717] kthread+0x1d1/0x200 [ 4280.237863] ? set_kthread_struct+0x70/0x70 [ 4280.240624] ret_from_fork+0x1f/0x30 [ 4283.367410] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1646s! [ 4283.376896] Showing busy workqueues and worker pools: [ 4283.379396] workqueue events: flags=0x0 [ 4283.381774] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4283.387284] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4283.395480] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4283.399240] pending: cpu_latency_work [ptlrpc] [ 4283.404010] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4283.408700] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4283.414782] workqueue events_long: flags=0x0 [ 4283.417284] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4283.420904] pending: lu_objects_flush [obdclass] [ 4283.423958] workqueue events_unbound: flags=0x2 [ 4283.425876] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4283.429891] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4283.433192] workqueue events_power_efficient: flags=0x80 [ 4283.436093] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4283.439886] pending: check_lifetime, do_cache_clean [ 4283.443601] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4283.448448] pending: neigh_periodic_work, neigh_periodic_work [ 4283.452361] workqueue mm_percpu_wq: flags=0x8 [ 4283.457219] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4283.467361] pending: vmstat_update, lru_add_drain_per_cpu [ 4283.472174] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4283.477239] pending: vmstat_update, lru_add_drain_per_cpu [ 4283.480413] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4283.485947] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4283.492297] workqueue xprtiod: flags=0xa [ 4283.495222] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4283.504216] pending: xprt_autoclose, xs_error_handle [ 4283.506925] workqueue ipv6_addrconf: flags=0x40008 [ 4283.514092] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4283.532230] pending: addrconf_verify_work [ 4283.539375] workqueue knbd0-recv: flags=0x1a [ 4283.544178] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4283.548367] in-flight: 651:recv_work [ 4283.561337] workqueue ptlrpc_pinger: flags=0x2 [ 4283.569194] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4283.574448] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4283.581851] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1646s workers=4 idle: 15979 31824 8 [ 4283.588482] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4274s workers=2 idle: 654 [ 4292.062268] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 4292.064907] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4292.097598] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4292.106490] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4292.121332] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4292.129597] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4292.154170] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4292.168648] RAX: 0000000000005109 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4292.181966] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4292.192632] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4292.200479] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4292.213384] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4292.229381] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4292.238341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4292.243224] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4292.252509] Call Trace: [ 4292.256603] [ 4292.260153] ? show_regs.cold.9+0x22/0x2f [ 4292.268530] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4292.275950] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4292.284201] ? watchdog+0x40/0x40 [ 4292.291451] ? hrtimer_interrupt+0x128/0x2c0 [ 4292.298445] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4292.307238] ? apic_timer_interrupt+0xf/0x20 [ 4292.310488] [ 4292.318632] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4292.322609] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4292.335104] do_raw_spin_lock+0x177/0x190 [ 4292.355876] _raw_spin_lock+0x21/0x30 [ 4292.357534] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4292.370015] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4292.378166] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4292.395933] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4292.410379] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4292.423228] ? woken_wake_function+0x30/0x30 [ 4292.449624] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4292.467931] kthread+0x1d1/0x200 [ 4292.471601] ? set_kthread_struct+0x70/0x70 [ 4292.478530] ret_from_fork+0x1f/0x30 [ 4300.055032] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 4300.057023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 4300.060004] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4300.060092] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4300.060099] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4300.060102] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4300.060116] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4300.060120] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4300.060124] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4300.060126] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4300.060128] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4300.060130] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4300.060132] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4300.060135] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4300.060138] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4300.060140] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4300.060149] Call Trace: [ 4300.060155] [ 4300.060161] ? show_regs.cold.9+0x22/0x2f [ 4300.060170] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4300.060176] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4300.060181] ? watchdog+0x40/0x40 [ 4300.060185] ? hrtimer_interrupt+0x128/0x2c0 [ 4300.060188] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4300.060194] ? apic_timer_interrupt+0xf/0x20 [ 4300.060199] [ 4300.060201] ? native_safe_halt+0x2e/0x50 [ 4300.060205] kvm_wait+0xcd/0xe0 [ 4300.060263] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4300.065277] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) [ 4300.094933] do_raw_spin_lock+0x177/0x190 [ 4300.094953] _raw_spin_lock+0x21/0x30 [ 4300.094959] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4300.095450] ? do_raw_spin_unlock+0x75/0x190 [ 4300.095458] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4300.106049] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4300.106119] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4300.106127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4300.106130] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4300.106143] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4300.106147] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4300.106152] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4300.106154] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4300.106156] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4300.106158] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4300.106161] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4300.106163] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4300.106166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4300.106168] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4300.106177] Call Trace: [ 4300.106182] [ 4300.106188] ? show_regs.cold.9+0x22/0x2f [ 4300.106198] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4300.106204] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4300.106208] ? watchdog+0x40/0x40 [ 4300.116760] ? do_wait_intr+0xf0/0xf0 [ 4300.116783] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4300.117051] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4300.119691] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4300.119918] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4300.120008] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4300.120037] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4300.120056] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4300.120074] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4300.120129] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4300.120494] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4300.120515] lmv_intent_open+0x338/0x1150 [lmv] [ 4300.120536] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4300.120593] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4300.120780] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4300.120797] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4300.120853] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4300.120908] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4300.120962] ? do_raw_spin_unlock+0x75/0x190 [ 4300.120968] ? _raw_spin_unlock+0x12/0x30 [ 4300.120973] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4300.120993] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4300.121006] ll_atomic_open+0x36a/0xd70 [lustre] [ 4300.121062] lookup_open+0x356/0xb40 [ 4300.121069] do_last+0x313/0xfc0 [ 4300.121073] ? nd_jump_root+0xe5/0x160 [ 4300.121076] ? path_init+0x437/0x520 [ 4300.121079] path_openat+0xf7/0x500 [ 4300.121082] ? do_raw_spin_unlock+0x75/0x190 [ 4300.121085] do_filp_open+0x99/0x140 [ 4300.121090] ? getname_flags+0x6e/0x330 [ 4300.121093] ? __check_object_size+0xff/0x256 [ 4300.121098] ? do_raw_spin_unlock+0x75/0x190 [ 4300.121101] ? _raw_spin_unlock+0x12/0x30 [ 4300.121104] do_sys_openat2+0x2b4/0x410 [ 4300.121108] do_sys_open+0x73/0xa0 [ 4300.121111] __x64_sys_openat+0x24/0x30 [ 4300.121114] do_syscall_64+0xc1/0x440 [ 4300.121119] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4300.121124] RIP: 0033:0x7f53425a5332 [ 4300.121130] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4300.121133] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4300.121137] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4300.121139] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4300.121141] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4300.121142] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4300.121144] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4300.419622] ? hrtimer_interrupt+0x128/0x2c0 [ 4300.420697] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4300.421828] ? apic_timer_interrupt+0xf/0x20 [ 4300.422840] [ 4300.423590] ? native_safe_halt+0x2e/0x50 [ 4300.424816] kvm_wait+0xcd/0xe0 [ 4300.425840] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4300.427972] do_raw_spin_lock+0x177/0x190 [ 4300.428931] _raw_spin_lock+0x21/0x30 [ 4300.429858] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4300.431494] obd_mod_free+0x46/0x140 [ptlrpc] [ 4300.432938] mdc_commit_open+0x91/0xb0 [mdc] [ 4300.433982] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4300.435449] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4300.437017] after_reply+0xe2c/0x1780 [ptlrpc] [ 4300.438237] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4300.440360] ? __next_timer_interrupt+0x160/0x160 [ 4300.442107] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4300.444157] ? do_wait_intr+0xf0/0xf0 [ 4300.445351] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4300.446836] kthread+0x1d1/0x200 [ 4300.447691] ? set_kthread_struct+0x70/0x70 [ 4300.448743] ret_from_fork+0x1f/0x30 [ 4308.059657] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 4308.064547] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4308.115219] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4308.131589] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4308.142604] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4308.153226] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4308.155459] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4308.188515] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4308.196600] RAX: 0000000000006073 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4308.206652] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4308.220219] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4308.228593] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4308.238644] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4308.243888] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4308.255790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4308.257480] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4308.271514] Call Trace: [ 4308.273840] [ 4308.283336] ? show_regs.cold.9+0x22/0x2f [ 4308.294454] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4308.305017] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4308.312886] ? watchdog+0x40/0x40 [ 4308.319460] ? hrtimer_interrupt+0x128/0x2c0 [ 4308.329836] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4308.336004] ? apic_timer_interrupt+0xf/0x20 [ 4308.344868] [ 4308.349653] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4308.359616] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4308.366598] ? newidle_balance.isra.93+0x207/0x580 [ 4308.368059] do_raw_spin_lock+0x177/0x190 [ 4308.378744] _raw_spin_lock+0x21/0x30 [ 4308.385823] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4308.396693] process_one_work+0x2c8/0x700 [ 4308.409750] worker_thread+0x296/0x6e0 [ 4308.411251] ? rescuer_thread+0x570/0x570 [ 4308.416828] kthread+0x1d1/0x200 [ 4308.426728] ? set_kthread_struct+0x70/0x70 [ 4308.436382] ret_from_fork+0x1f/0x30 [ 4314.084021] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1676s! [ 4314.088797] Showing busy workqueues and worker pools: [ 4314.094521] workqueue events: flags=0x0 [ 4314.098762] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4314.105529] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4314.114708] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4314.120924] pending: cpu_latency_work [ptlrpc] [ 4314.127623] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4314.134868] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4314.141733] workqueue events_long: flags=0x0 [ 4314.146406] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4314.154255] pending: lu_objects_flush [obdclass] [ 4314.159913] workqueue events_unbound: flags=0x2 [ 4314.169666] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4314.174656] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4314.182227] workqueue events_power_efficient: flags=0x80 [ 4314.189539] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4314.196807] pending: check_lifetime, do_cache_clean [ 4314.200819] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4314.205693] pending: neigh_periodic_work, neigh_periodic_work [ 4314.211674] workqueue mm_percpu_wq: flags=0x8 [ 4314.215748] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4314.224292] pending: vmstat_update, lru_add_drain_per_cpu [ 4314.228887] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4314.236686] pending: vmstat_update, lru_add_drain_per_cpu [ 4314.238775] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4314.248098] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4314.255213] workqueue xprtiod: flags=0xa [ 4314.259892] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4314.268226] pending: xprt_autoclose, xs_error_handle [ 4314.271145] workqueue ipv6_addrconf: flags=0x40008 [ 4314.277454] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4314.282215] pending: addrconf_verify_work [ 4314.287059] workqueue knbd0-recv: flags=0x1a [ 4314.292822] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4314.298422] in-flight: 651:recv_work [ 4314.300454] workqueue ptlrpc_pinger: flags=0x2 [ 4314.301842] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4314.308822] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4314.319691] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1677s workers=4 idle: 15979 31824 8 [ 4314.328492] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4304s workers=2 idle: 654 [ 4314.484395] rcu: INFO: rcu_sched self-detected stall on CPU [ 4314.488750] rcu: 2-....: (1161951 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 4314.505468] (t=2220058 jiffies g=274185 q=64216) [ 4314.516177] rcu: rcu_sched kthread starved for 1678424 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 4314.539565] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4314.553830] rcu: RCU grace-period kthread stack dump: [ 4314.561776] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4314.575313] Call Trace: [ 4314.581204] __schedule+0x351/0xcb0 [ 4314.586559] schedule+0xc0/0x180 [ 4314.587685] schedule_timeout+0xb4/0x190 [ 4314.589892] ? do_raw_spin_unlock+0x75/0x190 [ 4314.592884] ? __next_timer_interrupt+0x160/0x160 [ 4314.602627] ? prepare_to_swait_event+0xf3/0x140 [ 4314.607111] rcu_gp_kthread+0x8dd/0x15b0 [ 4314.613509] ? force_qs_rnp+0x2c0/0x2c0 [ 4314.614944] kthread+0x1d1/0x200 [ 4314.623023] ? set_kthread_struct+0x70/0x70 [ 4314.629156] ret_from_fork+0x1f/0x30 [ 4314.631892] rcu: Stack dump where RCU GP kthread last ran: [ 4314.638257] Sending NMI from CPU 2 to CPUs 1: [ 4314.651763] NMI backtrace for cpu 1 [ 4314.651771] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4314.651773] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4314.651775] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4314.651777] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4314.651779] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [ 4314.651781] RAX: 0000000000007f89 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4314.651783] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4314.651784] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4314.651785] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4314.651786] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4314.651787] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4314.651788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4314.651790] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4314.651791] Call Trace: [ 4314.651792] [ 4314.651792] ? show_regs.cold.9+0x22/0x2f [ 4314.651793] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4314.651794] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4314.651799] ? nmi_handle+0xbd/0x210 [ 4314.651800] ? default_do_nmi+0x4b/0x1b0 [ 4314.651802] ? do_nmi+0x280/0x380 [ 4314.651802] ? end_repeat_nmi+0x16/0x1f [ 4314.651803] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4314.651804] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4314.651805] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4314.651807] [ 4314.651808] do_raw_spin_lock+0x177/0x190 [ 4314.651809] _raw_spin_lock+0x21/0x30 [ 4314.651810] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4314.651811] ? do_raw_spin_unlock+0x75/0x190 [ 4314.651812] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4314.651813] ? do_wait_intr+0xf0/0xf0 [ 4314.651814] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4314.651815] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4314.651816] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4314.651817] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4314.651819] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4314.651820] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4314.651821] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4314.651822] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4314.651823] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4314.651824] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4314.651825] lmv_intent_open+0x338/0x1150 [lmv] [ 4314.651826] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4314.651827] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4314.651828] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4314.651829] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4314.651831] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4314.651832] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4314.651833] ? do_raw_spin_unlock+0x75/0x190 [ 4314.651834] ? _raw_spin_unlock+0x12/0x30 [ 4314.651835] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4314.651836] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4314.651837] ll_atomic_open+0x36a/0xd70 [lustre] [ 4314.651841] lookup_open+0x356/0xb40 [ 4314.651843] do_last+0x313/0xfc0 [ 4314.651844] ? nd_jump_root+0xe5/0x160 [ 4314.651846] ? path_init+0x437/0x520 [ 4314.651847] path_openat+0xf7/0x500 [ 4314.651848] ? do_raw_spin_unlock+0x75/0x190 [ 4314.651849] do_filp_open+0x99/0x140 [ 4314.651850] ? getname_flags+0x6e/0x330 [ 4314.651851] ? __check_object_size+0xff/0x256 [ 4314.651852] ? do_raw_spin_unlock+0x75/0x190 [ 4314.651852] ? _raw_spin_unlock+0x12/0x30 [ 4314.651854] do_sys_openat2+0x2b4/0x410 [ 4314.651855] do_sys_open+0x73/0xa0 [ 4314.651856] __x64_sys_openat+0x24/0x30 [ 4314.651857] do_syscall_64+0xc1/0x440 [ 4314.651858] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4314.651859] RIP: 0033:0x7f53425a5332 [ 4314.651860] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4314.651863] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4314.651866] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4314.651867] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4314.651868] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4314.651869] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4314.651870] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4314.657081] Sending NMI from CPU 2 to CPUs 1: [ 4315.027494] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 4315.029618] NMI backtrace for cpu 2 [ 4315.030577] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4315.035923] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4315.039502] Call Trace: [ 4315.040342] [ 4315.045132] ? dump_stack+0xbb/0x10e [ 4315.047997] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4315.055412] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4315.059317] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4315.070347] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4315.074314] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4315.083580] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4315.088267] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4315.091277] ? update_process_times+0x6d/0xc0 [ 4315.093943] ? tick_sched_handle.isra.18+0x27/0x90 [ 4315.098394] ? tick_sched_timer+0x53/0xb0 [ 4315.103847] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4315.105305] ? tick_sched_handle.isra.18+0x90/0x90 [ 4315.111340] ? hrtimer_interrupt+0x128/0x2c0 [ 4315.114441] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4315.118294] ? apic_timer_interrupt+0xf/0x20 [ 4315.121081] [ 4315.122221] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4315.124324] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4315.128032] ? do_raw_spin_lock+0x177/0x190 [ 4315.130273] ? _raw_spin_lock+0x21/0x30 [ 4315.132779] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4315.135697] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4315.141338] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4315.145202] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4315.149446] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4315.158112] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4315.160201] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4315.164452] ? __next_timer_interrupt+0x160/0x160 [ 4315.167094] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4315.169445] ? do_wait_intr+0xf0/0xf0 [ 4315.171187] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4315.173757] ? kthread+0x1d1/0x200 [ 4315.175331] ? set_kthread_struct+0x70/0x70 [ 4315.177413] ? ret_from_fork+0x1f/0x30 [ 4320.060129] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4320.071043] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4320.106470] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4320.115405] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4320.120655] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4320.125547] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4320.142341] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4320.152469] RAX: 0000000000007e08 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4320.157482] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4320.163672] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4320.174634] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4320.181506] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4320.186601] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4320.197771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4320.206043] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4320.214104] Call Trace: [ 4320.215534] [ 4320.216657] ? show_regs.cold.9+0x22/0x2f [ 4320.218929] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4320.222001] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4320.224053] ? watchdog+0x40/0x40 [ 4320.225652] ? hrtimer_interrupt+0x128/0x2c0 [ 4320.227176] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4320.233097] ? apic_timer_interrupt+0xf/0x20 [ 4320.235185] [ 4320.237032] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4320.240611] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4320.245192] do_raw_spin_lock+0x177/0x190 [ 4320.247746] _raw_spin_lock+0x21/0x30 [ 4320.253124] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4320.255448] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4320.266698] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4320.270654] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4320.277247] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4320.281311] ? woken_wake_function+0x30/0x30 [ 4320.283761] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4320.288757] kthread+0x1d1/0x200 [ 4320.291602] ? set_kthread_struct+0x70/0x70 [ 4320.300755] ret_from_fork+0x1f/0x30 [ 4340.055281] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 4340.059669] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 4340.060359] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4340.060539] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4340.060545] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4340.060549] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4340.060573] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4340.060577] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4340.060581] RAX: 0000000000003c85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4340.060583] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4340.060586] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4340.060588] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4340.060590] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4340.060594] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4340.060597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4340.060599] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4340.060607] Call Trace: [ 4340.060624] [ 4340.060630] ? show_regs.cold.9+0x22/0x2f [ 4340.060641] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4340.060647] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4340.060651] ? watchdog+0x40/0x40 [ 4340.060655] ? hrtimer_interrupt+0x128/0x2c0 [ 4340.060659] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4340.060667] ? apic_timer_interrupt+0xf/0x20 [ 4340.060671] [ 4340.060674] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4340.060680] do_raw_spin_lock+0x177/0x190 [ 4340.060685] _raw_spin_lock+0x21/0x30 [ 4340.063353] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 4340.060690] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4340.063709] ? do_raw_spin_unlock+0x75/0x190 [ 4340.063719] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4340.063955] ? do_wait_intr+0xf0/0xf0 [ 4340.063964] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4340.064157] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4340.064381] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4340.064593] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4340.066069] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4340.066115] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4340.066148] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4340.066177] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4340.066257] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4340.066562] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4340.066597] lmv_intent_open+0x338/0x1150 [lmv] [ 4340.066631] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4340.066922] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4340.067038] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4340.067063] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4340.067158] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4340.067254] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4340.067351] ? do_raw_spin_unlock+0x75/0x190 [ 4340.067358] ? _raw_spin_unlock+0x12/0x30 [ 4340.067365] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4340.067393] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4340.067414] ll_atomic_open+0x36a/0xd70 [lustre] [ 4340.067499] lookup_open+0x356/0xb40 [ 4340.067508] do_last+0x313/0xfc0 [ 4340.067531] ? nd_jump_root+0xe5/0x160 [ 4340.067536] ? path_init+0x437/0x520 [ 4340.067541] path_openat+0xf7/0x500 [ 4340.067546] ? do_raw_spin_unlock+0x75/0x190 [ 4340.067551] do_filp_open+0x99/0x140 [ 4340.067556] ? getname_flags+0x6e/0x330 [ 4340.067562] ? __check_object_size+0xff/0x256 [ 4340.067569] ? do_raw_spin_unlock+0x75/0x190 [ 4340.067573] ? _raw_spin_unlock+0x12/0x30 [ 4340.067577] do_sys_openat2+0x2b4/0x410 [ 4340.067581] do_sys_open+0x73/0xa0 [ 4340.067585] __x64_sys_openat+0x24/0x30 [ 4340.067588] do_syscall_64+0xc1/0x440 [ 4340.067594] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4340.067601] RIP: 0033:0x7f53425a5332 [ 4340.067609] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4340.067612] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4340.067617] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4340.067619] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4340.067621] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4340.067623] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4340.067625] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4340.070438] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4340.070737] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4340.070744] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4340.070746] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4340.070759] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4340.070763] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4340.070768] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4340.070770] RDX: 0000000000007525 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4340.070772] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4340.070774] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4340.070776] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4340.070779] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4340.070782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4340.070785] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4340.070794] Call Trace: [ 4340.070799] [ 4340.070804] ? show_regs.cold.9+0x22/0x2f [ 4340.070814] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4340.070820] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4340.070824] ? watchdog+0x40/0x40 [ 4340.070828] ? hrtimer_interrupt+0x128/0x2c0 [ 4340.070832] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4340.070840] ? apic_timer_interrupt+0xf/0x20 [ 4340.070845] [ 4340.070848] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4340.070852] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4340.070857] do_raw_spin_lock+0x177/0x190 [ 4340.070861] _raw_spin_lock+0x21/0x30 [ 4340.070868] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4340.071127] obd_mod_free+0x46/0x140 [ptlrpc] [ 4340.071803] mdc_commit_open+0x91/0xb0 [mdc] [ 4340.071842] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4340.072032] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4340.072734] after_reply+0xe2c/0x1780 [ptlrpc] [ 4340.072985] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4340.073502] ? __next_timer_interrupt+0x160/0x160 [ 4340.073577] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4340.073836] ? do_wait_intr+0xf0/0xf0 [ 4340.073845] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4340.074039] kthread+0x1d1/0x200 [ 4340.074045] ? set_kthread_struct+0x70/0x70 [ 4340.074049] ret_from_fork+0x1f/0x30 [ 4340.881619] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4340.923023] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4340.934471] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4340.942840] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4340.950145] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4340.957460] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4340.974570] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4340.984759] RAX: 0000000000003e0e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4340.992458] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4340.999297] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4341.003178] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4341.007600] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4341.022870] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4341.035038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4341.037646] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4341.044686] Call Trace: [ 4341.046354] [ 4341.050227] ? show_regs.cold.9+0x22/0x2f [ 4341.056458] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4341.066086] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4341.068819] ? watchdog+0x40/0x40 [ 4341.077655] ? hrtimer_interrupt+0x128/0x2c0 [ 4341.079146] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4341.083721] ? apic_timer_interrupt+0xf/0x20 [ 4341.088318] [ 4341.089349] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4341.092491] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4341.095895] ? newidle_balance.isra.93+0x207/0x580 [ 4341.098427] do_raw_spin_lock+0x177/0x190 [ 4341.100299] _raw_spin_lock+0x21/0x30 [ 4341.101949] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4341.105491] process_one_work+0x2c8/0x700 [ 4341.107233] worker_thread+0x296/0x6e0 [ 4341.110231] ? rescuer_thread+0x570/0x570 [ 4341.113291] kthread+0x1d1/0x200 [ 4341.115892] ? set_kthread_struct+0x70/0x70 [ 4341.118183] ret_from_fork+0x1f/0x30 [ 4344.800118] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1707s! [ 4344.802850] Showing busy workqueues and worker pools: [ 4344.805396] workqueue events: flags=0x0 [ 4344.807444] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4344.810828] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4344.816569] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4344.819604] pending: cpu_latency_work [ptlrpc] [ 4344.822509] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4344.825572] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4344.830107] workqueue events_long: flags=0x0 [ 4344.832821] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4344.836301] pending: lu_objects_flush [obdclass] [ 4344.839670] workqueue events_unbound: flags=0x2 [ 4344.843642] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4344.847225] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4344.850978] workqueue events_power_efficient: flags=0x80 [ 4344.855810] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4344.859153] pending: check_lifetime, do_cache_clean [ 4344.863099] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4344.866488] pending: neigh_periodic_work, neigh_periodic_work [ 4344.870172] workqueue mm_percpu_wq: flags=0x8 [ 4344.872526] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4344.876179] pending: vmstat_update, lru_add_drain_per_cpu [ 4344.879538] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4344.883411] pending: vmstat_update, lru_add_drain_per_cpu [ 4344.886531] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4344.890337] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4344.894224] workqueue xprtiod: flags=0xa [ 4344.896295] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4344.899696] pending: xprt_autoclose, xs_error_handle [ 4344.902140] workqueue ipv6_addrconf: flags=0x40008 [ 4344.904839] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4344.908284] pending: addrconf_verify_work [ 4344.910709] workqueue knbd0-recv: flags=0x1a [ 4344.913583] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4344.916855] in-flight: 651:recv_work [ 4344.918994] workqueue ptlrpc_pinger: flags=0x2 [ 4344.921915] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4344.925423] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4344.928710] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1707s workers=4 idle: 15979 31824 8 [ 4344.933377] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4335s workers=2 idle: 654 [ 4348.058010] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4348.068346] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4348.109655] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4348.114689] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4348.119519] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4348.126280] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4348.135970] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4348.140505] RAX: 000000000000430f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4348.143763] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4348.147047] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4348.150436] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4348.152491] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4348.155025] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4348.162059] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4348.166849] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4348.172263] Call Trace: [ 4348.173218] [ 4348.173926] ? show_regs.cold.9+0x22/0x2f [ 4348.175266] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4348.182307] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4348.185307] ? watchdog+0x40/0x40 [ 4348.187770] ? hrtimer_interrupt+0x128/0x2c0 [ 4348.190036] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4348.194618] ? apic_timer_interrupt+0xf/0x20 [ 4348.196767] [ 4348.198660] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4348.203958] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4348.206275] do_raw_spin_lock+0x177/0x190 [ 4348.208524] _raw_spin_lock+0x21/0x30 [ 4348.209841] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4348.213735] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4348.216253] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4348.218270] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4348.223171] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4348.226688] ? woken_wake_function+0x30/0x30 [ 4348.228297] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4348.232627] kthread+0x1d1/0x200 [ 4348.235615] ? set_kthread_struct+0x70/0x70 [ 4348.237683] ret_from_fork+0x1f/0x30 [ 4368.055616] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 4368.058744] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 4368.062758] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4368.062849] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4368.062859] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4368.062862] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4368.062877] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4368.062882] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4368.062886] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4368.062889] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4368.062891] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4368.062894] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4368.062896] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4368.062900] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4368.062903] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4368.062906] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4368.062915] Call Trace: [ 4368.062921] [ 4368.062927] ? show_regs.cold.9+0x22/0x2f [ 4368.062937] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4368.062942] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4368.062947] ? watchdog+0x40/0x40 [ 4368.062952] ? hrtimer_interrupt+0x128/0x2c0 [ 4368.062956] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4368.062963] ? apic_timer_interrupt+0xf/0x20 [ 4368.062968] [ 4368.062971] ? native_safe_halt+0x2e/0x50 [ 4368.062975] kvm_wait+0xcd/0xe0 [ 4368.062983] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4368.062991] do_raw_spin_lock+0x177/0x190 [ 4368.062996] _raw_spin_lock+0x21/0x30 [ 4368.063000] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4368.070777] Modules linked in: [ 4368.097013] ? do_raw_spin_unlock+0x75/0x190 [ 4368.113410] lustre(O) [ 4368.116591] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4368.120477] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4368.120627] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4368.120636] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4368.120639] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4368.120654] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4368.120660] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4368.120664] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4368.120667] RDX: 0000000000007d21 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4368.120669] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4368.120671] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4368.120674] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4368.120678] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4368.120680] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4368.120683] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4368.120691] Call Trace: [ 4368.120698] [ 4368.120706] ? show_regs.cold.9+0x22/0x2f [ 4368.120720] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4368.120728] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4368.120732] ? watchdog+0x40/0x40 [ 4368.120736] ? hrtimer_interrupt+0x128/0x2c0 [ 4368.120740] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4368.120750] ? apic_timer_interrupt+0xf/0x20 [ 4368.120755] [ 4368.120758] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4368.120762] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4368.120767] do_raw_spin_lock+0x177/0x190 [ 4368.120771] _raw_spin_lock+0x21/0x30 [ 4368.120778] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4368.121599] obd_mod_free+0x46/0x140 [ptlrpc] [ 4368.121816] mdc_commit_open+0x91/0xb0 [mdc] [ 4368.121858] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4368.135494] ? do_wait_intr+0xf0/0xf0 [ 4368.142755] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4368.145510] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4368.151695] after_reply+0xe2c/0x1780 [ptlrpc] [ 4368.158402] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4368.158781] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4368.158992] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4368.159163] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4368.159220] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4368.159251] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4368.159278] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4368.159364] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4368.159577] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4368.159604] lmv_intent_open+0x338/0x1150 [lmv] [ 4368.159633] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4368.159718] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4368.159800] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4368.159822] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4368.159904] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4368.159988] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4368.167746] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4368.176040] ? do_raw_spin_unlock+0x75/0x190 [ 4368.176069] ? _raw_spin_unlock+0x12/0x30 [ 4368.176078] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4368.176113] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4368.176133] ll_atomic_open+0x36a/0xd70 [lustre] [ 4368.176273] lookup_open+0x356/0xb40 [ 4368.176283] do_last+0x313/0xfc0 [ 4368.176288] ? nd_jump_root+0xe5/0x160 [ 4368.176293] ? path_init+0x437/0x520 [ 4368.176298] path_openat+0xf7/0x500 [ 4368.176303] ? do_raw_spin_unlock+0x75/0x190 [ 4368.176308] do_filp_open+0x99/0x140 [ 4368.176316] ? getname_flags+0x6e/0x330 [ 4368.176320] ? __check_object_size+0xff/0x256 [ 4368.176328] ? do_raw_spin_unlock+0x75/0x190 [ 4368.176332] ? _raw_spin_unlock+0x12/0x30 [ 4368.176336] do_sys_openat2+0x2b4/0x410 [ 4368.176341] do_sys_open+0x73/0xa0 [ 4368.176346] __x64_sys_openat+0x24/0x30 [ 4368.176349] do_syscall_64+0xc1/0x440 [ 4368.176358] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4368.176364] RIP: 0033:0x7f53425a5332 [ 4368.176425] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4368.176432] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4368.176439] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4368.176441] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4368.176444] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4368.176446] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4368.176448] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4368.696823] ? __next_timer_interrupt+0x160/0x160 [ 4368.701979] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4368.707404] ? do_wait_intr+0xf0/0xf0 [ 4368.709745] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4368.713756] kthread+0x1d1/0x200 [ 4368.715719] ? set_kthread_struct+0x70/0x70 [ 4368.718736] ret_from_fork+0x1f/0x30 [ 4372.052735] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 4372.061140] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4372.085740] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4372.096057] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4372.102644] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4372.114293] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4372.122268] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4372.143187] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4372.147928] RAX: 0000000000007f86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4372.152748] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4372.157152] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4372.161965] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4372.168857] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4372.176943] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4372.184379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4372.190874] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4372.195791] Call Trace: [ 4372.197214] [ 4372.198306] ? show_regs.cold.9+0x22/0x2f [ 4372.200356] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4372.203261] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4372.204909] ? watchdog+0x40/0x40 [ 4372.207267] ? hrtimer_interrupt+0x128/0x2c0 [ 4372.210026] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4372.217763] ? apic_timer_interrupt+0xf/0x20 [ 4372.221391] [ 4372.222505] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4372.226388] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4372.230208] ? newidle_balance.isra.93+0x207/0x580 [ 4372.233026] do_raw_spin_lock+0x177/0x190 [ 4372.239941] _raw_spin_lock+0x21/0x30 [ 4372.243458] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4372.246837] process_one_work+0x2c8/0x700 [ 4372.248923] worker_thread+0x296/0x6e0 [ 4372.250809] ? rescuer_thread+0x570/0x570 [ 4372.252924] kthread+0x1d1/0x200 [ 4372.254798] ? set_kthread_struct+0x70/0x70 [ 4372.257256] ret_from_fork+0x1f/0x30 [ 4375.018155] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1737s! [ 4375.024700] Showing busy workqueues and worker pools: [ 4375.027601] workqueue events: flags=0x0 [ 4375.030457] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4375.034292] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4375.067465] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4375.075891] pending: cpu_latency_work [ptlrpc] [ 4375.088773] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4375.104744] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4375.132106] workqueue events_long: flags=0x0 [ 4375.138753] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4375.150653] pending: lu_objects_flush [obdclass] [ 4375.152747] workqueue events_unbound: flags=0x2 [ 4375.166893] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4375.177674] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4375.179793] workqueue events_power_efficient: flags=0x80 [ 4375.186605] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4375.199944] pending: check_lifetime, do_cache_clean [ 4375.201837] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4375.214005] pending: neigh_periodic_work, neigh_periodic_work [ 4375.220649] workqueue mm_percpu_wq: flags=0x8 [ 4375.227069] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4375.238579] pending: vmstat_update, lru_add_drain_per_cpu [ 4375.243835] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4375.253665] pending: vmstat_update, lru_add_drain_per_cpu [ 4375.262352] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4375.271536] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4375.276525] workqueue xprtiod: flags=0xa [ 4375.281375] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4375.286786] pending: xprt_autoclose, xs_error_handle [ 4375.291856] workqueue ipv6_addrconf: flags=0x40008 [ 4375.298521] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4375.302661] pending: addrconf_verify_work [ 4375.306075] workqueue knbd0-recv: flags=0x1a [ 4375.310225] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4375.318735] in-flight: 651:recv_work [ 4375.321544] workqueue ptlrpc_pinger: flags=0x2 [ 4375.329142] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4375.333659] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4375.345163] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1738s workers=4 idle: 15979 31824 8 [ 4375.355184] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4365s workers=2 idle: 654 [ 4376.061515] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4376.066223] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4376.106518] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4376.117053] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4376.120526] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4376.123405] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4376.136683] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4376.146564] RAX: 0000000000005f08 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4376.149618] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4376.157739] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4376.171723] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4376.175475] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4376.180790] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4376.194190] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4376.204406] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4376.219129] Call Trace: [ 4376.220107] [ 4376.220812] ? show_regs.cold.9+0x22/0x2f [ 4376.227383] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4376.229213] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4376.242370] ? watchdog+0x40/0x40 [ 4376.246838] ? hrtimer_interrupt+0x128/0x2c0 [ 4376.254114] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4376.255669] ? apic_timer_interrupt+0xf/0x20 [ 4376.262049] [ 4376.263284] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4376.265497] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4376.269720] do_raw_spin_lock+0x177/0x190 [ 4376.277796] _raw_spin_lock+0x21/0x30 [ 4376.286187] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4376.295807] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4376.299713] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4376.306464] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4376.309047] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4376.312935] ? woken_wake_function+0x30/0x30 [ 4376.314502] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4376.321329] kthread+0x1d1/0x200 [ 4376.322304] ? set_kthread_struct+0x70/0x70 [ 4376.330243] ret_from_fork+0x1f/0x30 [ 4396.055130] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 4396.061146] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4396.064110] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 4396.108134] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4396.108149] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4396.108152] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4396.108166] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4396.108170] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4396.108176] RAX: 0000000000007b90 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4396.108179] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4396.108181] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4396.108184] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4396.108186] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4396.108189] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4396.108192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4396.108195] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4396.108203] Call Trace: [ 4396.108209] [ 4396.108213] ? show_regs.cold.9+0x22/0x2f [ 4396.108223] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4396.108229] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4396.108234] ? watchdog+0x40/0x40 [ 4396.108238] ? hrtimer_interrupt+0x128/0x2c0 [ 4396.108242] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4396.108250] ? apic_timer_interrupt+0xf/0x20 [ 4396.108255] [ 4396.108257] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4396.108266] do_raw_spin_lock+0x177/0x190 [ 4396.108271] _raw_spin_lock+0x21/0x30 [ 4396.108275] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4396.108997] ? do_raw_spin_unlock+0x75/0x190 [ 4396.109004] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4396.109230] ? do_wait_intr+0xf0/0xf0 [ 4396.109240] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4396.109431] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4396.110070] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4396.110295] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4396.110424] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4396.110932] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4396.110965] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4396.110995] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4396.111154] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4396.111417] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4396.111449] lmv_intent_open+0x338/0x1150 [lmv] [ 4396.112132] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4396.112237] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4396.112322] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4396.112344] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4396.112428] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4396.124355] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4396.124529] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4396.124548] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4396.124551] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4396.124565] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4396.124569] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4396.124574] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4396.124577] RDX: 00000000000005fb RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4396.124579] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4396.124581] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4396.124584] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4396.124587] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4396.124590] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4396.124593] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4396.124601] Call Trace: [ 4396.124606] [ 4396.124612] ? show_regs.cold.9+0x22/0x2f [ 4396.124621] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4396.124628] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4396.124633] ? watchdog+0x40/0x40 [ 4396.124638] ? hrtimer_interrupt+0x128/0x2c0 [ 4396.124642] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4396.124649] ? apic_timer_interrupt+0xf/0x20 [ 4396.124654] [ 4396.124657] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4396.124662] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4396.124667] do_raw_spin_lock+0x177/0x190 [ 4396.124733] _raw_spin_lock+0x21/0x30 [ 4396.124740] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4396.125116] obd_mod_free+0x46/0x140 [ptlrpc] [ 4396.125319] mdc_commit_open+0x91/0xb0 [mdc] [ 4396.125990] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4396.126219] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4396.127387] after_reply+0xe2c/0x1780 [ptlrpc] [ 4396.127630] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4396.129517] ? __next_timer_interrupt+0x160/0x160 [ 4396.129528] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4396.130482] ? do_wait_intr+0xf0/0xf0 [ 4396.130494] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4396.130740] kthread+0x1d1/0x200 [ 4396.130747] ? set_kthread_struct+0x70/0x70 [ 4396.130752] ret_from_fork+0x1f/0x30 [ 4396.676277] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4396.678390] ? do_raw_spin_unlock+0x75/0x190 [ 4396.680087] ? _raw_spin_unlock+0x12/0x30 [ 4396.681976] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4396.690921] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4396.698782] ll_atomic_open+0x36a/0xd70 [lustre] [ 4396.700421] lookup_open+0x356/0xb40 [ 4396.706273] do_last+0x313/0xfc0 [ 4396.707516] ? nd_jump_root+0xe5/0x160 [ 4396.712022] ? path_init+0x437/0x520 [ 4396.715355] path_openat+0xf7/0x500 [ 4396.719087] ? do_raw_spin_unlock+0x75/0x190 [ 4396.728921] do_filp_open+0x99/0x140 [ 4396.733180] ? getname_flags+0x6e/0x330 [ 4396.736422] ? __check_object_size+0xff/0x256 [ 4396.739030] ? do_raw_spin_unlock+0x75/0x190 [ 4396.740974] ? _raw_spin_unlock+0x12/0x30 [ 4396.743275] do_sys_openat2+0x2b4/0x410 [ 4396.745083] do_sys_open+0x73/0xa0 [ 4396.748997] __x64_sys_openat+0x24/0x30 [ 4396.754894] do_syscall_64+0xc1/0x440 [ 4396.758760] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4396.764455] RIP: 0033:0x7f53425a5332 [ 4396.766821] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4396.783010] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4396.789773] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4396.792697] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4396.804679] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4396.807094] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4396.824549] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4400.053563] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 4400.068810] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4400.125487] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4400.141297] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4400.154288] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4400.159448] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4400.169547] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4400.192312] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4400.199927] RAX: 000000000000670f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4400.214266] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4400.231863] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4400.236658] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4400.244364] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4400.252746] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4400.266962] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4400.277948] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4400.286489] Call Trace: [ 4400.287102] [ 4400.287618] ? show_regs.cold.9+0x22/0x2f [ 4400.297821] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4400.305728] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4400.311714] ? watchdog+0x40/0x40 [ 4400.316865] ? hrtimer_interrupt+0x128/0x2c0 [ 4400.322972] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4400.327138] ? apic_timer_interrupt+0xf/0x20 [ 4400.330598] [ 4400.331943] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4400.334920] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4400.337719] ? newidle_balance.isra.93+0x207/0x580 [ 4400.342944] do_raw_spin_lock+0x177/0x190 [ 4400.345581] _raw_spin_lock+0x21/0x30 [ 4400.347967] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4400.350707] process_one_work+0x2c8/0x700 [ 4400.354515] worker_thread+0x296/0x6e0 [ 4400.355875] ? rescuer_thread+0x570/0x570 [ 4400.358276] kthread+0x1d1/0x200 [ 4400.360697] ? set_kthread_struct+0x70/0x70 [ 4400.362777] ret_from_fork+0x1f/0x30 [ 4404.058567] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4404.071726] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4404.141583] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4404.151942] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4404.161078] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4404.166118] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4404.182619] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4404.191602] RAX: 0000000000000f8d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4404.197210] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4404.212216] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4404.231009] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4404.237881] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4404.246201] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4404.271075] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4404.292035] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4404.298854] Call Trace: [ 4404.303263] [ 4404.308882] ? show_regs.cold.9+0x22/0x2f [ 4404.318876] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4404.330670] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4404.334216] ? watchdog+0x40/0x40 [ 4404.337084] ? hrtimer_interrupt+0x128/0x2c0 [ 4404.339868] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4404.342344] ? apic_timer_interrupt+0xf/0x20 [ 4404.345317] [ 4404.349780] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4404.355373] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4404.359282] do_raw_spin_lock+0x177/0x190 [ 4404.361767] _raw_spin_lock+0x21/0x30 [ 4404.364137] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4404.366910] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4404.369543] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4404.373039] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4404.376311] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4404.379794] ? woken_wake_function+0x30/0x30 [ 4404.381970] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4404.384814] kthread+0x1d1/0x200 [ 4404.386167] ? set_kthread_struct+0x70/0x70 [ 4404.390790] ret_from_fork+0x1f/0x30 [ 4405.730077] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1768s! [ 4405.750593] Showing busy workqueues and worker pools: [ 4405.762258] workqueue events: flags=0x0 [ 4405.773938] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4405.790696] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4405.822687] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4405.836028] pending: cpu_latency_work [ptlrpc] [ 4405.838108] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4405.856732] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4405.864842] workqueue events_long: flags=0x0 [ 4405.867789] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4405.873178] pending: lu_objects_flush [obdclass] [ 4405.877129] workqueue events_unbound: flags=0x2 [ 4405.879272] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4405.886093] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4405.889649] workqueue events_power_efficient: flags=0x80 [ 4405.891756] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4405.895476] pending: check_lifetime, do_cache_clean [ 4405.898295] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4405.901202] pending: neigh_periodic_work, neigh_periodic_work [ 4405.903114] workqueue mm_percpu_wq: flags=0x8 [ 4405.906665] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4405.909911] pending: vmstat_update, lru_add_drain_per_cpu [ 4405.912667] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4405.915539] pending: vmstat_update, lru_add_drain_per_cpu [ 4405.921084] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4405.926899] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4405.929565] workqueue xprtiod: flags=0xa [ 4405.932269] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4405.949225] pending: xprt_autoclose, xs_error_handle [ 4405.955552] workqueue ipv6_addrconf: flags=0x40008 [ 4405.962231] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4405.966609] pending: addrconf_verify_work [ 4405.969573] workqueue knbd0-recv: flags=0x1a [ 4405.972665] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4405.976347] in-flight: 651:recv_work [ 4405.980532] workqueue ptlrpc_pinger: flags=0x2 [ 4405.987789] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4405.995409] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4406.010765] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1768s workers=4 idle: 15979 31824 8 [ 4406.020644] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4396s workers=2 idle: 654 [ 4424.055028] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 4424.062257] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 4424.063469] Modules linked in: [ 4424.080316] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul [ 4424.082521] lustre(O) osp(O) ofd(O) [ 4424.131411] crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4424.131444] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4424.131452] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4424.131457] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4424.131475] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4424.131480] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4424.131484] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4424.131487] RDX: 0000000000006609 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4424.131489] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4424.131492] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4424.131494] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4424.131498] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4424.131501] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4424.131503] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4424.131511] Call Trace: [ 4424.131518] [ 4424.131525] ? show_regs.cold.9+0x22/0x2f [ 4424.131535] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4424.131541] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4424.131545] ? watchdog+0x40/0x40 [ 4424.131549] ? hrtimer_interrupt+0x128/0x2c0 [ 4424.131553] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4424.131562] ? apic_timer_interrupt+0xf/0x20 [ 4424.131567] [ 4424.131570] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4424.131575] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4424.131580] do_raw_spin_lock+0x177/0x190 [ 4424.131585] _raw_spin_lock+0x21/0x30 [ 4424.131594] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4424.133413] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4424.133483] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4424.133490] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4424.133494] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4424.133506] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4424.133510] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4424.133514] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4424.133516] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4424.133518] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4424.133521] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4424.133523] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4424.133526] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4424.133529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4424.133531] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4424.133540] Call Trace: [ 4424.133545] [ 4424.133549] ? show_regs.cold.9+0x22/0x2f [ 4424.133559] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4424.133566] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4424.133570] ? watchdog+0x40/0x40 [ 4424.133574] ? hrtimer_interrupt+0x128/0x2c0 [ 4424.133578] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4424.133583] ? apic_timer_interrupt+0xf/0x20 [ 4424.133588] [ 4424.133590] ? native_safe_halt+0x2e/0x50 [ 4424.133593] kvm_wait+0xcd/0xe0 [ 4424.133600] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4424.133609] do_raw_spin_lock+0x177/0x190 [ 4424.133613] _raw_spin_lock+0x21/0x30 [ 4424.133617] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4424.133891] ? do_raw_spin_unlock+0x75/0x190 [ 4424.133898] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4424.143949] obd_mod_free+0x46/0x140 [ptlrpc] [ 4424.148475] ? do_wait_intr+0xf0/0xf0 [ 4424.159572] mdc_commit_open+0x91/0xb0 [mdc] [ 4424.159625] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4424.162169] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4424.176403] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4424.176754] after_reply+0xe2c/0x1780 [ptlrpc] [ 4424.176938] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4424.177209] ? __next_timer_interrupt+0x160/0x160 [ 4424.177293] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4424.177508] ? do_wait_intr+0xf0/0xf0 [ 4424.177517] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4424.177714] kthread+0x1d1/0x200 [ 4424.177720] ? set_kthread_struct+0x70/0x70 [ 4424.177724] ret_from_fork+0x1f/0x30 [ 4424.414574] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4424.416686] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4424.419303] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4424.422682] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4424.433161] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4424.435899] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4424.439336] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4424.442323] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4424.448594] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4424.451706] lmv_intent_open+0x338/0x1150 [lmv] [ 4424.454193] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4424.458024] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4424.461309] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4424.464740] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4424.466873] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4424.470429] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4424.472681] ? do_raw_spin_unlock+0x75/0x190 [ 4424.474490] ? _raw_spin_unlock+0x12/0x30 [ 4424.478296] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4424.481862] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4424.484951] ll_atomic_open+0x36a/0xd70 [lustre] [ 4424.487589] lookup_open+0x356/0xb40 [ 4424.489783] do_last+0x313/0xfc0 [ 4424.493197] ? nd_jump_root+0xe5/0x160 [ 4424.496327] ? path_init+0x437/0x520 [ 4424.497897] path_openat+0xf7/0x500 [ 4424.500336] ? do_raw_spin_unlock+0x75/0x190 [ 4424.504354] do_filp_open+0x99/0x140 [ 4424.506532] ? getname_flags+0x6e/0x330 [ 4424.508835] ? __check_object_size+0xff/0x256 [ 4424.511896] ? do_raw_spin_unlock+0x75/0x190 [ 4424.516679] ? _raw_spin_unlock+0x12/0x30 [ 4424.521441] do_sys_openat2+0x2b4/0x410 [ 4424.523195] do_sys_open+0x73/0xa0 [ 4424.524595] __x64_sys_openat+0x24/0x30 [ 4424.527023] do_syscall_64+0xc1/0x440 [ 4424.529385] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4424.532683] RIP: 0033:0x7f53425a5332 [ 4424.539162] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4424.557717] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4424.566134] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4424.569802] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4424.578154] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4424.583894] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4424.596449] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4432.052038] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 4432.057674] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4432.058324] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4432.082869] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4432.082884] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4432.082892] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4432.083243] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4432.083261] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4432.083269] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4432.083276] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4432.083279] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4432.083282] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4432.083285] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4432.083289] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4432.083294] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4432.083298] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4432.083301] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4432.083315] Call Trace: [ 4432.083324] [ 4432.083335] ? show_regs.cold.9+0x22/0x2f [ 4432.083351] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4432.083357] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4432.083362] ? watchdog+0x40/0x40 [ 4432.083365] ? hrtimer_interrupt+0x128/0x2c0 [ 4432.083368] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4432.083375] ? apic_timer_interrupt+0xf/0x20 [ 4432.083380] [ 4432.083383] ? native_safe_halt+0x2e/0x50 [ 4432.083387] kvm_wait+0xcd/0xe0 [ 4432.083395] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4432.083402] ? newidle_balance.isra.93+0x207/0x580 [ 4432.083411] do_raw_spin_lock+0x177/0x190 [ 4432.083425] _raw_spin_lock+0x21/0x30 [ 4432.083437] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4432.083813] process_one_work+0x2c8/0x700 [ 4432.083830] worker_thread+0x296/0x6e0 [ 4432.083841] ? rescuer_thread+0x570/0x570 [ 4432.083851] kthread+0x1d1/0x200 [ 4432.083862] ? set_kthread_struct+0x70/0x70 [ 4432.083871] ret_from_fork+0x1f/0x30 [ 4432.326297] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4432.386895] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4432.404378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4432.424609] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4432.440377] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4432.467695] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4432.476863] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4432.485409] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4432.495268] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4432.500222] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4432.510004] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4432.522195] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4432.526652] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4432.529684] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4432.542859] Call Trace: [ 4432.546561] [ 4432.547493] ? show_regs.cold.9+0x22/0x2f [ 4432.548773] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4432.552083] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4432.554893] ? watchdog+0x40/0x40 [ 4432.556572] ? hrtimer_interrupt+0x128/0x2c0 [ 4432.558011] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4432.560457] ? apic_timer_interrupt+0xf/0x20 [ 4432.562224] [ 4432.563199] ? native_safe_halt+0x2e/0x50 [ 4432.572491] kvm_wait+0xcd/0xe0 [ 4432.574798] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4432.583300] do_raw_spin_lock+0x177/0x190 [ 4432.586894] _raw_spin_lock+0x21/0x30 [ 4432.594547] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4432.602078] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4432.609005] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4432.617413] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4432.621659] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4432.625773] ? woken_wake_function+0x30/0x30 [ 4432.629244] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4432.636186] kthread+0x1d1/0x200 [ 4432.638257] ? set_kthread_struct+0x70/0x70 [ 4432.640490] ret_from_fork+0x1f/0x30 [ 4436.447062] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1799s! [ 4436.452871] Showing busy workqueues and worker pools: [ 4436.467674] workqueue events: flags=0x0 [ 4436.469069] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4436.477826] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4436.500914] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4436.516642] pending: cpu_latency_work [ptlrpc] [ 4436.522291] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4436.534047] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4436.549622] workqueue events_long: flags=0x0 [ 4436.558192] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4436.574483] pending: lu_objects_flush [obdclass] [ 4436.582067] workqueue events_unbound: flags=0x2 [ 4436.586139] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4436.588336] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4436.592760] workqueue events_power_efficient: flags=0x80 [ 4436.599661] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4436.603587] pending: check_lifetime, do_cache_clean [ 4436.606063] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4436.611621] pending: neigh_periodic_work, neigh_periodic_work [ 4436.614655] workqueue mm_percpu_wq: flags=0x8 [ 4436.616768] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4436.621157] pending: vmstat_update, lru_add_drain_per_cpu [ 4436.628519] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4436.630777] pending: vmstat_update, lru_add_drain_per_cpu [ 4436.638343] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4436.641336] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4436.643498] workqueue xprtiod: flags=0xa [ 4436.646768] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4436.650495] pending: xprt_autoclose, xs_error_handle [ 4436.653097] workqueue ipv6_addrconf: flags=0x40008 [ 4436.655903] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4436.659839] pending: addrconf_verify_work [ 4436.661569] workqueue knbd0-recv: flags=0x1a [ 4436.663975] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4436.667131] in-flight: 651:recv_work [ 4436.669582] workqueue ptlrpc_pinger: flags=0x2 [ 4436.671395] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4436.673976] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4436.677674] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1799s workers=4 idle: 15979 31824 8 [ 4436.682482] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4427s workers=2 idle: 654 [ 4452.061586] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 4452.068086] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4452.068405] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 4452.130955] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4452.130970] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4452.130973] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4452.130986] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4452.130991] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4452.130996] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4452.130999] RDX: 0000000000005b34 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4452.131002] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4452.131005] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4452.131007] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4452.131011] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4452.131013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4452.131016] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4452.131024] Call Trace: [ 4452.131029] [ 4452.131034] ? show_regs.cold.9+0x22/0x2f [ 4452.131045] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4452.131052] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4452.131057] ? watchdog+0x40/0x40 [ 4452.131061] ? hrtimer_interrupt+0x128/0x2c0 [ 4452.131065] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4452.131073] ? apic_timer_interrupt+0xf/0x20 [ 4452.131077] [ 4452.131079] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4452.131084] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4452.131089] do_raw_spin_lock+0x177/0x190 [ 4452.131093] _raw_spin_lock+0x21/0x30 [ 4452.131100] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4452.131542] obd_mod_free+0x46/0x140 [ptlrpc] [ 4452.131960] mdc_commit_open+0x91/0xb0 [mdc] [ 4452.132006] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4452.132566] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4452.133070] after_reply+0xe2c/0x1780 [ptlrpc] [ 4452.135825] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4452.141677] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4452.143726] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4452.143802] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4452.143806] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4452.143822] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4452.143827] RSP: 0018:ffffbb5381503500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4452.143832] RAX: 00000000000021ab RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4452.143835] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4452.143837] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4452.143840] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4452.143842] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4452.143846] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4452.143849] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4452.143851] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4452.143861] Call Trace: [ 4452.143867] [ 4452.143873] ? show_regs.cold.9+0x22/0x2f [ 4452.143884] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4452.143891] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4452.143895] ? watchdog+0x40/0x40 [ 4452.143899] ? hrtimer_interrupt+0x128/0x2c0 [ 4452.143904] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4452.143913] ? apic_timer_interrupt+0xf/0x20 [ 4452.143918] [ 4452.143921] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4452.143929] do_raw_spin_lock+0x177/0x190 [ 4452.143934] _raw_spin_lock+0x21/0x30 [ 4452.143939] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4452.155230] ? __next_timer_interrupt+0x160/0x160 [ 4452.155253] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4452.157112] ? do_wait_intr+0xf0/0xf0 [ 4452.157127] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4452.157366] kthread+0x1d1/0x200 [ 4452.157374] ? set_kthread_struct+0x70/0x70 [ 4452.157378] ret_from_fork+0x1f/0x30 [ 4452.551110] ? do_raw_spin_unlock+0x75/0x190 [ 4452.553044] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4452.561527] ? do_wait_intr+0xf0/0xf0 [ 4452.565903] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4452.570874] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4452.577284] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4452.587664] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4452.589581] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4452.594195] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4452.599382] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4452.611156] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4452.619442] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4452.625477] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4452.628677] lmv_intent_open+0x338/0x1150 [lmv] [ 4452.632830] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4452.643198] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4452.646025] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4452.648334] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4452.650102] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4452.660354] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4452.666271] ? do_raw_spin_unlock+0x75/0x190 [ 4452.671617] ? _raw_spin_unlock+0x12/0x30 [ 4452.677629] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4452.683894] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4452.690154] ll_atomic_open+0x36a/0xd70 [lustre] [ 4452.697068] lookup_open+0x356/0xb40 [ 4452.699298] do_last+0x313/0xfc0 [ 4452.706432] ? nd_jump_root+0xe5/0x160 [ 4452.710233] ? path_init+0x437/0x520 [ 4452.712327] path_openat+0xf7/0x500 [ 4452.716319] ? do_raw_spin_unlock+0x75/0x190 [ 4452.722136] do_filp_open+0x99/0x140 [ 4452.724047] ? getname_flags+0x6e/0x330 [ 4452.726712] ? __check_object_size+0xff/0x256 [ 4452.729047] ? do_raw_spin_unlock+0x75/0x190 [ 4452.739672] ? _raw_spin_unlock+0x12/0x30 [ 4452.745265] do_sys_openat2+0x2b4/0x410 [ 4452.748003] do_sys_open+0x73/0xa0 [ 4452.751672] __x64_sys_openat+0x24/0x30 [ 4452.754416] do_syscall_64+0xc1/0x440 [ 4452.761961] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4452.764312] RIP: 0033:0x7f53425a5332 [ 4452.771270] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4452.782879] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4452.789859] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4452.793805] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4452.798501] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4452.804643] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4452.808211] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4460.062870] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4460.073213] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4460.141357] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4460.156292] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4460.161248] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4460.169151] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4460.185395] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4460.193218] RAX: 0000000000004385 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4460.201583] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4460.204930] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4460.212387] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4460.214581] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4460.225669] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4460.234679] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4460.244474] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4460.246140] Call Trace: [ 4460.246775] [ 4460.247229] ? show_regs.cold.9+0x22/0x2f [ 4460.249627] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4460.252130] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4460.253966] ? watchdog+0x40/0x40 [ 4460.257068] ? hrtimer_interrupt+0x128/0x2c0 [ 4460.259808] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4460.261528] ? apic_timer_interrupt+0xf/0x20 [ 4460.264635] [ 4460.265548] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4460.275883] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4460.278471] do_raw_spin_lock+0x177/0x190 [ 4460.279864] _raw_spin_lock+0x21/0x30 [ 4460.283300] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4460.291223] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4460.294774] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4460.297315] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4460.299341] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4460.304409] ? woken_wake_function+0x30/0x30 [ 4460.307272] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4460.314311] kthread+0x1d1/0x200 [ 4460.319277] ? set_kthread_struct+0x70/0x70 [ 4460.324255] ret_from_fork+0x1f/0x30 [ 4464.059758] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 4464.071696] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4464.126256] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4464.139513] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4464.151185] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4464.162801] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4464.169042] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4464.180171] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4464.196535] RAX: 0000000000005b8b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4464.206243] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4464.217205] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4464.225658] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4464.232463] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4464.234903] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4464.246783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4464.253799] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4464.261505] Call Trace: [ 4464.262487] [ 4464.263967] ? show_regs.cold.9+0x22/0x2f [ 4464.266262] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4464.273758] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4464.281969] ? watchdog+0x40/0x40 [ 4464.283887] ? hrtimer_interrupt+0x128/0x2c0 [ 4464.285373] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4464.294128] ? apic_timer_interrupt+0xf/0x20 [ 4464.297836] [ 4464.302189] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4464.309653] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4464.314915] ? newidle_balance.isra.93+0x207/0x580 [ 4464.321071] do_raw_spin_lock+0x177/0x190 [ 4464.325515] _raw_spin_lock+0x21/0x30 [ 4464.335250] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4464.341078] process_one_work+0x2c8/0x700 [ 4464.343505] worker_thread+0x296/0x6e0 [ 4464.351252] ? rescuer_thread+0x570/0x570 [ 4464.355251] kthread+0x1d1/0x200 [ 4464.358969] ? set_kthread_struct+0x70/0x70 [ 4464.362204] ret_from_fork+0x1f/0x30 [ 4467.168045] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1830s! [ 4467.173068] Showing busy workqueues and worker pools: [ 4467.180182] workqueue events: flags=0x0 [ 4467.182881] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4467.187647] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4467.196145] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4467.205632] pending: cpu_latency_work [ptlrpc] [ 4467.211606] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4467.218206] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4467.226307] workqueue events_long: flags=0x0 [ 4467.234930] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4467.237095] pending: lu_objects_flush [obdclass] [ 4467.240612] workqueue events_unbound: flags=0x2 [ 4467.243829] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4467.249140] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4467.256587] workqueue events_power_efficient: flags=0x80 [ 4467.263173] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4467.270247] pending: check_lifetime, do_cache_clean [ 4467.277574] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4467.281363] pending: neigh_periodic_work, neigh_periodic_work [ 4467.293831] workqueue mm_percpu_wq: flags=0x8 [ 4467.298848] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4467.305792] pending: vmstat_update, lru_add_drain_per_cpu [ 4467.313579] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4467.317250] pending: vmstat_update, lru_add_drain_per_cpu [ 4467.321819] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4467.328157] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4467.338786] workqueue xprtiod: flags=0xa [ 4467.340425] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4467.347625] pending: xprt_autoclose, xs_error_handle [ 4467.359673] workqueue ipv6_addrconf: flags=0x40008 [ 4467.361876] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4467.365861] pending: addrconf_verify_work [ 4467.370286] workqueue knbd0-recv: flags=0x1a [ 4467.372850] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4467.382907] in-flight: 651:recv_work [ 4467.385833] workqueue ptlrpc_pinger: flags=0x2 [ 4467.388191] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4467.392176] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4467.397598] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1830s workers=4 idle: 15979 31824 8 [ 4467.404311] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4457s workers=2 idle: 654 [ 4480.055439] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 4480.058588] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 4480.060497] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4480.060581] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4480.060589] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4480.060593] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4480.060608] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4480.060613] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4480.060617] RAX: 000000000000248a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4480.060619] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4480.060622] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4480.060624] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4480.060627] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4480.060748] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4480.060753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4480.060755] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4480.060763] Call Trace: [ 4480.060768] [ 4480.060775] ? show_regs.cold.9+0x22/0x2f [ 4480.060784] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4480.060792] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4480.060796] ? watchdog+0x40/0x40 [ 4480.060799] ? hrtimer_interrupt+0x128/0x2c0 [ 4480.060803] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4480.060811] ? apic_timer_interrupt+0xf/0x20 [ 4480.060815] [ 4480.060818] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4480.060825] do_raw_spin_lock+0x177/0x190 [ 4480.060829] _raw_spin_lock+0x21/0x30 [ 4480.060834] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4480.061192] ? do_raw_spin_unlock+0x75/0x190 [ 4480.061199] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4480.061559] ? do_wait_intr+0xf0/0xf0 [ 4480.061570] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4480.061926] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4480.062247] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4480.062743] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4480.067821] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4480.067908] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4480.067915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4480.067919] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4480.067935] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4480.067939] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4480.067944] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4480.067946] RDX: 0000000000005871 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4480.067949] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4480.067951] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4480.067954] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4480.067957] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4480.067960] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4480.067963] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4480.067971] Call Trace: [ 4480.067978] [ 4480.067984] ? show_regs.cold.9+0x22/0x2f [ 4480.067994] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4480.068001] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4480.068006] ? watchdog+0x40/0x40 [ 4480.068011] ? hrtimer_interrupt+0x128/0x2c0 [ 4480.068015] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4480.068024] ? apic_timer_interrupt+0xf/0x20 [ 4480.068029] [ 4480.068032] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4480.068036] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4480.068042] do_raw_spin_lock+0x177/0x190 [ 4480.068046] _raw_spin_lock+0x21/0x30 [ 4480.068053] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4480.068549] obd_mod_free+0x46/0x140 [ptlrpc] [ 4480.099382] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4480.099434] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4480.099467] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4480.099498] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4480.115502] mdc_commit_open+0x91/0xb0 [mdc] [ 4480.115556] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4480.115845] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4480.116460] after_reply+0xe2c/0x1780 [ptlrpc] [ 4480.116680] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4480.116914] ? __next_timer_interrupt+0x160/0x160 [ 4480.116925] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4480.117160] ? do_wait_intr+0xf0/0xf0 [ 4480.117170] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4480.118546] kthread+0x1d1/0x200 [ 4480.118553] ? set_kthread_struct+0x70/0x70 [ 4480.118558] ret_from_fork+0x1f/0x30 [ 4480.432543] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4480.435496] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4480.438717] lmv_intent_open+0x338/0x1150 [lmv] [ 4480.441406] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4480.445267] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4480.448927] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4480.450997] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4480.453231] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4480.455367] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4480.460541] ? do_raw_spin_unlock+0x75/0x190 [ 4480.463764] ? _raw_spin_unlock+0x12/0x30 [ 4480.465117] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4480.467424] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4480.472777] ll_atomic_open+0x36a/0xd70 [lustre] [ 4480.474945] lookup_open+0x356/0xb40 [ 4480.476588] do_last+0x313/0xfc0 [ 4480.477760] ? nd_jump_root+0xe5/0x160 [ 4480.479341] ? path_init+0x437/0x520 [ 4480.480366] path_openat+0xf7/0x500 [ 4480.481825] ? do_raw_spin_unlock+0x75/0x190 [ 4480.485496] do_filp_open+0x99/0x140 [ 4480.486975] ? getname_flags+0x6e/0x330 [ 4480.488646] ? __check_object_size+0xff/0x256 [ 4480.492344] ? do_raw_spin_unlock+0x75/0x190 [ 4480.495938] ? _raw_spin_unlock+0x12/0x30 [ 4480.497563] do_sys_openat2+0x2b4/0x410 [ 4480.501609] do_sys_open+0x73/0xa0 [ 4480.504668] __x64_sys_openat+0x24/0x30 [ 4480.508216] do_syscall_64+0xc1/0x440 [ 4480.509938] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4480.519637] RIP: 0033:0x7f53425a5332 [ 4480.522350] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4480.546836] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4480.548590] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4480.559298] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4480.570322] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4480.573399] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4480.579643] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4488.059465] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4488.062317] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4488.081395] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4488.085521] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4488.090361] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4488.092188] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4488.105358] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4488.107285] RAX: 0000000000007322 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4488.114162] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4488.119032] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4488.122376] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4488.125221] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4488.127960] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4488.131361] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4488.134105] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4488.138327] Call Trace: [ 4488.139344] [ 4488.140193] ? show_regs.cold.9+0x22/0x2f [ 4488.141910] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4488.144702] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4488.145895] ? watchdog+0x40/0x40 [ 4488.146770] ? hrtimer_interrupt+0x128/0x2c0 [ 4488.147807] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4488.152318] ? apic_timer_interrupt+0xf/0x20 [ 4488.159382] [ 4488.161574] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4488.169076] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4488.175907] do_raw_spin_lock+0x177/0x190 [ 4488.181752] _raw_spin_lock+0x21/0x30 [ 4488.184558] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4488.187775] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4488.190680] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4488.193671] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4488.198140] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4488.202571] ? woken_wake_function+0x30/0x30 [ 4488.204119] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4488.207729] kthread+0x1d1/0x200 [ 4488.209185] ? set_kthread_struct+0x70/0x70 [ 4488.211011] ret_from_fork+0x1f/0x30 [ 4492.058017] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 4492.066496] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4492.116704] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4492.136711] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4492.142224] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4492.159823] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4492.170278] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4492.192923] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4492.205618] RAX: 0000000000004c57 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4492.217604] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4492.226005] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4492.237729] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4492.250564] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4492.260688] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4492.273930] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4492.282212] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4492.289686] Call Trace: [ 4492.292111] [ 4492.293274] ? show_regs.cold.9+0x22/0x2f [ 4492.295667] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4492.298757] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4492.300347] ? watchdog+0x40/0x40 [ 4492.307328] ? hrtimer_interrupt+0x128/0x2c0 [ 4492.308963] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4492.321962] ? apic_timer_interrupt+0xf/0x20 [ 4492.323979] [ 4492.325036] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4492.328705] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4492.332048] ? newidle_balance.isra.93+0x207/0x580 [ 4492.337141] do_raw_spin_lock+0x177/0x190 [ 4492.346334] _raw_spin_lock+0x21/0x30 [ 4492.349480] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4492.359287] process_one_work+0x2c8/0x700 [ 4492.364445] worker_thread+0x296/0x6e0 [ 4492.369336] ? rescuer_thread+0x570/0x570 [ 4492.374804] kthread+0x1d1/0x200 [ 4492.379745] ? set_kthread_struct+0x70/0x70 [ 4492.384260] ret_from_fork+0x1f/0x30 [ 4494.487742] rcu: INFO: rcu_sched self-detected stall on CPU [ 4494.490113] rcu: 2-....: (1247540 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 4494.497623] (t=2400061 jiffies g=274185 q=64295) [ 4494.502123] rcu: rcu_sched kthread starved for 1858427 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 4494.507254] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4494.511445] rcu: RCU grace-period kthread stack dump: [ 4494.514374] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4494.531056] Call Trace: [ 4494.531925] __schedule+0x351/0xcb0 [ 4494.534528] schedule+0xc0/0x180 [ 4494.536407] schedule_timeout+0xb4/0x190 [ 4494.539530] ? do_raw_spin_unlock+0x75/0x190 [ 4494.542426] ? __next_timer_interrupt+0x160/0x160 [ 4494.546783] ? prepare_to_swait_event+0xf3/0x140 [ 4494.552386] rcu_gp_kthread+0x8dd/0x15b0 [ 4494.554803] ? force_qs_rnp+0x2c0/0x2c0 [ 4494.559478] kthread+0x1d1/0x200 [ 4494.570680] ? set_kthread_struct+0x70/0x70 [ 4494.572194] ret_from_fork+0x1f/0x30 [ 4494.573466] rcu: Stack dump where RCU GP kthread last ran: [ 4494.579402] NMI backtrace for cpu 2 [ 4494.580577] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4494.592755] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4494.598173] Call Trace: [ 4494.600480] [ 4494.601731] ? dump_stack+0xbb/0x10e [ 4494.611395] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4494.616770] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4494.621556] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4494.625189] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4494.628404] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4494.638100] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 4494.643038] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 4494.647504] ? update_process_times+0x6d/0xc0 [ 4494.649014] ? tick_sched_handle.isra.18+0x27/0x90 [ 4494.652435] ? tick_sched_timer+0x53/0xb0 [ 4494.655214] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4494.659686] ? tick_sched_handle.isra.18+0x90/0x90 [ 4494.661434] ? hrtimer_interrupt+0x128/0x2c0 [ 4494.662731] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4494.665966] ? apic_timer_interrupt+0xf/0x20 [ 4494.671185] [ 4494.672228] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4494.675253] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4494.676889] ? do_raw_spin_lock+0x177/0x190 [ 4494.681113] ? _raw_spin_lock+0x21/0x30 [ 4494.684022] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4494.689682] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4494.694223] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4494.701375] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4494.705781] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4494.715409] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4494.720038] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4494.722065] ? __next_timer_interrupt+0x160/0x160 [ 4494.730878] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4494.736224] ? do_wait_intr+0xf0/0xf0 [ 4494.739632] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4494.744417] ? kthread+0x1d1/0x200 [ 4494.751216] ? set_kthread_struct+0x70/0x70 [ 4494.753186] ? ret_from_fork+0x1f/0x30 [ 4494.756837] Sending NMI from CPU 2 to CPUs 1: [ 4494.764513] NMI backtrace for cpu 1 [ 4494.764521] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4494.764523] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4494.764525] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4494.764527] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4494.764530] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [ 4494.764533] RAX: 0000000000007f89 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4494.764534] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4494.764536] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4494.764685] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4494.764686] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4494.764688] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4494.764690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4494.764691] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4494.764693] Call Trace: [ 4494.764693] [ 4494.764694] ? show_regs.cold.9+0x22/0x2f [ 4494.764696] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4494.764697] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4494.764699] ? nmi_handle+0xbd/0x210 [ 4494.764700] ? default_do_nmi+0x4b/0x1b0 [ 4494.764701] ? do_nmi+0x280/0x380 [ 4494.764702] ? end_repeat_nmi+0x16/0x1f [ 4494.764703] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4494.764704] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4494.764706] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4494.764707] [ 4494.764708] do_raw_spin_lock+0x177/0x190 [ 4494.764709] _raw_spin_lock+0x21/0x30 [ 4494.764710] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4494.764712] ? do_raw_spin_unlock+0x75/0x190 [ 4494.764713] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4494.764714] ? do_wait_intr+0xf0/0xf0 [ 4494.764715] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4494.764717] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4494.764718] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4494.764719] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4494.764720] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4494.764722] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4494.764723] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4494.764724] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4494.764725] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4494.764727] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4494.764728] lmv_intent_open+0x338/0x1150 [lmv] [ 4494.764729] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4494.764731] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4494.764732] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4494.764733] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4494.764734] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4494.764735] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4494.764737] ? do_raw_spin_unlock+0x75/0x190 [ 4494.764738] ? _raw_spin_unlock+0x12/0x30 [ 4494.764739] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4494.764740] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4494.764742] ll_atomic_open+0x36a/0xd70 [lustre] [ 4494.764743] lookup_open+0x356/0xb40 [ 4494.764744] do_last+0x313/0xfc0 [ 4494.764746] ? nd_jump_root+0xe5/0x160 [ 4494.764747] ? path_init+0x437/0x520 [ 4494.764748] path_openat+0xf7/0x500 [ 4494.764750] ? do_raw_spin_unlock+0x75/0x190 [ 4494.764751] do_filp_open+0x99/0x140 [ 4494.764752] ? getname_flags+0x6e/0x330 [ 4494.764754] ? __check_object_size+0xff/0x256 [ 4494.764755] ? do_raw_spin_unlock+0x75/0x190 [ 4494.764756] ? _raw_spin_unlock+0x12/0x30 [ 4494.764758] do_sys_openat2+0x2b4/0x410 [ 4494.764759] do_sys_open+0x73/0xa0 [ 4494.764760] __x64_sys_openat+0x24/0x30 [ 4494.764762] do_syscall_64+0xc1/0x440 [ 4494.764763] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4494.764765] RIP: 0033:0x7f53425a5332 [ 4494.764766] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4494.764769] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4494.764772] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4494.764774] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4494.764776] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4494.764778] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4494.764779] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4494.764789] NMI backtrace for cpu 2 [ 4495.135281] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4495.141601] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4495.150205] Call Trace: [ 4495.151595] [ 4495.153625] ? dump_stack+0xbb/0x10e [ 4495.155469] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4495.157698] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4495.159701] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4495.162526] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4495.165478] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4495.168343] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4495.170582] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4495.172788] ? update_process_times+0x6d/0xc0 [ 4495.175712] ? tick_sched_handle.isra.18+0x27/0x90 [ 4495.177777] ? tick_sched_timer+0x53/0xb0 [ 4495.181082] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4495.183440] ? tick_sched_handle.isra.18+0x90/0x90 [ 4495.189770] ? hrtimer_interrupt+0x128/0x2c0 [ 4495.194557] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4495.197121] ? apic_timer_interrupt+0xf/0x20 [ 4495.202232] [ 4495.203399] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4495.207002] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4495.208794] ? do_raw_spin_lock+0x177/0x190 [ 4495.212339] ? _raw_spin_lock+0x21/0x30 [ 4495.216827] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4495.221790] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4495.228362] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4495.232270] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4495.235458] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4495.238695] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4495.248366] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4495.252239] ? __next_timer_interrupt+0x160/0x160 [ 4495.254901] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4495.258676] ? do_wait_intr+0xf0/0xf0 [ 4495.261108] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4495.265406] ? kthread+0x1d1/0x200 [ 4495.269675] ? set_kthread_struct+0x70/0x70 [ 4495.273779] ? ret_from_fork+0x1f/0x30 [ 4497.888098] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1860s! [ 4497.890893] Showing busy workqueues and worker pools: [ 4497.892514] workqueue events: flags=0x0 [ 4497.903409] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4497.911314] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4497.927157] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4497.932139] pending: cpu_latency_work [ptlrpc] [ 4497.935440] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4497.939192] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4497.951189] workqueue events_long: flags=0x0 [ 4497.956017] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4497.963514] pending: lu_objects_flush [obdclass] [ 4497.965221] workqueue events_unbound: flags=0x2 [ 4497.968020] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4497.974060] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4497.975987] workqueue events_power_efficient: flags=0x80 [ 4497.981104] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4497.986088] pending: check_lifetime, do_cache_clean [ 4497.988217] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4497.990938] pending: neigh_periodic_work, neigh_periodic_work [ 4497.995150] workqueue mm_percpu_wq: flags=0x8 [ 4497.998297] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4498.008453] pending: vmstat_update, lru_add_drain_per_cpu [ 4498.013629] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4498.018270] pending: vmstat_update, lru_add_drain_per_cpu [ 4498.026815] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4498.029011] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4498.035157] workqueue xprtiod: flags=0xa [ 4498.039312] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4498.043668] pending: xprt_autoclose, xs_error_handle [ 4498.046685] workqueue ipv6_addrconf: flags=0x40008 [ 4498.048448] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4498.053446] pending: addrconf_verify_work [ 4498.057210] workqueue knbd0-recv: flags=0x1a [ 4498.059727] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4498.069865] in-flight: 651:recv_work [ 4498.071197] workqueue ptlrpc_pinger: flags=0x2 [ 4498.072638] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4498.079230] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4498.088526] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1860s workers=4 idle: 15979 31824 8 [ 4498.098381] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4488s workers=2 idle: 654 [ 4516.060690] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4516.063436] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4516.097587] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4516.101843] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4516.107767] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4516.111655] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4516.133400] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4516.139751] RAX: 0000000000007509 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4516.145868] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4516.152770] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4516.158506] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4516.163181] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4516.170632] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4516.173420] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4516.178263] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4516.184180] Call Trace: [ 4516.185059] [ 4516.186853] ? show_regs.cold.9+0x22/0x2f [ 4516.190302] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4516.195089] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4516.196618] ? watchdog+0x40/0x40 [ 4516.200101] ? hrtimer_interrupt+0x128/0x2c0 [ 4516.202871] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4516.205507] ? apic_timer_interrupt+0xf/0x20 [ 4516.209926] [ 4516.210931] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4516.215836] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4516.218775] do_raw_spin_lock+0x177/0x190 [ 4516.223363] _raw_spin_lock+0x21/0x30 [ 4516.224782] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4516.230630] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4516.236905] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4516.242790] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4516.248995] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4516.251212] ? woken_wake_function+0x30/0x30 [ 4516.253863] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4516.256663] kthread+0x1d1/0x200 [ 4516.257757] ? set_kthread_struct+0x70/0x70 [ 4516.259946] ret_from_fork+0x1f/0x30 [ 4520.056871] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 4520.059472] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 4520.063064] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4520.063145] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4520.063154] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4520.063157] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4520.063245] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4520.063252] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4520.063257] RAX: 0000000000007c86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4520.063260] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4520.063263] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4520.063265] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4520.063268] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4520.063272] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4520.063275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4520.063277] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4520.063286] Call Trace: [ 4520.063302] [ 4520.063309] ? show_regs.cold.9+0x22/0x2f [ 4520.063319] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4520.063325] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4520.063329] ? watchdog+0x40/0x40 [ 4520.063333] ? hrtimer_interrupt+0x128/0x2c0 [ 4520.063337] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4520.063345] ? apic_timer_interrupt+0xf/0x20 [ 4520.063349] [ 4520.063352] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4520.066041] do_raw_spin_lock+0x177/0x190 [ 4520.066051] _raw_spin_lock+0x21/0x30 [ 4520.066057] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4520.066351] ? do_raw_spin_unlock+0x75/0x190 [ 4520.066378] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4520.069228] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4520.069304] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4520.069312] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4520.069314] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4520.069327] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4520.069331] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4520.069336] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4520.069338] RDX: 00000000000003e2 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4520.069340] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4520.069343] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4520.069345] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4520.069348] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4520.069351] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4520.069353] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4520.069362] Call Trace: [ 4520.071896] [ 4520.102748] ? do_wait_intr+0xf0/0xf0 [ 4520.102771] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4520.103056] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4520.103266] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4520.103786] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4520.103908] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4520.103950] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4520.103983] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4520.104014] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4520.104102] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4520.104339] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4520.107477] lmv_intent_open+0x338/0x1150 [lmv] [ 4520.107518] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4520.107644] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4520.107735] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4520.107759] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4520.107846] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4520.107926] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4520.108007] ? do_raw_spin_unlock+0x75/0x190 [ 4520.108015] ? _raw_spin_unlock+0x12/0x30 [ 4520.108023] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4520.108048] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4520.108067] ll_atomic_open+0x36a/0xd70 [lustre] [ 4520.117679] ? show_regs.cold.9+0x22/0x2f [ 4520.117709] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4520.117717] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4520.117722] ? watchdog+0x40/0x40 [ 4520.117726] ? hrtimer_interrupt+0x128/0x2c0 [ 4520.117736] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4520.117746] ? apic_timer_interrupt+0xf/0x20 [ 4520.117751] [ 4520.117756] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4520.117762] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4520.117767] do_raw_spin_lock+0x177/0x190 [ 4520.117773] _raw_spin_lock+0x21/0x30 [ 4520.117780] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4520.118083] obd_mod_free+0x46/0x140 [ptlrpc] [ 4520.118321] mdc_commit_open+0x91/0xb0 [mdc] [ 4520.118357] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4520.118657] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4520.118900] after_reply+0xe2c/0x1780 [ptlrpc] [ 4520.119133] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4520.119359] ? __next_timer_interrupt+0x160/0x160 [ 4520.121489] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4520.121789] ? do_wait_intr+0xf0/0xf0 [ 4520.121799] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4520.130069] lookup_open+0x356/0xb40 [ 4520.130090] do_last+0x313/0xfc0 [ 4520.130097] ? nd_jump_root+0xe5/0x160 [ 4520.130102] ? path_init+0x437/0x520 [ 4520.130107] path_openat+0xf7/0x500 [ 4520.130112] ? do_raw_spin_unlock+0x75/0x190 [ 4520.130119] do_filp_open+0x99/0x140 [ 4520.130126] ? getname_flags+0x6e/0x330 [ 4520.130130] ? __check_object_size+0xff/0x256 [ 4520.130137] ? do_raw_spin_unlock+0x75/0x190 [ 4520.130141] ? _raw_spin_unlock+0x12/0x30 [ 4520.130148] do_sys_openat2+0x2b4/0x410 [ 4520.130152] do_sys_open+0x73/0xa0 [ 4520.130156] __x64_sys_openat+0x24/0x30 [ 4520.130159] do_syscall_64+0xc1/0x440 [ 4520.130165] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4520.130171] RIP: 0033:0x7f53425a5332 [ 4520.130198] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4520.130203] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4520.130209] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4520.130211] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4520.130213] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4520.130215] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4520.130218] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4520.883760] kthread+0x1d1/0x200 [ 4520.886742] ? set_kthread_struct+0x70/0x70 [ 4520.892968] ret_from_fork+0x1f/0x30 [ 4524.053915] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 4524.072523] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4524.118901] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4524.124216] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4524.128582] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4524.132676] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4524.136394] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4524.148740] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4524.158274] RAX: 0000000000007c17 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4524.166870] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4524.175992] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4524.183768] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4524.192840] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4524.201200] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4524.205900] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4524.208642] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4524.213615] Call Trace: [ 4524.214336] [ 4524.214914] ? show_regs.cold.9+0x22/0x2f [ 4524.222434] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4524.224538] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4524.231791] ? watchdog+0x40/0x40 [ 4524.237393] ? hrtimer_interrupt+0x128/0x2c0 [ 4524.240684] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4524.246373] ? apic_timer_interrupt+0xf/0x20 [ 4524.248586] [ 4524.249857] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4524.252906] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4524.255772] ? newidle_balance.isra.93+0x207/0x580 [ 4524.258339] do_raw_spin_lock+0x177/0x190 [ 4524.260076] _raw_spin_lock+0x21/0x30 [ 4524.262270] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4524.271934] process_one_work+0x2c8/0x700 [ 4524.274079] worker_thread+0x296/0x6e0 [ 4524.281223] ? rescuer_thread+0x570/0x570 [ 4524.283205] kthread+0x1d1/0x200 [ 4524.289495] ? set_kthread_struct+0x70/0x70 [ 4524.290958] ret_from_fork+0x1f/0x30 [ 4528.608642] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1891s! [ 4528.611527] Showing busy workqueues and worker pools: [ 4528.614509] workqueue events: flags=0x0 [ 4528.626467] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4528.633515] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4528.642305] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4528.644672] pending: cpu_latency_work [ptlrpc] [ 4528.655814] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4528.660306] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4528.666924] workqueue events_long: flags=0x0 [ 4528.673736] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4528.679063] pending: lu_objects_flush [obdclass] [ 4528.688932] workqueue events_unbound: flags=0x2 [ 4528.693475] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4528.695614] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4528.701672] workqueue events_power_efficient: flags=0x80 [ 4528.707882] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4528.712019] pending: check_lifetime, do_cache_clean [ 4528.716720] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4528.719806] pending: neigh_periodic_work, neigh_periodic_work [ 4528.723046] workqueue mm_percpu_wq: flags=0x8 [ 4528.725333] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4528.728649] pending: vmstat_update, lru_add_drain_per_cpu [ 4528.732501] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4528.741551] pending: vmstat_update, lru_add_drain_per_cpu [ 4528.753548] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4528.756871] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4528.763415] workqueue xprtiod: flags=0xa [ 4528.770142] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4528.773517] pending: xprt_autoclose, xs_error_handle [ 4528.775185] workqueue ipv6_addrconf: flags=0x40008 [ 4528.776669] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4528.779098] pending: addrconf_verify_work [ 4528.785765] workqueue knbd0-recv: flags=0x1a [ 4528.790138] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4528.795822] in-flight: 651:recv_work [ 4528.797721] workqueue ptlrpc_pinger: flags=0x2 [ 4528.800346] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4528.802070] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4528.804193] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1891s workers=4 idle: 15979 31824 8 [ 4528.806914] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4519s workers=2 idle: 654 [ 4544.066105] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4544.076828] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4544.126135] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4544.140143] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4544.145344] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4544.147546] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4544.159465] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4544.164989] RAX: 0000000000006028 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4544.166993] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4544.192836] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4544.201464] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4544.210968] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4544.212812] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4544.222653] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4544.225295] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4544.228439] Call Trace: [ 4544.229418] [ 4544.230246] ? show_regs.cold.9+0x22/0x2f [ 4544.232095] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4544.234521] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4544.236521] ? watchdog+0x40/0x40 [ 4544.238124] ? hrtimer_interrupt+0x128/0x2c0 [ 4544.240273] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4544.242564] ? apic_timer_interrupt+0xf/0x20 [ 4544.245632] [ 4544.248881] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4544.252634] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4544.255887] do_raw_spin_lock+0x177/0x190 [ 4544.262377] _raw_spin_lock+0x21/0x30 [ 4544.266169] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4544.269726] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4544.276231] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4544.281799] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4544.285111] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4544.286829] ? woken_wake_function+0x30/0x30 [ 4544.293276] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4544.295897] kthread+0x1d1/0x200 [ 4544.297147] ? set_kthread_struct+0x70/0x70 [ 4544.300049] ret_from_fork+0x1f/0x30 [ 4548.056176] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 4548.058161] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 4548.059292] Modules linked in: [ 4548.063379] Modules linked in: lustre(O) [ 4548.064483] lustre(O) [ 4548.064484] osp(O) ofd(O) [ 4548.065707] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4548.065783] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4548.065790] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4548.065794] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4548.065808] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4548.065813] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4548.065818] RAX: 0000000000005c8b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4548.065820] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4548.065823] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4548.065825] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4548.065828] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4548.065831] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4548.065833] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4548.065836] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4548.065844] Call Trace: [ 4548.065850] [ 4548.065855] ? show_regs.cold.9+0x22/0x2f [ 4548.065864] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4548.065872] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4548.065876] ? watchdog+0x40/0x40 [ 4548.065890] ? hrtimer_interrupt+0x128/0x2c0 [ 4548.065895] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4548.065904] ? apic_timer_interrupt+0xf/0x20 [ 4548.065908] [ 4548.065912] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4548.065919] do_raw_spin_lock+0x177/0x190 [ 4548.065923] _raw_spin_lock+0x21/0x30 [ 4548.065928] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4548.066205] ? do_raw_spin_unlock+0x75/0x190 [ 4548.066211] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4548.066399] ? do_wait_intr+0xf0/0xf0 [ 4548.066407] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4548.066602] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4548.066830] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4548.067238] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4548.067344] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4548.067384] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4548.067413] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4548.067438] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4548.068313] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4548.068378] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4548.068384] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4548.068386] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4548.068396] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4548.068401] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 [ 4548.069574] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4548.071239] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4548.071274] lmv_intent_open+0x338/0x1150 [lmv] [ 4548.071307] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4548.071418] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4548.071499] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4548.071523] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4548.071614] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4548.072347] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4548.072434] ? do_raw_spin_unlock+0x75/0x190 [ 4548.072442] ? _raw_spin_unlock+0x12/0x30 [ 4548.072449] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4548.072476] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4548.072497] ll_atomic_open+0x36a/0xd70 [lustre] [ 4548.072586] lookup_open+0x356/0xb40 [ 4548.072596] do_last+0x313/0xfc0 [ 4548.072602] ? nd_jump_root+0xe5/0x160 [ 4548.072607] ? path_init+0x437/0x520 [ 4548.072612] path_openat+0xf7/0x500 [ 4548.072617] ? do_raw_spin_unlock+0x75/0x190 [ 4548.072622] do_filp_open+0x99/0x140 [ 4548.072629] ? getname_flags+0x6e/0x330 [ 4548.072634] ? __check_object_size+0xff/0x256 [ 4548.072642] ? do_raw_spin_unlock+0x75/0x190 [ 4548.072646] ? _raw_spin_unlock+0x12/0x30 [ 4548.072651] do_sys_openat2+0x2b4/0x410 [ 4548.072657] do_sys_open+0x73/0xa0 [ 4548.072661] __x64_sys_openat+0x24/0x30 [ 4548.072665] do_syscall_64+0xc1/0x440 [ 4548.072673] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4548.073778] RIP: 0033:0x7f53425a5332 [ 4548.073830] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4548.073836] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4548.073841] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4548.073844] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4548.073847] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4548.073849] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4548.073851] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4548.570422] ORIG_RAX: ffffffffffffff13 [ 4548.578081] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4548.585519] RDX: 0000000000007941 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4548.594438] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4548.596865] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4548.621999] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4548.634806] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4548.641635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4548.652965] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4548.665969] Call Trace: [ 4548.675165] [ 4548.675785] ? show_regs.cold.9+0x22/0x2f [ 4548.677190] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4548.684149] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4548.690446] ? watchdog+0x40/0x40 [ 4548.694578] ? hrtimer_interrupt+0x128/0x2c0 [ 4548.697812] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4548.702214] ? apic_timer_interrupt+0xf/0x20 [ 4548.704717] [ 4548.706618] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4548.710017] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4548.713967] do_raw_spin_lock+0x177/0x190 [ 4548.720317] _raw_spin_lock+0x21/0x30 [ 4548.726332] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4548.733819] obd_mod_free+0x46/0x140 [ptlrpc] [ 4548.740816] mdc_commit_open+0x91/0xb0 [mdc] [ 4548.746250] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4548.748768] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4548.756846] after_reply+0xe2c/0x1780 [ptlrpc] [ 4548.763516] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4548.768732] ? __next_timer_interrupt+0x160/0x160 [ 4548.773461] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4548.785094] ? do_wait_intr+0xf0/0xf0 [ 4548.786265] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4548.790384] kthread+0x1d1/0x200 [ 4548.792341] ? set_kthread_struct+0x70/0x70 [ 4548.800332] ret_from_fork+0x1f/0x30 [ 4556.052049] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 4556.055449] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4556.090239] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4556.101308] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4556.106733] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4556.109106] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4556.110624] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4556.121304] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4556.124081] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4556.128448] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4556.131906] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4556.135352] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4556.138535] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4556.141655] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4556.145173] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4556.147903] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4556.151191] Call Trace: [ 4556.151903] [ 4556.152393] ? show_regs.cold.9+0x22/0x2f [ 4556.157429] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4556.160501] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4556.164723] ? watchdog+0x40/0x40 [ 4556.166209] ? hrtimer_interrupt+0x128/0x2c0 [ 4556.168295] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4556.170298] ? apic_timer_interrupt+0xf/0x20 [ 4556.174266] [ 4556.175233] ? native_safe_halt+0x2e/0x50 [ 4556.177858] kvm_wait+0xcd/0xe0 [ 4556.179312] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4556.184620] ? newidle_balance.isra.93+0x207/0x580 [ 4556.189070] do_raw_spin_lock+0x177/0x190 [ 4556.190149] _raw_spin_lock+0x21/0x30 [ 4556.193493] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4556.198666] process_one_work+0x2c8/0x700 [ 4556.201371] worker_thread+0x296/0x6e0 [ 4556.202708] ? rescuer_thread+0x570/0x570 [ 4556.207679] kthread+0x1d1/0x200 [ 4556.210558] ? set_kthread_struct+0x70/0x70 [ 4556.212937] ret_from_fork+0x1f/0x30 [ 4558.820935] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1921s! [ 4558.828513] Showing busy workqueues and worker pools: [ 4558.834549] workqueue events: flags=0x0 [ 4558.839523] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4558.847506] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4558.856117] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4558.859438] pending: cpu_latency_work [ptlrpc] [ 4558.863627] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4558.867240] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4558.872072] workqueue events_long: flags=0x0 [ 4558.874961] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4558.878456] pending: lu_objects_flush [obdclass] [ 4558.881233] workqueue events_unbound: flags=0x2 [ 4558.883548] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4558.886271] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4558.891325] workqueue events_power_efficient: flags=0x80 [ 4558.893499] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4558.897104] pending: check_lifetime, do_cache_clean [ 4558.899318] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4558.902566] pending: neigh_periodic_work, neigh_periodic_work [ 4558.911473] workqueue mm_percpu_wq: flags=0x8 [ 4558.914588] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4558.916898] pending: vmstat_update, lru_add_drain_per_cpu [ 4558.919405] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4558.922177] pending: vmstat_update, lru_add_drain_per_cpu [ 4558.924602] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4558.928314] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4558.931457] workqueue xprtiod: flags=0xa [ 4558.932645] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4558.938306] pending: xprt_autoclose, xs_error_handle [ 4558.944300] workqueue ipv6_addrconf: flags=0x40008 [ 4558.945786] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4558.948326] pending: addrconf_verify_work [ 4558.950871] workqueue knbd0-recv: flags=0x1a [ 4558.956227] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4558.961318] in-flight: 651:recv_work [ 4558.967934] workqueue ptlrpc_pinger: flags=0x2 [ 4558.972470] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4558.976725] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4558.978450] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1921s workers=4 idle: 15979 31824 8 [ 4558.988484] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4549s workers=2 idle: 654 [ 4572.058873] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 4572.067453] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4572.094868] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4572.099892] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4572.103797] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4572.106009] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4572.114655] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4572.117447] RAX: 0000000000005ab2 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4572.120283] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4572.125753] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4572.129318] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4572.132415] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4572.135373] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4572.138655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4572.140632] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4572.143569] Call Trace: [ 4572.144409] [ 4572.145065] ? show_regs.cold.9+0x22/0x2f [ 4572.146649] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4572.148467] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4572.154912] ? watchdog+0x40/0x40 [ 4572.156202] ? hrtimer_interrupt+0x128/0x2c0 [ 4572.158640] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4572.162577] ? apic_timer_interrupt+0xf/0x20 [ 4572.166220] [ 4572.167225] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4572.172529] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4572.176646] do_raw_spin_lock+0x177/0x190 [ 4572.178797] _raw_spin_lock+0x21/0x30 [ 4572.180022] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4572.182903] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4572.186598] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4572.190235] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4572.197107] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4572.203241] ? woken_wake_function+0x30/0x30 [ 4572.204642] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4572.210561] kthread+0x1d1/0x200 [ 4572.211656] ? set_kthread_struct+0x70/0x70 [ 4572.220014] ret_from_fork+0x1f/0x30 [ 4576.055032] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 4576.058984] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4576.061952] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 4576.077289] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4576.086143] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4576.086222] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4576.086230] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4576.086233] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4576.086247] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4576.086252] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4576.086256] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4576.086258] RDX: 00000000000078b4 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4576.086261] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4576.086262] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4576.086265] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4576.086267] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4576.086270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4576.086272] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4576.086280] Call Trace: [ 4576.086286] [ 4576.086291] ? show_regs.cold.9+0x22/0x2f [ 4576.086299] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4576.086306] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4576.086310] ? watchdog+0x40/0x40 [ 4576.086314] ? hrtimer_interrupt+0x128/0x2c0 [ 4576.086318] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4576.086326] ? apic_timer_interrupt+0xf/0x20 [ 4576.086331] [ 4576.086333] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4576.086338] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4576.086343] do_raw_spin_lock+0x177/0x190 [ 4576.086347] _raw_spin_lock+0x21/0x30 [ 4576.086352] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4576.086620] obd_mod_free+0x46/0x140 [ptlrpc] [ 4576.087139] mdc_commit_open+0x91/0xb0 [mdc] [ 4576.087176] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4576.087373] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4576.087639] after_reply+0xe2c/0x1780 [ptlrpc] [ 4576.087991] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4576.088194] ? __next_timer_interrupt+0x160/0x160 [ 4576.088204] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4576.090596] ? do_wait_intr+0xf0/0xf0 [ 4576.090611] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4576.091568] kthread+0x1d1/0x200 [ 4576.091577] ? set_kthread_struct+0x70/0x70 [ 4576.091581] ret_from_fork+0x1f/0x30 [ 4576.448013] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4576.451621] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4576.453118] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4576.466943] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4576.484572] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4576.497157] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4576.508278] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4576.512685] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4576.523648] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4576.535570] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4576.549881] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4576.555928] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4576.563555] Call Trace: [ 4576.566002] [ 4576.567568] ? show_regs.cold.9+0x22/0x2f [ 4576.573174] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4576.579072] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4576.585214] ? watchdog+0x40/0x40 [ 4576.586429] ? hrtimer_interrupt+0x128/0x2c0 [ 4576.594978] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4576.600883] ? apic_timer_interrupt+0xf/0x20 [ 4576.606666] [ 4576.607397] ? native_safe_halt+0x2e/0x50 [ 4576.617739] kvm_wait+0xcd/0xe0 [ 4576.623833] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4576.631617] do_raw_spin_lock+0x177/0x190 [ 4576.637329] _raw_spin_lock+0x21/0x30 [ 4576.641869] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4576.648403] ? do_raw_spin_unlock+0x75/0x190 [ 4576.650553] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4576.653895] ? do_wait_intr+0xf0/0xf0 [ 4576.665557] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4576.674908] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4576.681906] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4576.691694] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4576.697869] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4576.708870] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4576.717135] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4576.724760] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4576.728167] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4576.730181] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4576.737484] lmv_intent_open+0x338/0x1150 [lmv] [ 4576.741338] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4576.747374] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4576.752369] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4576.758488] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4576.760861] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4576.764389] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4576.769594] ? do_raw_spin_unlock+0x75/0x190 [ 4576.772867] ? _raw_spin_unlock+0x12/0x30 [ 4576.774772] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4576.777618] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4576.784831] ll_atomic_open+0x36a/0xd70 [lustre] [ 4576.788169] lookup_open+0x356/0xb40 [ 4576.789815] do_last+0x313/0xfc0 [ 4576.792985] ? nd_jump_root+0xe5/0x160 [ 4576.798744] ? path_init+0x437/0x520 [ 4576.804107] path_openat+0xf7/0x500 [ 4576.806854] ? do_raw_spin_unlock+0x75/0x190 [ 4576.809401] do_filp_open+0x99/0x140 [ 4576.812860] ? getname_flags+0x6e/0x330 [ 4576.819296] ? __check_object_size+0xff/0x256 [ 4576.821863] ? do_raw_spin_unlock+0x75/0x190 [ 4576.826105] ? _raw_spin_unlock+0x12/0x30 [ 4576.828359] do_sys_openat2+0x2b4/0x410 [ 4576.833595] do_sys_open+0x73/0xa0 [ 4576.845236] __x64_sys_openat+0x24/0x30 [ 4576.849309] do_syscall_64+0xc1/0x440 [ 4576.857441] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4576.859568] RIP: 0033:0x7f53425a5332 [ 4576.860854] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4576.905274] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4576.920131] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4576.928860] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4576.937559] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4576.945485] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4576.959407] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4584.057316] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 4584.067200] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4584.121238] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4584.142619] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4584.153416] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4584.160281] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4584.163102] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4584.170823] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4584.173789] RAX: 000000000000757c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4584.176351] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4584.178798] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4584.181523] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4584.183958] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4584.186607] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4584.189774] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4584.191784] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4584.194560] Call Trace: [ 4584.195445] [ 4584.196066] ? show_regs.cold.9+0x22/0x2f [ 4584.197467] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4584.198983] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4584.200569] ? watchdog+0x40/0x40 [ 4584.202113] ? hrtimer_interrupt+0x128/0x2c0 [ 4584.203662] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4584.205438] ? apic_timer_interrupt+0xf/0x20 [ 4584.206852] [ 4584.207648] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4584.209518] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4584.211410] ? newidle_balance.isra.93+0x207/0x580 [ 4584.213196] do_raw_spin_lock+0x177/0x190 [ 4584.214741] _raw_spin_lock+0x21/0x30 [ 4584.215979] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4584.218156] process_one_work+0x2c8/0x700 [ 4584.219674] worker_thread+0x296/0x6e0 [ 4584.221015] ? rescuer_thread+0x570/0x570 [ 4584.222475] kthread+0x1d1/0x200 [ 4584.223630] ? set_kthread_struct+0x70/0x70 [ 4584.225487] ret_from_fork+0x1f/0x30 [ 4589.027029] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1951s! [ 4589.030709] Showing busy workqueues and worker pools: [ 4589.033763] workqueue events: flags=0x0 [ 4589.035319] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4589.038066] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4589.043275] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4589.045950] pending: cpu_latency_work [ptlrpc] [ 4589.049221] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4589.052458] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4589.057354] workqueue events_long: flags=0x0 [ 4589.059740] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4589.066936] pending: lu_objects_flush [obdclass] [ 4589.069292] workqueue events_unbound: flags=0x2 [ 4589.071349] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4589.073850] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4589.076352] workqueue events_power_efficient: flags=0x80 [ 4589.078479] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4589.081432] pending: check_lifetime, do_cache_clean [ 4589.083611] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4589.086191] pending: neigh_periodic_work, neigh_periodic_work [ 4589.089689] workqueue mm_percpu_wq: flags=0x8 [ 4589.091728] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4589.094914] pending: vmstat_update, lru_add_drain_per_cpu [ 4589.097368] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4589.100448] pending: vmstat_update, lru_add_drain_per_cpu [ 4589.103615] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4589.107430] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4589.121378] workqueue xprtiod: flags=0xa [ 4589.123556] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4589.129178] pending: xprt_autoclose, xs_error_handle [ 4589.132497] workqueue ipv6_addrconf: flags=0x40008 [ 4589.136908] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4589.142085] pending: addrconf_verify_work [ 4589.145084] workqueue knbd0-recv: flags=0x1a [ 4589.150178] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4589.157059] in-flight: 651:recv_work [ 4589.160495] workqueue ptlrpc_pinger: flags=0x2 [ 4589.165967] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4589.169557] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4589.173287] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1952s workers=4 idle: 15979 31824 8 [ 4589.181614] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4579s workers=2 idle: 654 [ 4600.058430] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 4600.061536] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4600.091965] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4600.097477] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4600.102582] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4600.105555] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4600.122159] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4600.125651] RAX: 0000000000007e85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4600.133947] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4600.143484] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4600.148819] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4600.152903] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4600.156734] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4600.161469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4600.165413] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4600.173697] Call Trace: [ 4600.175419] [ 4600.176681] ? show_regs.cold.9+0x22/0x2f [ 4600.178989] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4600.180842] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4600.185458] ? watchdog+0x40/0x40 [ 4600.186692] ? hrtimer_interrupt+0x128/0x2c0 [ 4600.189709] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4600.192714] ? apic_timer_interrupt+0xf/0x20 [ 4600.195218] [ 4600.198893] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4600.202043] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4600.206389] do_raw_spin_lock+0x177/0x190 [ 4600.209766] _raw_spin_lock+0x21/0x30 [ 4600.211554] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4600.215114] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4600.216865] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4600.229472] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4600.235034] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4600.241490] ? woken_wake_function+0x30/0x30 [ 4600.248017] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4600.255533] kthread+0x1d1/0x200 [ 4600.257224] ? set_kthread_struct+0x70/0x70 [ 4600.259169] ret_from_fork+0x1f/0x30 [ 4604.055013] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 4604.058446] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 4604.059334] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) [ 4604.068830] Modules linked in: [ 4604.068832] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [ 4604.076111] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 [ 4604.077510] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4604.077631] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4604.077638] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4604.077641] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30a/0x6d0 [ 4604.077655] Code: 03 00 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 [ 4604.077660] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4604.077664] RAX: 0000000000003ce6 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4604.077666] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4604.077668] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4604.077670] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4604.077672] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4604.077675] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4604.077678] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4604.077680] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4604.077688] Call Trace: [ 4604.077694] [ 4604.077699] ? show_regs.cold.9+0x22/0x2f [ 4604.077709] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4604.077717] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4604.077722] ? watchdog+0x40/0x40 [ 4604.077726] ? hrtimer_interrupt+0x128/0x2c0 [ 4604.077730] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4604.077740] ? apic_timer_interrupt+0xf/0x20 [ 4604.077744] [ 4604.077747] ? __pv_queued_spin_lock_slowpath+0x30a/0x6d0 [ 4604.077753] do_raw_spin_lock+0x177/0x190 [ 4604.077758] _raw_spin_lock+0x21/0x30 [ 4604.077763] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4604.078306] ? do_raw_spin_unlock+0x75/0x190 [ 4604.078312] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4604.078506] ? do_wait_intr+0xf0/0xf0 [ 4604.078514] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4604.078795] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4604.079365] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4604.079658] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4604.079771] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4604.079808] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4604.079836] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4604.079861] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4604.080029] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4604.080409] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4604.080438] lmv_intent_open+0x338/0x1150 [lmv] [ 4604.080465] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4604.080616] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4604.080698] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4604.080720] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4604.080801] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4604.080885] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4604.081044] ? do_raw_spin_unlock+0x75/0x190 [ 4604.081051] ? _raw_spin_unlock+0x12/0x30 [ 4604.081058] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4604.081153] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4604.081242] ll_atomic_open+0x36a/0xd70 [lustre] [ 4604.081330] lookup_open+0x356/0xb40 [ 4604.081338] do_last+0x313/0xfc0 [ 4604.081343] ? nd_jump_root+0xe5/0x160 [ 4604.081347] ? path_init+0x437/0x520 [ 4604.081352] path_openat+0xf7/0x500 [ 4604.081357] ? do_raw_spin_unlock+0x75/0x190 [ 4604.081361] do_filp_open+0x99/0x140 [ 4604.081367] ? getname_flags+0x6e/0x330 [ 4604.081375] ? __check_object_size+0xff/0x256 [ 4604.081381] ? do_raw_spin_unlock+0x75/0x190 [ 4604.081385] ? _raw_spin_unlock+0x12/0x30 [ 4604.081389] do_sys_openat2+0x2b4/0x410 [ 4604.081392] do_sys_open+0x73/0xa0 [ 4604.081397] __x64_sys_openat+0x24/0x30 [ 4604.081400] do_syscall_64+0xc1/0x440 [ 4604.081407] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4604.081413] RIP: 0033:0x7f53425a5332 [ 4604.081419] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4604.081423] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4604.081428] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4604.081431] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4604.081432] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4604.081434] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4604.081436] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4604.798811] auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4604.850179] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4604.868494] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4604.878311] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4604.890643] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4604.926650] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4604.950620] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4604.968475] RDX: 0000000000006fe4 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4604.984420] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4604.994555] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4605.016597] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4605.023212] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4605.045305] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4605.053826] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4605.060183] Call Trace: [ 4605.061803] [ 4605.063180] ? show_regs.cold.9+0x22/0x2f [ 4605.069402] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4605.076401] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4605.088531] ? watchdog+0x40/0x40 [ 4605.090582] ? hrtimer_interrupt+0x128/0x2c0 [ 4605.094952] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4605.098586] ? apic_timer_interrupt+0xf/0x20 [ 4605.101131] [ 4605.102475] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4605.105563] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4605.112152] do_raw_spin_lock+0x177/0x190 [ 4605.116654] _raw_spin_lock+0x21/0x30 [ 4605.122016] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4605.129282] obd_mod_free+0x46/0x140 [ptlrpc] [ 4605.141683] mdc_commit_open+0x91/0xb0 [mdc] [ 4605.153117] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4605.156528] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4605.159475] after_reply+0xe2c/0x1780 [ptlrpc] [ 4605.162227] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4605.164987] ? __next_timer_interrupt+0x160/0x160 [ 4605.167345] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4605.169488] ? do_wait_intr+0xf0/0xf0 [ 4605.170833] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4605.173477] kthread+0x1d1/0x200 [ 4605.175129] ? set_kthread_struct+0x70/0x70 [ 4605.177339] ret_from_fork+0x1f/0x30 [ 4616.052590] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 4616.056652] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4616.081765] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4616.088978] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4616.093948] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4616.097265] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4616.099562] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4616.108497] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4616.112342] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4616.116553] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4616.120378] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4616.123653] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4616.127377] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4616.130956] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4616.135058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4616.137881] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4616.141484] Call Trace: [ 4616.142622] [ 4616.143778] ? show_regs.cold.9+0x22/0x2f [ 4616.145538] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4616.148249] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4616.150589] ? watchdog+0x40/0x40 [ 4616.152416] ? hrtimer_interrupt+0x128/0x2c0 [ 4616.155096] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4616.157613] ? apic_timer_interrupt+0xf/0x20 [ 4616.159666] [ 4616.160932] ? native_safe_halt+0x2e/0x50 [ 4616.162864] kvm_wait+0xcd/0xe0 [ 4616.163955] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4616.166510] ? newidle_balance.isra.93+0x207/0x580 [ 4616.169072] do_raw_spin_lock+0x177/0x190 [ 4616.171077] _raw_spin_lock+0x21/0x30 [ 4616.173057] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4616.176027] process_one_work+0x2c8/0x700 [ 4616.178265] worker_thread+0x296/0x6e0 [ 4616.179940] ? rescuer_thread+0x570/0x570 [ 4616.181349] kthread+0x1d1/0x200 [ 4616.182404] ? set_kthread_struct+0x70/0x70 [ 4616.184481] ret_from_fork+0x1f/0x30 [ 4619.234622] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 1982s! [ 4619.240370] Showing busy workqueues and worker pools: [ 4619.245699] workqueue events: flags=0x0 [ 4619.247424] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4619.250062] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4619.257678] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4619.264146] pending: cpu_latency_work [ptlrpc] [ 4619.269881] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4619.274466] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4619.286190] workqueue events_long: flags=0x0 [ 4619.287616] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4619.294481] pending: lu_objects_flush [obdclass] [ 4619.297520] workqueue events_unbound: flags=0x2 [ 4619.299646] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4619.303111] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4619.305009] workqueue events_power_efficient: flags=0x80 [ 4619.310369] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4619.317148] pending: check_lifetime, do_cache_clean [ 4619.319968] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4619.324269] pending: neigh_periodic_work, neigh_periodic_work [ 4619.333140] workqueue mm_percpu_wq: flags=0x8 [ 4619.339484] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4619.348431] pending: vmstat_update, lru_add_drain_per_cpu [ 4619.354860] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4619.360492] pending: vmstat_update, lru_add_drain_per_cpu [ 4619.364918] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4619.371104] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4619.375768] workqueue xprtiod: flags=0xa [ 4619.377404] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4619.379969] pending: xprt_autoclose, xs_error_handle [ 4619.385811] workqueue ipv6_addrconf: flags=0x40008 [ 4619.389550] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4619.393593] pending: addrconf_verify_work [ 4619.396687] workqueue knbd0-recv: flags=0x1a [ 4619.398370] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4619.401297] in-flight: 651:recv_work [ 4619.403108] workqueue ptlrpc_pinger: flags=0x2 [ 4619.407721] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4619.414599] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4619.417902] pool 8: cpus=0-3 flags=0x4 nice=0 hung=1982s workers=4 idle: 15979 31824 8 [ 4619.424207] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4609s workers=2 idle: 654 [ 4628.060306] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 4628.070129] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4628.146160] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4628.163975] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4628.177802] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4628.181872] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4628.204686] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4628.217545] RAX: 0000000000007f0c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4628.230494] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4628.236327] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4628.241116] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4628.246430] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4628.253059] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4628.263826] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4628.273206] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4628.283366] Call Trace: [ 4628.286415] [ 4628.287121] ? show_regs.cold.9+0x22/0x2f [ 4628.288444] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4628.291085] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4628.294777] ? watchdog+0x40/0x40 [ 4628.296310] ? hrtimer_interrupt+0x128/0x2c0 [ 4628.297996] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4628.302282] ? apic_timer_interrupt+0xf/0x20 [ 4628.304367] [ 4628.305293] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4628.308719] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4628.311097] do_raw_spin_lock+0x177/0x190 [ 4628.312700] _raw_spin_lock+0x21/0x30 [ 4628.314875] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4628.318780] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4628.322456] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4628.327701] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4628.330650] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4628.332904] ? woken_wake_function+0x30/0x30 [ 4628.334080] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4628.336192] kthread+0x1d1/0x200 [ 4628.337760] ? set_kthread_struct+0x70/0x70 [ 4628.340695] ret_from_fork+0x1f/0x30 [ 4632.056670] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 4632.057949] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 4632.068649] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4632.068733] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4632.068740] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4632.068743] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4632.068757] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4632.068761] RSP: 0018:ffffbb5381503500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4632.068766] RAX: 0000000000002d0f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4632.068769] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4632.068771] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4632.068774] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4632.068776] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4632.068779] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4632.068782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4632.068785] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4632.068794] Call Trace: [ 4632.068799] [ 4632.068804] ? show_regs.cold.9+0x22/0x2f [ 4632.068814] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4632.068820] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4632.068825] ? watchdog+0x40/0x40 [ 4632.068829] ? hrtimer_interrupt+0x128/0x2c0 [ 4632.068834] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4632.068842] ? apic_timer_interrupt+0xf/0x20 [ 4632.068847] [ 4632.068851] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4632.068858] do_raw_spin_lock+0x177/0x190 [ 4632.068863] _raw_spin_lock+0x21/0x30 [ 4632.068868] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4632.071730] Modules linked in: lustre(O) [ 4632.111172] ? do_raw_spin_unlock+0x75/0x190 [ 4632.111196] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4632.112606] ? do_wait_intr+0xf0/0xf0 [ 4632.112617] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4632.113652] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4632.113905] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4632.115258] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4632.115396] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4632.116174] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4632.116209] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4632.116241] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4632.116344] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4632.116647] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4632.116679] lmv_intent_open+0x338/0x1150 [lmv] [ 4632.116716] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4632.116812] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4632.116903] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4632.116928] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4632.117022] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4632.118380] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4632.118765] ? do_raw_spin_unlock+0x75/0x190 [ 4632.118774] ? _raw_spin_unlock+0x12/0x30 [ 4632.118782] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4632.118807] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4632.118829] ll_atomic_open+0x36a/0xd70 [lustre] [ 4632.118920] lookup_open+0x356/0xb40 [ 4632.118931] do_last+0x313/0xfc0 [ 4632.118937] ? nd_jump_root+0xe5/0x160 [ 4632.118942] ? path_init+0x437/0x520 [ 4632.118947] path_openat+0xf7/0x500 [ 4632.118952] ? do_raw_spin_unlock+0x75/0x190 [ 4632.118957] do_filp_open+0x99/0x140 [ 4632.118964] ? getname_flags+0x6e/0x330 [ 4632.118969] ? __check_object_size+0xff/0x256 [ 4632.118977] ? do_raw_spin_unlock+0x75/0x190 [ 4632.118981] ? _raw_spin_unlock+0x12/0x30 [ 4632.118985] do_sys_openat2+0x2b4/0x410 [ 4632.118990] do_sys_open+0x73/0xa0 [ 4632.118995] __x64_sys_openat+0x24/0x30 [ 4632.118999] do_syscall_64+0xc1/0x440 [ 4632.119006] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4632.119013] RIP: 0033:0x7f53425a5332 [ 4632.119035] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4632.119039] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4632.119046] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4632.119049] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4632.119052] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4632.120207] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4632.120210] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4632.455077] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4632.475790] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4632.481295] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4632.486888] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4632.489340] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4632.512977] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4632.521770] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4632.537622] RDX: 0000000000005180 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4632.542939] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4632.548934] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4632.553345] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4632.557944] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4632.563574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4632.569321] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4632.577489] Call Trace: [ 4632.579989] [ 4632.581120] ? show_regs.cold.9+0x22/0x2f [ 4632.583900] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4632.586031] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4632.588003] ? watchdog+0x40/0x40 [ 4632.589360] ? hrtimer_interrupt+0x128/0x2c0 [ 4632.592779] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4632.595299] ? apic_timer_interrupt+0xf/0x20 [ 4632.598287] [ 4632.599480] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4632.601939] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4632.603851] do_raw_spin_lock+0x177/0x190 [ 4632.605257] _raw_spin_lock+0x21/0x30 [ 4632.608514] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4632.613523] obd_mod_free+0x46/0x140 [ptlrpc] [ 4632.616428] mdc_commit_open+0x91/0xb0 [mdc] [ 4632.619195] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4632.623546] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4632.626149] after_reply+0xe2c/0x1780 [ptlrpc] [ 4632.629744] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4632.635646] ? __next_timer_interrupt+0x160/0x160 [ 4632.637664] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4632.642824] ? do_wait_intr+0xf0/0xf0 [ 4632.645970] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4632.649292] kthread+0x1d1/0x200 [ 4632.650326] ? set_kthread_struct+0x70/0x70 [ 4632.653932] ret_from_fork+0x1f/0x30 [ 4644.052131] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 4644.056379] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4644.126121] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4644.151826] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4644.160880] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4644.174633] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4644.179616] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4644.205331] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4644.212772] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4644.231654] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4644.244086] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4644.258514] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4644.265185] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4644.290934] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4644.304213] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4644.327018] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4644.346277] Call Trace: [ 4644.347138] [ 4644.347744] ? show_regs.cold.9+0x22/0x2f [ 4644.348724] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4644.359861] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4644.363983] ? watchdog+0x40/0x40 [ 4644.366033] ? hrtimer_interrupt+0x128/0x2c0 [ 4644.368068] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4644.371369] ? apic_timer_interrupt+0xf/0x20 [ 4644.373308] [ 4644.374437] ? native_safe_halt+0x2e/0x50 [ 4644.377047] kvm_wait+0xcd/0xe0 [ 4644.378886] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4644.383721] ? newidle_balance.isra.93+0x207/0x580 [ 4644.386894] do_raw_spin_lock+0x177/0x190 [ 4644.388353] _raw_spin_lock+0x21/0x30 [ 4644.390824] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4644.393968] process_one_work+0x2c8/0x700 [ 4644.396984] worker_thread+0x296/0x6e0 [ 4644.398882] ? rescuer_thread+0x570/0x570 [ 4644.401124] kthread+0x1d1/0x200 [ 4644.402910] ? set_kthread_struct+0x70/0x70 [ 4644.405906] ret_from_fork+0x1f/0x30 [ 4649.439906] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2012s! [ 4649.447951] Showing busy workqueues and worker pools: [ 4649.451285] workqueue events: flags=0x0 [ 4649.461913] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4649.472141] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4649.497315] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4649.509216] pending: cpu_latency_work [ptlrpc] [ 4649.520021] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4649.530818] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4649.551318] workqueue events_long: flags=0x0 [ 4649.558748] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4649.561413] pending: lu_objects_flush [obdclass] [ 4649.573672] workqueue events_unbound: flags=0x2 [ 4649.586079] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4649.600035] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4649.607297] workqueue events_power_efficient: flags=0x80 [ 4649.621515] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4649.627661] pending: check_lifetime, do_cache_clean [ 4649.644186] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4649.658830] pending: neigh_periodic_work, neigh_periodic_work [ 4649.675624] workqueue mm_percpu_wq: flags=0x8 [ 4649.685057] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4649.688764] pending: vmstat_update, lru_add_drain_per_cpu [ 4649.704854] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4649.714129] pending: vmstat_update, lru_add_drain_per_cpu [ 4649.718813] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4649.723194] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4649.744021] workqueue xprtiod: flags=0xa [ 4649.746081] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4649.754926] pending: xprt_autoclose, xs_error_handle [ 4649.773862] workqueue ipv6_addrconf: flags=0x40008 [ 4649.780503] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4649.790912] pending: addrconf_verify_work [ 4649.793659] workqueue knbd0-recv: flags=0x1a [ 4649.797120] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4649.808161] in-flight: 651:recv_work [ 4649.814170] workqueue ptlrpc_pinger: flags=0x2 [ 4649.826123] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4649.835377] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4649.847211] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2012s workers=4 idle: 15979 31824 8 [ 4649.857427] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4640s workers=2 idle: 654 [ 4656.058049] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 4656.062178] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4656.084976] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4656.091527] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4656.094867] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4656.096886] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4656.106957] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4656.110587] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4656.117601] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4656.121674] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4656.124436] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4656.137305] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4656.145829] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4656.155370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4656.158260] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4656.161928] Call Trace: [ 4656.164187] [ 4656.164783] ? show_regs.cold.9+0x22/0x2f [ 4656.165961] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4656.169980] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4656.172468] ? watchdog+0x40/0x40 [ 4656.173527] ? hrtimer_interrupt+0x128/0x2c0 [ 4656.176673] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4656.178783] ? apic_timer_interrupt+0xf/0x20 [ 4656.186248] [ 4656.186959] ? native_safe_halt+0x2e/0x50 [ 4656.191227] kvm_wait+0xcd/0xe0 [ 4656.192889] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4656.195911] do_raw_spin_lock+0x177/0x190 [ 4656.203172] _raw_spin_lock+0x21/0x30 [ 4656.205762] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4656.213944] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4656.222096] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4656.243105] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4656.244812] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4656.263941] ? woken_wake_function+0x30/0x30 [ 4656.269602] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4656.278783] kthread+0x1d1/0x200 [ 4656.282395] ? set_kthread_struct+0x70/0x70 [ 4656.283425] ret_from_fork+0x1f/0x30 [ 4660.057011] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 4660.057963] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 4660.062584] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4660.062668] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4660.062677] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4660.062681] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4660.062702] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4660.062707] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4660.062712] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4660.062714] RDX: 0000000000000e6b RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4660.062717] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4660.062719] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4660.062721] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4660.062725] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4660.062728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4660.062731] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4660.062740] Call Trace: [ 4660.062747] [ 4660.062755] ? show_regs.cold.9+0x22/0x2f [ 4660.062766] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4660.062773] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4660.062777] ? watchdog+0x40/0x40 [ 4660.062781] ? hrtimer_interrupt+0x128/0x2c0 [ 4660.062784] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4660.062795] ? apic_timer_interrupt+0xf/0x20 [ 4660.062800] [ 4660.062803] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4660.062808] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4660.062813] do_raw_spin_lock+0x177/0x190 [ 4660.062817] _raw_spin_lock+0x21/0x30 [ 4660.062824] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4660.068520] Modules linked in: [ 4660.083182] obd_mod_free+0x46/0x140 [ptlrpc] [ 4660.088771] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4660.088854] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4660.088862] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4660.088865] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4660.088878] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4660.088884] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4660.088889] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4660.088891] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4660.088893] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4660.088896] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4660.088898] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4660.088901] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4660.088905] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4660.088907] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4660.088917] Call Trace: [ 4660.089644] [ 4660.091706] mdc_commit_open+0x91/0xb0 [mdc] [ 4660.093301] ? show_regs.cold.9+0x22/0x2f [ 4660.093322] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4660.093330] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4660.093334] ? watchdog+0x40/0x40 [ 4660.093338] ? hrtimer_interrupt+0x128/0x2c0 [ 4660.093342] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4660.093362] ? apic_timer_interrupt+0xf/0x20 [ 4660.093367] [ 4660.093373] ? native_safe_halt+0x2e/0x50 [ 4660.093379] kvm_wait+0xcd/0xe0 [ 4660.093388] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4660.093400] do_raw_spin_lock+0x177/0x190 [ 4660.093404] _raw_spin_lock+0x21/0x30 [ 4660.093411] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4660.093755] ? do_raw_spin_unlock+0x75/0x190 [ 4660.093760] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4660.094144] ? do_wait_intr+0xf0/0xf0 [ 4660.094157] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4660.112190] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4660.115750] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4660.119520] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4660.121312] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4660.123281] after_reply+0xe2c/0x1780 [ptlrpc] [ 4660.125232] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4660.127793] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4660.133067] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4660.136482] ? __next_timer_interrupt+0x160/0x160 [ 4660.136512] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4660.142770] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4660.142828] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4660.143135] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4660.143245] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4660.147815] ? do_wait_intr+0xf0/0xf0 [ 4660.152678] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4660.152752] lmv_intent_open+0x338/0x1150 [lmv] [ 4660.152796] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4660.152981] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4660.153076] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4660.153100] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4660.153186] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4660.153267] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4660.153353] ? do_raw_spin_unlock+0x75/0x190 [ 4660.153361] ? _raw_spin_unlock+0x12/0x30 [ 4660.153369] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4660.153410] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4660.153432] ll_atomic_open+0x36a/0xd70 [lustre] [ 4660.153515] lookup_open+0x356/0xb40 [ 4660.153525] do_last+0x313/0xfc0 [ 4660.153531] ? nd_jump_root+0xe5/0x160 [ 4660.153535] ? path_init+0x437/0x520 [ 4660.153540] path_openat+0xf7/0x500 [ 4660.153544] ? do_raw_spin_unlock+0x75/0x190 [ 4660.153549] do_filp_open+0x99/0x140 [ 4660.153555] ? getname_flags+0x6e/0x330 [ 4660.153559] ? __check_object_size+0xff/0x256 [ 4660.153567] ? do_raw_spin_unlock+0x75/0x190 [ 4660.153570] ? _raw_spin_unlock+0x12/0x30 [ 4660.153575] do_sys_openat2+0x2b4/0x410 [ 4660.153579] do_sys_open+0x73/0xa0 [ 4660.154962] __x64_sys_openat+0x24/0x30 [ 4660.157312] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4660.159357] do_syscall_64+0xc1/0x440 [ 4660.163279] kthread+0x1d1/0x200 [ 4660.164339] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4660.167373] ? set_kthread_struct+0x70/0x70 [ 4660.167402] ret_from_fork+0x1f/0x30 [ 4660.310029] RIP: 0033:0x7f53425a5332 [ 4660.310845] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4660.314816] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4660.316383] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4660.317846] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4660.319670] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4660.321527] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4660.323421] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4674.493505] rcu: INFO: rcu_sched self-detected stall on CPU [ 4674.499549] rcu: 2-....: (1340619 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 4674.508180] (t=2580067 jiffies g=274185 q=64375) [ 4674.512479] rcu: rcu_sched kthread starved for 2038433 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 4674.519647] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4674.528802] rcu: RCU grace-period kthread stack dump: [ 4674.535344] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4674.541361] Call Trace: [ 4674.545242] __schedule+0x351/0xcb0 [ 4674.550181] schedule+0xc0/0x180 [ 4674.552221] schedule_timeout+0xb4/0x190 [ 4674.557430] ? do_raw_spin_unlock+0x75/0x190 [ 4674.560108] ? __next_timer_interrupt+0x160/0x160 [ 4674.563321] ? prepare_to_swait_event+0xf3/0x140 [ 4674.568219] rcu_gp_kthread+0x8dd/0x15b0 [ 4674.576138] ? force_qs_rnp+0x2c0/0x2c0 [ 4674.580423] kthread+0x1d1/0x200 [ 4674.585913] ? set_kthread_struct+0x70/0x70 [ 4674.589414] ret_from_fork+0x1f/0x30 [ 4674.593553] rcu: Stack dump where RCU GP kthread last ran: [ 4674.601351] NMI backtrace for cpu 2 [ 4674.604418] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4674.616238] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4674.620207] Call Trace: [ 4674.621206] [ 4674.622087] ? dump_stack+0xbb/0x10e [ 4674.623411] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4674.625473] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4674.627780] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4674.631421] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4674.633242] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4674.636329] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 4674.645982] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 4674.647574] ? update_process_times+0x6d/0xc0 [ 4674.658197] ? tick_sched_handle.isra.18+0x27/0x90 [ 4674.664885] ? tick_sched_timer+0x53/0xb0 [ 4674.670141] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4674.671635] ? tick_sched_handle.isra.18+0x90/0x90 [ 4674.680213] ? hrtimer_interrupt+0x128/0x2c0 [ 4674.684607] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4674.690660] ? apic_timer_interrupt+0xf/0x20 [ 4674.697467] [ 4674.698503] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4674.707871] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4674.709619] ? do_raw_spin_lock+0x177/0x190 [ 4674.719590] ? _raw_spin_lock+0x21/0x30 [ 4674.722343] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4674.731941] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4674.739866] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4674.743470] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4674.748435] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4674.752561] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4674.755824] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4674.759250] ? __next_timer_interrupt+0x160/0x160 [ 4674.760914] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4674.764207] ? do_wait_intr+0xf0/0xf0 [ 4674.770874] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4674.779209] ? kthread+0x1d1/0x200 [ 4674.782523] ? set_kthread_struct+0x70/0x70 [ 4674.788432] ? ret_from_fork+0x1f/0x30 [ 4674.793011] Sending NMI from CPU 2 to CPUs 1: [ 4674.804296] NMI backtrace for cpu 1 [ 4674.804303] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4674.804306] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4674.804307] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4674.804309] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4674.804311] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [ 4674.804314] RAX: 0000000000000f0c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4674.804316] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4674.804317] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4674.804318] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4674.804320] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4674.804321] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4674.804322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4674.804324] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4674.804325] Call Trace: [ 4674.804326] [ 4674.804326] ? show_regs.cold.9+0x22/0x2f [ 4674.804328] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4674.804328] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4674.804329] ? nmi_handle+0xbd/0x210 [ 4674.804330] ? default_do_nmi+0x4b/0x1b0 [ 4674.804331] ? do_nmi+0x280/0x380 [ 4674.804332] ? end_repeat_nmi+0x16/0x1f [ 4674.804333] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4674.804334] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4674.804335] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4674.804336] [ 4674.804337] do_raw_spin_lock+0x177/0x190 [ 4674.804338] _raw_spin_lock+0x21/0x30 [ 4674.804339] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4674.804340] ? do_raw_spin_unlock+0x75/0x190 [ 4674.804341] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4674.804342] ? do_wait_intr+0xf0/0xf0 [ 4674.804343] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4674.804344] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4674.804345] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4674.804346] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4674.804347] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4674.804348] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4674.804350] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4674.804351] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4674.804352] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4674.804353] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4674.804354] lmv_intent_open+0x338/0x1150 [lmv] [ 4674.804355] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4674.804356] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4674.804358] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4674.804358] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4674.804360] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4674.804361] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4674.804362] ? do_raw_spin_unlock+0x75/0x190 [ 4674.804363] ? _raw_spin_unlock+0x12/0x30 [ 4674.804365] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4674.804366] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4674.804367] ll_atomic_open+0x36a/0xd70 [lustre] [ 4674.804368] lookup_open+0x356/0xb40 [ 4674.804369] do_last+0x313/0xfc0 [ 4674.804370] ? nd_jump_root+0xe5/0x160 [ 4674.804372] ? path_init+0x437/0x520 [ 4674.804373] path_openat+0xf7/0x500 [ 4674.804374] ? do_raw_spin_unlock+0x75/0x190 [ 4674.804376] do_filp_open+0x99/0x140 [ 4674.804377] ? getname_flags+0x6e/0x330 [ 4674.804378] ? __check_object_size+0xff/0x256 [ 4674.804379] ? do_raw_spin_unlock+0x75/0x190 [ 4674.804380] ? _raw_spin_unlock+0x12/0x30 [ 4674.804381] do_sys_openat2+0x2b4/0x410 [ 4674.804382] do_sys_open+0x73/0xa0 [ 4674.804383] __x64_sys_openat+0x24/0x30 [ 4674.804384] do_syscall_64+0xc1/0x440 [ 4674.804385] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4674.804386] RIP: 0033:0x7f53425a5332 [ 4674.804388] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4674.804390] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4674.804398] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4674.804399] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4674.804401] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4674.804402] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4674.804403] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4674.807259] NMI backtrace for cpu 2 [ 4675.123359] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4675.126976] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4675.129716] Call Trace: [ 4675.130446] [ 4675.131143] ? dump_stack+0xbb/0x10e [ 4675.132239] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4675.133576] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4675.135177] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4675.136964] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4675.138894] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4675.140452] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4675.142042] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4675.143605] ? update_process_times+0x6d/0xc0 [ 4675.144997] ? tick_sched_handle.isra.18+0x27/0x90 [ 4675.146523] ? tick_sched_timer+0x53/0xb0 [ 4675.147950] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4675.149375] ? tick_sched_handle.isra.18+0x90/0x90 [ 4675.151523] ? hrtimer_interrupt+0x128/0x2c0 [ 4675.153470] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4675.155000] ? apic_timer_interrupt+0xf/0x20 [ 4675.156366] [ 4675.160252] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4675.168037] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4675.170577] ? do_raw_spin_lock+0x177/0x190 [ 4675.173087] ? _raw_spin_lock+0x21/0x30 [ 4675.174332] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4675.178525] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4675.181913] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4675.184762] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4675.190323] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4675.195924] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4675.198784] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4675.201923] ? __next_timer_interrupt+0x160/0x160 [ 4675.204102] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4675.209767] ? do_wait_intr+0xf0/0xf0 [ 4675.212383] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4675.222088] ? kthread+0x1d1/0x200 [ 4675.227721] ? set_kthread_struct+0x70/0x70 [ 4675.229157] ? ret_from_fork+0x1f/0x30 [ 4676.053484] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 4676.056211] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4676.105223] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4676.125097] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4676.133875] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4676.142191] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4676.147399] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4676.168402] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4676.175846] RAX: 0000000000007e18 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4676.186872] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4676.195276] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4676.198723] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4676.200634] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4676.204081] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4676.208387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4676.211210] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4676.214929] Call Trace: [ 4676.216284] [ 4676.217283] ? show_regs.cold.9+0x22/0x2f [ 4676.219386] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4676.221614] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4676.223855] ? watchdog+0x40/0x40 [ 4676.226345] ? hrtimer_interrupt+0x128/0x2c0 [ 4676.229803] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4676.235962] ? apic_timer_interrupt+0xf/0x20 [ 4676.240870] [ 4676.243244] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4676.246295] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4676.248133] ? newidle_balance.isra.93+0x207/0x580 [ 4676.253305] do_raw_spin_lock+0x177/0x190 [ 4676.256289] _raw_spin_lock+0x21/0x30 [ 4676.260911] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4676.274312] process_one_work+0x2c8/0x700 [ 4676.282258] worker_thread+0x296/0x6e0 [ 4676.285629] ? rescuer_thread+0x570/0x570 [ 4676.286737] kthread+0x1d1/0x200 [ 4676.287408] ? set_kthread_struct+0x70/0x70 [ 4676.290223] ret_from_fork+0x1f/0x30 [ 4680.166613] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2043s! [ 4680.177316] Showing busy workqueues and worker pools: [ 4680.181095] workqueue events: flags=0x0 [ 4680.185428] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4680.190093] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4680.208348] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4680.224778] pending: cpu_latency_work [ptlrpc] [ 4680.235723] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4680.244128] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4680.255774] workqueue events_long: flags=0x0 [ 4680.258438] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4680.265277] pending: lu_objects_flush [obdclass] [ 4680.270757] workqueue events_unbound: flags=0x2 [ 4680.273518] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4680.281704] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4680.289942] workqueue events_power_efficient: flags=0x80 [ 4680.291959] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4680.302478] pending: check_lifetime, do_cache_clean [ 4680.309419] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4680.315427] pending: neigh_periodic_work, neigh_periodic_work [ 4680.322474] workqueue mm_percpu_wq: flags=0x8 [ 4680.327254] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4680.329488] pending: vmstat_update, lru_add_drain_per_cpu [ 4680.336031] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4680.338832] pending: vmstat_update, lru_add_drain_per_cpu [ 4680.342098] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4680.349377] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4680.353926] workqueue xprtiod: flags=0xa [ 4680.358983] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4680.362698] pending: xprt_autoclose, xs_error_handle [ 4680.365776] workqueue ipv6_addrconf: flags=0x40008 [ 4680.368431] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4680.371804] pending: addrconf_verify_work [ 4680.373850] workqueue knbd0-recv: flags=0x1a [ 4680.375957] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4680.382934] in-flight: 651:recv_work [ 4680.386849] workqueue ptlrpc_pinger: flags=0x2 [ 4680.390537] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4680.393386] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4680.397626] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2043s workers=4 idle: 15979 31824 8 [ 4680.403017] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4670s workers=2 idle: 654 [ 4684.059228] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4684.062029] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4684.101602] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4684.119770] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4684.129480] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4684.138021] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4684.154404] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4684.163256] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4684.176035] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4684.179160] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4684.189178] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4684.202090] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4684.214859] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4684.228111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4684.230288] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4684.234807] Call Trace: [ 4684.235724] [ 4684.241295] ? show_regs.cold.9+0x22/0x2f [ 4684.242693] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4684.244207] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4684.252206] ? watchdog+0x40/0x40 [ 4684.253416] ? hrtimer_interrupt+0x128/0x2c0 [ 4684.257894] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4684.261368] ? apic_timer_interrupt+0xf/0x20 [ 4684.263145] [ 4684.264239] ? native_safe_halt+0x2e/0x50 [ 4684.267272] kvm_wait+0xcd/0xe0 [ 4684.268378] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4684.269893] do_raw_spin_lock+0x177/0x190 [ 4684.274738] _raw_spin_lock+0x21/0x30 [ 4684.276621] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4684.288718] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4684.292899] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4684.296183] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4684.301925] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4684.308396] ? woken_wake_function+0x30/0x30 [ 4684.309978] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4684.317318] kthread+0x1d1/0x200 [ 4684.321016] ? set_kthread_struct+0x70/0x70 [ 4684.324247] ret_from_fork+0x1f/0x30 [ 4700.059428] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 4700.061470] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 4700.066654] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4700.066731] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4700.066738] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4700.066741] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4700.066755] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4700.066759] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4700.066763] RAX: 0000000000007e99 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4700.066766] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4700.066768] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4700.066770] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4700.066772] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4700.066775] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4700.066778] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4700.066780] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4700.066788] Call Trace: [ 4700.066794] [ 4700.066800] ? show_regs.cold.9+0x22/0x2f [ 4700.066809] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4700.066814] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4700.066818] ? watchdog+0x40/0x40 [ 4700.066823] ? hrtimer_interrupt+0x128/0x2c0 [ 4700.066828] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4700.066836] ? apic_timer_interrupt+0xf/0x20 [ 4700.066841] [ 4700.066843] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4700.066850] do_raw_spin_lock+0x177/0x190 [ 4700.066854] _raw_spin_lock+0x21/0x30 [ 4700.066859] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4700.072391] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4700.072469] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4700.072478] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4700.072481] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4700.072496] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4700.072501] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4700.072506] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4700.072508] RDX: 0000000000007d8e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4700.072511] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4700.072513] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4700.072516] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4700.072519] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4700.072522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4700.072525] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4700.072535] Call Trace: [ 4700.107742] ? do_raw_spin_unlock+0x75/0x190 [ 4700.107767] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4700.111179] ? do_wait_intr+0xf0/0xf0 [ 4700.111190] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4700.111494] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4700.111712] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4700.111927] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4700.112050] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4700.112096] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4700.112127] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4700.112156] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4700.112254] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4700.113203] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4700.113315] lmv_intent_open+0x338/0x1150 [lmv] [ 4700.113346] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4700.113448] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4700.113537] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4700.113561] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4700.113655] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4700.113751] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4700.113835] ? do_raw_spin_unlock+0x75/0x190 [ 4700.113843] ? _raw_spin_unlock+0x12/0x30 [ 4700.113851] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4700.113881] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4700.114063] ll_atomic_open+0x36a/0xd70 [lustre] [ 4700.123361] [ 4700.123374] ? show_regs.cold.9+0x22/0x2f [ 4700.123390] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4700.123397] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4700.123401] ? watchdog+0x40/0x40 [ 4700.123407] ? hrtimer_interrupt+0x128/0x2c0 [ 4700.123410] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4700.123418] ? apic_timer_interrupt+0xf/0x20 [ 4700.123423] [ 4700.123426] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4700.123433] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4700.123437] do_raw_spin_lock+0x177/0x190 [ 4700.123441] _raw_spin_lock+0x21/0x30 [ 4700.123448] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4700.123727] obd_mod_free+0x46/0x140 [ptlrpc] [ 4700.123919] mdc_commit_open+0x91/0xb0 [mdc] [ 4700.123959] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4700.124153] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4700.124367] after_reply+0xe2c/0x1780 [ptlrpc] [ 4700.124563] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4700.124757] ? __next_timer_interrupt+0x160/0x160 [ 4700.124767] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4700.124964] ? do_wait_intr+0xf0/0xf0 [ 4700.124974] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4700.125180] kthread+0x1d1/0x200 [ 4700.125185] ? set_kthread_struct+0x70/0x70 [ 4700.125189] ret_from_fork+0x1f/0x30 [ 4700.529148] lookup_open+0x356/0xb40 [ 4700.530891] do_last+0x313/0xfc0 [ 4700.532363] ? nd_jump_root+0xe5/0x160 [ 4700.534040] ? path_init+0x437/0x520 [ 4700.535146] path_openat+0xf7/0x500 [ 4700.536914] ? do_raw_spin_unlock+0x75/0x190 [ 4700.539944] do_filp_open+0x99/0x140 [ 4700.543171] ? getname_flags+0x6e/0x330 [ 4700.553125] ? __check_object_size+0xff/0x256 [ 4700.559506] ? do_raw_spin_unlock+0x75/0x190 [ 4700.563161] ? _raw_spin_unlock+0x12/0x30 [ 4700.565691] do_sys_openat2+0x2b4/0x410 [ 4700.568923] do_sys_open+0x73/0xa0 [ 4700.575206] __x64_sys_openat+0x24/0x30 [ 4700.577718] do_syscall_64+0xc1/0x440 [ 4700.578618] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4700.586132] RIP: 0033:0x7f53425a5332 [ 4700.591802] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4700.610655] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4700.614812] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4700.625016] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4700.630622] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4700.636703] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4700.641157] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4708.052016] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 4708.055159] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4708.084359] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4708.089481] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4708.093715] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4708.097073] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4708.099498] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4708.110423] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4708.113282] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4708.117684] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4708.121635] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4708.125311] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4708.128939] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4708.135863] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4708.144842] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4708.148869] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4708.155097] Call Trace: [ 4708.155688] [ 4708.156361] ? show_regs.cold.9+0x22/0x2f [ 4708.162369] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4708.164936] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4708.168153] ? watchdog+0x40/0x40 [ 4708.170141] ? hrtimer_interrupt+0x128/0x2c0 [ 4708.173370] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4708.176715] ? apic_timer_interrupt+0xf/0x20 [ 4708.182401] [ 4708.183431] ? native_safe_halt+0x2e/0x50 [ 4708.191348] kvm_wait+0xcd/0xe0 [ 4708.192865] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4708.194805] ? newidle_balance.isra.93+0x207/0x580 [ 4708.196878] do_raw_spin_lock+0x177/0x190 [ 4708.198667] _raw_spin_lock+0x21/0x30 [ 4708.200772] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4708.205227] process_one_work+0x2c8/0x700 [ 4708.213804] worker_thread+0x296/0x6e0 [ 4708.216239] ? rescuer_thread+0x570/0x570 [ 4708.218804] kthread+0x1d1/0x200 [ 4708.220872] ? set_kthread_struct+0x70/0x70 [ 4708.223386] ret_from_fork+0x1f/0x30 [ 4710.879818] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2073s! [ 4710.890467] Showing busy workqueues and worker pools: [ 4710.895644] workqueue events: flags=0x0 [ 4710.901713] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4710.903972] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4710.922294] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4710.931570] pending: cpu_latency_work [ptlrpc] [ 4710.933520] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4710.942213] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4710.952479] workqueue events_long: flags=0x0 [ 4710.958146] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4710.965760] pending: lu_objects_flush [obdclass] [ 4710.967609] workqueue events_unbound: flags=0x2 [ 4710.973746] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4710.978972] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4710.986869] workqueue events_power_efficient: flags=0x80 [ 4710.988741] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4710.993868] pending: check_lifetime, do_cache_clean [ 4710.997559] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4710.999873] pending: neigh_periodic_work, neigh_periodic_work [ 4711.009168] workqueue mm_percpu_wq: flags=0x8 [ 4711.011861] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4711.016247] pending: vmstat_update, lru_add_drain_per_cpu [ 4711.019276] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4711.028863] pending: vmstat_update, lru_add_drain_per_cpu [ 4711.035479] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4711.037835] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4711.048855] workqueue xprtiod: flags=0xa [ 4711.051136] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4711.057097] pending: xprt_autoclose, xs_error_handle [ 4711.060138] workqueue ipv6_addrconf: flags=0x40008 [ 4711.063166] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4711.070436] pending: addrconf_verify_work [ 4711.072857] workqueue knbd0-recv: flags=0x1a [ 4711.076180] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4711.083347] in-flight: 651:recv_work [ 4711.085547] workqueue ptlrpc_pinger: flags=0x2 [ 4711.087124] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4711.094149] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4711.100793] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2074s workers=4 idle: 15979 31824 8 [ 4711.105642] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4701s workers=2 idle: 654 [ 4712.058086] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4712.060910] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4712.097637] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4712.102494] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4712.105034] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4712.110526] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4712.117510] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4712.120765] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4712.129353] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4712.138310] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4712.141798] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4712.144422] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4712.147890] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4712.154940] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4712.156764] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4712.162795] Call Trace: [ 4712.166229] [ 4712.167191] ? show_regs.cold.9+0x22/0x2f [ 4712.168972] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4712.172820] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4712.175766] ? watchdog+0x40/0x40 [ 4712.176931] ? hrtimer_interrupt+0x128/0x2c0 [ 4712.186987] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4712.191591] ? apic_timer_interrupt+0xf/0x20 [ 4712.196490] [ 4712.197316] ? native_safe_halt+0x2e/0x50 [ 4712.199545] kvm_wait+0xcd/0xe0 [ 4712.200598] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4712.202909] do_raw_spin_lock+0x177/0x190 [ 4712.204434] _raw_spin_lock+0x21/0x30 [ 4712.205991] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4712.208740] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4712.210956] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4712.215414] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4712.219294] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4712.223071] ? woken_wake_function+0x30/0x30 [ 4712.225381] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4712.228659] kthread+0x1d1/0x200 [ 4712.231270] ? set_kthread_struct+0x70/0x70 [ 4712.238851] ret_from_fork+0x1f/0x30 [ 4728.057573] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 4728.058393] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 4728.063388] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4728.063460] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4728.063467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4728.063710] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4728.074580] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4728.074660] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4728.074666] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4728.074669] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4728.074683] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4728.074687] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4728.074691] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4728.074693] RDX: 00000000000044fa RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4728.074695] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4728.074697] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4728.074700] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4728.074703] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4728.074706] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4728.074708] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4728.122235] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4728.122250] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4728.122256] RAX: 0000000000007b0a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4728.122258] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4728.122260] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4728.122262] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4728.122264] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4728.122267] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4728.122270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4728.122273] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4728.122283] Call Trace: [ 4728.122289] [ 4728.122294] ? show_regs.cold.9+0x22/0x2f [ 4728.122307] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4728.122313] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4728.122317] ? watchdog+0x40/0x40 [ 4728.122322] ? hrtimer_interrupt+0x128/0x2c0 [ 4728.122325] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4728.122334] ? apic_timer_interrupt+0xf/0x20 [ 4728.122338] [ 4728.122341] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4728.122351] do_raw_spin_lock+0x177/0x190 [ 4728.122356] _raw_spin_lock+0x21/0x30 [ 4728.122361] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4728.122628] ? do_raw_spin_unlock+0x75/0x190 [ 4728.122633] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4728.123161] ? do_wait_intr+0xf0/0xf0 [ 4728.123175] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4728.123371] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4728.124846] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4728.125090] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4728.125205] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4728.125248] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4728.125275] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4728.125303] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4728.125387] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4728.136159] Call Trace: [ 4728.136171] [ 4728.136177] ? show_regs.cold.9+0x22/0x2f [ 4728.136191] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4728.136198] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4728.136202] ? watchdog+0x40/0x40 [ 4728.136206] ? hrtimer_interrupt+0x128/0x2c0 [ 4728.136210] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4728.136217] ? apic_timer_interrupt+0xf/0x20 [ 4728.136221] [ 4728.136225] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4728.136231] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4728.136236] do_raw_spin_lock+0x177/0x190 [ 4728.136241] _raw_spin_lock+0x21/0x30 [ 4728.136248] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4728.136518] obd_mod_free+0x46/0x140 [ptlrpc] [ 4728.137954] mdc_commit_open+0x91/0xb0 [mdc] [ 4728.138000] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4728.138235] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4728.138431] after_reply+0xe2c/0x1780 [ptlrpc] [ 4728.138626] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4728.141537] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4728.141589] lmv_intent_open+0x338/0x1150 [lmv] [ 4728.141618] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4728.141743] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4728.141825] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4728.141848] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4728.141931] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4728.142017] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4728.142114] ? do_raw_spin_unlock+0x75/0x190 [ 4728.142122] ? _raw_spin_unlock+0x12/0x30 [ 4728.142129] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4728.142156] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4728.142184] ll_atomic_open+0x36a/0xd70 [lustre] [ 4728.142293] lookup_open+0x356/0xb40 [ 4728.142303] do_last+0x313/0xfc0 [ 4728.142308] ? nd_jump_root+0xe5/0x160 [ 4728.142312] ? path_init+0x437/0x520 [ 4728.142316] path_openat+0xf7/0x500 [ 4728.142321] ? do_raw_spin_unlock+0x75/0x190 [ 4728.142326] do_filp_open+0x99/0x140 [ 4728.142332] ? getname_flags+0x6e/0x330 [ 4728.142337] ? __check_object_size+0xff/0x256 [ 4728.142345] ? do_raw_spin_unlock+0x75/0x190 [ 4728.142349] ? _raw_spin_unlock+0x12/0x30 [ 4728.142354] do_sys_openat2+0x2b4/0x410 [ 4728.142358] do_sys_open+0x73/0xa0 [ 4728.142363] __x64_sys_openat+0x24/0x30 [ 4728.142366] do_syscall_64+0xc1/0x440 [ 4728.142372] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4728.142379] RIP: 0033:0x7f53425a5332 [ 4728.142387] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4728.142390] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4728.142395] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4728.142398] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4728.142400] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4728.142402] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4728.142404] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4728.515870] ? __next_timer_interrupt+0x160/0x160 [ 4728.517469] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4728.519042] ? do_wait_intr+0xf0/0xf0 [ 4728.520229] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4728.522049] kthread+0x1d1/0x200 [ 4728.523160] ? set_kthread_struct+0x70/0x70 [ 4728.524484] ret_from_fork+0x1f/0x30 [ 4736.056389] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 4736.061028] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4736.096297] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4736.101159] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4736.110165] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4736.116053] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4736.121007] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4736.136400] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4736.140192] RAX: 0000000000007c8b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4736.144471] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4736.149806] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4736.154306] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4736.161037] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4736.165817] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4736.170459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4736.172151] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4736.174499] Call Trace: [ 4736.175216] [ 4736.175956] ? show_regs.cold.9+0x22/0x2f [ 4736.177348] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4736.178859] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4736.180268] ? watchdog+0x40/0x40 [ 4736.181481] ? hrtimer_interrupt+0x128/0x2c0 [ 4736.182834] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4736.184194] ? apic_timer_interrupt+0xf/0x20 [ 4736.185795] [ 4736.186336] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4736.187967] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4736.190212] ? newidle_balance.isra.93+0x207/0x580 [ 4736.193336] do_raw_spin_lock+0x177/0x190 [ 4736.197093] _raw_spin_lock+0x21/0x30 [ 4736.199863] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4736.207568] process_one_work+0x2c8/0x700 [ 4736.216761] worker_thread+0x296/0x6e0 [ 4736.219534] ? rescuer_thread+0x570/0x570 [ 4736.222005] kthread+0x1d1/0x200 [ 4736.223324] ? set_kthread_struct+0x70/0x70 [ 4736.225974] ret_from_fork+0x1f/0x30 [ 4740.061899] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4740.068832] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4740.101889] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4740.110361] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4740.120365] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4740.125765] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4740.137888] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4740.141697] RAX: 0000000000004c3c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4740.144748] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4740.147716] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4740.151144] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4740.154237] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4740.158330] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4740.161915] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4740.164923] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4740.168737] Call Trace: [ 4740.169964] [ 4740.170821] ? show_regs.cold.9+0x22/0x2f [ 4740.172498] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4740.174432] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4740.176373] ? watchdog+0x40/0x40 [ 4740.177803] ? hrtimer_interrupt+0x128/0x2c0 [ 4740.179731] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4740.181976] ? apic_timer_interrupt+0xf/0x20 [ 4740.183690] [ 4740.184739] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4740.187322] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4740.189410] do_raw_spin_lock+0x177/0x190 [ 4740.191007] _raw_spin_lock+0x21/0x30 [ 4740.193072] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4740.196231] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4740.201329] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4740.204304] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4740.208222] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4740.211679] ? woken_wake_function+0x30/0x30 [ 4740.213229] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4740.216387] kthread+0x1d1/0x200 [ 4740.217310] ? set_kthread_struct+0x70/0x70 [ 4740.218913] ret_from_fork+0x1f/0x30 [ 4741.602025] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2104s! [ 4741.605892] Showing busy workqueues and worker pools: [ 4741.609896] workqueue events: flags=0x0 [ 4741.611928] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4741.620324] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4741.628555] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4741.633126] pending: cpu_latency_work [ptlrpc] [ 4741.636284] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4741.640984] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4741.653703] workqueue events_long: flags=0x0 [ 4741.655383] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4741.665167] pending: lu_objects_flush [obdclass] [ 4741.671933] workqueue events_unbound: flags=0x2 [ 4741.677240] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4741.683861] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4741.689414] workqueue events_power_efficient: flags=0x80 [ 4741.692814] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4741.696863] pending: check_lifetime, do_cache_clean [ 4741.699829] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4741.703165] pending: neigh_periodic_work, neigh_periodic_work [ 4741.705661] workqueue mm_percpu_wq: flags=0x8 [ 4741.707744] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4741.710814] pending: vmstat_update, lru_add_drain_per_cpu [ 4741.713314] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4741.716620] pending: vmstat_update, lru_add_drain_per_cpu [ 4741.719089] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4741.723683] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4741.727969] workqueue xprtiod: flags=0xa [ 4741.729860] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4741.736901] pending: xprt_autoclose, xs_error_handle [ 4741.740207] workqueue ipv6_addrconf: flags=0x40008 [ 4741.742583] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4741.745785] pending: addrconf_verify_work [ 4741.747540] workqueue knbd0-recv: flags=0x1a [ 4741.749355] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4741.751423] in-flight: 651:recv_work [ 4741.753754] workqueue ptlrpc_pinger: flags=0x2 [ 4741.757728] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4741.762222] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4741.765415] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2104s workers=4 idle: 15979 31824 8 [ 4741.771852] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4732s workers=2 idle: 654 [ 4756.055042] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 4756.057017] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 4756.057898] Modules linked in: [ 4756.061122] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4756.061203] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4756.061210] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4756.061213] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4756.061227] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4756.061231] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4756.061235] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4756.061238] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4756.061240] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4756.061243] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4756.061257] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4756.061260] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4756.061263] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4756.061266] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4756.061274] Call Trace: [ 4756.061280] [ 4756.061286] ? show_regs.cold.9+0x22/0x2f [ 4756.061296] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4756.061302] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4756.061306] ? watchdog+0x40/0x40 [ 4756.061310] ? hrtimer_interrupt+0x128/0x2c0 [ 4756.061315] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4756.061322] ? apic_timer_interrupt+0xf/0x20 [ 4756.061326] [ 4756.061330] ? native_safe_halt+0x2e/0x50 [ 4756.061334] kvm_wait+0xcd/0xe0 [ 4756.061465] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4756.061473] do_raw_spin_lock+0x177/0x190 [ 4756.061478] _raw_spin_lock+0x21/0x30 [ 4756.061539] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4756.062091] obd_mod_free+0x46/0x140 [ptlrpc] [ 4756.062320] mdc_commit_open+0x91/0xb0 [mdc] [ 4756.062356] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4756.064122] lustre(O) [ 4756.085734] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4756.086091] after_reply+0xe2c/0x1780 [ptlrpc] [ 4756.086448] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4756.086670] ? __next_timer_interrupt+0x160/0x160 [ 4756.086680] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4756.097030] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4756.097105] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4756.097111] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4756.097114] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4756.097127] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4756.097131] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4756.097135] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4756.097137] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4756.097139] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4756.097141] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4756.097144] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4756.097147] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4756.097150] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4756.097152] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4756.097162] Call Trace: [ 4756.097168] [ 4756.097174] ? show_regs.cold.9+0x22/0x2f [ 4756.097183] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4756.097190] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4756.097195] ? watchdog+0x40/0x40 [ 4756.097199] ? hrtimer_interrupt+0x128/0x2c0 [ 4756.097203] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4756.097209] ? apic_timer_interrupt+0xf/0x20 [ 4756.097214] [ 4756.097217] ? native_safe_halt+0x2e/0x50 [ 4756.097221] kvm_wait+0xcd/0xe0 [ 4756.097228] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4756.097237] do_raw_spin_lock+0x177/0x190 [ 4756.098348] _raw_spin_lock+0x21/0x30 [ 4756.106171] ? do_wait_intr+0xf0/0xf0 [ 4756.106194] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4756.106480] kthread+0x1d1/0x200 [ 4756.106486] ? set_kthread_struct+0x70/0x70 [ 4756.106491] ret_from_fork+0x1f/0x30 [ 4756.403737] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4756.406133] ? do_raw_spin_unlock+0x75/0x190 [ 4756.408332] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4756.410025] ? do_wait_intr+0xf0/0xf0 [ 4756.411065] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4756.414770] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4756.417979] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4756.432957] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4756.445196] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4756.446891] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4756.449126] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4756.450575] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4756.457651] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4756.463984] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4756.470509] lmv_intent_open+0x338/0x1150 [lmv] [ 4756.472218] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4756.478018] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4756.481056] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4756.483202] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4756.486830] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4756.489055] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4756.493934] ? do_raw_spin_unlock+0x75/0x190 [ 4756.499547] ? _raw_spin_unlock+0x12/0x30 [ 4756.507113] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4756.511231] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4756.515417] ll_atomic_open+0x36a/0xd70 [lustre] [ 4756.519778] lookup_open+0x356/0xb40 [ 4756.524127] do_last+0x313/0xfc0 [ 4756.530122] ? nd_jump_root+0xe5/0x160 [ 4756.531912] ? path_init+0x437/0x520 [ 4756.533373] path_openat+0xf7/0x500 [ 4756.534463] ? do_raw_spin_unlock+0x75/0x190 [ 4756.536043] do_filp_open+0x99/0x140 [ 4756.537725] ? getname_flags+0x6e/0x330 [ 4756.540711] ? __check_object_size+0xff/0x256 [ 4756.544209] ? do_raw_spin_unlock+0x75/0x190 [ 4756.546067] ? _raw_spin_unlock+0x12/0x30 [ 4756.548176] do_sys_openat2+0x2b4/0x410 [ 4756.549971] do_sys_open+0x73/0xa0 [ 4756.551092] __x64_sys_openat+0x24/0x30 [ 4756.552812] do_syscall_64+0xc1/0x440 [ 4756.554977] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4756.556923] RIP: 0033:0x7f53425a5332 [ 4756.558392] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4756.565608] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4756.569181] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4756.571796] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4756.574774] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4756.577041] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4756.582185] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4768.052018] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 4768.057458] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4768.058032] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4768.089451] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4768.094414] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) [ 4768.100864] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4768.100880] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4768.107188] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) [ 4768.111847] [ 4768.111857] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4768.111873] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4768.111876] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4768.111881] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4768.111883] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4768.111886] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4768.111887] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4768.111890] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4768.111892] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4768.118857] ksocklnd(O) [ 4768.123310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4768.123320] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4768.123412] Call Trace: [ 4768.123420] [ 4768.123426] ? show_regs.cold.9+0x22/0x2f [ 4768.123447] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4768.123453] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4768.123461] ? watchdog+0x40/0x40 [ 4768.123465] ? hrtimer_interrupt+0x128/0x2c0 [ 4768.123469] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4768.123476] ? apic_timer_interrupt+0xf/0x20 [ 4768.123480] [ 4768.123487] ? native_safe_halt+0x2e/0x50 [ 4768.123494] kvm_wait+0xcd/0xe0 [ 4768.123501] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4768.123507] ? newidle_balance.isra.93+0x207/0x580 [ 4768.123514] do_raw_spin_lock+0x177/0x190 [ 4768.123518] _raw_spin_lock+0x21/0x30 [ 4768.123526] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4768.123899] process_one_work+0x2c8/0x700 [ 4768.123911] worker_thread+0x296/0x6e0 [ 4768.123915] ? rescuer_thread+0x570/0x570 [ 4768.123919] kthread+0x1d1/0x200 [ 4768.123927] ? set_kthread_struct+0x70/0x70 [ 4768.123931] ret_from_fork+0x1f/0x30 [ 4768.257461] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4768.281407] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4768.289429] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4768.296539] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4768.299701] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4768.311581] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4768.321580] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4768.324586] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4768.327798] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4768.333862] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4768.339117] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4768.342868] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4768.347420] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4768.351925] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4768.354938] Call Trace: [ 4768.355861] [ 4768.356654] ? show_regs.cold.9+0x22/0x2f [ 4768.358245] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4768.360042] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4768.364291] ? watchdog+0x40/0x40 [ 4768.366538] ? hrtimer_interrupt+0x128/0x2c0 [ 4768.373674] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4768.376880] ? apic_timer_interrupt+0xf/0x20 [ 4768.378922] [ 4768.380426] ? native_safe_halt+0x2e/0x50 [ 4768.382617] kvm_wait+0xcd/0xe0 [ 4768.386023] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4768.388003] do_raw_spin_lock+0x177/0x190 [ 4768.392800] _raw_spin_lock+0x21/0x30 [ 4768.398222] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4768.402237] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4768.405481] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4768.413515] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4768.417308] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4768.425507] ? woken_wake_function+0x30/0x30 [ 4768.429537] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4768.436533] kthread+0x1d1/0x200 [ 4768.439726] ? set_kthread_struct+0x70/0x70 [ 4768.445072] ret_from_fork+0x1f/0x30 [ 4771.808043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2134s! [ 4771.815758] Showing busy workqueues and worker pools: [ 4771.817788] workqueue events: flags=0x0 [ 4771.819498] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4771.822287] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4771.830303] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4771.842120] pending: cpu_latency_work [ptlrpc] [ 4771.850407] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4771.858544] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4771.865373] workqueue events_long: flags=0x0 [ 4771.866861] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4771.878525] pending: lu_objects_flush [obdclass] [ 4771.881428] workqueue events_unbound: flags=0x2 [ 4771.883298] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4771.885897] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4771.887929] workqueue events_power_efficient: flags=0x80 [ 4771.890836] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4771.894248] pending: check_lifetime, do_cache_clean [ 4771.896519] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4771.899426] pending: neigh_periodic_work, neigh_periodic_work [ 4771.902245] workqueue mm_percpu_wq: flags=0x8 [ 4771.903771] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4771.906436] pending: vmstat_update, lru_add_drain_per_cpu [ 4771.908696] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4771.911291] pending: vmstat_update, lru_add_drain_per_cpu [ 4771.913436] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4771.916201] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4771.918828] workqueue xprtiod: flags=0xa [ 4771.920261] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4771.923904] pending: xprt_autoclose, xs_error_handle [ 4771.927707] workqueue ipv6_addrconf: flags=0x40008 [ 4771.934793] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4771.936781] pending: addrconf_verify_work [ 4771.942491] workqueue knbd0-recv: flags=0x1a [ 4771.944550] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4771.949879] in-flight: 651:recv_work [ 4771.952177] workqueue ptlrpc_pinger: flags=0x2 [ 4771.954283] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4771.957844] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4771.960792] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2134s workers=4 idle: 15979 31824 8 [ 4771.969549] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4762s workers=2 idle: 654 [ 4784.055011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 4784.057023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 4784.069666] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4784.069743] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4784.069751] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4784.069754] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4784.077827] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4784.077900] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4784.077907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4784.077910] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4784.077924] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4784.077927] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4784.077932] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4784.077934] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4784.077936] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4784.077937] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4784.077939] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4784.077942] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4784.077944] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4784.077946] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4784.077955] Call Trace: [ 4784.077960] [ 4784.077965] ? show_regs.cold.9+0x22/0x2f [ 4784.077975] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4784.077982] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4784.077987] ? watchdog+0x40/0x40 [ 4784.077990] ? hrtimer_interrupt+0x128/0x2c0 [ 4784.077994] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4784.078000] ? apic_timer_interrupt+0xf/0x20 [ 4784.078004] [ 4784.078006] ? native_safe_halt+0x2e/0x50 [ 4784.078010] kvm_wait+0xcd/0xe0 [ 4784.078017] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4784.078024] do_raw_spin_lock+0x177/0x190 [ 4784.078027] _raw_spin_lock+0x21/0x30 [ 4784.078033] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4784.122418] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4784.122669] RSP: 0018:ffffbb5381503500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4784.122677] RAX: 00000000000033a1 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4784.122679] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4784.122682] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4784.122684] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4784.122687] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4784.122691] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4784.122694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4784.122697] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4784.122705] Call Trace: [ 4784.122711] [ 4784.122717] ? show_regs.cold.9+0x22/0x2f [ 4784.122730] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4784.122736] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4784.122740] ? watchdog+0x40/0x40 [ 4784.122744] ? hrtimer_interrupt+0x128/0x2c0 [ 4784.122748] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4784.123118] ? apic_timer_interrupt+0xf/0x20 [ 4784.123122] [ 4784.123125] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4784.123134] do_raw_spin_lock+0x177/0x190 [ 4784.123139] _raw_spin_lock+0x21/0x30 [ 4784.123144] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4784.123415] ? do_raw_spin_unlock+0x75/0x190 [ 4784.123421] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4784.124333] ? do_wait_intr+0xf0/0xf0 [ 4784.124344] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4784.124559] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4784.134146] obd_mod_free+0x46/0x140 [ptlrpc] [ 4784.142329] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4784.142640] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4784.142901] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4784.143014] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4784.143108] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4784.143139] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4784.143225] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4784.143595] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4784.143622] lmv_intent_open+0x338/0x1150 [lmv] [ 4784.143653] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4784.143741] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4784.143885] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4784.143908] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4784.144137] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4784.144217] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4784.144297] ? do_raw_spin_unlock+0x75/0x190 [ 4784.144305] ? _raw_spin_unlock+0x12/0x30 [ 4784.144312] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4784.144345] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4784.144365] ll_atomic_open+0x36a/0xd70 [lustre] [ 4784.144589] lookup_open+0x356/0xb40 [ 4784.144599] do_last+0x313/0xfc0 [ 4784.144604] ? nd_jump_root+0xe5/0x160 [ 4784.144608] ? path_init+0x437/0x520 [ 4784.144612] path_openat+0xf7/0x500 [ 4784.144616] ? do_raw_spin_unlock+0x75/0x190 [ 4784.144620] do_filp_open+0x99/0x140 [ 4784.144626] ? getname_flags+0x6e/0x330 [ 4784.144630] ? __check_object_size+0xff/0x256 [ 4784.144637] ? do_raw_spin_unlock+0x75/0x190 [ 4784.144641] ? _raw_spin_unlock+0x12/0x30 [ 4784.144645] do_sys_openat2+0x2b4/0x410 [ 4784.144649] do_sys_open+0x73/0xa0 [ 4784.144654] __x64_sys_openat+0x24/0x30 [ 4784.144657] do_syscall_64+0xc1/0x440 [ 4784.144664] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4784.144670] RIP: 0033:0x7f53425a5332 [ 4784.144677] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4784.144681] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4784.144686] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4784.144688] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4784.144690] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4784.144692] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4784.144694] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4784.850721] mdc_commit_open+0x91/0xb0 [mdc] [ 4784.860612] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4784.867460] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4784.879303] after_reply+0xe2c/0x1780 [ptlrpc] [ 4784.888473] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4784.893117] ? __next_timer_interrupt+0x160/0x160 [ 4784.894765] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4784.901693] ? do_wait_intr+0xf0/0xf0 [ 4784.905722] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4784.912445] kthread+0x1d1/0x200 [ 4784.915525] ? set_kthread_struct+0x70/0x70 [ 4784.923985] ret_from_fork+0x1f/0x30 [ 4796.052023] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 4796.058413] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4796.059536] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4796.099190] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4796.099203] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4796.099210] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4796.099498] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4796.099509] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4796.099513] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4796.099518] RAX: 0000000000001f87 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4796.099520] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4796.099523] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4796.099525] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4796.099527] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4796.099531] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4796.099534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4796.099537] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4796.101123] Call Trace: [ 4796.101134] [ 4796.101139] ? show_regs.cold.9+0x22/0x2f [ 4796.101152] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4796.101158] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4796.101162] ? watchdog+0x40/0x40 [ 4796.101166] ? hrtimer_interrupt+0x128/0x2c0 [ 4796.101170] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4796.101178] ? apic_timer_interrupt+0xf/0x20 [ 4796.101183] [ 4796.101186] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4796.101192] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4796.101195] ? newidle_balance.isra.93+0x207/0x580 [ 4796.101202] do_raw_spin_lock+0x177/0x190 [ 4796.101207] _raw_spin_lock+0x21/0x30 [ 4796.101212] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4796.101500] process_one_work+0x2c8/0x700 [ 4796.101508] worker_thread+0x296/0x6e0 [ 4796.101512] ? rescuer_thread+0x570/0x570 [ 4796.101516] kthread+0x1d1/0x200 [ 4796.101521] ? set_kthread_struct+0x70/0x70 [ 4796.101524] ret_from_fork+0x1f/0x30 [ 4796.352172] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4796.381452] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4796.388596] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4796.399881] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4796.412858] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4796.422942] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4796.434164] RAX: 0000000000007f91 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4796.439967] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4796.443309] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4796.448165] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4796.452118] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4796.456526] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4796.461153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4796.473599] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4796.478508] Call Trace: [ 4796.485620] [ 4796.486357] ? show_regs.cold.9+0x22/0x2f [ 4796.487699] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4796.489501] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4796.491607] ? watchdog+0x40/0x40 [ 4796.493435] ? hrtimer_interrupt+0x128/0x2c0 [ 4796.497217] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4796.501423] ? apic_timer_interrupt+0xf/0x20 [ 4796.504514] [ 4796.506328] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4796.508055] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4796.512022] do_raw_spin_lock+0x177/0x190 [ 4796.513804] _raw_spin_lock+0x21/0x30 [ 4796.515239] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4796.518887] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4796.521271] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4796.523489] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4796.525887] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4796.531881] ? woken_wake_function+0x30/0x30 [ 4796.533534] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4796.541408] kthread+0x1d1/0x200 [ 4796.542906] ? set_kthread_struct+0x70/0x70 [ 4796.546635] ret_from_fork+0x1f/0x30 [ 4802.018080] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2164s! [ 4802.021074] Showing busy workqueues and worker pools: [ 4802.022477] workqueue events: flags=0x0 [ 4802.023857] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4802.026316] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4802.030355] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4802.032664] pending: cpu_latency_work [ptlrpc] [ 4802.034875] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4802.037215] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4802.040160] workqueue events_long: flags=0x0 [ 4802.041679] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4802.045622] pending: lu_objects_flush [obdclass] [ 4802.049621] workqueue events_unbound: flags=0x2 [ 4802.052618] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4802.058299] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4802.064859] workqueue events_power_efficient: flags=0x80 [ 4802.070948] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4802.082814] pending: check_lifetime, do_cache_clean [ 4802.085748] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4802.089395] pending: neigh_periodic_work, neigh_periodic_work [ 4802.096630] workqueue mm_percpu_wq: flags=0x8 [ 4802.103688] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4802.109659] pending: vmstat_update, lru_add_drain_per_cpu [ 4802.114337] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4802.120143] pending: vmstat_update, lru_add_drain_per_cpu [ 4802.122781] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4802.125114] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4802.127960] workqueue xprtiod: flags=0xa [ 4802.131795] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4802.138607] pending: xprt_autoclose, xs_error_handle [ 4802.140355] workqueue ipv6_addrconf: flags=0x40008 [ 4802.148598] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4802.154156] pending: addrconf_verify_work [ 4802.157930] workqueue knbd0-recv: flags=0x1a [ 4802.161791] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4802.167968] in-flight: 651:recv_work [ 4802.173527] workqueue ptlrpc_pinger: flags=0x2 [ 4802.176884] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4802.182582] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4802.190911] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2165s workers=4 idle: 15979 31824 8 [ 4802.202117] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4792s workers=2 idle: 654 [ 4812.055027] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 4812.057030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 4812.057965] Modules linked in: [ 4812.060970] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4812.061042] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4812.061049] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4812.061051] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4812.061065] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4812.061069] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4812.061073] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4812.061075] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4812.061077] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4812.061079] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4812.061082] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4812.061085] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4812.061087] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4812.061090] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4812.061098] Call Trace: [ 4812.061103] [ 4812.061109] ? show_regs.cold.9+0x22/0x2f [ 4812.061118] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4812.061124] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4812.061128] ? watchdog+0x40/0x40 [ 4812.061132] ? hrtimer_interrupt+0x128/0x2c0 [ 4812.061136] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4812.061142] ? apic_timer_interrupt+0xf/0x20 [ 4812.061146] [ 4812.061149] ? native_safe_halt+0x2e/0x50 [ 4812.061153] kvm_wait+0xcd/0xe0 [ 4812.061160] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4812.061166] do_raw_spin_lock+0x177/0x190 [ 4812.061170] _raw_spin_lock+0x21/0x30 [ 4812.061175] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4812.063183] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4812.063248] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4812.063254] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4812.063257] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4812.108638] obd_mod_free+0x46/0x140 [ptlrpc] [ 4812.108929] mdc_commit_open+0x91/0xb0 [mdc] [ 4812.108967] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4812.109244] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4812.120111] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4812.120125] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4812.120132] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4812.120135] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4812.120137] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4812.120139] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4812.120142] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4812.120146] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4812.120148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4812.120150] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4812.120159] Call Trace: [ 4812.120166] [ 4812.120173] ? show_regs.cold.9+0x22/0x2f [ 4812.120187] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4812.120194] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4812.120199] ? watchdog+0x40/0x40 [ 4812.120203] ? hrtimer_interrupt+0x128/0x2c0 [ 4812.120207] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4812.120214] ? apic_timer_interrupt+0xf/0x20 [ 4812.120219] [ 4812.120221] ? native_safe_halt+0x2e/0x50 [ 4812.120225] kvm_wait+0xcd/0xe0 [ 4812.120231] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4812.120240] do_raw_spin_lock+0x177/0x190 [ 4812.120244] _raw_spin_lock+0x21/0x30 [ 4812.120248] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4812.120558] ? do_raw_spin_unlock+0x75/0x190 [ 4812.120565] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4812.120765] ? do_wait_intr+0xf0/0xf0 [ 4812.120775] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4812.120977] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4812.121196] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4812.135507] after_reply+0xe2c/0x1780 [ptlrpc] [ 4812.138113] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4812.138239] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4812.138342] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4812.138374] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4812.138405] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4812.138497] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4812.138787] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4812.138817] lmv_intent_open+0x338/0x1150 [lmv] [ 4812.138846] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4812.138938] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4812.139025] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4812.139123] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4812.139211] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4812.140877] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4812.140969] ? do_raw_spin_unlock+0x75/0x190 [ 4812.140978] ? _raw_spin_unlock+0x12/0x30 [ 4812.140984] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4812.141009] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4812.141041] ll_atomic_open+0x36a/0xd70 [lustre] [ 4812.141135] lookup_open+0x356/0xb40 [ 4812.141144] do_last+0x313/0xfc0 [ 4812.141149] ? nd_jump_root+0xe5/0x160 [ 4812.141153] ? path_init+0x437/0x520 [ 4812.141159] path_openat+0xf7/0x500 [ 4812.141163] ? do_raw_spin_unlock+0x75/0x190 [ 4812.141168] do_filp_open+0x99/0x140 [ 4812.141174] ? getname_flags+0x6e/0x330 [ 4812.141179] ? __check_object_size+0xff/0x256 [ 4812.141187] ? do_raw_spin_unlock+0x75/0x190 [ 4812.141191] ? _raw_spin_unlock+0x12/0x30 [ 4812.141195] do_sys_openat2+0x2b4/0x410 [ 4812.141199] do_sys_open+0x73/0xa0 [ 4812.141203] __x64_sys_openat+0x24/0x30 [ 4812.141207] do_syscall_64+0xc1/0x440 [ 4812.141214] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4812.141219] RIP: 0033:0x7f53425a5332 [ 4812.141226] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4812.141231] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4812.141236] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4812.141238] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4812.141240] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4812.141242] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4812.141244] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4812.621414] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4812.626884] ? __next_timer_interrupt+0x160/0x160 [ 4812.631642] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4812.636467] ? do_wait_intr+0xf0/0xf0 [ 4812.637680] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4812.642418] kthread+0x1d1/0x200 [ 4812.644344] ? set_kthread_struct+0x70/0x70 [ 4812.646695] ret_from_fork+0x1f/0x30 [ 4824.058214] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4824.076373] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4824.127239] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4824.134949] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4824.139752] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4824.142033] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4824.154741] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4824.160037] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4824.162257] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4824.167544] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4824.170718] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4824.174621] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4824.177419] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4824.181971] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4824.188804] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4824.191660] Call Trace: [ 4824.192891] [ 4824.194119] ? show_regs.cold.9+0x22/0x2f [ 4824.198871] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4824.204601] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4824.206417] ? watchdog+0x40/0x40 [ 4824.208171] ? hrtimer_interrupt+0x128/0x2c0 [ 4824.210413] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4824.212650] ? apic_timer_interrupt+0xf/0x20 [ 4824.215873] [ 4824.216807] ? native_safe_halt+0x2e/0x50 [ 4824.222827] kvm_wait+0xcd/0xe0 [ 4824.228373] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4824.230974] do_raw_spin_lock+0x177/0x190 [ 4824.235837] _raw_spin_lock+0x21/0x30 [ 4824.239932] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4824.248265] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4824.253787] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4824.257037] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4824.260169] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4824.266480] ? woken_wake_function+0x30/0x30 [ 4824.267882] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4824.278421] kthread+0x1d1/0x200 [ 4824.280572] ? set_kthread_struct+0x70/0x70 [ 4824.282377] ret_from_fork+0x1f/0x30 [ 4828.052025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 4828.054572] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4828.068738] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4828.073101] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4828.075455] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4828.077558] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4828.078670] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4828.084484] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4828.086801] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4828.088883] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4828.092266] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4828.093801] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4828.095670] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4828.097548] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4828.100964] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4828.104112] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4828.106724] Call Trace: [ 4828.107838] [ 4828.108634] ? show_regs.cold.9+0x22/0x2f [ 4828.111348] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4828.113454] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4828.117175] ? watchdog+0x40/0x40 [ 4828.119121] ? hrtimer_interrupt+0x128/0x2c0 [ 4828.121325] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4828.123759] ? apic_timer_interrupt+0xf/0x20 [ 4828.127269] [ 4828.128327] ? native_safe_halt+0x2e/0x50 [ 4828.129292] kvm_wait+0xcd/0xe0 [ 4828.132801] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4828.134577] ? newidle_balance.isra.93+0x207/0x580 [ 4828.136173] do_raw_spin_lock+0x177/0x190 [ 4828.138209] _raw_spin_lock+0x21/0x30 [ 4828.139664] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4828.141993] process_one_work+0x2c8/0x700 [ 4828.143545] worker_thread+0x296/0x6e0 [ 4828.144795] ? rescuer_thread+0x570/0x570 [ 4828.146978] kthread+0x1d1/0x200 [ 4828.149341] ? set_kthread_struct+0x70/0x70 [ 4828.151291] ret_from_fork+0x1f/0x30 [ 4832.227961] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2195s! [ 4832.232739] Showing busy workqueues and worker pools: [ 4832.238450] workqueue events: flags=0x0 [ 4832.241020] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4832.244411] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4832.254930] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4832.258588] pending: cpu_latency_work [ptlrpc] [ 4832.261062] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4832.263172] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4832.265961] workqueue events_long: flags=0x0 [ 4832.267379] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4832.270519] pending: lu_objects_flush [obdclass] [ 4832.276010] workqueue events_unbound: flags=0x2 [ 4832.278142] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4832.281141] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4832.283545] workqueue events_power_efficient: flags=0x80 [ 4832.285584] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4832.287608] pending: check_lifetime, do_cache_clean [ 4832.289571] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4832.292236] pending: neigh_periodic_work, neigh_periodic_work [ 4832.294559] workqueue mm_percpu_wq: flags=0x8 [ 4832.296322] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4832.299414] pending: vmstat_update, lru_add_drain_per_cpu [ 4832.301769] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4832.305072] pending: vmstat_update, lru_add_drain_per_cpu [ 4832.307256] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4832.310323] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4832.316430] workqueue xprtiod: flags=0xa [ 4832.318581] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4832.321420] pending: xprt_autoclose, xs_error_handle [ 4832.323736] workqueue ipv6_addrconf: flags=0x40008 [ 4832.325563] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4832.328198] pending: addrconf_verify_work [ 4832.329841] workqueue knbd0-recv: flags=0x1a [ 4832.331358] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4832.333871] in-flight: 651:recv_work [ 4832.335310] workqueue ptlrpc_pinger: flags=0x2 [ 4832.336816] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4832.338832] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4832.341139] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2195s workers=4 idle: 15979 31824 8 [ 4832.346347] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4822s workers=2 idle: 654 [ 4840.055090] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 4840.057629] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 4840.059024] Modules linked in: lustre(O) [ 4840.062507] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4840.062587] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4840.062594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4840.062596] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4840.062611] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4840.062615] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4840.062620] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4840.066972] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr [ 4840.088152] RDX: 0000000000006104 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4840.088163] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4840.088165] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4840.088168] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4840.088171] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4840.088174] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4840.088177] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4840.088186] Call Trace: [ 4840.088198] [ 4840.088204] ? show_regs.cold.9+0x22/0x2f [ 4840.088215] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4840.088225] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4840.088230] ? watchdog+0x40/0x40 [ 4840.088234] ? hrtimer_interrupt+0x128/0x2c0 [ 4840.088237] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4840.088244] ? apic_timer_interrupt+0xf/0x20 [ 4840.088249] [ 4840.088251] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4840.088257] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4840.088266] do_raw_spin_lock+0x177/0x190 [ 4840.088271] _raw_spin_lock+0x21/0x30 [ 4840.088278] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4840.088681] obd_mod_free+0x46/0x140 [ptlrpc] [ 4840.089156] mdc_commit_open+0x91/0xb0 [mdc] [ 4840.089192] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4840.096447] intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4840.096486] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4840.096493] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4840.096748] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4840.101192] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4840.107624] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4840.107638] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4840.107645] RAX: 0000000000006bd9 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4840.107647] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4840.107649] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4840.107652] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4840.107654] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4840.107656] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4840.107659] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4840.107662] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4840.107671] Call Trace: [ 4840.107677] [ 4840.107683] ? show_regs.cold.9+0x22/0x2f [ 4840.107696] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4840.107702] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4840.107706] ? watchdog+0x40/0x40 [ 4840.107710] ? hrtimer_interrupt+0x128/0x2c0 [ 4840.107713] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4840.107723] ? apic_timer_interrupt+0xf/0x20 [ 4840.107728] [ 4840.107731] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4840.107741] do_raw_spin_lock+0x177/0x190 [ 4840.107746] _raw_spin_lock+0x21/0x30 [ 4840.107751] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4840.108170] ? do_raw_spin_unlock+0x75/0x190 [ 4840.108230] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4840.108646] ? do_wait_intr+0xf0/0xf0 [ 4840.108661] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4840.109082] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4840.109301] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4840.109517] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4840.109630] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4840.109668] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4840.109700] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4840.109735] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4840.109836] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4840.110231] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4840.110260] lmv_intent_open+0x338/0x1150 [lmv] [ 4840.110290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4840.110376] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4840.110518] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4840.110541] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4840.110626] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4840.110709] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4840.119660] after_reply+0xe2c/0x1780 [ptlrpc] [ 4840.120066] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4840.120377] ? __next_timer_interrupt+0x160/0x160 [ 4840.120386] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4840.120600] ? do_wait_intr+0xf0/0xf0 [ 4840.120614] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4840.120938] kthread+0x1d1/0x200 [ 4840.121007] ? set_kthread_struct+0x70/0x70 [ 4840.121011] ret_from_fork+0x1f/0x30 [ 4840.425053] ? do_raw_spin_unlock+0x75/0x190 [ 4840.428304] ? _raw_spin_unlock+0x12/0x30 [ 4840.430544] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4840.433193] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4840.435353] ll_atomic_open+0x36a/0xd70 [lustre] [ 4840.437226] lookup_open+0x356/0xb40 [ 4840.438933] do_last+0x313/0xfc0 [ 4840.440213] ? nd_jump_root+0xe5/0x160 [ 4840.442032] ? path_init+0x437/0x520 [ 4840.443635] path_openat+0xf7/0x500 [ 4840.444866] ? do_raw_spin_unlock+0x75/0x190 [ 4840.446164] do_filp_open+0x99/0x140 [ 4840.447468] ? getname_flags+0x6e/0x330 [ 4840.449306] ? __check_object_size+0xff/0x256 [ 4840.451316] ? do_raw_spin_unlock+0x75/0x190 [ 4840.453124] ? _raw_spin_unlock+0x12/0x30 [ 4840.455003] do_sys_openat2+0x2b4/0x410 [ 4840.456557] do_sys_open+0x73/0xa0 [ 4840.458301] __x64_sys_openat+0x24/0x30 [ 4840.459846] do_syscall_64+0xc1/0x440 [ 4840.461506] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4840.464702] RIP: 0033:0x7f53425a5332 [ 4840.466937] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4840.474908] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4840.478289] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4840.481444] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4840.484518] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4840.487452] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4840.490992] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4852.059216] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4852.074822] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4852.115243] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4852.125867] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4852.130502] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4852.139300] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4852.154195] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4852.160193] RAX: 0000000000007f09 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4852.165226] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4852.169064] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4852.172800] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4852.176162] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4852.187011] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4852.194547] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4852.199220] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4852.202302] Call Trace: [ 4852.203484] [ 4852.204859] ? show_regs.cold.9+0x22/0x2f [ 4852.207072] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4852.209185] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4852.214836] ? watchdog+0x40/0x40 [ 4852.216392] ? hrtimer_interrupt+0x128/0x2c0 [ 4852.218340] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4852.220396] ? apic_timer_interrupt+0xf/0x20 [ 4852.222382] [ 4852.223346] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4852.225625] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4852.228546] do_raw_spin_lock+0x177/0x190 [ 4852.230589] _raw_spin_lock+0x21/0x30 [ 4852.232408] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4852.235806] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4852.238899] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4852.242289] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4852.247260] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4852.254141] ? woken_wake_function+0x30/0x30 [ 4852.258745] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4852.263720] kthread+0x1d1/0x200 [ 4852.265774] ? set_kthread_struct+0x70/0x70 [ 4852.268770] ret_from_fork+0x1f/0x30 [ 4854.496755] rcu: INFO: rcu_sched self-detected stall on CPU [ 4854.514908] rcu: 2-....: (1439954 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 4854.526064] (t=2760070 jiffies g=274185 q=64424) [ 4854.529677] rcu: rcu_sched kthread starved for 2218436 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 4854.534604] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 4854.538706] rcu: RCU grace-period kthread stack dump: [ 4854.541473] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 4854.554096] Call Trace: [ 4854.554904] __schedule+0x351/0xcb0 [ 4854.556468] schedule+0xc0/0x180 [ 4854.559175] schedule_timeout+0xb4/0x190 [ 4854.560513] ? do_raw_spin_unlock+0x75/0x190 [ 4854.563654] ? __next_timer_interrupt+0x160/0x160 [ 4854.565274] ? prepare_to_swait_event+0xf3/0x140 [ 4854.566818] rcu_gp_kthread+0x8dd/0x15b0 [ 4854.569706] ? force_qs_rnp+0x2c0/0x2c0 [ 4854.572930] kthread+0x1d1/0x200 [ 4854.576684] ? set_kthread_struct+0x70/0x70 [ 4854.582325] ret_from_fork+0x1f/0x30 [ 4854.586439] rcu: Stack dump where RCU GP kthread last ran: [ 4854.589422] Sending NMI from CPU 2 to CPUs 0: [ 4854.595485] NMI backtrace for cpu 0 [ 4854.595491] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4854.595493] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4854.595494] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4854.595497] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4854.595499] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4854.595501] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 [ 4854.595503] RAX: 0000000000003048 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4854.595504] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4854.595505] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4854.595507] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4854.595508] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4854.595510] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4854.595511] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4854.595512] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4854.595513] Call Trace: [ 4854.595514] [ 4854.595515] ? show_regs.cold.9+0x22/0x2f [ 4854.595516] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4854.595517] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4854.595518] ? nmi_handle+0xbd/0x210 [ 4854.595519] ? default_do_nmi+0x4b/0x1b0 [ 4854.595520] ? do_nmi+0x280/0x380 [ 4854.595521] ? end_repeat_nmi+0x16/0x1f [ 4854.595522] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4854.595532] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4854.595533] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4854.595534] [ 4854.595534] ? newidle_balance.isra.93+0x207/0x580 [ 4854.595535] do_raw_spin_lock+0x177/0x190 [ 4854.595536] _raw_spin_lock+0x21/0x30 [ 4854.595537] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4854.595538] process_one_work+0x2c8/0x700 [ 4854.595539] worker_thread+0x296/0x6e0 [ 4854.595540] ? rescuer_thread+0x570/0x570 [ 4854.595541] kthread+0x1d1/0x200 [ 4854.595542] ? set_kthread_struct+0x70/0x70 [ 4854.595543] ret_from_fork+0x1f/0x30 [ 4854.596090] Sending NMI from CPU 2 to CPUs 1: [ 4854.758866] NMI backtrace for cpu 1 [ 4854.758874] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4854.758876] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4854.758878] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4854.758879] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4854.758881] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [ 4854.758884] RAX: 0000000000004980 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4854.758886] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4854.758887] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4854.758888] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4854.758889] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4854.758891] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4854.758892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4854.758893] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4854.758894] Call Trace: [ 4854.758895] [ 4854.758896] ? show_regs.cold.9+0x22/0x2f [ 4854.758897] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 4854.758898] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 4854.758899] ? nmi_handle+0xbd/0x210 [ 4854.758900] ? default_do_nmi+0x4b/0x1b0 [ 4854.758900] ? do_nmi+0x280/0x380 [ 4854.758901] ? end_repeat_nmi+0x16/0x1f [ 4854.758902] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4854.758903] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4854.758904] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4854.758905] [ 4854.758906] do_raw_spin_lock+0x177/0x190 [ 4854.758906] _raw_spin_lock+0x21/0x30 [ 4854.758907] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4854.758908] ? do_raw_spin_unlock+0x75/0x190 [ 4854.758909] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4854.758910] ? do_wait_intr+0xf0/0xf0 [ 4854.758911] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4854.758912] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4854.758914] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4854.758915] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4854.758916] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4854.758917] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4854.758918] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4854.758919] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4854.758920] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4854.758921] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4854.758922] lmv_intent_open+0x338/0x1150 [lmv] [ 4854.758923] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4854.758924] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4854.758926] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4854.758926] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4854.758927] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4854.758928] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4854.758929] ? do_raw_spin_unlock+0x75/0x190 [ 4854.758930] ? _raw_spin_unlock+0x12/0x30 [ 4854.758930] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4854.758931] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4854.758932] ll_atomic_open+0x36a/0xd70 [lustre] [ 4854.758933] lookup_open+0x356/0xb40 [ 4854.758934] do_last+0x313/0xfc0 [ 4854.758935] ? nd_jump_root+0xe5/0x160 [ 4854.758935] ? path_init+0x437/0x520 [ 4854.758936] path_openat+0xf7/0x500 [ 4854.758937] ? do_raw_spin_unlock+0x75/0x190 [ 4854.758937] do_filp_open+0x99/0x140 [ 4854.758938] ? getname_flags+0x6e/0x330 [ 4854.758939] ? __check_object_size+0xff/0x256 [ 4854.758940] ? do_raw_spin_unlock+0x75/0x190 [ 4854.758941] ? _raw_spin_unlock+0x12/0x30 [ 4854.758942] do_sys_openat2+0x2b4/0x410 [ 4854.758944] do_sys_open+0x73/0xa0 [ 4854.758945] __x64_sys_openat+0x24/0x30 [ 4854.758946] do_syscall_64+0xc1/0x440 [ 4854.758947] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4854.758948] RIP: 0033:0x7f53425a5332 [ 4854.758949] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4854.758951] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4854.758954] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4854.758955] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4854.758957] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4854.758958] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4854.758959] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4854.759837] NMI backtrace for cpu 2 [ 4855.022116] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4855.033790] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4855.038032] Call Trace: [ 4855.039130] [ 4855.040117] ? dump_stack+0xbb/0x10e [ 4855.046787] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 4855.051896] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 4855.053449] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 4855.058525] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 4855.061263] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 4855.066454] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 4855.068491] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 4855.071113] ? update_process_times+0x6d/0xc0 [ 4855.076801] ? tick_sched_handle.isra.18+0x27/0x90 [ 4855.079971] ? tick_sched_timer+0x53/0xb0 [ 4855.083569] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4855.085224] ? tick_sched_handle.isra.18+0x90/0x90 [ 4855.090991] ? hrtimer_interrupt+0x128/0x2c0 [ 4855.093024] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4855.100841] ? apic_timer_interrupt+0xf/0x20 [ 4855.102346] [ 4855.105667] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4855.109740] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4855.117143] ? do_raw_spin_lock+0x177/0x190 [ 4855.124312] ? _raw_spin_lock+0x21/0x30 [ 4855.125480] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4855.136738] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 4855.139808] ? mdc_commit_open+0x91/0xb0 [mdc] [ 4855.142143] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4855.148863] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4855.153914] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 4855.158301] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4855.161454] ? __next_timer_interrupt+0x160/0x160 [ 4855.163414] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4855.165697] ? do_wait_intr+0xf0/0xf0 [ 4855.167285] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4855.170152] ? kthread+0x1d1/0x200 [ 4855.171793] ? set_kthread_struct+0x70/0x70 [ 4855.173341] ? ret_from_fork+0x1f/0x30 [ 4862.431719] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2225s! [ 4862.440720] Showing busy workqueues and worker pools: [ 4862.445811] workqueue events: flags=0x0 [ 4862.447906] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4862.450604] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4862.456515] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4862.459839] pending: cpu_latency_work [ptlrpc] [ 4862.464272] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4862.468979] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4862.472755] workqueue events_long: flags=0x0 [ 4862.474283] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4862.477364] pending: lu_objects_flush [obdclass] [ 4862.480450] workqueue events_unbound: flags=0x2 [ 4862.482561] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4862.485883] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4862.488732] workqueue events_power_efficient: flags=0x80 [ 4862.492234] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4862.495264] pending: check_lifetime, do_cache_clean [ 4862.497682] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4862.501180] pending: neigh_periodic_work, neigh_periodic_work [ 4862.504403] workqueue mm_percpu_wq: flags=0x8 [ 4862.505757] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4862.508443] pending: vmstat_update, lru_add_drain_per_cpu [ 4862.511205] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4862.514490] pending: vmstat_update, lru_add_drain_per_cpu [ 4862.517587] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4862.520563] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4862.523303] workqueue xprtiod: flags=0xa [ 4862.524884] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4862.527631] pending: xprt_autoclose, xs_error_handle [ 4862.529624] workqueue ipv6_addrconf: flags=0x40008 [ 4862.531615] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4862.534167] pending: addrconf_verify_work [ 4862.536246] workqueue knbd0-recv: flags=0x1a [ 4862.537990] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4862.541182] in-flight: 651:recv_work [ 4862.542790] workqueue ptlrpc_pinger: flags=0x2 [ 4862.544778] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4862.547634] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4862.550779] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2225s workers=4 idle: 15979 31824 8 [ 4862.554589] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4853s workers=2 idle: 654 [ 4880.055025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 4880.057042] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 4880.058031] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4880.058039] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4880.058131] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4880.058136] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4880.058140] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4880.058153] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4880.058158] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4880.058163] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4880.058165] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4880.058168] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4880.058170] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4880.058172] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4880.058176] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4880.058179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4880.058181] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4880.058189] Call Trace: [ 4880.058195] [ 4880.058201] ? show_regs.cold.9+0x22/0x2f [ 4880.058211] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4880.058217] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4880.058221] ? watchdog+0x40/0x40 [ 4880.058225] ? hrtimer_interrupt+0x128/0x2c0 [ 4880.058229] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4880.058236] ? apic_timer_interrupt+0xf/0x20 [ 4880.058240] [ 4880.058243] ? native_safe_halt+0x2e/0x50 [ 4880.058247] kvm_wait+0xcd/0xe0 [ 4880.058254] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4880.058261] do_raw_spin_lock+0x177/0x190 [ 4880.058265] _raw_spin_lock+0x21/0x30 [ 4880.058269] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4880.058530] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4880.058824] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) [ 4880.058731] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4880.058835] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4880.058891] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4880.058895] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4880.058897] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4880.058905] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4880.058908] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4880.058912] RAX: 000000000000610d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4880.058914] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4880.058917] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4880.058919] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4880.058921] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4880.058924] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4880.058927] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4880.058929] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4880.058936] Call Trace: [ 4880.058938] [ 4880.058946] ? show_regs.cold.9+0x22/0x2f [ 4880.058955] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4880.058959] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4880.058963] ? watchdog+0x40/0x40 [ 4880.058968] ? hrtimer_interrupt+0x128/0x2c0 [ 4880.058971] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4880.058976] ? apic_timer_interrupt+0xf/0x20 [ 4880.058980] [ 4880.058983] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4880.058991] do_raw_spin_lock+0x177/0x190 [ 4880.058996] _raw_spin_lock+0x21/0x30 [ 4880.058947] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4880.059321] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4880.059000] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4880.059781] ? do_raw_spin_unlock+0x75/0x190 [ 4880.059787] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4880.059993] ? do_wait_intr+0xf0/0xf0 [ 4880.060001] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4880.060421] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4880.060736] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4880.060941] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4880.061118] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4880.061157] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4880.061185] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4880.061210] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4880.061460] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4880.061676] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4880.061705] lmv_intent_open+0x338/0x1150 [lmv] [ 4880.061736] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4880.061819] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4880.061898] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4880.061920] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4880.062002] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4880.062149] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4880.062233] ? do_raw_spin_unlock+0x75/0x190 [ 4880.062239] ? _raw_spin_unlock+0x12/0x30 [ 4880.062245] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4880.062446] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4880.062468] ll_atomic_open+0x36a/0xd70 [lustre] [ 4880.062551] lookup_open+0x356/0xb40 [ 4880.062561] do_last+0x313/0xfc0 [ 4880.062567] ? nd_jump_root+0xe5/0x160 [ 4880.062571] ? path_init+0x437/0x520 [ 4880.062576] path_openat+0xf7/0x500 [ 4880.062582] ? do_raw_spin_unlock+0x75/0x190 [ 4880.062587] do_filp_open+0x99/0x140 [ 4880.062593] ? getname_flags+0x6e/0x330 [ 4880.062598] ? __check_object_size+0xff/0x256 [ 4880.062605] ? do_raw_spin_unlock+0x75/0x190 [ 4880.062609] ? _raw_spin_unlock+0x12/0x30 [ 4880.062614] do_sys_openat2+0x2b4/0x410 [ 4880.062619] do_sys_open+0x73/0xa0 [ 4880.062624] __x64_sys_openat+0x24/0x30 [ 4880.062628] do_syscall_64+0xc1/0x440 [ 4880.062634] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4880.062640] RIP: 0033:0x7f53425a5332 [ 4880.062647] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4880.062651] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4880.062656] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4880.062659] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4880.062661] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4880.062663] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4880.062666] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4880.063924] Modules linked in: [ 4880.066534] ? woken_wake_function+0x30/0x30 [ 4880.066553] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4880.098744] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4880.098823] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4880.098830] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4880.098833] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4880.098847] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4880.098851] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4880.098856] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4880.098858] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4880.098860] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4880.098862] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4880.098865] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4880.098869] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4880.098872] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4880.098875] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4880.098885] Call Trace: [ 4880.098890] [ 4880.098897] ? show_regs.cold.9+0x22/0x2f [ 4880.098907] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4880.098913] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4880.098918] ? watchdog+0x40/0x40 [ 4880.098923] ? hrtimer_interrupt+0x128/0x2c0 [ 4880.098927] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4880.098934] ? apic_timer_interrupt+0xf/0x20 [ 4880.098939] [ 4880.098942] ? native_safe_halt+0x2e/0x50 [ 4880.098947] kvm_wait+0xcd/0xe0 [ 4880.098954] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4880.098962] do_raw_spin_lock+0x177/0x190 [ 4880.098967] _raw_spin_lock+0x21/0x30 [ 4880.098973] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4880.105629] kthread+0x1d1/0x200 [ 4880.105648] ? set_kthread_struct+0x70/0x70 [ 4880.105653] ret_from_fork+0x1f/0x30 [ 4880.553135] obd_mod_free+0x46/0x140 [ptlrpc] [ 4880.556281] mdc_commit_open+0x91/0xb0 [mdc] [ 4880.558902] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4880.567804] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4880.571676] after_reply+0xe2c/0x1780 [ptlrpc] [ 4880.578338] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4880.584474] ? __next_timer_interrupt+0x160/0x160 [ 4880.588275] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4880.591890] ? do_wait_intr+0xf0/0xf0 [ 4880.595524] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4880.599225] kthread+0x1d1/0x200 [ 4880.600926] ? set_kthread_struct+0x70/0x70 [ 4880.607302] ret_from_fork+0x1f/0x30 [ 4888.052984] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 4888.055917] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4888.088315] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4888.104895] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4888.115280] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4888.123105] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4888.128565] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4888.144640] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4888.152372] RAX: 0000000000007d8a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4888.159744] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4888.167360] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4888.173380] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4888.180461] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4888.188163] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4888.193353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4888.198535] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4888.204501] Call Trace: [ 4888.206403] [ 4888.208420] ? show_regs.cold.9+0x22/0x2f [ 4888.212835] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4888.217733] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4888.222452] ? watchdog+0x40/0x40 [ 4888.224265] ? hrtimer_interrupt+0x128/0x2c0 [ 4888.226302] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4888.228910] ? apic_timer_interrupt+0xf/0x20 [ 4888.231542] [ 4888.234385] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4888.240307] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4888.245508] ? newidle_balance.isra.93+0x207/0x580 [ 4888.250638] do_raw_spin_lock+0x177/0x190 [ 4888.252781] _raw_spin_lock+0x21/0x30 [ 4888.255414] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4888.261264] process_one_work+0x2c8/0x700 [ 4888.267072] worker_thread+0x296/0x6e0 [ 4888.268225] ? rescuer_thread+0x570/0x570 [ 4888.272437] kthread+0x1d1/0x200 [ 4888.275376] ? set_kthread_struct+0x70/0x70 [ 4888.279854] ret_from_fork+0x1f/0x30 [ 4892.642969] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2255s! [ 4892.650584] Showing busy workqueues and worker pools: [ 4892.654192] workqueue events: flags=0x0 [ 4892.659636] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4892.666273] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4892.674713] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4892.684556] pending: cpu_latency_work [ptlrpc] [ 4892.686731] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4892.689563] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4892.697173] workqueue events_long: flags=0x0 [ 4892.699240] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4892.701622] pending: lu_objects_flush [obdclass] [ 4892.704970] workqueue events_unbound: flags=0x2 [ 4892.708779] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4892.714721] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4892.720760] workqueue events_power_efficient: flags=0x80 [ 4892.722800] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4892.728273] pending: check_lifetime, do_cache_clean [ 4892.731615] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4892.734818] pending: neigh_periodic_work, neigh_periodic_work [ 4892.740454] workqueue mm_percpu_wq: flags=0x8 [ 4892.743510] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4892.746583] pending: vmstat_update, lru_add_drain_per_cpu [ 4892.751397] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4892.756176] pending: vmstat_update, lru_add_drain_per_cpu [ 4892.759154] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4892.762715] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4892.770617] workqueue xprtiod: flags=0xa [ 4892.781692] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4892.786647] pending: xprt_autoclose, xs_error_handle [ 4892.793572] workqueue ipv6_addrconf: flags=0x40008 [ 4892.798466] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4892.802249] pending: addrconf_verify_work [ 4892.811527] workqueue knbd0-recv: flags=0x1a [ 4892.815133] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4892.819484] in-flight: 651:recv_work [ 4892.823377] workqueue ptlrpc_pinger: flags=0x2 [ 4892.826656] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4892.832190] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4892.839591] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2255s workers=4 idle: 15979 31824 8 [ 4892.845780] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4883s workers=2 idle: 654 [ 4908.055041] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 4908.057041] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 4908.058024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 4908.058032] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4908.058114] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4908.058120] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4908.058124] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4908.058136] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4908.058142] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 4908.058146] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4908.058149] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4908.058151] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4908.058153] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4908.058156] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4908.058159] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4908.058162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4908.058165] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4908.058174] Call Trace: [ 4908.058179] [ 4908.058186] ? show_regs.cold.9+0x22/0x2f [ 4908.058196] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4908.058202] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4908.058207] ? watchdog+0x40/0x40 [ 4908.058212] ? hrtimer_interrupt+0x128/0x2c0 [ 4908.058216] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4908.058224] ? apic_timer_interrupt+0xf/0x20 [ 4908.058230] [ 4908.058233] ? native_safe_halt+0x2e/0x50 [ 4908.058237] kvm_wait+0xcd/0xe0 [ 4908.058245] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4908.058255] do_raw_spin_lock+0x177/0x190 [ 4908.058260] _raw_spin_lock+0x21/0x30 [ 4908.058264] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4908.058566] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4908.058782] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4908.059187] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4908.059403] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4908.059642] ? woken_wake_function+0x30/0x30 [ 4908.059651] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4908.060094] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4908.060353] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4908.060359] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4908.060361] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4908.060372] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4908.060375] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4908.060385] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4908.060387] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4908.060389] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4908.060391] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4908.060393] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4908.060397] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4908.060400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4908.060405] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4908.060413] Call Trace: [ 4908.060417] [ 4908.060420] ? show_regs.cold.9+0x22/0x2f [ 4908.060442] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4908.060451] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4908.060455] ? watchdog+0x40/0x40 [ 4908.060460] ? hrtimer_interrupt+0x128/0x2c0 [ 4908.060464] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4908.060470] ? apic_timer_interrupt+0xf/0x20 [ 4908.060475] [ 4908.060478] ? native_safe_halt+0x2e/0x50 [ 4908.060482] kvm_wait+0xcd/0xe0 [ 4908.060489] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4908.060495] do_raw_spin_lock+0x177/0x190 [ 4908.060500] _raw_spin_lock+0x21/0x30 [ 4908.060576] kthread+0x1d1/0x200 [ 4908.060583] ? set_kthread_struct+0x70/0x70 [ 4908.060587] ret_from_fork+0x1f/0x30 [ 4908.060504] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4908.061055] ? do_raw_spin_unlock+0x75/0x190 [ 4908.069213] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4908.069289] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4908.073523] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4908.104120] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4908.104131] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4908.104144] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4908.104148] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4908.104154] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4908.104157] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4908.104159] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4908.104161] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4908.104163] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4908.104166] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4908.104169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4908.104171] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4908.104180] Call Trace: [ 4908.104197] [ 4908.104203] ? show_regs.cold.9+0x22/0x2f [ 4908.104213] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4908.109740] ? do_wait_intr+0xf0/0xf0 [ 4908.109760] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4908.110027] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4908.117094] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4908.120047] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4908.141233] ? watchdog+0x40/0x40 [ 4908.141252] ? hrtimer_interrupt+0x128/0x2c0 [ 4908.141259] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4908.141267] ? apic_timer_interrupt+0xf/0x20 [ 4908.141271] [ 4908.141276] ? native_safe_halt+0x2e/0x50 [ 4908.141280] kvm_wait+0xcd/0xe0 [ 4908.141289] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4908.141296] do_raw_spin_lock+0x177/0x190 [ 4908.141301] _raw_spin_lock+0x21/0x30 [ 4908.141307] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4908.141576] obd_mod_free+0x46/0x140 [ptlrpc] [ 4908.141818] mdc_commit_open+0x91/0xb0 [mdc] [ 4908.141858] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4908.142094] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4908.142321] after_reply+0xe2c/0x1780 [ptlrpc] [ 4908.142534] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4908.142756] ? __next_timer_interrupt+0x160/0x160 [ 4908.142765] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4908.142987] ? do_wait_intr+0xf0/0xf0 [ 4908.142996] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4908.143214] kthread+0x1d1/0x200 [ 4908.143220] ? set_kthread_struct+0x70/0x70 [ 4908.143224] ret_from_fork+0x1f/0x30 [ 4908.639934] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4908.645493] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4908.651749] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4908.656905] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4908.661449] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4908.665824] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4908.673471] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4908.678236] lmv_intent_open+0x338/0x1150 [lmv] [ 4908.679753] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4908.687548] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4908.695518] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4908.698410] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4908.705403] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4908.710702] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4908.713795] ? do_raw_spin_unlock+0x75/0x190 [ 4908.716523] ? _raw_spin_unlock+0x12/0x30 [ 4908.718005] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4908.726143] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4908.730522] ll_atomic_open+0x36a/0xd70 [lustre] [ 4908.733189] lookup_open+0x356/0xb40 [ 4908.734630] do_last+0x313/0xfc0 [ 4908.735531] ? nd_jump_root+0xe5/0x160 [ 4908.739446] ? path_init+0x437/0x520 [ 4908.742480] path_openat+0xf7/0x500 [ 4908.747671] ? do_raw_spin_unlock+0x75/0x190 [ 4908.752370] do_filp_open+0x99/0x140 [ 4908.756088] ? getname_flags+0x6e/0x330 [ 4908.757217] ? __check_object_size+0xff/0x256 [ 4908.758443] ? do_raw_spin_unlock+0x75/0x190 [ 4908.760459] ? _raw_spin_unlock+0x12/0x30 [ 4908.762337] do_sys_openat2+0x2b4/0x410 [ 4908.771522] do_sys_open+0x73/0xa0 [ 4908.774247] __x64_sys_openat+0x24/0x30 [ 4908.776412] do_syscall_64+0xc1/0x440 [ 4908.782176] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4908.785673] RIP: 0033:0x7f53425a5332 [ 4908.790169] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4908.806223] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4908.811744] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4908.820596] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4908.834186] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4908.841741] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4908.846840] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4920.053684] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 4920.059364] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4920.089063] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4920.096096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4920.100993] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4920.103692] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4920.107295] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4920.117986] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4920.122486] RAX: 0000000000002812 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4920.127147] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4920.131098] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4920.140584] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4920.144849] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4920.154189] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4920.160747] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4920.163597] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4920.167952] Call Trace: [ 4920.169565] [ 4920.170488] ? show_regs.cold.9+0x22/0x2f [ 4920.172822] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4920.176206] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4920.177875] ? watchdog+0x40/0x40 [ 4920.179082] ? hrtimer_interrupt+0x128/0x2c0 [ 4920.192763] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4920.201212] ? apic_timer_interrupt+0xf/0x20 [ 4920.205284] [ 4920.206059] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4920.215289] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4920.222640] ? newidle_balance.isra.93+0x207/0x580 [ 4920.226582] do_raw_spin_lock+0x177/0x190 [ 4920.230207] _raw_spin_lock+0x21/0x30 [ 4920.232987] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4920.240828] process_one_work+0x2c8/0x700 [ 4920.244848] worker_thread+0x296/0x6e0 [ 4920.253411] ? rescuer_thread+0x570/0x570 [ 4920.256398] kthread+0x1d1/0x200 [ 4920.258198] ? set_kthread_struct+0x70/0x70 [ 4920.259816] ret_from_fork+0x1f/0x30 [ 4922.848042] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2285s! [ 4922.851013] Showing busy workqueues and worker pools: [ 4922.852694] workqueue events: flags=0x0 [ 4922.854187] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4922.857525] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4922.863668] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4922.867422] pending: cpu_latency_work [ptlrpc] [ 4922.870437] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4922.875095] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4922.880423] workqueue events_long: flags=0x0 [ 4922.882484] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4922.887671] pending: lu_objects_flush [obdclass] [ 4922.891902] workqueue events_unbound: flags=0x2 [ 4922.894113] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4922.898405] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4922.902881] workqueue events_power_efficient: flags=0x80 [ 4922.905447] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4922.908791] pending: check_lifetime, do_cache_clean [ 4922.911182] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4922.914397] pending: neigh_periodic_work, neigh_periodic_work [ 4922.917101] workqueue mm_percpu_wq: flags=0x8 [ 4922.918463] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4922.922428] pending: vmstat_update, lru_add_drain_per_cpu [ 4922.930533] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4922.935092] pending: vmstat_update, lru_add_drain_per_cpu [ 4922.938285] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4922.940545] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4922.945271] workqueue xprtiod: flags=0xa [ 4922.946703] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4922.949820] pending: xprt_autoclose, xs_error_handle [ 4922.953884] workqueue ipv6_addrconf: flags=0x40008 [ 4922.955888] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4922.961135] pending: addrconf_verify_work [ 4922.963150] workqueue knbd0-recv: flags=0x1a [ 4922.965466] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4922.969408] in-flight: 651:recv_work [ 4922.972556] workqueue ptlrpc_pinger: flags=0x2 [ 4922.976108] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4922.979579] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4922.983140] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2285s workers=4 idle: 15979 31824 8 [ 4922.986308] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4913s workers=2 idle: 654 [ 4936.057012] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 4936.061178] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4936.061346] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 4936.066116] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 4936.066129] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4936.066200] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4936.066206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4936.066208] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4936.066223] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4936.066228] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4936.066232] RAX: 0000000000000b18 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4936.066235] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4936.066237] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4936.066240] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4936.066242] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4936.066245] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4936.066248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4936.066251] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4936.066260] Call Trace: [ 4936.066266] [ 4936.066272] ? show_regs.cold.9+0x22/0x2f [ 4936.066283] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4936.066289] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4936.066294] ? watchdog+0x40/0x40 [ 4936.066298] ? hrtimer_interrupt+0x128/0x2c0 [ 4936.066301] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4936.066309] ? apic_timer_interrupt+0xf/0x20 [ 4936.066314] [ 4936.066317] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4936.066324] do_raw_spin_lock+0x177/0x190 [ 4936.066329] _raw_spin_lock+0x21/0x30 [ 4936.066334] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4936.066636] ? do_raw_spin_unlock+0x75/0x190 [ 4936.066643] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4936.066843] ? do_wait_intr+0xf0/0xf0 [ 4936.066852] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4936.067137] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4936.067357] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4936.067572] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4936.067683] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4936.067723] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4936.067752] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4936.067779] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4936.067863] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4936.068904] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4936.068939] lmv_intent_open+0x338/0x1150 [lmv] [ 4936.068968] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4936.069156] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4936.069237] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4936.069261] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4936.069806] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4936.069891] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4936.070198] ? do_raw_spin_unlock+0x75/0x190 [ 4936.070208] ? _raw_spin_unlock+0x12/0x30 [ 4936.070215] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4936.070242] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4936.070263] ll_atomic_open+0x36a/0xd70 [lustre] [ 4936.070361] lookup_open+0x356/0xb40 [ 4936.070371] do_last+0x313/0xfc0 [ 4936.070376] ? nd_jump_root+0xe5/0x160 [ 4936.070381] ? path_init+0x437/0x520 [ 4936.070386] path_openat+0xf7/0x500 [ 4936.070391] ? do_raw_spin_unlock+0x75/0x190 [ 4936.070397] do_filp_open+0x99/0x140 [ 4936.070404] ? getname_flags+0x6e/0x330 [ 4936.070409] ? __check_object_size+0xff/0x256 [ 4936.070417] ? do_raw_spin_unlock+0x75/0x190 [ 4936.070422] ? _raw_spin_unlock+0x12/0x30 [ 4936.070427] do_sys_openat2+0x2b4/0x410 [ 4936.070432] do_sys_open+0x73/0xa0 [ 4936.070436] __x64_sys_openat+0x24/0x30 [ 4936.070440] do_syscall_64+0xc1/0x440 [ 4936.070458] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4936.070464] RIP: 0033:0x7f53425a5332 [ 4936.070473] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4936.070477] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4936.070482] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4936.070484] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4936.070486] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4936.070488] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4936.070491] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4936.086067] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4936.086080] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4936.086083] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4936.086098] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4936.086113] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4936.086119] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4936.086121] RDX: 0000000000007871 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4936.086123] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4936.086126] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4936.086128] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4936.086131] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4936.086134] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4936.086136] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4936.086144] Call Trace: [ 4936.086149] [ 4936.086156] ? show_regs.cold.9+0x22/0x2f [ 4936.086164] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4936.086169] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4936.086172] ? watchdog+0x40/0x40 [ 4936.086174] ? hrtimer_interrupt+0x128/0x2c0 [ 4936.086177] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4936.086183] ? apic_timer_interrupt+0xf/0x20 [ 4936.086186] [ 4936.086188] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4936.086191] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4936.086194] do_raw_spin_lock+0x177/0x190 [ 4936.086197] _raw_spin_lock+0x21/0x30 [ 4936.086203] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4936.086601] obd_mod_free+0x46/0x140 [ptlrpc] [ 4936.086729] mdc_commit_open+0x91/0xb0 [mdc] [ 4936.086763] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4936.086943] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4936.087757] after_reply+0xe2c/0x1780 [ptlrpc] [ 4936.087961] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4936.088389] ? __next_timer_interrupt+0x160/0x160 [ 4936.088399] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4936.092079] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror [ 4936.095939] ? do_wait_intr+0xf0/0xf0 [ 4936.095960] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4936.096296] kthread+0x1d1/0x200 [ 4936.096615] ? set_kthread_struct+0x70/0x70 [ 4936.096621] ret_from_fork+0x1f/0x30 [ 4936.567064] dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4936.569812] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4936.576705] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4936.580003] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4936.584863] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4936.595078] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4936.598396] RAX: 0000000000007d85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4936.602009] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4936.611974] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4936.616297] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4936.620696] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4936.623864] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4936.628675] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4936.633079] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4936.637544] Call Trace: [ 4936.641709] [ 4936.642425] ? show_regs.cold.9+0x22/0x2f [ 4936.643760] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4936.648820] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4936.650829] ? watchdog+0x40/0x40 [ 4936.652526] ? hrtimer_interrupt+0x128/0x2c0 [ 4936.654477] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4936.657489] ? apic_timer_interrupt+0xf/0x20 [ 4936.658630] [ 4936.659716] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4936.664018] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4936.666959] do_raw_spin_lock+0x177/0x190 [ 4936.668923] _raw_spin_lock+0x21/0x30 [ 4936.670900] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4936.678424] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4936.683437] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4936.686673] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4936.691722] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4936.696080] ? woken_wake_function+0x30/0x30 [ 4936.697890] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4936.700024] kthread+0x1d1/0x200 [ 4936.701652] ? set_kthread_struct+0x70/0x70 [ 4936.703507] ret_from_fork+0x1f/0x30 [ 4948.052070] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 4948.058774] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4948.124534] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4948.139993] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4948.157151] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4948.165481] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4948.170254] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4948.185079] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4948.190149] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4948.197237] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4948.204420] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4948.211359] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4948.219446] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4948.225681] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4948.244811] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4948.246776] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4948.270201] Call Trace: [ 4948.274672] [ 4948.282002] ? show_regs.cold.9+0x22/0x2f [ 4948.283450] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4948.292942] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4948.294453] ? watchdog+0x40/0x40 [ 4948.295537] ? hrtimer_interrupt+0x128/0x2c0 [ 4948.303638] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4948.308660] ? apic_timer_interrupt+0xf/0x20 [ 4948.312630] [ 4948.315570] ? native_safe_halt+0x2e/0x50 [ 4948.319351] kvm_wait+0xcd/0xe0 [ 4948.320548] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4948.329510] ? newidle_balance.isra.93+0x207/0x580 [ 4948.336476] do_raw_spin_lock+0x177/0x190 [ 4948.338510] _raw_spin_lock+0x21/0x30 [ 4948.339785] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4948.343268] process_one_work+0x2c8/0x700 [ 4948.347261] worker_thread+0x296/0x6e0 [ 4948.351182] ? rescuer_thread+0x570/0x570 [ 4948.359490] kthread+0x1d1/0x200 [ 4948.361577] ? set_kthread_struct+0x70/0x70 [ 4948.364269] ret_from_fork+0x1f/0x30 [ 4953.056331] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2315s! [ 4953.063869] Showing busy workqueues and worker pools: [ 4953.072482] workqueue events: flags=0x0 [ 4953.081310] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4953.087562] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4953.096271] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4953.100951] pending: cpu_latency_work [ptlrpc] [ 4953.104124] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4953.107738] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4953.115436] workqueue events_long: flags=0x0 [ 4953.122142] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4953.129471] pending: lu_objects_flush [obdclass] [ 4953.139539] workqueue events_unbound: flags=0x2 [ 4953.144951] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4953.163200] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4953.171716] workqueue events_power_efficient: flags=0x80 [ 4953.178204] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4953.192539] pending: check_lifetime, do_cache_clean [ 4953.208142] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4953.212235] pending: neigh_periodic_work, neigh_periodic_work [ 4953.221110] workqueue mm_percpu_wq: flags=0x8 [ 4953.229894] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4953.239917] pending: vmstat_update, lru_add_drain_per_cpu [ 4953.241898] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4953.259857] pending: vmstat_update, lru_add_drain_per_cpu [ 4953.267675] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4953.271712] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4953.276395] workqueue xprtiod: flags=0xa [ 4953.278256] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4953.281487] pending: xprt_autoclose, xs_error_handle [ 4953.284669] workqueue ipv6_addrconf: flags=0x40008 [ 4953.288892] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4953.292474] pending: addrconf_verify_work [ 4953.295709] workqueue knbd0-recv: flags=0x1a [ 4953.299040] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4953.301965] in-flight: 651:recv_work [ 4953.303718] workqueue ptlrpc_pinger: flags=0x2 [ 4953.305534] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4953.308121] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4953.310931] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2316s workers=4 idle: 15979 31824 8 [ 4953.316474] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4943s workers=2 idle: 654 [ 4964.055066] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 4964.057626] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4964.057715] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 4964.058030] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 4964.058036] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4964.058124] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4964.058130] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4964.058133] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4964.058148] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4964.058152] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4964.058157] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4964.058160] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4964.058163] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4964.058166] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4964.058169] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4964.058172] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4964.058175] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4964.058178] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4964.058187] Call Trace: [ 4964.058194] [ 4964.058200] ? show_regs.cold.9+0x22/0x2f [ 4964.058211] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4964.058217] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4964.058222] ? watchdog+0x40/0x40 [ 4964.058226] ? hrtimer_interrupt+0x128/0x2c0 [ 4964.058230] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4964.058237] ? apic_timer_interrupt+0xf/0x20 [ 4964.058241] [ 4964.058245] ? native_safe_halt+0x2e/0x50 [ 4964.058249] kvm_wait+0xcd/0xe0 [ 4964.058256] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4964.058265] do_raw_spin_lock+0x177/0x190 [ 4964.058270] _raw_spin_lock+0x21/0x30 [ 4964.058274] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4964.058544] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4964.058869] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4964.059217] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4964.059433] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4964.059784] ? woken_wake_function+0x30/0x30 [ 4964.059792] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4964.060212] kthread+0x1d1/0x200 [ 4964.060217] ? set_kthread_struct+0x70/0x70 [ 4964.060222] ret_from_fork+0x1f/0x30 [ 4964.076240] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4964.076255] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4964.076258] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 4964.076271] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 4964.076275] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4964.076281] RAX: 0000000000005743 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4964.076283] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4964.076286] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4964.076288] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4964.076290] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4964.076294] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4964.076296] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4964.076298] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4964.076307] Call Trace: [ 4964.076313] [ 4964.076318] ? show_regs.cold.9+0x22/0x2f [ 4964.076327] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4964.076333] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4964.076337] ? watchdog+0x40/0x40 [ 4964.076341] ? hrtimer_interrupt+0x128/0x2c0 [ 4964.076346] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4964.076354] ? apic_timer_interrupt+0xf/0x20 [ 4964.076359] [ 4964.076362] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 4964.076370] do_raw_spin_lock+0x177/0x190 [ 4964.076375] _raw_spin_lock+0x21/0x30 [ 4964.076451] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4964.076880] ? do_raw_spin_unlock+0x75/0x190 [ 4964.076886] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4964.079947] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4964.080029] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4964.080037] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4964.080040] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4964.080052] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 4964.080056] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4964.080061] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4964.080064] RDX: 00000000000021a5 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4964.080066] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4964.080068] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4964.080071] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4964.080074] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4964.080077] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4964.080079] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4964.080088] Call Trace: [ 4964.080093] [ 4964.080098] ? show_regs.cold.9+0x22/0x2f [ 4964.080106] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4964.080112] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4964.080116] ? watchdog+0x40/0x40 [ 4964.080120] ? hrtimer_interrupt+0x128/0x2c0 [ 4964.080123] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4964.080131] ? apic_timer_interrupt+0xf/0x20 [ 4964.080136] [ 4964.080139] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 4964.080143] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4964.080148] do_raw_spin_lock+0x177/0x190 [ 4964.080153] _raw_spin_lock+0x21/0x30 [ 4964.080159] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4964.080414] obd_mod_free+0x46/0x140 [ptlrpc] [ 4964.084315] ? do_wait_intr+0xf0/0xf0 [ 4964.084333] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4964.084624] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4964.084857] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4964.085084] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4964.085201] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4964.085244] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4964.085273] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4964.085301] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4964.085390] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4964.085628] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4964.085658] lmv_intent_open+0x338/0x1150 [lmv] [ 4964.085689] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4964.085775] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4964.085861] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4964.085884] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4964.085970] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4964.086054] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4964.086137] ? do_raw_spin_unlock+0x75/0x190 [ 4964.086144] ? _raw_spin_unlock+0x12/0x30 [ 4964.086151] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4964.086179] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4964.086197] ll_atomic_open+0x36a/0xd70 [lustre] [ 4964.086285] lookup_open+0x356/0xb40 [ 4964.086294] do_last+0x313/0xfc0 [ 4964.086299] ? nd_jump_root+0xe5/0x160 [ 4964.086303] ? path_init+0x437/0x520 [ 4964.086308] path_openat+0xf7/0x500 [ 4964.086312] ? do_raw_spin_unlock+0x75/0x190 [ 4964.086316] do_filp_open+0x99/0x140 [ 4964.086323] ? getname_flags+0x6e/0x330 [ 4964.086327] ? __check_object_size+0xff/0x256 [ 4964.086334] ? do_raw_spin_unlock+0x75/0x190 [ 4964.086337] ? _raw_spin_unlock+0x12/0x30 [ 4964.086342] do_sys_openat2+0x2b4/0x410 [ 4964.086347] do_sys_open+0x73/0xa0 [ 4964.086351] __x64_sys_openat+0x24/0x30 [ 4964.086355] do_syscall_64+0xc1/0x440 [ 4964.086363] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4964.086369] RIP: 0033:0x7f53425a5332 [ 4964.086376] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4964.086381] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4964.086386] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4964.086389] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4964.086392] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4964.086394] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4964.086397] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 4964.494855] mdc_commit_open+0x91/0xb0 [mdc] [ 4964.497249] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4964.499872] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4964.502590] after_reply+0xe2c/0x1780 [ptlrpc] [ 4964.504993] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4964.507684] ? __next_timer_interrupt+0x160/0x160 [ 4964.510024] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4964.511958] ? do_wait_intr+0xf0/0xf0 [ 4964.514061] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4964.516367] kthread+0x1d1/0x200 [ 4964.517761] ? set_kthread_struct+0x70/0x70 [ 4964.519378] ret_from_fork+0x1f/0x30 [ 4980.052011] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 4980.055265] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4980.101831] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4980.122616] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4980.136851] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 4980.144085] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4980.153622] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 4980.173545] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4980.183968] RAX: 0000000000003ca7 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4980.189773] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 4980.195464] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 4980.200146] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 4980.209425] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 4980.218054] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 4980.228422] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4980.233294] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 4980.239943] Call Trace: [ 4980.241264] [ 4980.243027] ? show_regs.cold.9+0x22/0x2f [ 4980.245396] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4980.248140] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4980.249280] ? watchdog+0x40/0x40 [ 4980.250803] ? hrtimer_interrupt+0x128/0x2c0 [ 4980.252746] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4980.254682] ? apic_timer_interrupt+0xf/0x20 [ 4980.256046] [ 4980.256758] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 4980.258242] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4980.259681] ? newidle_balance.isra.93+0x207/0x580 [ 4980.261020] do_raw_spin_lock+0x177/0x190 [ 4980.262193] _raw_spin_lock+0x21/0x30 [ 4980.263219] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 4980.265117] process_one_work+0x2c8/0x700 [ 4980.266317] worker_thread+0x296/0x6e0 [ 4980.267345] ? rescuer_thread+0x570/0x570 [ 4980.268657] kthread+0x1d1/0x200 [ 4980.269600] ? set_kthread_struct+0x70/0x70 [ 4980.270938] ret_from_fork+0x1f/0x30 [ 4983.777432] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2346s! [ 4983.787066] Showing busy workqueues and worker pools: [ 4983.788424] workqueue events: flags=0x0 [ 4983.789380] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 4983.797071] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 4983.811241] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4983.818714] pending: cpu_latency_work [ptlrpc] [ 4983.823154] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 4983.830784] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 4983.840013] workqueue events_long: flags=0x0 [ 4983.842775] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 4983.857615] pending: lu_objects_flush [obdclass] [ 4983.863710] workqueue events_unbound: flags=0x2 [ 4983.874331] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 4983.881200] pending: flush_memcg_stats_dwork, idle_cull_fn [ 4983.886335] workqueue events_power_efficient: flags=0x80 [ 4983.889152] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4983.892522] pending: check_lifetime, do_cache_clean [ 4983.894726] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4983.898499] pending: neigh_periodic_work, neigh_periodic_work [ 4983.901805] workqueue mm_percpu_wq: flags=0x8 [ 4983.905733] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4983.911662] pending: vmstat_update, lru_add_drain_per_cpu [ 4983.913121] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 4983.915430] pending: vmstat_update, lru_add_drain_per_cpu [ 4983.919613] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 4983.924558] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 4983.932239] workqueue xprtiod: flags=0xa [ 4983.934859] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 4983.951067] pending: xprt_autoclose, xs_error_handle [ 4983.957580] workqueue ipv6_addrconf: flags=0x40008 [ 4983.958745] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 4983.960678] pending: addrconf_verify_work [ 4983.962255] workqueue knbd0-recv: flags=0x1a [ 4983.963922] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 4983.966236] in-flight: 651:recv_work [ 4983.967585] workqueue ptlrpc_pinger: flags=0x2 [ 4983.969090] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 4983.972605] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 4983.974926] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2346s workers=4 idle: 15979 31824 8 [ 4983.979658] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=4974s workers=2 idle: 654 [ 4992.055375] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 4992.057028] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 4992.058024] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 4992.058031] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4992.058117] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4992.058133] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4992.058136] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4992.058150] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4992.058154] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 4992.058158] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4992.058160] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 4992.058162] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4992.058165] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 4992.058168] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 4992.058170] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 4992.058173] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4992.058176] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 4992.058184] Call Trace: [ 4992.058189] [ 4992.058194] ? show_regs.cold.9+0x22/0x2f [ 4992.058204] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4992.058210] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4992.058214] ? watchdog+0x40/0x40 [ 4992.058219] ? hrtimer_interrupt+0x128/0x2c0 [ 4992.058223] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4992.058230] ? apic_timer_interrupt+0xf/0x20 [ 4992.058235] [ 4992.058238] ? native_safe_halt+0x2e/0x50 [ 4992.058242] kvm_wait+0xcd/0xe0 [ 4992.058249] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4992.058258] do_raw_spin_lock+0x177/0x190 [ 4992.058262] _raw_spin_lock+0x21/0x30 [ 4992.058266] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 4992.058549] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 4992.058941] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 4992.059175] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 4992.059392] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 4992.059606] ? woken_wake_function+0x30/0x30 [ 4992.059614] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 4992.060097] kthread+0x1d1/0x200 [ 4992.060105] ? set_kthread_struct+0x70/0x70 [ 4992.060109] ret_from_fork+0x1f/0x30 [ 4992.063078] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw [ 4992.066002] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 4992.066076] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4992.066083] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4992.066085] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4992.066098] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4992.066102] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 4992.066106] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 4992.066109] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 4992.066111] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 4992.066113] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 4992.066115] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 4992.066118] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 4992.066120] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4992.066131] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 4992.066138] Call Trace: [ 4992.066144] [ 4992.066149] ? show_regs.cold.9+0x22/0x2f [ 4992.066158] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4992.066163] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4992.066167] ? watchdog+0x40/0x40 [ 4992.066172] ? hrtimer_interrupt+0x128/0x2c0 [ 4992.066175] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4992.066182] ? apic_timer_interrupt+0xf/0x20 [ 4992.066186] [ 4992.066189] ? native_safe_halt+0x2e/0x50 [ 4992.066193] kvm_wait+0xcd/0xe0 [ 4992.066200] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 4992.078086] libata [ 4992.107005] do_raw_spin_lock+0x177/0x190 [ 4992.110402] dm_mirror [ 4992.114459] _raw_spin_lock+0x21/0x30 [ 4992.115767] dm_region_hash [ 4992.131065] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 4992.131520] obd_mod_free+0x46/0x140 [ptlrpc] [ 4992.131842] mdc_commit_open+0x91/0xb0 [mdc] [ 4992.131880] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 4992.132075] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 4992.135464] dm_log [ 4992.138910] after_reply+0xe2c/0x1780 [ptlrpc] [ 4992.139197] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 4992.139570] ? __next_timer_interrupt+0x160/0x160 [ 4992.139581] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 4992.149345] dm_mod [ 4992.153262] ? do_wait_intr+0xf0/0xf0 [ 4992.153283] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 4992.153561] kthread+0x1d1/0x200 [ 4992.153567] ? set_kthread_struct+0x70/0x70 [ 4992.153572] ret_from_fork+0x1f/0x30 [ 4992.442828] sha512_ssse3 sha512_generic [ 4992.444644] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 4992.449430] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 4992.453041] RIP: 0010:native_safe_halt+0x2e/0x50 [ 4992.454982] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 4992.462796] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 4992.465942] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 4992.468715] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 4992.471506] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 4992.478366] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 4992.485661] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 4992.495144] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 4992.499654] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4992.504693] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 4992.508811] Call Trace: [ 4992.512162] [ 4992.512965] ? show_regs.cold.9+0x22/0x2f [ 4992.514991] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 4992.521811] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 4992.527986] ? watchdog+0x40/0x40 [ 4992.530850] ? hrtimer_interrupt+0x128/0x2c0 [ 4992.534536] ? smp_apic_timer_interrupt+0xa6/0x270 [ 4992.537104] ? apic_timer_interrupt+0xf/0x20 [ 4992.542858] [ 4992.545577] ? native_safe_halt+0x2e/0x50 [ 4992.549347] kvm_wait+0xcd/0xe0 [ 4992.550760] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 4992.553790] do_raw_spin_lock+0x177/0x190 [ 4992.556599] _raw_spin_lock+0x21/0x30 [ 4992.558543] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 4992.561758] ? do_raw_spin_unlock+0x75/0x190 [ 4992.566088] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 4992.571938] ? do_wait_intr+0xf0/0xf0 [ 4992.573102] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 4992.578764] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 4992.584740] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4992.590880] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4992.593778] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4992.596552] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 4992.599945] mdc_intent_lock+0x353/0x7e0 [mdc] [ 4992.605902] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4992.617355] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 4992.622196] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 4992.624667] lmv_intent_open+0x338/0x1150 [lmv] [ 4992.626011] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4992.633927] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4992.636911] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 4992.638906] ll_intent_lock+0xc5/0xaa0 [lustre] [ 4992.641364] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 4992.647034] ll_lookup_it+0xe0e/0x2270 [lustre] [ 4992.650786] ? do_raw_spin_unlock+0x75/0x190 [ 4992.652911] ? _raw_spin_unlock+0x12/0x30 [ 4992.653990] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 4992.655524] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 4992.659007] ll_atomic_open+0x36a/0xd70 [lustre] [ 4992.661331] lookup_open+0x356/0xb40 [ 4992.662585] do_last+0x313/0xfc0 [ 4992.669650] ? nd_jump_root+0xe5/0x160 [ 4992.671453] ? path_init+0x437/0x520 [ 4992.674445] path_openat+0xf7/0x500 [ 4992.675591] ? do_raw_spin_unlock+0x75/0x190 [ 4992.679102] do_filp_open+0x99/0x140 [ 4992.683840] ? getname_flags+0x6e/0x330 [ 4992.687077] ? __check_object_size+0xff/0x256 [ 4992.688993] ? do_raw_spin_unlock+0x75/0x190 [ 4992.692055] ? _raw_spin_unlock+0x12/0x30 [ 4992.695971] do_sys_openat2+0x2b4/0x410 [ 4992.701929] do_sys_open+0x73/0xa0 [ 4992.703088] __x64_sys_openat+0x24/0x30 [ 4992.709067] do_syscall_64+0xc1/0x440 [ 4992.711526] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 4992.719410] RIP: 0033:0x7f53425a5332 [ 4992.724461] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 4992.740649] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 4992.748440] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 4992.750497] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 4992.758665] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 4992.761537] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 4992.769295] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5008.054530] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 5008.059545] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5008.091372] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5008.098958] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5008.104363] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5008.108249] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5008.110567] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5008.123666] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5008.140585] RAX: 0000000000007f21 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5008.148419] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5008.152721] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5008.159712] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5008.165005] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5008.168309] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5008.171430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5008.174725] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5008.177769] Call Trace: [ 5008.178511] [ 5008.179594] ? show_regs.cold.9+0x22/0x2f [ 5008.181962] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5008.183560] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5008.186822] ? watchdog+0x40/0x40 [ 5008.189730] ? hrtimer_interrupt+0x128/0x2c0 [ 5008.196159] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5008.200829] ? apic_timer_interrupt+0xf/0x20 [ 5008.204725] [ 5008.205447] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5008.207567] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5008.209383] ? newidle_balance.isra.93+0x207/0x580 [ 5008.211274] do_raw_spin_lock+0x177/0x190 [ 5008.213618] _raw_spin_lock+0x21/0x30 [ 5008.218568] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5008.223084] process_one_work+0x2c8/0x700 [ 5008.228445] worker_thread+0x296/0x6e0 [ 5008.231101] ? rescuer_thread+0x570/0x570 [ 5008.235055] kthread+0x1d1/0x200 [ 5008.236484] ? set_kthread_struct+0x70/0x70 [ 5008.238081] ret_from_fork+0x1f/0x30 [ 5014.495266] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2377s! [ 5014.500010] Showing busy workqueues and worker pools: [ 5014.513789] workqueue events: flags=0x0 [ 5014.517061] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5014.524662] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5014.536226] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5014.544088] pending: cpu_latency_work [ptlrpc] [ 5014.551168] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5014.558644] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5014.565153] workqueue events_long: flags=0x0 [ 5014.570116] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5014.578232] pending: lu_objects_flush [obdclass] [ 5014.584159] workqueue events_unbound: flags=0x2 [ 5014.585818] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5014.590858] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5014.596708] workqueue events_power_efficient: flags=0x80 [ 5014.599719] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5014.606149] pending: check_lifetime, do_cache_clean [ 5014.610537] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5014.619908] pending: neigh_periodic_work, neigh_periodic_work [ 5014.626365] workqueue mm_percpu_wq: flags=0x8 [ 5014.627932] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5014.641713] pending: vmstat_update, lru_add_drain_per_cpu [ 5014.643742] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5014.649023] pending: vmstat_update, lru_add_drain_per_cpu [ 5014.651703] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5014.656966] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5014.667131] workqueue xprtiod: flags=0xa [ 5014.669577] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5014.682966] pending: xprt_autoclose, xs_error_handle [ 5014.691498] workqueue ipv6_addrconf: flags=0x40008 [ 5014.701613] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5014.707878] pending: addrconf_verify_work [ 5014.713867] workqueue knbd0-recv: flags=0x1a [ 5014.719154] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5014.729709] in-flight: 651:recv_work [ 5014.733866] workqueue ptlrpc_pinger: flags=0x2 [ 5014.739675] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5014.746026] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5014.749806] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2377s workers=4 idle: 15979 31824 8 [ 5014.754703] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5005s workers=2 idle: 654 [ 5020.056257] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 5020.058714] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 5020.059002] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 5020.059010] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5020.059086] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5020.059148] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5020.059151] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5020.059168] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5020.059172] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5020.059177] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5020.059179] RDX: 0000000000007b1b RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5020.059182] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5020.059184] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5020.059187] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5020.059191] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5020.059193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5020.059196] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5020.059204] Call Trace: [ 5020.059210] [ 5020.059216] ? show_regs.cold.9+0x22/0x2f [ 5020.059230] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5020.059237] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5020.059243] ? watchdog+0x40/0x40 [ 5020.059247] ? hrtimer_interrupt+0x128/0x2c0 [ 5020.059251] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5020.059261] ? apic_timer_interrupt+0xf/0x20 [ 5020.059265] [ 5020.059268] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5020.059274] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5020.059278] do_raw_spin_lock+0x177/0x190 [ 5020.059283] _raw_spin_lock+0x21/0x30 [ 5020.059289] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5020.059669] obd_mod_free+0x46/0x140 [ptlrpc] [ 5020.060158] mdc_commit_open+0x91/0xb0 [mdc] [ 5020.060200] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5020.060407] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5020.063556] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [ 5020.060597] after_reply+0xe2c/0x1780 [ptlrpc] [ 5020.065730] Modules linked in: lustre(O) osp(O) [ 5020.066434] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5020.067391] ? __next_timer_interrupt+0x160/0x160 [ 5020.067402] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5020.067612] ? do_wait_intr+0xf0/0xf0 [ 5020.067622] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5020.068654] kthread+0x1d1/0x200 [ 5020.068661] ? set_kthread_struct+0x70/0x70 [ 5020.068666] ret_from_fork+0x1f/0x30 [ 5020.069140] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5020.069195] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5020.069201] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5020.069203] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5020.069214] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5020.069218] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [ 5020.109736] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5020.109823] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5020.109831] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5020.109833] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5020.109847] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5020.109852] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5020.109856] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5020.109859] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5020.109861] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5020.109863] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5020.109866] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5020.109869] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5020.109872] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5020.109874] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5020.109884] Call Trace: [ 5020.109889] [ 5020.109895] ? show_regs.cold.9+0x22/0x2f [ 5020.109905] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5020.109912] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5020.109916] ? watchdog+0x40/0x40 [ 5020.109920] ? hrtimer_interrupt+0x128/0x2c0 [ 5020.109925] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5020.109932] ? apic_timer_interrupt+0xf/0x20 [ 5020.109937] [ 5020.109940] ? native_safe_halt+0x2e/0x50 [ 5020.109944] kvm_wait+0xcd/0xe0 [ 5020.109955] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5020.115132] ORIG_RAX: ffffffffffffff13 [ 5020.121619] do_raw_spin_lock+0x177/0x190 [ 5020.121704] _raw_spin_lock+0x21/0x30 [ 5020.121711] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5020.121992] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5020.122182] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5020.127379] RAX: 0000000000003c86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5020.127390] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5020.127392] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5020.127394] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5020.127397] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5020.127400] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5020.127403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5020.127405] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5020.127415] Call Trace: [ 5020.127423] [ 5020.127429] ? show_regs.cold.9+0x22/0x2f [ 5020.127443] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5020.127450] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5020.127455] ? watchdog+0x40/0x40 [ 5020.127460] ? hrtimer_interrupt+0x128/0x2c0 [ 5020.127464] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5020.127473] ? apic_timer_interrupt+0xf/0x20 [ 5020.127478] [ 5020.127481] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5020.127491] do_raw_spin_lock+0x177/0x190 [ 5020.127496] _raw_spin_lock+0x21/0x30 [ 5020.127501] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5020.141267] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5020.141714] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5020.141930] ? woken_wake_function+0x30/0x30 [ 5020.141943] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5020.142335] kthread+0x1d1/0x200 [ 5020.142342] ? set_kthread_struct+0x70/0x70 [ 5020.142346] ret_from_fork+0x1f/0x30 [ 5020.857313] ? do_raw_spin_unlock+0x75/0x190 [ 5020.864971] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5020.873685] ? do_wait_intr+0xf0/0xf0 [ 5020.879829] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5020.882914] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5020.892602] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5020.901314] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5020.909208] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5020.915983] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5020.921831] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5020.927468] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5020.937446] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5020.948093] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5020.950507] lmv_intent_open+0x338/0x1150 [lmv] [ 5020.961057] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5020.966159] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5020.973158] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5020.978601] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5020.984497] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5020.988389] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5020.997224] ? do_raw_spin_unlock+0x75/0x190 [ 5021.001522] ? _raw_spin_unlock+0x12/0x30 [ 5021.007579] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5021.012982] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5021.015796] ll_atomic_open+0x36a/0xd70 [lustre] [ 5021.031054] lookup_open+0x356/0xb40 [ 5021.032516] do_last+0x313/0xfc0 [ 5021.036391] ? nd_jump_root+0xe5/0x160 [ 5021.041554] ? path_init+0x437/0x520 [ 5021.046307] path_openat+0xf7/0x500 [ 5021.050382] ? do_raw_spin_unlock+0x75/0x190 [ 5021.055310] do_filp_open+0x99/0x140 [ 5021.059363] ? getname_flags+0x6e/0x330 [ 5021.062523] ? __check_object_size+0xff/0x256 [ 5021.069664] ? do_raw_spin_unlock+0x75/0x190 [ 5021.071394] ? _raw_spin_unlock+0x12/0x30 [ 5021.076577] do_sys_openat2+0x2b4/0x410 [ 5021.080972] do_sys_open+0x73/0xa0 [ 5021.085662] __x64_sys_openat+0x24/0x30 [ 5021.086987] do_syscall_64+0xc1/0x440 [ 5021.088251] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5021.099163] RIP: 0033:0x7f53425a5332 [ 5021.105345] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5021.122209] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5021.129758] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5021.136804] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5021.142607] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5021.150463] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5021.160203] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5034.503751] rcu: INFO: rcu_sched self-detected stall on CPU [ 5034.517751] rcu: 2-....: (1541972 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 5034.524886] (t=2940077 jiffies g=274185 q=64451) [ 5034.533059] rcu: rcu_sched kthread starved for 2398443 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 5034.553928] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5034.564454] rcu: RCU grace-period kthread stack dump: [ 5034.569988] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5034.577900] Call Trace: [ 5034.579293] __schedule+0x351/0xcb0 [ 5034.588366] schedule+0xc0/0x180 [ 5034.598207] schedule_timeout+0xb4/0x190 [ 5034.606613] ? do_raw_spin_unlock+0x75/0x190 [ 5034.614256] ? __next_timer_interrupt+0x160/0x160 [ 5034.621835] ? prepare_to_swait_event+0xf3/0x140 [ 5034.630852] rcu_gp_kthread+0x8dd/0x15b0 [ 5034.638047] ? force_qs_rnp+0x2c0/0x2c0 [ 5034.642092] kthread+0x1d1/0x200 [ 5034.651304] ? set_kthread_struct+0x70/0x70 [ 5034.654063] ret_from_fork+0x1f/0x30 [ 5034.656771] rcu: Stack dump where RCU GP kthread last ran: [ 5034.663302] Sending NMI from CPU 2 to CPUs 3: [ 5034.671509] NMI backtrace for cpu 3 [ 5034.671518] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5034.671521] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5034.671522] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5034.671523] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5034.671525] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 [ 5034.671528] RAX: 0000000000000e8e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5034.671529] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5034.671531] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5034.671532] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5034.671533] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5034.671534] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5034.671536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5034.671537] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5034.671538] Call Trace: [ 5034.671539] [ 5034.671540] ? show_regs.cold.9+0x22/0x2f [ 5034.671540] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 5034.671542] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 5034.671543] ? nmi_handle+0xbd/0x210 [ 5034.671544] ? default_do_nmi+0x4b/0x1b0 [ 5034.671545] ? do_nmi+0x280/0x380 [ 5034.671546] ? end_repeat_nmi+0x16/0x1f [ 5034.671547] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5034.671548] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5034.671549] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5034.671550] [ 5034.671625] do_raw_spin_lock+0x177/0x190 [ 5034.671628] _raw_spin_lock+0x21/0x30 [ 5034.671629] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5034.671630] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5034.671631] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5034.671632] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5034.671633] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5034.671634] ? woken_wake_function+0x30/0x30 [ 5034.671635] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5034.671636] kthread+0x1d1/0x200 [ 5034.671637] ? set_kthread_struct+0x70/0x70 [ 5034.671638] ret_from_fork+0x1f/0x30 [ 5034.673127] Sending NMI from CPU 2 to CPUs 1: [ 5034.790316] NMI backtrace for cpu 1 [ 5034.790325] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5034.790327] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5034.790329] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5034.790330] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5034.790333] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [ 5034.790336] RAX: 0000000000000211 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5034.790337] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5034.790339] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5034.790340] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5034.790342] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5034.790343] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5034.790345] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5034.790346] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5034.790347] Call Trace: [ 5034.790348] [ 5034.790349] ? show_regs.cold.9+0x22/0x2f [ 5034.790350] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 5034.790351] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 5034.790352] ? nmi_handle+0xbd/0x210 [ 5034.790353] ? default_do_nmi+0x4b/0x1b0 [ 5034.790355] ? do_nmi+0x280/0x380 [ 5034.790355] ? end_repeat_nmi+0x16/0x1f [ 5034.790356] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5034.790358] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5034.790359] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5034.790360] [ 5034.790361] do_raw_spin_lock+0x177/0x190 [ 5034.790362] _raw_spin_lock+0x21/0x30 [ 5034.790363] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5034.790364] ? do_raw_spin_unlock+0x75/0x190 [ 5034.790365] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5034.790366] ? do_wait_intr+0xf0/0xf0 [ 5034.790367] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5034.790369] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5034.790370] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5034.790371] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5034.790373] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5034.790374] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5034.790387] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5034.790388] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5034.790390] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5034.790391] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5034.790392] lmv_intent_open+0x338/0x1150 [lmv] [ 5034.790393] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5034.790394] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5034.790396] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5034.790396] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5034.790398] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5034.790399] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5034.790400] ? do_raw_spin_unlock+0x75/0x190 [ 5034.790401] ? _raw_spin_unlock+0x12/0x30 [ 5034.790402] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5034.790403] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5034.790404] ll_atomic_open+0x36a/0xd70 [lustre] [ 5034.790406] lookup_open+0x356/0xb40 [ 5034.790407] do_last+0x313/0xfc0 [ 5034.790408] ? nd_jump_root+0xe5/0x160 [ 5034.790409] ? path_init+0x437/0x520 [ 5034.790410] path_openat+0xf7/0x500 [ 5034.790411] ? do_raw_spin_unlock+0x75/0x190 [ 5034.790412] do_filp_open+0x99/0x140 [ 5034.790414] ? getname_flags+0x6e/0x330 [ 5034.790415] ? __check_object_size+0xff/0x256 [ 5034.790415] ? do_raw_spin_unlock+0x75/0x190 [ 5034.790416] ? _raw_spin_unlock+0x12/0x30 [ 5034.790417] do_sys_openat2+0x2b4/0x410 [ 5034.790418] do_sys_open+0x73/0xa0 [ 5034.790419] __x64_sys_openat+0x24/0x30 [ 5034.790420] do_syscall_64+0xc1/0x440 [ 5034.790421] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5034.790422] RIP: 0033:0x7f53425a5332 [ 5034.790423] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5034.790425] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5034.790428] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5034.790429] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5034.790431] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5034.790433] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5034.790434] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5034.790724] NMI backtrace for cpu 2 [ 5035.072076] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5035.081549] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5035.086096] Call Trace: [ 5035.086926] [ 5035.088642] ? dump_stack+0xbb/0x10e [ 5035.090895] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5035.096259] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5035.103688] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5035.107508] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5035.110149] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5035.113348] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5035.122414] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5035.124271] ? update_process_times+0x6d/0xc0 [ 5035.127973] ? tick_sched_handle.isra.18+0x27/0x90 [ 5035.133039] ? tick_sched_timer+0x53/0xb0 [ 5035.138269] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5035.143016] ? tick_sched_handle.isra.18+0x90/0x90 [ 5035.144476] ? hrtimer_interrupt+0x128/0x2c0 [ 5035.152580] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5035.154341] ? apic_timer_interrupt+0xf/0x20 [ 5035.159776] [ 5035.161980] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5035.167524] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5035.169997] ? do_raw_spin_lock+0x177/0x190 [ 5035.175166] ? _raw_spin_lock+0x21/0x30 [ 5035.178936] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5035.183348] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5035.188533] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5035.191229] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5035.194211] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5035.197531] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5035.200683] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5035.205882] ? __next_timer_interrupt+0x160/0x160 [ 5035.208935] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5035.213948] ? do_wait_intr+0xf0/0xf0 [ 5035.215600] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5035.223250] ? kthread+0x1d1/0x200 [ 5035.224919] ? set_kthread_struct+0x70/0x70 [ 5035.228930] ? ret_from_fork+0x1f/0x30 [ 5040.052083] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 5040.057396] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5040.073620] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5040.081976] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5040.093216] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5040.101511] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5040.103744] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5040.114961] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5040.119466] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5040.124472] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5040.128217] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5040.130895] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5040.133952] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5040.137136] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5040.142373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5040.144764] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5040.150853] Call Trace: [ 5040.153237] [ 5040.154482] ? show_regs.cold.9+0x22/0x2f [ 5040.155714] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5040.157147] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5040.158448] ? watchdog+0x40/0x40 [ 5040.159179] ? hrtimer_interrupt+0x128/0x2c0 [ 5040.160996] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5040.162487] ? apic_timer_interrupt+0xf/0x20 [ 5040.163794] [ 5040.164440] ? native_safe_halt+0x2e/0x50 [ 5040.165398] kvm_wait+0xcd/0xe0 [ 5040.166223] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5040.167570] ? newidle_balance.isra.93+0x207/0x580 [ 5040.168979] do_raw_spin_lock+0x177/0x190 [ 5040.169969] _raw_spin_lock+0x21/0x30 [ 5040.170816] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5040.172443] process_one_work+0x2c8/0x700 [ 5040.173264] worker_thread+0x296/0x6e0 [ 5040.174109] ? rescuer_thread+0x570/0x570 [ 5040.175008] kthread+0x1d1/0x200 [ 5040.175857] ? set_kthread_struct+0x70/0x70 [ 5040.176818] ret_from_fork+0x1f/0x30 [ 5045.217218] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2408s! [ 5045.221017] Showing busy workqueues and worker pools: [ 5045.224765] workqueue events: flags=0x0 [ 5045.230717] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5045.236793] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5045.245011] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5045.248623] pending: cpu_latency_work [ptlrpc] [ 5045.254799] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5045.259835] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5045.268923] workqueue events_long: flags=0x0 [ 5045.273175] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5045.281760] pending: lu_objects_flush [obdclass] [ 5045.294568] workqueue events_unbound: flags=0x2 [ 5045.296633] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5045.301173] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5045.307540] workqueue events_power_efficient: flags=0x80 [ 5045.313795] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5045.318874] pending: check_lifetime, do_cache_clean [ 5045.320597] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5045.326744] pending: neigh_periodic_work, neigh_periodic_work [ 5045.330096] workqueue mm_percpu_wq: flags=0x8 [ 5045.332521] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5045.338369] pending: vmstat_update, lru_add_drain_per_cpu [ 5045.341995] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5045.349967] pending: vmstat_update, lru_add_drain_per_cpu [ 5045.356597] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5045.365016] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5045.369414] workqueue xprtiod: flags=0xa [ 5045.373258] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5045.377148] pending: xprt_autoclose, xs_error_handle [ 5045.380614] workqueue ipv6_addrconf: flags=0x40008 [ 5045.384616] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5045.389501] pending: addrconf_verify_work [ 5045.392026] workqueue knbd0-recv: flags=0x1a [ 5045.393051] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5045.396126] in-flight: 651:recv_work [ 5045.403965] workqueue ptlrpc_pinger: flags=0x2 [ 5045.417067] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5045.424848] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5045.434035] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2408s workers=4 idle: 15979 31824 8 [ 5045.442825] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5035s workers=2 idle: 654 [ 5060.057052] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 5060.057472] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 5060.060166] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5060.060243] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5060.060253] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5060.060256] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5060.060270] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5060.060275] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5060.060279] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5060.060281] RDX: 0000000000003c15 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5060.060283] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5060.060285] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5060.060287] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5060.060290] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5060.060292] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5060.060295] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5060.060303] Call Trace: [ 5060.060308] [ 5060.060313] ? show_regs.cold.9+0x22/0x2f [ 5060.060322] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5060.060328] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5060.060332] ? watchdog+0x40/0x40 [ 5060.060336] ? hrtimer_interrupt+0x128/0x2c0 [ 5060.060340] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5060.060348] ? apic_timer_interrupt+0xf/0x20 [ 5060.060353] [ 5060.060355] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5060.060674] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5060.060679] do_raw_spin_lock+0x177/0x190 [ 5060.060683] _raw_spin_lock+0x21/0x30 [ 5060.060689] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5060.060963] obd_mod_free+0x46/0x140 [ptlrpc] [ 5060.061150] mdc_commit_open+0x91/0xb0 [mdc] [ 5060.061182] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5060.061446] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5060.063084] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5060.063093] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5060.063690] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5060.063697] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5060.063699] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5060.063714] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5060.063718] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5060.063723] RAX: 0000000000007a89 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5060.063726] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5060.063728] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5060.063731] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5060.063733] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5060.063737] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5060.063740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5060.063743] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5060.063752] Call Trace: [ 5060.063757] [ 5060.063763] ? show_regs.cold.9+0x22/0x2f [ 5060.063772] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5060.063779] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5060.063784] ? watchdog+0x40/0x40 [ 5060.063789] ? hrtimer_interrupt+0x128/0x2c0 [ 5060.063794] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5060.063802] ? apic_timer_interrupt+0xf/0x20 [ 5060.063808] [ 5060.063811] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5060.063816] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5060.063823] do_raw_spin_lock+0x177/0x190 [ 5060.063828] _raw_spin_lock+0x21/0x30 [ 5060.063833] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5060.065384] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5060.065792] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5060.067555] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5060.067806] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5060.068047] ? woken_wake_function+0x30/0x30 [ 5060.069240] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5060.069315] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5060.069593] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5060.069599] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5060.069611] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5060.069615] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5060.069619] RAX: 0000000000000c08 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5060.069622] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5060.069624] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5060.069626] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5060.069629] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5060.069633] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5060.069635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5060.069637] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5060.069646] Call Trace: [ 5060.069651] [ 5060.069656] ? show_regs.cold.9+0x22/0x2f [ 5060.069664] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5060.069671] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5060.069676] ? watchdog+0x40/0x40 [ 5060.069679] ? hrtimer_interrupt+0x128/0x2c0 [ 5060.069683] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5060.069692] ? apic_timer_interrupt+0xf/0x20 [ 5060.069697] [ 5060.069699] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5060.069705] do_raw_spin_lock+0x177/0x190 [ 5060.069709] _raw_spin_lock+0x21/0x30 [ 5060.069713] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5060.068057] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5060.070466] ? do_raw_spin_unlock+0x75/0x190 [ 5060.070472] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5060.071123] ? do_wait_intr+0xf0/0xf0 [ 5060.071132] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5060.072439] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5060.072746] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5060.072946] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5060.073056] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5060.073091] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5060.073121] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5060.073146] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5060.073224] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5060.074240] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5060.074276] lmv_intent_open+0x338/0x1150 [lmv] [ 5060.074699] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5060.074796] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5060.074878] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5060.074900] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5060.074994] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5060.075079] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5060.075157] ? do_raw_spin_unlock+0x75/0x190 [ 5060.075164] ? _raw_spin_unlock+0x12/0x30 [ 5060.075171] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5060.075195] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5060.075214] ll_atomic_open+0x36a/0xd70 [lustre] [ 5060.075299] lookup_open+0x356/0xb40 [ 5060.075308] do_last+0x313/0xfc0 [ 5060.075313] ? nd_jump_root+0xe5/0x160 [ 5060.075318] ? path_init+0x437/0x520 [ 5060.075333] path_openat+0xf7/0x500 [ 5060.075340] ? do_raw_spin_unlock+0x75/0x190 [ 5060.075344] do_filp_open+0x99/0x140 [ 5060.075351] ? getname_flags+0x6e/0x330 [ 5060.075356] ? __check_object_size+0xff/0x256 [ 5060.075363] ? do_raw_spin_unlock+0x75/0x190 [ 5060.075367] ? _raw_spin_unlock+0x12/0x30 [ 5060.076451] do_sys_openat2+0x2b4/0x410 [ 5060.076458] do_sys_open+0x73/0xa0 [ 5060.076464] __x64_sys_openat+0x24/0x30 [ 5060.076467] do_syscall_64+0xc1/0x440 [ 5060.076473] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5060.076480] RIP: 0033:0x7f53425a5332 [ 5060.076489] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5060.076492] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5060.076498] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5060.076500] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5060.076502] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5060.076504] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5060.076506] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5060.089544] after_reply+0xe2c/0x1780 [ptlrpc] [ 5060.096305] kthread+0x1d1/0x200 [ 5060.102096] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5060.105204] ? set_kthread_struct+0x70/0x70 [ 5060.105228] ret_from_fork+0x1f/0x30 [ 5060.114383] ? __next_timer_interrupt+0x160/0x160 [ 5060.566542] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5060.568785] ? do_wait_intr+0xf0/0xf0 [ 5060.570067] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5060.572290] kthread+0x1d1/0x200 [ 5060.573536] ? set_kthread_struct+0x70/0x70 [ 5060.575163] ret_from_fork+0x1f/0x30 [ 5072.055708] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 5072.062716] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5072.093854] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5072.101150] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5072.105873] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5072.109596] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5072.113103] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5072.122003] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5072.126127] RAX: 0000000000002d58 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5072.130081] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5072.133180] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5072.137646] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5072.140565] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5072.143985] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5072.147691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5072.150042] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5072.155869] Call Trace: [ 5072.156708] [ 5072.157540] ? show_regs.cold.9+0x22/0x2f [ 5072.162918] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5072.165190] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5072.170990] ? watchdog+0x40/0x40 [ 5072.172976] ? hrtimer_interrupt+0x128/0x2c0 [ 5072.178158] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5072.179964] ? apic_timer_interrupt+0xf/0x20 [ 5072.183114] [ 5072.184966] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5072.187957] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5072.190325] ? newidle_balance.isra.93+0x207/0x580 [ 5072.192448] do_raw_spin_lock+0x177/0x190 [ 5072.195549] _raw_spin_lock+0x21/0x30 [ 5072.196839] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5072.201678] process_one_work+0x2c8/0x700 [ 5072.204177] worker_thread+0x296/0x6e0 [ 5072.206449] ? rescuer_thread+0x570/0x570 [ 5072.208760] kthread+0x1d1/0x200 [ 5072.210108] ? set_kthread_struct+0x70/0x70 [ 5072.212196] ret_from_fork+0x1f/0x30 [ 5075.943010] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2438s! [ 5075.945772] Showing busy workqueues and worker pools: [ 5075.949432] workqueue events: flags=0x0 [ 5075.953519] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5075.959502] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5075.969955] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5075.976111] pending: cpu_latency_work [ptlrpc] [ 5075.982423] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5075.986352] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5075.997614] workqueue events_long: flags=0x0 [ 5076.001989] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5076.009111] pending: lu_objects_flush [obdclass] [ 5076.015503] workqueue events_unbound: flags=0x2 [ 5076.019295] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5076.022841] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5076.028612] workqueue events_power_efficient: flags=0x80 [ 5076.033116] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5076.038809] pending: check_lifetime, do_cache_clean [ 5076.045971] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5076.056176] pending: neigh_periodic_work, neigh_periodic_work [ 5076.065497] workqueue mm_percpu_wq: flags=0x8 [ 5076.070860] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5076.079356] pending: vmstat_update, lru_add_drain_per_cpu [ 5076.082919] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5076.085319] pending: vmstat_update, lru_add_drain_per_cpu [ 5076.092074] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5076.099628] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5076.106527] workqueue xprtiod: flags=0xa [ 5076.107423] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5076.108726] pending: xprt_autoclose, xs_error_handle [ 5076.116951] workqueue ipv6_addrconf: flags=0x40008 [ 5076.122454] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5076.126389] pending: addrconf_verify_work [ 5076.130087] workqueue knbd0-recv: flags=0x1a [ 5076.130952] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5076.136868] in-flight: 651:recv_work [ 5076.138888] workqueue ptlrpc_pinger: flags=0x2 [ 5076.142606] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5076.147553] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5076.153762] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2439s workers=4 idle: 15979 31824 8 [ 5076.161677] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5066s workers=2 idle: 654 [ 5088.055400] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 5088.059280] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5088.059790] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 5088.059797] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5088.059867] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5088.059873] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5088.059876] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5088.060103] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5088.060108] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5088.060113] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5088.060115] RDX: 00000000000050b9 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5088.060117] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5088.060120] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5088.060122] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5088.060124] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5088.060126] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5088.060128] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5088.060137] Call Trace: [ 5088.060142] [ 5088.060148] ? show_regs.cold.9+0x22/0x2f [ 5088.060158] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5088.060164] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5088.060169] ? watchdog+0x40/0x40 [ 5088.060173] ? hrtimer_interrupt+0x128/0x2c0 [ 5088.060177] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5088.060185] ? apic_timer_interrupt+0xf/0x20 [ 5088.060190] [ 5088.060193] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5088.060197] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5088.060201] do_raw_spin_lock+0x177/0x190 [ 5088.060206] _raw_spin_lock+0x21/0x30 [ 5088.060212] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5088.060499] obd_mod_free+0x46/0x140 [ptlrpc] [ 5088.060694] mdc_commit_open+0x91/0xb0 [mdc] [ 5088.060732] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5088.064145] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5088.064384] after_reply+0xe2c/0x1780 [ptlrpc] [ 5088.064591] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5088.064784] ? __next_timer_interrupt+0x160/0x160 [ 5088.064793] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5088.067131] ? do_wait_intr+0xf0/0xf0 [ 5088.067142] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5088.067346] kthread+0x1d1/0x200 [ 5088.067353] ? set_kthread_struct+0x70/0x70 [ 5088.067357] ret_from_fork+0x1f/0x30 [ 5088.069933] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5088.070007] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5088.070016] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5088.070018] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5088.070031] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5088.070035] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5088.070039] RAX: 0000000000005f8e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5088.070041] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5088.070043] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5088.070045] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5088.070047] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5088.070049] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5088.070052] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5088.070054] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5088.070065] Call Trace: [ 5088.070070] [ 5088.070075] ? show_regs.cold.9+0x22/0x2f [ 5088.070084] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5088.070089] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5088.070093] ? watchdog+0x40/0x40 [ 5088.070097] ? hrtimer_interrupt+0x128/0x2c0 [ 5088.070100] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5088.070107] ? apic_timer_interrupt+0xf/0x20 [ 5088.070112] [ 5088.070114] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5088.072814] do_raw_spin_lock+0x177/0x190 [ 5088.072841] _raw_spin_lock+0x21/0x30 [ 5088.072852] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5088.073143] ? do_raw_spin_unlock+0x75/0x190 [ 5088.073149] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5088.088719] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5088.088812] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5088.088819] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5088.088823] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5088.088837] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5088.088841] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5088.088846] RAX: 0000000000001684 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5088.088849] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5088.088851] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5088.088853] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5088.088856] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5088.088860] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5088.088863] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5088.088865] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5088.088874] Call Trace: [ 5088.088879] [ 5088.088885] ? show_regs.cold.9+0x22/0x2f [ 5088.088894] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5088.088901] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5088.088905] ? watchdog+0x40/0x40 [ 5088.088910] ? hrtimer_interrupt+0x128/0x2c0 [ 5088.088915] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5088.088922] ? apic_timer_interrupt+0xf/0x20 [ 5088.088926] [ 5088.088929] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5088.088934] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5088.088940] do_raw_spin_lock+0x177/0x190 [ 5088.088945] _raw_spin_lock+0x21/0x30 [ 5088.088949] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5088.090504] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5088.090727] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5088.091014] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5088.102151] ? do_wait_intr+0xf0/0xf0 [ 5088.161328] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5088.161682] ? woken_wake_function+0x30/0x30 [ 5088.161693] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5088.161956] kthread+0x1d1/0x200 [ 5088.161962] ? set_kthread_struct+0x70/0x70 [ 5088.161966] ret_from_fork+0x1f/0x30 [ 5088.835290] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5088.840805] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5088.846135] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5088.848384] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5088.855414] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5088.859416] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5088.865145] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5088.867251] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5088.872142] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5088.874923] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5088.882018] lmv_intent_open+0x338/0x1150 [lmv] [ 5088.884069] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5088.888856] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5088.890378] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5088.894302] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5088.901058] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5088.903105] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5088.906347] ? do_raw_spin_unlock+0x75/0x190 [ 5088.911077] ? _raw_spin_unlock+0x12/0x30 [ 5088.914421] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5088.919413] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5088.920940] ll_atomic_open+0x36a/0xd70 [lustre] [ 5088.922565] lookup_open+0x356/0xb40 [ 5088.923884] do_last+0x313/0xfc0 [ 5088.926632] ? nd_jump_root+0xe5/0x160 [ 5088.928958] ? path_init+0x437/0x520 [ 5088.938782] path_openat+0xf7/0x500 [ 5088.942864] ? do_raw_spin_unlock+0x75/0x190 [ 5088.946927] do_filp_open+0x99/0x140 [ 5088.956050] ? getname_flags+0x6e/0x330 [ 5088.958135] ? __check_object_size+0xff/0x256 [ 5088.964086] ? do_raw_spin_unlock+0x75/0x190 [ 5088.977618] ? _raw_spin_unlock+0x12/0x30 [ 5088.979086] do_sys_openat2+0x2b4/0x410 [ 5088.980186] do_sys_open+0x73/0xa0 [ 5088.983312] __x64_sys_openat+0x24/0x30 [ 5088.989421] do_syscall_64+0xc1/0x440 [ 5088.996295] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5089.005230] RIP: 0033:0x7f53425a5332 [ 5089.007751] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5089.024860] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5089.029211] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5089.037060] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5089.047829] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5089.052834] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5089.060684] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5104.053529] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 5104.063214] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5104.150123] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5104.179140] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5104.195448] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5104.207208] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5104.219051] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5104.254406] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5104.266728] RAX: 0000000000006e89 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5104.271850] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5104.284363] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5104.293045] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5104.297901] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5104.304005] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5104.311789] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5104.318765] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5104.322462] Call Trace: [ 5104.323589] [ 5104.324597] ? show_regs.cold.9+0x22/0x2f [ 5104.326826] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5104.331559] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5104.334049] ? watchdog+0x40/0x40 [ 5104.335757] ? hrtimer_interrupt+0x128/0x2c0 [ 5104.337602] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5104.339822] ? apic_timer_interrupt+0xf/0x20 [ 5104.341805] [ 5104.342800] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5104.345544] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5104.347856] ? newidle_balance.isra.93+0x207/0x580 [ 5104.350335] do_raw_spin_lock+0x177/0x190 [ 5104.352270] _raw_spin_lock+0x21/0x30 [ 5104.354595] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5104.358781] process_one_work+0x2c8/0x700 [ 5104.361513] worker_thread+0x296/0x6e0 [ 5104.367603] ? rescuer_thread+0x570/0x570 [ 5104.372742] kthread+0x1d1/0x200 [ 5104.374934] ? set_kthread_struct+0x70/0x70 [ 5104.378366] ret_from_fork+0x1f/0x30 [ 5106.656049] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2469s! [ 5106.658485] Showing busy workqueues and worker pools: [ 5106.660107] workqueue events: flags=0x0 [ 5106.661375] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5106.663640] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5106.669708] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5106.674309] pending: cpu_latency_work [ptlrpc] [ 5106.679404] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5106.681602] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5106.686423] workqueue events_long: flags=0x0 [ 5106.688902] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5106.691107] pending: lu_objects_flush [obdclass] [ 5106.692936] workqueue events_unbound: flags=0x2 [ 5106.697905] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5106.700574] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5106.702776] workqueue events_power_efficient: flags=0x80 [ 5106.705695] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5106.709615] pending: check_lifetime, do_cache_clean [ 5106.712501] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5106.715109] pending: neigh_periodic_work, neigh_periodic_work [ 5106.721489] workqueue mm_percpu_wq: flags=0x8 [ 5106.722761] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5106.726546] pending: vmstat_update, lru_add_drain_per_cpu [ 5106.730113] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5106.734847] pending: vmstat_update, lru_add_drain_per_cpu [ 5106.736844] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5106.740493] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5106.742606] workqueue xprtiod: flags=0xa [ 5106.743975] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5106.746673] pending: xprt_autoclose, xs_error_handle [ 5106.747901] workqueue ipv6_addrconf: flags=0x40008 [ 5106.750935] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5106.756558] pending: addrconf_verify_work [ 5106.758063] workqueue knbd0-recv: flags=0x1a [ 5106.759566] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5106.762768] in-flight: 651:recv_work [ 5106.765690] workqueue ptlrpc_pinger: flags=0x2 [ 5106.766919] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5106.771939] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5106.773789] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2469s workers=4 idle: 15979 31824 8 [ 5106.777818] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5097s workers=2 idle: 654 [ 5116.055600] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 5116.058142] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5116.060347] Modules linked in: lustre(O) [ 5116.064682] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 5116.064696] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5116.064767] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5116.064773] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5116.064776] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5116.064789] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5116.064793] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5116.064797] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5116.064800] RDX: 0000000000005ba7 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5116.064802] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5116.064804] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5116.064806] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5116.064810] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5116.064813] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5116.064815] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5116.064823] Call Trace: [ 5116.064830] [ 5116.064836] ? show_regs.cold.9+0x22/0x2f [ 5116.064846] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5116.064853] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5116.064857] ? watchdog+0x40/0x40 [ 5116.064862] ? hrtimer_interrupt+0x128/0x2c0 [ 5116.064866] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5116.064874] ? apic_timer_interrupt+0xf/0x20 [ 5116.064878] [ 5116.064880] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5116.064884] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5116.064888] do_raw_spin_lock+0x177/0x190 [ 5116.064892] _raw_spin_lock+0x21/0x30 [ 5116.064898] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5116.065189] obd_mod_free+0x46/0x140 [ptlrpc] [ 5116.065382] mdc_commit_open+0x91/0xb0 [mdc] [ 5116.065420] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5116.067887] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) [ 5116.067767] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5116.067903] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common [ 5116.068022] after_reply+0xe2c/0x1780 [ptlrpc] [ 5116.068219] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5116.068425] ? __next_timer_interrupt+0x160/0x160 [ 5116.068436] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5116.069968] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5116.070044] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5116.070053] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5116.070055] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5116.070068] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5116.070072] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5116.070077] RAX: 0000000000007e11 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5116.070079] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5116.070081] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5116.070083] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5116.070086] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5116.070089] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5116.070092] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5116.070095] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5116.070104] Call Trace: [ 5116.070111] [ 5116.070116] ? show_regs.cold.9+0x22/0x2f [ 5116.070126] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5116.070132] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5116.070136] ? watchdog+0x40/0x40 [ 5116.070141] ? hrtimer_interrupt+0x128/0x2c0 [ 5116.070145] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5116.070152] ? apic_timer_interrupt+0xf/0x20 [ 5116.070156] [ 5116.070159] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5116.070165] do_raw_spin_lock+0x177/0x190 [ 5116.070169] _raw_spin_lock+0x21/0x30 [ 5116.070173] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5116.070586] ? do_raw_spin_unlock+0x75/0x190 [ 5116.070593] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5116.070878] ? do_wait_intr+0xf0/0xf0 [ 5116.070887] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5116.071095] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5116.071316] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5116.073632] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5116.073768] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5116.073812] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5116.073841] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5116.073867] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5116.073948] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5116.074183] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5116.074213] lmv_intent_open+0x338/0x1150 [lmv] [ 5116.074243] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5116.074333] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5116.074432] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5116.074453] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5116.074537] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5116.074623] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5116.074705] ? do_raw_spin_unlock+0x75/0x190 [ 5116.074713] ? _raw_spin_unlock+0x12/0x30 [ 5116.074720] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5116.074747] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5116.074769] ll_atomic_open+0x36a/0xd70 [lustre] [ 5116.074877] lookup_open+0x356/0xb40 [ 5116.074887] do_last+0x313/0xfc0 [ 5116.074891] ? nd_jump_root+0xe5/0x160 [ 5116.074895] ? path_init+0x437/0x520 [ 5116.074899] path_openat+0xf7/0x500 [ 5116.074904] ? do_raw_spin_unlock+0x75/0x190 [ 5116.074908] do_filp_open+0x99/0x140 [ 5116.074914] ? getname_flags+0x6e/0x330 [ 5116.074918] ? __check_object_size+0xff/0x256 [ 5116.074923] ? do_raw_spin_unlock+0x75/0x190 [ 5116.074927] ? _raw_spin_unlock+0x12/0x30 [ 5116.074931] do_sys_openat2+0x2b4/0x410 [ 5116.074934] do_sys_open+0x73/0xa0 [ 5116.074938] __x64_sys_openat+0x24/0x30 [ 5116.074942] do_syscall_64+0xc1/0x440 [ 5116.074947] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5116.074953] RIP: 0033:0x7f53425a5332 [ 5116.074971] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5116.074974] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5116.074979] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5116.074982] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5116.074984] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5116.074985] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5116.074988] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5116.077151] sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5116.077187] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5116.077194] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5116.077196] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5116.077209] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5116.077214] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5116.077218] RAX: 0000000000001883 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5116.077220] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5116.077223] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5116.077225] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5116.077227] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5116.077231] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5116.077234] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5116.077237] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5116.077246] Call Trace: [ 5116.077252] [ 5116.077258] ? show_regs.cold.9+0x22/0x2f [ 5116.077270] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5116.077277] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5116.077281] ? watchdog+0x40/0x40 [ 5116.077285] ? hrtimer_interrupt+0x128/0x2c0 [ 5116.077289] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5116.077298] ? apic_timer_interrupt+0xf/0x20 [ 5116.077303] [ 5116.077306] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5116.077312] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5116.077319] do_raw_spin_lock+0x177/0x190 [ 5116.077325] _raw_spin_lock+0x21/0x30 [ 5116.077331] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5116.118926] ? do_wait_intr+0xf0/0xf0 [ 5116.118950] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5116.119297] kthread+0x1d1/0x200 [ 5116.119303] ? set_kthread_struct+0x70/0x70 [ 5116.119307] ret_from_fork+0x1f/0x30 [ 5116.963052] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5116.966607] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5116.972424] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5116.978501] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5116.984135] ? woken_wake_function+0x30/0x30 [ 5116.986758] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5116.995803] kthread+0x1d1/0x200 [ 5116.999376] ? set_kthread_struct+0x70/0x70 [ 5117.001136] ret_from_fork+0x1f/0x30 [ 5132.052022] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 5132.056008] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5132.083461] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5132.092569] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5132.104705] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5132.116044] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5132.125530] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5132.155203] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5132.163368] RAX: 0000000000004f7e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5132.178323] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5132.187728] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5132.199044] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5132.207947] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5132.220752] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5132.234443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5132.236549] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5132.249601] Call Trace: [ 5132.252579] [ 5132.254429] ? show_regs.cold.9+0x22/0x2f [ 5132.256560] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5132.265952] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5132.271404] ? watchdog+0x40/0x40 [ 5132.273729] ? hrtimer_interrupt+0x128/0x2c0 [ 5132.279794] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5132.284840] ? apic_timer_interrupt+0xf/0x20 [ 5132.287944] [ 5132.288933] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5132.294446] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5132.301982] ? newidle_balance.isra.93+0x207/0x580 [ 5132.305813] do_raw_spin_lock+0x177/0x190 [ 5132.307314] _raw_spin_lock+0x21/0x30 [ 5132.308690] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5132.311115] process_one_work+0x2c8/0x700 [ 5132.312865] worker_thread+0x296/0x6e0 [ 5132.314009] ? rescuer_thread+0x570/0x570 [ 5132.315199] kthread+0x1d1/0x200 [ 5132.316322] ? set_kthread_struct+0x70/0x70 [ 5132.317946] ret_from_fork+0x1f/0x30 [ 5136.864108] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2499s! [ 5136.867858] Showing busy workqueues and worker pools: [ 5136.871883] workqueue events: flags=0x0 [ 5136.875020] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5136.883145] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5136.896206] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5136.900386] pending: cpu_latency_work [ptlrpc] [ 5136.901886] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5136.915745] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5136.921717] workqueue events_long: flags=0x0 [ 5136.925535] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5136.927692] pending: lu_objects_flush [obdclass] [ 5136.929433] workqueue events_unbound: flags=0x2 [ 5136.931558] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5136.942503] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5136.949009] workqueue events_power_efficient: flags=0x80 [ 5136.952314] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5136.959114] pending: check_lifetime, do_cache_clean [ 5136.961572] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5136.968648] pending: neigh_periodic_work, neigh_periodic_work [ 5136.973012] workqueue mm_percpu_wq: flags=0x8 [ 5136.975900] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5136.982783] pending: vmstat_update, lru_add_drain_per_cpu [ 5136.990525] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5136.993513] pending: vmstat_update, lru_add_drain_per_cpu [ 5137.002462] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5137.009135] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5137.011632] workqueue xprtiod: flags=0xa [ 5137.013653] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5137.017338] pending: xprt_autoclose, xs_error_handle [ 5137.022834] workqueue ipv6_addrconf: flags=0x40008 [ 5137.028018] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5137.032836] pending: addrconf_verify_work [ 5137.035066] workqueue knbd0-recv: flags=0x1a [ 5137.037093] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5137.039739] in-flight: 651:recv_work [ 5137.041831] workqueue ptlrpc_pinger: flags=0x2 [ 5137.043546] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5137.046260] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5137.048884] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2499s workers=4 idle: 15979 31824 8 [ 5137.052216] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5127s workers=2 idle: 654 [ 5144.055059] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 5144.058076] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5144.060859] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5144.062208] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 5144.062217] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5144.062290] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5144.062297] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5144.062299] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5144.062313] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5144.062318] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5144.062322] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5144.062325] RDX: 0000000000007efa RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5144.062326] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5144.062328] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5144.062331] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5144.062334] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5144.062337] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5144.062339] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5144.062348] Call Trace: [ 5144.062354] [ 5144.062359] ? show_regs.cold.9+0x22/0x2f [ 5144.062368] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5144.062375] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5144.062379] ? watchdog+0x40/0x40 [ 5144.062383] ? hrtimer_interrupt+0x128/0x2c0 [ 5144.062387] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5144.062396] ? apic_timer_interrupt+0xf/0x20 [ 5144.062401] [ 5144.062403] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5144.062407] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5144.062411] do_raw_spin_lock+0x177/0x190 [ 5144.062415] _raw_spin_lock+0x21/0x30 [ 5144.062421] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5144.062720] obd_mod_free+0x46/0x140 [ptlrpc] [ 5144.063639] mdc_commit_open+0x91/0xb0 [mdc] [ 5144.063689] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5144.063902] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5144.064131] after_reply+0xe2c/0x1780 [ptlrpc] [ 5144.064332] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5144.064541] ? __next_timer_interrupt+0x160/0x160 [ 5144.064551] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5144.064755] ? do_wait_intr+0xf0/0xf0 [ 5144.064764] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5144.064954] kthread+0x1d1/0x200 [ 5144.064960] ? set_kthread_struct+0x70/0x70 [ 5144.064963] ret_from_fork+0x1f/0x30 [ 5144.074333] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5144.074346] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5144.074350] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5144.074363] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5144.074367] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5144.074372] RAX: 0000000000005a88 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5144.074375] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5144.074377] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5144.074379] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5144.074382] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5144.074385] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5144.074388] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5144.074390] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5144.074398] Call Trace: [ 5144.074403] [ 5144.074408] ? show_regs.cold.9+0x22/0x2f [ 5144.074417] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5144.074422] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5144.074529] ? watchdog+0x40/0x40 [ 5144.074534] ? hrtimer_interrupt+0x128/0x2c0 [ 5144.074538] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5144.074546] ? apic_timer_interrupt+0xf/0x20 [ 5144.074551] [ 5144.074554] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5144.074562] do_raw_spin_lock+0x177/0x190 [ 5144.074567] _raw_spin_lock+0x21/0x30 [ 5144.074571] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5144.074858] ? do_raw_spin_unlock+0x75/0x190 [ 5144.074863] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5144.075120] ? do_wait_intr+0xf0/0xf0 [ 5144.075130] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5144.075350] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5144.075703] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5144.076094] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5144.081010] Modules linked in: lustre(O) [ 5144.083568] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5144.083619] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5144.083651] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5144.083679] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5144.083796] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5144.084302] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5144.084337] lmv_intent_open+0x338/0x1150 [lmv] [ 5144.084368] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5144.084536] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5144.084620] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5144.084642] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5144.084729] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5144.084808] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5144.084891] ? do_raw_spin_unlock+0x75/0x190 [ 5144.084899] ? _raw_spin_unlock+0x12/0x30 [ 5144.084906] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5144.085250] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5144.085272] ll_atomic_open+0x36a/0xd70 [lustre] [ 5144.085367] lookup_open+0x356/0xb40 [ 5144.085376] do_last+0x313/0xfc0 [ 5144.085381] ? nd_jump_root+0xe5/0x160 [ 5144.085386] ? path_init+0x437/0x520 [ 5144.085390] path_openat+0xf7/0x500 [ 5144.085395] ? do_raw_spin_unlock+0x75/0x190 [ 5144.085400] do_filp_open+0x99/0x140 [ 5144.085406] ? getname_flags+0x6e/0x330 [ 5144.085411] ? __check_object_size+0xff/0x256 [ 5144.085417] ? do_raw_spin_unlock+0x75/0x190 [ 5144.085421] ? _raw_spin_unlock+0x12/0x30 [ 5144.085426] do_sys_openat2+0x2b4/0x410 [ 5144.085430] do_sys_open+0x73/0xa0 [ 5144.085434] __x64_sys_openat+0x24/0x30 [ 5144.085437] do_syscall_64+0xc1/0x440 [ 5144.085443] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5144.085448] RIP: 0033:0x7f53425a5332 [ 5144.085455] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5144.085460] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5144.085527] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5144.085530] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5144.085532] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5144.085535] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5144.085537] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5144.349369] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5144.363762] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5144.367231] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5144.369729] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5144.371135] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5144.375713] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5144.377439] RAX: 000000000000780e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5144.379108] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5144.380952] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5144.382626] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5144.384615] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5144.386406] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5144.388341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5144.389898] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5144.391698] Call Trace: [ 5144.392234] [ 5144.392832] ? show_regs.cold.9+0x22/0x2f [ 5144.393897] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5144.395049] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5144.396129] ? watchdog+0x40/0x40 [ 5144.397253] ? hrtimer_interrupt+0x128/0x2c0 [ 5144.398780] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5144.400627] ? apic_timer_interrupt+0xf/0x20 [ 5144.402284] [ 5144.403114] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5144.405351] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5144.407240] do_raw_spin_lock+0x177/0x190 [ 5144.408718] _raw_spin_lock+0x21/0x30 [ 5144.410136] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5144.412712] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5144.414825] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5144.417099] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5144.419344] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5144.421442] ? woken_wake_function+0x30/0x30 [ 5144.422988] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5144.425449] kthread+0x1d1/0x200 [ 5144.426853] ? set_kthread_struct+0x70/0x70 [ 5144.428873] ret_from_fork+0x1f/0x30 [ 5164.055632] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 5164.066715] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5164.125076] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5164.137445] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5164.142485] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5164.145027] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5164.148680] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5164.158453] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5164.161637] RAX: 0000000000007f8a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5164.164421] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5164.167315] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5164.170459] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5164.173313] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5164.176203] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5164.180311] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5164.182795] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5164.185530] Call Trace: [ 5164.186863] [ 5164.188477] ? show_regs.cold.9+0x22/0x2f [ 5164.193040] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5164.194629] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5164.198602] ? watchdog+0x40/0x40 [ 5164.199588] ? hrtimer_interrupt+0x128/0x2c0 [ 5164.200905] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5164.202624] ? apic_timer_interrupt+0xf/0x20 [ 5164.204987] [ 5164.205972] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5164.208582] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5164.210744] ? newidle_balance.isra.93+0x207/0x580 [ 5164.213071] do_raw_spin_lock+0x177/0x190 [ 5164.214445] _raw_spin_lock+0x21/0x30 [ 5164.215585] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5164.217526] process_one_work+0x2c8/0x700 [ 5164.222555] worker_thread+0x296/0x6e0 [ 5164.226267] ? rescuer_thread+0x570/0x570 [ 5164.230925] kthread+0x1d1/0x200 [ 5164.233506] ? set_kthread_struct+0x70/0x70 [ 5164.237499] ret_from_fork+0x1f/0x30 [ 5167.077933] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2529s! [ 5167.086985] Showing busy workqueues and worker pools: [ 5167.094010] workqueue events: flags=0x0 [ 5167.096333] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5167.104544] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5167.135538] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5167.148493] pending: cpu_latency_work [ptlrpc] [ 5167.159684] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5167.166363] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5167.171970] workqueue events_long: flags=0x0 [ 5167.181291] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5167.187396] pending: lu_objects_flush [obdclass] [ 5167.196454] workqueue events_unbound: flags=0x2 [ 5167.202157] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5167.213630] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5167.218106] workqueue events_power_efficient: flags=0x80 [ 5167.224549] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5167.231542] pending: check_lifetime, do_cache_clean [ 5167.241128] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5167.249729] pending: neigh_periodic_work, neigh_periodic_work [ 5167.255218] workqueue mm_percpu_wq: flags=0x8 [ 5167.258659] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5167.263076] pending: vmstat_update, lru_add_drain_per_cpu [ 5167.265906] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5167.269472] pending: vmstat_update, lru_add_drain_per_cpu [ 5167.272611] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5167.276735] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5167.280543] workqueue xprtiod: flags=0xa [ 5167.283531] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5167.295744] pending: xprt_autoclose, xs_error_handle [ 5167.305883] workqueue ipv6_addrconf: flags=0x40008 [ 5167.307593] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5167.315347] pending: addrconf_verify_work [ 5167.320529] workqueue knbd0-recv: flags=0x1a [ 5167.323407] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5167.330397] in-flight: 651:recv_work [ 5167.332597] workqueue ptlrpc_pinger: flags=0x2 [ 5167.337363] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5167.343485] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5167.348071] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2530s workers=4 idle: 15979 31824 8 [ 5167.352651] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5157s workers=2 idle: 654 [ 5172.055013] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 5172.059025] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5172.060039] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 5172.061511] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5172.061524] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5172.061605] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5172.061611] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5172.061614] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5172.061632] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5172.061637] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 5172.061641] RAX: 000000000000553b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5172.061644] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5172.061645] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5172.061647] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5172.061650] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5172.061654] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5172.061656] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5172.061658] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5172.061667] Call Trace: [ 5172.061673] [ 5172.061679] ? show_regs.cold.9+0x22/0x2f [ 5172.061689] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5172.061695] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5172.061699] ? watchdog+0x40/0x40 [ 5172.061704] ? hrtimer_interrupt+0x128/0x2c0 [ 5172.061707] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5172.061715] ? apic_timer_interrupt+0xf/0x20 [ 5172.061719] [ 5172.061722] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5172.061726] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5172.061732] do_raw_spin_lock+0x177/0x190 [ 5172.061736] _raw_spin_lock+0x21/0x30 [ 5172.061741] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5172.062057] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5172.062258] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5172.064747] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5172.065000] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5172.065221] ? woken_wake_function+0x30/0x30 [ 5172.065231] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5172.066933] kthread+0x1d1/0x200 [ 5172.066987] ? set_kthread_struct+0x70/0x70 [ 5172.066992] ret_from_fork+0x1f/0x30 [ 5172.089685] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5172.092819] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5172.092902] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5172.092910] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5172.092914] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5172.092927] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5172.092931] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5172.092935] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5172.092937] RDX: 00000000000055a1 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5172.092940] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5172.092941] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5172.092943] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5172.092946] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5172.092949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5172.092951] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5172.092959] Call Trace: [ 5172.092966] [ 5172.092972] ? show_regs.cold.9+0x22/0x2f [ 5172.092995] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5172.093002] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5172.093007] ? watchdog+0x40/0x40 [ 5172.093011] ? hrtimer_interrupt+0x128/0x2c0 [ 5172.093015] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5172.093024] ? apic_timer_interrupt+0xf/0x20 [ 5172.093029] [ 5172.093313] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5172.093324] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5172.093329] do_raw_spin_lock+0x177/0x190 [ 5172.093335] _raw_spin_lock+0x21/0x30 [ 5172.093345] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5172.093821] obd_mod_free+0x46/0x140 [ptlrpc] [ 5172.098172] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5172.098184] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5172.098202] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5172.098207] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5172.098213] RAX: 0000000000002c81 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5172.098216] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5172.098218] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5172.098220] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5172.098223] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5172.098227] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5172.098230] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5172.098232] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5172.098242] Call Trace: [ 5172.098249] [ 5172.098256] ? show_regs.cold.9+0x22/0x2f [ 5172.098281] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5172.098289] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5172.098294] ? watchdog+0x40/0x40 [ 5172.098298] ? hrtimer_interrupt+0x128/0x2c0 [ 5172.098302] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5172.098311] ? apic_timer_interrupt+0xf/0x20 [ 5172.098316] [ 5172.098319] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5172.098328] do_raw_spin_lock+0x177/0x190 [ 5172.098333] _raw_spin_lock+0x21/0x30 [ 5172.098338] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5172.098632] ? do_raw_spin_unlock+0x75/0x190 [ 5172.098638] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5172.098844] ? do_wait_intr+0xf0/0xf0 [ 5172.098854] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5172.099455] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5172.099671] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5172.099882] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5172.100021] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5172.100366] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5172.100396] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5172.100423] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5172.100511] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5172.100755] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5172.100782] lmv_intent_open+0x338/0x1150 [lmv] [ 5172.100820] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5172.100921] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5172.101237] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5172.101379] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5172.123072] mdc_commit_open+0x91/0xb0 [mdc] [ 5172.123130] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5172.123566] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5172.123786] after_reply+0xe2c/0x1780 [ptlrpc] [ 5172.124005] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5172.124482] ? __next_timer_interrupt+0x160/0x160 [ 5172.124499] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5172.124717] ? do_wait_intr+0xf0/0xf0 [ 5172.124725] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5172.124956] kthread+0x1d1/0x200 [ 5172.124962] ? set_kthread_struct+0x70/0x70 [ 5172.124966] ret_from_fork+0x1f/0x30 [ 5172.574772] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5172.578772] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5172.580519] ? do_raw_spin_unlock+0x75/0x190 [ 5172.582487] ? _raw_spin_unlock+0x12/0x30 [ 5172.584291] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5172.586809] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5172.589128] ll_atomic_open+0x36a/0xd70 [lustre] [ 5172.591221] lookup_open+0x356/0xb40 [ 5172.592789] do_last+0x313/0xfc0 [ 5172.594200] ? nd_jump_root+0xe5/0x160 [ 5172.595897] ? path_init+0x437/0x520 [ 5172.597532] path_openat+0xf7/0x500 [ 5172.599049] ? do_raw_spin_unlock+0x75/0x190 [ 5172.600931] do_filp_open+0x99/0x140 [ 5172.602433] ? getname_flags+0x6e/0x330 [ 5172.604122] ? __check_object_size+0xff/0x256 [ 5172.606094] ? do_raw_spin_unlock+0x75/0x190 [ 5172.609119] ? _raw_spin_unlock+0x12/0x30 [ 5172.611676] do_sys_openat2+0x2b4/0x410 [ 5172.614139] do_sys_open+0x73/0xa0 [ 5172.615785] __x64_sys_openat+0x24/0x30 [ 5172.618254] do_syscall_64+0xc1/0x440 [ 5172.620346] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5172.623097] RIP: 0033:0x7f53425a5332 [ 5172.625551] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5172.635063] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5172.639338] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5172.643283] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5172.648761] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5172.652209] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5172.656581] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5192.052017] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 5192.054654] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5192.097484] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5192.108486] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5192.119366] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5192.124381] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5192.132751] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5192.150151] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5192.163807] RAX: 00000000000034fb RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5192.176649] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5192.182831] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5192.186578] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5192.198378] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5192.204569] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5192.212873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5192.220884] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5192.236459] Call Trace: [ 5192.238454] [ 5192.239403] ? show_regs.cold.9+0x22/0x2f [ 5192.242963] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5192.247457] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5192.250523] ? watchdog+0x40/0x40 [ 5192.252205] ? hrtimer_interrupt+0x128/0x2c0 [ 5192.257391] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5192.263613] ? apic_timer_interrupt+0xf/0x20 [ 5192.268501] [ 5192.271260] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5192.276303] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5192.282827] ? newidle_balance.isra.93+0x207/0x580 [ 5192.286805] do_raw_spin_lock+0x177/0x190 [ 5192.289441] _raw_spin_lock+0x21/0x30 [ 5192.291783] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5192.297767] process_one_work+0x2c8/0x700 [ 5192.304349] worker_thread+0x296/0x6e0 [ 5192.307581] ? rescuer_thread+0x570/0x570 [ 5192.309384] kthread+0x1d1/0x200 [ 5192.311256] ? set_kthread_struct+0x70/0x70 [ 5192.312734] ret_from_fork+0x1f/0x30 [ 5197.791132] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2560s! [ 5197.799688] Showing busy workqueues and worker pools: [ 5197.801417] workqueue events: flags=0x0 [ 5197.803968] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5197.810251] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5197.817701] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5197.823143] pending: cpu_latency_work [ptlrpc] [ 5197.827703] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5197.831084] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5197.835297] workqueue events_long: flags=0x0 [ 5197.837073] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5197.844897] pending: lu_objects_flush [obdclass] [ 5197.848174] workqueue events_unbound: flags=0x2 [ 5197.853883] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5197.863030] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5197.866909] workqueue events_power_efficient: flags=0x80 [ 5197.871234] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5197.874082] pending: check_lifetime, do_cache_clean [ 5197.877505] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5197.879901] pending: neigh_periodic_work, neigh_periodic_work [ 5197.881824] workqueue mm_percpu_wq: flags=0x8 [ 5197.883816] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5197.887071] pending: vmstat_update, lru_add_drain_per_cpu [ 5197.889385] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5197.891853] pending: vmstat_update, lru_add_drain_per_cpu [ 5197.894045] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5197.896292] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5197.898339] workqueue xprtiod: flags=0xa [ 5197.899716] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5197.902152] pending: xprt_autoclose, xs_error_handle [ 5197.904087] workqueue ipv6_addrconf: flags=0x40008 [ 5197.905939] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5197.913947] pending: addrconf_verify_work [ 5197.915571] workqueue knbd0-recv: flags=0x1a [ 5197.917180] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5197.919696] in-flight: 651:recv_work [ 5197.921120] workqueue ptlrpc_pinger: flags=0x2 [ 5197.922878] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5197.925307] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5197.927807] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2560s workers=4 idle: 15979 31824 8 [ 5197.930678] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5188s workers=2 idle: 654 [ 5200.055026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 5200.057060] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 5200.058802] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5200.058813] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5200.058886] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5200.058891] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5200.058894] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5200.058908] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5200.058912] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5200.058916] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5200.058918] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5200.058920] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5200.058922] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5200.058925] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5200.058928] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5200.058931] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5200.058933] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5200.058941] Call Trace: [ 5200.058946] [ 5200.058952] ? show_regs.cold.9+0x22/0x2f [ 5200.058961] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5200.058966] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5200.058970] ? watchdog+0x40/0x40 [ 5200.058974] ? hrtimer_interrupt+0x128/0x2c0 [ 5200.058978] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5200.058984] ? apic_timer_interrupt+0xf/0x20 [ 5200.058989] [ 5200.058992] ? native_safe_halt+0x2e/0x50 [ 5200.058996] kvm_wait+0xcd/0xe0 [ 5200.059002] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5200.059010] do_raw_spin_lock+0x177/0x190 [ 5200.059014] _raw_spin_lock+0x21/0x30 [ 5200.059019] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5200.059398] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5200.059722] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5200.059936] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5200.060144] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5200.060212] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5200.060217] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5200.060219] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5200.060226] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5200.060230] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5200.060234] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5200.060237] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5200.060239] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5200.060241] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5200.060244] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5200.060246] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5200.060249] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5200.060251] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5200.060258] Call Trace: [ 5200.060261] [ 5200.060264] ? show_regs.cold.9+0x22/0x2f [ 5200.060272] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5200.060278] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5200.060282] ? watchdog+0x40/0x40 [ 5200.060285] ? hrtimer_interrupt+0x128/0x2c0 [ 5200.060289] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5200.060295] ? apic_timer_interrupt+0xf/0x20 [ 5200.060299] [ 5200.060302] ? native_safe_halt+0x2e/0x50 [ 5200.060306] kvm_wait+0xcd/0xe0 [ 5200.060312] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5200.060318] do_raw_spin_lock+0x177/0x190 [ 5200.060323] _raw_spin_lock+0x21/0x30 [ 5200.060327] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5200.060535] ? do_raw_spin_unlock+0x75/0x190 [ 5200.060542] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5200.060729] ? do_wait_intr+0xf0/0xf0 [ 5200.060738] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5200.061250] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5200.061553] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5200.061782] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5200.061890] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5200.060157] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5200.062197] ? woken_wake_function+0x30/0x30 [ 5200.063476] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5200.063526] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5200.063553] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5200.063662] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5200.063915] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5200.064194] lmv_intent_open+0x338/0x1150 [lmv] [ 5200.064228] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5200.064322] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5200.064404] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5200.064426] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5200.065039] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5200.065129] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5200.065215] ? do_raw_spin_unlock+0x75/0x190 [ 5200.065222] ? _raw_spin_unlock+0x12/0x30 [ 5200.065228] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5200.065254] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5200.065274] ll_atomic_open+0x36a/0xd70 [lustre] [ 5200.065358] lookup_open+0x356/0xb40 [ 5200.065368] do_last+0x313/0xfc0 [ 5200.065373] ? nd_jump_root+0xe5/0x160 [ 5200.065378] ? path_init+0x437/0x520 [ 5200.065382] path_openat+0xf7/0x500 [ 5200.065388] ? do_raw_spin_unlock+0x75/0x190 [ 5200.065392] do_filp_open+0x99/0x140 [ 5200.065398] ? getname_flags+0x6e/0x330 [ 5200.065402] ? __check_object_size+0xff/0x256 [ 5200.065410] ? do_raw_spin_unlock+0x75/0x190 [ 5200.065414] ? _raw_spin_unlock+0x12/0x30 [ 5200.065418] do_sys_openat2+0x2b4/0x410 [ 5200.065422] do_sys_open+0x73/0xa0 [ 5200.065426] __x64_sys_openat+0x24/0x30 [ 5200.065429] do_syscall_64+0xc1/0x440 [ 5200.065999] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5200.066011] RIP: 0033:0x7f53425a5332 [ 5200.066020] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5200.066025] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5200.066031] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5200.066034] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5200.066036] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5200.066039] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5200.066042] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5200.062205] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5200.066296] kthread+0x1d1/0x200 [ 5200.066303] ? set_kthread_struct+0x70/0x70 [ 5200.066306] ret_from_fork+0x1f/0x30 [ 5200.558255] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5200.580902] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5200.586340] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5200.590205] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5200.593494] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5200.610114] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5200.616285] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5200.625045] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5200.631415] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5200.634027] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5200.642452] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5200.649160] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5200.655893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5200.658799] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5200.663187] Call Trace: [ 5200.665286] [ 5200.667318] ? show_regs.cold.9+0x22/0x2f [ 5200.670915] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5200.672641] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5200.679785] ? watchdog+0x40/0x40 [ 5200.681374] ? hrtimer_interrupt+0x128/0x2c0 [ 5200.685035] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5200.687394] ? apic_timer_interrupt+0xf/0x20 [ 5200.692211] [ 5200.692917] ? native_safe_halt+0x2e/0x50 [ 5200.694200] kvm_wait+0xcd/0xe0 [ 5200.701838] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5200.706134] do_raw_spin_lock+0x177/0x190 [ 5200.708212] _raw_spin_lock+0x21/0x30 [ 5200.709887] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5200.712356] obd_mod_free+0x46/0x140 [ptlrpc] [ 5200.715465] mdc_commit_open+0x91/0xb0 [mdc] [ 5200.717517] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5200.719336] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5200.722444] after_reply+0xe2c/0x1780 [ptlrpc] [ 5200.729045] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5200.731894] ? __next_timer_interrupt+0x160/0x160 [ 5200.738058] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5200.744906] ? do_wait_intr+0xf0/0xf0 [ 5200.746166] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5200.755384] kthread+0x1d1/0x200 [ 5200.762458] ? set_kthread_struct+0x70/0x70 [ 5200.766493] ret_from_fork+0x1f/0x30 [ 5214.506039] rcu: INFO: rcu_sched self-detected stall on CPU [ 5214.509076] rcu: 2-....: (1645767 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 5214.515295] (t=3120080 jiffies g=274185 q=64460) [ 5214.517589] rcu: rcu_sched kthread starved for 2578446 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 5214.523352] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5214.528128] rcu: RCU grace-period kthread stack dump: [ 5214.530868] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5214.535076] Call Trace: [ 5214.536191] __schedule+0x351/0xcb0 [ 5214.539298] schedule+0xc0/0x180 [ 5214.541398] schedule_timeout+0xb4/0x190 [ 5214.543408] ? do_raw_spin_unlock+0x75/0x190 [ 5214.544880] ? __next_timer_interrupt+0x160/0x160 [ 5214.547143] ? prepare_to_swait_event+0xf3/0x140 [ 5214.549616] rcu_gp_kthread+0x8dd/0x15b0 [ 5214.551736] ? force_qs_rnp+0x2c0/0x2c0 [ 5214.557307] kthread+0x1d1/0x200 [ 5214.561745] ? set_kthread_struct+0x70/0x70 [ 5214.564555] ret_from_fork+0x1f/0x30 [ 5214.568315] rcu: Stack dump where RCU GP kthread last ran: [ 5214.573384] Sending NMI from CPU 2 to CPUs 0: [ 5214.578221] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 5214.579687] Sending NMI from CPU 2 to CPUs 1: [ 5214.590280] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 5214.591271] NMI backtrace for cpu 2 [ 5214.596673] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5214.604332] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5214.610098] Call Trace: [ 5214.611549] [ 5214.612994] ? dump_stack+0xbb/0x10e [ 5214.616157] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5214.619344] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5214.621248] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5214.623842] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5214.626346] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5214.629575] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5214.632199] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5214.634558] ? update_process_times+0x6d/0xc0 [ 5214.637328] ? tick_sched_handle.isra.18+0x27/0x90 [ 5214.639495] ? tick_sched_timer+0x53/0xb0 [ 5214.641666] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5214.644208] ? tick_sched_handle.isra.18+0x90/0x90 [ 5214.646495] ? hrtimer_interrupt+0x128/0x2c0 [ 5214.648582] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5214.650878] ? apic_timer_interrupt+0xf/0x20 [ 5214.652896] [ 5214.653913] ? native_safe_halt+0x2e/0x50 [ 5214.656527] ? kvm_wait+0xcd/0xe0 [ 5214.657677] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5214.665515] ? do_raw_spin_lock+0x177/0x190 [ 5214.670608] ? _raw_spin_lock+0x21/0x30 [ 5214.671983] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5214.684727] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5214.692932] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5214.700853] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5214.704118] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5214.712347] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5214.718334] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5214.721649] ? __next_timer_interrupt+0x160/0x160 [ 5214.723827] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5214.726286] ? do_wait_intr+0xf0/0xf0 [ 5214.727854] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5214.731618] ? kthread+0x1d1/0x200 [ 5214.735022] ? set_kthread_struct+0x70/0x70 [ 5214.738180] ? ret_from_fork+0x1f/0x30 [ 5227.999551] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2590s! [ 5228.014855] Showing busy workqueues and worker pools: [ 5228.019720] workqueue events: flags=0x0 [ 5228.023415] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5228.025619] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5228.042967] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5228.045046] pending: cpu_latency_work [ptlrpc] [ 5228.046755] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5228.051915] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5228.056088] workqueue events_long: flags=0x0 [ 5228.059727] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5228.061256] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 5228.066865] pending: lu_objects_flush [obdclass] [ 5228.067073] workqueue events_unbound: flags=0x2 [ 5228.067078] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5228.067093] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5228.067149] workqueue events_power_efficient: flags=0x80 [ 5228.067152] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5228.067160] pending: check_lifetime, do_cache_clean [ 5228.067173] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5228.067181] pending: neigh_periodic_work, neigh_periodic_work [ 5228.067194] workqueue mm_percpu_wq: flags=0x8 [ 5228.067196] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5228.067202] pending: vmstat_update, lru_add_drain_per_cpu [ 5228.067212] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5228.067219] pending: vmstat_update, lru_add_drain_per_cpu [ 5228.067226] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5228.067231] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5228.067257] workqueue xprtiod: flags=0xa [ 5228.067261] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5228.067269] pending: xprt_autoclose, xs_error_handle [ 5228.067283] workqueue ipv6_addrconf: flags=0x40008 [ 5228.067285] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5228.067292] pending: addrconf_verify_work [ 5228.067304] workqueue knbd0-recv: flags=0x1a [ 5228.067306] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5228.067312] in-flight: 651:recv_work [ 5228.067336] workqueue ptlrpc_pinger: flags=0x2 [ 5228.067339] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5228.067345] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5228.075106] Modules linked in: lustre(O) [ 5228.084578] [ 5228.084616] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2590s workers=4 idle: 15979 31824 8 [ 5228.084636] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5218s workers=2 idle: 654 [ 5228.224306] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5228.266042] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5228.275252] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5228.282968] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5228.285534] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5228.302498] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5228.307423] RAX: 000000000000069a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5228.312454] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5228.318529] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5228.325380] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5228.328209] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5228.333638] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5228.344140] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5228.352913] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5228.358660] Call Trace: [ 5228.366738] [ 5228.368651] ? show_regs.cold.9+0x22/0x2f [ 5228.371694] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5228.374492] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5228.375706] ? watchdog+0x40/0x40 [ 5228.378233] ? hrtimer_interrupt+0x128/0x2c0 [ 5228.383527] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5228.385839] ? apic_timer_interrupt+0xf/0x20 [ 5228.386964] [ 5228.387670] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5228.391262] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5228.397274] do_raw_spin_lock+0x177/0x190 [ 5228.401265] _raw_spin_lock+0x21/0x30 [ 5228.403657] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5228.410946] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5228.415595] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5228.422943] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5228.429937] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5228.439549] ? woken_wake_function+0x30/0x30 [ 5228.441113] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5228.447684] kthread+0x1d1/0x200 [ 5228.449332] ? set_kthread_struct+0x70/0x70 [ 5228.451193] ret_from_fork+0x1f/0x30 [ 5240.055637] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 5240.057972] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 5240.058941] Modules linked in: lustre(O) [ 5240.069331] Modules linked in: [ 5240.069334] osp(O) ofd(O) [ 5240.071743] lustre(O) [ 5240.074709] lod(O) [ 5240.083075] osp(O) [ 5240.084122] mdt(O) [ 5240.088015] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) [ 5240.088963] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5240.089076] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5240.089083] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5240.089086] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5240.089098] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5240.089102] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5240.089106] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5240.089108] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5240.089111] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5240.089113] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5240.089115] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5240.089118] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5240.089121] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5240.089123] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5240.089131] Call Trace: [ 5240.089137] [ 5240.089142] ? show_regs.cold.9+0x22/0x2f [ 5240.089151] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5240.089157] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5240.089161] ? watchdog+0x40/0x40 [ 5240.089166] ? hrtimer_interrupt+0x128/0x2c0 [ 5240.089170] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5240.089176] ? apic_timer_interrupt+0xf/0x20 [ 5240.089181] [ 5240.089184] ? native_safe_halt+0x2e/0x50 [ 5240.089188] kvm_wait+0xcd/0xe0 [ 5240.089194] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5240.089203] do_raw_spin_lock+0x177/0x190 [ 5240.089207] _raw_spin_lock+0x21/0x30 [ 5240.089210] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5240.095171] fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5240.095226] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5240.095234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5240.095237] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5240.095250] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5240.095254] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5240.095258] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5240.095261] RDX: 00000000000006de RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5240.095263] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5240.095266] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5240.095268] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5240.095271] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5240.095274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5240.095276] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5240.095285] Call Trace: [ 5240.095290] [ 5240.095307] ? show_regs.cold.9+0x22/0x2f [ 5240.095318] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5240.095325] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5240.095329] ? watchdog+0x40/0x40 [ 5240.095333] ? hrtimer_interrupt+0x128/0x2c0 [ 5240.095338] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5240.095346] ? apic_timer_interrupt+0xf/0x20 [ 5240.095351] [ 5240.095355] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5240.095360] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5240.095364] do_raw_spin_lock+0x177/0x190 [ 5240.095369] _raw_spin_lock+0x21/0x30 [ 5240.095374] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5240.095641] obd_mod_free+0x46/0x140 [ptlrpc] [ 5240.095861] mdc_commit_open+0x91/0xb0 [mdc] [ 5240.095903] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5240.096107] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5240.096314] after_reply+0xe2c/0x1780 [ptlrpc] [ 5240.096517] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5240.096710] ? __next_timer_interrupt+0x160/0x160 [ 5240.096720] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5240.096935] ? do_wait_intr+0xf0/0xf0 [ 5240.096944] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5240.097132] kthread+0x1d1/0x200 [ 5240.097138] ? set_kthread_struct+0x70/0x70 [ 5240.097142] ret_from_fork+0x1f/0x30 [ 5240.382750] ? do_raw_spin_unlock+0x75/0x190 [ 5240.385848] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5240.389708] ? do_wait_intr+0xf0/0xf0 [ 5240.391426] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5240.393083] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5240.395738] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5240.398643] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5240.403721] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5240.409699] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5240.413906] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5240.416112] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5240.418576] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5240.421727] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5240.425905] lmv_intent_open+0x338/0x1150 [lmv] [ 5240.432293] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5240.435518] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5240.438168] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5240.439561] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5240.441514] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5240.444195] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5240.448383] ? do_raw_spin_unlock+0x75/0x190 [ 5240.452177] ? _raw_spin_unlock+0x12/0x30 [ 5240.456966] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5240.459805] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5240.461124] ll_atomic_open+0x36a/0xd70 [lustre] [ 5240.467649] lookup_open+0x356/0xb40 [ 5240.470301] do_last+0x313/0xfc0 [ 5240.472489] ? nd_jump_root+0xe5/0x160 [ 5240.475477] ? path_init+0x437/0x520 [ 5240.476946] path_openat+0xf7/0x500 [ 5240.478074] ? do_raw_spin_unlock+0x75/0x190 [ 5240.479961] do_filp_open+0x99/0x140 [ 5240.481074] ? getname_flags+0x6e/0x330 [ 5240.482180] ? __check_object_size+0xff/0x256 [ 5240.483501] ? do_raw_spin_unlock+0x75/0x190 [ 5240.484706] ? _raw_spin_unlock+0x12/0x30 [ 5240.485846] do_sys_openat2+0x2b4/0x410 [ 5240.486949] do_sys_open+0x73/0xa0 [ 5240.488087] __x64_sys_openat+0x24/0x30 [ 5240.489106] do_syscall_64+0xc1/0x440 [ 5240.490157] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5240.491622] RIP: 0033:0x7f53425a5332 [ 5240.492732] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5240.501605] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5240.503679] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5240.506137] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5240.508549] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5240.510600] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5240.512452] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5256.053079] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 5256.058022] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5256.062586] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 5256.086427] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5256.086439] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5256.086445] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5256.086766] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5256.086774] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5256.086777] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5256.086780] RAX: 0000000000007009 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5256.086782] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5256.086783] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5256.086785] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5256.086786] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5256.086789] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5256.086790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5256.086792] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5256.086798] Call Trace: [ 5256.086802] [ 5256.086806] ? show_regs.cold.9+0x22/0x2f [ 5256.086813] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5256.086817] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5256.086820] ? watchdog+0x40/0x40 [ 5256.086822] ? hrtimer_interrupt+0x128/0x2c0 [ 5256.086825] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5256.086831] ? apic_timer_interrupt+0xf/0x20 [ 5256.086834] [ 5256.086836] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5256.086839] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5256.086841] ? newidle_balance.isra.93+0x207/0x580 [ 5256.086845] do_raw_spin_lock+0x177/0x190 [ 5256.086848] _raw_spin_lock+0x21/0x30 [ 5256.086851] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5256.086983] process_one_work+0x2c8/0x700 [ 5256.086988] worker_thread+0x296/0x6e0 [ 5256.086991] ? rescuer_thread+0x570/0x570 [ 5256.086993] kthread+0x1d1/0x200 [ 5256.086996] ? set_kthread_struct+0x70/0x70 [ 5256.086999] ret_from_fork+0x1f/0x30 [ 5256.199846] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5256.223808] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5256.229490] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5256.235053] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5256.236743] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5256.247333] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5256.250360] RAX: 0000000000007191 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5256.255292] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5256.259119] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5256.267254] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5256.269663] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5256.279453] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5256.282193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5256.290455] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5256.296293] Call Trace: [ 5256.297402] [ 5256.300639] ? show_regs.cold.9+0x22/0x2f [ 5256.301862] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5256.304488] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5256.307605] ? watchdog+0x40/0x40 [ 5256.309019] ? hrtimer_interrupt+0x128/0x2c0 [ 5256.312141] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5256.315312] ? apic_timer_interrupt+0xf/0x20 [ 5256.316428] [ 5256.318069] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5256.321417] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5256.324476] do_raw_spin_lock+0x177/0x190 [ 5256.327810] _raw_spin_lock+0x21/0x30 [ 5256.330769] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5256.332410] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5256.337253] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5256.338927] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5256.343348] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5256.346777] ? woken_wake_function+0x30/0x30 [ 5256.349865] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5256.353201] kthread+0x1d1/0x200 [ 5256.355954] ? set_kthread_struct+0x70/0x70 [ 5256.358108] ret_from_fork+0x1f/0x30 [ 5258.208026] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2621s! [ 5258.213865] Showing busy workqueues and worker pools: [ 5258.215934] workqueue events: flags=0x0 [ 5258.218017] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5258.222079] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5258.228364] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5258.232683] pending: cpu_latency_work [ptlrpc] [ 5258.235361] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5258.239861] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5258.246788] workqueue events_long: flags=0x0 [ 5258.250127] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5258.253202] pending: lu_objects_flush [obdclass] [ 5258.259031] workqueue events_unbound: flags=0x2 [ 5258.264738] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5258.270706] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5258.274354] workqueue events_power_efficient: flags=0x80 [ 5258.276683] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5258.279411] pending: check_lifetime, do_cache_clean [ 5258.281765] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5258.284418] pending: neigh_periodic_work, neigh_periodic_work [ 5258.286753] workqueue mm_percpu_wq: flags=0x8 [ 5258.288384] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5258.290966] pending: vmstat_update, lru_add_drain_per_cpu [ 5258.293685] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5258.296182] pending: vmstat_update, lru_add_drain_per_cpu [ 5258.297709] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5258.299660] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5258.302303] workqueue xprtiod: flags=0xa [ 5258.303676] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5258.305464] pending: xprt_autoclose, xs_error_handle [ 5258.307190] workqueue ipv6_addrconf: flags=0x40008 [ 5258.308842] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5258.310932] pending: addrconf_verify_work [ 5258.312922] workqueue knbd0-recv: flags=0x1a [ 5258.314705] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5258.317552] in-flight: 651:recv_work [ 5258.319407] workqueue ptlrpc_pinger: flags=0x2 [ 5258.321323] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5258.323677] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5258.326527] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2621s workers=4 idle: 15979 31824 8 [ 5258.329566] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5248s workers=2 idle: 654 [ 5268.057878] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 5268.058341] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 5268.065235] Modules linked in: [ 5268.072348] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5268.072422] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5268.072430] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5268.072551] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5268.072565] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5268.072570] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5268.072574] RAX: 0000000000005555 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5268.072577] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5268.072579] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5268.072581] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5268.072584] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5268.072587] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5268.072590] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5268.072592] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5268.072600] Call Trace: [ 5268.072606] [ 5268.072612] ? show_regs.cold.9+0x22/0x2f [ 5268.072690] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5268.072697] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5268.072701] ? watchdog+0x40/0x40 [ 5268.072706] ? hrtimer_interrupt+0x128/0x2c0 [ 5268.072710] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5268.072718] ? apic_timer_interrupt+0xf/0x20 [ 5268.072724] [ 5268.072726] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5268.072734] do_raw_spin_lock+0x177/0x190 [ 5268.072739] _raw_spin_lock+0x21/0x30 [ 5268.072745] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5268.073916] ? do_raw_spin_unlock+0x75/0x190 [ 5268.073923] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5268.074130] ? do_wait_intr+0xf0/0xf0 [ 5268.074139] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5268.075550] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5268.079456] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5268.079529] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5268.079537] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5268.079540] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5268.126041] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5268.130678] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5268.137272] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5268.138784] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5268.138794] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5268.138796] RDX: 0000000000007201 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5268.138799] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5268.138801] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5268.138804] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5268.138807] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5268.138810] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5268.138812] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5268.148335] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5268.149586] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5268.149630] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5268.149661] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5268.149775] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5268.155134] Call Trace: [ 5268.155145] [ 5268.155152] ? show_regs.cold.9+0x22/0x2f [ 5268.155168] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5268.155176] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5268.155180] ? watchdog+0x40/0x40 [ 5268.155185] ? hrtimer_interrupt+0x128/0x2c0 [ 5268.155189] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5268.155197] ? apic_timer_interrupt+0xf/0x20 [ 5268.155202] [ 5268.155205] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5268.155212] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5268.155218] do_raw_spin_lock+0x177/0x190 [ 5268.155223] _raw_spin_lock+0x21/0x30 [ 5268.155231] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5268.160947] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5268.161003] lmv_intent_open+0x338/0x1150 [lmv] [ 5268.167134] obd_mod_free+0x46/0x140 [ptlrpc] [ 5268.179017] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5268.179143] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5268.179911] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5268.180093] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5268.180841] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5268.183111] mdc_commit_open+0x91/0xb0 [mdc] [ 5268.183161] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5268.183444] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5268.183648] after_reply+0xe2c/0x1780 [ptlrpc] [ 5268.189965] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5268.190087] ? do_raw_spin_unlock+0x75/0x190 [ 5268.190096] ? _raw_spin_unlock+0x12/0x30 [ 5268.190104] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5268.190127] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5268.190148] ll_atomic_open+0x36a/0xd70 [lustre] [ 5268.190235] lookup_open+0x356/0xb40 [ 5268.190245] do_last+0x313/0xfc0 [ 5268.190250] ? nd_jump_root+0xe5/0x160 [ 5268.190255] ? path_init+0x437/0x520 [ 5268.190260] path_openat+0xf7/0x500 [ 5268.190266] ? do_raw_spin_unlock+0x75/0x190 [ 5268.190271] do_filp_open+0x99/0x140 [ 5268.190276] ? getname_flags+0x6e/0x330 [ 5268.190280] ? __check_object_size+0xff/0x256 [ 5268.190287] ? do_raw_spin_unlock+0x75/0x190 [ 5268.190291] ? _raw_spin_unlock+0x12/0x30 [ 5268.190295] do_sys_openat2+0x2b4/0x410 [ 5268.190312] do_sys_open+0x73/0xa0 [ 5268.190316] __x64_sys_openat+0x24/0x30 [ 5268.190319] do_syscall_64+0xc1/0x440 [ 5268.190326] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5268.190332] RIP: 0033:0x7f53425a5332 [ 5268.190339] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5268.190343] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5268.190350] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5268.190352] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5268.190355] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5268.190357] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5268.190359] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5268.645687] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5268.647812] ? __next_timer_interrupt+0x160/0x160 [ 5268.649687] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5268.652136] ? do_wait_intr+0xf0/0xf0 [ 5268.653698] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5268.655947] kthread+0x1d1/0x200 [ 5268.657046] ? set_kthread_struct+0x70/0x70 [ 5268.659727] ret_from_fork+0x1f/0x30 [ 5284.052566] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 5284.057315] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5284.058598] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 5284.091457] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5284.091470] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5284.091476] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5284.096484] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5284.096563] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5284.096571] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5284.096574] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5284.096620] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5284.096624] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5284.096628] RAX: 000000000000548d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5284.097089] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5284.097093] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5284.097095] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5284.097098] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5284.097101] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5284.097104] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5284.097106] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5284.097416] Call Trace: [ 5284.097439] [ 5284.097444] ? show_regs.cold.9+0x22/0x2f [ 5284.097455] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5284.097461] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5284.097465] ? watchdog+0x40/0x40 [ 5284.097470] ? hrtimer_interrupt+0x128/0x2c0 [ 5284.097474] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5284.097483] ? apic_timer_interrupt+0xf/0x20 [ 5284.097488] [ 5284.097491] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5284.097496] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5284.097503] do_raw_spin_lock+0x177/0x190 [ 5284.097508] _raw_spin_lock+0x21/0x30 [ 5284.097512] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5284.107860] [ 5284.107869] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5284.107887] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5284.107892] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5284.107897] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5284.107900] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5284.107902] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5284.107904] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5284.107906] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5284.107910] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5284.107913] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5284.107915] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5284.107924] Call Trace: [ 5284.107929] [ 5284.107934] ? show_regs.cold.9+0x22/0x2f [ 5284.107943] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5284.107950] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5284.107954] ? watchdog+0x40/0x40 [ 5284.107958] ? hrtimer_interrupt+0x128/0x2c0 [ 5284.107962] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5284.107969] ? apic_timer_interrupt+0xf/0x20 [ 5284.107973] [ 5284.107976] ? native_safe_halt+0x2e/0x50 [ 5284.107981] kvm_wait+0xcd/0xe0 [ 5284.107988] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5284.107994] ? newidle_balance.isra.93+0x207/0x580 [ 5284.108003] do_raw_spin_lock+0x177/0x190 [ 5284.108007] _raw_spin_lock+0x21/0x30 [ 5284.108012] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5284.108962] process_one_work+0x2c8/0x700 [ 5284.108971] worker_thread+0x296/0x6e0 [ 5284.108976] ? rescuer_thread+0x570/0x570 [ 5284.108980] kthread+0x1d1/0x200 [ 5284.108985] ? set_kthread_struct+0x70/0x70 [ 5284.108989] ret_from_fork+0x1f/0x30 [ 5284.531421] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5284.539898] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5284.547981] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5284.557943] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5284.564901] ? woken_wake_function+0x30/0x30 [ 5284.572101] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5284.578660] kthread+0x1d1/0x200 [ 5284.582962] ? set_kthread_struct+0x70/0x70 [ 5284.590377] ret_from_fork+0x1f/0x30 [ 5288.419875] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2651s! [ 5288.444216] Showing busy workqueues and worker pools: [ 5288.445828] workqueue events: flags=0x0 [ 5288.456634] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5288.459446] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5288.482308] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5288.491483] pending: cpu_latency_work [ptlrpc] [ 5288.495629] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5288.503852] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5288.510828] workqueue events_long: flags=0x0 [ 5288.526918] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5288.542184] pending: lu_objects_flush [obdclass] [ 5288.549567] workqueue events_unbound: flags=0x2 [ 5288.557895] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5288.569749] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5288.577676] workqueue events_power_efficient: flags=0x80 [ 5288.585701] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5288.592227] pending: check_lifetime, do_cache_clean [ 5288.607829] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5288.618456] pending: neigh_periodic_work, neigh_periodic_work [ 5288.627457] workqueue mm_percpu_wq: flags=0x8 [ 5288.631758] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5288.639636] pending: vmstat_update, lru_add_drain_per_cpu [ 5288.642378] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5288.653800] pending: vmstat_update, lru_add_drain_per_cpu [ 5288.662521] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5288.667877] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5288.673949] workqueue xprtiod: flags=0xa [ 5288.675846] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5288.680850] pending: xprt_autoclose, xs_error_handle [ 5288.688869] workqueue ipv6_addrconf: flags=0x40008 [ 5288.695699] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5288.703356] pending: addrconf_verify_work [ 5288.704858] workqueue knbd0-recv: flags=0x1a [ 5288.717462] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5288.722621] in-flight: 651:recv_work [ 5288.725117] workqueue ptlrpc_pinger: flags=0x2 [ 5288.728809] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5288.735682] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5288.742054] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2651s workers=4 idle: 15979 31824 8 [ 5288.750776] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5279s workers=2 idle: 654 [ 5296.057313] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 5296.060706] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 5296.068402] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5296.068581] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5296.068588] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5296.068592] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5296.068605] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5296.068609] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5296.068613] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5296.068615] RDX: 000000000000129f RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5296.068618] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5296.068620] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5296.068622] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5296.068625] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5296.068628] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5296.068631] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5296.068640] Call Trace: [ 5296.068646] [ 5296.068652] ? show_regs.cold.9+0x22/0x2f [ 5296.068662] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5296.068668] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5296.068672] ? watchdog+0x40/0x40 [ 5296.068676] ? hrtimer_interrupt+0x128/0x2c0 [ 5296.068680] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5296.068687] ? apic_timer_interrupt+0xf/0x20 [ 5296.068692] [ 5296.068696] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5296.068700] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5296.068705] do_raw_spin_lock+0x177/0x190 [ 5296.068710] _raw_spin_lock+0x21/0x30 [ 5296.068716] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5296.069003] obd_mod_free+0x46/0x140 [ptlrpc] [ 5296.069311] mdc_commit_open+0x91/0xb0 [mdc] [ 5296.069350] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5296.070676] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5296.072991] after_reply+0xe2c/0x1780 [ptlrpc] [ 5296.097509] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5296.097590] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5296.097600] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5296.097603] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5296.097617] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5296.097621] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5296.097625] RAX: 000000000000730f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5296.097627] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5296.097629] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5296.097631] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5296.097633] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5296.097636] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5296.097639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5296.097641] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5296.097649] Call Trace: [ 5296.097655] [ 5296.097660] ? show_regs.cold.9+0x22/0x2f [ 5296.097670] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5296.097676] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5296.097681] ? watchdog+0x40/0x40 [ 5296.097685] ? hrtimer_interrupt+0x128/0x2c0 [ 5296.097690] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5296.097699] ? apic_timer_interrupt+0xf/0x20 [ 5296.097704] [ 5296.097707] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5296.097713] do_raw_spin_lock+0x177/0x190 [ 5296.097718] _raw_spin_lock+0x21/0x30 [ 5296.097723] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5296.098017] ? do_raw_spin_unlock+0x75/0x190 [ 5296.098022] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5296.098235] ? do_wait_intr+0xf0/0xf0 [ 5296.098244] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5296.147396] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5296.147792] ? __next_timer_interrupt+0x160/0x160 [ 5296.147868] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5296.148163] ? do_wait_intr+0xf0/0xf0 [ 5296.148172] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5296.148378] kthread+0x1d1/0x200 [ 5296.148384] ? set_kthread_struct+0x70/0x70 [ 5296.148388] ret_from_fork+0x1f/0x30 [ 5296.847764] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5296.849792] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5296.866441] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5296.868391] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5296.885605] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5296.890737] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5296.901936] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5296.918429] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5296.925514] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5296.930662] lmv_intent_open+0x338/0x1150 [lmv] [ 5296.938277] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5296.950896] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5296.961533] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5296.966461] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5296.974419] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5296.983546] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5296.991290] ? do_raw_spin_unlock+0x75/0x190 [ 5296.997877] ? _raw_spin_unlock+0x12/0x30 [ 5296.999371] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5297.009025] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5297.013636] ll_atomic_open+0x36a/0xd70 [lustre] [ 5297.019327] lookup_open+0x356/0xb40 [ 5297.022292] do_last+0x313/0xfc0 [ 5297.031717] ? nd_jump_root+0xe5/0x160 [ 5297.034205] ? path_init+0x437/0x520 [ 5297.036566] path_openat+0xf7/0x500 [ 5297.038161] ? do_raw_spin_unlock+0x75/0x190 [ 5297.042642] do_filp_open+0x99/0x140 [ 5297.044206] ? getname_flags+0x6e/0x330 [ 5297.047381] ? __check_object_size+0xff/0x256 [ 5297.049604] ? do_raw_spin_unlock+0x75/0x190 [ 5297.052180] ? _raw_spin_unlock+0x12/0x30 [ 5297.055395] do_sys_openat2+0x2b4/0x410 [ 5297.057585] do_sys_open+0x73/0xa0 [ 5297.062656] __x64_sys_openat+0x24/0x30 [ 5297.065231] do_syscall_64+0xc1/0x440 [ 5297.066627] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5297.073583] RIP: 0033:0x7f53425a5332 [ 5297.075396] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5297.084683] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5297.089312] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5297.099893] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5297.106334] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5297.110428] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5297.114607] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5312.060494] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 5312.063649] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5312.088668] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5312.103412] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5312.110970] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5312.116173] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5312.159743] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5312.166922] RAX: 0000000000005c1e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5312.173682] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5312.180076] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5312.185091] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5312.189722] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5312.201460] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5312.214599] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5312.221151] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5312.236410] Call Trace: [ 5312.237806] [ 5312.239001] ? show_regs.cold.9+0x22/0x2f [ 5312.245812] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5312.247323] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5312.256162] ? watchdog+0x40/0x40 [ 5312.260054] ? hrtimer_interrupt+0x128/0x2c0 [ 5312.268406] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5312.284439] ? apic_timer_interrupt+0xf/0x20 [ 5312.295149] [ 5312.302766] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5312.315268] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5312.321983] do_raw_spin_lock+0x177/0x190 [ 5312.327846] _raw_spin_lock+0x21/0x30 [ 5312.330162] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5312.346363] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5312.352589] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5312.354597] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5312.362145] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5312.370848] ? woken_wake_function+0x30/0x30 [ 5312.374748] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5312.377825] kthread+0x1d1/0x200 [ 5312.379306] ? set_kthread_struct+0x70/0x70 [ 5312.381944] ret_from_fork+0x1f/0x30 [ 5316.056235] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 5316.074579] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5316.114574] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5316.125970] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5316.133706] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5316.142275] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5316.147916] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5316.157631] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5316.160926] RAX: 0000000000001e13 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5316.163943] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5316.166906] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5316.170010] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5316.173049] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5316.175924] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5316.178917] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5316.180987] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5316.183606] Call Trace: [ 5316.184229] [ 5316.184761] ? show_regs.cold.9+0x22/0x2f [ 5316.189848] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5316.191560] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5316.196911] ? watchdog+0x40/0x40 [ 5316.198439] ? hrtimer_interrupt+0x128/0x2c0 [ 5316.200696] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5316.201881] ? apic_timer_interrupt+0xf/0x20 [ 5316.202933] [ 5316.203438] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5316.204704] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5316.205846] ? newidle_balance.isra.93+0x207/0x580 [ 5316.207252] do_raw_spin_lock+0x177/0x190 [ 5316.208790] _raw_spin_lock+0x21/0x30 [ 5316.209915] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5316.211374] process_one_work+0x2c8/0x700 [ 5316.212250] worker_thread+0x296/0x6e0 [ 5316.213097] ? rescuer_thread+0x570/0x570 [ 5316.213952] kthread+0x1d1/0x200 [ 5316.214686] ? set_kthread_struct+0x70/0x70 [ 5316.215546] ret_from_fork+0x1f/0x30 [ 5319.136066] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2682s! [ 5319.144129] Showing busy workqueues and worker pools: [ 5319.156511] workqueue events: flags=0x0 [ 5319.159571] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5319.173689] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5319.195763] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5319.202103] pending: cpu_latency_work [ptlrpc] [ 5319.206460] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5319.214622] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5319.223830] workqueue events_long: flags=0x0 [ 5319.233164] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5319.240864] pending: lu_objects_flush [obdclass] [ 5319.248801] workqueue events_unbound: flags=0x2 [ 5319.254584] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5319.264513] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5319.271417] workqueue events_power_efficient: flags=0x80 [ 5319.273392] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5319.276116] pending: check_lifetime, do_cache_clean [ 5319.278922] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5319.281764] pending: neigh_periodic_work, neigh_periodic_work [ 5319.284461] workqueue mm_percpu_wq: flags=0x8 [ 5319.286420] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5319.288691] pending: vmstat_update, lru_add_drain_per_cpu [ 5319.290272] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5319.291834] pending: vmstat_update, lru_add_drain_per_cpu [ 5319.293778] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5319.296068] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5319.298828] workqueue xprtiod: flags=0xa [ 5319.300158] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5319.302095] pending: xprt_autoclose, xs_error_handle [ 5319.303386] workqueue ipv6_addrconf: flags=0x40008 [ 5319.304946] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5319.307365] pending: addrconf_verify_work [ 5319.309085] workqueue knbd0-recv: flags=0x1a [ 5319.316752] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5319.323000] in-flight: 651:recv_work [ 5319.324944] workqueue ptlrpc_pinger: flags=0x2 [ 5319.327039] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5319.330018] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5319.332968] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2682s workers=4 idle: 15979 31824 8 [ 5319.336839] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5309s workers=2 idle: 654 [ 5324.055026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 5324.057472] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5324.060402] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 5324.070170] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5324.070181] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5324.070184] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5324.070195] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5324.070199] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5324.070204] RAX: 0000000000000fbb RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5324.070205] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5324.070207] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5324.070208] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5324.070210] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5324.070212] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5324.070214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5324.070216] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5324.070222] Call Trace: [ 5324.070226] [ 5324.070231] ? show_regs.cold.9+0x22/0x2f [ 5324.070240] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5324.070246] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5324.070249] ? watchdog+0x40/0x40 [ 5324.070254] ? hrtimer_interrupt+0x128/0x2c0 [ 5324.070257] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5324.070264] ? apic_timer_interrupt+0xf/0x20 [ 5324.070269] [ 5324.070272] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5324.070280] do_raw_spin_lock+0x177/0x190 [ 5324.070285] _raw_spin_lock+0x21/0x30 [ 5324.070289] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5324.074044] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) [ 5324.076850] ? do_raw_spin_unlock+0x75/0x190 [ 5324.082771] lmv(O) [ 5324.084165] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5324.084419] ? do_wait_intr+0xf0/0xf0 [ 5324.084427] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5324.084551] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5324.084694] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5324.084835] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5324.084921] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5324.084949] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5324.084972] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5324.084999] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5324.085121] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5324.085278] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5324.085305] lmv_intent_open+0x338/0x1150 [lmv] [ 5324.085345] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5324.098774] ec(O) [ 5324.101595] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5324.101718] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5324.101750] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5324.101835] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5324.101919] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5324.102003] ? do_raw_spin_unlock+0x75/0x190 [ 5324.102012] ? _raw_spin_unlock+0x12/0x30 [ 5324.102019] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5324.102101] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5324.102123] ll_atomic_open+0x36a/0xd70 [lustre] [ 5324.102212] lookup_open+0x356/0xb40 [ 5324.102221] do_last+0x313/0xfc0 [ 5324.102226] ? nd_jump_root+0xe5/0x160 [ 5324.102231] ? path_init+0x437/0x520 [ 5324.102236] path_openat+0xf7/0x500 [ 5324.102241] ? do_raw_spin_unlock+0x75/0x190 [ 5324.102245] do_filp_open+0x99/0x140 [ 5324.102251] ? getname_flags+0x6e/0x330 [ 5324.102256] ? __check_object_size+0xff/0x256 [ 5324.102262] ? do_raw_spin_unlock+0x75/0x190 [ 5324.102266] ? _raw_spin_unlock+0x12/0x30 [ 5324.102271] do_sys_openat2+0x2b4/0x410 [ 5324.102275] do_sys_open+0x73/0xa0 [ 5324.102279] __x64_sys_openat+0x24/0x30 [ 5324.102283] do_syscall_64+0xc1/0x440 [ 5324.102291] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5324.102296] RIP: 0033:0x7f53425a5332 [ 5324.102304] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5324.102308] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5324.102314] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5324.102378] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5324.102380] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5324.102382] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5324.102384] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5324.372789] fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5324.392519] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5324.399930] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5324.407877] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5324.410701] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5324.424657] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5324.432218] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5324.437938] RDX: 0000000000003463 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5324.441804] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5324.444842] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5324.446454] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5324.450551] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5324.456148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5324.465818] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5324.468190] Call Trace: [ 5324.471129] [ 5324.471817] ? show_regs.cold.9+0x22/0x2f [ 5324.472971] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5324.479617] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5324.482179] ? watchdog+0x40/0x40 [ 5324.484114] ? hrtimer_interrupt+0x128/0x2c0 [ 5324.486083] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5324.490605] ? apic_timer_interrupt+0xf/0x20 [ 5324.492901] [ 5324.493896] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5324.496440] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5324.499510] do_raw_spin_lock+0x177/0x190 [ 5324.502018] _raw_spin_lock+0x21/0x30 [ 5324.503766] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5324.506516] obd_mod_free+0x46/0x140 [ptlrpc] [ 5324.509943] mdc_commit_open+0x91/0xb0 [mdc] [ 5324.511635] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5324.515127] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5324.520235] after_reply+0xe2c/0x1780 [ptlrpc] [ 5324.525144] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5324.530604] ? __next_timer_interrupt+0x160/0x160 [ 5324.535934] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5324.539554] ? do_wait_intr+0xf0/0xf0 [ 5324.546892] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5324.549683] kthread+0x1d1/0x200 [ 5324.553190] ? set_kthread_struct+0x70/0x70 [ 5324.558042] ret_from_fork+0x1f/0x30 [ 5340.059981] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 5340.067644] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5340.121014] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5340.135692] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5340.151942] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5340.154135] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5340.172896] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5340.181539] RAX: 000000000000466f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5340.188361] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5340.196327] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5340.200924] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5340.211288] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5340.216923] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5340.228191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5340.236844] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5340.243877] Call Trace: [ 5340.246461] [ 5340.247460] ? show_regs.cold.9+0x22/0x2f [ 5340.250025] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5340.254750] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5340.256379] ? watchdog+0x40/0x40 [ 5340.257546] ? hrtimer_interrupt+0x128/0x2c0 [ 5340.260513] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5340.266666] ? apic_timer_interrupt+0xf/0x20 [ 5340.268667] [ 5340.270154] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5340.275347] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5340.283266] do_raw_spin_lock+0x177/0x190 [ 5340.287332] _raw_spin_lock+0x21/0x30 [ 5340.289259] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5340.292981] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5340.299368] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5340.302660] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5340.305395] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5340.308269] ? woken_wake_function+0x30/0x30 [ 5340.309842] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5340.312552] kthread+0x1d1/0x200 [ 5340.314215] ? set_kthread_struct+0x70/0x70 [ 5340.316306] ret_from_fork+0x1f/0x30 [ 5344.052584] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 5344.059556] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5344.093925] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5344.099810] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5344.103818] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5344.110844] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5344.113774] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5344.120647] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5344.123135] RAX: 0000000000007c26 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5344.125141] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5344.127359] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5344.129784] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5344.132255] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5344.134646] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5344.137590] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5344.139719] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5344.142346] Call Trace: [ 5344.143318] [ 5344.143944] ? show_regs.cold.9+0x22/0x2f [ 5344.145447] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5344.147275] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5344.149136] ? watchdog+0x40/0x40 [ 5344.160245] ? hrtimer_interrupt+0x128/0x2c0 [ 5344.162928] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5344.168944] ? apic_timer_interrupt+0xf/0x20 [ 5344.171774] [ 5344.172858] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5344.184957] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5344.188871] ? newidle_balance.isra.93+0x207/0x580 [ 5344.195051] do_raw_spin_lock+0x177/0x190 [ 5344.198148] _raw_spin_lock+0x21/0x30 [ 5344.199766] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5344.201921] process_one_work+0x2c8/0x700 [ 5344.203882] worker_thread+0x296/0x6e0 [ 5344.205264] ? rescuer_thread+0x570/0x570 [ 5344.206556] kthread+0x1d1/0x200 [ 5344.209535] ? set_kthread_struct+0x70/0x70 [ 5344.214198] ret_from_fork+0x1f/0x30 [ 5349.345812] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2712s! [ 5349.351354] Showing busy workqueues and worker pools: [ 5349.356029] workqueue events: flags=0x0 [ 5349.361719] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5349.369988] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5349.383428] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5349.387703] pending: cpu_latency_work [ptlrpc] [ 5349.391825] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5349.401641] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5349.408956] workqueue events_long: flags=0x0 [ 5349.413058] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5349.418352] pending: lu_objects_flush [obdclass] [ 5349.424521] workqueue events_unbound: flags=0x2 [ 5349.429090] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5349.433347] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5349.436512] workqueue events_power_efficient: flags=0x80 [ 5349.438172] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5349.440837] pending: check_lifetime, do_cache_clean [ 5349.442944] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5349.444872] pending: neigh_periodic_work, neigh_periodic_work [ 5349.447546] workqueue mm_percpu_wq: flags=0x8 [ 5349.456439] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5349.464864] pending: vmstat_update, lru_add_drain_per_cpu [ 5349.470631] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5349.473760] pending: vmstat_update, lru_add_drain_per_cpu [ 5349.475982] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5349.482665] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5349.485085] workqueue xprtiod: flags=0xa [ 5349.486602] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5349.489714] pending: xprt_autoclose, xs_error_handle [ 5349.494581] workqueue ipv6_addrconf: flags=0x40008 [ 5349.496037] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5349.498506] pending: addrconf_verify_work [ 5349.502413] workqueue knbd0-recv: flags=0x1a [ 5349.503763] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5349.506693] in-flight: 651:recv_work [ 5349.509300] workqueue ptlrpc_pinger: flags=0x2 [ 5349.511106] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5349.513318] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5349.515274] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2712s workers=4 idle: 15979 31824 8 [ 5349.518082] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5339s workers=2 idle: 654 [ 5352.057377] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 5352.058374] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 5352.068102] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5352.068992] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5352.068999] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5352.069002] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5352.069015] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5352.069020] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5352.069024] RAX: 0000000000001615 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5352.069026] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5352.069028] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5352.069030] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5352.069032] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5352.069035] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5352.069038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5352.069040] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5352.069049] Call Trace: [ 5352.069055] [ 5352.069061] ? show_regs.cold.9+0x22/0x2f [ 5352.069070] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5352.069077] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5352.069082] ? watchdog+0x40/0x40 [ 5352.069086] ? hrtimer_interrupt+0x128/0x2c0 [ 5352.069090] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5352.069097] ? apic_timer_interrupt+0xf/0x20 [ 5352.069102] [ 5352.069104] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5352.069111] do_raw_spin_lock+0x177/0x190 [ 5352.069116] _raw_spin_lock+0x21/0x30 [ 5352.069121] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5352.080201] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5352.080297] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5352.080305] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5352.080309] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5352.080323] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5352.080328] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5352.080334] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5352.080336] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5352.080338] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5352.080340] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5352.080352] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5352.080356] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5352.080359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5352.080361] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5352.080369] Call Trace: [ 5352.080374] [ 5352.080379] ? show_regs.cold.9+0x22/0x2f [ 5352.080388] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5352.080394] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5352.080398] ? watchdog+0x40/0x40 [ 5352.080402] ? hrtimer_interrupt+0x128/0x2c0 [ 5352.080405] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5352.080412] ? apic_timer_interrupt+0xf/0x20 [ 5352.080417] [ 5352.080419] ? native_safe_halt+0x2e/0x50 [ 5352.080423] kvm_wait+0xcd/0xe0 [ 5352.080431] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5352.080439] do_raw_spin_lock+0x177/0x190 [ 5352.080444] _raw_spin_lock+0x21/0x30 [ 5352.080449] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5352.139586] ? do_raw_spin_unlock+0x75/0x190 [ 5352.139807] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5352.140292] ? do_wait_intr+0xf0/0xf0 [ 5352.140303] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5352.158844] obd_mod_free+0x46/0x140 [ptlrpc] [ 5352.159748] mdc_commit_open+0x91/0xb0 [mdc] [ 5352.159787] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5352.160075] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5352.168802] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5352.174001] after_reply+0xe2c/0x1780 [ptlrpc] [ 5352.174331] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5352.175423] ? __next_timer_interrupt+0x160/0x160 [ 5352.175435] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5352.176003] ? do_wait_intr+0xf0/0xf0 [ 5352.176014] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5352.176229] kthread+0x1d1/0x200 [ 5352.176234] ? set_kthread_struct+0x70/0x70 [ 5352.176238] ret_from_fork+0x1f/0x30 [ 5352.401409] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5352.404485] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5352.407062] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5352.408489] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5352.413456] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5352.415856] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5352.418191] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5352.421193] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5352.423741] lmv_intent_open+0x338/0x1150 [lmv] [ 5352.425779] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5352.428273] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5352.430599] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5352.432838] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5352.434723] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5352.437003] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5352.439419] ? do_raw_spin_unlock+0x75/0x190 [ 5352.445797] ? _raw_spin_unlock+0x12/0x30 [ 5352.449562] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5352.453385] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5352.457169] ll_atomic_open+0x36a/0xd70 [lustre] [ 5352.459319] lookup_open+0x356/0xb40 [ 5352.460506] do_last+0x313/0xfc0 [ 5352.467595] ? nd_jump_root+0xe5/0x160 [ 5352.472485] ? path_init+0x437/0x520 [ 5352.474207] path_openat+0xf7/0x500 [ 5352.476554] ? do_raw_spin_unlock+0x75/0x190 [ 5352.481075] do_filp_open+0x99/0x140 [ 5352.483650] ? getname_flags+0x6e/0x330 [ 5352.484846] ? __check_object_size+0xff/0x256 [ 5352.485913] ? do_raw_spin_unlock+0x75/0x190 [ 5352.489365] ? _raw_spin_unlock+0x12/0x30 [ 5352.493399] do_sys_openat2+0x2b4/0x410 [ 5352.494774] do_sys_open+0x73/0xa0 [ 5352.497310] __x64_sys_openat+0x24/0x30 [ 5352.500600] do_syscall_64+0xc1/0x440 [ 5352.502457] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5352.506889] RIP: 0033:0x7f53425a5332 [ 5352.513639] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5352.523673] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5352.527587] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5352.530731] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5352.533577] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5352.536218] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5352.543513] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5368.063089] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5368.071444] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5368.102473] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5368.107882] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5368.113239] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5368.116357] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5368.133077] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5368.139485] RAX: 0000000000007f86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5368.144184] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5368.147625] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5368.161511] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5368.173265] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5368.181472] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5368.196541] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5368.200595] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5368.210043] Call Trace: [ 5368.211309] [ 5368.212916] ? show_regs.cold.9+0x22/0x2f [ 5368.215363] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5368.216479] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5368.225475] ? watchdog+0x40/0x40 [ 5368.232527] ? hrtimer_interrupt+0x128/0x2c0 [ 5368.240075] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5368.243745] ? apic_timer_interrupt+0xf/0x20 [ 5368.246366] [ 5368.248356] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5368.255272] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5368.259502] do_raw_spin_lock+0x177/0x190 [ 5368.262994] _raw_spin_lock+0x21/0x30 [ 5368.267725] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5368.271718] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5368.278086] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5368.284300] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5368.292401] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5368.298437] ? woken_wake_function+0x30/0x30 [ 5368.299927] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5368.304743] kthread+0x1d1/0x200 [ 5368.307025] ? set_kthread_struct+0x70/0x70 [ 5368.310658] ret_from_fork+0x1f/0x30 [ 5376.052028] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 5376.056887] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5376.093903] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5376.106725] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5376.112777] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5376.120089] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5376.123910] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5376.135774] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5376.141208] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5376.147249] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5376.149597] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5376.157788] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5376.163720] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5376.168260] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5376.179916] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5376.187671] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5376.192710] Call Trace: [ 5376.193700] [ 5376.194525] ? show_regs.cold.9+0x22/0x2f [ 5376.196032] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5376.197775] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5376.199363] ? watchdog+0x40/0x40 [ 5376.200315] ? hrtimer_interrupt+0x128/0x2c0 [ 5376.201629] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5376.203515] ? apic_timer_interrupt+0xf/0x20 [ 5376.205113] [ 5376.205878] ? native_safe_halt+0x2e/0x50 [ 5376.207372] kvm_wait+0xcd/0xe0 [ 5376.208597] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5376.210536] ? newidle_balance.isra.93+0x207/0x580 [ 5376.212223] do_raw_spin_lock+0x177/0x190 [ 5376.213796] _raw_spin_lock+0x21/0x30 [ 5376.215335] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5376.217584] process_one_work+0x2c8/0x700 [ 5376.219657] worker_thread+0x296/0x6e0 [ 5376.221086] ? rescuer_thread+0x570/0x570 [ 5376.222613] kthread+0x1d1/0x200 [ 5376.223784] ? set_kthread_struct+0x70/0x70 [ 5376.225300] ret_from_fork+0x1f/0x30 [ 5379.555104] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2742s! [ 5379.561136] Showing busy workqueues and worker pools: [ 5379.568971] workqueue events: flags=0x0 [ 5379.574449] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5379.581484] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5379.593132] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5379.600332] pending: cpu_latency_work [ptlrpc] [ 5379.603189] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5379.607047] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5379.615245] workqueue events_long: flags=0x0 [ 5379.617421] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5379.622956] pending: lu_objects_flush [obdclass] [ 5379.625730] workqueue events_unbound: flags=0x2 [ 5379.629547] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5379.632850] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5379.636629] workqueue events_power_efficient: flags=0x80 [ 5379.640728] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5379.644338] pending: check_lifetime, do_cache_clean [ 5379.647747] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5379.651330] pending: neigh_periodic_work, neigh_periodic_work [ 5379.657057] workqueue mm_percpu_wq: flags=0x8 [ 5379.661673] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5379.665475] pending: vmstat_update, lru_add_drain_per_cpu [ 5379.669747] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5379.677009] pending: vmstat_update, lru_add_drain_per_cpu [ 5379.682208] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5379.691064] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5379.694807] workqueue xprtiod: flags=0xa [ 5379.697913] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5379.701847] pending: xprt_autoclose, xs_error_handle [ 5379.708114] workqueue ipv6_addrconf: flags=0x40008 [ 5379.714167] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5379.722306] pending: addrconf_verify_work [ 5379.725293] workqueue knbd0-recv: flags=0x1a [ 5379.727945] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5379.730100] in-flight: 651:recv_work [ 5379.732746] workqueue ptlrpc_pinger: flags=0x2 [ 5379.735322] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5379.739049] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5379.742297] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2742s workers=4 idle: 15979 31824 8 [ 5379.748220] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5370s workers=2 idle: 654 [ 5380.057077] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 5380.059403] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5380.059769] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 5380.083909] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5380.083923] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5380.083926] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5380.083939] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5380.083944] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5380.083949] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5380.083952] RDX: 0000000000003cc1 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5380.083954] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5380.083956] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5380.083959] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5380.083962] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5380.083966] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5380.083968] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5380.083976] Call Trace: [ 5380.083981] [ 5380.085326] ? show_regs.cold.9+0x22/0x2f [ 5380.085357] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5380.085364] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5380.085368] ? watchdog+0x40/0x40 [ 5380.085372] ? hrtimer_interrupt+0x128/0x2c0 [ 5380.085377] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5380.085386] ? apic_timer_interrupt+0xf/0x20 [ 5380.085390] [ 5380.085394] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5380.085400] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5380.085404] do_raw_spin_lock+0x177/0x190 [ 5380.085410] _raw_spin_lock+0x21/0x30 [ 5380.085426] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5380.085699] obd_mod_free+0x46/0x140 [ptlrpc] [ 5380.085892] mdc_commit_open+0x91/0xb0 [mdc] [ 5380.085928] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5380.086138] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5380.086331] after_reply+0xe2c/0x1780 [ptlrpc] [ 5380.086562] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5380.086689] ? __next_timer_interrupt+0x160/0x160 [ 5380.086696] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5380.086842] ? do_wait_intr+0xf0/0xf0 [ 5380.086851] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5380.093185] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5380.093261] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5380.093269] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5380.093271] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5380.093284] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5380.093289] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5380.093293] RAX: 0000000000007e93 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5380.093296] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5380.093298] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5380.093300] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5380.093303] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5380.093306] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5380.093308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5380.093310] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5380.093319] Call Trace: [ 5380.093324] [ 5380.093330] ? show_regs.cold.9+0x22/0x2f [ 5380.093403] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5380.093410] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5380.093424] ? watchdog+0x40/0x40 [ 5380.093429] ? hrtimer_interrupt+0x128/0x2c0 [ 5380.093433] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5380.093506] ? apic_timer_interrupt+0xf/0x20 [ 5380.093512] [ 5380.093515] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5380.093522] do_raw_spin_lock+0x177/0x190 [ 5380.093527] _raw_spin_lock+0x21/0x30 [ 5380.093532] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5380.093808] ? do_raw_spin_unlock+0x75/0x190 [ 5380.093813] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5380.094010] ? do_wait_intr+0xf0/0xf0 [ 5380.094019] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5380.094218] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5380.100386] kthread+0x1d1/0x200 [ 5380.100405] ? set_kthread_struct+0x70/0x70 [ 5380.100409] ret_from_fork+0x1f/0x30 [ 5380.329935] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5380.334240] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5380.337744] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5380.345001] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5380.347398] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5380.349598] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5380.351731] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5380.354978] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5380.357755] lmv_intent_open+0x338/0x1150 [lmv] [ 5380.362571] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5380.366689] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5380.371558] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5380.374825] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5380.379744] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5380.381899] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5380.383884] ? do_raw_spin_unlock+0x75/0x190 [ 5380.386015] ? _raw_spin_unlock+0x12/0x30 [ 5380.387970] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5380.390802] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5380.394551] ll_atomic_open+0x36a/0xd70 [lustre] [ 5380.396878] lookup_open+0x356/0xb40 [ 5380.398258] do_last+0x313/0xfc0 [ 5380.400717] ? nd_jump_root+0xe5/0x160 [ 5380.402369] ? path_init+0x437/0x520 [ 5380.403560] path_openat+0xf7/0x500 [ 5380.405124] ? do_raw_spin_unlock+0x75/0x190 [ 5380.406848] do_filp_open+0x99/0x140 [ 5380.408260] ? getname_flags+0x6e/0x330 [ 5380.409932] ? __check_object_size+0xff/0x256 [ 5380.411766] ? do_raw_spin_unlock+0x75/0x190 [ 5380.413396] ? _raw_spin_unlock+0x12/0x30 [ 5380.415139] do_sys_openat2+0x2b4/0x410 [ 5380.416828] do_sys_open+0x73/0xa0 [ 5380.418128] __x64_sys_openat+0x24/0x30 [ 5380.419760] do_syscall_64+0xc1/0x440 [ 5380.421131] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5380.423939] RIP: 0033:0x7f53425a5332 [ 5380.426567] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5380.435625] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5380.440243] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5380.443277] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5380.446281] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5380.449295] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5380.452670] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5394.510785] rcu: INFO: rcu_sched self-detected stall on CPU [ 5394.515988] rcu: 2-....: (1746320 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 5394.527888] (t=3300084 jiffies g=274185 q=64494) [ 5394.530269] rcu: rcu_sched kthread starved for 2758450 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 5394.541462] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5394.553346] rcu: RCU grace-period kthread stack dump: [ 5394.558633] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5394.570916] Call Trace: [ 5394.571747] __schedule+0x351/0xcb0 [ 5394.572915] schedule+0xc0/0x180 [ 5394.578354] schedule_timeout+0xb4/0x190 [ 5394.581994] ? do_raw_spin_unlock+0x75/0x190 [ 5394.586993] ? __next_timer_interrupt+0x160/0x160 [ 5394.591006] ? prepare_to_swait_event+0xf3/0x140 [ 5394.593311] rcu_gp_kthread+0x8dd/0x15b0 [ 5394.595765] ? force_qs_rnp+0x2c0/0x2c0 [ 5394.598077] kthread+0x1d1/0x200 [ 5394.600046] ? set_kthread_struct+0x70/0x70 [ 5394.602782] ret_from_fork+0x1f/0x30 [ 5394.604382] rcu: Stack dump where RCU GP kthread last ran: [ 5394.607065] Sending NMI from CPU 2 to CPUs 1: [ 5394.609190] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 5394.610075] Sending NMI from CPU 2 to CPUs 1: [ 5394.622022] NMI backtrace for cpu 1 [ 5394.622030] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5394.622032] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5394.622034] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5394.622035] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5394.622038] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [ 5394.622040] RAX: 0000000000007dae RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5394.622041] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5394.622043] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5394.622044] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5394.622045] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5394.622046] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5394.622048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5394.622049] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5394.622050] Call Trace: [ 5394.622050] [ 5394.622051] ? show_regs.cold.9+0x22/0x2f [ 5394.622052] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 5394.622053] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 5394.622054] ? nmi_handle+0xbd/0x210 [ 5394.622056] ? default_do_nmi+0x4b/0x1b0 [ 5394.622056] ? do_nmi+0x280/0x380 [ 5394.622057] ? end_repeat_nmi+0x16/0x1f [ 5394.622058] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5394.622060] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5394.622061] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5394.622062] [ 5394.622063] do_raw_spin_lock+0x177/0x190 [ 5394.622064] _raw_spin_lock+0x21/0x30 [ 5394.622065] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5394.622066] ? do_raw_spin_unlock+0x75/0x190 [ 5394.622067] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5394.622068] ? do_wait_intr+0xf0/0xf0 [ 5394.622069] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5394.622070] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5394.622071] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5394.622072] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5394.622073] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5394.622074] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5394.622075] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5394.622076] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5394.622077] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5394.622079] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5394.622080] lmv_intent_open+0x338/0x1150 [lmv] [ 5394.622081] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5394.622092] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5394.622093] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5394.622094] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5394.622095] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5394.622096] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5394.622097] ? do_raw_spin_unlock+0x75/0x190 [ 5394.622098] ? _raw_spin_unlock+0x12/0x30 [ 5394.622099] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5394.622100] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5394.622102] ll_atomic_open+0x36a/0xd70 [lustre] [ 5394.622103] lookup_open+0x356/0xb40 [ 5394.622104] do_last+0x313/0xfc0 [ 5394.622104] ? nd_jump_root+0xe5/0x160 [ 5394.622106] ? path_init+0x437/0x520 [ 5394.622106] path_openat+0xf7/0x500 [ 5394.622107] ? do_raw_spin_unlock+0x75/0x190 [ 5394.622108] do_filp_open+0x99/0x140 [ 5394.622109] ? getname_flags+0x6e/0x330 [ 5394.622110] ? __check_object_size+0xff/0x256 [ 5394.622111] ? do_raw_spin_unlock+0x75/0x190 [ 5394.622112] ? _raw_spin_unlock+0x12/0x30 [ 5394.622113] do_sys_openat2+0x2b4/0x410 [ 5394.622114] do_sys_open+0x73/0xa0 [ 5394.622114] __x64_sys_openat+0x24/0x30 [ 5394.622115] do_syscall_64+0xc1/0x440 [ 5394.622116] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5394.622117] RIP: 0033:0x7f53425a5332 [ 5394.622119] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5394.622121] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5394.622123] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5394.622125] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5394.622126] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5394.622127] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5394.622129] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5394.624765] NMI backtrace for cpu 2 [ 5394.978631] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5394.994922] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5395.002043] Call Trace: [ 5395.003483] [ 5395.004466] ? dump_stack+0xbb/0x10e [ 5395.008620] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5395.012680] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5395.015385] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5395.017881] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5395.020197] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5395.026679] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5395.035257] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5395.040604] ? update_process_times+0x6d/0xc0 [ 5395.046148] ? tick_sched_handle.isra.18+0x27/0x90 [ 5395.047883] ? tick_sched_timer+0x53/0xb0 [ 5395.051658] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5395.056027] ? tick_sched_handle.isra.18+0x90/0x90 [ 5395.058027] ? hrtimer_interrupt+0x128/0x2c0 [ 5395.060687] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5395.066803] ? apic_timer_interrupt+0xf/0x20 [ 5395.069953] [ 5395.071505] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5395.074068] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5395.076668] ? do_raw_spin_lock+0x177/0x190 [ 5395.078639] ? _raw_spin_lock+0x21/0x30 [ 5395.080011] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5395.087331] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5395.091347] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5395.093016] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5395.101424] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5395.107711] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5395.114880] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5395.118716] ? __next_timer_interrupt+0x160/0x160 [ 5395.121414] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5395.126003] ? do_wait_intr+0xf0/0xf0 [ 5395.128900] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5395.133553] ? kthread+0x1d1/0x200 [ 5395.134542] ? set_kthread_struct+0x70/0x70 [ 5395.135637] ? ret_from_fork+0x1f/0x30 [ 5396.058172] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5396.061138] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5396.094570] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5396.111205] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5396.127141] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5396.129001] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5396.152476] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5396.159883] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5396.169812] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5396.177192] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5396.184188] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5396.191601] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5396.196774] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5396.202687] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5396.207402] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5396.211571] Call Trace: [ 5396.213982] [ 5396.214894] ? show_regs.cold.9+0x22/0x2f [ 5396.217255] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5396.219742] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5396.221978] ? watchdog+0x40/0x40 [ 5396.225307] ? hrtimer_interrupt+0x128/0x2c0 [ 5396.226732] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5396.229182] ? apic_timer_interrupt+0xf/0x20 [ 5396.231268] [ 5396.232194] ? native_safe_halt+0x2e/0x50 [ 5396.234442] kvm_wait+0xcd/0xe0 [ 5396.236222] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5396.238957] do_raw_spin_lock+0x177/0x190 [ 5396.240589] _raw_spin_lock+0x21/0x30 [ 5396.241963] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5396.244757] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5396.247555] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5396.252773] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5396.255497] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5396.261928] ? woken_wake_function+0x30/0x30 [ 5396.267070] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5396.272424] kthread+0x1d1/0x200 [ 5396.273910] ? set_kthread_struct+0x70/0x70 [ 5396.275628] ret_from_fork+0x1f/0x30 [ 5404.052231] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 5404.060776] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5404.112365] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5404.120411] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5404.125312] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5404.129798] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5404.135608] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5404.150245] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5404.153807] RAX: 0000000000007e8a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5404.156438] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5404.160614] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5404.163993] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5404.169071] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5404.172789] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5404.180076] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5404.182744] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5404.187393] Call Trace: [ 5404.188504] [ 5404.189709] ? show_regs.cold.9+0x22/0x2f [ 5404.191825] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5404.195929] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5404.199461] ? watchdog+0x40/0x40 [ 5404.200859] ? hrtimer_interrupt+0x128/0x2c0 [ 5404.202616] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5404.206062] ? apic_timer_interrupt+0xf/0x20 [ 5404.207471] [ 5404.208205] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5404.214182] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5404.219755] ? newidle_balance.isra.93+0x207/0x580 [ 5404.222731] do_raw_spin_lock+0x177/0x190 [ 5404.226409] _raw_spin_lock+0x21/0x30 [ 5404.227677] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5404.229586] process_one_work+0x2c8/0x700 [ 5404.234649] worker_thread+0x296/0x6e0 [ 5404.235910] ? rescuer_thread+0x570/0x570 [ 5404.238714] kthread+0x1d1/0x200 [ 5404.239786] ? set_kthread_struct+0x70/0x70 [ 5404.241264] ret_from_fork+0x1f/0x30 [ 5409.760049] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2772s! [ 5409.767981] Showing busy workqueues and worker pools: [ 5409.769873] workqueue events: flags=0x0 [ 5409.782631] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5409.795988] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5409.812376] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5409.814491] pending: cpu_latency_work [ptlrpc] [ 5409.826429] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5409.837544] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5409.847143] workqueue events_long: flags=0x0 [ 5409.850316] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5409.858469] pending: lu_objects_flush [obdclass] [ 5409.860750] workqueue events_unbound: flags=0x2 [ 5409.862733] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5409.865478] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5409.868184] workqueue events_power_efficient: flags=0x80 [ 5409.870076] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5409.872801] pending: check_lifetime, do_cache_clean [ 5409.886838] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5409.891343] pending: neigh_periodic_work, neigh_periodic_work [ 5409.894193] workqueue mm_percpu_wq: flags=0x8 [ 5409.896733] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5409.915323] pending: vmstat_update, lru_add_drain_per_cpu [ 5409.928220] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5409.942722] pending: vmstat_update, lru_add_drain_per_cpu [ 5409.950328] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5409.952593] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5409.965754] workqueue xprtiod: flags=0xa [ 5409.968096] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5409.972331] pending: xprt_autoclose, xs_error_handle [ 5409.978225] workqueue ipv6_addrconf: flags=0x40008 [ 5409.982392] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5409.987159] pending: addrconf_verify_work [ 5409.990027] workqueue knbd0-recv: flags=0x1a [ 5409.992234] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5409.998411] in-flight: 651:recv_work [ 5409.999717] workqueue ptlrpc_pinger: flags=0x2 [ 5410.007207] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5410.011105] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5410.014134] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2772s workers=4 idle: 15979 31824 8 [ 5410.022529] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5400s workers=2 idle: 654 [ 5420.056647] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 5420.058096] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 5420.064141] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5420.064214] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5420.064222] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5420.064225] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5420.064239] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5420.064244] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5420.064247] RAX: 0000000000007e67 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5420.064249] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5420.070853] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5420.070937] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5420.070945] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5420.070949] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5420.070963] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5420.070968] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5420.070973] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5420.070975] RDX: 0000000000007d90 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5420.070978] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5420.070981] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5420.070983] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5420.070986] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5420.070989] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5420.070991] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5420.070999] Call Trace: [ 5420.071004] [ 5420.071009] ? show_regs.cold.9+0x22/0x2f [ 5420.071019] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5420.071025] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5420.071029] ? watchdog+0x40/0x40 [ 5420.071033] ? hrtimer_interrupt+0x128/0x2c0 [ 5420.071037] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5420.071046] ? apic_timer_interrupt+0xf/0x20 [ 5420.071051] [ 5420.071054] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5420.071059] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5420.071063] do_raw_spin_lock+0x177/0x190 [ 5420.071067] _raw_spin_lock+0x21/0x30 [ 5420.071073] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5420.071395] obd_mod_free+0x46/0x140 [ptlrpc] [ 5420.071590] mdc_commit_open+0x91/0xb0 [mdc] [ 5420.071626] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5420.071838] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5420.072043] after_reply+0xe2c/0x1780 [ptlrpc] [ 5420.072247] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5420.072490] ? __next_timer_interrupt+0x160/0x160 [ 5420.072500] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5420.075262] ? do_wait_intr+0xf0/0xf0 [ 5420.075817] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5420.135049] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5420.135062] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5420.135064] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5420.135068] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5420.135071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5420.135073] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5420.135082] Call Trace: [ 5420.135088] [ 5420.135095] ? show_regs.cold.9+0x22/0x2f [ 5420.135109] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5420.135115] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5420.135119] ? watchdog+0x40/0x40 [ 5420.135124] ? hrtimer_interrupt+0x128/0x2c0 [ 5420.135128] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5420.135138] ? apic_timer_interrupt+0xf/0x20 [ 5420.135143] [ 5420.135146] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5420.135156] do_raw_spin_lock+0x177/0x190 [ 5420.135161] _raw_spin_lock+0x21/0x30 [ 5420.135166] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5420.136319] ? do_raw_spin_unlock+0x75/0x190 [ 5420.136328] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5420.136941] ? do_wait_intr+0xf0/0xf0 [ 5420.136951] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5420.137555] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5420.149024] kthread+0x1d1/0x200 [ 5420.149046] ? set_kthread_struct+0x70/0x70 [ 5420.149050] ret_from_fork+0x1f/0x30 [ 5420.494981] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5420.499923] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5420.504269] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5420.513528] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5420.516276] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5420.518340] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5420.521356] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5420.525816] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5420.529224] lmv_intent_open+0x338/0x1150 [lmv] [ 5420.531102] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5420.533444] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5420.541232] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5420.543625] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5420.545803] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5420.548533] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5420.551170] ? do_raw_spin_unlock+0x75/0x190 [ 5420.553467] ? _raw_spin_unlock+0x12/0x30 [ 5420.557475] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5420.563359] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5420.567921] ll_atomic_open+0x36a/0xd70 [lustre] [ 5420.570335] lookup_open+0x356/0xb40 [ 5420.572155] do_last+0x313/0xfc0 [ 5420.573375] ? nd_jump_root+0xe5/0x160 [ 5420.574801] ? path_init+0x437/0x520 [ 5420.579623] path_openat+0xf7/0x500 [ 5420.583290] ? do_raw_spin_unlock+0x75/0x190 [ 5420.586327] do_filp_open+0x99/0x140 [ 5420.592069] ? getname_flags+0x6e/0x330 [ 5420.595133] ? __check_object_size+0xff/0x256 [ 5420.597121] ? do_raw_spin_unlock+0x75/0x190 [ 5420.602200] ? _raw_spin_unlock+0x12/0x30 [ 5420.606154] do_sys_openat2+0x2b4/0x410 [ 5420.608244] do_sys_open+0x73/0xa0 [ 5420.610335] __x64_sys_openat+0x24/0x30 [ 5420.612539] do_syscall_64+0xc1/0x440 [ 5420.620276] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5420.623662] RIP: 0033:0x7f53425a5332 [ 5420.625049] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5420.639185] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5420.644580] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5420.652071] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5420.656140] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5420.663494] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5420.667688] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5424.058774] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5424.063582] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5424.103753] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5424.119642] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5424.125400] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5424.131367] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5424.150791] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5424.159819] RAX: 0000000000007589 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5424.170693] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5424.176418] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5424.181232] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5424.188723] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5424.194309] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5424.199762] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5424.202802] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5424.206765] Call Trace: [ 5424.207407] [ 5424.207979] ? show_regs.cold.9+0x22/0x2f [ 5424.208982] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5424.213887] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5424.216433] ? watchdog+0x40/0x40 [ 5424.220792] ? hrtimer_interrupt+0x128/0x2c0 [ 5424.225762] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5424.232205] ? apic_timer_interrupt+0xf/0x20 [ 5424.237032] [ 5424.244000] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5424.247086] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5424.250282] do_raw_spin_lock+0x177/0x190 [ 5424.252827] _raw_spin_lock+0x21/0x30 [ 5424.258503] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5424.269014] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5424.272902] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5424.281040] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5424.287777] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5424.291910] ? woken_wake_function+0x30/0x30 [ 5424.296027] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5424.299730] kthread+0x1d1/0x200 [ 5424.302404] ? set_kthread_struct+0x70/0x70 [ 5424.304646] ret_from_fork+0x1f/0x30 [ 5436.053526] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 5436.060632] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5436.131270] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5436.148820] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5436.165254] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5436.175050] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5436.181056] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5436.215949] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5436.222233] RAX: 0000000000002687 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5436.229478] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5436.250408] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5436.253021] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5436.261423] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5436.272120] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5436.282304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5436.293517] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5436.315110] Call Trace: [ 5436.315984] [ 5436.328052] ? show_regs.cold.9+0x22/0x2f [ 5436.332118] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5436.340207] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5436.355728] ? watchdog+0x40/0x40 [ 5436.363568] ? hrtimer_interrupt+0x128/0x2c0 [ 5436.371750] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5436.386606] ? apic_timer_interrupt+0xf/0x20 [ 5436.393476] [ 5436.400076] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5436.406772] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5436.414215] ? newidle_balance.isra.93+0x207/0x580 [ 5436.425865] do_raw_spin_lock+0x177/0x190 [ 5436.429317] _raw_spin_lock+0x21/0x30 [ 5436.431768] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5436.442941] process_one_work+0x2c8/0x700 [ 5436.456872] worker_thread+0x296/0x6e0 [ 5436.465579] ? rescuer_thread+0x570/0x570 [ 5436.484520] kthread+0x1d1/0x200 [ 5436.485666] ? set_kthread_struct+0x70/0x70 [ 5436.501053] ret_from_fork+0x1f/0x30 [ 5440.483335] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2803s! [ 5440.490488] Showing busy workqueues and worker pools: [ 5440.492209] workqueue events: flags=0x0 [ 5440.498070] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5440.507090] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5440.520990] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5440.529150] pending: cpu_latency_work [ptlrpc] [ 5440.534216] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5440.537538] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5440.544061] workqueue events_long: flags=0x0 [ 5440.549880] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5440.561222] pending: lu_objects_flush [obdclass] [ 5440.570128] workqueue events_unbound: flags=0x2 [ 5440.573947] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5440.580168] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5440.582493] workqueue events_power_efficient: flags=0x80 [ 5440.584677] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5440.588659] pending: check_lifetime, do_cache_clean [ 5440.591431] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5440.594674] pending: neigh_periodic_work, neigh_periodic_work [ 5440.597157] workqueue mm_percpu_wq: flags=0x8 [ 5440.599042] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5440.601904] pending: vmstat_update, lru_add_drain_per_cpu [ 5440.604175] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5440.606826] pending: vmstat_update, lru_add_drain_per_cpu [ 5440.609201] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5440.611934] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5440.614501] workqueue xprtiod: flags=0xa [ 5440.616083] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5440.618503] pending: xprt_autoclose, xs_error_handle [ 5440.620691] workqueue ipv6_addrconf: flags=0x40008 [ 5440.622530] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5440.624785] pending: addrconf_verify_work [ 5440.635690] workqueue knbd0-recv: flags=0x1a [ 5440.637144] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5440.639687] in-flight: 651:recv_work [ 5440.641059] workqueue ptlrpc_pinger: flags=0x2 [ 5440.643063] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5440.645560] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5440.648072] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2803s workers=4 idle: 15979 31824 8 [ 5440.653423] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5431s workers=2 idle: 654 [ 5448.057020] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 5448.061620] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5448.064649] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 5448.113292] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5448.113307] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5448.113310] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5448.113324] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5448.113328] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5448.113334] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5448.113337] RDX: 0000000000000d47 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5448.113339] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5448.113342] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5448.113344] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5448.113347] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5448.113350] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5448.113352] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5448.113360] Call Trace: [ 5448.113366] [ 5448.113371] ? show_regs.cold.9+0x22/0x2f [ 5448.113380] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5448.113386] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5448.113390] ? watchdog+0x40/0x40 [ 5448.113395] ? hrtimer_interrupt+0x128/0x2c0 [ 5448.113399] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5448.113407] ? apic_timer_interrupt+0xf/0x20 [ 5448.113411] [ 5448.113414] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5448.113419] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5448.113423] do_raw_spin_lock+0x177/0x190 [ 5448.113428] _raw_spin_lock+0x21/0x30 [ 5448.113436] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5448.113746] obd_mod_free+0x46/0x140 [ptlrpc] [ 5448.113966] mdc_commit_open+0x91/0xb0 [mdc] [ 5448.114007] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5448.114232] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5448.114451] after_reply+0xe2c/0x1780 [ptlrpc] [ 5448.120486] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5448.120569] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5448.120577] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5448.120580] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5448.120593] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5448.120597] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5448.120602] RAX: 0000000000000786 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5448.120605] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5448.120607] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5448.120610] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5448.120612] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5448.120615] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5448.120618] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5448.120621] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5448.120629] Call Trace: [ 5448.120634] [ 5448.120639] ? show_regs.cold.9+0x22/0x2f [ 5448.120649] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5448.120671] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5448.120675] ? watchdog+0x40/0x40 [ 5448.120680] ? hrtimer_interrupt+0x128/0x2c0 [ 5448.120684] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5448.120692] ? apic_timer_interrupt+0xf/0x20 [ 5448.120697] [ 5448.120700] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5448.120707] do_raw_spin_lock+0x177/0x190 [ 5448.121337] _raw_spin_lock+0x21/0x30 [ 5448.121343] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5448.121617] ? do_raw_spin_unlock+0x75/0x190 [ 5448.121624] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5448.124449] ? do_wait_intr+0xf0/0xf0 [ 5448.124461] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5448.125895] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5448.128239] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5448.128466] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5448.128584] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5448.128622] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5448.128669] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5448.128700] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5448.130113] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5448.141004] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5448.141503] ? __next_timer_interrupt+0x160/0x160 [ 5448.141514] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5448.141898] ? do_wait_intr+0xf0/0xf0 [ 5448.141908] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5448.154247] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5448.154300] lmv_intent_open+0x338/0x1150 [lmv] [ 5448.154331] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5448.154449] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5448.154886] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5448.155210] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5448.164993] kthread+0x1d1/0x200 [ 5448.165011] ? set_kthread_struct+0x70/0x70 [ 5448.165017] ret_from_fork+0x1f/0x30 [ 5448.815528] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5448.821155] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5448.830858] ? do_raw_spin_unlock+0x75/0x190 [ 5448.834589] ? _raw_spin_unlock+0x12/0x30 [ 5448.839094] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5448.841397] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5448.845606] ll_atomic_open+0x36a/0xd70 [lustre] [ 5448.850987] lookup_open+0x356/0xb40 [ 5448.853810] do_last+0x313/0xfc0 [ 5448.859035] ? nd_jump_root+0xe5/0x160 [ 5448.860297] ? path_init+0x437/0x520 [ 5448.861355] path_openat+0xf7/0x500 [ 5448.865274] ? do_raw_spin_unlock+0x75/0x190 [ 5448.867899] do_filp_open+0x99/0x140 [ 5448.873577] ? getname_flags+0x6e/0x330 [ 5448.875430] ? __check_object_size+0xff/0x256 [ 5448.881910] ? do_raw_spin_unlock+0x75/0x190 [ 5448.885456] ? _raw_spin_unlock+0x12/0x30 [ 5448.887592] do_sys_openat2+0x2b4/0x410 [ 5448.891290] do_sys_open+0x73/0xa0 [ 5448.895307] __x64_sys_openat+0x24/0x30 [ 5448.900517] do_syscall_64+0xc1/0x440 [ 5448.907028] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5448.912897] RIP: 0033:0x7f53425a5332 [ 5448.919333] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5448.935978] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5448.943339] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5448.951619] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5448.961072] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5448.966027] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5448.973231] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5452.060418] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5452.064655] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5452.101515] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5452.112612] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5452.119166] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5452.123131] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5452.134826] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5452.143148] RAX: 0000000000005480 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5452.147777] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5452.154505] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5452.162408] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5452.169597] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5452.173790] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5452.178592] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5452.183309] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5452.188331] Call Trace: [ 5452.190016] [ 5452.190567] ? show_regs.cold.9+0x22/0x2f [ 5452.192332] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5452.195128] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5452.197141] ? watchdog+0x40/0x40 [ 5452.198286] ? hrtimer_interrupt+0x128/0x2c0 [ 5452.203151] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5452.205668] ? apic_timer_interrupt+0xf/0x20 [ 5452.209333] [ 5452.211541] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5452.215317] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5452.217784] do_raw_spin_lock+0x177/0x190 [ 5452.220174] _raw_spin_lock+0x21/0x30 [ 5452.222641] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5452.231968] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5452.234058] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5452.245489] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5452.250424] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5452.260010] ? woken_wake_function+0x30/0x30 [ 5452.267219] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5452.270580] kthread+0x1d1/0x200 [ 5452.274089] ? set_kthread_struct+0x70/0x70 [ 5452.275566] ret_from_fork+0x1f/0x30 [ 5468.052032] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 5468.056877] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5468.104242] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5468.114009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5468.120930] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5468.124788] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5468.128629] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5468.140649] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5468.143649] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5468.146674] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5468.150033] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5468.153014] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5468.156285] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5468.160169] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5468.163196] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5468.165333] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5468.168514] Call Trace: [ 5468.170169] [ 5468.171246] ? show_regs.cold.9+0x22/0x2f [ 5468.173685] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5468.175717] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5468.177645] ? watchdog+0x40/0x40 [ 5468.178880] ? hrtimer_interrupt+0x128/0x2c0 [ 5468.180614] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5468.182342] ? apic_timer_interrupt+0xf/0x20 [ 5468.183874] [ 5468.184786] ? native_safe_halt+0x2e/0x50 [ 5468.186483] kvm_wait+0xcd/0xe0 [ 5468.187940] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5468.190040] ? newidle_balance.isra.93+0x207/0x580 [ 5468.191923] do_raw_spin_lock+0x177/0x190 [ 5468.193285] _raw_spin_lock+0x21/0x30 [ 5468.194527] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5468.197536] process_one_work+0x2c8/0x700 [ 5468.199526] worker_thread+0x296/0x6e0 [ 5468.201533] ? rescuer_thread+0x570/0x570 [ 5468.203360] kthread+0x1d1/0x200 [ 5468.204738] ? set_kthread_struct+0x70/0x70 [ 5468.206183] ret_from_fork+0x1f/0x30 [ 5470.687106] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2833s! [ 5470.691256] Showing busy workqueues and worker pools: [ 5470.695029] workqueue events: flags=0x0 [ 5470.696545] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5470.699154] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5470.704712] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5470.707312] pending: cpu_latency_work [ptlrpc] [ 5470.710108] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5470.712977] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5470.716806] workqueue events_long: flags=0x0 [ 5470.718533] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5470.721238] pending: lu_objects_flush [obdclass] [ 5470.723739] workqueue events_unbound: flags=0x2 [ 5470.725849] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5470.728305] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5470.730712] workqueue events_power_efficient: flags=0x80 [ 5470.733184] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5470.736195] pending: check_lifetime, do_cache_clean [ 5470.744570] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5470.749976] pending: neigh_periodic_work, neigh_periodic_work [ 5470.755212] workqueue mm_percpu_wq: flags=0x8 [ 5470.760862] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5470.768003] pending: vmstat_update, lru_add_drain_per_cpu [ 5470.773854] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5470.776187] pending: vmstat_update, lru_add_drain_per_cpu [ 5470.782229] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5470.787194] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5470.790196] workqueue xprtiod: flags=0xa [ 5470.793110] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5470.795275] pending: xprt_autoclose, xs_error_handle [ 5470.800741] workqueue ipv6_addrconf: flags=0x40008 [ 5470.806012] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5470.809154] pending: addrconf_verify_work [ 5470.811213] workqueue knbd0-recv: flags=0x1a [ 5470.812719] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5470.815267] in-flight: 651:recv_work [ 5470.817191] workqueue ptlrpc_pinger: flags=0x2 [ 5470.818951] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5470.821532] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5470.824558] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2833s workers=4 idle: 15979 31824 8 [ 5470.828138] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5461s workers=2 idle: 654 [ 5476.055049] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 5476.057015] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 5476.059091] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror [ 5476.062847] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5476.062926] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5476.062933] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5476.062937] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5476.062952] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5476.062956] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5476.062961] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5476.062963] RDX: 0000000000006f07 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5476.062965] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5476.062967] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5476.062970] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5476.062973] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5476.062976] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5476.062978] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5476.062987] Call Trace: [ 5476.062993] [ 5476.062999] ? show_regs.cold.9+0x22/0x2f [ 5476.063007] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5476.063013] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5476.063017] ? watchdog+0x40/0x40 [ 5476.063021] ? hrtimer_interrupt+0x128/0x2c0 [ 5476.063025] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5476.063033] ? apic_timer_interrupt+0xf/0x20 [ 5476.063038] [ 5476.063040] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5476.063045] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5476.063049] do_raw_spin_lock+0x177/0x190 [ 5476.063054] _raw_spin_lock+0x21/0x30 [ 5476.063061] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5476.063410] obd_mod_free+0x46/0x140 [ptlrpc] [ 5476.094982] dm_region_hash dm_log dm_mod [ 5476.117859] mdc_commit_open+0x91/0xb0 [mdc] [ 5476.117909] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5476.123323] sha512_ssse3 sha512_generic [ 5476.123337] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5476.123345] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5476.123348] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5476.123362] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5476.123367] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5476.123371] RAX: 000000000000040f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5476.123373] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5476.123375] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5476.123378] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5476.123380] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5476.123384] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5476.123386] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5476.123389] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5476.123412] Call Trace: [ 5476.123418] [ 5476.123424] ? show_regs.cold.9+0x22/0x2f [ 5476.123435] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5476.123442] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5476.123447] ? watchdog+0x40/0x40 [ 5476.123452] ? hrtimer_interrupt+0x128/0x2c0 [ 5476.123457] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5476.123466] ? apic_timer_interrupt+0xf/0x20 [ 5476.123471] [ 5476.123475] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5476.123484] do_raw_spin_lock+0x177/0x190 [ 5476.123488] _raw_spin_lock+0x21/0x30 [ 5476.123494] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5476.123944] ? do_raw_spin_unlock+0x75/0x190 [ 5476.123950] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5476.124170] ? do_wait_intr+0xf0/0xf0 [ 5476.124179] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5476.124394] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5476.124834] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5476.125715] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5476.125848] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5476.125958] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5476.125986] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5476.126013] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5476.126099] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5476.129950] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5476.132237] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5476.132283] lmv_intent_open+0x338/0x1150 [lmv] [ 5476.132315] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5476.132542] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5476.132763] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5476.132795] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5476.132886] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5476.132976] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5476.133064] ? do_raw_spin_unlock+0x75/0x190 [ 5476.133071] ? _raw_spin_unlock+0x12/0x30 [ 5476.133078] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5476.133105] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5476.133126] ll_atomic_open+0x36a/0xd70 [lustre] [ 5476.133211] lookup_open+0x356/0xb40 [ 5476.133221] do_last+0x313/0xfc0 [ 5476.133226] ? nd_jump_root+0xe5/0x160 [ 5476.133230] ? path_init+0x437/0x520 [ 5476.133235] path_openat+0xf7/0x500 [ 5476.133239] ? do_raw_spin_unlock+0x75/0x190 [ 5476.133244] do_filp_open+0x99/0x140 [ 5476.133251] ? getname_flags+0x6e/0x330 [ 5476.133256] ? __check_object_size+0xff/0x256 [ 5476.133263] ? do_raw_spin_unlock+0x75/0x190 [ 5476.133267] ? _raw_spin_unlock+0x12/0x30 [ 5476.133271] do_sys_openat2+0x2b4/0x410 [ 5476.133275] do_sys_open+0x73/0xa0 [ 5476.133279] __x64_sys_openat+0x24/0x30 [ 5476.133282] do_syscall_64+0xc1/0x440 [ 5476.133289] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5476.133294] RIP: 0033:0x7f53425a5332 [ 5476.133320] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5476.133325] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5476.133330] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5476.133332] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5476.133334] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5476.133336] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5476.133339] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5476.447867] after_reply+0xe2c/0x1780 [ptlrpc] [ 5476.450938] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5476.453324] ? __next_timer_interrupt+0x160/0x160 [ 5476.456981] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5476.459993] ? do_wait_intr+0xf0/0xf0 [ 5476.461892] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5476.466360] kthread+0x1d1/0x200 [ 5476.473028] ? set_kthread_struct+0x70/0x70 [ 5476.474521] ret_from_fork+0x1f/0x30 [ 5480.058132] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5480.067237] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5480.093261] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5480.107714] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5480.119722] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5480.121428] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5480.132301] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5480.139106] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5480.147153] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5480.154277] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5480.162039] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5480.168142] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5480.174821] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5480.184355] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5480.192989] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5480.199717] Call Trace: [ 5480.204150] [ 5480.205056] ? show_regs.cold.9+0x22/0x2f [ 5480.208193] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5480.214073] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5480.218279] ? watchdog+0x40/0x40 [ 5480.220370] ? hrtimer_interrupt+0x128/0x2c0 [ 5480.224801] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5480.230159] ? apic_timer_interrupt+0xf/0x20 [ 5480.233741] [ 5480.235475] ? native_safe_halt+0x2e/0x50 [ 5480.239195] kvm_wait+0xcd/0xe0 [ 5480.241129] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5480.246561] do_raw_spin_lock+0x177/0x190 [ 5480.251171] _raw_spin_lock+0x21/0x30 [ 5480.255362] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5480.263268] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5480.270271] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5480.276158] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5480.280911] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5480.284048] ? woken_wake_function+0x30/0x30 [ 5480.285892] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5480.288972] kthread+0x1d1/0x200 [ 5480.292900] ? set_kthread_struct+0x70/0x70 [ 5480.295218] ret_from_fork+0x1f/0x30 [ 5496.054134] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 5496.062569] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5496.112563] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5496.120530] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5496.129128] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5496.140711] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5496.144115] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5496.156320] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5496.159884] RAX: 0000000000007d87 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5496.174960] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5496.190747] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5496.206396] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5496.208614] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5496.222884] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5496.236236] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5496.237910] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5496.248290] Call Trace: [ 5496.249067] [ 5496.254960] ? show_regs.cold.9+0x22/0x2f [ 5496.260652] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5496.264824] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5496.267867] ? watchdog+0x40/0x40 [ 5496.269501] ? hrtimer_interrupt+0x128/0x2c0 [ 5496.272911] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5496.277646] ? apic_timer_interrupt+0xf/0x20 [ 5496.281057] [ 5496.284562] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5496.288052] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5496.294000] ? newidle_balance.isra.93+0x207/0x580 [ 5496.298117] do_raw_spin_lock+0x177/0x190 [ 5496.302263] _raw_spin_lock+0x21/0x30 [ 5496.304778] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5496.309078] process_one_work+0x2c8/0x700 [ 5496.316122] worker_thread+0x296/0x6e0 [ 5496.317757] ? rescuer_thread+0x570/0x570 [ 5496.319500] kthread+0x1d1/0x200 [ 5496.323967] ? set_kthread_struct+0x70/0x70 [ 5496.327714] ret_from_fork+0x1f/0x30 [ 5500.897259] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2863s! [ 5500.899436] Showing busy workqueues and worker pools: [ 5500.900532] workqueue events: flags=0x0 [ 5500.902793] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5500.904888] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5500.910975] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5500.914704] pending: cpu_latency_work [ptlrpc] [ 5500.917144] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5500.919992] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5500.925116] workqueue events_long: flags=0x0 [ 5500.926040] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5500.928436] pending: lu_objects_flush [obdclass] [ 5500.931177] workqueue events_unbound: flags=0x2 [ 5500.932323] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5500.937651] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5500.945007] workqueue events_power_efficient: flags=0x80 [ 5500.956123] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5500.970302] pending: check_lifetime, do_cache_clean [ 5500.975638] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5500.989809] pending: neigh_periodic_work, neigh_periodic_work [ 5501.001782] workqueue mm_percpu_wq: flags=0x8 [ 5501.003879] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5501.009371] pending: vmstat_update, lru_add_drain_per_cpu [ 5501.014946] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5501.018097] pending: vmstat_update, lru_add_drain_per_cpu [ 5501.021868] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5501.026037] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5501.034812] workqueue xprtiod: flags=0xa [ 5501.039940] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5501.052724] pending: xprt_autoclose, xs_error_handle [ 5501.054528] workqueue ipv6_addrconf: flags=0x40008 [ 5501.076708] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5501.078802] pending: addrconf_verify_work [ 5501.090882] workqueue knbd0-recv: flags=0x1a [ 5501.094783] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5501.101938] in-flight: 651:recv_work [ 5501.110860] workqueue ptlrpc_pinger: flags=0x2 [ 5501.112464] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5501.124056] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5501.146599] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2864s workers=4 idle: 15979 31824 8 [ 5501.167157] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5491s workers=2 idle: 654 [ 5504.055196] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 5504.057009] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 5504.060620] Modules linked in: lustre(O) osp(O) ofd(O) [ 5504.068876] Modules linked in: [ 5504.068882] lod(O) [ 5504.079129] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [ 5504.081190] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5504.081256] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5504.081262] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5504.081264] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5504.081276] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5504.081280] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5504.081284] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5504.081286] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5504.081288] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5504.081290] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5504.081292] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5504.081295] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5504.081297] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5504.081300] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5504.081309] Call Trace: [ 5504.081314] [ 5504.081320] ? show_regs.cold.9+0x22/0x2f [ 5504.081329] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5504.081335] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5504.081402] ? watchdog+0x40/0x40 [ 5504.081406] ? hrtimer_interrupt+0x128/0x2c0 [ 5504.081409] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5504.081415] ? apic_timer_interrupt+0xf/0x20 [ 5504.081419] [ 5504.081422] ? native_safe_halt+0x2e/0x50 [ 5504.081426] kvm_wait+0xcd/0xe0 [ 5504.081433] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5504.081441] do_raw_spin_lock+0x177/0x190 [ 5504.081445] _raw_spin_lock+0x21/0x30 [ 5504.081449] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5504.081816] ? do_raw_spin_unlock+0x75/0x190 [ 5504.081822] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5504.082457] ? do_wait_intr+0xf0/0xf0 [ 5504.082466] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5504.082739] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5504.082946] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5504.083249] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5504.083491] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5504.083532] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5504.083642] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5504.083666] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5504.083752] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5504.083992] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5504.084022] lmv_intent_open+0x338/0x1150 [lmv] [ 5504.084054] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5504.084261] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5504.084516] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5504.084543] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5504.084645] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5504.084734] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5504.084823] ? do_raw_spin_unlock+0x75/0x190 [ 5504.084831] ? _raw_spin_unlock+0x12/0x30 [ 5504.084837] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5504.084864] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5504.084886] ll_atomic_open+0x36a/0xd70 [lustre] [ 5504.084974] lookup_open+0x356/0xb40 [ 5504.084983] do_last+0x313/0xfc0 [ 5504.084988] ? nd_jump_root+0xe5/0x160 [ 5504.084993] ? path_init+0x437/0x520 [ 5504.084998] path_openat+0xf7/0x500 [ 5504.085003] ? do_raw_spin_unlock+0x75/0x190 [ 5504.085007] do_filp_open+0x99/0x140 [ 5504.085013] ? getname_flags+0x6e/0x330 [ 5504.085017] ? __check_object_size+0xff/0x256 [ 5504.085025] ? do_raw_spin_unlock+0x75/0x190 [ 5504.085030] ? _raw_spin_unlock+0x12/0x30 [ 5504.085036] do_sys_openat2+0x2b4/0x410 [ 5504.085041] do_sys_open+0x73/0xa0 [ 5504.085045] __x64_sys_openat+0x24/0x30 [ 5504.085048] do_syscall_64+0xc1/0x440 [ 5504.085055] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5504.085070] RIP: 0033:0x7f53425a5332 [ 5504.085156] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5504.085161] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5504.085176] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5504.085179] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5504.085181] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5504.085182] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5504.085185] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5504.816781] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5504.851720] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5504.859972] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5504.871894] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5504.879580] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5504.915974] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5504.926812] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5504.934316] RDX: 00000000000002c8 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5504.944166] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5504.958035] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5504.977073] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5504.984652] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5505.001888] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5505.011026] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5505.021891] Call Trace: [ 5505.023611] [ 5505.025309] ? show_regs.cold.9+0x22/0x2f [ 5505.029940] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5505.035836] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5505.048558] ? watchdog+0x40/0x40 [ 5505.051005] ? hrtimer_interrupt+0x128/0x2c0 [ 5505.061034] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5505.073324] ? apic_timer_interrupt+0xf/0x20 [ 5505.080175] [ 5505.084718] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5505.097626] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5505.107089] do_raw_spin_lock+0x177/0x190 [ 5505.119981] _raw_spin_lock+0x21/0x30 [ 5505.127019] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5505.137527] obd_mod_free+0x46/0x140 [ptlrpc] [ 5505.146945] mdc_commit_open+0x91/0xb0 [mdc] [ 5505.157710] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5505.170293] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5505.188447] after_reply+0xe2c/0x1780 [ptlrpc] [ 5505.190235] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5505.202383] ? __next_timer_interrupt+0x160/0x160 [ 5505.210816] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5505.221642] ? do_wait_intr+0xf0/0xf0 [ 5505.236015] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5505.257663] kthread+0x1d1/0x200 [ 5505.263936] ? set_kthread_struct+0x70/0x70 [ 5505.275780] ret_from_fork+0x1f/0x30 [ 5508.065208] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5508.069431] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5508.102983] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5508.111741] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5508.114629] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5508.117189] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5508.138219] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5508.142015] RAX: 00000000000027a0 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5508.146184] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5508.151628] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5508.159703] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5508.163589] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5508.167955] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5508.173567] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5508.176075] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5508.179383] Call Trace: [ 5508.181609] [ 5508.182943] ? show_regs.cold.9+0x22/0x2f [ 5508.185004] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5508.186713] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5508.189286] ? watchdog+0x40/0x40 [ 5508.191324] ? hrtimer_interrupt+0x128/0x2c0 [ 5508.193316] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5508.197054] ? apic_timer_interrupt+0xf/0x20 [ 5508.199016] [ 5508.200272] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5508.204658] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5508.208280] do_raw_spin_lock+0x177/0x190 [ 5508.210588] _raw_spin_lock+0x21/0x30 [ 5508.215581] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5508.219238] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5508.222328] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5508.223725] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5508.231038] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5508.235077] ? woken_wake_function+0x30/0x30 [ 5508.241411] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5508.245808] kthread+0x1d1/0x200 [ 5508.247294] ? set_kthread_struct+0x70/0x70 [ 5508.249251] ret_from_fork+0x1f/0x30 [ 5528.052375] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 5528.062731] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5528.112012] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5528.118684] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5528.125684] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5528.131594] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5528.134174] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5528.151015] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5528.163317] RAX: 0000000000005009 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5528.170542] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5528.177422] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5528.189064] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5528.201964] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5528.212479] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5528.224289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5528.228990] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5528.245175] Call Trace: [ 5528.249606] [ 5528.253597] ? show_regs.cold.9+0x22/0x2f [ 5528.261719] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5528.272346] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5528.277865] ? watchdog+0x40/0x40 [ 5528.286662] ? hrtimer_interrupt+0x128/0x2c0 [ 5528.294231] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5528.299535] ? apic_timer_interrupt+0xf/0x20 [ 5528.305371] [ 5528.308276] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5528.314272] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5528.325352] ? newidle_balance.isra.93+0x207/0x580 [ 5528.331391] do_raw_spin_lock+0x177/0x190 [ 5528.334055] _raw_spin_lock+0x21/0x30 [ 5528.347551] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5528.355618] process_one_work+0x2c8/0x700 [ 5528.362526] worker_thread+0x296/0x6e0 [ 5528.364708] ? rescuer_thread+0x570/0x570 [ 5528.376616] kthread+0x1d1/0x200 [ 5528.382992] ? set_kthread_struct+0x70/0x70 [ 5528.389800] ret_from_fork+0x1f/0x30 [ 5531.619546] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2894s! [ 5531.625312] Showing busy workqueues and worker pools: [ 5531.634396] workqueue events: flags=0x0 [ 5531.638153] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5531.643146] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5531.661562] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5531.666632] pending: cpu_latency_work [ptlrpc] [ 5531.669913] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5531.675294] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5531.682536] workqueue events_long: flags=0x0 [ 5531.688119] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5531.690349] pending: lu_objects_flush [obdclass] [ 5531.699697] workqueue events_unbound: flags=0x2 [ 5531.701232] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5531.707385] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5531.713863] workqueue events_power_efficient: flags=0x80 [ 5531.719515] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5531.722547] pending: check_lifetime, do_cache_clean [ 5531.725600] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5531.733407] pending: neigh_periodic_work, neigh_periodic_work [ 5531.737316] workqueue mm_percpu_wq: flags=0x8 [ 5531.740494] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5531.746697] pending: vmstat_update, lru_add_drain_per_cpu [ 5531.755235] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5531.759484] pending: vmstat_update, lru_add_drain_per_cpu [ 5531.763956] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5531.773575] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5531.778167] workqueue xprtiod: flags=0xa [ 5531.780138] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5531.787458] pending: xprt_autoclose, xs_error_handle [ 5531.789354] workqueue ipv6_addrconf: flags=0x40008 [ 5531.794945] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5531.798418] pending: addrconf_verify_work [ 5531.801767] workqueue knbd0-recv: flags=0x1a [ 5531.805019] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5531.807159] in-flight: 651:recv_work [ 5531.809215] workqueue ptlrpc_pinger: flags=0x2 [ 5531.812823] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5531.818005] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5531.831500] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2894s workers=4 idle: 15979 31824 8 [ 5531.834233] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5522s workers=2 idle: 654 [ 5532.056042] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 5532.057196] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 5532.070697] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5532.070777] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5532.070784] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5532.070787] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5532.070802] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5532.070806] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5532.070811] RAX: 0000000000007e83 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5532.070813] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5532.070815] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5532.070817] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5532.070819] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5532.070822] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5532.070825] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5532.070827] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5532.070836] Call Trace: [ 5532.070842] [ 5532.070849] ? show_regs.cold.9+0x22/0x2f [ 5532.070858] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5532.070864] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5532.070868] ? watchdog+0x40/0x40 [ 5532.070871] ? hrtimer_interrupt+0x128/0x2c0 [ 5532.070875] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5532.070883] ? apic_timer_interrupt+0xf/0x20 [ 5532.070888] [ 5532.070891] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5532.070898] do_raw_spin_lock+0x177/0x190 [ 5532.070903] _raw_spin_lock+0x21/0x30 [ 5532.070907] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5532.075252] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5532.075329] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5532.075337] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5532.075340] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5532.075353] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5532.075357] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5532.075361] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5532.075363] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5532.075365] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5532.075367] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5532.075369] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5532.075373] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5532.075375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5532.075377] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5532.075387] Call Trace: [ 5532.075392] [ 5532.075398] ? show_regs.cold.9+0x22/0x2f [ 5532.075409] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5532.075415] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5532.075420] ? watchdog+0x40/0x40 [ 5532.075424] ? hrtimer_interrupt+0x128/0x2c0 [ 5532.075428] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5532.075608] ? apic_timer_interrupt+0xf/0x20 [ 5532.075613] [ 5532.075617] ? native_safe_halt+0x2e/0x50 [ 5532.075621] kvm_wait+0xcd/0xe0 [ 5532.075628] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5532.075636] do_raw_spin_lock+0x177/0x190 [ 5532.075640] _raw_spin_lock+0x21/0x30 [ 5532.075645] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5532.076052] obd_mod_free+0x46/0x140 [ptlrpc] [ 5532.076315] mdc_commit_open+0x91/0xb0 [mdc] [ 5532.076355] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5532.076555] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5532.077208] after_reply+0xe2c/0x1780 [ptlrpc] [ 5532.077412] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5532.077605] ? __next_timer_interrupt+0x160/0x160 [ 5532.077615] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5532.119372] ? do_raw_spin_unlock+0x75/0x190 [ 5532.120767] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5532.121065] ? do_wait_intr+0xf0/0xf0 [ 5532.121075] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5532.121302] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5532.121529] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5532.129606] ? do_wait_intr+0xf0/0xf0 [ 5532.129629] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5532.130043] kthread+0x1d1/0x200 [ 5532.130050] ? set_kthread_struct+0x70/0x70 [ 5532.130054] ret_from_fork+0x1f/0x30 [ 5532.557906] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5532.560218] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5532.562342] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5532.564382] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5532.566144] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5532.568239] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5532.570655] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5532.572930] lmv_intent_open+0x338/0x1150 [lmv] [ 5532.574379] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5532.576608] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5532.584649] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5532.586331] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5532.594253] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5532.597734] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5532.600093] ? do_raw_spin_unlock+0x75/0x190 [ 5532.602424] ? _raw_spin_unlock+0x12/0x30 [ 5532.606012] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5532.608348] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5532.610810] ll_atomic_open+0x36a/0xd70 [lustre] [ 5532.612801] lookup_open+0x356/0xb40 [ 5532.614125] do_last+0x313/0xfc0 [ 5532.621639] ? nd_jump_root+0xe5/0x160 [ 5532.625097] ? path_init+0x437/0x520 [ 5532.626411] path_openat+0xf7/0x500 [ 5532.632589] ? do_raw_spin_unlock+0x75/0x190 [ 5532.636480] do_filp_open+0x99/0x140 [ 5532.640311] ? getname_flags+0x6e/0x330 [ 5532.642893] ? __check_object_size+0xff/0x256 [ 5532.648549] ? do_raw_spin_unlock+0x75/0x190 [ 5532.651462] ? _raw_spin_unlock+0x12/0x30 [ 5532.660121] do_sys_openat2+0x2b4/0x410 [ 5532.661521] do_sys_open+0x73/0xa0 [ 5532.663402] __x64_sys_openat+0x24/0x30 [ 5532.667038] do_syscall_64+0xc1/0x440 [ 5532.669014] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5532.674633] RIP: 0033:0x7f53425a5332 [ 5532.676584] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5532.692739] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5532.698772] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5532.705777] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5532.712396] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5532.719120] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5532.730534] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5536.064008] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5536.070926] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5536.112309] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5536.125923] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5536.132775] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5536.140982] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5536.163182] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5536.170418] RAX: 0000000000007082 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5536.179435] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5536.186744] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5536.190511] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5536.194047] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5536.202774] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5536.210917] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5536.216824] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5536.231756] Call Trace: [ 5536.232744] [ 5536.235935] ? show_regs.cold.9+0x22/0x2f [ 5536.245232] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5536.247397] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5536.249433] ? watchdog+0x40/0x40 [ 5536.252287] ? hrtimer_interrupt+0x128/0x2c0 [ 5536.256243] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5536.258146] ? apic_timer_interrupt+0xf/0x20 [ 5536.262032] [ 5536.263010] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5536.266254] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5536.270457] do_raw_spin_lock+0x177/0x190 [ 5536.272369] _raw_spin_lock+0x21/0x30 [ 5536.273717] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5536.281770] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5536.286589] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5536.289576] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5536.292526] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5536.299282] ? woken_wake_function+0x30/0x30 [ 5536.302467] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5536.308557] kthread+0x1d1/0x200 [ 5536.312647] ? set_kthread_struct+0x70/0x70 [ 5536.316044] ret_from_fork+0x1f/0x30 [ 5556.052637] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 5556.055330] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5556.072035] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5556.076049] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5556.082924] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5556.086882] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5556.089624] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5556.097088] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5556.100977] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5556.110165] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5556.117957] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5556.127993] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5556.133946] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5556.139006] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5556.144636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5556.152562] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5556.159126] Call Trace: [ 5556.160305] [ 5556.163180] ? show_regs.cold.9+0x22/0x2f [ 5556.171654] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5556.178988] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5556.182704] ? watchdog+0x40/0x40 [ 5556.186395] ? hrtimer_interrupt+0x128/0x2c0 [ 5556.191692] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5556.195279] ? apic_timer_interrupt+0xf/0x20 [ 5556.197651] [ 5556.199120] ? native_safe_halt+0x2e/0x50 [ 5556.204062] kvm_wait+0xcd/0xe0 [ 5556.205136] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5556.213632] ? newidle_balance.isra.93+0x207/0x580 [ 5556.220066] do_raw_spin_lock+0x177/0x190 [ 5556.221447] _raw_spin_lock+0x21/0x30 [ 5556.228103] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5556.232784] process_one_work+0x2c8/0x700 [ 5556.235395] worker_thread+0x296/0x6e0 [ 5556.237963] ? rescuer_thread+0x570/0x570 [ 5556.243001] kthread+0x1d1/0x200 [ 5556.245097] ? set_kthread_struct+0x70/0x70 [ 5556.250592] ret_from_fork+0x1f/0x30 [ 5560.055881] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 5560.057129] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 5560.063321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) [ 5560.067182] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) [ 5560.070478] lfsck(O) obdecho(O) mgc(O) [ 5560.077692] ec(O) fid(O) fld(O) [ 5560.079398] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel [ 5560.081321] ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5560.081430] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5560.081436] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5560.081439] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5560.081452] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5560.081457] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5560.081461] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5560.081464] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5560.081466] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5560.081469] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5560.081471] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5560.081474] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5560.081477] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5560.081480] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5560.081489] Call Trace: [ 5560.081495] [ 5560.081568] ? show_regs.cold.9+0x22/0x2f [ 5560.081578] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5560.081584] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5560.081589] ? watchdog+0x40/0x40 [ 5560.081593] ? hrtimer_interrupt+0x128/0x2c0 [ 5560.081597] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5560.081603] ? apic_timer_interrupt+0xf/0x20 [ 5560.081608] [ 5560.081610] ? native_safe_halt+0x2e/0x50 [ 5560.081614] kvm_wait+0xcd/0xe0 [ 5560.081621] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5560.081629] do_raw_spin_lock+0x177/0x190 [ 5560.081634] _raw_spin_lock+0x21/0x30 [ 5560.081640] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5560.082129] obd_mod_free+0x46/0x140 [ptlrpc] [ 5560.104575] ata_piix serio_raw libata dm_mirror dm_region_hash [ 5560.122134] mdc_commit_open+0x91/0xb0 [mdc] [ 5560.122202] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5560.122710] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5560.123059] after_reply+0xe2c/0x1780 [ptlrpc] [ 5560.123340] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5560.123680] ? __next_timer_interrupt+0x160/0x160 [ 5560.123691] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5560.124046] ? do_wait_intr+0xf0/0xf0 [ 5560.124056] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5560.124346] kthread+0x1d1/0x200 [ 5560.124352] ? set_kthread_struct+0x70/0x70 [ 5560.124426] ret_from_fork+0x1f/0x30 [ 5560.255304] dm_log dm_mod sha512_ssse3 sha512_generic [ 5560.258332] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5560.263129] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5560.267012] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5560.269725] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5560.277784] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5560.281602] RAX: 0000000000007c96 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5560.285104] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5560.288512] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5560.292509] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5560.296570] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5560.301160] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5560.306400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5560.309644] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5560.313427] Call Trace: [ 5560.314866] [ 5560.316130] ? show_regs.cold.9+0x22/0x2f [ 5560.318510] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5560.321074] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5560.324595] ? watchdog+0x40/0x40 [ 5560.326656] ? hrtimer_interrupt+0x128/0x2c0 [ 5560.329316] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5560.331876] ? apic_timer_interrupt+0xf/0x20 [ 5560.334259] [ 5560.335354] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5560.338514] do_raw_spin_lock+0x177/0x190 [ 5560.340699] _raw_spin_lock+0x21/0x30 [ 5560.342841] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5560.345780] ? do_raw_spin_unlock+0x75/0x190 [ 5560.347692] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5560.350924] ? do_wait_intr+0xf0/0xf0 [ 5560.355744] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5560.358529] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5560.361158] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5560.364175] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5560.366487] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5560.368886] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5560.371154] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5560.373105] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5560.375443] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5560.378408] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5560.381261] lmv_intent_open+0x338/0x1150 [lmv] [ 5560.383367] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5560.385644] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5560.388124] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5560.389970] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5560.392070] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5560.394581] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5560.396551] ? do_raw_spin_unlock+0x75/0x190 [ 5560.398003] ? _raw_spin_unlock+0x12/0x30 [ 5560.399379] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5560.400626] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5560.402050] ll_atomic_open+0x36a/0xd70 [lustre] [ 5560.404259] lookup_open+0x356/0xb40 [ 5560.405897] do_last+0x313/0xfc0 [ 5560.407448] ? nd_jump_root+0xe5/0x160 [ 5560.408897] ? path_init+0x437/0x520 [ 5560.410380] path_openat+0xf7/0x500 [ 5560.411662] ? do_raw_spin_unlock+0x75/0x190 [ 5560.413312] do_filp_open+0x99/0x140 [ 5560.414747] ? getname_flags+0x6e/0x330 [ 5560.416516] ? __check_object_size+0xff/0x256 [ 5560.418276] ? do_raw_spin_unlock+0x75/0x190 [ 5560.420049] ? _raw_spin_unlock+0x12/0x30 [ 5560.421909] do_sys_openat2+0x2b4/0x410 [ 5560.423466] do_sys_open+0x73/0xa0 [ 5560.424469] __x64_sys_openat+0x24/0x30 [ 5560.425818] do_syscall_64+0xc1/0x440 [ 5560.427172] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5560.429219] RIP: 0033:0x7f53425a5332 [ 5560.430670] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5560.451882] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5560.458536] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5560.466323] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5560.473208] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5560.475356] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5560.480764] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5562.335904] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2925s! [ 5562.338851] Showing busy workqueues and worker pools: [ 5562.341371] workqueue events: flags=0x0 [ 5562.342905] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5562.345716] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5562.350224] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5562.353105] pending: cpu_latency_work [ptlrpc] [ 5562.355274] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5562.360726] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5562.365503] workqueue events_long: flags=0x0 [ 5562.367568] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5562.370071] pending: lu_objects_flush [obdclass] [ 5562.371656] workqueue events_unbound: flags=0x2 [ 5562.372795] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5562.375071] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5562.377262] workqueue events_power_efficient: flags=0x80 [ 5562.379357] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5562.382053] pending: check_lifetime, do_cache_clean [ 5562.384020] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5562.386370] pending: neigh_periodic_work, neigh_periodic_work [ 5562.388610] workqueue mm_percpu_wq: flags=0x8 [ 5562.390523] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5562.392974] pending: vmstat_update, lru_add_drain_per_cpu [ 5562.395086] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5562.398517] pending: vmstat_update, lru_add_drain_per_cpu [ 5562.400785] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5562.403379] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5562.405856] workqueue xprtiod: flags=0xa [ 5562.407352] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5562.409637] pending: xprt_autoclose, xs_error_handle [ 5562.411586] workqueue ipv6_addrconf: flags=0x40008 [ 5562.413713] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5562.415972] pending: addrconf_verify_work [ 5562.417613] workqueue knbd0-recv: flags=0x1a [ 5562.419323] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5562.422042] in-flight: 651:recv_work [ 5562.423635] workqueue ptlrpc_pinger: flags=0x2 [ 5562.425336] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5562.427520] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5562.429761] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2925s workers=4 idle: 15979 31824 8 [ 5562.432755] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5552s workers=2 idle: 654 [ 5564.058009] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5564.062067] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5564.087007] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5564.094811] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5564.099349] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 5564.102290] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 5564.110890] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5564.114897] RAX: 0000000000002904 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5564.120917] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5564.127301] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5564.130961] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5564.137044] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5564.143085] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5564.151958] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5564.154802] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5564.161891] Call Trace: [ 5564.163262] [ 5564.164082] ? show_regs.cold.9+0x22/0x2f [ 5564.167157] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5564.168192] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5564.170961] ? watchdog+0x40/0x40 [ 5564.172896] ? hrtimer_interrupt+0x128/0x2c0 [ 5564.174342] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5564.176102] ? apic_timer_interrupt+0xf/0x20 [ 5564.177109] [ 5564.177616] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 5564.178846] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5564.181303] do_raw_spin_lock+0x177/0x190 [ 5564.183936] _raw_spin_lock+0x21/0x30 [ 5564.186865] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5564.193254] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5564.196338] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5564.199290] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5564.201331] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5564.204891] ? woken_wake_function+0x30/0x30 [ 5564.207127] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5564.214499] kthread+0x1d1/0x200 [ 5564.216069] ? set_kthread_struct+0x70/0x70 [ 5564.218346] ret_from_fork+0x1f/0x30 [ 5574.515831] rcu: INFO: rcu_sched self-detected stall on CPU [ 5574.519687] rcu: 2-....: (1840321 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 5574.524514] (t=3480089 jiffies g=274185 q=64583) [ 5574.526754] rcu: rcu_sched kthread starved for 2938455 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 5574.537264] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5574.545451] rcu: RCU grace-period kthread stack dump: [ 5574.548520] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5574.553592] Call Trace: [ 5574.554517] __schedule+0x351/0xcb0 [ 5574.560817] schedule+0xc0/0x180 [ 5574.565761] schedule_timeout+0xb4/0x190 [ 5574.567919] ? do_raw_spin_unlock+0x75/0x190 [ 5574.570793] ? __next_timer_interrupt+0x160/0x160 [ 5574.573336] ? prepare_to_swait_event+0xf3/0x140 [ 5574.578036] rcu_gp_kthread+0x8dd/0x15b0 [ 5574.581621] ? force_qs_rnp+0x2c0/0x2c0 [ 5574.583697] kthread+0x1d1/0x200 [ 5574.585547] ? set_kthread_struct+0x70/0x70 [ 5574.591115] ret_from_fork+0x1f/0x30 [ 5574.595717] rcu: Stack dump where RCU GP kthread last ran: [ 5574.600454] Sending NMI from CPU 2 to CPUs 3: [ 5574.602006] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [ 5574.605771] Sending NMI from CPU 2 to CPUs 1: [ 5574.619814] NMI backtrace for cpu 1 [ 5574.619822] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5574.619824] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5574.619825] RIP: 0010:apic_timer_interrupt+0x0/0x20 [ 5574.619826] Code: 04 24 48 89 54 24 08 48 89 ef e8 7b ee 2b ff 48 89 c7 48 8b 54 24 08 48 8b 04 24 48 83 c4 18 ff e7 c3 90 90 90 90 90 90 90 90 <68> 13 ff ff ff e8 86 f0 ff ff e8 d1 12 00 00 e9 7b f1 ff ff 66 66 [ 5574.619830] RSP: 0018:ffffbb53815034d8 EFLAGS: 00000012 [ 5574.619832] RAX: 0000000000001454 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5574.619833] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5574.619835] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5574.619836] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5574.619838] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5574.619839] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5574.619840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5574.619842] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5574.619843] Call Trace: [ 5574.619844] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5574.619846] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5574.619848] RSP: 0018:ffffbb5381503500 EFLAGS: 00000212 ? do_raw_spin_lock+0x177/0x190 [ 5574.619850] ? _raw_spin_lock+0x21/0x30 [ 5574.619851] ? ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5574.619852] ? do_raw_spin_unlock+0x75/0x190 [ 5574.619853] ? ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5574.619854] ? do_wait_intr+0xf0/0xf0 [ 5574.619855] ? ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5574.619856] ? ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5574.619857] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5574.619858] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5574.619860] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5574.619861] ? mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5574.619863] ? mdc_intent_lock+0x353/0x7e0 [mdc] [ 5574.619864] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5574.619865] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5574.619866] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5574.619867] ? lmv_intent_open+0x338/0x1150 [lmv] [ 5574.619868] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5574.619869] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5574.619870] ? lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5574.619872] ? ll_intent_lock+0xc5/0xaa0 [lustre] [ 5574.619873] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5574.619873] ? ll_lookup_it+0xe0e/0x2270 [lustre] [ 5574.619875] ? do_raw_spin_unlock+0x75/0x190 [ 5574.619876] ? _raw_spin_unlock+0x12/0x30 [ 5574.619877] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5574.619878] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5574.619879] ? ll_atomic_open+0x36a/0xd70 [lustre] [ 5574.619880] ? lookup_open+0x356/0xb40 [ 5574.619881] ? do_last+0x313/0xfc0 [ 5574.619882] ? nd_jump_root+0xe5/0x160 [ 5574.619883] ? path_init+0x437/0x520 [ 5574.619884] ? path_openat+0xf7/0x500 [ 5574.619885] ? do_raw_spin_unlock+0x75/0x190 [ 5574.619886] ? do_filp_open+0x99/0x140 [ 5574.619886] ? getname_flags+0x6e/0x330 [ 5574.619888] ? __check_object_size+0xff/0x256 [ 5574.619889] ? do_raw_spin_unlock+0x75/0x190 [ 5574.619890] ? _raw_spin_unlock+0x12/0x30 [ 5574.619891] ? do_sys_openat2+0x2b4/0x410 [ 5574.619892] ? do_sys_open+0x73/0xa0 [ 5574.619893] ? __x64_sys_openat+0x24/0x30 [ 5574.619894] ? do_syscall_64+0xc1/0x440 [ 5574.619895] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5574.629730] NMI backtrace for cpu 2 [ 5574.888801] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5574.900964] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5574.908527] Call Trace: [ 5574.910863] [ 5574.911614] ? dump_stack+0xbb/0x10e [ 5574.914427] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5574.917657] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5574.919304] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5574.923883] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5574.926576] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5574.929476] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5574.932170] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5574.935066] ? update_process_times+0x6d/0xc0 [ 5574.942210] ? tick_sched_handle.isra.18+0x27/0x90 [ 5574.948289] ? tick_sched_timer+0x53/0xb0 [ 5574.952306] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5574.963248] ? tick_sched_handle.isra.18+0x90/0x90 [ 5574.968419] ? hrtimer_interrupt+0x128/0x2c0 [ 5574.973706] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5574.976948] ? apic_timer_interrupt+0xf/0x20 [ 5574.981527] [ 5574.982214] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5574.984900] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5574.987209] ? do_raw_spin_lock+0x177/0x190 [ 5574.988665] ? _raw_spin_lock+0x21/0x30 [ 5574.992217] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5574.999181] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5575.003086] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5575.006210] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5575.009118] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5575.011308] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5575.014321] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5575.017130] ? __next_timer_interrupt+0x160/0x160 [ 5575.019367] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5575.021732] ? do_wait_intr+0xf0/0xf0 [ 5575.025640] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5575.030600] ? kthread+0x1d1/0x200 [ 5575.033330] ? set_kthread_struct+0x70/0x70 [ 5575.036027] ? ret_from_fork+0x1f/0x30 [ 5588.055610] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 5588.068328] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5588.125204] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5588.133971] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5588.140514] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5588.147178] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5588.155778] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5588.175720] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5588.178133] RAX: 0000000000007ea8 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5588.185344] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5588.191275] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5588.198612] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5588.208021] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5588.210364] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5588.218557] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5588.223875] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5588.233533] Call Trace: [ 5588.235219] [ 5588.237498] ? show_regs.cold.9+0x22/0x2f [ 5588.242015] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5588.250825] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5588.252497] ? watchdog+0x40/0x40 [ 5588.255310] ? hrtimer_interrupt+0x128/0x2c0 [ 5588.258384] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5588.261978] ? apic_timer_interrupt+0xf/0x20 [ 5588.265350] [ 5588.266273] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5588.269951] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5588.272214] ? newidle_balance.isra.93+0x207/0x580 [ 5588.274314] do_raw_spin_lock+0x177/0x190 [ 5588.276261] _raw_spin_lock+0x21/0x30 [ 5588.277634] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5588.279552] process_one_work+0x2c8/0x700 [ 5588.280572] worker_thread+0x296/0x6e0 [ 5588.282513] ? rescuer_thread+0x570/0x570 [ 5588.284443] kthread+0x1d1/0x200 [ 5588.291977] ? set_kthread_struct+0x70/0x70 [ 5588.293508] ret_from_fork+0x1f/0x30 [ 5592.544100] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2955s! [ 5592.546969] Showing busy workqueues and worker pools: [ 5592.549052] workqueue events: flags=0x0 [ 5592.550556] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5592.559106] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5592.568321] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5592.573387] pending: cpu_latency_work [ptlrpc] [ 5592.579226] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5592.584528] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5592.591327] workqueue events_long: flags=0x0 [ 5592.598090] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5592.603994] pending: lu_objects_flush [obdclass] [ 5592.606118] workqueue events_unbound: flags=0x2 [ 5592.618173] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5592.620293] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5592.625027] workqueue events_power_efficient: flags=0x80 [ 5592.631884] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5592.638307] pending: check_lifetime, do_cache_clean [ 5592.645115] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5592.656573] pending: neigh_periodic_work, neigh_periodic_work [ 5592.662398] workqueue mm_percpu_wq: flags=0x8 [ 5592.666945] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5592.671826] pending: vmstat_update, lru_add_drain_per_cpu [ 5592.673864] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5592.676422] pending: vmstat_update, lru_add_drain_per_cpu [ 5592.681573] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5592.684022] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5592.688322] workqueue xprtiod: flags=0xa [ 5592.693188] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5592.696455] pending: xprt_autoclose, xs_error_handle [ 5592.700232] workqueue ipv6_addrconf: flags=0x40008 [ 5592.702062] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5592.705488] pending: addrconf_verify_work [ 5592.707745] workqueue knbd0-recv: flags=0x1a [ 5592.709450] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5592.717609] in-flight: 651:recv_work [ 5592.719137] workqueue ptlrpc_pinger: flags=0x2 [ 5592.723044] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5592.728993] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5592.731160] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2955s workers=4 idle: 15979 31824 8 [ 5592.737915] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5583s workers=2 idle: 654 [ 5600.056510] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 5600.058122] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5600.058358] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 5600.058365] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5600.058442] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5600.058449] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5600.058451] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5600.058467] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5600.058472] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5600.058478] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5600.058480] RDX: 0000000000003eee RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5600.058494] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5600.058497] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5600.058499] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5600.058503] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5600.058507] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5600.058510] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5600.058518] Call Trace: [ 5600.058524] [ 5600.058530] ? show_regs.cold.9+0x22/0x2f [ 5600.058540] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5600.058548] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5600.058553] ? watchdog+0x40/0x40 [ 5600.058558] ? hrtimer_interrupt+0x128/0x2c0 [ 5600.058563] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5600.058574] ? apic_timer_interrupt+0xf/0x20 [ 5600.058579] [ 5600.058582] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5600.058588] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5600.058593] do_raw_spin_lock+0x177/0x190 [ 5600.058598] _raw_spin_lock+0x21/0x30 [ 5600.058604] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5600.058905] obd_mod_free+0x46/0x140 [ptlrpc] [ 5600.061583] mdc_commit_open+0x91/0xb0 [mdc] [ 5600.061622] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5600.061822] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5600.062103] after_reply+0xe2c/0x1780 [ptlrpc] [ 5600.062406] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5600.062663] ? __next_timer_interrupt+0x160/0x160 [ 5600.062673] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5600.062868] ? do_wait_intr+0xf0/0xf0 [ 5600.062877] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5600.063078] kthread+0x1d1/0x200 [ 5600.063084] ? set_kthread_struct+0x70/0x70 [ 5600.063088] ret_from_fork+0x1f/0x30 [ 5600.067092] Modules linked in: [ 5600.073343] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5600.073420] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5600.073427] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5600.073429] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5600.073444] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5600.073448] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5600.073453] RAX: 000000000000028c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5600.073455] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5600.073457] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5600.073459] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5600.073461] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5600.073464] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5600.073467] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5600.073469] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5600.073478] Call Trace: [ 5600.073492] [ 5600.073498] ? show_regs.cold.9+0x22/0x2f [ 5600.073508] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5600.073514] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5600.073519] ? watchdog+0x40/0x40 [ 5600.073523] ? hrtimer_interrupt+0x128/0x2c0 [ 5600.073528] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5600.073536] ? apic_timer_interrupt+0xf/0x20 [ 5600.073540] [ 5600.073543] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5600.073548] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5600.073554] do_raw_spin_lock+0x177/0x190 [ 5600.073559] _raw_spin_lock+0x21/0x30 [ 5600.073564] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5600.075121] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5600.076287] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5600.077101] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5600.077634] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5600.078072] ? woken_wake_function+0x30/0x30 [ 5600.078080] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5600.078912] kthread+0x1d1/0x200 [ 5600.078920] ? set_kthread_struct+0x70/0x70 [ 5600.078924] ret_from_fork+0x1f/0x30 [ 5600.546691] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5600.586134] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5600.594656] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5600.604911] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5600.611376] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5600.624653] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5600.626455] RAX: 0000000000003427 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5600.629614] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5600.635408] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5600.641166] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5600.647943] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5600.652670] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5600.661813] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5600.666874] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5600.670822] Call Trace: [ 5600.674441] [ 5600.675916] ? show_regs.cold.9+0x22/0x2f [ 5600.683824] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5600.686368] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5600.691097] ? watchdog+0x40/0x40 [ 5600.691899] ? hrtimer_interrupt+0x128/0x2c0 [ 5600.694780] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5600.699539] ? apic_timer_interrupt+0xf/0x20 [ 5600.701946] [ 5600.706646] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5600.711772] do_raw_spin_lock+0x177/0x190 [ 5600.714389] _raw_spin_lock+0x21/0x30 [ 5600.716242] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5600.727635] ? do_raw_spin_unlock+0x75/0x190 [ 5600.732096] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5600.734915] ? do_wait_intr+0xf0/0xf0 [ 5600.739792] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5600.742300] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5600.750274] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5600.756391] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5600.759606] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5600.763908] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5600.766305] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5600.768662] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5600.774801] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5600.779857] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5600.785232] lmv_intent_open+0x338/0x1150 [lmv] [ 5600.789291] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5600.795231] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5600.800703] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5600.802280] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5600.805820] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5600.810505] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5600.812200] ? do_raw_spin_unlock+0x75/0x190 [ 5600.815240] ? _raw_spin_unlock+0x12/0x30 [ 5600.817119] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5600.822261] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5600.826493] ll_atomic_open+0x36a/0xd70 [lustre] [ 5600.830869] lookup_open+0x356/0xb40 [ 5600.832737] do_last+0x313/0xfc0 [ 5600.834026] ? nd_jump_root+0xe5/0x160 [ 5600.839443] ? path_init+0x437/0x520 [ 5600.847304] path_openat+0xf7/0x500 [ 5600.850692] ? do_raw_spin_unlock+0x75/0x190 [ 5600.855295] do_filp_open+0x99/0x140 [ 5600.858070] ? getname_flags+0x6e/0x330 [ 5600.860428] ? __check_object_size+0xff/0x256 [ 5600.863831] ? do_raw_spin_unlock+0x75/0x190 [ 5600.865841] ? _raw_spin_unlock+0x12/0x30 [ 5600.869984] do_sys_openat2+0x2b4/0x410 [ 5600.875759] do_sys_open+0x73/0xa0 [ 5600.877980] __x64_sys_openat+0x24/0x30 [ 5600.879937] do_syscall_64+0xc1/0x440 [ 5600.883140] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5600.886890] RIP: 0033:0x7f53425a5332 [ 5600.887802] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5600.905996] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5600.910875] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5600.915353] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5600.922869] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5600.932670] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5600.942903] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5620.055646] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 5620.065508] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5620.111370] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5620.120796] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5620.128407] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5620.132498] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5620.134615] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5620.154507] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5620.165117] RAX: 0000000000007f0e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5620.169976] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5620.172125] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5620.180633] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5620.188572] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5620.192842] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5620.198929] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5620.203872] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5620.209003] Call Trace: [ 5620.213393] [ 5620.214737] ? show_regs.cold.9+0x22/0x2f [ 5620.216949] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5620.219318] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5620.221537] ? watchdog+0x40/0x40 [ 5620.224326] ? hrtimer_interrupt+0x128/0x2c0 [ 5620.227012] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5620.230614] ? apic_timer_interrupt+0xf/0x20 [ 5620.232496] [ 5620.233513] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5620.236046] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5620.239003] ? newidle_balance.isra.93+0x207/0x580 [ 5620.241383] do_raw_spin_lock+0x177/0x190 [ 5620.245186] _raw_spin_lock+0x21/0x30 [ 5620.248622] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5620.255025] process_one_work+0x2c8/0x700 [ 5620.258164] worker_thread+0x296/0x6e0 [ 5620.260106] ? rescuer_thread+0x570/0x570 [ 5620.266188] kthread+0x1d1/0x200 [ 5620.268366] ? set_kthread_struct+0x70/0x70 [ 5620.270841] ret_from_fork+0x1f/0x30 [ 5622.756024] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 2985s! [ 5622.758444] Showing busy workqueues and worker pools: [ 5622.761186] workqueue events: flags=0x0 [ 5622.763572] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5622.766735] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5622.777782] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5622.786012] pending: cpu_latency_work [ptlrpc] [ 5622.788562] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5622.793955] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5622.800077] workqueue events_long: flags=0x0 [ 5622.812342] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5622.827798] pending: lu_objects_flush [obdclass] [ 5622.840189] workqueue events_unbound: flags=0x2 [ 5622.841874] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5622.847884] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5622.851385] workqueue events_power_efficient: flags=0x80 [ 5622.853038] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5622.856671] pending: check_lifetime, do_cache_clean [ 5622.859055] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5622.868238] pending: neigh_periodic_work, neigh_periodic_work [ 5622.873660] workqueue mm_percpu_wq: flags=0x8 [ 5622.876077] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5622.880660] pending: vmstat_update, lru_add_drain_per_cpu [ 5622.883292] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5622.885727] pending: vmstat_update, lru_add_drain_per_cpu [ 5622.887970] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5622.891481] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5622.894427] workqueue xprtiod: flags=0xa [ 5622.897363] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5622.900466] pending: xprt_autoclose, xs_error_handle [ 5622.903534] workqueue ipv6_addrconf: flags=0x40008 [ 5622.905956] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5622.910009] pending: addrconf_verify_work [ 5622.918050] workqueue knbd0-recv: flags=0x1a [ 5622.927487] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5622.932979] in-flight: 651:recv_work [ 5622.940482] workqueue ptlrpc_pinger: flags=0x2 [ 5622.942763] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5622.950590] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5622.954797] pool 8: cpus=0-3 flags=0x4 nice=0 hung=2985s workers=4 idle: 15979 31824 8 [ 5622.959442] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5613s workers=2 idle: 654 [ 5628.055041] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 5628.057917] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 5628.058666] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5628.058680] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5628.058757] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5628.058762] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5628.058765] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5628.058777] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5628.058781] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5628.058785] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5628.058787] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5628.058789] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5628.058791] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5628.058793] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5628.058797] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5628.058800] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5628.058802] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5628.058811] Call Trace: [ 5628.058817] [ 5628.058823] ? show_regs.cold.9+0x22/0x2f [ 5628.058832] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5628.058839] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5628.058843] ? watchdog+0x40/0x40 [ 5628.058847] ? hrtimer_interrupt+0x128/0x2c0 [ 5628.058850] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5628.058856] ? apic_timer_interrupt+0xf/0x20 [ 5628.058861] [ 5628.058864] ? native_safe_halt+0x2e/0x50 [ 5628.058868] kvm_wait+0xcd/0xe0 [ 5628.058875] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5628.058885] do_raw_spin_lock+0x177/0x190 [ 5628.058890] _raw_spin_lock+0x21/0x30 [ 5628.058894] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5628.059183] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5628.059512] Modules linked in: [ 5628.059811] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5628.060052] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5628.060271] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5628.061844] ? woken_wake_function+0x30/0x30 [ 5628.061854] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5628.062090] kthread+0x1d1/0x200 [ 5628.062097] ? set_kthread_struct+0x70/0x70 [ 5628.062100] ret_from_fork+0x1f/0x30 [ 5628.069813] Modules linked in: lustre(O) [ 5628.076192] lustre(O) [ 5628.107394] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5628.108363] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5628.108370] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5628.108373] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5628.108385] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5628.108390] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5628.108393] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5628.108396] RDX: 00000000000053ae RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5628.108398] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5628.108400] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5628.108402] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5628.108412] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5628.108415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5628.108417] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5628.108424] Call Trace: [ 5628.108429] [ 5628.108439] ? show_regs.cold.9+0x22/0x2f [ 5628.108455] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5628.108461] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5628.108465] ? watchdog+0x40/0x40 [ 5628.108470] ? hrtimer_interrupt+0x128/0x2c0 [ 5628.108473] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5628.108482] ? apic_timer_interrupt+0xf/0x20 [ 5628.108487] [ 5628.108489] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5628.108494] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5628.108499] do_raw_spin_lock+0x177/0x190 [ 5628.108503] _raw_spin_lock+0x21/0x30 [ 5628.108509] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5628.108787] obd_mod_free+0x46/0x140 [ptlrpc] [ 5628.109128] mdc_commit_open+0x91/0xb0 [mdc] [ 5628.109163] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5628.109356] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5628.109611] after_reply+0xe2c/0x1780 [ptlrpc] [ 5628.109827] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5628.110159] ? __next_timer_interrupt+0x160/0x160 [ 5628.110170] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5628.110383] ? do_wait_intr+0xf0/0xf0 [ 5628.110392] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5628.110667] kthread+0x1d1/0x200 [ 5628.110673] ? set_kthread_struct+0x70/0x70 [ 5628.110676] ret_from_fork+0x1f/0x30 [ 5628.559189] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5628.592577] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5628.602552] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5628.613331] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5628.615603] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5628.634271] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5628.637656] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5628.642655] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5628.654232] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5628.664189] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5628.677447] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5628.687360] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5628.695905] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5628.699487] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5628.705638] Call Trace: [ 5628.706477] [ 5628.707179] ? show_regs.cold.9+0x22/0x2f [ 5628.708421] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5628.714392] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5628.718176] ? watchdog+0x40/0x40 [ 5628.721901] ? hrtimer_interrupt+0x128/0x2c0 [ 5628.726404] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5628.729230] ? apic_timer_interrupt+0xf/0x20 [ 5628.731266] [ 5628.732314] ? native_safe_halt+0x2e/0x50 [ 5628.735465] kvm_wait+0xcd/0xe0 [ 5628.737345] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5628.740856] do_raw_spin_lock+0x177/0x190 [ 5628.742829] _raw_spin_lock+0x21/0x30 [ 5628.745347] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5628.749763] ? do_raw_spin_unlock+0x75/0x190 [ 5628.751748] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5628.754550] ? do_wait_intr+0xf0/0xf0 [ 5628.759159] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5628.763324] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5628.766049] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5628.769776] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5628.774402] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5628.783283] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5628.784793] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5628.796227] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5628.799547] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5628.801791] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5628.809946] lmv_intent_open+0x338/0x1150 [lmv] [ 5628.819413] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5628.826450] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5628.832833] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5628.835949] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5628.839930] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5628.843152] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5628.850424] ? do_raw_spin_unlock+0x75/0x190 [ 5628.852507] ? _raw_spin_unlock+0x12/0x30 [ 5628.855701] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5628.862822] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5628.867903] ll_atomic_open+0x36a/0xd70 [lustre] [ 5628.871885] lookup_open+0x356/0xb40 [ 5628.877171] do_last+0x313/0xfc0 [ 5628.878820] ? nd_jump_root+0xe5/0x160 [ 5628.884529] ? path_init+0x437/0x520 [ 5628.886561] path_openat+0xf7/0x500 [ 5628.890214] ? do_raw_spin_unlock+0x75/0x190 [ 5628.894082] do_filp_open+0x99/0x140 [ 5628.896495] ? getname_flags+0x6e/0x330 [ 5628.897833] ? __check_object_size+0xff/0x256 [ 5628.899391] ? do_raw_spin_unlock+0x75/0x190 [ 5628.907343] ? _raw_spin_unlock+0x12/0x30 [ 5628.913944] do_sys_openat2+0x2b4/0x410 [ 5628.916855] do_sys_open+0x73/0xa0 [ 5628.919534] __x64_sys_openat+0x24/0x30 [ 5628.921550] do_syscall_64+0xc1/0x440 [ 5628.922943] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5628.929468] RIP: 0033:0x7f53425a5332 [ 5628.931366] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5628.949735] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5628.954264] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5628.957926] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5628.962250] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5628.966548] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5628.971265] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5648.052687] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 5648.056532] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5648.080715] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5648.099243] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5648.111592] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5648.122914] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5648.136026] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5648.153227] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5648.159704] RAX: 0000000000005429 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5648.162208] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5648.168043] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5648.171094] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5648.177825] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5648.187225] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5648.197492] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5648.203221] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5648.212306] Call Trace: [ 5648.215932] [ 5648.217053] ? show_regs.cold.9+0x22/0x2f [ 5648.219504] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5648.221406] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5648.222543] ? watchdog+0x40/0x40 [ 5648.223409] ? hrtimer_interrupt+0x128/0x2c0 [ 5648.224904] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5648.226631] ? apic_timer_interrupt+0xf/0x20 [ 5648.228364] [ 5648.229123] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5648.231226] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5648.233296] ? newidle_balance.isra.93+0x207/0x580 [ 5648.234895] do_raw_spin_lock+0x177/0x190 [ 5648.236410] _raw_spin_lock+0x21/0x30 [ 5648.237762] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5648.239872] process_one_work+0x2c8/0x700 [ 5648.241563] worker_thread+0x296/0x6e0 [ 5648.242939] ? rescuer_thread+0x570/0x570 [ 5648.244583] kthread+0x1d1/0x200 [ 5648.245918] ? set_kthread_struct+0x70/0x70 [ 5648.247482] ret_from_fork+0x1f/0x30 [ 5653.472025] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3016s! [ 5653.476679] Showing busy workqueues and worker pools: [ 5653.479219] workqueue events: flags=0x0 [ 5653.481479] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5653.495209] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5653.509667] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5653.524279] pending: cpu_latency_work [ptlrpc] [ 5653.538038] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5653.540242] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5653.560499] workqueue events_long: flags=0x0 [ 5653.564065] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5653.572560] pending: lu_objects_flush [obdclass] [ 5653.577667] workqueue events_unbound: flags=0x2 [ 5653.580726] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5653.587761] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5653.593852] workqueue events_power_efficient: flags=0x80 [ 5653.602908] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5653.608430] pending: check_lifetime, do_cache_clean [ 5653.612257] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5653.617797] pending: neigh_periodic_work, neigh_periodic_work [ 5653.626417] workqueue mm_percpu_wq: flags=0x8 [ 5653.629708] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5653.637843] pending: vmstat_update, lru_add_drain_per_cpu [ 5653.641772] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5653.648267] pending: vmstat_update, lru_add_drain_per_cpu [ 5653.660384] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5653.671322] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5653.674479] workqueue xprtiod: flags=0xa [ 5653.678165] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5653.685339] pending: xprt_autoclose, xs_error_handle [ 5653.692135] workqueue ipv6_addrconf: flags=0x40008 [ 5653.700622] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5653.707599] pending: addrconf_verify_work [ 5653.709181] workqueue knbd0-recv: flags=0x1a [ 5653.718831] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5653.724805] in-flight: 651:recv_work [ 5653.730904] workqueue ptlrpc_pinger: flags=0x2 [ 5653.734906] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5653.738825] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5653.746351] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3016s workers=4 idle: 15979 31824 8 [ 5653.751821] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5644s workers=2 idle: 654 [ 5656.060024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5656.063164] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 5656.063161] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 5656.063180] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5656.063344] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5656.063349] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5656.063352] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5656.063370] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5656.063375] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5656.063380] RAX: 00000000000075e5 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5656.063382] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5656.063384] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5656.063387] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5656.063389] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5656.063392] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5656.063395] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5656.063398] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5656.063406] Call Trace: [ 5656.063413] [ 5656.063418] ? show_regs.cold.9+0x22/0x2f [ 5656.063429] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5656.063435] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5656.063439] ? watchdog+0x40/0x40 [ 5656.063444] ? hrtimer_interrupt+0x128/0x2c0 [ 5656.063448] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5656.063459] ? apic_timer_interrupt+0xf/0x20 [ 5656.063465] [ 5656.063468] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5656.063475] do_raw_spin_lock+0x177/0x190 [ 5656.063481] _raw_spin_lock+0x21/0x30 [ 5656.063487] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5656.063777] ? do_raw_spin_unlock+0x75/0x190 [ 5656.063783] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5656.064158] ? do_wait_intr+0xf0/0xf0 [ 5656.064168] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5656.064385] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5656.064631] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5656.065568] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5656.065683] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5656.065720] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5656.065750] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5656.065779] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5656.065863] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5656.066424] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5656.066458] lmv_intent_open+0x338/0x1150 [lmv] [ 5656.066490] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5656.066601] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5656.066686] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5656.066709] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5656.066800] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5656.067164] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5656.067266] ? do_raw_spin_unlock+0x75/0x190 [ 5656.067274] ? _raw_spin_unlock+0x12/0x30 [ 5656.067281] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5656.067312] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5656.067335] ll_atomic_open+0x36a/0xd70 [lustre] [ 5656.067423] lookup_open+0x356/0xb40 [ 5656.067432] do_last+0x313/0xfc0 [ 5656.067436] ? nd_jump_root+0xe5/0x160 [ 5656.067441] ? path_init+0x437/0x520 [ 5656.067446] path_openat+0xf7/0x500 [ 5656.067450] ? do_raw_spin_unlock+0x75/0x190 [ 5656.067454] do_filp_open+0x99/0x140 [ 5656.067461] ? getname_flags+0x6e/0x330 [ 5656.067465] ? __check_object_size+0xff/0x256 [ 5656.067473] ? do_raw_spin_unlock+0x75/0x190 [ 5656.067477] ? _raw_spin_unlock+0x12/0x30 [ 5656.067482] do_sys_openat2+0x2b4/0x410 [ 5656.067487] do_sys_open+0x73/0xa0 [ 5656.067491] __x64_sys_openat+0x24/0x30 [ 5656.067494] do_syscall_64+0xc1/0x440 [ 5656.068086] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5656.068097] RIP: 0033:0x7f53425a5332 [ 5656.068104] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5656.068108] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5656.068113] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5656.068116] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5656.068119] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5656.068121] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5656.068123] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5656.070180] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5656.070255] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5656.070263] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5656.070267] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5656.070282] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5656.070287] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5656.070291] RAX: 0000000000007662 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5656.070295] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5656.070297] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5656.070299] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5656.070301] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5656.070305] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5656.070308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5656.070310] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5656.070319] Call Trace: [ 5656.070324] [ 5656.070329] ? show_regs.cold.9+0x22/0x2f [ 5656.070339] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5656.070346] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5656.070351] ? watchdog+0x40/0x40 [ 5656.070355] ? hrtimer_interrupt+0x128/0x2c0 [ 5656.070359] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5656.070367] ? apic_timer_interrupt+0xf/0x20 [ 5656.070372] [ 5656.070375] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5656.070379] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5656.070386] do_raw_spin_lock+0x177/0x190 [ 5656.070391] _raw_spin_lock+0x21/0x30 [ 5656.070396] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5656.070680] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5656.070885] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5656.071100] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5656.081450] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5656.083080] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5656.083089] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5656.083092] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5656.083105] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5656.083109] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5656.083113] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5656.083115] RDX: 0000000000000c66 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5656.083117] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5656.083119] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5656.083122] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5656.083124] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5656.083127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5656.083129] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5656.083138] Call Trace: [ 5656.083143] [ 5656.083148] ? show_regs.cold.9+0x22/0x2f [ 5656.083156] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5656.083162] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5656.083166] ? watchdog+0x40/0x40 [ 5656.083170] ? hrtimer_interrupt+0x128/0x2c0 [ 5656.083174] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5656.083182] ? apic_timer_interrupt+0xf/0x20 [ 5656.083187] [ 5656.083190] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5656.083194] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5656.083200] do_raw_spin_lock+0x177/0x190 [ 5656.083204] _raw_spin_lock+0x21/0x30 [ 5656.083211] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5656.083467] obd_mod_free+0x46/0x140 [ptlrpc] [ 5656.084926] mdc_commit_open+0x91/0xb0 [mdc] [ 5656.084977] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5656.085175] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5656.085382] after_reply+0xe2c/0x1780 [ptlrpc] [ 5656.085596] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5656.085799] ? __next_timer_interrupt+0x160/0x160 [ 5656.085811] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5656.086009] ? do_wait_intr+0xf0/0xf0 [ 5656.086017] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5656.086220] kthread+0x1d1/0x200 [ 5656.086225] ? set_kthread_struct+0x70/0x70 [ 5656.086229] ret_from_fork+0x1f/0x30 [ 5656.703353] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5656.706006] ? woken_wake_function+0x30/0x30 [ 5656.707629] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5656.710442] kthread+0x1d1/0x200 [ 5656.711926] ? set_kthread_struct+0x70/0x70 [ 5656.713551] ret_from_fork+0x1f/0x30 [ 5680.053527] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 5680.057718] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5680.095873] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5680.107128] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5680.111692] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5680.117991] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5680.124821] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5680.140956] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5680.144938] RAX: 0000000000007f06 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5680.149506] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5680.152016] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5680.159073] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5680.167911] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5680.176101] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5680.181029] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5680.183895] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5680.187226] Call Trace: [ 5680.188307] [ 5680.189692] ? show_regs.cold.9+0x22/0x2f [ 5680.191956] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5680.195672] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5680.201577] ? watchdog+0x40/0x40 [ 5680.202738] ? hrtimer_interrupt+0x128/0x2c0 [ 5680.205704] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5680.208667] ? apic_timer_interrupt+0xf/0x20 [ 5680.211293] [ 5680.212035] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5680.218484] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5680.220351] ? newidle_balance.isra.93+0x207/0x580 [ 5680.221969] do_raw_spin_lock+0x177/0x190 [ 5680.224910] _raw_spin_lock+0x21/0x30 [ 5680.233799] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5680.237237] process_one_work+0x2c8/0x700 [ 5680.241309] worker_thread+0x296/0x6e0 [ 5680.244793] ? rescuer_thread+0x570/0x570 [ 5680.249875] kthread+0x1d1/0x200 [ 5680.254314] ? set_kthread_struct+0x70/0x70 [ 5680.257249] ret_from_fork+0x1f/0x30 [ 5684.055162] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 5684.058103] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 5684.058517] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5684.058523] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5684.058594] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5684.058600] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5684.058603] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5684.058617] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5684.058621] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5684.058626] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5684.058628] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5684.058631] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5684.058633] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5684.058636] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5684.058639] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5684.058642] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5684.058644] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5684.058654] Call Trace: [ 5684.058659] [ 5684.058665] ? show_regs.cold.9+0x22/0x2f [ 5684.058676] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5684.058683] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5684.058687] ? watchdog+0x40/0x40 [ 5684.058691] ? hrtimer_interrupt+0x128/0x2c0 [ 5684.058696] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5684.058704] ? apic_timer_interrupt+0xf/0x20 [ 5684.058709] [ 5684.058712] ? native_safe_halt+0x2e/0x50 [ 5684.058734] kvm_wait+0xcd/0xe0 [ 5684.058742] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5684.058770] do_raw_spin_lock+0x177/0x190 [ 5684.058775] _raw_spin_lock+0x21/0x30 [ 5684.058779] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5684.059095] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5684.059719] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5684.059949] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5684.060806] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5684.061099] ? woken_wake_function+0x30/0x30 [ 5684.061110] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5684.061409] kthread+0x1d1/0x200 [ 5684.061415] ? set_kthread_struct+0x70/0x70 [ 5684.061419] ret_from_fork+0x1f/0x30 [ 5684.062962] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5684.063107] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5684.063114] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5684.063117] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5684.063129] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5684.063133] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5684.063137] RAX: 0000000000004017 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5684.063139] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5684.063141] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5684.063143] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5684.063152] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5684.063155] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5684.063158] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5684.063160] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5684.063168] Call Trace: [ 5684.063173] [ 5684.063177] ? show_regs.cold.9+0x22/0x2f [ 5684.063186] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5684.063193] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5684.063197] ? watchdog+0x40/0x40 [ 5684.063201] ? hrtimer_interrupt+0x128/0x2c0 [ 5684.063206] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5684.063352] ? apic_timer_interrupt+0xf/0x20 [ 5684.063359] [ 5684.063361] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5684.063368] do_raw_spin_lock+0x177/0x190 [ 5684.063373] _raw_spin_lock+0x21/0x30 [ 5684.063377] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5684.063654] ? do_raw_spin_unlock+0x75/0x190 [ 5684.063660] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5684.063931] ? do_wait_intr+0xf0/0xf0 [ 5684.063940] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5684.064211] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5684.064533] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5684.064759] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5684.064872] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5684.064910] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5684.064939] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5684.064965] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5684.065127] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5684.065434] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5684.065464] lmv_intent_open+0x338/0x1150 [lmv] [ 5684.065491] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5684.065576] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5684.065659] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5684.065683] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5684.065779] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5684.066165] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5684.066329] ? do_raw_spin_unlock+0x75/0x190 [ 5684.066335] ? _raw_spin_unlock+0x12/0x30 [ 5684.066343] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5684.066374] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5684.066393] ll_atomic_open+0x36a/0xd70 [lustre] [ 5684.066481] lookup_open+0x356/0xb40 [ 5684.066490] do_last+0x313/0xfc0 [ 5684.066495] ? nd_jump_root+0xe5/0x160 [ 5684.066501] ? path_init+0x437/0x520 [ 5684.066505] path_openat+0xf7/0x500 [ 5684.066509] ? do_raw_spin_unlock+0x75/0x190 [ 5684.066514] do_filp_open+0x99/0x140 [ 5684.066520] ? getname_flags+0x6e/0x330 [ 5684.066524] ? __check_object_size+0xff/0x256 [ 5684.066532] ? do_raw_spin_unlock+0x75/0x190 [ 5684.066536] ? _raw_spin_unlock+0x12/0x30 [ 5684.066540] do_sys_openat2+0x2b4/0x410 [ 5684.066544] do_sys_open+0x73/0xa0 [ 5684.066548] __x64_sys_openat+0x24/0x30 [ 5684.066551] do_syscall_64+0xc1/0x440 [ 5684.066557] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5684.066563] RIP: 0033:0x7f53425a5332 [ 5684.066570] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5684.066574] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5684.066579] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5684.066581] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5684.066583] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5684.066585] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5684.066587] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5684.384911] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5684.402824] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5684.408422] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5684.414176] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5684.417499] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5684.429364] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5684.433597] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5684.437686] RDX: 0000000000001c18 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5684.441085] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5684.444056] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5684.447145] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5684.450363] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5684.453777] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5684.456739] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5684.459790] Call Trace: [ 5684.461187] [ 5684.462193] ? show_regs.cold.9+0x22/0x2f [ 5684.463633] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5684.465775] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5684.467603] ? watchdog+0x40/0x40 [ 5684.469188] ? hrtimer_interrupt+0x128/0x2c0 [ 5684.471255] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5684.472847] ? apic_timer_interrupt+0xf/0x20 [ 5684.474901] [ 5684.475608] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5684.477878] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5684.480133] do_raw_spin_lock+0x177/0x190 [ 5684.481612] _raw_spin_lock+0x21/0x30 [ 5684.483255] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5684.485256] obd_mod_free+0x46/0x140 [ptlrpc] [ 5684.486763] mdc_commit_open+0x91/0xb0 [mdc] [ 5684.488788] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5684.491720] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5684.494819] after_reply+0xe2c/0x1780 [ptlrpc] [ 5684.498281] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5684.502246] ? __next_timer_interrupt+0x160/0x160 [ 5684.504664] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5684.508138] ? do_wait_intr+0xf0/0xf0 [ 5684.509809] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5684.513172] kthread+0x1d1/0x200 [ 5684.515414] ? set_kthread_struct+0x70/0x70 [ 5684.517934] ret_from_fork+0x1f/0x30 [ 5684.521051] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3047s! [ 5684.524712] Showing busy workqueues and worker pools: [ 5684.527256] workqueue events: flags=0x0 [ 5684.529095] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5684.532600] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5684.536431] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5684.543996] pending: cpu_latency_work [ptlrpc] [ 5684.550040] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5684.555862] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5684.568080] workqueue events_long: flags=0x0 [ 5684.572873] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5684.581375] pending: lu_objects_flush [obdclass] [ 5684.585302] workqueue events_unbound: flags=0x2 [ 5684.588992] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5684.596613] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5684.599261] workqueue events_power_efficient: flags=0x80 [ 5684.601567] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5684.603746] pending: check_lifetime, do_cache_clean [ 5684.605901] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5684.608827] pending: neigh_periodic_work, neigh_periodic_work [ 5684.615886] workqueue mm_percpu_wq: flags=0x8 [ 5684.617756] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5684.627816] pending: vmstat_update, lru_add_drain_per_cpu [ 5684.633752] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5684.642772] pending: vmstat_update, lru_add_drain_per_cpu [ 5684.651681] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5684.658745] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5684.662597] workqueue xprtiod: flags=0xa [ 5684.667749] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5684.669834] pending: xprt_autoclose, xs_error_handle [ 5684.678648] workqueue ipv6_addrconf: flags=0x40008 [ 5684.684718] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5684.694872] pending: addrconf_verify_work [ 5684.697271] workqueue knbd0-recv: flags=0x1a [ 5684.703866] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5684.709042] in-flight: 651:recv_work [ 5684.711070] workqueue ptlrpc_pinger: flags=0x2 [ 5684.715817] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5684.720077] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5684.726580] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3047s workers=4 idle: 15979 31824 8 [ 5684.730900] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5675s workers=2 idle: 654 [ 5712.052028] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 5712.055033] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 5712.057491] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [ 5712.058438] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5712.058446] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5712.058622] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5712.058627] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5712.058630] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5712.058676] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5712.058681] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5712.058686] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5712.058688] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5712.058690] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5712.058692] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5712.058695] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5712.058698] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5712.058701] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5712.058704] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5712.058713] Call Trace: [ 5712.058856] [ 5712.058864] ? show_regs.cold.9+0x22/0x2f [ 5712.058875] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5712.058881] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5712.058886] ? watchdog+0x40/0x40 [ 5712.058891] ? hrtimer_interrupt+0x128/0x2c0 [ 5712.058896] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5712.058902] ? apic_timer_interrupt+0xf/0x20 [ 5712.058907] [ 5712.058910] ? native_safe_halt+0x2e/0x50 [ 5712.058914] kvm_wait+0xcd/0xe0 [ 5712.058922] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5712.058931] do_raw_spin_lock+0x177/0x190 [ 5712.058936] _raw_spin_lock+0x21/0x30 [ 5712.058940] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5712.059417] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5712.059697] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5712.060394] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5712.060607] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5712.060677] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5712.060683] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5712.060685] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5712.060695] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5712.060700] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5712.060704] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5712.060706] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5712.060708] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5712.060710] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5712.060713] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5712.060716] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5712.060719] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5712.060721] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5712.060730] Call Trace: [ 5712.060734] [ 5712.060737] ? show_regs.cold.9+0x22/0x2f [ 5712.060749] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5712.060755] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5712.060759] ? watchdog+0x40/0x40 [ 5712.060764] ? hrtimer_interrupt+0x128/0x2c0 [ 5712.060768] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5712.060775] ? apic_timer_interrupt+0xf/0x20 [ 5712.060780] [ 5712.060783] ? native_safe_halt+0x2e/0x50 [ 5712.060787] kvm_wait+0xcd/0xe0 [ 5712.060795] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5712.060803] do_raw_spin_lock+0x177/0x190 [ 5712.060807] _raw_spin_lock+0x21/0x30 [ 5712.060812] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5712.061049] ? do_raw_spin_unlock+0x75/0x190 [ 5712.061056] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5712.061280] ? do_wait_intr+0xf0/0xf0 [ 5712.061289] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5712.061498] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5712.061922] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5712.062263] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5712.062374] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5712.062415] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5712.062490] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5712.062574] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5712.062476] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5712.062849] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5712.063387] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5712.063420] lmv_intent_open+0x338/0x1150 [lmv] [ 5712.063451] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5712.063552] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5712.063636] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5712.063660] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5712.064867] ? woken_wake_function+0x30/0x30 [ 5712.064878] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5712.068022] kthread+0x1d1/0x200 [ 5712.068024] obdecho(O) [ 5712.068032] mgc(O) mdc(O) [ 5712.068036] ? set_kthread_struct+0x70/0x70 [ 5712.068039] lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 [ 5712.068048] ret_from_fork+0x1f/0x30 [ 5712.068067] dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5712.068110] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5712.068116] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5712.068122] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5712.068363] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5712.068371] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5712.068375] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 [ 5712.068392] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 5712.068400] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5712.068458] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5712.068463] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5712.068464] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5712.068473] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5712.068477] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5712.068481] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5712.068484] RDX: 0000000000006f78 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5712.068486] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5712.068489] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5712.068491] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5712.068494] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5712.068496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5712.068499] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5712.068507] Call Trace: [ 5712.068511] [ 5712.068513] ? show_regs.cold.9+0x22/0x2f [ 5712.068533] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5712.068540] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5712.068544] ? watchdog+0x40/0x40 [ 5712.068548] ? hrtimer_interrupt+0x128/0x2c0 [ 5712.068993] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5712.069014] ? apic_timer_interrupt+0xf/0x20 [ 5712.069019] [ 5712.069022] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5712.069028] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5712.069032] do_raw_spin_lock+0x177/0x190 [ 5712.069037] _raw_spin_lock+0x21/0x30 [ 5712.069043] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5712.069665] obd_mod_free+0x46/0x140 [ptlrpc] [ 5712.069944] mdc_commit_open+0x91/0xb0 [mdc] [ 5712.071305] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5712.071428] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5712.071513] ? do_raw_spin_unlock+0x75/0x190 [ 5712.071536] ? _raw_spin_unlock+0x12/0x30 [ 5712.071542] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5712.071568] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5712.071588] ll_atomic_open+0x36a/0xd70 [lustre] [ 5712.069982] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5712.071958] lookup_open+0x356/0xb40 [ 5712.071967] do_last+0x313/0xfc0 [ 5712.071972] ? nd_jump_root+0xe5/0x160 [ 5712.071976] ? path_init+0x437/0x520 [ 5712.071981] path_openat+0xf7/0x500 [ 5712.071985] ? do_raw_spin_unlock+0x75/0x190 [ 5712.071990] do_filp_open+0x99/0x140 [ 5712.071997] ? getname_flags+0x6e/0x330 [ 5712.072001] ? __check_object_size+0xff/0x256 [ 5712.072008] ? do_raw_spin_unlock+0x75/0x190 [ 5712.072014] ? _raw_spin_unlock+0x12/0x30 [ 5712.072022] do_sys_openat2+0x2b4/0x410 [ 5712.072027] do_sys_open+0x73/0xa0 [ 5712.072032] __x64_sys_openat+0x24/0x30 [ 5712.072035] do_syscall_64+0xc1/0x440 [ 5712.072042] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5712.072048] RIP: 0033:0x7f53425a5332 [ 5712.072056] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5712.072059] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5712.072065] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5712.072067] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5712.072070] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5712.072071] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5712.072074] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5712.072020] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5712.073661] after_reply+0xe2c/0x1780 [ptlrpc] [ 5712.075458] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5712.076176] ? __next_timer_interrupt+0x160/0x160 [ 5712.076203] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5712.076896] ? do_wait_intr+0xf0/0xf0 [ 5712.076906] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5712.077112] kthread+0x1d1/0x200 [ 5712.077118] ? set_kthread_struct+0x70/0x70 [ 5712.077121] ret_from_fork+0x1f/0x30 [ 5712.879433] ORIG_RAX: ffffffffffffff13 [ 5712.880853] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5712.883449] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5712.887709] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5712.892216] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5712.896948] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5712.900267] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5712.904210] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5712.908800] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5712.912077] Call Trace: [ 5712.914600] [ 5712.917094] ? show_regs.cold.9+0x22/0x2f [ 5712.921206] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5712.924087] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5712.926915] ? watchdog+0x40/0x40 [ 5712.930499] ? hrtimer_interrupt+0x128/0x2c0 [ 5712.933292] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5712.939029] ? apic_timer_interrupt+0xf/0x20 [ 5712.943683] [ 5712.947108] ? native_safe_halt+0x2e/0x50 [ 5712.949102] kvm_wait+0xcd/0xe0 [ 5712.951500] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5712.960577] ? newidle_balance.isra.93+0x207/0x580 [ 5712.963182] do_raw_spin_lock+0x177/0x190 [ 5712.966032] _raw_spin_lock+0x21/0x30 [ 5712.969190] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5712.975753] process_one_work+0x2c8/0x700 [ 5712.978975] worker_thread+0x296/0x6e0 [ 5712.982151] ? rescuer_thread+0x570/0x570 [ 5712.987442] kthread+0x1d1/0x200 [ 5712.988642] ? set_kthread_struct+0x70/0x70 [ 5712.993733] ret_from_fork+0x1f/0x30 [ 5714.912082] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3077s! [ 5714.919339] Showing busy workqueues and worker pools: [ 5714.921794] workqueue events: flags=0x0 [ 5714.926949] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5714.930792] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5714.940099] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5714.946870] pending: cpu_latency_work [ptlrpc] [ 5714.949311] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5714.959961] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5714.968281] workqueue events_long: flags=0x0 [ 5714.974749] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5714.981184] pending: lu_objects_flush [obdclass] [ 5714.994053] workqueue events_unbound: flags=0x2 [ 5714.995733] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5715.015121] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5715.020676] workqueue events_power_efficient: flags=0x80 [ 5715.026426] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5715.040931] pending: check_lifetime, do_cache_clean [ 5715.042822] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5715.048143] pending: neigh_periodic_work, neigh_periodic_work [ 5715.051490] workqueue mm_percpu_wq: flags=0x8 [ 5715.052970] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5715.057750] pending: vmstat_update, lru_add_drain_per_cpu [ 5715.060724] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5715.064044] pending: vmstat_update, lru_add_drain_per_cpu [ 5715.066736] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5715.069848] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5715.072901] workqueue xprtiod: flags=0xa [ 5715.074537] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5715.076929] pending: xprt_autoclose, xs_error_handle [ 5715.078884] workqueue ipv6_addrconf: flags=0x40008 [ 5715.080539] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5715.085881] pending: addrconf_verify_work [ 5715.087343] workqueue knbd0-recv: flags=0x1a [ 5715.088686] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5715.091629] in-flight: 651:recv_work [ 5715.093406] workqueue ptlrpc_pinger: flags=0x2 [ 5715.096285] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5715.101557] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5715.104756] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3078s workers=4 idle: 15979 31824 8 [ 5715.108684] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5705s workers=2 idle: 654 [ 5740.054565] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 5740.057243] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5740.058025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5740.058780] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 5740.058787] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5740.059657] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5740.059665] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5740.059668] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5740.059788] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5740.059792] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5740.059797] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5740.059799] RDX: 0000000000003062 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5740.059802] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5740.059804] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5740.059820] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5740.059824] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5740.059827] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5740.059830] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5740.059838] Call Trace: [ 5740.059843] [ 5740.059850] ? show_regs.cold.9+0x22/0x2f [ 5740.059859] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5740.059865] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5740.059870] ? watchdog+0x40/0x40 [ 5740.060245] ? hrtimer_interrupt+0x128/0x2c0 [ 5740.060249] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5740.060258] ? apic_timer_interrupt+0xf/0x20 [ 5740.060263] [ 5740.060266] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5740.060272] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5740.060277] do_raw_spin_lock+0x177/0x190 [ 5740.060282] _raw_spin_lock+0x21/0x30 [ 5740.060289] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5740.060472] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 5740.060479] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5740.060610] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5740.060615] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5740.060618] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5740.060627] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5740.060631] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5740.060635] RAX: 0000000000005b74 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5740.060637] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5740.060640] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5740.060642] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5740.060644] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5740.060646] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5740.060649] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5740.060651] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5740.060659] Call Trace: [ 5740.060662] [ 5740.060664] ? show_regs.cold.9+0x22/0x2f [ 5740.060673] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5740.060679] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5740.060576] obd_mod_free+0x46/0x140 [ptlrpc] [ 5740.060683] ? watchdog+0x40/0x40 [ 5740.060690] ? hrtimer_interrupt+0x128/0x2c0 [ 5740.060694] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5740.060699] ? apic_timer_interrupt+0xf/0x20 [ 5740.060704] [ 5740.060707] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5740.060713] do_raw_spin_lock+0x177/0x190 [ 5740.060718] _raw_spin_lock+0x21/0x30 [ 5740.060722] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5740.061073] ? do_raw_spin_unlock+0x75/0x190 [ 5740.061079] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5740.061597] ? do_wait_intr+0xf0/0xf0 [ 5740.061607] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5740.061866] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5740.062233] mdc_commit_open+0x91/0xb0 [mdc] [ 5740.062271] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5740.062470] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5740.063076] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5740.063596] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5740.063708] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5740.063760] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5740.063788] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5740.063814] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5740.063893] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5740.064133] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5740.062899] after_reply+0xe2c/0x1780 [ptlrpc] [ 5740.065399] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5740.064178] lmv_intent_open+0x338/0x1150 [lmv] [ 5740.065776] ? __next_timer_interrupt+0x160/0x160 [ 5740.065609] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5740.066077] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5740.066160] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5740.066185] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5740.066337] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5740.066489] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5740.066571] ? do_raw_spin_unlock+0x75/0x190 [ 5740.066579] ? _raw_spin_unlock+0x12/0x30 [ 5740.066587] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5740.066612] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5740.066629] ll_atomic_open+0x36a/0xd70 [lustre] [ 5740.066712] lookup_open+0x356/0xb40 [ 5740.066721] do_last+0x313/0xfc0 [ 5740.066726] ? nd_jump_root+0xe5/0x160 [ 5740.066730] ? path_init+0x437/0x520 [ 5740.066735] path_openat+0xf7/0x500 [ 5740.066740] ? do_raw_spin_unlock+0x75/0x190 [ 5740.066745] do_filp_open+0x99/0x140 [ 5740.066820] ? getname_flags+0x6e/0x330 [ 5740.066825] ? __check_object_size+0xff/0x256 [ 5740.066833] ? do_raw_spin_unlock+0x75/0x190 [ 5740.066837] ? _raw_spin_unlock+0x12/0x30 [ 5740.066842] do_sys_openat2+0x2b4/0x410 [ 5740.066846] do_sys_open+0x73/0xa0 [ 5740.066851] __x64_sys_openat+0x24/0x30 [ 5740.066855] do_syscall_64+0xc1/0x440 [ 5740.066861] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5740.066867] RIP: 0033:0x7f53425a5332 [ 5740.066917] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5740.066923] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5740.066928] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5740.066931] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5740.066933] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5740.066935] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5740.067026] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5740.065787] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5740.070624] ? do_wait_intr+0xf0/0xf0 [ 5740.070634] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5740.070847] kthread+0x1d1/0x200 [ 5740.070854] ? set_kthread_struct+0x70/0x70 [ 5740.070858] ret_from_fork+0x1f/0x30 [ 5740.092641] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5740.096238] Modules linked in: [ 5740.105131] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5740.136218] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5740.136288] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5740.136295] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5740.136297] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5740.136311] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5740.136315] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5740.136319] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5740.136321] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5740.136323] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5740.136325] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5740.136327] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5740.136331] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5740.136333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5740.136336] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5740.136427] Call Trace: [ 5740.136434] [ 5740.136441] ? show_regs.cold.9+0x22/0x2f [ 5740.136450] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5740.136456] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5740.136461] ? watchdog+0x40/0x40 [ 5740.136465] ? hrtimer_interrupt+0x128/0x2c0 [ 5740.136469] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5740.136476] ? apic_timer_interrupt+0xf/0x20 [ 5740.136480] [ 5740.136483] ? native_safe_halt+0x2e/0x50 [ 5740.136487] kvm_wait+0xcd/0xe0 [ 5740.136493] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5740.136503] do_raw_spin_lock+0x177/0x190 [ 5740.136508] _raw_spin_lock+0x21/0x30 [ 5740.136512] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5740.136856] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5740.137185] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5740.137635] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5740.137909] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5740.138650] ? woken_wake_function+0x30/0x30 [ 5740.138659] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5740.138939] kthread+0x1d1/0x200 [ 5740.139014] ? set_kthread_struct+0x70/0x70 [ 5740.139017] ret_from_fork+0x1f/0x30 [ 5740.779873] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5740.783265] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5740.785787] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5740.794654] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5740.799101] RAX: 000000000000398c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5740.804246] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5740.807455] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5740.810737] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5740.814466] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5740.816665] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5740.820182] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5740.822629] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5740.826455] Call Trace: [ 5740.827908] [ 5740.829086] ? show_regs.cold.9+0x22/0x2f [ 5740.830898] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5740.833424] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5740.835474] ? watchdog+0x40/0x40 [ 5740.837194] ? hrtimer_interrupt+0x128/0x2c0 [ 5740.839411] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5740.842161] ? apic_timer_interrupt+0xf/0x20 [ 5740.844334] [ 5740.845557] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5740.849185] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5740.851460] ? newidle_balance.isra.93+0x207/0x580 [ 5740.853284] do_raw_spin_lock+0x177/0x190 [ 5740.854471] _raw_spin_lock+0x21/0x30 [ 5740.856200] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5740.857929] process_one_work+0x2c8/0x700 [ 5740.859418] worker_thread+0x296/0x6e0 [ 5740.860775] ? rescuer_thread+0x570/0x570 [ 5740.862390] kthread+0x1d1/0x200 [ 5740.863524] ? set_kthread_struct+0x70/0x70 [ 5740.865386] ret_from_fork+0x1f/0x30 [ 5745.119608] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3108s! [ 5745.126229] Showing busy workqueues and worker pools: [ 5745.130918] workqueue events: flags=0x0 [ 5745.135659] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5745.143719] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5745.165836] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5745.173079] pending: cpu_latency_work [ptlrpc] [ 5745.182452] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5745.192359] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5745.207104] workqueue events_long: flags=0x0 [ 5745.208549] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5745.210673] pending: lu_objects_flush [obdclass] [ 5745.212344] workqueue events_unbound: flags=0x2 [ 5745.213544] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5745.215398] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5745.217410] workqueue events_power_efficient: flags=0x80 [ 5745.219543] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5745.222134] pending: check_lifetime, do_cache_clean [ 5745.224244] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5745.226575] pending: neigh_periodic_work, neigh_periodic_work [ 5745.229037] workqueue mm_percpu_wq: flags=0x8 [ 5745.230665] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5745.233329] pending: vmstat_update, lru_add_drain_per_cpu [ 5745.235508] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5745.238112] pending: vmstat_update, lru_add_drain_per_cpu [ 5745.243344] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5745.247426] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5745.257596] workqueue xprtiod: flags=0xa [ 5745.265905] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5745.280269] pending: xprt_autoclose, xs_error_handle [ 5745.295520] workqueue ipv6_addrconf: flags=0x40008 [ 5745.298714] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5745.302833] pending: addrconf_verify_work [ 5745.306713] workqueue knbd0-recv: flags=0x1a [ 5745.310662] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5745.314519] in-flight: 651:recv_work [ 5745.316903] workqueue ptlrpc_pinger: flags=0x2 [ 5745.318887] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5745.324662] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5745.331234] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3108s workers=4 idle: 15979 31824 8 [ 5745.340046] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5735s workers=2 idle: 654 [ 5754.518616] rcu: INFO: rcu_sched self-detected stall on CPU [ 5754.522477] rcu: 2-....: (1949344 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 5754.529682] (t=3660092 jiffies g=274185 q=64583) [ 5754.533462] rcu: rcu_sched kthread starved for 3118458 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 5754.541738] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5754.548496] rcu: RCU grace-period kthread stack dump: [ 5754.553856] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5754.566113] Call Trace: [ 5754.569299] __schedule+0x351/0xcb0 [ 5754.575076] schedule+0xc0/0x180 [ 5754.579505] schedule_timeout+0xb4/0x190 [ 5754.580949] ? do_raw_spin_unlock+0x75/0x190 [ 5754.585157] ? __next_timer_interrupt+0x160/0x160 [ 5754.590678] ? prepare_to_swait_event+0xf3/0x140 [ 5754.597651] rcu_gp_kthread+0x8dd/0x15b0 [ 5754.600582] ? force_qs_rnp+0x2c0/0x2c0 [ 5754.603308] kthread+0x1d1/0x200 [ 5754.607267] ? set_kthread_struct+0x70/0x70 [ 5754.612205] ret_from_fork+0x1f/0x30 [ 5754.615621] rcu: Stack dump where RCU GP kthread last ran: [ 5754.627130] Sending NMI from CPU 2 to CPUs 1: [ 5754.637485] NMI backtrace for cpu 1 [ 5754.637492] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5754.637495] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5754.637496] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5754.637498] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5754.637500] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [ 5754.637502] RAX: 0000000000002493 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5754.637504] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5754.637505] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5754.637507] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5754.637508] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5754.637510] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5754.637511] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5754.637512] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5754.637514] Call Trace: [ 5754.637515] [ 5754.637516] ? show_regs.cold.9+0x22/0x2f [ 5754.637517] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 5754.637518] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 5754.637519] ? nmi_handle+0xbd/0x210 [ 5754.637520] ? default_do_nmi+0x4b/0x1b0 [ 5754.637521] ? do_nmi+0x280/0x380 [ 5754.637522] ? end_repeat_nmi+0x16/0x1f [ 5754.637523] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5754.637524] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5754.637525] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5754.637526] [ 5754.637527] do_raw_spin_lock+0x177/0x190 [ 5754.637528] _raw_spin_lock+0x21/0x30 [ 5754.637529] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5754.637530] ? do_raw_spin_unlock+0x75/0x190 [ 5754.637532] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5754.637533] ? do_wait_intr+0xf0/0xf0 [ 5754.637534] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5754.637535] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5754.637536] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5754.637537] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5754.637539] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5754.637540] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5754.637541] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5754.637542] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5754.637544] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5754.637545] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5754.637546] lmv_intent_open+0x338/0x1150 [lmv] [ 5754.637547] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5754.637548] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5754.637549] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5754.637550] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5754.637551] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5754.637552] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5754.637553] ? do_raw_spin_unlock+0x75/0x190 [ 5754.637554] ? _raw_spin_unlock+0x12/0x30 [ 5754.637555] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5754.637556] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5754.637557] ll_atomic_open+0x36a/0xd70 [lustre] [ 5754.637559] lookup_open+0x356/0xb40 [ 5754.637559] do_last+0x313/0xfc0 [ 5754.637561] ? nd_jump_root+0xe5/0x160 [ 5754.637562] ? path_init+0x437/0x520 [ 5754.637562] path_openat+0xf7/0x500 [ 5754.637563] ? do_raw_spin_unlock+0x75/0x190 [ 5754.637564] do_filp_open+0x99/0x140 [ 5754.637566] ? getname_flags+0x6e/0x330 [ 5754.637567] ? __check_object_size+0xff/0x256 [ 5754.637568] ? do_raw_spin_unlock+0x75/0x190 [ 5754.637569] ? _raw_spin_unlock+0x12/0x30 [ 5754.637570] do_sys_openat2+0x2b4/0x410 [ 5754.637571] do_sys_open+0x73/0xa0 [ 5754.637572] __x64_sys_openat+0x24/0x30 [ 5754.637573] do_syscall_64+0xc1/0x440 [ 5754.637574] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5754.637575] RIP: 0033:0x7f53425a5332 [ 5754.637576] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5754.637579] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5754.637582] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5754.637583] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5754.637585] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5754.637586] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5754.637588] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5754.641431] Sending NMI from CPU 2 to CPUs 1: [ 5754.832915] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 5754.833359] NMI backtrace for cpu 2 [ 5754.839436] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5754.844079] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5754.848763] Call Trace: [ 5754.850064] [ 5754.850596] ? dump_stack+0xbb/0x10e [ 5754.851867] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5754.853582] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5754.855552] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5754.857695] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5754.860120] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5754.867865] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5754.869357] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5754.870933] ? update_process_times+0x6d/0xc0 [ 5754.873063] ? tick_sched_handle.isra.18+0x27/0x90 [ 5754.877826] ? tick_sched_timer+0x53/0xb0 [ 5754.880536] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5754.881819] ? tick_sched_handle.isra.18+0x90/0x90 [ 5754.883416] ? hrtimer_interrupt+0x128/0x2c0 [ 5754.884468] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5754.886012] ? apic_timer_interrupt+0xf/0x20 [ 5754.887188] [ 5754.887695] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5754.889594] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5754.891037] ? do_raw_spin_lock+0x177/0x190 [ 5754.892556] ? _raw_spin_lock+0x21/0x30 [ 5754.894045] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5754.896143] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5754.898093] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5754.899883] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5754.901993] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5754.904243] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5754.906303] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5754.908437] ? __next_timer_interrupt+0x160/0x160 [ 5754.910321] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5754.912058] ? do_wait_intr+0xf0/0xf0 [ 5754.913322] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5754.915363] ? kthread+0x1d1/0x200 [ 5754.916562] ? set_kthread_struct+0x70/0x70 [ 5754.917971] ? ret_from_fork+0x1f/0x30 [ 5768.058563] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5768.062029] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5768.096295] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5768.104262] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5768.111483] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5768.115112] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5768.127304] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5768.130397] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5768.137221] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5768.141067] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5768.147172] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5768.154370] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5768.158997] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5768.165512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5768.172179] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5768.178102] Call Trace: [ 5768.180269] [ 5768.181721] ? show_regs.cold.9+0x22/0x2f [ 5768.185344] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5768.187470] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5768.192104] ? watchdog+0x40/0x40 [ 5768.194381] ? hrtimer_interrupt+0x128/0x2c0 [ 5768.196491] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5768.199913] ? apic_timer_interrupt+0xf/0x20 [ 5768.202096] [ 5768.202688] ? native_safe_halt+0x2e/0x50 [ 5768.204121] kvm_wait+0xcd/0xe0 [ 5768.208323] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5768.210420] do_raw_spin_lock+0x177/0x190 [ 5768.213766] _raw_spin_lock+0x21/0x30 [ 5768.216363] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5768.221410] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5768.226248] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5768.229380] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5768.232448] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5768.234853] ? woken_wake_function+0x30/0x30 [ 5768.237388] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5768.240608] kthread+0x1d1/0x200 [ 5768.241897] ? set_kthread_struct+0x70/0x70 [ 5768.243503] ret_from_fork+0x1f/0x30 [ 5772.053009] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 5772.062031] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5772.102344] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5772.117666] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5772.127052] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5772.134278] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5772.137974] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5772.156281] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5772.160139] RAX: 0000000000007eb6 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5772.164173] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5772.167726] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5772.174557] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5772.181126] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5772.184596] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5772.188298] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5772.190948] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5772.195428] Call Trace: [ 5772.197033] [ 5772.197724] ? show_regs.cold.9+0x22/0x2f [ 5772.203208] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5772.206070] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5772.207677] ? watchdog+0x40/0x40 [ 5772.209662] ? hrtimer_interrupt+0x128/0x2c0 [ 5772.212594] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5772.215613] ? apic_timer_interrupt+0xf/0x20 [ 5772.219659] [ 5772.221595] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5772.226322] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5772.229288] ? newidle_balance.isra.93+0x207/0x580 [ 5772.231112] do_raw_spin_lock+0x177/0x190 [ 5772.238256] _raw_spin_lock+0x21/0x30 [ 5772.239611] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5772.242287] process_one_work+0x2c8/0x700 [ 5772.243739] worker_thread+0x296/0x6e0 [ 5772.245694] ? rescuer_thread+0x570/0x570 [ 5772.246747] kthread+0x1d1/0x200 [ 5772.248520] ? set_kthread_struct+0x70/0x70 [ 5772.250210] ret_from_fork+0x1f/0x30 [ 5775.840038] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3138s! [ 5775.842089] Showing busy workqueues and worker pools: [ 5775.843283] workqueue events: flags=0x0 [ 5775.844936] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5775.849085] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5775.854967] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5775.857793] pending: cpu_latency_work [ptlrpc] [ 5775.860193] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5775.863038] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5775.866880] workqueue events_long: flags=0x0 [ 5775.868370] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5775.871196] pending: lu_objects_flush [obdclass] [ 5775.873522] workqueue events_unbound: flags=0x2 [ 5775.875265] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5775.877577] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5775.879672] workqueue events_power_efficient: flags=0x80 [ 5775.881347] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5775.883876] pending: check_lifetime, do_cache_clean [ 5775.886039] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5775.888895] pending: neigh_periodic_work, neigh_periodic_work [ 5775.891063] workqueue mm_percpu_wq: flags=0x8 [ 5775.892951] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5775.895997] pending: vmstat_update, lru_add_drain_per_cpu [ 5775.898376] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5775.901591] pending: vmstat_update, lru_add_drain_per_cpu [ 5775.904079] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5775.906724] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5775.908965] workqueue xprtiod: flags=0xa [ 5775.910170] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5775.911935] pending: xprt_autoclose, xs_error_handle [ 5775.913336] workqueue ipv6_addrconf: flags=0x40008 [ 5775.914655] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5775.917196] pending: addrconf_verify_work [ 5775.924067] workqueue knbd0-recv: flags=0x1a [ 5775.927057] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5775.933562] in-flight: 651:recv_work [ 5775.936181] workqueue ptlrpc_pinger: flags=0x2 [ 5775.940541] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5775.945879] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5775.951987] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3138s workers=4 idle: 15979 31824 8 [ 5775.958086] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5766s workers=2 idle: 654 [ 5780.058271] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 5780.062377] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 5780.064655] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5780.064729] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5780.064736] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5780.064739] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5780.064754] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5780.064759] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5780.064763] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5780.064765] RDX: 000000000000471b RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5780.064767] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5780.064770] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5780.064772] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5780.064775] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5780.064778] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5780.064780] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5780.064788] Call Trace: [ 5780.064795] [ 5780.064802] ? show_regs.cold.9+0x22/0x2f [ 5780.064812] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5780.064818] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5780.064822] ? watchdog+0x40/0x40 [ 5780.064826] ? hrtimer_interrupt+0x128/0x2c0 [ 5780.064830] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5780.064840] ? apic_timer_interrupt+0xf/0x20 [ 5780.064844] [ 5780.064847] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5780.064852] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5780.064857] do_raw_spin_lock+0x177/0x190 [ 5780.064862] _raw_spin_lock+0x21/0x30 [ 5780.068735] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5780.069753] obd_mod_free+0x46/0x140 [ptlrpc] [ 5780.077289] Modules linked in: lustre(O) [ 5780.097524] mdc_commit_open+0x91/0xb0 [mdc] [ 5780.097584] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5780.097885] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5780.098085] after_reply+0xe2c/0x1780 [ptlrpc] [ 5780.098274] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5780.098480] ? __next_timer_interrupt+0x160/0x160 [ 5780.098493] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5780.098693] ? do_wait_intr+0xf0/0xf0 [ 5780.098702] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5780.098889] kthread+0x1d1/0x200 [ 5780.098894] ? set_kthread_struct+0x70/0x70 [ 5780.098898] ret_from_fork+0x1f/0x30 [ 5780.248799] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5780.275746] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5780.278846] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5780.283423] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5780.290895] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5780.302044] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5780.305414] RAX: 0000000000007f89 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5780.308630] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5780.311763] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5780.314948] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5780.318929] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5780.322890] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5780.326892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5780.333320] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5780.342796] Call Trace: [ 5780.343801] [ 5780.347142] ? show_regs.cold.9+0x22/0x2f [ 5780.350893] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5780.357538] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5780.359027] ? watchdog+0x40/0x40 [ 5780.369809] ? hrtimer_interrupt+0x128/0x2c0 [ 5780.376025] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5780.378674] ? apic_timer_interrupt+0xf/0x20 [ 5780.381699] [ 5780.382970] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5780.387460] do_raw_spin_lock+0x177/0x190 [ 5780.389572] _raw_spin_lock+0x21/0x30 [ 5780.391298] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5780.393811] ? do_raw_spin_unlock+0x75/0x190 [ 5780.395625] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5780.398047] ? do_wait_intr+0xf0/0xf0 [ 5780.401799] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5780.404007] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5780.406688] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5780.409840] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5780.412434] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5780.414864] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5780.417076] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5780.419312] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5780.422097] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5780.425688] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5780.428181] lmv_intent_open+0x338/0x1150 [lmv] [ 5780.429795] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5780.432272] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5780.434823] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5780.437007] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5780.440103] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5780.442934] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5780.445469] ? do_raw_spin_unlock+0x75/0x190 [ 5780.447572] ? _raw_spin_unlock+0x12/0x30 [ 5780.449562] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5780.453339] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5780.455572] ll_atomic_open+0x36a/0xd70 [lustre] [ 5780.459003] lookup_open+0x356/0xb40 [ 5780.461442] do_last+0x313/0xfc0 [ 5780.463530] ? nd_jump_root+0xe5/0x160 [ 5780.468600] ? path_init+0x437/0x520 [ 5780.471237] path_openat+0xf7/0x500 [ 5780.473768] ? do_raw_spin_unlock+0x75/0x190 [ 5780.477334] do_filp_open+0x99/0x140 [ 5780.480227] ? getname_flags+0x6e/0x330 [ 5780.481978] ? __check_object_size+0xff/0x256 [ 5780.489656] ? do_raw_spin_unlock+0x75/0x190 [ 5780.493013] ? _raw_spin_unlock+0x12/0x30 [ 5780.495629] do_sys_openat2+0x2b4/0x410 [ 5780.501982] do_sys_open+0x73/0xa0 [ 5780.503503] __x64_sys_openat+0x24/0x30 [ 5780.508697] do_syscall_64+0xc1/0x440 [ 5780.512286] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5780.518541] RIP: 0033:0x7f53425a5332 [ 5780.521175] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5780.540577] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5780.542907] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5780.550533] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5780.559245] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5780.564575] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5780.569236] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5796.060992] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 5796.067257] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5796.101131] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5796.106180] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5796.110408] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5796.112860] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5796.122641] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5796.129977] RAX: 0000000000006f19 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5796.136292] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5796.142806] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5796.149645] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5796.156348] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5796.160831] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5796.165255] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5796.168271] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5796.178527] Call Trace: [ 5796.181035] [ 5796.184984] ? show_regs.cold.9+0x22/0x2f [ 5796.193033] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5796.196522] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5796.202528] ? watchdog+0x40/0x40 [ 5796.208539] ? hrtimer_interrupt+0x128/0x2c0 [ 5796.214478] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5796.221464] ? apic_timer_interrupt+0xf/0x20 [ 5796.223944] [ 5796.225138] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5796.232006] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5796.238847] do_raw_spin_lock+0x177/0x190 [ 5796.240843] _raw_spin_lock+0x21/0x30 [ 5796.243076] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5796.251942] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5796.255547] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5796.264989] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5796.275914] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5796.280176] ? woken_wake_function+0x30/0x30 [ 5796.282752] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5796.288971] kthread+0x1d1/0x200 [ 5796.290876] ? set_kthread_struct+0x70/0x70 [ 5796.293271] ret_from_fork+0x1f/0x30 [ 5800.052056] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 5800.056629] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5800.087693] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5800.098402] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5800.103774] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5800.106489] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5800.108920] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5800.125774] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5800.132928] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5800.138861] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5800.142757] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5800.146741] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5800.150373] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5800.153968] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5800.159597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5800.162426] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5800.165931] Call Trace: [ 5800.167262] [ 5800.168371] ? show_regs.cold.9+0x22/0x2f [ 5800.170787] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5800.172905] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5800.174601] ? watchdog+0x40/0x40 [ 5800.175864] ? hrtimer_interrupt+0x128/0x2c0 [ 5800.178167] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5800.180579] ? apic_timer_interrupt+0xf/0x20 [ 5800.182663] [ 5800.183764] ? native_safe_halt+0x2e/0x50 [ 5800.185692] kvm_wait+0xcd/0xe0 [ 5800.186915] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5800.190227] ? newidle_balance.isra.93+0x207/0x580 [ 5800.193253] do_raw_spin_lock+0x177/0x190 [ 5800.195261] _raw_spin_lock+0x21/0x30 [ 5800.197535] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5800.200734] process_one_work+0x2c8/0x700 [ 5800.202599] worker_thread+0x296/0x6e0 [ 5800.203918] ? rescuer_thread+0x570/0x570 [ 5800.206419] kthread+0x1d1/0x200 [ 5800.207936] ? set_kthread_struct+0x70/0x70 [ 5800.210195] ret_from_fork+0x1f/0x30 [ 5806.047081] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3168s! [ 5806.049794] Showing busy workqueues and worker pools: [ 5806.051969] workqueue events: flags=0x0 [ 5806.053266] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5806.056566] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5806.061825] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5806.065335] pending: cpu_latency_work [ptlrpc] [ 5806.069330] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5806.072603] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5806.076517] workqueue events_long: flags=0x0 [ 5806.078735] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5806.081546] pending: lu_objects_flush [obdclass] [ 5806.083534] workqueue events_unbound: flags=0x2 [ 5806.086139] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5806.089155] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5806.091667] workqueue events_power_efficient: flags=0x80 [ 5806.097534] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5806.100829] pending: check_lifetime, do_cache_clean [ 5806.103266] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5806.107005] pending: neigh_periodic_work, neigh_periodic_work [ 5806.110335] workqueue mm_percpu_wq: flags=0x8 [ 5806.112383] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5806.117310] pending: vmstat_update, lru_add_drain_per_cpu [ 5806.120248] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5806.123647] pending: vmstat_update, lru_add_drain_per_cpu [ 5806.126389] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5806.130047] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5806.132981] workqueue xprtiod: flags=0xa [ 5806.134527] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5806.137389] pending: xprt_autoclose, xs_error_handle [ 5806.139302] workqueue ipv6_addrconf: flags=0x40008 [ 5806.141473] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5806.144928] pending: addrconf_verify_work [ 5806.146396] workqueue knbd0-recv: flags=0x1a [ 5806.148476] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5806.152060] in-flight: 651:recv_work [ 5806.153744] workqueue ptlrpc_pinger: flags=0x2 [ 5806.155789] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5806.159320] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5806.162587] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3169s workers=4 idle: 15979 31824 8 [ 5806.166374] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5796s workers=2 idle: 654 [ 5808.055074] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 5808.057007] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 5808.058417] Modules linked in: [ 5808.061959] Modules linked in: [ 5808.061963] lustre(O) osp(O) [ 5808.063611] lustre(O) [ 5808.064725] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5808.064786] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5808.064793] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5808.064796] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5808.064809] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5808.064813] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5808.064817] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5808.064820] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5808.064821] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5808.064823] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5808.064825] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5808.064828] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5808.064832] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5808.064834] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5808.064852] Call Trace: [ 5808.064859] [ 5808.064864] ? show_regs.cold.9+0x22/0x2f [ 5808.064942] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5808.064951] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5808.064955] ? watchdog+0x40/0x40 [ 5808.064960] ? hrtimer_interrupt+0x128/0x2c0 [ 5808.064964] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5808.064969] ? apic_timer_interrupt+0xf/0x20 [ 5808.064974] [ 5808.064977] ? native_safe_halt+0x2e/0x50 [ 5808.064981] kvm_wait+0xcd/0xe0 [ 5808.064988] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5808.064996] do_raw_spin_lock+0x177/0x190 [ 5808.065000] _raw_spin_lock+0x21/0x30 [ 5808.065004] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5808.065329] ? do_raw_spin_unlock+0x75/0x190 [ 5808.065335] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5808.065528] ? do_wait_intr+0xf0/0xf0 [ 5808.065535] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5808.065728] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5808.066092] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5808.066297] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5808.066414] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5808.066458] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5808.066484] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5808.068213] osp(O) [ 5808.069210] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5808.069312] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5808.069563] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5808.069607] lmv_intent_open+0x338/0x1150 [lmv] [ 5808.069641] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5808.069728] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5808.069789] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5808.069804] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5808.069940] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5808.098462] ofd(O) [ 5808.103344] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5808.110742] lod(O) [ 5808.112706] ? do_raw_spin_unlock+0x75/0x190 [ 5808.112722] ? _raw_spin_unlock+0x12/0x30 [ 5808.112729] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5808.112755] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5808.112776] ll_atomic_open+0x36a/0xd70 [lustre] [ 5808.123869] mdt(O) [ 5808.127136] lookup_open+0x356/0xb40 [ 5808.127159] do_last+0x313/0xfc0 [ 5808.127165] ? nd_jump_root+0xe5/0x160 [ 5808.127170] ? path_init+0x437/0x520 [ 5808.127175] path_openat+0xf7/0x500 [ 5808.127180] ? do_raw_spin_unlock+0x75/0x190 [ 5808.127186] do_filp_open+0x99/0x140 [ 5808.127193] ? getname_flags+0x6e/0x330 [ 5808.127197] ? __check_object_size+0xff/0x256 [ 5808.127204] ? do_raw_spin_unlock+0x75/0x190 [ 5808.127207] ? _raw_spin_unlock+0x12/0x30 [ 5808.127214] do_sys_openat2+0x2b4/0x410 [ 5808.127219] do_sys_open+0x73/0xa0 [ 5808.127222] __x64_sys_openat+0x24/0x30 [ 5808.127225] do_syscall_64+0xc1/0x440 [ 5808.127231] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5808.127238] RIP: 0033:0x7f53425a5332 [ 5808.127247] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5808.127251] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5808.127256] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5808.127259] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5808.127261] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5808.127263] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5808.127266] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5808.352579] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5808.384903] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5808.395991] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5808.403054] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5808.405576] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5808.415071] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5808.418591] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5808.423422] RDX: 00000000000017bf RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5808.426913] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5808.430227] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5808.433070] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5808.435649] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5808.438270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5808.439994] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5808.442708] Call Trace: [ 5808.443445] [ 5808.444199] ? show_regs.cold.9+0x22/0x2f [ 5808.447279] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5808.449311] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5808.451238] ? watchdog+0x40/0x40 [ 5808.452610] ? hrtimer_interrupt+0x128/0x2c0 [ 5808.456319] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5808.460473] ? apic_timer_interrupt+0xf/0x20 [ 5808.462165] [ 5808.462887] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5808.465670] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5808.468415] do_raw_spin_lock+0x177/0x190 [ 5808.470384] _raw_spin_lock+0x21/0x30 [ 5808.474716] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5808.485843] obd_mod_free+0x46/0x140 [ptlrpc] [ 5808.494087] mdc_commit_open+0x91/0xb0 [mdc] [ 5808.496718] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5808.499578] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5808.501993] after_reply+0xe2c/0x1780 [ptlrpc] [ 5808.504466] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5808.506397] ? __next_timer_interrupt+0x160/0x160 [ 5808.508249] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5808.510362] ? do_wait_intr+0xf0/0xf0 [ 5808.511958] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5808.515727] kthread+0x1d1/0x200 [ 5808.517972] ? set_kthread_struct+0x70/0x70 [ 5808.519705] ret_from_fork+0x1f/0x30 [ 5824.058038] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 5824.061832] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5824.096308] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5824.100127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5824.103036] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5824.105311] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5824.115704] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 5824.118534] RAX: 0000000000007a0b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5824.121392] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5824.124242] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5824.134708] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5824.139199] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5824.142803] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5824.153826] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5824.158154] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5824.165160] Call Trace: [ 5824.177078] [ 5824.182999] ? show_regs.cold.9+0x22/0x2f [ 5824.188277] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5824.191372] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5824.198941] ? watchdog+0x40/0x40 [ 5824.205577] ? hrtimer_interrupt+0x128/0x2c0 [ 5824.207083] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5824.210182] ? apic_timer_interrupt+0xf/0x20 [ 5824.213290] [ 5824.216010] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5824.221547] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5824.225051] do_raw_spin_lock+0x177/0x190 [ 5824.229155] _raw_spin_lock+0x21/0x30 [ 5824.233165] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5824.240847] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5824.242421] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5824.247474] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5824.255832] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5824.261321] ? woken_wake_function+0x30/0x30 [ 5824.268602] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5824.271081] kthread+0x1d1/0x200 [ 5824.272199] ? set_kthread_struct+0x70/0x70 [ 5824.277784] ret_from_fork+0x1f/0x30 [ 5832.052011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 5832.056931] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5832.083056] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5832.089770] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5832.093182] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5832.099058] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 5832.102706] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 5832.115875] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5832.120036] RAX: 0000000000001a26 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5832.132405] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5832.136786] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5832.139802] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5832.142826] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5832.151057] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5832.157551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5832.163792] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5832.173065] Call Trace: [ 5832.174321] [ 5832.175069] ? show_regs.cold.9+0x22/0x2f [ 5832.180639] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5832.183083] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5832.185368] ? watchdog+0x40/0x40 [ 5832.186687] ? hrtimer_interrupt+0x128/0x2c0 [ 5832.188052] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5832.190387] ? apic_timer_interrupt+0xf/0x20 [ 5832.192738] [ 5832.193780] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 5832.196606] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5832.199028] ? newidle_balance.isra.93+0x207/0x580 [ 5832.201134] do_raw_spin_lock+0x177/0x190 [ 5832.206376] _raw_spin_lock+0x21/0x30 [ 5832.208290] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5832.210688] process_one_work+0x2c8/0x700 [ 5832.213058] worker_thread+0x296/0x6e0 [ 5832.214621] ? rescuer_thread+0x570/0x570 [ 5832.216369] kthread+0x1d1/0x200 [ 5832.217379] ? set_kthread_struct+0x70/0x70 [ 5832.218800] ret_from_fork+0x1f/0x30 [ 5836.059975] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 5836.062848] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 5836.063382] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5836.063453] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5836.063460] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5836.063463] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5836.063478] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5836.063483] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5836.063487] RAX: 0000000000007d1b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5836.063490] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5836.063492] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5836.063494] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5836.063497] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5836.063500] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5836.063503] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5836.063505] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5836.063514] Call Trace: [ 5836.063519] [ 5836.063525] ? show_regs.cold.9+0x22/0x2f [ 5836.063534] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5836.063541] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5836.063545] ? watchdog+0x40/0x40 [ 5836.063548] ? hrtimer_interrupt+0x128/0x2c0 [ 5836.063552] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5836.063560] ? apic_timer_interrupt+0xf/0x20 [ 5836.063565] [ 5836.063568] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5836.063573] do_raw_spin_lock+0x177/0x190 [ 5836.063578] _raw_spin_lock+0x21/0x30 [ 5836.063584] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5836.063861] ? do_raw_spin_unlock+0x75/0x190 [ 5836.063866] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5836.064065] ? do_wait_intr+0xf0/0xf0 [ 5836.064072] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5836.064266] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5836.114106] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5836.114494] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5836.114501] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5836.114504] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5836.114517] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5836.114522] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5836.114526] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5836.114528] RDX: 00000000000006ee RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5836.114530] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5836.114532] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5836.114535] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5836.114537] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5836.114540] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5836.114543] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5836.114551] Call Trace: [ 5836.114557] [ 5836.114562] ? show_regs.cold.9+0x22/0x2f [ 5836.114573] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5836.114580] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5836.114584] ? watchdog+0x40/0x40 [ 5836.114587] ? hrtimer_interrupt+0x128/0x2c0 [ 5836.114591] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5836.114599] ? apic_timer_interrupt+0xf/0x20 [ 5836.114604] [ 5836.114607] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5836.114611] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5836.114616] do_raw_spin_lock+0x177/0x190 [ 5836.114622] _raw_spin_lock+0x21/0x30 [ 5836.114628] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5836.114897] obd_mod_free+0x46/0x140 [ptlrpc] [ 5836.115304] mdc_commit_open+0x91/0xb0 [mdc] [ 5836.115417] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5836.115614] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5836.115806] after_reply+0xe2c/0x1780 [ptlrpc] [ 5836.133415] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5836.133695] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5836.133805] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5836.133844] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5836.133874] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5836.133902] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5836.137155] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5836.138166] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5836.138207] lmv_intent_open+0x338/0x1150 [lmv] [ 5836.138242] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5836.138533] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5836.138611] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5836.138632] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5836.138716] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5836.138796] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5836.138881] ? do_raw_spin_unlock+0x75/0x190 [ 5836.138888] ? _raw_spin_unlock+0x12/0x30 [ 5836.138895] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5836.138923] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5836.138943] ll_atomic_open+0x36a/0xd70 [lustre] [ 5836.148043] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5836.149407] ? __next_timer_interrupt+0x160/0x160 [ 5836.149420] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5836.149628] ? do_wait_intr+0xf0/0xf0 [ 5836.149638] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5836.149839] kthread+0x1d1/0x200 [ 5836.149846] ? set_kthread_struct+0x70/0x70 [ 5836.149850] ret_from_fork+0x1f/0x30 [ 5836.264027] BUG: workqueue lockup - pool [ 5836.269504] lookup_open+0x356/0xb40 [ 5836.269530] do_last+0x313/0xfc0 [ 5836.269535] ? nd_jump_root+0xe5/0x160 [ 5836.269541] ? path_init+0x437/0x520 [ 5836.269545] path_openat+0xf7/0x500 [ 5836.269550] ? do_raw_spin_unlock+0x75/0x190 [ 5836.269557] do_filp_open+0x99/0x140 [ 5836.269563] ? getname_flags+0x6e/0x330 [ 5836.269568] ? __check_object_size+0xff/0x256 [ 5836.269575] ? do_raw_spin_unlock+0x75/0x190 [ 5836.269579] ? _raw_spin_unlock+0x12/0x30 [ 5836.269585] do_sys_openat2+0x2b4/0x410 [ 5836.269590] do_sys_open+0x73/0xa0 [ 5836.269595] __x64_sys_openat+0x24/0x30 [ 5836.269599] do_syscall_64+0xc1/0x440 [ 5836.269605] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5836.269612] RIP: 0033:0x7f53425a5332 [ 5836.269620] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5836.269625] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5836.269630] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5836.269633] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5836.269635] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5836.269638] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5836.269640] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5836.475218] cpus=0-3 flags=0x4 nice=0 stuck for 3199s! [ 5836.477447] Showing busy workqueues and worker pools: [ 5836.479950] workqueue events: flags=0x0 [ 5836.481572] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5836.484686] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5836.489315] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5836.492433] pending: cpu_latency_work [ptlrpc] [ 5836.494652] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5836.497450] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5836.501488] workqueue events_long: flags=0x0 [ 5836.503463] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5836.509469] pending: lu_objects_flush [obdclass] [ 5836.514980] workqueue events_unbound: flags=0x2 [ 5836.517382] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5836.522972] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5836.525517] workqueue events_power_efficient: flags=0x80 [ 5836.528754] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5836.536577] pending: check_lifetime, do_cache_clean [ 5836.538046] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5836.550867] pending: neigh_periodic_work, neigh_periodic_work [ 5836.553689] workqueue mm_percpu_wq: flags=0x8 [ 5836.558957] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5836.571899] pending: vmstat_update, lru_add_drain_per_cpu [ 5836.574793] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5836.584064] pending: vmstat_update, lru_add_drain_per_cpu [ 5836.591965] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5836.597514] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5836.600041] workqueue xprtiod: flags=0xa [ 5836.601930] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5836.603931] pending: xprt_autoclose, xs_error_handle [ 5836.606287] workqueue ipv6_addrconf: flags=0x40008 [ 5836.608301] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5836.611100] pending: addrconf_verify_work [ 5836.614169] workqueue knbd0-recv: flags=0x1a [ 5836.615885] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5836.621935] in-flight: 651:recv_work [ 5836.623553] workqueue ptlrpc_pinger: flags=0x2 [ 5836.626893] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5836.630426] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5836.633182] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3199s workers=4 idle: 15979 31824 8 [ 5836.638581] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5827s workers=2 idle: 654 [ 5852.058898] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 5852.064596] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5852.120809] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5852.129353] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5852.134778] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5852.142737] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5852.156209] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5852.162325] RAX: 0000000000002286 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5852.168880] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5852.178776] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5852.184044] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5852.190659] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5852.194767] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5852.200355] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5852.205581] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5852.212695] Call Trace: [ 5852.214548] [ 5852.220925] ? show_regs.cold.9+0x22/0x2f [ 5852.226728] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5852.229701] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5852.232467] ? watchdog+0x40/0x40 [ 5852.234577] ? hrtimer_interrupt+0x128/0x2c0 [ 5852.236966] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5852.238633] ? apic_timer_interrupt+0xf/0x20 [ 5852.244575] [ 5852.245744] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5852.248642] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5852.252850] do_raw_spin_lock+0x177/0x190 [ 5852.256836] _raw_spin_lock+0x21/0x30 [ 5852.261293] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5852.270696] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5852.274304] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5852.280278] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5852.282306] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5852.290826] ? woken_wake_function+0x30/0x30 [ 5852.292786] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5852.295782] kthread+0x1d1/0x200 [ 5852.305925] ? set_kthread_struct+0x70/0x70 [ 5852.312762] ret_from_fork+0x1f/0x30 [ 5864.055062] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 5864.057025] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 5864.057883] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 5864.057892] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5864.057975] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5864.057981] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5864.059072] Modules linked in: lustre(O) osp(O) [ 5864.057987] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5864.059650] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5864.059663] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5864.059668] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5864.059672] RAX: 0000000000000dab RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5864.059674] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5864.059677] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5864.059679] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5864.059681] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5864.059684] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5864.061058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5864.061063] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5864.061071] Call Trace: [ 5864.061078] [ 5864.061084] ? show_regs.cold.9+0x22/0x2f [ 5864.061095] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5864.061101] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5864.061105] ? watchdog+0x40/0x40 [ 5864.061108] ? hrtimer_interrupt+0x128/0x2c0 [ 5864.061111] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5864.061118] ? apic_timer_interrupt+0xf/0x20 [ 5864.061123] [ 5864.061126] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5864.061133] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5864.061137] ? newidle_balance.isra.93+0x207/0x580 [ 5864.061144] do_raw_spin_lock+0x177/0x190 [ 5864.061148] _raw_spin_lock+0x21/0x30 [ 5864.061153] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5864.061410] process_one_work+0x2c8/0x700 [ 5864.061418] worker_thread+0x296/0x6e0 [ 5864.061422] ? rescuer_thread+0x570/0x570 [ 5864.061425] kthread+0x1d1/0x200 [ 5864.061429] ? set_kthread_struct+0x70/0x70 [ 5864.061433] ret_from_fork+0x1f/0x30 [ 5864.063319] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5864.063404] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5864.063412] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5864.063414] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5864.063427] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5864.063432] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5864.063436] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5864.063439] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5864.063441] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5864.063443] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5864.063446] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5864.063449] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5864.063452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5864.063455] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5864.063463] Call Trace: [ 5864.063468] [ 5864.063473] ? show_regs.cold.9+0x22/0x2f [ 5864.063483] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5864.063488] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5864.063492] ? watchdog+0x40/0x40 [ 5864.063497] ? hrtimer_interrupt+0x128/0x2c0 [ 5864.063501] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5864.063507] ? apic_timer_interrupt+0xf/0x20 [ 5864.063512] [ 5864.063515] ? native_safe_halt+0x2e/0x50 [ 5864.063519] kvm_wait+0xcd/0xe0 [ 5864.063526] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5864.063534] do_raw_spin_lock+0x177/0x190 [ 5864.063538] _raw_spin_lock+0x21/0x30 [ 5864.063543] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5864.064332] obd_mod_free+0x46/0x140 [ptlrpc] [ 5864.064546] mdc_commit_open+0x91/0xb0 [mdc] [ 5864.064595] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5864.064793] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5864.065193] after_reply+0xe2c/0x1780 [ptlrpc] [ 5864.065469] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5864.065792] ? __next_timer_interrupt+0x160/0x160 [ 5864.065802] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5864.066074] ? do_wait_intr+0xf0/0xf0 [ 5864.066083] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5864.072046] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5864.072221] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5864.072229] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5864.072232] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5864.072245] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5864.072250] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5864.072255] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5864.072257] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5864.072259] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5864.072262] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5864.072264] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5864.072268] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5864.072271] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5864.072273] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5864.072281] Call Trace: [ 5864.072286] [ 5864.072291] ? show_regs.cold.9+0x22/0x2f [ 5864.098203] kthread+0x1d1/0x200 [ 5864.098224] ? set_kthread_struct+0x70/0x70 [ 5864.098229] ret_from_fork+0x1f/0x30 [ 5864.369088] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5864.370307] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5864.371449] ? watchdog+0x40/0x40 [ 5864.372338] ? hrtimer_interrupt+0x128/0x2c0 [ 5864.373433] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5864.374632] ? apic_timer_interrupt+0xf/0x20 [ 5864.375747] [ 5864.376323] ? native_safe_halt+0x2e/0x50 [ 5864.377310] kvm_wait+0xcd/0xe0 [ 5864.378143] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5864.379584] do_raw_spin_lock+0x177/0x190 [ 5864.380687] _raw_spin_lock+0x21/0x30 [ 5864.381669] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5864.383439] ? do_raw_spin_unlock+0x75/0x190 [ 5864.384564] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5864.386183] ? do_wait_intr+0xf0/0xf0 [ 5864.387234] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5864.388718] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5864.390235] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5864.391923] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5864.393375] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5864.394868] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5864.396199] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5864.397383] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5864.398840] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5864.400564] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5864.402628] lmv_intent_open+0x338/0x1150 [lmv] [ 5864.403821] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5864.405581] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5864.407302] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5864.408720] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5864.410223] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5864.411831] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5864.413478] ? do_raw_spin_unlock+0x75/0x190 [ 5864.414557] ? _raw_spin_unlock+0x12/0x30 [ 5864.415618] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5864.417093] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5864.419033] ll_atomic_open+0x36a/0xd70 [lustre] [ 5864.420733] lookup_open+0x356/0xb40 [ 5864.421712] do_last+0x313/0xfc0 [ 5864.422690] ? nd_jump_root+0xe5/0x160 [ 5864.423660] ? path_init+0x437/0x520 [ 5864.424582] path_openat+0xf7/0x500 [ 5864.425548] ? do_raw_spin_unlock+0x75/0x190 [ 5864.426683] do_filp_open+0x99/0x140 [ 5864.427600] ? getname_flags+0x6e/0x330 [ 5864.428650] ? __check_object_size+0xff/0x256 [ 5864.429782] ? do_raw_spin_unlock+0x75/0x190 [ 5864.431109] ? _raw_spin_unlock+0x12/0x30 [ 5864.432077] do_sys_openat2+0x2b4/0x410 [ 5864.433049] do_sys_open+0x73/0xa0 [ 5864.433825] __x64_sys_openat+0x24/0x30 [ 5864.435038] do_syscall_64+0xc1/0x440 [ 5864.435923] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5864.437224] RIP: 0033:0x7f53425a5332 [ 5864.438204] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5864.443270] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5864.445190] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5864.447105] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5864.448788] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5864.450792] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5864.453258] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5866.975166] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3229s! [ 5866.983965] Showing busy workqueues and worker pools: [ 5866.992914] workqueue events: flags=0x0 [ 5867.000597] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5867.002800] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5867.020911] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5867.026973] pending: cpu_latency_work [ptlrpc] [ 5867.038307] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5867.045704] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5867.062070] workqueue events_long: flags=0x0 [ 5867.065904] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5867.079147] pending: lu_objects_flush [obdclass] [ 5867.089277] workqueue events_unbound: flags=0x2 [ 5867.094300] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5867.108374] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5867.117465] workqueue events_power_efficient: flags=0x80 [ 5867.122471] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5867.133599] pending: check_lifetime, do_cache_clean [ 5867.135421] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5867.146768] pending: neigh_periodic_work, neigh_periodic_work [ 5867.157778] workqueue mm_percpu_wq: flags=0x8 [ 5867.160712] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5867.164081] pending: vmstat_update, lru_add_drain_per_cpu [ 5867.166228] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5867.173908] pending: vmstat_update, lru_add_drain_per_cpu [ 5867.176963] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5867.179616] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5867.181331] workqueue xprtiod: flags=0xa [ 5867.182723] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5867.185363] pending: xprt_autoclose, xs_error_handle [ 5867.188276] workqueue ipv6_addrconf: flags=0x40008 [ 5867.193530] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5867.198212] pending: addrconf_verify_work [ 5867.200531] workqueue knbd0-recv: flags=0x1a [ 5867.202607] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5867.208734] in-flight: 651:recv_work [ 5867.210227] workqueue ptlrpc_pinger: flags=0x2 [ 5867.211552] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5867.213948] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5867.215759] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3230s workers=4 idle: 15979 31824 8 [ 5867.218473] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5857s workers=2 idle: 654 [ 5880.058945] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 5880.067702] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5880.114120] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5880.123939] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5880.130238] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5880.132491] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5880.145437] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5880.147851] RAX: 0000000000007f12 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5880.150464] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5880.153194] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5880.155762] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5880.158470] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5880.162100] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5880.166534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5880.168467] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5880.171039] Call Trace: [ 5880.171919] [ 5880.172660] ? show_regs.cold.9+0x22/0x2f [ 5880.174122] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5880.175868] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5880.177501] ? watchdog+0x40/0x40 [ 5880.178776] ? hrtimer_interrupt+0x128/0x2c0 [ 5880.180243] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5880.181891] ? apic_timer_interrupt+0xf/0x20 [ 5880.183469] [ 5880.184270] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5880.186233] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5880.188641] do_raw_spin_lock+0x177/0x190 [ 5880.190001] _raw_spin_lock+0x21/0x30 [ 5880.191262] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5880.193164] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5880.195210] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5880.201082] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5880.203977] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5880.207036] ? woken_wake_function+0x30/0x30 [ 5880.208850] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5880.211355] kthread+0x1d1/0x200 [ 5880.212761] ? set_kthread_struct+0x70/0x70 [ 5880.214690] ret_from_fork+0x1f/0x30 [ 5892.053992] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 5892.055009] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 5892.058560] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 5892.058573] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5892.058659] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5892.058665] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5892.058667] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5892.058680] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5892.058685] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5892.058689] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5892.058692] RDX: 0000000000001499 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5892.058694] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5892.058697] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5892.058700] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5892.058703] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5892.058705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5892.058708] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5892.058790] Call Trace: [ 5892.058795] [ 5892.058799] ? show_regs.cold.9+0x22/0x2f [ 5892.058808] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5892.058814] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5892.058818] ? watchdog+0x40/0x40 [ 5892.058950] ? hrtimer_interrupt+0x128/0x2c0 [ 5892.058957] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5892.058965] ? apic_timer_interrupt+0xf/0x20 [ 5892.058969] [ 5892.058973] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5892.058977] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5892.058982] do_raw_spin_lock+0x177/0x190 [ 5892.058987] _raw_spin_lock+0x21/0x30 [ 5892.059052] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5892.059327] obd_mod_free+0x46/0x140 [ptlrpc] [ 5892.059548] mdc_commit_open+0x91/0xb0 [mdc] [ 5892.059582] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5892.061384] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5892.061697] after_reply+0xe2c/0x1780 [ptlrpc] [ 5892.062280] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5892.063682] ? __next_timer_interrupt+0x160/0x160 [ 5892.063882] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5892.063957] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5892.063964] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5892.063970] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5892.063694] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5892.065185] ? do_wait_intr+0xf0/0xf0 [ 5892.065195] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5892.065410] kthread+0x1d1/0x200 [ 5892.065416] ? set_kthread_struct+0x70/0x70 [ 5892.065420] ret_from_fork+0x1f/0x30 [ 5892.069730] Modules linked in: [ 5892.073800] [ 5892.114653] lustre(O) [ 5892.125239] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5892.125263] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5892.125268] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5892.125275] RAX: 0000000000007f87 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5892.125277] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5892.133583] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5892.133656] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5892.133663] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5892.133666] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5892.133680] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5892.133684] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5892.133689] RAX: 0000000000006720 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5892.133691] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5892.133694] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5892.133697] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5892.133700] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5892.133704] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5892.133708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5892.133710] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5892.133814] Call Trace: [ 5892.133821] [ 5892.133826] ? show_regs.cold.9+0x22/0x2f [ 5892.133836] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5892.133843] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5892.133847] ? watchdog+0x40/0x40 [ 5892.133851] ? hrtimer_interrupt+0x128/0x2c0 [ 5892.133855] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5892.133870] ? apic_timer_interrupt+0xf/0x20 [ 5892.133875] [ 5892.133878] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5892.133884] do_raw_spin_lock+0x177/0x190 [ 5892.133889] _raw_spin_lock+0x21/0x30 [ 5892.133894] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5892.134177] ? do_raw_spin_unlock+0x75/0x190 [ 5892.134183] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5892.134595] ? do_wait_intr+0xf0/0xf0 [ 5892.134605] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5892.134902] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5892.135129] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5892.135562] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5892.138892] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5892.138902] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5892.138904] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5892.138907] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5892.138911] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5892.138913] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5892.138924] Call Trace: [ 5892.138931] [ 5892.138936] ? show_regs.cold.9+0x22/0x2f [ 5892.138950] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5892.138958] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5892.138962] ? watchdog+0x40/0x40 [ 5892.138967] ? hrtimer_interrupt+0x128/0x2c0 [ 5892.138971] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5892.138979] ? apic_timer_interrupt+0xf/0x20 [ 5892.138984] [ 5892.138995] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5892.139002] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5892.139007] ? newidle_balance.isra.93+0x207/0x580 [ 5892.139014] do_raw_spin_lock+0x177/0x190 [ 5892.139020] _raw_spin_lock+0x21/0x30 [ 5892.139024] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5892.139343] process_one_work+0x2c8/0x700 [ 5892.139351] worker_thread+0x296/0x6e0 [ 5892.139356] ? rescuer_thread+0x570/0x570 [ 5892.139360] kthread+0x1d1/0x200 [ 5892.139384] ? set_kthread_struct+0x70/0x70 [ 5892.139387] ret_from_fork+0x1f/0x30 [ 5892.629834] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5892.633058] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5892.635659] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5892.639473] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5892.642226] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5892.647879] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5892.650541] lmv_intent_open+0x338/0x1150 [lmv] [ 5892.652599] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5892.655213] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5892.659013] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5892.661741] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5892.664668] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5892.666938] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5892.669228] ? do_raw_spin_unlock+0x75/0x190 [ 5892.673658] ? _raw_spin_unlock+0x12/0x30 [ 5892.675503] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5892.677711] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5892.679657] ll_atomic_open+0x36a/0xd70 [lustre] [ 5892.684219] lookup_open+0x356/0xb40 [ 5892.686573] do_last+0x313/0xfc0 [ 5892.687801] ? nd_jump_root+0xe5/0x160 [ 5892.689539] ? path_init+0x437/0x520 [ 5892.691967] path_openat+0xf7/0x500 [ 5892.694079] ? do_raw_spin_unlock+0x75/0x190 [ 5892.697683] do_filp_open+0x99/0x140 [ 5892.699349] ? getname_flags+0x6e/0x330 [ 5892.703563] ? __check_object_size+0xff/0x256 [ 5892.706142] ? do_raw_spin_unlock+0x75/0x190 [ 5892.710569] ? _raw_spin_unlock+0x12/0x30 [ 5892.714489] do_sys_openat2+0x2b4/0x410 [ 5892.717616] do_sys_open+0x73/0xa0 [ 5892.723475] __x64_sys_openat+0x24/0x30 [ 5892.728882] do_syscall_64+0xc1/0x440 [ 5892.730174] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5892.732917] RIP: 0033:0x7f53425a5332 [ 5892.734824] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5892.758704] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5892.764215] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5892.774425] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5892.779554] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5892.783120] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5892.793314] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5897.697103] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3260s! [ 5897.705809] Showing busy workqueues and worker pools: [ 5897.708777] workqueue events: flags=0x0 [ 5897.713915] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5897.722468] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5897.729136] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5897.734969] pending: cpu_latency_work [ptlrpc] [ 5897.738085] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5897.741077] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5897.745639] workqueue events_long: flags=0x0 [ 5897.751582] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5897.759444] pending: lu_objects_flush [obdclass] [ 5897.764716] workqueue events_unbound: flags=0x2 [ 5897.771069] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5897.774035] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5897.776594] workqueue events_power_efficient: flags=0x80 [ 5897.778655] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5897.781236] pending: check_lifetime, do_cache_clean [ 5897.783233] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5897.785777] pending: neigh_periodic_work, neigh_periodic_work [ 5897.788094] workqueue mm_percpu_wq: flags=0x8 [ 5897.790424] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5897.792593] pending: vmstat_update, lru_add_drain_per_cpu [ 5897.794080] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5897.796719] pending: vmstat_update, lru_add_drain_per_cpu [ 5897.798895] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5897.801754] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5897.804199] workqueue xprtiod: flags=0xa [ 5897.805735] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5897.808106] pending: xprt_autoclose, xs_error_handle [ 5897.810220] workqueue ipv6_addrconf: flags=0x40008 [ 5897.812015] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5897.814618] pending: addrconf_verify_work [ 5897.816230] workqueue knbd0-recv: flags=0x1a [ 5897.817995] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5897.824514] in-flight: 651:recv_work [ 5897.825749] workqueue ptlrpc_pinger: flags=0x2 [ 5897.827912] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5897.831250] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5897.835558] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3260s workers=4 idle: 15979 31824 8 [ 5897.838897] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5888s workers=2 idle: 654 [ 5908.061579] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 5908.068915] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5908.100721] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5908.105148] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5908.108549] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5908.110488] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5908.116349] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5908.118707] RAX: 0000000000005d4f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5908.121396] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5908.123539] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5908.125807] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5908.127957] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5908.129932] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5908.132283] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5908.133858] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5908.135927] Call Trace: [ 5908.136643] [ 5908.137163] ? show_regs.cold.9+0x22/0x2f [ 5908.138160] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5908.139616] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5908.141201] ? watchdog+0x40/0x40 [ 5908.142230] ? hrtimer_interrupt+0x128/0x2c0 [ 5908.143393] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5908.144773] ? apic_timer_interrupt+0xf/0x20 [ 5908.146093] [ 5908.146782] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5908.148244] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5908.149672] do_raw_spin_lock+0x177/0x190 [ 5908.150792] _raw_spin_lock+0x21/0x30 [ 5908.151968] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5908.154002] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5908.156215] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5908.159076] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5908.162745] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5908.165778] ? woken_wake_function+0x30/0x30 [ 5908.167207] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5908.169628] kthread+0x1d1/0x200 [ 5908.170934] ? set_kthread_struct+0x70/0x70 [ 5908.173807] ret_from_fork+0x1f/0x30 [ 5920.055052] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 5920.057027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 5920.061434] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5920.061512] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5920.061519] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5920.061522] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5920.061535] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5920.061540] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5920.061544] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5920.061547] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5920.061548] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5920.061550] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5920.061553] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5920.061556] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5920.061560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5920.061562] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5920.061571] Call Trace: [ 5920.061576] [ 5920.061582] ? show_regs.cold.9+0x22/0x2f [ 5920.061592] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5920.061612] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5920.061617] ? watchdog+0x40/0x40 [ 5920.061621] ? hrtimer_interrupt+0x128/0x2c0 [ 5920.061625] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5920.061631] ? apic_timer_interrupt+0xf/0x20 [ 5920.061637] [ 5920.061640] ? native_safe_halt+0x2e/0x50 [ 5920.061644] kvm_wait+0xcd/0xe0 [ 5920.061653] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5920.061662] do_raw_spin_lock+0x177/0x190 [ 5920.061666] _raw_spin_lock+0x21/0x30 [ 5920.061671] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5920.065503] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) [ 5920.114542] ? do_raw_spin_unlock+0x75/0x190 [ 5920.118276] ec(O) [ 5920.125816] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5920.127996] fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5920.128054] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5920.128061] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5920.128064] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5920.128076] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5920.128081] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5920.128086] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5920.128088] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5920.128090] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5920.128092] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5920.128094] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5920.128096] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5920.128099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5920.128102] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5920.128112] Call Trace: [ 5920.128119] [ 5920.128125] ? show_regs.cold.9+0x22/0x2f [ 5920.128135] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5920.128142] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5920.128147] ? watchdog+0x40/0x40 [ 5920.128151] ? hrtimer_interrupt+0x128/0x2c0 [ 5920.128155] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5920.128234] ? apic_timer_interrupt+0xf/0x20 [ 5920.128239] [ 5920.128242] ? native_safe_halt+0x2e/0x50 [ 5920.128246] kvm_wait+0xcd/0xe0 [ 5920.128254] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5920.128262] do_raw_spin_lock+0x177/0x190 [ 5920.128268] _raw_spin_lock+0x21/0x30 [ 5920.128273] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5920.142442] ? do_wait_intr+0xf0/0xf0 [ 5920.146105] obd_mod_free+0x46/0x140 [ptlrpc] [ 5920.151928] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5920.154611] mdc_commit_open+0x91/0xb0 [mdc] [ 5920.154663] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5920.160537] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5920.163697] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5920.169499] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5920.173282] after_reply+0xe2c/0x1780 [ptlrpc] [ 5920.177212] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5920.182040] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5920.183448] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5920.183489] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5920.183517] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5920.183544] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5920.183800] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5920.185414] ? __next_timer_interrupt+0x160/0x160 [ 5920.185430] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5920.185689] ? do_wait_intr+0xf0/0xf0 [ 5920.185699] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5920.189459] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5920.195603] kthread+0x1d1/0x200 [ 5920.195624] ? set_kthread_struct+0x70/0x70 [ 5920.195629] ret_from_fork+0x1f/0x30 [ 5920.527382] lmv_intent_open+0x338/0x1150 [lmv] [ 5920.535645] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5920.540642] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5920.544990] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5920.547724] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5920.551757] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5920.555711] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5920.559445] ? do_raw_spin_unlock+0x75/0x190 [ 5920.560846] ? _raw_spin_unlock+0x12/0x30 [ 5920.566065] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5920.569586] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5920.572731] ll_atomic_open+0x36a/0xd70 [lustre] [ 5920.575415] lookup_open+0x356/0xb40 [ 5920.576565] do_last+0x313/0xfc0 [ 5920.577687] ? nd_jump_root+0xe5/0x160 [ 5920.583757] ? path_init+0x437/0x520 [ 5920.588789] path_openat+0xf7/0x500 [ 5920.592656] ? do_raw_spin_unlock+0x75/0x190 [ 5920.596832] do_filp_open+0x99/0x140 [ 5920.599999] ? getname_flags+0x6e/0x330 [ 5920.601308] ? __check_object_size+0xff/0x256 [ 5920.604973] ? do_raw_spin_unlock+0x75/0x190 [ 5920.609152] ? _raw_spin_unlock+0x12/0x30 [ 5920.617534] do_sys_openat2+0x2b4/0x410 [ 5920.620976] do_sys_open+0x73/0xa0 [ 5920.623226] __x64_sys_openat+0x24/0x30 [ 5920.624590] do_syscall_64+0xc1/0x440 [ 5920.628624] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5920.633607] RIP: 0033:0x7f53425a5332 [ 5920.636190] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5920.651302] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5920.655987] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5920.661493] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5920.665992] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5920.670780] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5920.674519] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5924.056055] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 5924.060620] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5924.103081] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5924.112783] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5924.118759] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5924.126468] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5924.130863] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5924.145873] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5924.149690] RAX: 0000000000002108 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5924.154484] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5924.160632] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5924.166005] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5924.174407] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5924.178762] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5924.188404] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5924.195322] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5924.200598] Call Trace: [ 5924.201777] [ 5924.204201] ? show_regs.cold.9+0x22/0x2f [ 5924.206160] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5924.207488] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5924.208750] ? watchdog+0x40/0x40 [ 5924.218482] ? hrtimer_interrupt+0x128/0x2c0 [ 5924.220759] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5924.223676] ? apic_timer_interrupt+0xf/0x20 [ 5924.226316] [ 5924.227821] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5924.233696] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5924.238602] ? newidle_balance.isra.93+0x207/0x580 [ 5924.241352] do_raw_spin_lock+0x177/0x190 [ 5924.244530] _raw_spin_lock+0x21/0x30 [ 5924.246444] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5924.249828] process_one_work+0x2c8/0x700 [ 5924.259280] worker_thread+0x296/0x6e0 [ 5924.260683] ? rescuer_thread+0x570/0x570 [ 5924.261890] kthread+0x1d1/0x200 [ 5924.262725] ? set_kthread_struct+0x70/0x70 [ 5924.267209] ret_from_fork+0x1f/0x30 [ 5927.903108] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3290s! [ 5927.906712] Showing busy workqueues and worker pools: [ 5927.909016] workqueue events: flags=0x0 [ 5927.910830] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5927.913659] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5927.917577] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5927.920255] pending: cpu_latency_work [ptlrpc] [ 5927.922416] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5927.924854] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5927.929052] workqueue events_long: flags=0x0 [ 5927.930982] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5927.933729] pending: lu_objects_flush [obdclass] [ 5927.936966] workqueue events_unbound: flags=0x2 [ 5927.938711] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5927.940713] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5927.943020] workqueue events_power_efficient: flags=0x80 [ 5927.944889] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5927.947365] pending: check_lifetime, do_cache_clean [ 5927.949337] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5927.951839] pending: neigh_periodic_work, neigh_periodic_work [ 5927.954326] workqueue mm_percpu_wq: flags=0x8 [ 5927.955930] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5927.958797] pending: vmstat_update, lru_add_drain_per_cpu [ 5927.961598] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5927.964239] pending: vmstat_update, lru_add_drain_per_cpu [ 5927.966521] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5927.968799] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5927.970561] workqueue xprtiod: flags=0xa [ 5927.972347] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5927.974565] pending: xprt_autoclose, xs_error_handle [ 5927.976774] workqueue ipv6_addrconf: flags=0x40008 [ 5927.978459] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5927.985067] pending: addrconf_verify_work [ 5927.987260] workqueue knbd0-recv: flags=0x1a [ 5927.989236] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5927.992335] in-flight: 651:recv_work [ 5927.994562] workqueue ptlrpc_pinger: flags=0x2 [ 5927.998770] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5928.004852] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5928.012936] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3290s workers=4 idle: 15979 31824 8 [ 5928.020499] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5918s workers=2 idle: 654 [ 5934.521127] rcu: INFO: rcu_sched self-detected stall on CPU [ 5934.524532] rcu: 2-....: (2061972 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 5934.530042] (t=3840095 jiffies g=274185 q=64611) [ 5934.532117] rcu: rcu_sched kthread starved for 3298461 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 5934.537310] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 5934.541491] rcu: RCU grace-period kthread stack dump: [ 5934.543746] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 5934.548285] Call Trace: [ 5934.549351] __schedule+0x351/0xcb0 [ 5934.550885] schedule+0xc0/0x180 [ 5934.552522] schedule_timeout+0xb4/0x190 [ 5934.554339] ? do_raw_spin_unlock+0x75/0x190 [ 5934.556274] ? __next_timer_interrupt+0x160/0x160 [ 5934.558279] ? prepare_to_swait_event+0xf3/0x140 [ 5934.559812] rcu_gp_kthread+0x8dd/0x15b0 [ 5934.563383] ? force_qs_rnp+0x2c0/0x2c0 [ 5934.565376] kthread+0x1d1/0x200 [ 5934.567160] ? set_kthread_struct+0x70/0x70 [ 5934.569316] ret_from_fork+0x1f/0x30 [ 5934.571102] rcu: Stack dump where RCU GP kthread last ran: [ 5934.573708] Sending NMI from CPU 2 to CPUs 0: [ 5934.575929] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 5934.578875] Sending NMI from CPU 2 to CPUs 1: [ 5934.581358] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 5934.584936] NMI backtrace for cpu 2 [ 5934.587154] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5934.592835] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5934.596979] Call Trace: [ 5934.599784] [ 5934.600977] ? dump_stack+0xbb/0x10e [ 5934.603484] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 5934.605068] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 5934.607087] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 5934.609256] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 5934.611468] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 5934.613142] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 5934.615019] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 5934.617075] ? update_process_times+0x6d/0xc0 [ 5934.618758] ? tick_sched_handle.isra.18+0x27/0x90 [ 5934.620802] ? tick_sched_timer+0x53/0xb0 [ 5934.622521] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5934.624406] ? tick_sched_handle.isra.18+0x90/0x90 [ 5934.626467] ? hrtimer_interrupt+0x128/0x2c0 [ 5934.628275] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5934.630343] ? apic_timer_interrupt+0xf/0x20 [ 5934.632370] [ 5934.633374] ? native_safe_halt+0x2e/0x50 [ 5934.635404] ? kvm_wait+0xcd/0xe0 [ 5934.636810] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5934.641603] ? do_raw_spin_lock+0x177/0x190 [ 5934.643501] ? _raw_spin_lock+0x21/0x30 [ 5934.645338] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5934.647965] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 5934.650397] ? mdc_commit_open+0x91/0xb0 [mdc] [ 5934.652586] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5934.655363] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5934.658069] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 5934.660571] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5934.663301] ? __next_timer_interrupt+0x160/0x160 [ 5934.665249] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5934.666846] ? do_wait_intr+0xf0/0xf0 [ 5934.668296] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5934.671000] ? kthread+0x1d1/0x200 [ 5934.672793] ? set_kthread_struct+0x70/0x70 [ 5934.674925] ? ret_from_fork+0x1f/0x30 [ 5936.060224] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5936.070160] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5936.128214] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5936.140031] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5936.147941] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5936.151313] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5936.170923] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5936.180427] RAX: 0000000000007290 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5936.190744] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5936.208980] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5936.226147] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5936.243529] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5936.253531] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5936.257838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5936.261575] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5936.265681] Call Trace: [ 5936.266572] [ 5936.268223] ? show_regs.cold.9+0x22/0x2f [ 5936.269698] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5936.277761] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5936.281434] ? watchdog+0x40/0x40 [ 5936.284155] ? hrtimer_interrupt+0x128/0x2c0 [ 5936.286841] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5936.290041] ? apic_timer_interrupt+0xf/0x20 [ 5936.291541] [ 5936.295498] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5936.300079] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5936.302945] do_raw_spin_lock+0x177/0x190 [ 5936.306189] _raw_spin_lock+0x21/0x30 [ 5936.309092] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5936.313569] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5936.317375] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5936.320512] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5936.325053] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5936.329016] ? woken_wake_function+0x30/0x30 [ 5936.332385] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5936.335710] kthread+0x1d1/0x200 [ 5936.339644] ? set_kthread_struct+0x70/0x70 [ 5936.342347] ret_from_fork+0x1f/0x30 [ 5958.122651] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3321s! [ 5958.127455] Showing busy workqueues and worker pools: [ 5958.130857] workqueue events: flags=0x0 [ 5958.133008] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5958.135897] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5958.141746] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5958.144329] pending: cpu_latency_work [ptlrpc] [ 5958.146874] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5958.149790] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5958.154057] workqueue events_long: flags=0x0 [ 5958.155849] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5958.158725] pending: lu_objects_flush [obdclass] [ 5958.160823] workqueue events_unbound: flags=0x2 [ 5958.163014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5958.165743] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5958.168714] workqueue events_power_efficient: flags=0x80 [ 5958.170416] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5958.173826] pending: check_lifetime, do_cache_clean [ 5958.176531] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5958.183679] pending: neigh_periodic_work, neigh_periodic_work [ 5958.186686] workqueue mm_percpu_wq: flags=0x8 [ 5958.189577] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5958.194044] pending: vmstat_update, lru_add_drain_per_cpu [ 5958.197110] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5958.199278] pending: vmstat_update, lru_add_drain_per_cpu [ 5958.202156] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5958.208899] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5958.212741] workqueue xprtiod: flags=0xa [ 5958.215474] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5958.218457] pending: xprt_autoclose, xs_error_handle [ 5958.220954] workqueue ipv6_addrconf: flags=0x40008 [ 5958.223172] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5958.226847] pending: addrconf_verify_work [ 5958.229019] workqueue knbd0-recv: flags=0x1a [ 5958.231066] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5958.234708] in-flight: 651:recv_work [ 5958.237445] workqueue ptlrpc_pinger: flags=0x2 [ 5958.239495] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5958.243056] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5958.247051] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3321s workers=4 idle: 15979 31824 8 [ 5958.253132] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5948s workers=2 idle: 654 [ 5960.057586] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 5960.061150] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5960.062659] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 5960.103081] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5960.103096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5960.103098] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5960.103112] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5960.103116] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5960.103122] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5960.103124] RDX: 0000000000005d57 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5960.103126] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5960.103128] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5960.103130] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5960.103133] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5960.103136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5960.103138] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5960.103146] Call Trace: [ 5960.103151] [ 5960.103157] ? show_regs.cold.9+0x22/0x2f [ 5960.103167] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5960.103183] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5960.103188] ? watchdog+0x40/0x40 [ 5960.103192] ? hrtimer_interrupt+0x128/0x2c0 [ 5960.103195] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5960.103204] ? apic_timer_interrupt+0xf/0x20 [ 5960.103209] [ 5960.103212] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5960.103216] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5960.103221] do_raw_spin_lock+0x177/0x190 [ 5960.103226] _raw_spin_lock+0x21/0x30 [ 5960.103234] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5960.103780] obd_mod_free+0x46/0x140 [ptlrpc] [ 5960.103995] mdc_commit_open+0x91/0xb0 [mdc] [ 5960.104037] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5960.104250] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5960.104615] after_reply+0xe2c/0x1780 [ptlrpc] [ 5960.104864] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5960.105061] ? __next_timer_interrupt+0x160/0x160 [ 5960.105073] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5960.105533] ? do_wait_intr+0xf0/0xf0 [ 5960.105544] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5960.105880] kthread+0x1d1/0x200 [ 5960.105886] ? set_kthread_struct+0x70/0x70 [ 5960.105891] ret_from_fork+0x1f/0x30 [ 5960.233674] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5960.257510] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5960.263998] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5960.267730] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5960.270360] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5960.278087] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5960.282416] RAX: 0000000000007e08 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5960.286473] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5960.289524] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5960.298414] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5960.304590] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5960.307593] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5960.312278] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5960.318078] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5960.330559] Call Trace: [ 5960.331380] [ 5960.332125] ? show_regs.cold.9+0x22/0x2f [ 5960.335635] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5960.339380] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5960.344984] ? watchdog+0x40/0x40 [ 5960.349404] ? hrtimer_interrupt+0x128/0x2c0 [ 5960.353451] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5960.358136] ? apic_timer_interrupt+0xf/0x20 [ 5960.363045] [ 5960.365368] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5960.369348] do_raw_spin_lock+0x177/0x190 [ 5960.372471] _raw_spin_lock+0x21/0x30 [ 5960.376595] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5960.384749] ? do_raw_spin_unlock+0x75/0x190 [ 5960.388428] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5960.394475] ? do_wait_intr+0xf0/0xf0 [ 5960.400651] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5960.403574] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5960.405499] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5960.412344] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5960.418394] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5960.421663] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5960.425298] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5960.428683] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5960.434151] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5960.438426] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5960.447385] lmv_intent_open+0x338/0x1150 [lmv] [ 5960.451519] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5960.457004] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5960.459320] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5960.460534] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5960.461817] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5960.471010] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5960.474319] ? do_raw_spin_unlock+0x75/0x190 [ 5960.480236] ? _raw_spin_unlock+0x12/0x30 [ 5960.481499] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5960.486107] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5960.487414] ll_atomic_open+0x36a/0xd70 [lustre] [ 5960.488561] lookup_open+0x356/0xb40 [ 5960.492550] do_last+0x313/0xfc0 [ 5960.494239] ? nd_jump_root+0xe5/0x160 [ 5960.495998] ? path_init+0x437/0x520 [ 5960.497379] path_openat+0xf7/0x500 [ 5960.498866] ? do_raw_spin_unlock+0x75/0x190 [ 5960.501670] do_filp_open+0x99/0x140 [ 5960.505530] ? getname_flags+0x6e/0x330 [ 5960.509736] ? __check_object_size+0xff/0x256 [ 5960.511984] ? do_raw_spin_unlock+0x75/0x190 [ 5960.514268] ? _raw_spin_unlock+0x12/0x30 [ 5960.516495] do_sys_openat2+0x2b4/0x410 [ 5960.517405] do_sys_open+0x73/0xa0 [ 5960.520323] __x64_sys_openat+0x24/0x30 [ 5960.521949] do_syscall_64+0xc1/0x440 [ 5960.523301] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5960.525888] RIP: 0033:0x7f53425a5332 [ 5960.527502] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5960.538991] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5960.543401] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5960.547661] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5960.553633] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5960.561640] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5960.571134] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5964.059957] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5964.066064] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5964.109075] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5964.119296] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5964.126080] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5964.132392] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5964.148716] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5964.152233] RAX: 0000000000000ea3 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5964.155763] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5964.158959] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5964.161652] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5964.164780] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5964.167655] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5964.170868] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5964.173082] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5964.176308] Call Trace: [ 5964.177182] [ 5964.178037] ? show_regs.cold.9+0x22/0x2f [ 5964.179776] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5964.181914] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5964.183872] ? watchdog+0x40/0x40 [ 5964.185387] ? hrtimer_interrupt+0x128/0x2c0 [ 5964.187230] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5964.189336] ? apic_timer_interrupt+0xf/0x20 [ 5964.191002] [ 5964.191925] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5964.198607] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5964.204234] do_raw_spin_lock+0x177/0x190 [ 5964.206933] _raw_spin_lock+0x21/0x30 [ 5964.211165] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5964.215248] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5964.217386] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5964.219937] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5964.222679] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5964.225810] ? woken_wake_function+0x30/0x30 [ 5964.228096] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5964.233628] kthread+0x1d1/0x200 [ 5964.237250] ? set_kthread_struct+0x70/0x70 [ 5964.241811] ret_from_fork+0x1f/0x30 [ 5984.054142] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 5984.069197] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5984.140934] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5984.147552] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5984.157626] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 5984.162433] RIP: 0010:native_safe_halt+0x2e/0x50 [ 5984.168206] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 5984.187036] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5984.191743] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5984.200486] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 5984.213085] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 5984.226933] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 5984.239723] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 5984.256711] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 5984.266899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5984.270208] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 5984.283312] Call Trace: [ 5984.289081] [ 5984.292005] ? show_regs.cold.9+0x22/0x2f [ 5984.298622] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5984.307197] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5984.310195] ? watchdog+0x40/0x40 [ 5984.312866] ? hrtimer_interrupt+0x128/0x2c0 [ 5984.317246] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5984.321327] ? apic_timer_interrupt+0xf/0x20 [ 5984.325070] [ 5984.326163] ? native_safe_halt+0x2e/0x50 [ 5984.330300] kvm_wait+0xcd/0xe0 [ 5984.336669] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5984.346947] ? newidle_balance.isra.93+0x207/0x580 [ 5984.354323] do_raw_spin_lock+0x177/0x190 [ 5984.362271] _raw_spin_lock+0x21/0x30 [ 5984.370502] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 5984.376234] process_one_work+0x2c8/0x700 [ 5984.379233] worker_thread+0x296/0x6e0 [ 5984.384238] ? rescuer_thread+0x570/0x570 [ 5984.388172] kthread+0x1d1/0x200 [ 5984.394097] ? set_kthread_struct+0x70/0x70 [ 5984.398741] ret_from_fork+0x1f/0x30 [ 5988.063978] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 5988.081768] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 5988.095586] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5988.095660] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5988.095669] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5988.095671] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5988.095699] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 5988.095703] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5988.095707] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 5988.095709] RDX: 0000000000001876 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 5988.095711] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 5988.095713] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 5988.095715] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 5988.095718] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 5988.095721] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5988.095724] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 5988.095732] Call Trace: [ 5988.095737] [ 5988.095744] ? show_regs.cold.9+0x22/0x2f [ 5988.095753] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5988.095760] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5988.095764] ? watchdog+0x40/0x40 [ 5988.095770] ? hrtimer_interrupt+0x128/0x2c0 [ 5988.095774] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5988.095783] ? apic_timer_interrupt+0xf/0x20 [ 5988.095788] [ 5988.095791] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 5988.095797] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 5988.095802] do_raw_spin_lock+0x177/0x190 [ 5988.095807] _raw_spin_lock+0x21/0x30 [ 5988.095812] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 5988.096084] obd_mod_free+0x46/0x140 [ptlrpc] [ 5988.096276] mdc_commit_open+0x91/0xb0 [mdc] [ 5988.096310] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 5988.096504] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 5988.096717] after_reply+0xe2c/0x1780 [ptlrpc] [ 5988.144780] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5988.144857] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5988.144864] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5988.144867] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5988.144880] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5988.144885] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 5988.144889] RAX: 0000000000002c1d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5988.144892] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 5988.144894] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5988.144896] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 5988.144899] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 5988.144903] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 5988.144906] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5988.144908] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 5988.144917] Call Trace: [ 5988.144923] [ 5988.144929] ? show_regs.cold.9+0x22/0x2f [ 5988.144939] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5988.144946] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5988.144950] ? watchdog+0x40/0x40 [ 5988.144954] ? hrtimer_interrupt+0x128/0x2c0 [ 5988.144958] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5988.144968] ? apic_timer_interrupt+0xf/0x20 [ 5988.144973] [ 5988.144976] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5988.144983] do_raw_spin_lock+0x177/0x190 [ 5988.144987] _raw_spin_lock+0x21/0x30 [ 5988.144992] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 5988.145272] ? do_raw_spin_unlock+0x75/0x190 [ 5988.145278] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 5988.145492] ? do_wait_intr+0xf0/0xf0 [ 5988.145502] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 5988.145730] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 5988.145955] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5988.213269] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 5988.213783] ? __next_timer_interrupt+0x160/0x160 [ 5988.213793] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 5988.214161] ? do_wait_intr+0xf0/0xf0 [ 5988.214173] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 5988.214566] kthread+0x1d1/0x200 [ 5988.214575] ? set_kthread_struct+0x70/0x70 [ 5988.214579] ret_from_fork+0x1f/0x30 [ 5988.331657] BUG: workqueue lockup - pool [ 5988.335448] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5988.337214] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5988.338091] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 5988.338130] mdc_intent_lock+0x353/0x7e0 [mdc] [ 5988.338157] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5988.338264] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 5988.355849] cpus=0-3 flags=0x4 nice=0 stuck for 3351s! [ 5988.355873] Showing busy workqueues and worker pools: [ 5988.355876] workqueue events: flags=0x0 [ 5988.355879] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 5988.355887] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 5988.356349] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5988.356360] pending: cpu_latency_work [ptlrpc] [ 5988.356840] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 5988.356850] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 5988.357193] workqueue events_long: flags=0x0 [ 5988.357197] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 5988.357205] pending: lu_objects_flush [obdclass] [ 5988.357447] workqueue events_unbound: flags=0x2 [ 5988.357449] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 5988.357457] pending: flush_memcg_stats_dwork, idle_cull_fn [ 5988.357470] workqueue events_power_efficient: flags=0x80 [ 5988.357473] pwq 4: [ 5988.361863] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 5988.361916] lmv_intent_open+0x338/0x1150 [lmv] [ 5988.361950] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5988.362263] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5988.362352] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 5988.362397] ll_intent_lock+0xc5/0xaa0 [lustre] [ 5988.362873] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 5988.362959] ll_lookup_it+0xe0e/0x2270 [lustre] [ 5988.363123] ? do_raw_spin_unlock+0x75/0x190 [ 5988.363131] ? _raw_spin_unlock+0x12/0x30 [ 5988.363139] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 5988.363168] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 5988.363188] ll_atomic_open+0x36a/0xd70 [lustre] [ 5988.363279] lookup_open+0x356/0xb40 [ 5988.363288] do_last+0x313/0xfc0 [ 5988.363293] ? nd_jump_root+0xe5/0x160 [ 5988.363298] ? path_init+0x437/0x520 [ 5988.363303] path_openat+0xf7/0x500 [ 5988.363308] ? do_raw_spin_unlock+0x75/0x190 [ 5988.363313] do_filp_open+0x99/0x140 [ 5988.363320] ? getname_flags+0x6e/0x330 [ 5988.363325] ? __check_object_size+0xff/0x256 [ 5988.363333] ? do_raw_spin_unlock+0x75/0x190 [ 5988.363337] ? _raw_spin_unlock+0x12/0x30 [ 5988.363342] do_sys_openat2+0x2b4/0x410 [ 5988.363347] do_sys_open+0x73/0xa0 [ 5988.363352] __x64_sys_openat+0x24/0x30 [ 5988.363356] do_syscall_64+0xc1/0x440 [ 5988.363364] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 5988.363437] RIP: 0033:0x7f53425a5332 [ 5988.363447] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 5988.363451] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 5988.363457] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 5988.363460] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 5988.363462] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 5988.363464] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 5988.363467] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 5988.729930] cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5988.732138] pending: check_lifetime, do_cache_clean [ 5988.734396] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5988.737111] pending: neigh_periodic_work, neigh_periodic_work [ 5988.739074] workqueue mm_percpu_wq: flags=0x8 [ 5988.740414] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5988.742583] pending: vmstat_update, lru_add_drain_per_cpu [ 5988.745202] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 5988.747353] pending: vmstat_update, lru_add_drain_per_cpu [ 5988.749461] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 5988.756595] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 5988.759336] workqueue xprtiod: flags=0xa [ 5988.761651] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 5988.764066] pending: xprt_autoclose, xs_error_handle [ 5988.766637] workqueue ipv6_addrconf: flags=0x40008 [ 5988.769428] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 5988.772633] pending: addrconf_verify_work [ 5988.774903] workqueue knbd0-recv: flags=0x1a [ 5988.776802] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 5988.779760] in-flight: 651:recv_work [ 5988.781587] workqueue ptlrpc_pinger: flags=0x2 [ 5988.783608] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 5988.786394] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 5988.789892] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3351s workers=4 idle: 15979 31824 8 [ 5988.793902] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=5979s workers=2 idle: 654 [ 5992.064717] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 5992.074152] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 5992.126986] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 5992.151941] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 5992.160643] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5992.168478] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 5992.195499] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 5992.211757] RAX: 000000000000658e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 5992.216640] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 5992.224865] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 5992.236905] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 5992.243838] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 5992.252984] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 5992.259276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5992.263913] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 5992.270631] Call Trace: [ 5992.271322] [ 5992.276393] ? show_regs.cold.9+0x22/0x2f [ 5992.278271] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 5992.282602] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 5992.285296] ? watchdog+0x40/0x40 [ 5992.287007] ? hrtimer_interrupt+0x128/0x2c0 [ 5992.289568] ? smp_apic_timer_interrupt+0xa6/0x270 [ 5992.291444] ? apic_timer_interrupt+0xf/0x20 [ 5992.293830] [ 5992.294619] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 5992.297099] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 5992.305326] do_raw_spin_lock+0x177/0x190 [ 5992.307001] _raw_spin_lock+0x21/0x30 [ 5992.309469] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 5992.314096] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 5992.316393] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 5992.320783] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 5992.322447] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 5992.332099] ? woken_wake_function+0x30/0x30 [ 5992.337516] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 5992.342148] kthread+0x1d1/0x200 [ 5992.343232] ? set_kthread_struct+0x70/0x70 [ 5992.347706] ret_from_fork+0x1f/0x30 [ 6016.052288] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 6016.055036] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 6016.058670] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 6016.058685] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6016.058758] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6016.058764] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6016.058767] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6016.058780] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6016.058785] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6016.058789] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6016.058791] RDX: 0000000000007e12 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6016.058793] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6016.058795] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6016.058797] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6016.058800] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6016.058803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6016.058805] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6016.058813] Call Trace: [ 6016.058819] [ 6016.058825] ? show_regs.cold.9+0x22/0x2f [ 6016.058834] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6016.058842] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6016.058846] ? watchdog+0x40/0x40 [ 6016.058850] ? hrtimer_interrupt+0x128/0x2c0 [ 6016.058854] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6016.058862] ? apic_timer_interrupt+0xf/0x20 [ 6016.058867] [ 6016.058869] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6016.058874] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6016.058878] do_raw_spin_lock+0x177/0x190 [ 6016.058883] _raw_spin_lock+0x21/0x30 [ 6016.058889] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6016.059325] obd_mod_free+0x46/0x140 [ptlrpc] [ 6016.059598] mdc_commit_open+0x91/0xb0 [mdc] [ 6016.059635] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6016.059827] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6016.060388] after_reply+0xe2c/0x1780 [ptlrpc] [ 6016.060583] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6016.060771] ? __next_timer_interrupt+0x160/0x160 [ 6016.060780] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6016.061174] ? do_wait_intr+0xf0/0xf0 [ 6016.061182] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6016.061399] kthread+0x1d1/0x200 [ 6016.061405] ? set_kthread_struct+0x70/0x70 [ 6016.061409] ret_from_fork+0x1f/0x30 [ 6016.064553] Modules linked in: [ 6016.069576] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6016.069663] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6016.069670] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6016.069673] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6016.069688] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6016.069693] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6016.069697] RAX: 0000000000007e16 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6016.069700] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6016.069702] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6016.069703] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6016.069705] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6016.069709] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6016.069712] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6016.069714] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6016.069723] Call Trace: [ 6016.069729] [ 6016.069734] ? show_regs.cold.9+0x22/0x2f [ 6016.069744] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6016.069750] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6016.069755] ? watchdog+0x40/0x40 [ 6016.069759] ? hrtimer_interrupt+0x128/0x2c0 [ 6016.069763] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6016.069771] ? apic_timer_interrupt+0xf/0x20 [ 6016.069776] [ 6016.069779] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6016.069786] do_raw_spin_lock+0x177/0x190 [ 6016.069791] _raw_spin_lock+0x21/0x30 [ 6016.069797] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6016.075132] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6016.075220] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6016.075228] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6016.075235] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6016.075498] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6016.075510] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6016.075515] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6016.075519] RAX: 0000000000007a86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6016.075522] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6016.075524] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6016.075526] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6016.075529] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6016.075533] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6016.075535] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6016.075538] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6016.075547] Call Trace: [ 6016.075552] [ 6016.075558] ? show_regs.cold.9+0x22/0x2f [ 6016.075568] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6016.075576] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6016.075580] ? watchdog+0x40/0x40 [ 6016.075584] ? hrtimer_interrupt+0x128/0x2c0 [ 6016.075587] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6016.075595] ? apic_timer_interrupt+0xf/0x20 [ 6016.075706] [ 6016.075710] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6016.075715] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6016.075719] ? newidle_balance.isra.93+0x207/0x580 [ 6016.075724] do_raw_spin_lock+0x177/0x190 [ 6016.075728] _raw_spin_lock+0x21/0x30 [ 6016.075732] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6016.076412] process_one_work+0x2c8/0x700 [ 6016.076419] worker_thread+0x296/0x6e0 [ 6016.076423] ? rescuer_thread+0x570/0x570 [ 6016.076426] kthread+0x1d1/0x200 [ 6016.076431] ? set_kthread_struct+0x70/0x70 [ 6016.076435] ret_from_fork+0x1f/0x30 [ 6016.436576] ? do_raw_spin_unlock+0x75/0x190 [ 6016.440880] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6016.443903] ? do_wait_intr+0xf0/0xf0 [ 6016.445468] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6016.447705] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6016.450414] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6016.452992] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6016.456335] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6016.461248] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6016.462926] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6016.466042] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6016.468683] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6016.473053] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6016.475645] lmv_intent_open+0x338/0x1150 [lmv] [ 6016.478831] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6016.481679] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6016.484650] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6016.486792] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6016.488644] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6016.491422] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6016.493856] ? do_raw_spin_unlock+0x75/0x190 [ 6016.496000] ? _raw_spin_unlock+0x12/0x30 [ 6016.497580] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6016.500556] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6016.507669] ll_atomic_open+0x36a/0xd70 [lustre] [ 6016.510854] lookup_open+0x356/0xb40 [ 6016.513303] do_last+0x313/0xfc0 [ 6016.514678] ? nd_jump_root+0xe5/0x160 [ 6016.517316] ? path_init+0x437/0x520 [ 6016.519127] path_openat+0xf7/0x500 [ 6016.525118] ? do_raw_spin_unlock+0x75/0x190 [ 6016.528672] do_filp_open+0x99/0x140 [ 6016.530468] ? getname_flags+0x6e/0x330 [ 6016.532381] ? __check_object_size+0xff/0x256 [ 6016.534352] ? do_raw_spin_unlock+0x75/0x190 [ 6016.536322] ? _raw_spin_unlock+0x12/0x30 [ 6016.538540] do_sys_openat2+0x2b4/0x410 [ 6016.540497] do_sys_open+0x73/0xa0 [ 6016.542056] __x64_sys_openat+0x24/0x30 [ 6016.543979] do_syscall_64+0xc1/0x440 [ 6016.545855] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6016.550567] RIP: 0033:0x7f53425a5332 [ 6016.558834] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6016.573315] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6016.576631] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6016.583479] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6016.587603] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6016.593777] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6016.597052] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6019.043944] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3381s! [ 6019.046797] Showing busy workqueues and worker pools: [ 6019.059110] workqueue events: flags=0x0 [ 6019.060572] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6019.074496] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6019.089659] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6019.096414] pending: cpu_latency_work [ptlrpc] [ 6019.100978] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6019.104405] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6019.109663] workqueue events_long: flags=0x0 [ 6019.110789] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6019.114192] pending: lu_objects_flush [obdclass] [ 6019.116895] workqueue events_unbound: flags=0x2 [ 6019.121544] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6019.125770] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6019.128040] workqueue events_power_efficient: flags=0x80 [ 6019.130195] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6019.132484] pending: check_lifetime, do_cache_clean [ 6019.139513] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6019.143370] pending: neigh_periodic_work, neigh_periodic_work [ 6019.148120] workqueue mm_percpu_wq: flags=0x8 [ 6019.149640] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6019.157580] pending: vmstat_update, lru_add_drain_per_cpu [ 6019.159829] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6019.162575] pending: vmstat_update, lru_add_drain_per_cpu [ 6019.165264] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6019.168031] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6019.170738] workqueue xprtiod: flags=0xa [ 6019.172795] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6019.175065] pending: xprt_autoclose, xs_error_handle [ 6019.177107] workqueue ipv6_addrconf: flags=0x40008 [ 6019.178793] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6019.181193] pending: addrconf_verify_work [ 6019.182798] workqueue knbd0-recv: flags=0x1a [ 6019.184605] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6019.187034] in-flight: 651:recv_work [ 6019.188613] workqueue ptlrpc_pinger: flags=0x2 [ 6019.190338] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6019.192634] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6019.195210] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3382s workers=4 idle: 15979 31824 8 [ 6019.198067] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6009s workers=2 idle: 654 [ 6020.060993] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 6020.064014] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6020.100682] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6020.122302] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6020.130365] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6020.131994] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6020.154914] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6020.160047] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6020.162750] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6020.172604] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6020.180088] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6020.187022] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6020.194380] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6020.198024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6020.200886] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6020.216174] Call Trace: [ 6020.218980] [ 6020.221606] ? show_regs.cold.9+0x22/0x2f [ 6020.225519] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6020.232435] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6020.239478] ? watchdog+0x40/0x40 [ 6020.243100] ? hrtimer_interrupt+0x128/0x2c0 [ 6020.249100] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6020.252863] ? apic_timer_interrupt+0xf/0x20 [ 6020.255093] [ 6020.266592] ? native_safe_halt+0x2e/0x50 [ 6020.272806] kvm_wait+0xcd/0xe0 [ 6020.274105] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6020.280848] do_raw_spin_lock+0x177/0x190 [ 6020.284475] _raw_spin_lock+0x21/0x30 [ 6020.286129] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6020.290964] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6020.293794] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6020.301336] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6020.304020] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6020.313414] ? woken_wake_function+0x30/0x30 [ 6020.316008] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6020.322778] kthread+0x1d1/0x200 [ 6020.325996] ? set_kthread_struct+0x70/0x70 [ 6020.327521] ret_from_fork+0x1f/0x30 [ 6044.052254] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 6044.055076] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 6044.064789] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 6044.064802] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6044.064875] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6044.064880] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6044.064883] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6044.064896] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6044.064900] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6044.064905] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6044.064907] RDX: 0000000000000723 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6044.064910] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6044.064912] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6044.064915] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6044.064918] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6044.064920] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6044.064923] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6044.064931] Call Trace: [ 6044.064937] [ 6044.064943] ? show_regs.cold.9+0x22/0x2f [ 6044.064951] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6044.064958] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6044.064962] ? watchdog+0x40/0x40 [ 6044.064966] ? hrtimer_interrupt+0x128/0x2c0 [ 6044.064970] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6044.064978] ? apic_timer_interrupt+0xf/0x20 [ 6044.064982] [ 6044.064985] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6044.064990] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6044.064995] do_raw_spin_lock+0x177/0x190 [ 6044.065000] _raw_spin_lock+0x21/0x30 [ 6044.065005] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6044.069013] obd_mod_free+0x46/0x140 [ptlrpc] [ 6044.071220] Modules linked in: [ 6044.071337] mdc_commit_open+0x91/0xb0 [mdc] [ 6044.071373] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6044.071613] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6044.075293] Modules linked in: [ 6044.076078] after_reply+0xe2c/0x1780 [ptlrpc] [ 6044.078631] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6044.079005] lustre(O) [ 6044.079326] ? __next_timer_interrupt+0x160/0x160 [ 6044.079337] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6044.079544] ? do_wait_intr+0xf0/0xf0 [ 6044.079554] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6044.079766] kthread+0x1d1/0x200 [ 6044.079773] ? set_kthread_struct+0x70/0x70 [ 6044.079791] ret_from_fork+0x1f/0x30 [ 6044.099264] lustre(O) [ 6044.113552] osp(O) [ 6044.117425] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6044.117559] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6044.117566] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6044.117568] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6044.117582] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6044.117586] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6044.117590] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6044.117592] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6044.117595] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6044.117597] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6044.117599] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6044.117603] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6044.117606] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6044.117609] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6044.117618] Call Trace: [ 6044.117622] [ 6044.117627] ? show_regs.cold.9+0x22/0x2f [ 6044.117636] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6044.117642] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6044.117646] ? watchdog+0x40/0x40 [ 6044.117650] ? hrtimer_interrupt+0x128/0x2c0 [ 6044.117655] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6044.117661] ? apic_timer_interrupt+0xf/0x20 [ 6044.117665] [ 6044.117668] ? native_safe_halt+0x2e/0x50 [ 6044.117672] kvm_wait+0xcd/0xe0 [ 6044.117677] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6044.117684] do_raw_spin_lock+0x177/0x190 [ 6044.117688] _raw_spin_lock+0x21/0x30 [ 6044.117692] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6044.117972] ? do_raw_spin_unlock+0x75/0x190 [ 6044.117978] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6044.118181] ? do_wait_intr+0xf0/0xf0 [ 6044.118189] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6044.118389] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6044.118829] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6044.119048] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6044.119168] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6044.119207] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6044.119234] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6044.119261] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6044.119344] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6044.119627] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6044.119655] lmv_intent_open+0x338/0x1150 [lmv] [ 6044.119685] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6044.133207] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6044.133284] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6044.133291] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6044.133297] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6044.141039] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6044.141165] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6044.141200] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6044.141285] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6044.141368] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6044.141454] ? do_raw_spin_unlock+0x75/0x190 [ 6044.141462] ? _raw_spin_unlock+0x12/0x30 [ 6044.141468] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6044.141496] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6044.141517] ll_atomic_open+0x36a/0xd70 [lustre] [ 6044.141601] lookup_open+0x356/0xb40 [ 6044.141609] do_last+0x313/0xfc0 [ 6044.141614] ? nd_jump_root+0xe5/0x160 [ 6044.141619] ? path_init+0x437/0x520 [ 6044.141623] path_openat+0xf7/0x500 [ 6044.141628] ? do_raw_spin_unlock+0x75/0x190 [ 6044.141633] do_filp_open+0x99/0x140 [ 6044.141639] ? getname_flags+0x6e/0x330 [ 6044.141643] ? __check_object_size+0xff/0x256 [ 6044.141649] ? do_raw_spin_unlock+0x75/0x190 [ 6044.141653] ? _raw_spin_unlock+0x12/0x30 [ 6044.141657] do_sys_openat2+0x2b4/0x410 [ 6044.141661] do_sys_open+0x73/0xa0 [ 6044.141666] __x64_sys_openat+0x24/0x30 [ 6044.141669] do_syscall_64+0xc1/0x440 [ 6044.141677] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6044.141683] RIP: 0033:0x7f53425a5332 [ 6044.141690] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6044.141694] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6044.141699] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6044.141850] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6044.141854] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6044.141856] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6044.141858] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6045.005268] [ 6045.010957] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6045.015349] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6045.043194] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6045.053421] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6045.066865] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6045.076583] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6045.087758] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6045.099551] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6045.110965] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6045.119988] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6045.130813] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6045.133220] Call Trace: [ 6045.146169] [ 6045.149838] ? show_regs.cold.9+0x22/0x2f [ 6045.154972] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6045.159913] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6045.169082] ? watchdog+0x40/0x40 [ 6045.183846] ? hrtimer_interrupt+0x128/0x2c0 [ 6045.192751] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6045.197302] ? apic_timer_interrupt+0xf/0x20 [ 6045.199977] [ 6045.201278] ? native_safe_halt+0x2e/0x50 [ 6045.211248] kvm_wait+0xcd/0xe0 [ 6045.215222] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6045.221855] ? newidle_balance.isra.93+0x207/0x580 [ 6045.225327] do_raw_spin_lock+0x177/0x190 [ 6045.229932] _raw_spin_lock+0x21/0x30 [ 6045.235138] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6045.245043] process_one_work+0x2c8/0x700 [ 6045.249926] worker_thread+0x296/0x6e0 [ 6045.254208] ? rescuer_thread+0x570/0x570 [ 6045.259958] kthread+0x1d1/0x200 [ 6045.262467] ? set_kthread_struct+0x70/0x70 [ 6045.265892] ret_from_fork+0x1f/0x30 [ 6048.058017] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 6048.061260] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6048.083614] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6048.089816] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6048.098532] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6048.105889] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6048.118623] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6048.126765] RAX: 0000000000003222 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6048.133194] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6048.136884] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6048.142602] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6048.146076] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6048.152954] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6048.162293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6048.165743] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6048.172556] Call Trace: [ 6048.173922] [ 6048.174837] ? show_regs.cold.9+0x22/0x2f [ 6048.177036] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6048.180482] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6048.186812] ? watchdog+0x40/0x40 [ 6048.192817] ? hrtimer_interrupt+0x128/0x2c0 [ 6048.200064] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6048.201865] ? apic_timer_interrupt+0xf/0x20 [ 6048.203360] [ 6048.216182] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6048.224239] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6048.226534] do_raw_spin_lock+0x177/0x190 [ 6048.235112] _raw_spin_lock+0x21/0x30 [ 6048.240222] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6048.248275] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6048.253155] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6048.259113] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6048.267784] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6048.276669] ? woken_wake_function+0x30/0x30 [ 6048.278857] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6048.288125] kthread+0x1d1/0x200 [ 6048.289869] ? set_kthread_struct+0x70/0x70 [ 6048.296793] ret_from_fork+0x1f/0x30 [ 6049.251227] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3412s! [ 6049.263469] Showing busy workqueues and worker pools: [ 6049.266896] workqueue events: flags=0x0 [ 6049.269501] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6049.273096] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6049.279700] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6049.282191] pending: cpu_latency_work [ptlrpc] [ 6049.284452] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6049.287974] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6049.292316] workqueue events_long: flags=0x0 [ 6049.294409] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6049.297576] pending: lu_objects_flush [obdclass] [ 6049.300859] workqueue events_unbound: flags=0x2 [ 6049.304193] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6049.308032] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6049.311991] workqueue events_power_efficient: flags=0x80 [ 6049.315960] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6049.333573] pending: check_lifetime, do_cache_clean [ 6049.340453] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6049.349140] pending: neigh_periodic_work, neigh_periodic_work [ 6049.351400] workqueue mm_percpu_wq: flags=0x8 [ 6049.358996] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6049.362164] pending: vmstat_update, lru_add_drain_per_cpu [ 6049.365056] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6049.367923] pending: vmstat_update, lru_add_drain_per_cpu [ 6049.373288] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6049.388022] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6049.396942] workqueue xprtiod: flags=0xa [ 6049.398049] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6049.404095] pending: xprt_autoclose, xs_error_handle [ 6049.407328] workqueue ipv6_addrconf: flags=0x40008 [ 6049.409953] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6049.413563] pending: addrconf_verify_work [ 6049.415825] workqueue knbd0-recv: flags=0x1a [ 6049.417726] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6049.420096] in-flight: 651:recv_work [ 6049.421557] workqueue ptlrpc_pinger: flags=0x2 [ 6049.422766] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6049.424862] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6049.427078] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3412s workers=4 idle: 15979 31824 8 [ 6049.429584] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6039s workers=2 idle: 654 [ 6072.056345] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 6072.060243] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6072.064346] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 6072.099763] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6072.099781] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6072.099784] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6072.099800] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6072.099805] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6072.099811] RAX: 0000000000007d85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6072.099813] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6072.099816] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6072.099818] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6072.099821] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6072.099825] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6072.099828] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6072.099830] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6072.099905] Call Trace: [ 6072.099912] [ 6072.099917] ? show_regs.cold.9+0x22/0x2f [ 6072.099927] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6072.099934] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6072.099938] ? watchdog+0x40/0x40 [ 6072.099941] ? hrtimer_interrupt+0x128/0x2c0 [ 6072.099945] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6072.099954] ? apic_timer_interrupt+0xf/0x20 [ 6072.099959] [ 6072.099962] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6072.099970] do_raw_spin_lock+0x177/0x190 [ 6072.099974] _raw_spin_lock+0x21/0x30 [ 6072.099979] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6072.100936] ? do_raw_spin_unlock+0x75/0x190 [ 6072.100943] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6072.101441] ? do_wait_intr+0xf0/0xf0 [ 6072.101450] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6072.101729] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6072.102017] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6072.103812] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6072.103950] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6072.103991] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6072.104020] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6072.104044] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6072.104124] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6072.105790] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6072.105824] lmv_intent_open+0x338/0x1150 [lmv] [ 6072.119369] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6072.119527] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6072.119534] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6072.119537] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6072.119550] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6072.119555] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6072.119559] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6072.119562] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6072.119564] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6072.119567] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6072.119570] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6072.119572] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6072.119576] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6072.119578] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6072.119588] Call Trace: [ 6072.119594] [ 6072.119599] ? show_regs.cold.9+0x22/0x2f [ 6072.119610] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6072.119616] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6072.119621] ? watchdog+0x40/0x40 [ 6072.119625] ? hrtimer_interrupt+0x128/0x2c0 [ 6072.119629] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6072.119636] ? apic_timer_interrupt+0xf/0x20 [ 6072.119640] [ 6072.119643] ? native_safe_halt+0x2e/0x50 [ 6072.119646] kvm_wait+0xcd/0xe0 [ 6072.119653] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6072.119660] do_raw_spin_lock+0x177/0x190 [ 6072.119664] _raw_spin_lock+0x21/0x30 [ 6072.119669] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6072.147535] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6072.147652] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6072.147730] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6072.147761] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6072.147847] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6072.147925] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6072.148003] ? do_raw_spin_unlock+0x75/0x190 [ 6072.148011] ? _raw_spin_unlock+0x12/0x30 [ 6072.148019] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6072.148045] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6072.148064] ll_atomic_open+0x36a/0xd70 [lustre] [ 6072.148356] lookup_open+0x356/0xb40 [ 6072.148365] do_last+0x313/0xfc0 [ 6072.148370] ? nd_jump_root+0xe5/0x160 [ 6072.148376] ? path_init+0x437/0x520 [ 6072.148380] path_openat+0xf7/0x500 [ 6072.148385] ? do_raw_spin_unlock+0x75/0x190 [ 6072.148390] do_filp_open+0x99/0x140 [ 6072.148396] ? getname_flags+0x6e/0x330 [ 6072.148401] ? __check_object_size+0xff/0x256 [ 6072.148408] ? do_raw_spin_unlock+0x75/0x190 [ 6072.148412] ? _raw_spin_unlock+0x12/0x30 [ 6072.148417] do_sys_openat2+0x2b4/0x410 [ 6072.148421] do_sys_open+0x73/0xa0 [ 6072.148426] __x64_sys_openat+0x24/0x30 [ 6072.148429] do_syscall_64+0xc1/0x440 [ 6072.148435] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6072.148441] RIP: 0033:0x7f53425a5332 [ 6072.148448] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6072.148451] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6072.148456] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6072.148564] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6072.148567] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6072.148569] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6072.148571] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6072.605739] obd_mod_free+0x46/0x140 [ptlrpc] [ 6072.607917] mdc_commit_open+0x91/0xb0 [mdc] [ 6072.610261] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6072.613122] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6072.616820] after_reply+0xe2c/0x1780 [ptlrpc] [ 6072.619680] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6072.622812] ? __next_timer_interrupt+0x160/0x160 [ 6072.625027] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6072.627939] ? do_wait_intr+0xf0/0xf0 [ 6072.629746] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6072.632765] kthread+0x1d1/0x200 [ 6072.634715] ? set_kthread_struct+0x70/0x70 [ 6072.636718] ret_from_fork+0x1f/0x30 [ 6076.052013] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 6076.055066] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6076.059203] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 6076.074566] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6076.074582] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6076.074589] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6076.074895] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6076.074905] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6076.074909] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6076.074914] RAX: 00000000000040b9 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6076.074916] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6076.074919] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6076.074921] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6076.074923] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6076.074926] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6076.074929] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6076.074931] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6076.074940] Call Trace: [ 6076.074947] [ 6076.074952] ? show_regs.cold.9+0x22/0x2f [ 6076.074962] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6076.074968] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6076.074972] ? watchdog+0x40/0x40 [ 6076.074975] ? hrtimer_interrupt+0x128/0x2c0 [ 6076.074979] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6076.074987] ? apic_timer_interrupt+0xf/0x20 [ 6076.074992] [ 6076.074995] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6076.075000] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6076.075004] ? newidle_balance.isra.93+0x207/0x580 [ 6076.075009] do_raw_spin_lock+0x177/0x190 [ 6076.075014] _raw_spin_lock+0x21/0x30 [ 6076.075019] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6076.075446] process_one_work+0x2c8/0x700 [ 6076.075454] worker_thread+0x296/0x6e0 [ 6076.075459] ? rescuer_thread+0x570/0x570 [ 6076.075463] kthread+0x1d1/0x200 [ 6076.075468] ? set_kthread_struct+0x70/0x70 [ 6076.075472] ret_from_fork+0x1f/0x30 [ 6076.241333] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6076.274525] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6076.283680] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6076.288069] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6076.294860] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6076.311225] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6076.317038] RAX: 0000000000004091 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6076.322559] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6076.328856] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6076.335343] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6076.345676] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6076.354409] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6076.363508] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6076.370397] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6076.377129] Call Trace: [ 6076.378404] [ 6076.379538] ? show_regs.cold.9+0x22/0x2f [ 6076.385718] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6076.387625] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6076.392701] ? watchdog+0x40/0x40 [ 6076.396228] ? hrtimer_interrupt+0x128/0x2c0 [ 6076.399391] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6076.404022] ? apic_timer_interrupt+0xf/0x20 [ 6076.407549] [ 6076.408955] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6076.415312] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6076.420338] do_raw_spin_lock+0x177/0x190 [ 6076.422597] _raw_spin_lock+0x21/0x30 [ 6076.424856] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6076.430716] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6076.435634] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6076.440260] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6076.442249] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6076.445750] ? woken_wake_function+0x30/0x30 [ 6076.447824] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6076.455152] kthread+0x1d1/0x200 [ 6076.456321] ? set_kthread_struct+0x70/0x70 [ 6076.460621] ret_from_fork+0x1f/0x30 [ 6079.455061] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3442s! [ 6079.461479] Showing busy workqueues and worker pools: [ 6079.464807] workqueue events: flags=0x0 [ 6079.469419] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6079.476053] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6079.485473] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6079.488523] pending: cpu_latency_work [ptlrpc] [ 6079.491296] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6079.494707] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6079.498912] workqueue events_long: flags=0x0 [ 6079.501059] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6079.506787] pending: lu_objects_flush [obdclass] [ 6079.510030] workqueue events_unbound: flags=0x2 [ 6079.514149] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6079.517048] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6079.521593] workqueue events_power_efficient: flags=0x80 [ 6079.524656] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6079.527705] pending: check_lifetime, do_cache_clean [ 6079.531343] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6079.534915] pending: neigh_periodic_work, neigh_periodic_work [ 6079.538156] workqueue mm_percpu_wq: flags=0x8 [ 6079.541237] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6079.544266] pending: vmstat_update, lru_add_drain_per_cpu [ 6079.548024] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6079.551984] pending: vmstat_update, lru_add_drain_per_cpu [ 6079.555681] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6079.559343] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6079.563341] workqueue xprtiod: flags=0xa [ 6079.565062] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6079.568712] pending: xprt_autoclose, xs_error_handle [ 6079.574823] workqueue ipv6_addrconf: flags=0x40008 [ 6079.579436] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6079.582486] pending: addrconf_verify_work [ 6079.583987] workqueue knbd0-recv: flags=0x1a [ 6079.586438] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6079.589394] in-flight: 651:recv_work [ 6079.591216] workqueue ptlrpc_pinger: flags=0x2 [ 6079.592553] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6079.596045] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6079.598966] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3442s workers=4 idle: 15979 31824 8 [ 6079.603372] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6070s workers=2 idle: 654 [ 6100.055037] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 6100.057191] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 6100.062086] Modules linked in: [ 6100.070480] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) [ 6100.074637] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr [ 6100.088713] ksocklnd(O) lnet(O) [ 6100.098570] squashfs [ 6100.098577] libcfs(O) rpcsec_gss_krb5 [ 6100.100104] ata_generic crct10dif_pclmul [ 6100.103347] auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs [ 6100.105198] crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6100.105221] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6100.105228] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6100.105231] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6100.105244] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6100.105249] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6100.105254] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6100.105256] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6100.105258] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6100.105260] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6100.105263] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6100.105266] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6100.105269] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6100.105271] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6100.105280] Call Trace: [ 6100.105295] [ 6100.105301] ? show_regs.cold.9+0x22/0x2f [ 6100.105314] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6100.105324] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6100.105328] ? watchdog+0x40/0x40 [ 6100.105332] ? hrtimer_interrupt+0x128/0x2c0 [ 6100.105336] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6100.105344] ? apic_timer_interrupt+0xf/0x20 [ 6100.105348] [ 6100.105351] ? native_safe_halt+0x2e/0x50 [ 6100.105355] kvm_wait+0xcd/0xe0 [ 6100.105362] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6100.105371] do_raw_spin_lock+0x177/0x190 [ 6100.105376] _raw_spin_lock+0x21/0x30 [ 6100.105381] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6100.105663] ? do_raw_spin_unlock+0x75/0x190 [ 6100.105669] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6100.105890] ? do_wait_intr+0xf0/0xf0 [ 6100.105900] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6100.109345] ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6100.109377] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6100.109384] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6100.109388] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6100.109401] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6100.109405] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6100.109410] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6100.109412] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6100.109416] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6100.117638] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6100.118353] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6100.118588] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6100.118695] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6100.118751] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6100.127647] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6100.127658] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6100.127661] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6100.127664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6100.127667] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6100.127675] Call Trace: [ 6100.127683] [ 6100.127688] ? show_regs.cold.9+0x22/0x2f [ 6100.127700] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6100.127707] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6100.127712] ? watchdog+0x40/0x40 [ 6100.127881] ? hrtimer_interrupt+0x128/0x2c0 [ 6100.127886] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6100.127895] ? apic_timer_interrupt+0xf/0x20 [ 6100.127900] [ 6100.127904] ? native_safe_halt+0x2e/0x50 [ 6100.127910] kvm_wait+0xcd/0xe0 [ 6100.127919] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6100.127926] do_raw_spin_lock+0x177/0x190 [ 6100.127931] _raw_spin_lock+0x21/0x30 [ 6100.127937] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6100.128320] obd_mod_free+0x46/0x140 [ptlrpc] [ 6100.139299] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6100.139354] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6100.139477] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6100.139775] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6100.139806] lmv_intent_open+0x338/0x1150 [lmv] [ 6100.139838] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6100.139932] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6100.140013] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6100.140034] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6100.140122] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6100.140202] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6100.140282] ? do_raw_spin_unlock+0x75/0x190 [ 6100.140290] ? _raw_spin_unlock+0x12/0x30 [ 6100.140297] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6100.140325] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6100.140346] ll_atomic_open+0x36a/0xd70 [lustre] [ 6100.145602] mdc_commit_open+0x91/0xb0 [mdc] [ 6100.145652] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6100.146178] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6100.146442] after_reply+0xe2c/0x1780 [ptlrpc] [ 6100.146654] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6100.147346] ? __next_timer_interrupt+0x160/0x160 [ 6100.147356] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6100.147662] ? do_wait_intr+0xf0/0xf0 [ 6100.147671] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6100.148125] kthread+0x1d1/0x200 [ 6100.148131] ? set_kthread_struct+0x70/0x70 [ 6100.148135] ret_from_fork+0x1f/0x30 [ 6100.440062] lookup_open+0x356/0xb40 [ 6100.441522] do_last+0x313/0xfc0 [ 6100.443763] ? nd_jump_root+0xe5/0x160 [ 6100.447238] ? path_init+0x437/0x520 [ 6100.450690] path_openat+0xf7/0x500 [ 6100.452873] ? do_raw_spin_unlock+0x75/0x190 [ 6100.455343] do_filp_open+0x99/0x140 [ 6100.457815] ? getname_flags+0x6e/0x330 [ 6100.460568] ? __check_object_size+0xff/0x256 [ 6100.462268] ? do_raw_spin_unlock+0x75/0x190 [ 6100.464362] ? _raw_spin_unlock+0x12/0x30 [ 6100.466317] do_sys_openat2+0x2b4/0x410 [ 6100.468232] do_sys_open+0x73/0xa0 [ 6100.470586] __x64_sys_openat+0x24/0x30 [ 6100.472028] do_syscall_64+0xc1/0x440 [ 6100.476234] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6100.492291] RIP: 0033:0x7f53425a5332 [ 6100.493626] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6100.499509] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6100.501176] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6100.503456] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6100.505897] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6100.508191] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6100.512238] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6104.053704] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 6104.058012] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 6104.061019] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6104.062304] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6104.062311] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6104.062318] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6104.062580] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6104.062590] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6104.062594] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6104.062598] RAX: 00000000000053aa RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6104.062600] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6104.062602] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6104.062605] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6104.062607] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6104.062611] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6104.062614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6104.062616] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6104.062625] Call Trace: [ 6104.062630] [ 6104.062635] ? show_regs.cold.9+0x22/0x2f [ 6104.062645] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6104.062664] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6104.062668] ? watchdog+0x40/0x40 [ 6104.062672] ? hrtimer_interrupt+0x128/0x2c0 [ 6104.062676] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6104.062684] ? apic_timer_interrupt+0xf/0x20 [ 6104.062688] [ 6104.062690] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6104.062695] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6104.062699] ? newidle_balance.isra.93+0x207/0x580 [ 6104.062704] do_raw_spin_lock+0x177/0x190 [ 6104.062708] _raw_spin_lock+0x21/0x30 [ 6104.062711] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6104.062908] process_one_work+0x2c8/0x700 [ 6104.062915] worker_thread+0x296/0x6e0 [ 6104.062920] ? rescuer_thread+0x570/0x570 [ 6104.062923] kthread+0x1d1/0x200 [ 6104.062927] ? set_kthread_struct+0x70/0x70 [ 6104.062931] ret_from_fork+0x1f/0x30 [ 6104.287402] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6104.336338] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6104.349800] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6104.355400] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6104.364844] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6104.374037] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6104.379899] RAX: 00000000000058bc RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6104.384052] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6104.391254] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6104.400962] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6104.408198] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6104.417017] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6104.420724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6104.423925] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6104.429042] Call Trace: [ 6104.430256] [ 6104.433871] ? show_regs.cold.9+0x22/0x2f [ 6104.440577] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6104.442274] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6104.450009] ? watchdog+0x40/0x40 [ 6104.451199] ? hrtimer_interrupt+0x128/0x2c0 [ 6104.453207] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6104.457545] ? apic_timer_interrupt+0xf/0x20 [ 6104.460262] [ 6104.461346] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6104.463804] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6104.469814] do_raw_spin_lock+0x177/0x190 [ 6104.475299] _raw_spin_lock+0x21/0x30 [ 6104.479242] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6104.486083] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6104.490872] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6104.493817] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6104.498072] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6104.502003] ? woken_wake_function+0x30/0x30 [ 6104.507691] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6104.514452] kthread+0x1d1/0x200 [ 6104.517081] ? set_kthread_struct+0x70/0x70 [ 6104.519212] ret_from_fork+0x1f/0x30 [ 6109.668800] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3472s! [ 6109.672476] Showing busy workqueues and worker pools: [ 6109.676400] workqueue events: flags=0x0 [ 6109.680265] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6109.682519] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6109.694065] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6109.696334] pending: cpu_latency_work [ptlrpc] [ 6109.704178] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6109.707166] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6109.720853] workqueue events_long: flags=0x0 [ 6109.723160] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6109.730771] pending: lu_objects_flush [obdclass] [ 6109.735303] workqueue events_unbound: flags=0x2 [ 6109.741191] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6109.743282] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6109.759746] workqueue events_power_efficient: flags=0x80 [ 6109.761509] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6109.771010] pending: check_lifetime, do_cache_clean [ 6109.779726] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6109.783364] pending: neigh_periodic_work, neigh_periodic_work [ 6109.790090] workqueue mm_percpu_wq: flags=0x8 [ 6109.799232] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6109.807024] pending: vmstat_update, lru_add_drain_per_cpu [ 6109.809304] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6109.817096] pending: vmstat_update, lru_add_drain_per_cpu [ 6109.825726] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6109.827859] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6109.840051] workqueue xprtiod: flags=0xa [ 6109.845229] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6109.852440] pending: xprt_autoclose, xs_error_handle [ 6109.862740] workqueue ipv6_addrconf: flags=0x40008 [ 6109.864365] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6109.874833] pending: addrconf_verify_work [ 6109.876349] workqueue knbd0-recv: flags=0x1a [ 6109.881640] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6109.884215] in-flight: 651:recv_work [ 6109.886489] workqueue ptlrpc_pinger: flags=0x2 [ 6109.892197] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6109.895142] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6109.899747] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3472s workers=4 idle: 15979 31824 8 [ 6109.908020] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6100s workers=2 idle: 654 [ 6114.524068] rcu: INFO: rcu_sched self-detected stall on CPU [ 6114.530261] rcu: 2-....: (2158037 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 6114.545201] (t=4020098 jiffies g=274185 q=64691) [ 6114.553017] rcu: rcu_sched kthread starved for 3478464 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 6114.563038] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6114.580321] rcu: RCU grace-period kthread stack dump: [ 6114.590834] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6114.603843] Call Trace: [ 6114.607150] __schedule+0x351/0xcb0 [ 6114.612415] schedule+0xc0/0x180 [ 6114.613621] schedule_timeout+0xb4/0x190 [ 6114.630732] ? do_raw_spin_unlock+0x75/0x190 [ 6114.632291] ? __next_timer_interrupt+0x160/0x160 [ 6114.638319] ? prepare_to_swait_event+0xf3/0x140 [ 6114.648969] rcu_gp_kthread+0x8dd/0x15b0 [ 6114.652801] ? force_qs_rnp+0x2c0/0x2c0 [ 6114.655318] kthread+0x1d1/0x200 [ 6114.660882] ? set_kthread_struct+0x70/0x70 [ 6114.666127] ret_from_fork+0x1f/0x30 [ 6114.669019] rcu: Stack dump where RCU GP kthread last ran: [ 6114.670913] Sending NMI from CPU 2 to CPUs 1: [ 6114.680487] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 6114.681561] Sending NMI from CPU 2 to CPUs 1: [ 6114.692180] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 6114.693409] NMI backtrace for cpu 2 [ 6114.706854] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6114.726879] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6114.743479] Call Trace: [ 6114.744355] [ 6114.745106] ? dump_stack+0xbb/0x10e [ 6114.746355] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6114.757580] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6114.764271] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6114.770935] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6114.780312] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6114.791114] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6114.793385] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6114.800454] ? update_process_times+0x6d/0xc0 [ 6114.812374] ? tick_sched_handle.isra.18+0x27/0x90 [ 6114.823830] ? tick_sched_timer+0x53/0xb0 [ 6114.831827] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6114.833552] ? tick_sched_handle.isra.18+0x90/0x90 [ 6114.842352] ? hrtimer_interrupt+0x128/0x2c0 [ 6114.848737] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6114.854032] ? apic_timer_interrupt+0xf/0x20 [ 6114.855303] [ 6114.866752] ? native_safe_halt+0x2e/0x50 [ 6114.867886] ? kvm_wait+0xcd/0xe0 [ 6114.869021] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6114.884875] ? do_raw_spin_lock+0x177/0x190 [ 6114.892154] ? _raw_spin_lock+0x21/0x30 [ 6114.896879] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6114.900136] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6114.905064] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6114.907222] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6114.912194] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6114.917376] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6114.922780] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6114.928339] ? __next_timer_interrupt+0x160/0x160 [ 6114.931073] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6114.932963] ? do_wait_intr+0xf0/0xf0 [ 6114.934126] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6114.937738] ? kthread+0x1d1/0x200 [ 6114.939248] ? set_kthread_struct+0x70/0x70 [ 6114.941116] ? ret_from_fork+0x1f/0x30 [ 6132.062940] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 6132.066537] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6132.092397] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6132.097535] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6132.101155] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6132.104099] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6132.114459] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6132.118426] RAX: 0000000000007923 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6132.121408] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6132.124911] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6132.128528] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6132.131398] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6132.134876] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6132.138894] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6132.142295] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6132.145628] Call Trace: [ 6132.146534] [ 6132.147525] ? show_regs.cold.9+0x22/0x2f [ 6132.148704] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6132.150630] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6132.152947] ? watchdog+0x40/0x40 [ 6132.154068] ? hrtimer_interrupt+0x128/0x2c0 [ 6132.156146] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6132.157855] ? apic_timer_interrupt+0xf/0x20 [ 6132.159625] [ 6132.160431] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6132.164673] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6132.166412] do_raw_spin_lock+0x177/0x190 [ 6132.168047] _raw_spin_lock+0x21/0x30 [ 6132.170883] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6132.174305] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6132.177288] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6132.181510] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6132.183593] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6132.186072] ? woken_wake_function+0x30/0x30 [ 6132.187825] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6132.191859] kthread+0x1d1/0x200 [ 6132.192899] ? set_kthread_struct+0x70/0x70 [ 6132.194561] ret_from_fork+0x1f/0x30 [ 6136.052014] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 6136.058054] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6136.100231] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6136.113168] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6136.124260] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6136.138923] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6136.141031] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6136.171728] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6136.175497] RAX: 0000000000007c89 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6136.179166] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6136.182633] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6136.185827] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6136.190684] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6136.198023] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6136.206392] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6136.214473] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6136.225250] Call Trace: [ 6136.228894] [ 6136.229911] ? show_regs.cold.9+0x22/0x2f [ 6136.232275] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6136.236419] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6136.243896] ? watchdog+0x40/0x40 [ 6136.248546] ? hrtimer_interrupt+0x128/0x2c0 [ 6136.254610] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6136.257967] ? apic_timer_interrupt+0xf/0x20 [ 6136.259552] [ 6136.261142] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6136.264643] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6136.271884] ? newidle_balance.isra.93+0x207/0x580 [ 6136.283621] do_raw_spin_lock+0x177/0x190 [ 6136.285100] _raw_spin_lock+0x21/0x30 [ 6136.290863] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6136.295898] process_one_work+0x2c8/0x700 [ 6136.297557] worker_thread+0x296/0x6e0 [ 6136.314497] ? rescuer_thread+0x570/0x570 [ 6136.317277] kthread+0x1d1/0x200 [ 6136.320842] ? set_kthread_struct+0x70/0x70 [ 6136.329066] ret_from_fork+0x1f/0x30 [ 6140.055035] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 6140.057346] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 6140.062719] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6140.062867] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6140.062875] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6140.062932] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6140.062946] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6140.062950] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6140.062955] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6140.062957] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6140.062959] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6140.062961] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6140.062964] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6140.062967] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6140.062969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6140.062971] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6140.062980] Call Trace: [ 6140.062986] [ 6140.062991] ? show_regs.cold.9+0x22/0x2f [ 6140.063000] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6140.063006] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6140.063010] ? watchdog+0x40/0x40 [ 6140.063014] ? hrtimer_interrupt+0x128/0x2c0 [ 6140.063028] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6140.063036] ? apic_timer_interrupt+0xf/0x20 [ 6140.063042] [ 6140.063045] ? native_safe_halt+0x2e/0x50 [ 6140.063050] kvm_wait+0xcd/0xe0 [ 6140.063058] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6140.063068] do_raw_spin_lock+0x177/0x190 [ 6140.063072] _raw_spin_lock+0x21/0x30 [ 6140.063076] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6140.078515] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6140.079206] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6140.079213] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6140.079216] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6140.079230] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6140.079235] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6140.079304] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6140.079306] RDX: 0000000000001a1e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6140.079309] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6140.079311] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6140.079313] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6140.079317] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6140.079320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6140.079322] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6140.079330] Call Trace: [ 6140.079336] [ 6140.079343] ? show_regs.cold.9+0x22/0x2f [ 6140.079352] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6140.079358] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6140.079362] ? watchdog+0x40/0x40 [ 6140.079366] ? hrtimer_interrupt+0x128/0x2c0 [ 6140.079370] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6140.079380] ? apic_timer_interrupt+0xf/0x20 [ 6140.079385] [ 6140.079387] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6140.079392] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6140.079396] do_raw_spin_lock+0x177/0x190 [ 6140.079401] _raw_spin_lock+0x21/0x30 [ 6140.079407] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6140.116357] ? do_raw_spin_unlock+0x75/0x190 [ 6140.116379] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6140.116651] ? do_wait_intr+0xf0/0xf0 [ 6140.116661] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6140.116843] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6140.126978] obd_mod_free+0x46/0x140 [ptlrpc] [ 6140.137617] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6140.142104] mdc_commit_open+0x91/0xb0 [mdc] [ 6140.142152] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6140.156654] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6140.156778] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6140.156823] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6140.156857] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6140.156889] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6140.156974] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6140.163422] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6140.164756] after_reply+0xe2c/0x1780 [ptlrpc] [ 6140.171393] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6140.171446] lmv_intent_open+0x338/0x1150 [lmv] [ 6140.171478] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6140.171590] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6140.171680] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6140.171700] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6140.172292] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6140.172382] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6140.172469] ? do_raw_spin_unlock+0x75/0x190 [ 6140.172479] ? _raw_spin_unlock+0x12/0x30 [ 6140.172486] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6140.172513] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6140.172533] ll_atomic_open+0x36a/0xd70 [lustre] [ 6140.172617] lookup_open+0x356/0xb40 [ 6140.172628] do_last+0x313/0xfc0 [ 6140.172633] ? nd_jump_root+0xe5/0x160 [ 6140.172638] ? path_init+0x437/0x520 [ 6140.172642] path_openat+0xf7/0x500 [ 6140.172648] ? do_raw_spin_unlock+0x75/0x190 [ 6140.172652] do_filp_open+0x99/0x140 [ 6140.172658] ? getname_flags+0x6e/0x330 [ 6140.172662] ? __check_object_size+0xff/0x256 [ 6140.172669] ? do_raw_spin_unlock+0x75/0x190 [ 6140.172672] ? _raw_spin_unlock+0x12/0x30 [ 6140.172676] do_sys_openat2+0x2b4/0x410 [ 6140.172681] do_sys_open+0x73/0xa0 [ 6140.172686] __x64_sys_openat+0x24/0x30 [ 6140.172689] do_syscall_64+0xc1/0x440 [ 6140.172696] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6140.172701] RIP: 0033:0x7f53425a5332 [ 6140.172710] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6140.172714] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6140.172719] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6140.172721] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6140.172723] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6140.172725] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6140.172728] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6140.818812] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6140.834457] ? __next_timer_interrupt+0x160/0x160 [ 6140.844966] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6140.850647] ? do_wait_intr+0xf0/0xf0 [ 6140.851885] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6140.867289] kthread+0x1d1/0x200 [ 6140.874376] ? set_kthread_struct+0x70/0x70 [ 6140.879376] ret_from_fork+0x1f/0x30 [ 6140.884572] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3503s! [ 6140.890705] Showing busy workqueues and worker pools: [ 6140.897030] workqueue events: flags=0x0 [ 6140.900817] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6140.907742] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6140.914207] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6140.921748] pending: cpu_latency_work [ptlrpc] [ 6140.929526] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6140.933065] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6140.956501] workqueue events_long: flags=0x0 [ 6140.965718] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6140.971859] pending: lu_objects_flush [obdclass] [ 6140.977976] workqueue events_unbound: flags=0x2 [ 6140.985381] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6140.996100] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6141.001376] workqueue events_power_efficient: flags=0x80 [ 6141.010572] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6141.016159] pending: check_lifetime, do_cache_clean [ 6141.019645] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6141.027742] pending: neigh_periodic_work, neigh_periodic_work [ 6141.034347] workqueue mm_percpu_wq: flags=0x8 [ 6141.035874] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6141.041028] pending: vmstat_update, lru_add_drain_per_cpu [ 6141.048821] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6141.054874] pending: vmstat_update, lru_add_drain_per_cpu [ 6141.059857] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6141.065984] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6141.072150] workqueue xprtiod: flags=0xa [ 6141.075328] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6141.079296] pending: xprt_autoclose, xs_error_handle [ 6141.082877] workqueue ipv6_addrconf: flags=0x40008 [ 6141.088939] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6141.096151] pending: addrconf_verify_work [ 6141.099410] workqueue knbd0-recv: flags=0x1a [ 6141.103896] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6141.108140] in-flight: 651:recv_work [ 6141.112053] workqueue ptlrpc_pinger: flags=0x2 [ 6141.117095] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6141.123918] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6141.126453] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3504s workers=4 idle: 15979 31824 8 [ 6141.139994] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6131s workers=2 idle: 654 [ 6160.058784] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 6160.069518] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6160.100917] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6160.105393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6160.108857] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6160.111151] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6160.118429] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6160.121373] RAX: 0000000000003ac9 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6160.123801] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6160.126415] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6160.128897] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6160.131384] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6160.137573] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6160.142350] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6160.144979] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6160.148054] Call Trace: [ 6160.148871] [ 6160.149382] ? show_regs.cold.9+0x22/0x2f [ 6160.157148] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6160.162857] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6160.168580] ? watchdog+0x40/0x40 [ 6160.170126] ? hrtimer_interrupt+0x128/0x2c0 [ 6160.172128] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6160.175870] ? apic_timer_interrupt+0xf/0x20 [ 6160.178137] [ 6160.178803] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6160.181304] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6160.183340] do_raw_spin_lock+0x177/0x190 [ 6160.185142] _raw_spin_lock+0x21/0x30 [ 6160.186438] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6160.191619] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6160.195551] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6160.198276] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6160.203655] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6160.208640] ? woken_wake_function+0x30/0x30 [ 6160.210492] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6160.214238] kthread+0x1d1/0x200 [ 6160.216804] ? set_kthread_struct+0x70/0x70 [ 6160.219811] ret_from_fork+0x1f/0x30 [ 6168.052096] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 6168.055458] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6168.056941] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 6168.057014] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 6168.057019] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6168.057081] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6168.057087] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6168.057090] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6168.057104] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6168.057109] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6168.057113] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6168.057115] RDX: 0000000000003af1 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6168.057117] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6168.057119] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6168.057121] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6168.057124] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6168.057127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6168.057130] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6168.057139] Call Trace: [ 6168.057144] [ 6168.057150] ? show_regs.cold.9+0x22/0x2f [ 6168.057160] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6168.057165] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6168.057170] ? watchdog+0x40/0x40 [ 6168.057173] ? hrtimer_interrupt+0x128/0x2c0 [ 6168.057177] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6168.057185] ? apic_timer_interrupt+0xf/0x20 [ 6168.057189] [ 6168.057192] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6168.057197] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6168.057201] do_raw_spin_lock+0x177/0x190 [ 6168.057205] _raw_spin_lock+0x21/0x30 [ 6168.057210] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6168.057491] obd_mod_free+0x46/0x140 [ptlrpc] [ 6168.057698] mdc_commit_open+0x91/0xb0 [mdc] [ 6168.057739] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6168.060168] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6168.060378] after_reply+0xe2c/0x1780 [ptlrpc] [ 6168.060595] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6168.060789] ? __next_timer_interrupt+0x160/0x160 [ 6168.060798] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6168.061537] ? do_wait_intr+0xf0/0xf0 [ 6168.061548] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6168.061996] kthread+0x1d1/0x200 [ 6168.062002] ? set_kthread_struct+0x70/0x70 [ 6168.062006] ret_from_fork+0x1f/0x30 [ 6168.080727] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6168.080740] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6168.080747] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6168.081026] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6168.081037] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6168.081041] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6168.081045] RAX: 0000000000006d37 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6168.081048] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6168.081051] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6168.081054] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6168.081056] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6168.081060] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6168.081064] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6168.081066] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6168.081075] Call Trace: [ 6168.081082] [ 6168.081088] ? show_regs.cold.9+0x22/0x2f [ 6168.081102] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6168.081109] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6168.081113] ? watchdog+0x40/0x40 [ 6168.081118] ? hrtimer_interrupt+0x128/0x2c0 [ 6168.081122] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6168.081131] ? apic_timer_interrupt+0xf/0x20 [ 6168.081136] [ 6168.081139] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6168.081145] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6168.081159] ? newidle_balance.isra.93+0x207/0x580 [ 6168.081164] do_raw_spin_lock+0x177/0x190 [ 6168.081170] _raw_spin_lock+0x21/0x30 [ 6168.081175] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6168.081386] process_one_work+0x2c8/0x700 [ 6168.081394] worker_thread+0x296/0x6e0 [ 6168.081398] ? rescuer_thread+0x570/0x570 [ 6168.081403] kthread+0x1d1/0x200 [ 6168.081407] ? set_kthread_struct+0x70/0x70 [ 6168.081411] ret_from_fork+0x1f/0x30 [ 6168.278626] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6168.293519] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6168.297378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6168.300381] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6168.302390] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6168.308905] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6168.311224] RAX: 0000000000006a0f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6168.313456] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6168.315769] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6168.318016] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6168.320288] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6168.322660] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6168.325268] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6168.327032] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6168.329419] Call Trace: [ 6168.330251] [ 6168.330952] ? show_regs.cold.9+0x22/0x2f [ 6168.332402] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6168.333947] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6168.335404] ? watchdog+0x40/0x40 [ 6168.336423] ? hrtimer_interrupt+0x128/0x2c0 [ 6168.337819] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6168.339470] ? apic_timer_interrupt+0xf/0x20 [ 6168.340901] [ 6168.341590] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6168.343354] do_raw_spin_lock+0x177/0x190 [ 6168.344849] _raw_spin_lock+0x21/0x30 [ 6168.346141] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6168.348149] ? do_raw_spin_unlock+0x75/0x190 [ 6168.349797] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6168.351512] ? do_wait_intr+0xf0/0xf0 [ 6168.352864] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6168.355104] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6168.357293] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6168.359978] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6168.362144] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6168.364791] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6168.366612] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6168.368600] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6168.370965] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6168.373599] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6168.375911] lmv_intent_open+0x338/0x1150 [lmv] [ 6168.377811] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6168.380278] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6168.382585] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6168.384476] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6168.386453] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6168.388961] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6168.390582] ? do_raw_spin_unlock+0x75/0x190 [ 6168.392360] ? _raw_spin_unlock+0x12/0x30 [ 6168.394129] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6168.396326] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6168.398284] ll_atomic_open+0x36a/0xd70 [lustre] [ 6168.400693] lookup_open+0x356/0xb40 [ 6168.401981] do_last+0x313/0xfc0 [ 6168.403101] ? nd_jump_root+0xe5/0x160 [ 6168.404320] ? path_init+0x437/0x520 [ 6168.405453] path_openat+0xf7/0x500 [ 6168.406681] ? do_raw_spin_unlock+0x75/0x190 [ 6168.408003] do_filp_open+0x99/0x140 [ 6168.409151] ? getname_flags+0x6e/0x330 [ 6168.410429] ? __check_object_size+0xff/0x256 [ 6168.412053] ? do_raw_spin_unlock+0x75/0x190 [ 6168.413662] ? _raw_spin_unlock+0x12/0x30 [ 6168.415312] do_sys_openat2+0x2b4/0x410 [ 6168.416766] do_sys_open+0x73/0xa0 [ 6168.417944] __x64_sys_openat+0x24/0x30 [ 6168.419154] do_syscall_64+0xc1/0x440 [ 6168.420386] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6168.422665] RIP: 0033:0x7f53425a5332 [ 6168.423979] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6168.430340] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6168.432806] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6168.435268] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6168.437473] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6168.439885] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6168.442242] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6171.615044] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3534s! [ 6171.623924] Showing busy workqueues and worker pools: [ 6171.630711] workqueue events: flags=0x0 [ 6171.633075] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6171.638048] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6171.652323] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6171.658283] pending: cpu_latency_work [ptlrpc] [ 6171.666358] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6171.670417] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6171.675283] workqueue events_long: flags=0x0 [ 6171.678869] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6171.683508] pending: lu_objects_flush [obdclass] [ 6171.690513] workqueue events_unbound: flags=0x2 [ 6171.694335] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6171.701641] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6171.705336] workqueue events_power_efficient: flags=0x80 [ 6171.709371] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6171.716974] pending: check_lifetime, do_cache_clean [ 6171.722487] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6171.726955] pending: neigh_periodic_work, neigh_periodic_work [ 6171.730819] workqueue mm_percpu_wq: flags=0x8 [ 6171.733313] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6171.741850] pending: vmstat_update, lru_add_drain_per_cpu [ 6171.749455] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6171.761612] pending: vmstat_update, lru_add_drain_per_cpu [ 6171.765207] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6171.771268] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6171.778184] workqueue xprtiod: flags=0xa [ 6171.781869] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6171.787066] pending: xprt_autoclose, xs_error_handle [ 6171.791626] workqueue ipv6_addrconf: flags=0x40008 [ 6171.794214] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6171.796353] pending: addrconf_verify_work [ 6171.798530] workqueue knbd0-recv: flags=0x1a [ 6171.800985] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6171.803781] in-flight: 651:recv_work [ 6171.807549] workqueue ptlrpc_pinger: flags=0x2 [ 6171.810138] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6171.813661] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6171.817281] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3534s workers=4 idle: 15979 31824 8 [ 6171.823035] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6162s workers=2 idle: 654 [ 6188.061623] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 6188.067119] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6188.103131] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6188.113869] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6188.124783] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6188.137584] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6188.156731] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6188.166321] RAX: 0000000000007c98 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6188.172078] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6188.180067] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6188.184121] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6188.192912] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6188.200575] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6188.211564] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6188.218107] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6188.225838] Call Trace: [ 6188.230156] [ 6188.231612] ? show_regs.cold.9+0x22/0x2f [ 6188.236272] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6188.240903] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6188.243803] ? watchdog+0x40/0x40 [ 6188.245591] ? hrtimer_interrupt+0x128/0x2c0 [ 6188.248018] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6188.250586] ? apic_timer_interrupt+0xf/0x20 [ 6188.253343] [ 6188.254508] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6188.257062] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6188.260359] do_raw_spin_lock+0x177/0x190 [ 6188.262290] _raw_spin_lock+0x21/0x30 [ 6188.263794] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6188.266680] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6188.269238] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6188.271802] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6188.274517] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6188.276351] ? woken_wake_function+0x30/0x30 [ 6188.278083] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6188.281440] kthread+0x1d1/0x200 [ 6188.287095] ? set_kthread_struct+0x70/0x70 [ 6188.293302] ret_from_fork+0x1f/0x30 [ 6196.052027] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 6196.054691] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6196.055052] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 6196.057293] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 6196.057306] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6196.057472] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6196.057478] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6196.057481] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6196.057494] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6196.057498] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6196.057503] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6196.057505] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6196.057507] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6196.057509] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6196.057511] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6196.057514] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6196.057517] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6196.057519] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6196.057528] Call Trace: [ 6196.057534] [ 6196.057539] ? show_regs.cold.9+0x22/0x2f [ 6196.057551] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6196.057558] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6196.057562] ? watchdog+0x40/0x40 [ 6196.057567] ? hrtimer_interrupt+0x128/0x2c0 [ 6196.057571] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6196.057578] ? apic_timer_interrupt+0xf/0x20 [ 6196.057583] [ 6196.057586] ? native_safe_halt+0x2e/0x50 [ 6196.057590] kvm_wait+0xcd/0xe0 [ 6196.057597] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6196.057605] do_raw_spin_lock+0x177/0x190 [ 6196.057609] _raw_spin_lock+0x21/0x30 [ 6196.057614] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6196.057893] obd_mod_free+0x46/0x140 [ptlrpc] [ 6196.058098] mdc_commit_open+0x91/0xb0 [mdc] [ 6196.058138] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6196.058331] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6196.058539] after_reply+0xe2c/0x1780 [ptlrpc] [ 6196.058728] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6196.059608] ? __next_timer_interrupt+0x160/0x160 [ 6196.059619] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6196.059817] ? do_wait_intr+0xf0/0xf0 [ 6196.059826] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6196.060019] kthread+0x1d1/0x200 [ 6196.060025] ? set_kthread_struct+0x70/0x70 [ 6196.060030] ret_from_fork+0x1f/0x30 [ 6196.094313] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6196.094327] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6196.094333] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6196.094613] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6196.094623] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6196.094627] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6196.094632] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6196.094635] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6196.094638] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6196.094640] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6196.094643] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6196.094715] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6196.094720] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6196.094722] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6196.094730] Call Trace: [ 6196.094735] [ 6196.094740] ? show_regs.cold.9+0x22/0x2f [ 6196.094750] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6196.094756] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6196.094762] ? watchdog+0x40/0x40 [ 6196.094767] ? hrtimer_interrupt+0x128/0x2c0 [ 6196.094771] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6196.094778] ? apic_timer_interrupt+0xf/0x20 [ 6196.094784] [ 6196.094787] ? native_safe_halt+0x2e/0x50 [ 6196.094791] kvm_wait+0xcd/0xe0 [ 6196.094798] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6196.094805] ? newidle_balance.isra.93+0x207/0x580 [ 6196.094812] do_raw_spin_lock+0x177/0x190 [ 6196.094816] _raw_spin_lock+0x21/0x30 [ 6196.094822] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6196.095176] process_one_work+0x2c8/0x700 [ 6196.095184] worker_thread+0x296/0x6e0 [ 6196.095189] ? rescuer_thread+0x570/0x570 [ 6196.095193] kthread+0x1d1/0x200 [ 6196.095197] ? set_kthread_struct+0x70/0x70 [ 6196.095202] ret_from_fork+0x1f/0x30 [ 6196.425627] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6196.444569] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6196.448562] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6196.460514] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6196.465365] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6196.477766] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6196.481622] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6196.484801] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6196.489462] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6196.493339] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6196.500263] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6196.505718] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6196.512990] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6196.517555] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6196.526698] Call Trace: [ 6196.527636] [ 6196.533525] ? show_regs.cold.9+0x22/0x2f [ 6196.540290] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6196.544269] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6196.553223] ? watchdog+0x40/0x40 [ 6196.554821] ? hrtimer_interrupt+0x128/0x2c0 [ 6196.559909] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6196.565283] ? apic_timer_interrupt+0xf/0x20 [ 6196.568992] [ 6196.570446] ? native_safe_halt+0x2e/0x50 [ 6196.574197] kvm_wait+0xcd/0xe0 [ 6196.576880] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6196.579935] do_raw_spin_lock+0x177/0x190 [ 6196.583225] _raw_spin_lock+0x21/0x30 [ 6196.584489] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6196.590876] ? do_raw_spin_unlock+0x75/0x190 [ 6196.597388] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6196.599258] ? do_wait_intr+0xf0/0xf0 [ 6196.602348] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6196.606567] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6196.609967] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6196.617822] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6196.621433] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6196.624307] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6196.626333] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6196.629017] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6196.630967] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6196.633482] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6196.635285] lmv_intent_open+0x338/0x1150 [lmv] [ 6196.636650] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6196.638374] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6196.640123] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6196.641500] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6196.643247] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6196.644951] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6196.646501] ? do_raw_spin_unlock+0x75/0x190 [ 6196.648399] ? _raw_spin_unlock+0x12/0x30 [ 6196.649342] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6196.650984] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6196.652611] ll_atomic_open+0x36a/0xd70 [lustre] [ 6196.653893] lookup_open+0x356/0xb40 [ 6196.655104] do_last+0x313/0xfc0 [ 6196.656269] ? nd_jump_root+0xe5/0x160 [ 6196.657437] ? path_init+0x437/0x520 [ 6196.658325] path_openat+0xf7/0x500 [ 6196.659211] ? do_raw_spin_unlock+0x75/0x190 [ 6196.660451] do_filp_open+0x99/0x140 [ 6196.661450] ? getname_flags+0x6e/0x330 [ 6196.662750] ? __check_object_size+0xff/0x256 [ 6196.665986] ? do_raw_spin_unlock+0x75/0x190 [ 6196.668371] ? _raw_spin_unlock+0x12/0x30 [ 6196.670676] do_sys_openat2+0x2b4/0x410 [ 6196.672892] do_sys_open+0x73/0xa0 [ 6196.675263] __x64_sys_openat+0x24/0x30 [ 6196.677326] do_syscall_64+0xc1/0x440 [ 6196.679734] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6196.681639] RIP: 0033:0x7f53425a5332 [ 6196.683506] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6196.691357] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6196.696314] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6196.701936] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6196.706018] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6196.711008] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6196.722682] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6202.336092] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3565s! [ 6202.339922] Showing busy workqueues and worker pools: [ 6202.343332] workqueue events: flags=0x0 [ 6202.345353] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6202.349752] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6202.354781] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6202.358717] pending: cpu_latency_work [ptlrpc] [ 6202.361461] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6202.366035] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6202.371370] workqueue events_long: flags=0x0 [ 6202.372557] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6202.376101] pending: lu_objects_flush [obdclass] [ 6202.383433] workqueue events_unbound: flags=0x2 [ 6202.390034] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6202.395737] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6202.406022] workqueue events_power_efficient: flags=0x80 [ 6202.418990] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6202.423096] pending: check_lifetime, do_cache_clean [ 6202.427855] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6202.442832] pending: neigh_periodic_work, neigh_periodic_work [ 6202.447548] workqueue mm_percpu_wq: flags=0x8 [ 6202.452318] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6202.454900] pending: vmstat_update, lru_add_drain_per_cpu [ 6202.457197] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6202.459944] pending: vmstat_update, lru_add_drain_per_cpu [ 6202.462188] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6202.464087] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6202.465835] workqueue xprtiod: flags=0xa [ 6202.466875] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6202.468602] pending: xprt_autoclose, xs_error_handle [ 6202.469978] workqueue ipv6_addrconf: flags=0x40008 [ 6202.471115] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6202.474139] pending: addrconf_verify_work [ 6202.476630] workqueue knbd0-recv: flags=0x1a [ 6202.478467] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6202.480393] in-flight: 651:recv_work [ 6202.481583] workqueue ptlrpc_pinger: flags=0x2 [ 6202.483220] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6202.485001] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6202.487326] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3565s workers=4 idle: 15979 31824 8 [ 6202.489614] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6192s workers=2 idle: 654 [ 6216.058414] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 6216.070355] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6216.097138] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6216.102344] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6216.111472] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6216.114733] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6216.133023] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6216.139253] RAX: 000000000000720b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6216.149949] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6216.159956] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6216.166525] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6216.176502] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6216.183049] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6216.188412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6216.194473] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6216.198204] Call Trace: [ 6216.203702] [ 6216.204582] ? show_regs.cold.9+0x22/0x2f [ 6216.206369] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6216.208947] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6216.214369] ? watchdog+0x40/0x40 [ 6216.216279] ? hrtimer_interrupt+0x128/0x2c0 [ 6216.219844] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6216.223448] ? apic_timer_interrupt+0xf/0x20 [ 6216.225163] [ 6216.226925] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6216.229124] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6216.232343] do_raw_spin_lock+0x177/0x190 [ 6216.234387] _raw_spin_lock+0x21/0x30 [ 6216.235810] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6216.241957] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6216.243967] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6216.246480] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6216.251964] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6216.256813] ? woken_wake_function+0x30/0x30 [ 6216.259022] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6216.261830] kthread+0x1d1/0x200 [ 6216.264596] ? set_kthread_struct+0x70/0x70 [ 6216.267394] ret_from_fork+0x1f/0x30 [ 6224.059472] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 6224.062919] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 6224.066386] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6224.066468] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6224.066476] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6224.066479] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6224.066494] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6224.066499] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6224.066503] RAX: 0000000000007d97 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6224.066506] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6224.066508] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6224.066511] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6224.066513] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6224.066517] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6224.066520] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6224.066522] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6224.066533] Call Trace: [ 6224.066538] [ 6224.066543] ? show_regs.cold.9+0x22/0x2f [ 6224.066565] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6224.066571] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6224.066575] ? watchdog+0x40/0x40 [ 6224.066580] ? hrtimer_interrupt+0x128/0x2c0 [ 6224.066584] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6224.066592] ? apic_timer_interrupt+0xf/0x20 [ 6224.066596] [ 6224.066600] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6224.066607] do_raw_spin_lock+0x177/0x190 [ 6224.066613] _raw_spin_lock+0x21/0x30 [ 6224.066618] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6224.066899] ? do_raw_spin_unlock+0x75/0x190 [ 6224.066905] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6224.067111] ? do_wait_intr+0xf0/0xf0 [ 6224.067120] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6224.067328] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6224.067562] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6224.067785] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6224.067894] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6224.067929] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6224.067957] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6224.067983] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6224.068063] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6224.068277] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6224.068304] lmv_intent_open+0x338/0x1150 [lmv] [ 6224.068333] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6224.068414] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6224.068496] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6224.068520] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6224.068621] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6224.068703] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6224.068787] ? do_raw_spin_unlock+0x75/0x190 [ 6224.068793] ? _raw_spin_unlock+0x12/0x30 [ 6224.068799] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6224.068825] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6224.068846] ll_atomic_open+0x36a/0xd70 [lustre] [ 6224.068936] lookup_open+0x356/0xb40 [ 6224.068945] do_last+0x313/0xfc0 [ 6224.068950] ? nd_jump_root+0xe5/0x160 [ 6224.068955] ? path_init+0x437/0x520 [ 6224.068960] path_openat+0xf7/0x500 [ 6224.068965] ? do_raw_spin_unlock+0x75/0x190 [ 6224.068970] do_filp_open+0x99/0x140 [ 6224.068977] ? getname_flags+0x6e/0x330 [ 6224.068981] ? __check_object_size+0xff/0x256 [ 6224.068988] ? do_raw_spin_unlock+0x75/0x190 [ 6224.068993] ? _raw_spin_unlock+0x12/0x30 [ 6224.074430] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6224.074510] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6224.074518] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6224.074520] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6224.074534] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6224.074539] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6224.074543] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6224.074555] RDX: 0000000000007c7c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6224.074559] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6224.074561] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6224.074563] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6224.074567] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6224.074570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6224.074572] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6224.074580] Call Trace: [ 6224.074586] [ 6224.074592] ? show_regs.cold.9+0x22/0x2f [ 6224.074601] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6224.074608] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6224.074613] ? watchdog+0x40/0x40 [ 6224.074617] ? hrtimer_interrupt+0x128/0x2c0 [ 6224.074621] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6224.074629] ? apic_timer_interrupt+0xf/0x20 [ 6224.074633] [ 6224.074636] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6224.074641] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6224.074646] do_raw_spin_lock+0x177/0x190 [ 6224.074651] _raw_spin_lock+0x21/0x30 [ 6224.074657] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6224.074914] obd_mod_free+0x46/0x140 [ptlrpc] [ 6224.075112] mdc_commit_open+0x91/0xb0 [mdc] [ 6224.075151] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6224.075351] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6224.076495] after_reply+0xe2c/0x1780 [ptlrpc] [ 6224.076842] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6224.077039] ? __next_timer_interrupt+0x160/0x160 [ 6224.077052] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6224.155758] do_sys_openat2+0x2b4/0x410 [ 6224.155780] do_sys_open+0x73/0xa0 [ 6224.155785] __x64_sys_openat+0x24/0x30 [ 6224.155789] do_syscall_64+0xc1/0x440 [ 6224.155796] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6224.155805] RIP: 0033:0x7f53425a5332 [ 6224.155815] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6224.155819] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6224.155824] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6224.155827] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6224.155829] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6224.155831] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6224.155834] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6224.758261] ? do_wait_intr+0xf0/0xf0 [ 6224.765152] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6224.770033] kthread+0x1d1/0x200 [ 6224.781167] ? set_kthread_struct+0x70/0x70 [ 6224.783983] ret_from_fork+0x1f/0x30 [ 6228.053274] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 6228.063803] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6228.114749] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6228.120706] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6228.133360] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6228.141091] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6228.148597] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6228.164940] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6228.169747] RAX: 000000000000760a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6228.171944] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6228.175262] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6228.178091] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6228.181248] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6228.186964] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6228.190338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6228.192969] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6228.197861] Call Trace: [ 6228.200355] [ 6228.201107] ? show_regs.cold.9+0x22/0x2f [ 6228.202174] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6228.206993] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6228.209240] ? watchdog+0x40/0x40 [ 6228.213434] ? hrtimer_interrupt+0x128/0x2c0 [ 6228.214992] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6228.218011] ? apic_timer_interrupt+0xf/0x20 [ 6228.222960] [ 6228.224077] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6228.230669] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6228.235073] ? newidle_balance.isra.93+0x207/0x580 [ 6228.237184] do_raw_spin_lock+0x177/0x190 [ 6228.239830] _raw_spin_lock+0x21/0x30 [ 6228.241904] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6228.244933] process_one_work+0x2c8/0x700 [ 6228.250306] worker_thread+0x296/0x6e0 [ 6228.251564] ? rescuer_thread+0x570/0x570 [ 6228.256573] kthread+0x1d1/0x200 [ 6228.258951] ? set_kthread_struct+0x70/0x70 [ 6228.264108] ret_from_fork+0x1f/0x30 [ 6232.543612] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3595s! [ 6232.554060] Showing busy workqueues and worker pools: [ 6232.557833] workqueue events: flags=0x0 [ 6232.559395] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6232.563868] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6232.569681] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6232.575819] pending: cpu_latency_work [ptlrpc] [ 6232.578526] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6232.583278] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6232.590058] workqueue events_long: flags=0x0 [ 6232.594453] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6232.597998] pending: lu_objects_flush [obdclass] [ 6232.600772] workqueue events_unbound: flags=0x2 [ 6232.602767] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6232.607603] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6232.611484] workqueue events_power_efficient: flags=0x80 [ 6232.619197] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6232.625160] pending: check_lifetime, do_cache_clean [ 6232.627760] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6232.630653] pending: neigh_periodic_work, neigh_periodic_work [ 6232.635322] workqueue mm_percpu_wq: flags=0x8 [ 6232.637827] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6232.643213] pending: vmstat_update, lru_add_drain_per_cpu [ 6232.645023] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6232.647514] pending: vmstat_update, lru_add_drain_per_cpu [ 6232.649835] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6232.654260] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6232.658342] workqueue xprtiod: flags=0xa [ 6232.660026] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6232.663017] pending: xprt_autoclose, xs_error_handle [ 6232.666118] workqueue ipv6_addrconf: flags=0x40008 [ 6232.670016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6232.673002] pending: addrconf_verify_work [ 6232.675288] workqueue knbd0-recv: flags=0x1a [ 6232.676517] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6232.683038] in-flight: 651:recv_work [ 6232.686713] workqueue ptlrpc_pinger: flags=0x2 [ 6232.688742] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6232.691729] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6232.694915] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3595s workers=4 idle: 15979 31824 8 [ 6232.698690] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6223s workers=2 idle: 654 [ 6244.061243] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 6244.067607] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6244.103510] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6244.112744] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6244.118629] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6244.122877] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6244.140101] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6244.144045] RAX: 0000000000005ad4 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6244.149669] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6244.153826] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6244.160472] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6244.169486] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6244.185426] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6244.196746] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6244.201320] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6244.211705] Call Trace: [ 6244.213582] [ 6244.215803] ? show_regs.cold.9+0x22/0x2f [ 6244.218932] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6244.221847] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6244.224670] ? watchdog+0x40/0x40 [ 6244.226333] ? hrtimer_interrupt+0x128/0x2c0 [ 6244.232799] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6244.241694] ? apic_timer_interrupt+0xf/0x20 [ 6244.243299] [ 6244.244067] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6244.246608] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6244.250505] do_raw_spin_lock+0x177/0x190 [ 6244.254585] _raw_spin_lock+0x21/0x30 [ 6244.255945] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6244.262531] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6244.266986] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6244.272009] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6244.276685] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6244.282276] ? woken_wake_function+0x30/0x30 [ 6244.286148] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6244.291723] kthread+0x1d1/0x200 [ 6244.294662] ? set_kthread_struct+0x70/0x70 [ 6244.296588] ret_from_fork+0x1f/0x30 [ 6252.055033] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 6252.057287] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 6252.059208] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6252.059284] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6252.059291] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6252.059294] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6252.059306] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6252.059310] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6252.059314] RAX: 0000000000006484 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6252.059316] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6252.059318] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6252.059320] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6252.059323] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6252.059326] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6252.059329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6252.059332] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6252.059340] Call Trace: [ 6252.059345] [ 6252.059350] ? show_regs.cold.9+0x22/0x2f [ 6252.059358] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6252.059364] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6252.059368] ? watchdog+0x40/0x40 [ 6252.059373] ? hrtimer_interrupt+0x128/0x2c0 [ 6252.059377] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6252.059386] ? apic_timer_interrupt+0xf/0x20 [ 6252.059390] [ 6252.059392] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6252.059399] do_raw_spin_lock+0x177/0x190 [ 6252.059403] _raw_spin_lock+0x21/0x30 [ 6252.059407] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6252.062686] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6252.062849] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6252.062856] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6252.062859] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6252.062878] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6252.062883] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6252.062887] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6252.062890] RDX: 00000000000043d3 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6252.062892] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6252.062894] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6252.062896] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6252.062899] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6252.062902] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6252.062904] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6252.062911] Call Trace: [ 6252.062916] [ 6252.062922] ? show_regs.cold.9+0x22/0x2f [ 6252.062931] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6252.062937] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6252.062941] ? watchdog+0x40/0x40 [ 6252.062946] ? hrtimer_interrupt+0x128/0x2c0 [ 6252.062950] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6252.062958] ? apic_timer_interrupt+0xf/0x20 [ 6252.062962] [ 6252.062965] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6252.062969] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6252.062974] do_raw_spin_lock+0x177/0x190 [ 6252.062978] _raw_spin_lock+0x21/0x30 [ 6252.062984] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6252.063290] obd_mod_free+0x46/0x140 [ptlrpc] [ 6252.086973] ? do_raw_spin_unlock+0x75/0x190 [ 6252.086993] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6252.087391] ? do_wait_intr+0xf0/0xf0 [ 6252.087401] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6252.087726] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6252.096317] mdc_commit_open+0x91/0xb0 [mdc] [ 6252.100506] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6252.103983] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6252.113769] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6252.113898] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6252.113940] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6252.113973] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6252.114004] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6252.114116] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6252.114425] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6252.114455] lmv_intent_open+0x338/0x1150 [lmv] [ 6252.114485] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6252.114572] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6252.114797] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6252.114821] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6252.114909] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6252.115118] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6252.115210] ? do_raw_spin_unlock+0x75/0x190 [ 6252.115219] ? _raw_spin_unlock+0x12/0x30 [ 6252.115227] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6252.115254] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6252.115275] ll_atomic_open+0x36a/0xd70 [lustre] [ 6252.115359] lookup_open+0x356/0xb40 [ 6252.115370] do_last+0x313/0xfc0 [ 6252.115375] ? nd_jump_root+0xe5/0x160 [ 6252.115381] ? path_init+0x437/0x520 [ 6252.115385] path_openat+0xf7/0x500 [ 6252.115391] ? do_raw_spin_unlock+0x75/0x190 [ 6252.115396] do_filp_open+0x99/0x140 [ 6252.115403] ? getname_flags+0x6e/0x330 [ 6252.115407] ? __check_object_size+0xff/0x256 [ 6252.115415] ? do_raw_spin_unlock+0x75/0x190 [ 6252.115419] ? _raw_spin_unlock+0x12/0x30 [ 6252.115424] do_sys_openat2+0x2b4/0x410 [ 6252.115428] do_sys_open+0x73/0xa0 [ 6252.115432] __x64_sys_openat+0x24/0x30 [ 6252.115436] do_syscall_64+0xc1/0x440 [ 6252.115442] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6252.115448] RIP: 0033:0x7f53425a5332 [ 6252.115456] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6252.115460] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6252.115465] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6252.115467] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6252.115470] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6252.115472] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6252.115474] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6252.349422] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6252.352246] after_reply+0xe2c/0x1780 [ptlrpc] [ 6252.354651] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6252.356908] ? __next_timer_interrupt+0x160/0x160 [ 6252.358753] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6252.360608] ? do_wait_intr+0xf0/0xf0 [ 6252.361958] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6252.364456] kthread+0x1d1/0x200 [ 6252.365586] ? set_kthread_struct+0x70/0x70 [ 6252.367221] ret_from_fork+0x1f/0x30 [ 6260.052066] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 6260.054837] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6260.080659] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6260.091167] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6260.099888] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6260.105776] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6260.108403] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6260.129630] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6260.136780] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6260.140588] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6260.144602] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6260.151050] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6260.152871] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6260.162678] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6260.168594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6260.176962] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6260.185519] Call Trace: [ 6260.187483] [ 6260.191633] ? show_regs.cold.9+0x22/0x2f [ 6260.193770] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6260.203126] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6260.204840] ? watchdog+0x40/0x40 [ 6260.206881] ? hrtimer_interrupt+0x128/0x2c0 [ 6260.208744] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6260.211330] ? apic_timer_interrupt+0xf/0x20 [ 6260.213395] [ 6260.214656] ? native_safe_halt+0x2e/0x50 [ 6260.216482] kvm_wait+0xcd/0xe0 [ 6260.217911] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6260.220610] ? newidle_balance.isra.93+0x207/0x580 [ 6260.225559] do_raw_spin_lock+0x177/0x190 [ 6260.227420] _raw_spin_lock+0x21/0x30 [ 6260.229275] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6260.232229] process_one_work+0x2c8/0x700 [ 6260.234569] worker_thread+0x296/0x6e0 [ 6260.236920] ? rescuer_thread+0x570/0x570 [ 6260.240191] kthread+0x1d1/0x200 [ 6260.242847] ? set_kthread_struct+0x70/0x70 [ 6260.245401] ret_from_fork+0x1f/0x30 [ 6262.757586] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3625s! [ 6262.763343] Showing busy workqueues and worker pools: [ 6262.768539] workqueue events: flags=0x0 [ 6262.773337] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6262.782552] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6262.790925] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6262.796799] pending: cpu_latency_work [ptlrpc] [ 6262.800098] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6262.805623] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6262.812265] workqueue events_long: flags=0x0 [ 6262.813860] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6262.819676] pending: lu_objects_flush [obdclass] [ 6262.824665] workqueue events_unbound: flags=0x2 [ 6262.827474] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6262.832917] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6262.834866] workqueue events_power_efficient: flags=0x80 [ 6262.840181] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6262.846094] pending: check_lifetime, do_cache_clean [ 6262.852046] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6262.856598] pending: neigh_periodic_work, neigh_periodic_work [ 6262.860209] workqueue mm_percpu_wq: flags=0x8 [ 6262.862815] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6262.868361] pending: vmstat_update, lru_add_drain_per_cpu [ 6262.873534] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6262.879200] pending: vmstat_update, lru_add_drain_per_cpu [ 6262.881371] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6262.887089] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6262.890285] workqueue xprtiod: flags=0xa [ 6262.891605] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6262.896291] pending: xprt_autoclose, xs_error_handle [ 6262.901626] workqueue ipv6_addrconf: flags=0x40008 [ 6262.905192] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6262.907534] pending: addrconf_verify_work [ 6262.912492] workqueue knbd0-recv: flags=0x1a [ 6262.913656] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6262.919953] in-flight: 651:recv_work [ 6262.922388] workqueue ptlrpc_pinger: flags=0x2 [ 6262.926507] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6262.934068] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6262.936555] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3625s workers=4 idle: 15979 31824 8 [ 6262.943657] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6253s workers=2 idle: 654 [ 6272.060751] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 6272.068658] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6272.137193] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6272.157202] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6272.176325] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6272.183558] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6272.202665] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6272.214597] RAX: 0000000000007c3e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6272.223109] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6272.230119] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6272.240096] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6272.250200] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6272.256362] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6272.265682] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6272.270304] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6272.274407] Call Trace: [ 6272.275338] [ 6272.275936] ? show_regs.cold.9+0x22/0x2f [ 6272.277187] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6272.285291] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6272.291671] ? watchdog+0x40/0x40 [ 6272.297997] ? hrtimer_interrupt+0x128/0x2c0 [ 6272.299611] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6272.311263] ? apic_timer_interrupt+0xf/0x20 [ 6272.319279] [ 6272.319973] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6272.323160] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6272.326980] do_raw_spin_lock+0x177/0x190 [ 6272.335247] _raw_spin_lock+0x21/0x30 [ 6272.340110] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6272.349218] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6272.354492] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6272.359395] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6272.361597] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6272.366830] ? woken_wake_function+0x30/0x30 [ 6272.368300] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6272.371151] kthread+0x1d1/0x200 [ 6272.372364] ? set_kthread_struct+0x70/0x70 [ 6272.374280] ret_from_fork+0x1f/0x30 [ 6280.057012] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 6280.057233] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 6280.061290] Modules linked in: lustre(O) [ 6280.070270] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6280.070363] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6280.070371] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6280.070374] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6280.070388] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6280.070392] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6280.070397] RAX: 0000000000001089 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6280.070399] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6280.070402] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6280.070404] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6280.070406] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6280.070409] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6280.070413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6280.070415] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6280.070424] Call Trace: [ 6280.070431] [ 6280.070436] ? show_regs.cold.9+0x22/0x2f [ 6280.070445] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6280.070452] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6280.070456] ? watchdog+0x40/0x40 [ 6280.070461] ? hrtimer_interrupt+0x128/0x2c0 [ 6280.070465] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6280.070474] ? apic_timer_interrupt+0xf/0x20 [ 6280.070480] [ 6280.070482] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6280.070490] do_raw_spin_lock+0x177/0x190 [ 6280.070496] _raw_spin_lock+0x21/0x30 [ 6280.070502] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6280.072850] ? do_raw_spin_unlock+0x75/0x190 [ 6280.072860] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6280.073110] ? do_wait_intr+0xf0/0xf0 [ 6280.073119] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6280.074426] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6280.074809] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6280.078707] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6280.078791] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6280.078798] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6280.078801] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6280.078814] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6280.078819] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6280.078823] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6280.078826] RDX: 00000000000030d7 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6280.078829] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6280.078831] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6280.078834] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6280.078838] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6280.078841] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6280.078843] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6280.078914] Call Trace: [ 6280.078924] [ 6280.078930] ? show_regs.cold.9+0x22/0x2f [ 6280.078939] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6280.078945] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6280.078949] ? watchdog+0x40/0x40 [ 6280.078954] ? hrtimer_interrupt+0x128/0x2c0 [ 6280.078957] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6280.078967] ? apic_timer_interrupt+0xf/0x20 [ 6280.078972] [ 6280.078975] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6280.078980] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6280.078985] do_raw_spin_lock+0x177/0x190 [ 6280.078989] _raw_spin_lock+0x21/0x30 [ 6280.079001] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6280.123690] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6280.133510] obd_mod_free+0x46/0x140 [ptlrpc] [ 6280.137714] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6280.137784] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6280.137817] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6280.137846] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6280.137982] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6280.138308] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6280.138335] lmv_intent_open+0x338/0x1150 [lmv] [ 6280.138362] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6280.138448] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6280.138529] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6280.138550] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6280.138640] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6280.138722] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6280.138804] ? do_raw_spin_unlock+0x75/0x190 [ 6280.138812] ? _raw_spin_unlock+0x12/0x30 [ 6280.138819] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6280.138844] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6280.138863] ll_atomic_open+0x36a/0xd70 [lustre] [ 6280.138940] lookup_open+0x356/0xb40 [ 6280.138950] do_last+0x313/0xfc0 [ 6280.138954] ? nd_jump_root+0xe5/0x160 [ 6280.138959] ? path_init+0x437/0x520 [ 6280.138963] path_openat+0xf7/0x500 [ 6280.138968] ? do_raw_spin_unlock+0x75/0x190 [ 6280.138973] do_filp_open+0x99/0x140 [ 6280.138979] ? getname_flags+0x6e/0x330 [ 6280.138983] ? __check_object_size+0xff/0x256 [ 6280.138990] ? do_raw_spin_unlock+0x75/0x190 [ 6280.138994] ? _raw_spin_unlock+0x12/0x30 [ 6280.138999] do_sys_openat2+0x2b4/0x410 [ 6280.139004] do_sys_open+0x73/0xa0 [ 6280.139008] __x64_sys_openat+0x24/0x30 [ 6280.139011] do_syscall_64+0xc1/0x440 [ 6280.139018] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6280.139025] RIP: 0033:0x7f53425a5332 [ 6280.139034] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6280.139039] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6280.139045] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6280.139047] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6280.139050] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6280.139909] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6280.139915] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6280.475501] mdc_commit_open+0x91/0xb0 [mdc] [ 6280.481729] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6280.484455] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6280.486381] after_reply+0xe2c/0x1780 [ptlrpc] [ 6280.493422] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6280.497651] ? __next_timer_interrupt+0x160/0x160 [ 6280.500338] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6280.502553] ? do_wait_intr+0xf0/0xf0 [ 6280.507943] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6280.513679] kthread+0x1d1/0x200 [ 6280.516544] ? set_kthread_struct+0x70/0x70 [ 6280.518721] ret_from_fork+0x1f/0x30 [ 6288.052007] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 6288.063506] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6288.102520] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6288.108693] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6288.120172] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6288.124333] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6288.137464] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6288.166929] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6288.171226] RAX: 0000000000005d8e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6288.176612] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6288.181692] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6288.189067] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6288.197282] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6288.203364] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6288.213589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6288.216447] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6288.222196] Call Trace: [ 6288.224604] [ 6288.228194] ? show_regs.cold.9+0x22/0x2f [ 6288.230939] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6288.236752] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6288.239192] ? watchdog+0x40/0x40 [ 6288.240327] ? hrtimer_interrupt+0x128/0x2c0 [ 6288.249107] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6288.254989] ? apic_timer_interrupt+0xf/0x20 [ 6288.259394] [ 6288.260147] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6288.263227] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6288.265957] ? newidle_balance.isra.93+0x207/0x580 [ 6288.268038] do_raw_spin_lock+0x177/0x190 [ 6288.271351] _raw_spin_lock+0x21/0x30 [ 6288.273288] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6288.276288] process_one_work+0x2c8/0x700 [ 6288.278044] worker_thread+0x296/0x6e0 [ 6288.279970] ? rescuer_thread+0x570/0x570 [ 6288.281325] kthread+0x1d1/0x200 [ 6288.283741] ? set_kthread_struct+0x70/0x70 [ 6288.287987] ret_from_fork+0x1f/0x30 [ 6292.964152] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3655s! [ 6292.979882] Showing busy workqueues and worker pools: [ 6292.982382] workqueue events: flags=0x0 [ 6292.985015] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6292.993301] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6293.003210] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6293.011968] pending: cpu_latency_work [ptlrpc] [ 6293.019699] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6293.026588] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6293.040315] workqueue events_long: flags=0x0 [ 6293.043676] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6293.053779] pending: lu_objects_flush [obdclass] [ 6293.064900] workqueue events_unbound: flags=0x2 [ 6293.071283] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6293.083196] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6293.089495] workqueue events_power_efficient: flags=0x80 [ 6293.101312] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6293.112251] pending: check_lifetime, do_cache_clean [ 6293.115238] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6293.123792] pending: neigh_periodic_work, neigh_periodic_work [ 6293.130611] workqueue mm_percpu_wq: flags=0x8 [ 6293.132937] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6293.139009] pending: vmstat_update, lru_add_drain_per_cpu [ 6293.145181] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6293.153098] pending: vmstat_update, lru_add_drain_per_cpu [ 6293.156127] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6293.160923] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6293.167831] workqueue xprtiod: flags=0xa [ 6293.170080] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6293.178520] pending: xprt_autoclose, xs_error_handle [ 6293.183379] workqueue ipv6_addrconf: flags=0x40008 [ 6293.188796] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6293.195325] pending: addrconf_verify_work [ 6293.197023] workqueue knbd0-recv: flags=0x1a [ 6293.198657] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6293.201054] in-flight: 651:recv_work [ 6293.202605] workqueue ptlrpc_pinger: flags=0x2 [ 6293.204255] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6293.206657] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6293.209026] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3656s workers=4 idle: 15979 31824 8 [ 6293.212015] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6283s workers=2 idle: 654 [ 6294.531643] rcu: INFO: rcu_sched self-detected stall on CPU [ 6294.543778] rcu: 2-....: (2259820 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 6294.550765] (t=4200105 jiffies g=274185 q=64743) [ 6294.553711] rcu: rcu_sched kthread starved for 3658471 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 6294.566140] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6294.573145] rcu: RCU grace-period kthread stack dump: [ 6294.576064] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6294.585069] Call Trace: [ 6294.587227] __schedule+0x351/0xcb0 [ 6294.589822] schedule+0xc0/0x180 [ 6294.593117] schedule_timeout+0xb4/0x190 [ 6294.595549] ? do_raw_spin_unlock+0x75/0x190 [ 6294.599520] ? __next_timer_interrupt+0x160/0x160 [ 6294.602987] ? prepare_to_swait_event+0xf3/0x140 [ 6294.609137] rcu_gp_kthread+0x8dd/0x15b0 [ 6294.616770] ? force_qs_rnp+0x2c0/0x2c0 [ 6294.618905] kthread+0x1d1/0x200 [ 6294.627760] ? set_kthread_struct+0x70/0x70 [ 6294.629590] ret_from_fork+0x1f/0x30 [ 6294.632163] rcu: Stack dump where RCU GP kthread last ran: [ 6294.639629] Sending NMI from CPU 2 to CPUs 3: [ 6294.648280] NMI backtrace for cpu 3 [ 6294.648288] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6294.648290] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6294.648291] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6294.648293] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6294.648295] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 [ 6294.648297] RAX: 0000000000007683 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6294.648298] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6294.648299] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6294.648301] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6294.648302] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6294.648303] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6294.648305] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6294.648306] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6294.648572] Call Trace: [ 6294.648573] [ 6294.648574] ? show_regs.cold.9+0x22/0x2f [ 6294.648575] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6294.648576] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6294.648577] ? nmi_handle+0xbd/0x210 [ 6294.648578] ? default_do_nmi+0x4b/0x1b0 [ 6294.648579] ? do_nmi+0x280/0x380 [ 6294.648580] ? end_repeat_nmi+0x16/0x1f [ 6294.648581] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6294.648582] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6294.648583] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6294.648584] [ 6294.648585] do_raw_spin_lock+0x177/0x190 [ 6294.648586] _raw_spin_lock+0x21/0x30 [ 6294.648587] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6294.648588] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6294.648589] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6294.648590] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6294.648591] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6294.648592] ? woken_wake_function+0x30/0x30 [ 6294.648593] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6294.648594] kthread+0x1d1/0x200 [ 6294.648595] ? set_kthread_struct+0x70/0x70 [ 6294.648596] ret_from_fork+0x1f/0x30 [ 6294.649565] Sending NMI from CPU 2 to CPUs 1: [ 6294.919757] NMI backtrace for cpu 1 [ 6294.919765] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6294.919767] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6294.919769] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6294.919770] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6294.919773] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [ 6294.919775] RAX: 0000000000006a1f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6294.919776] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6294.919778] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6294.919780] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6294.919781] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6294.919782] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6294.919783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6294.919785] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6294.919786] Call Trace: [ 6294.919787] [ 6294.919788] ? show_regs.cold.9+0x22/0x2f [ 6294.919789] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6294.919790] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6294.919791] ? nmi_handle+0xbd/0x210 [ 6294.919792] ? default_do_nmi+0x4b/0x1b0 [ 6294.919793] ? do_nmi+0x280/0x380 [ 6294.919794] ? end_repeat_nmi+0x16/0x1f [ 6294.919795] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6294.919796] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6294.919798] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6294.919798] [ 6294.919799] do_raw_spin_lock+0x177/0x190 [ 6294.919800] _raw_spin_lock+0x21/0x30 [ 6294.919801] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6294.919802] ? do_raw_spin_unlock+0x75/0x190 [ 6294.919803] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6294.919804] ? do_wait_intr+0xf0/0xf0 [ 6294.919805] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6294.919807] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6294.919808] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6294.919809] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6294.919810] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6294.919811] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6294.919812] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6294.919813] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6294.919814] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6294.919816] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6294.919817] lmv_intent_open+0x338/0x1150 [lmv] [ 6294.919818] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6294.919819] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6294.919820] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6294.919821] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6294.919822] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6294.919823] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6294.919824] ? do_raw_spin_unlock+0x75/0x190 [ 6294.919825] ? _raw_spin_unlock+0x12/0x30 [ 6294.919826] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6294.919828] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6294.919829] ll_atomic_open+0x36a/0xd70 [lustre] [ 6294.919830] lookup_open+0x356/0xb40 [ 6294.919831] do_last+0x313/0xfc0 [ 6294.919832] ? nd_jump_root+0xe5/0x160 [ 6294.919833] ? path_init+0x437/0x520 [ 6294.919834] path_openat+0xf7/0x500 [ 6294.919834] ? do_raw_spin_unlock+0x75/0x190 [ 6294.919835] do_filp_open+0x99/0x140 [ 6294.919836] ? getname_flags+0x6e/0x330 [ 6294.919837] ? __check_object_size+0xff/0x256 [ 6294.919839] ? do_raw_spin_unlock+0x75/0x190 [ 6294.919840] ? _raw_spin_unlock+0x12/0x30 [ 6294.919841] do_sys_openat2+0x2b4/0x410 [ 6294.919842] do_sys_open+0x73/0xa0 [ 6294.919842] __x64_sys_openat+0x24/0x30 [ 6294.919843] do_syscall_64+0xc1/0x440 [ 6294.919844] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6294.919846] RIP: 0033:0x7f53425a5332 [ 6294.919847] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6294.919849] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6294.919851] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6294.919852] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6294.919854] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6294.919855] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6294.919857] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6294.923140] NMI backtrace for cpu 2 [ 6295.317481] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6295.326674] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6295.334079] Call Trace: [ 6295.334841] [ 6295.336629] ? dump_stack+0xbb/0x10e [ 6295.343197] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6295.344994] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6295.347953] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6295.350549] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6295.352678] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6295.355323] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6295.356807] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6295.358773] ? update_process_times+0x6d/0xc0 [ 6295.360287] ? tick_sched_handle.isra.18+0x27/0x90 [ 6295.362611] ? tick_sched_timer+0x53/0xb0 [ 6295.363973] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6295.365760] ? tick_sched_handle.isra.18+0x90/0x90 [ 6295.367894] ? hrtimer_interrupt+0x128/0x2c0 [ 6295.369418] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6295.371436] ? apic_timer_interrupt+0xf/0x20 [ 6295.373047] [ 6295.373955] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6295.375899] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6295.377682] ? do_raw_spin_lock+0x177/0x190 [ 6295.379054] ? _raw_spin_lock+0x21/0x30 [ 6295.380355] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6295.382429] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6295.384637] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6295.386738] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6295.389578] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6295.392150] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6295.394542] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6295.396473] ? __next_timer_interrupt+0x160/0x160 [ 6295.398146] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6295.400781] ? do_wait_intr+0xf0/0xf0 [ 6295.402586] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6295.405140] ? kthread+0x1d1/0x200 [ 6295.406964] ? set_kthread_struct+0x70/0x70 [ 6295.413365] ? ret_from_fork+0x1f/0x30 [ 6320.052249] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 6320.055019] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 6320.055428] Modules linked in: [ 6320.058861] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6320.059011] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6320.059019] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6320.059022] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6320.060531] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 6320.060542] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6320.060618] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6320.060624] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6320.060626] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6320.060638] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6320.060651] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6320.060656] RAX: 0000000000007f6f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6320.060659] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6320.060660] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 6320.060664] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6320.060670] Modules linked in: [ 6320.060671] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6320.060675] lustre(O) [ 6320.060676] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6320.060680] osp(O) ofd(O) [ 6320.060683] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6320.060687] lod(O) [ 6320.060690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6320.060692] mdt(O) [ 6320.060696] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6320.060697] mdd(O) mgs(O) lquota(O) lfsck(O) [ 6320.060707] Call Trace: [ 6320.060707] obdecho(O) mgc(O) mdc(O) lov(O) [ 6320.060715] [ 6320.060719] osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) [ 6320.060725] ? show_regs.cold.9+0x22/0x2f [ 6320.060732] ptlrpc(O) obdclass(O) ksocklnd(O) [ 6320.060738] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6320.060741] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss [ 6320.060751] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6320.060757] nfsv4 dns_resolver [ 6320.060762] ? watchdog+0x40/0x40 [ 6320.060765] intel_rapl_msr intel_rapl_common sb_edac [ 6320.060772] ? hrtimer_interrupt+0x128/0x2c0 [ 6320.060776] rapl i2c_piix4 pcspkr squashfs ata_generic [ 6320.060782] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6320.060790] crct10dif_pclmul crc32_pclmul crc32c_intel [ 6320.060796] ? apic_timer_interrupt+0xf/0x20 [ 6320.060799] ghash_clmulni_intel ata_piix [ 6320.060805] [ 6320.060806] serio_raw libata [ 6320.060811] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6320.060815] dm_mirror dm_region_hash [ 6320.060821] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6320.060824] dm_log dm_mod [ 6320.060833] do_raw_spin_lock+0x177/0x190 [ 6320.060836] sha512_ssse3 sha512_generic [ 6320.060843] _raw_spin_lock+0x21/0x30 [ 6320.060847] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6320.060856] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6320.060858] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6320.060874] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6320.060878] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6320.060882] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6320.060884] RDX: 0000000000003814 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6320.060887] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6320.060889] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6320.060892] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6320.060894] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6320.060897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6320.060900] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6320.060906] Call Trace: [ 6320.060909] [ 6320.060911] ? show_regs.cold.9+0x22/0x2f [ 6320.060918] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6320.060922] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6320.060926] ? watchdog+0x40/0x40 [ 6320.060930] ? hrtimer_interrupt+0x128/0x2c0 [ 6320.060934] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6320.060939] ? apic_timer_interrupt+0xf/0x20 [ 6320.060943] [ 6320.060946] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6320.060950] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6320.060955] do_raw_spin_lock+0x177/0x190 [ 6320.060959] _raw_spin_lock+0x21/0x30 [ 6320.060850] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6320.060964] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6320.061636] obd_mod_free+0x46/0x140 [ptlrpc] [ 6320.062235] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6320.062242] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6320.062247] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6320.062249] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6320.062251] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6320.062253] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6320.062255] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6320.062257] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6320.062260] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6320.062263] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6320.062270] Call Trace: [ 6320.062273] [ 6320.062276] ? show_regs.cold.9+0x22/0x2f [ 6320.062285] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6320.062290] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6320.062293] ? watchdog+0x40/0x40 [ 6320.062297] ? hrtimer_interrupt+0x128/0x2c0 [ 6320.062301] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6320.062307] ? apic_timer_interrupt+0xf/0x20 [ 6320.062311] [ 6320.062314] ? native_safe_halt+0x2e/0x50 [ 6320.062318] kvm_wait+0xcd/0xe0 [ 6320.062326] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6320.062333] do_raw_spin_lock+0x177/0x190 [ 6320.062337] _raw_spin_lock+0x21/0x30 [ 6320.062341] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6320.062544] ? do_raw_spin_unlock+0x75/0x190 [ 6320.062551] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6320.062757] ? do_wait_intr+0xf0/0xf0 [ 6320.062766] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6320.062990] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6320.063472] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6320.063657] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6320.063743] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6320.063772] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6320.063791] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6320.063808] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6320.063863] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6320.064392] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6320.064453] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6320.064459] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6320.064465] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6320.064778] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6320.065686] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6320.065781] lmv_intent_open+0x338/0x1150 [lmv] [ 6320.065893] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6320.066001] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6320.066325] [ 6320.066328] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6320.066340] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6320.066344] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6320.066349] RAX: 0000000000007842 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6320.066352] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6320.066354] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6320.066356] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6320.066358] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6320.066362] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6320.066365] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6320.066367] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6320.066374] Call Trace: [ 6320.066378] [ 6320.066382] ? show_regs.cold.9+0x22/0x2f [ 6320.066391] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6320.066397] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6320.066401] ? watchdog+0x40/0x40 [ 6320.066405] ? hrtimer_interrupt+0x128/0x2c0 [ 6320.066409] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6320.066418] ? apic_timer_interrupt+0xf/0x20 [ 6320.066423] [ 6320.066426] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6320.066431] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6320.066435] ? newidle_balance.isra.93+0x207/0x580 [ 6320.066441] do_raw_spin_lock+0x177/0x190 [ 6320.066446] _raw_spin_lock+0x21/0x30 [ 6320.066451] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6320.066735] process_one_work+0x2c8/0x700 [ 6320.066743] worker_thread+0x296/0x6e0 [ 6320.066747] ? rescuer_thread+0x570/0x570 [ 6320.066752] kthread+0x1d1/0x200 [ 6320.066757] ? set_kthread_struct+0x70/0x70 [ 6320.066761] ret_from_fork+0x1f/0x30 [ 6320.066929] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6320.066958] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6320.066805] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6320.069250] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6320.069497] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6320.069728] ? woken_wake_function+0x30/0x30 [ 6320.069738] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6320.069970] kthread+0x1d1/0x200 [ 6320.069976] ? set_kthread_struct+0x70/0x70 [ 6320.069980] ret_from_fork+0x1f/0x30 [ 6320.092245] mdc_commit_open+0x91/0xb0 [mdc] [ 6320.098292] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6320.098415] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6320.098503] ? do_raw_spin_unlock+0x75/0x190 [ 6320.098511] ? _raw_spin_unlock+0x12/0x30 [ 6320.105622] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6320.105963] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6320.106163] after_reply+0xe2c/0x1780 [ptlrpc] [ 6320.106364] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6320.109125] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6320.109161] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6320.109188] ll_atomic_open+0x36a/0xd70 [lustre] [ 6320.109310] lookup_open+0x356/0xb40 [ 6320.109320] do_last+0x313/0xfc0 [ 6320.109325] ? nd_jump_root+0xe5/0x160 [ 6320.109330] ? path_init+0x437/0x520 [ 6320.109335] path_openat+0xf7/0x500 [ 6320.109340] ? do_raw_spin_unlock+0x75/0x190 [ 6320.109346] do_filp_open+0x99/0x140 [ 6320.109354] ? getname_flags+0x6e/0x330 [ 6320.109358] ? __check_object_size+0xff/0x256 [ 6320.109364] ? do_raw_spin_unlock+0x75/0x190 [ 6320.109367] ? _raw_spin_unlock+0x12/0x30 [ 6320.109373] do_sys_openat2+0x2b4/0x410 [ 6320.109377] do_sys_open+0x73/0xa0 [ 6320.109382] __x64_sys_openat+0x24/0x30 [ 6320.109385] do_syscall_64+0xc1/0x440 [ 6320.109392] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6320.109398] RIP: 0033:0x7f53425a5332 [ 6320.109405] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6320.109409] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6320.109414] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6320.109417] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6320.109420] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6320.109422] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6320.109424] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6321.069298] ? __next_timer_interrupt+0x160/0x160 [ 6321.071732] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6321.074540] ? do_wait_intr+0xf0/0xf0 [ 6321.077606] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6321.081468] kthread+0x1d1/0x200 [ 6321.083328] ? set_kthread_struct+0x70/0x70 [ 6321.085500] ret_from_fork+0x1f/0x30 [ 6323.681638] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3686s! [ 6323.687565] Showing busy workqueues and worker pools: [ 6323.690795] workqueue events: flags=0x0 [ 6323.693783] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6323.698910] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6323.704757] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6323.708973] pending: cpu_latency_work [ptlrpc] [ 6323.713701] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6323.719304] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6323.731243] workqueue events_long: flags=0x0 [ 6323.741181] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6323.753593] pending: lu_objects_flush [obdclass] [ 6323.768011] workqueue events_unbound: flags=0x2 [ 6323.776828] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6323.787654] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6323.793773] workqueue events_power_efficient: flags=0x80 [ 6323.797068] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6323.818228] pending: check_lifetime, do_cache_clean [ 6323.831375] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6323.843427] pending: neigh_periodic_work, neigh_periodic_work [ 6323.857117] workqueue mm_percpu_wq: flags=0x8 [ 6323.862081] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6323.873447] pending: vmstat_update, lru_add_drain_per_cpu [ 6323.881765] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6323.890154] pending: vmstat_update, lru_add_drain_per_cpu [ 6323.895401] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6323.903405] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6323.911860] workqueue xprtiod: flags=0xa [ 6323.914275] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6323.918950] pending: xprt_autoclose, xs_error_handle [ 6323.920637] workqueue ipv6_addrconf: flags=0x40008 [ 6323.923570] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6323.931008] pending: addrconf_verify_work [ 6323.933930] workqueue knbd0-recv: flags=0x1a [ 6323.939014] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6323.942315] in-flight: 651:recv_work [ 6323.944531] workqueue ptlrpc_pinger: flags=0x2 [ 6323.948263] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6323.952028] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6323.955788] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3686s workers=4 idle: 15979 31824 8 [ 6323.960710] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6314s workers=2 idle: 654 [ 6348.052134] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 6348.055176] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6348.055716] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 6348.058030] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 6348.058042] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6348.058108] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6348.058113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6348.058116] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6348.058128] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6348.058133] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 6348.058137] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6348.058139] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6348.058142] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6348.058143] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6348.058146] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6348.058149] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6348.058152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6348.058154] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6348.058163] Call Trace: [ 6348.058169] [ 6348.058294] ? show_regs.cold.9+0x22/0x2f [ 6348.058303] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6348.058309] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6348.058312] ? watchdog+0x40/0x40 [ 6348.058316] ? hrtimer_interrupt+0x128/0x2c0 [ 6348.058319] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6348.058325] ? apic_timer_interrupt+0xf/0x20 [ 6348.058329] [ 6348.058332] ? native_safe_halt+0x2e/0x50 [ 6348.058335] kvm_wait+0xcd/0xe0 [ 6348.058343] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6348.058351] do_raw_spin_lock+0x177/0x190 [ 6348.058355] _raw_spin_lock+0x21/0x30 [ 6348.058359] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6348.058728] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6348.058947] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6348.060638] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6348.060865] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6348.061489] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 6348.061497] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6348.061571] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6348.061577] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6348.061580] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6348.061591] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6348.061596] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6348.061601] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6348.061604] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6348.061607] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6348.061609] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6348.061612] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6348.061615] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6348.061618] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6348.061620] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6348.061629] Call Trace: [ 6348.061633] [ 6348.061637] ? show_regs.cold.9+0x22/0x2f [ 6348.061646] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6348.061652] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6348.061656] ? watchdog+0x40/0x40 [ 6348.061660] ? hrtimer_interrupt+0x128/0x2c0 [ 6348.061665] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6348.061671] ? apic_timer_interrupt+0xf/0x20 [ 6348.061676] [ 6348.061679] ? native_safe_halt+0x2e/0x50 [ 6348.061683] kvm_wait+0xcd/0xe0 [ 6348.061690] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6348.061698] do_raw_spin_lock+0x177/0x190 [ 6348.061702] _raw_spin_lock+0x21/0x30 [ 6348.061707] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6348.061959] obd_mod_free+0x46/0x140 [ptlrpc] [ 6348.062246] mdc_commit_open+0x91/0xb0 [mdc] [ 6348.062398] ? woken_wake_function+0x30/0x30 [ 6348.062292] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6348.062407] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6348.063253] kthread+0x1d1/0x200 [ 6348.063260] ? set_kthread_struct+0x70/0x70 [ 6348.063264] ret_from_fork+0x1f/0x30 [ 6348.063993] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6348.064210] after_reply+0xe2c/0x1780 [ptlrpc] [ 6348.065767] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6348.065998] ? __next_timer_interrupt+0x160/0x160 [ 6348.066009] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6348.066247] ? do_wait_intr+0xf0/0xf0 [ 6348.066256] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6348.067282] kthread+0x1d1/0x200 [ 6348.067289] ? set_kthread_struct+0x70/0x70 [ 6348.067293] ret_from_fork+0x1f/0x30 [ 6348.083349] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6348.086603] Modules linked in: [ 6348.089653] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6348.116101] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6348.116182] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6348.116190] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6348.116193] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6348.116207] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6348.116212] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6348.116216] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6348.116219] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6348.116221] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6348.116224] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6348.116227] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6348.116230] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6348.116233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6348.116235] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6348.116244] Call Trace: [ 6348.116249] [ 6348.116255] ? show_regs.cold.9+0x22/0x2f [ 6348.116267] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6348.116275] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6348.116280] ? watchdog+0x40/0x40 [ 6348.116284] ? hrtimer_interrupt+0x128/0x2c0 [ 6348.116288] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6348.116294] ? apic_timer_interrupt+0xf/0x20 [ 6348.116299] [ 6348.116301] ? native_safe_halt+0x2e/0x50 [ 6348.116305] kvm_wait+0xcd/0xe0 [ 6348.116313] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6348.116321] do_raw_spin_lock+0x177/0x190 [ 6348.116326] _raw_spin_lock+0x21/0x30 [ 6348.116331] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6348.116622] ? do_raw_spin_unlock+0x75/0x190 [ 6348.116628] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6348.116845] ? do_wait_intr+0xf0/0xf0 [ 6348.116852] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6348.117070] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6348.117305] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6348.117554] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6348.117671] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6348.117711] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6348.117743] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6348.117774] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6348.117868] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6348.118102] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6348.118133] lmv_intent_open+0x338/0x1150 [lmv] [ 6348.124231] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6348.124649] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6348.124660] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6348.124665] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6348.124669] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6348.124671] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6348.124673] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6348.124675] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6348.124678] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6348.124682] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6348.124684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6348.124687] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6348.124696] Call Trace: [ 6348.124702] [ 6348.124707] ? show_regs.cold.9+0x22/0x2f [ 6348.124717] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6348.124722] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6348.124727] ? watchdog+0x40/0x40 [ 6348.124732] ? hrtimer_interrupt+0x128/0x2c0 [ 6348.124736] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6348.124742] ? apic_timer_interrupt+0xf/0x20 [ 6348.124746] [ 6348.124749] ? native_safe_halt+0x2e/0x50 [ 6348.124754] kvm_wait+0xcd/0xe0 [ 6348.124762] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6348.124769] ? newidle_balance.isra.93+0x207/0x580 [ 6348.124776] do_raw_spin_lock+0x177/0x190 [ 6348.124780] _raw_spin_lock+0x21/0x30 [ 6348.124785] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6348.128773] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6348.128901] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6348.135160] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6348.135195] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6348.135290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6348.135405] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6348.135508] ? do_raw_spin_unlock+0x75/0x190 [ 6348.135518] ? _raw_spin_unlock+0x12/0x30 [ 6348.135525] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6348.135554] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6348.135576] ll_atomic_open+0x36a/0xd70 [lustre] [ 6348.135669] lookup_open+0x356/0xb40 [ 6348.135680] do_last+0x313/0xfc0 [ 6348.135685] ? nd_jump_root+0xe5/0x160 [ 6348.135690] ? path_init+0x437/0x520 [ 6348.135696] path_openat+0xf7/0x500 [ 6348.135701] ? do_raw_spin_unlock+0x75/0x190 [ 6348.135707] do_filp_open+0x99/0x140 [ 6348.135714] ? getname_flags+0x6e/0x330 [ 6348.135718] ? __check_object_size+0xff/0x256 [ 6348.135726] ? do_raw_spin_unlock+0x75/0x190 [ 6348.135730] ? _raw_spin_unlock+0x12/0x30 [ 6348.135734] do_sys_openat2+0x2b4/0x410 [ 6348.135739] do_sys_open+0x73/0xa0 [ 6348.135743] __x64_sys_openat+0x24/0x30 [ 6348.135747] do_syscall_64+0xc1/0x440 [ 6348.135754] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6348.135760] RIP: 0033:0x7f53425a5332 [ 6348.135769] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6348.135773] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6348.135778] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6348.135780] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6348.135782] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6348.135785] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6348.135788] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6348.781019] process_one_work+0x2c8/0x700 [ 6348.782475] worker_thread+0x296/0x6e0 [ 6348.784619] ? rescuer_thread+0x570/0x570 [ 6348.786262] kthread+0x1d1/0x200 [ 6348.789142] ? set_kthread_struct+0x70/0x70 [ 6348.792863] ret_from_fork+0x1f/0x30 [ 6354.406641] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3717s! [ 6354.413272] Showing busy workqueues and worker pools: [ 6354.420576] workqueue events: flags=0x0 [ 6354.421749] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6354.430595] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6354.439191] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6354.443815] pending: cpu_latency_work [ptlrpc] [ 6354.446527] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6354.450402] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6354.456230] workqueue events_long: flags=0x0 [ 6354.459924] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6354.462976] pending: lu_objects_flush [obdclass] [ 6354.464763] workqueue events_unbound: flags=0x2 [ 6354.474629] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6354.478080] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6354.480811] workqueue events_power_efficient: flags=0x80 [ 6354.484954] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6354.492211] pending: check_lifetime, do_cache_clean [ 6354.493984] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6354.503936] pending: neigh_periodic_work, neigh_periodic_work [ 6354.511030] workqueue mm_percpu_wq: flags=0x8 [ 6354.517710] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6354.526556] pending: vmstat_update, lru_add_drain_per_cpu [ 6354.528421] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6354.535391] pending: vmstat_update, lru_add_drain_per_cpu [ 6354.545218] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6354.551141] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6354.559857] workqueue xprtiod: flags=0xa [ 6354.564321] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6354.570187] pending: xprt_autoclose, xs_error_handle [ 6354.572800] workqueue ipv6_addrconf: flags=0x40008 [ 6354.582955] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6354.590771] pending: addrconf_verify_work [ 6354.593192] workqueue knbd0-recv: flags=0x1a [ 6354.596874] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6354.601072] in-flight: 651:recv_work [ 6354.602839] workqueue ptlrpc_pinger: flags=0x2 [ 6354.607075] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6354.610556] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6354.620466] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3717s workers=4 idle: 15979 31824 8 [ 6354.626405] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6345s workers=2 idle: 654 [ 6376.055013] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 6376.058571] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6376.058672] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 6376.059243] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 6376.059249] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6376.059310] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6376.059315] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6376.059318] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6376.059333] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6376.059337] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6376.059342] RAX: 0000000000007f8d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6376.059344] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6376.059346] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6376.059349] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6376.059351] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6376.059354] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6376.059357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6376.059359] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6376.059368] Call Trace: [ 6376.059373] [ 6376.059379] ? show_regs.cold.9+0x22/0x2f [ 6376.059388] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6376.059395] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6376.059400] ? watchdog+0x40/0x40 [ 6376.059405] ? hrtimer_interrupt+0x128/0x2c0 [ 6376.059409] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6376.059419] ? apic_timer_interrupt+0xf/0x20 [ 6376.060092] [ 6376.060096] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6376.060102] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6376.060108] do_raw_spin_lock+0x177/0x190 [ 6376.060113] _raw_spin_lock+0x21/0x30 [ 6376.060119] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6376.060386] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6376.060924] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6376.061141] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6376.061352] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6376.062954] ? woken_wake_function+0x30/0x30 [ 6376.062964] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6376.063176] kthread+0x1d1/0x200 [ 6376.063181] ? set_kthread_struct+0x70/0x70 [ 6376.063185] ret_from_fork+0x1f/0x30 [ 6376.082605] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6376.082621] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6376.082624] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6376.082639] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6376.082644] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6376.082650] RAX: 0000000000005f0f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6376.082652] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6376.082654] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6376.082656] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6376.082659] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6376.082662] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6376.082665] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6376.082668] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6376.082676] Call Trace: [ 6376.082682] [ 6376.082688] ? show_regs.cold.9+0x22/0x2f [ 6376.082697] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6376.082703] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6376.082707] ? watchdog+0x40/0x40 [ 6376.082711] ? hrtimer_interrupt+0x128/0x2c0 [ 6376.082716] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6376.082725] ? apic_timer_interrupt+0xf/0x20 [ 6376.082730] [ 6376.082733] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6376.082742] do_raw_spin_lock+0x177/0x190 [ 6376.082747] _raw_spin_lock+0x21/0x30 [ 6376.082752] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6376.083258] ? do_raw_spin_unlock+0x75/0x190 [ 6376.083265] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6376.083469] ? do_wait_intr+0xf0/0xf0 [ 6376.083479] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6376.083679] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6376.093130] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) [ 6376.096432] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6376.130999] obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6376.131038] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6376.131043] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6376.131045] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6376.131107] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6376.131110] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6376.131114] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6376.131115] RDX: 0000000000002c06 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6376.131117] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6376.131119] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6376.131120] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6376.131122] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6376.131124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6376.131126] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6376.131132] Call Trace: [ 6376.131136] [ 6376.131141] ? show_regs.cold.9+0x22/0x2f [ 6376.131148] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6376.131152] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6376.131155] ? watchdog+0x40/0x40 [ 6376.131158] ? hrtimer_interrupt+0x128/0x2c0 [ 6376.131160] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6376.131167] ? apic_timer_interrupt+0xf/0x20 [ 6376.131170] [ 6376.131172] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6376.131175] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6376.131179] do_raw_spin_lock+0x177/0x190 [ 6376.131182] _raw_spin_lock+0x21/0x30 [ 6376.131187] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6376.131405] obd_mod_free+0x46/0x140 [ptlrpc] [ 6376.131609] mdc_commit_open+0x91/0xb0 [mdc] [ 6376.131645] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6376.131799] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6376.132155] after_reply+0xe2c/0x1780 [ptlrpc] [ 6376.137376] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6376.146686] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6376.147533] ? __next_timer_interrupt+0x160/0x160 [ 6376.147545] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6376.147762] ? do_wait_intr+0xf0/0xf0 [ 6376.147772] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6376.150096] kthread+0x1d1/0x200 [ 6376.150108] ? set_kthread_struct+0x70/0x70 [ 6376.150112] ret_from_fork+0x1f/0x30 [ 6376.540505] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6376.546919] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6376.549769] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6376.553268] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6376.559490] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6376.562699] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6376.565229] lmv_intent_open+0x338/0x1150 [lmv] [ 6376.567522] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6376.577511] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6376.581142] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6376.583639] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6376.586764] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6376.589762] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6376.593590] ? do_raw_spin_unlock+0x75/0x190 [ 6376.600635] ? _raw_spin_unlock+0x12/0x30 [ 6376.605593] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6376.609262] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6376.614533] ll_atomic_open+0x36a/0xd70 [lustre] [ 6376.618337] lookup_open+0x356/0xb40 [ 6376.619799] do_last+0x313/0xfc0 [ 6376.621225] ? nd_jump_root+0xe5/0x160 [ 6376.622635] ? path_init+0x437/0x520 [ 6376.623821] path_openat+0xf7/0x500 [ 6376.625219] ? do_raw_spin_unlock+0x75/0x190 [ 6376.627802] do_filp_open+0x99/0x140 [ 6376.628946] ? getname_flags+0x6e/0x330 [ 6376.629967] ? __check_object_size+0xff/0x256 [ 6376.631501] ? do_raw_spin_unlock+0x75/0x190 [ 6376.632745] ? _raw_spin_unlock+0x12/0x30 [ 6376.639461] do_sys_openat2+0x2b4/0x410 [ 6376.640652] do_sys_open+0x73/0xa0 [ 6376.641821] __x64_sys_openat+0x24/0x30 [ 6376.643136] do_syscall_64+0xc1/0x440 [ 6376.648794] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6376.654548] RIP: 0033:0x7f53425a5332 [ 6376.656214] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6376.669996] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6376.672719] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6376.681572] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6376.685797] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6376.688824] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6376.696157] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6380.052082] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 6380.055712] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6380.077370] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6380.082677] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6380.086622] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6380.089278] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6380.092277] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6380.101624] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6380.105485] RAX: 0000000000002c82 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6380.111034] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6380.116616] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6380.120649] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6380.123865] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6380.126914] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6380.130691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6380.133320] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6380.136490] Call Trace: [ 6380.137596] [ 6380.138448] ? show_regs.cold.9+0x22/0x2f [ 6380.140486] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6380.142364] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6380.144500] ? watchdog+0x40/0x40 [ 6380.145804] ? hrtimer_interrupt+0x128/0x2c0 [ 6380.147795] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6380.150006] ? apic_timer_interrupt+0xf/0x20 [ 6380.151977] [ 6380.152698] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6380.155016] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6380.157256] ? newidle_balance.isra.93+0x207/0x580 [ 6380.159436] do_raw_spin_lock+0x177/0x190 [ 6380.161250] _raw_spin_lock+0x21/0x30 [ 6380.163046] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6380.165694] process_one_work+0x2c8/0x700 [ 6380.167644] worker_thread+0x296/0x6e0 [ 6380.169270] ? rescuer_thread+0x570/0x570 [ 6380.171153] kthread+0x1d1/0x200 [ 6380.172411] ? set_kthread_struct+0x70/0x70 [ 6380.174422] ret_from_fork+0x1f/0x30 [ 6385.123025] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3748s! [ 6385.128347] Showing busy workqueues and worker pools: [ 6385.135492] workqueue events: flags=0x0 [ 6385.138500] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6385.142823] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6385.152129] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6385.157920] pending: cpu_latency_work [ptlrpc] [ 6385.161590] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6385.166254] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6385.173578] workqueue events_long: flags=0x0 [ 6385.176483] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6385.182399] pending: lu_objects_flush [obdclass] [ 6385.186615] workqueue events_unbound: flags=0x2 [ 6385.188297] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6385.192136] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6385.194870] workqueue events_power_efficient: flags=0x80 [ 6385.198273] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6385.203708] pending: check_lifetime, do_cache_clean [ 6385.205685] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6385.209600] pending: neigh_periodic_work, neigh_periodic_work [ 6385.212349] workqueue mm_percpu_wq: flags=0x8 [ 6385.214597] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6385.217208] pending: vmstat_update, lru_add_drain_per_cpu [ 6385.221025] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6385.226673] pending: vmstat_update, lru_add_drain_per_cpu [ 6385.233349] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6385.238599] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6385.244560] workqueue xprtiod: flags=0xa [ 6385.246673] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6385.253367] pending: xprt_autoclose, xs_error_handle [ 6385.258994] workqueue ipv6_addrconf: flags=0x40008 [ 6385.262767] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6385.267380] pending: addrconf_verify_work [ 6385.270437] workqueue knbd0-recv: flags=0x1a [ 6385.275778] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6385.278946] in-flight: 651:recv_work [ 6385.281427] workqueue ptlrpc_pinger: flags=0x2 [ 6385.284001] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6385.292290] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6385.295872] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3748s workers=4 idle: 15979 31824 8 [ 6385.303583] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6375s workers=2 idle: 654 [ 6404.055493] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 6404.057077] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 6404.058061] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 6404.058069] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6404.058140] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6404.058146] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6404.058149] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6404.058162] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6404.058166] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6404.058170] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6404.058173] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6404.058175] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6404.058188] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6404.058191] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6404.058194] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6404.058198] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6404.058200] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6404.058208] Call Trace: [ 6404.058213] [ 6404.058218] ? show_regs.cold.9+0x22/0x2f [ 6404.058228] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6404.058234] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6404.058239] ? watchdog+0x40/0x40 [ 6404.058244] ? hrtimer_interrupt+0x128/0x2c0 [ 6404.058248] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6404.058255] ? apic_timer_interrupt+0xf/0x20 [ 6404.058260] [ 6404.058263] ? native_safe_halt+0x2e/0x50 [ 6404.058268] kvm_wait+0xcd/0xe0 [ 6404.058275] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6404.058283] do_raw_spin_lock+0x177/0x190 [ 6404.058288] _raw_spin_lock+0x21/0x30 [ 6404.058291] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6404.058569] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6404.058775] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6404.059314] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6404.059532] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6404.059745] ? woken_wake_function+0x30/0x30 [ 6404.059755] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6404.060714] kthread+0x1d1/0x200 [ 6404.060721] ? set_kthread_struct+0x70/0x70 [ 6404.060725] ret_from_fork+0x1f/0x30 [ 6404.061883] Modules linked in: lustre(O) [ 6404.067162] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6404.067234] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6404.067241] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6404.067244] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6404.067256] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6404.067261] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6404.067265] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6404.067267] RDX: 00000000000055f0 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6404.067269] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6404.067271] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6404.067273] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6404.067276] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6404.067279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6404.067281] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6404.067289] Call Trace: [ 6404.067295] [ 6404.067301] ? show_regs.cold.9+0x22/0x2f [ 6404.067310] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6404.067316] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6404.067320] ? watchdog+0x40/0x40 [ 6404.067325] ? hrtimer_interrupt+0x128/0x2c0 [ 6404.067328] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6404.067336] ? apic_timer_interrupt+0xf/0x20 [ 6404.067340] [ 6404.067343] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6404.067348] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6404.067353] do_raw_spin_lock+0x177/0x190 [ 6404.067358] _raw_spin_lock+0x21/0x30 [ 6404.067363] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6404.067640] obd_mod_free+0x46/0x140 [ptlrpc] [ 6404.067827] mdc_commit_open+0x91/0xb0 [mdc] [ 6404.067866] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6404.068058] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6404.068253] after_reply+0xe2c/0x1780 [ptlrpc] [ 6404.068469] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6404.068680] ? __next_timer_interrupt+0x160/0x160 [ 6404.068689] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6404.068889] ? do_wait_intr+0xf0/0xf0 [ 6404.068899] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6404.069104] kthread+0x1d1/0x200 [ 6404.069110] ? set_kthread_struct+0x70/0x70 [ 6404.069114] ret_from_fork+0x1f/0x30 [ 6404.399492] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6404.414810] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6404.421165] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6404.424273] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6404.426318] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6404.435609] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6404.439032] RAX: 0000000000007c07 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6404.442066] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6404.444933] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6404.447769] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6404.452497] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6404.455743] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6404.459790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6404.464460] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6404.468530] Call Trace: [ 6404.469496] [ 6404.470240] ? show_regs.cold.9+0x22/0x2f [ 6404.471818] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6404.475838] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6404.478225] ? watchdog+0x40/0x40 [ 6404.480682] ? hrtimer_interrupt+0x128/0x2c0 [ 6404.482197] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6404.484731] ? apic_timer_interrupt+0xf/0x20 [ 6404.486635] [ 6404.488357] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6404.490696] do_raw_spin_lock+0x177/0x190 [ 6404.493328] _raw_spin_lock+0x21/0x30 [ 6404.496060] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6404.497883] ? do_raw_spin_unlock+0x75/0x190 [ 6404.501100] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6404.507371] ? do_wait_intr+0xf0/0xf0 [ 6404.510290] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6404.513827] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6404.520658] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6404.523011] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6404.527328] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6404.529949] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6404.532082] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6404.533853] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6404.536027] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6404.540036] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6404.541736] lmv_intent_open+0x338/0x1150 [lmv] [ 6404.543884] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6404.546019] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6404.549650] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6404.551095] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6404.552920] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6404.554486] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6404.555825] ? do_raw_spin_unlock+0x75/0x190 [ 6404.557285] ? _raw_spin_unlock+0x12/0x30 [ 6404.558654] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6404.560243] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6404.561952] ll_atomic_open+0x36a/0xd70 [lustre] [ 6404.563653] lookup_open+0x356/0xb40 [ 6404.565096] do_last+0x313/0xfc0 [ 6404.566489] ? nd_jump_root+0xe5/0x160 [ 6404.567914] ? path_init+0x437/0x520 [ 6404.569227] path_openat+0xf7/0x500 [ 6404.570535] ? do_raw_spin_unlock+0x75/0x190 [ 6404.571889] do_filp_open+0x99/0x140 [ 6404.573002] ? getname_flags+0x6e/0x330 [ 6404.574134] ? __check_object_size+0xff/0x256 [ 6404.575559] ? do_raw_spin_unlock+0x75/0x190 [ 6404.577012] ? _raw_spin_unlock+0x12/0x30 [ 6404.578316] do_sys_openat2+0x2b4/0x410 [ 6404.579539] do_sys_open+0x73/0xa0 [ 6404.580796] __x64_sys_openat+0x24/0x30 [ 6404.581964] do_syscall_64+0xc1/0x440 [ 6404.583096] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6404.584786] RIP: 0033:0x7f53425a5332 [ 6404.588010] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6404.599840] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6404.602797] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6404.606128] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6404.611835] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6404.617081] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6404.621269] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6412.054097] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 6412.061015] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6412.101731] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6412.123428] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6412.138273] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6412.150088] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6412.155214] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6412.182287] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6412.189700] RAX: 0000000000007f86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6412.196519] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6412.198826] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6412.210684] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6412.222194] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6412.229700] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6412.236008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6412.240330] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6412.245691] Call Trace: [ 6412.246433] [ 6412.247159] ? show_regs.cold.9+0x22/0x2f [ 6412.249646] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6412.253545] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6412.255130] ? watchdog+0x40/0x40 [ 6412.257756] ? hrtimer_interrupt+0x128/0x2c0 [ 6412.266380] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6412.269835] ? apic_timer_interrupt+0xf/0x20 [ 6412.271688] [ 6412.272711] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6412.282705] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6412.286027] ? newidle_balance.isra.93+0x207/0x580 [ 6412.289785] do_raw_spin_lock+0x177/0x190 [ 6412.295003] _raw_spin_lock+0x21/0x30 [ 6412.296418] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6412.300059] process_one_work+0x2c8/0x700 [ 6412.301850] worker_thread+0x296/0x6e0 [ 6412.304824] ? rescuer_thread+0x570/0x570 [ 6412.307367] kthread+0x1d1/0x200 [ 6412.308712] ? set_kthread_struct+0x70/0x70 [ 6412.310156] ret_from_fork+0x1f/0x30 [ 6415.329040] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3778s! [ 6415.333448] Showing busy workqueues and worker pools: [ 6415.335796] workqueue events: flags=0x0 [ 6415.337774] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6415.341393] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6415.351027] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6415.355551] pending: cpu_latency_work [ptlrpc] [ 6415.359460] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6415.364140] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6415.369147] workqueue events_long: flags=0x0 [ 6415.370641] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6415.373231] pending: lu_objects_flush [obdclass] [ 6415.376511] workqueue events_unbound: flags=0x2 [ 6415.379491] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6415.383648] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6415.388827] workqueue events_power_efficient: flags=0x80 [ 6415.394943] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6415.398458] pending: check_lifetime, do_cache_clean [ 6415.403800] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6415.406649] pending: neigh_periodic_work, neigh_periodic_work [ 6415.408723] workqueue mm_percpu_wq: flags=0x8 [ 6415.413182] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6415.420437] pending: vmstat_update, lru_add_drain_per_cpu [ 6415.425089] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6415.436106] pending: vmstat_update, lru_add_drain_per_cpu [ 6415.444739] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6415.451779] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6415.456789] workqueue xprtiod: flags=0xa [ 6415.459329] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6415.465571] pending: xprt_autoclose, xs_error_handle [ 6415.478468] workqueue ipv6_addrconf: flags=0x40008 [ 6415.485761] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6415.498892] pending: addrconf_verify_work [ 6415.501015] workqueue knbd0-recv: flags=0x1a [ 6415.507613] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6415.514482] in-flight: 651:recv_work [ 6415.515481] workqueue ptlrpc_pinger: flags=0x2 [ 6415.525106] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6415.529467] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6415.533996] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3778s workers=4 idle: 15979 31824 8 [ 6415.538670] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6405s workers=2 idle: 654 [ 6432.056184] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 6432.057981] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 6432.058030] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 6432.058035] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6432.058110] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6432.058116] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6432.058119] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6432.058132] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6432.058137] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6432.058141] RAX: 0000000000007d28 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6432.058143] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6432.058144] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6432.058147] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6432.058149] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6432.058152] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6432.058154] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6432.058157] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6432.058167] Call Trace: [ 6432.058172] [ 6432.058178] ? show_regs.cold.9+0x22/0x2f [ 6432.058187] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6432.058194] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6432.058199] ? watchdog+0x40/0x40 [ 6432.058204] ? hrtimer_interrupt+0x128/0x2c0 [ 6432.058208] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6432.058216] ? apic_timer_interrupt+0xf/0x20 [ 6432.058220] [ 6432.058222] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6432.058227] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6432.058233] do_raw_spin_lock+0x177/0x190 [ 6432.058237] _raw_spin_lock+0x21/0x30 [ 6432.058243] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6432.058499] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6432.058695] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6432.060661] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6432.060733] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6432.060740] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6432.060742] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6432.060755] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6432.060758] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6432.060762] RAX: 0000000000007d88 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6432.060765] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6432.060766] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6432.060768] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6432.060770] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6432.060773] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6432.060776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6432.060778] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6432.060786] Call Trace: [ 6432.060790] [ 6432.060794] ? show_regs.cold.9+0x22/0x2f [ 6432.060803] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6432.060809] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6432.060813] ? watchdog+0x40/0x40 [ 6432.060818] ? hrtimer_interrupt+0x128/0x2c0 [ 6432.060821] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6432.060829] ? apic_timer_interrupt+0xf/0x20 [ 6432.060833] [ 6432.060836] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6432.060842] do_raw_spin_lock+0x177/0x190 [ 6432.060847] _raw_spin_lock+0x21/0x30 [ 6432.061037] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6432.061307] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6432.061536] ? woken_wake_function+0x30/0x30 [ 6432.061546] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6432.061778] kthread+0x1d1/0x200 [ 6432.061783] ? set_kthread_struct+0x70/0x70 [ 6432.061787] ret_from_fork+0x1f/0x30 [ 6432.060852] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6432.069412] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6432.069489] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6432.069497] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6432.069500] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6432.069514] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6432.069518] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6432.069523] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6432.069526] RDX: 0000000000002c1a RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6432.069529] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6432.069531] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6432.069534] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6432.069537] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6432.069540] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6432.069542] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6432.069551] Call Trace: [ 6432.069556] [ 6432.069562] ? show_regs.cold.9+0x22/0x2f [ 6432.069572] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6432.069578] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6432.069582] ? watchdog+0x40/0x40 [ 6432.069587] ? hrtimer_interrupt+0x128/0x2c0 [ 6432.069591] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6432.069599] ? apic_timer_interrupt+0xf/0x20 [ 6432.069605] [ 6432.069608] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6432.069613] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6432.069618] do_raw_spin_lock+0x177/0x190 [ 6432.069624] _raw_spin_lock+0x21/0x30 [ 6432.069630] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6432.069913] obd_mod_free+0x46/0x140 [ptlrpc] [ 6432.070114] mdc_commit_open+0x91/0xb0 [mdc] [ 6432.070151] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6432.070345] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6432.070535] after_reply+0xe2c/0x1780 [ptlrpc] [ 6432.070724] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6432.093166] ? do_raw_spin_unlock+0x75/0x190 [ 6432.093188] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6432.093426] ? do_wait_intr+0xf0/0xf0 [ 6432.093433] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6432.093558] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6432.093713] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6432.093865] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6432.103649] ? __next_timer_interrupt+0x160/0x160 [ 6432.119449] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6432.121277] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6432.121333] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6432.121363] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6432.121475] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6432.121761] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6432.121792] lmv_intent_open+0x338/0x1150 [lmv] [ 6432.121820] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6432.121916] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6432.122005] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6432.122028] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6432.134866] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6432.143692] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6432.143823] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6432.148444] ? do_wait_intr+0xf0/0xf0 [ 6432.148462] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6432.151613] kthread+0x1d1/0x200 [ 6432.151621] ? set_kthread_struct+0x70/0x70 [ 6432.151626] ret_from_fork+0x1f/0x30 [ 6432.605148] ? do_raw_spin_unlock+0x75/0x190 [ 6432.607286] ? _raw_spin_unlock+0x12/0x30 [ 6432.612333] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6432.613649] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6432.615404] ll_atomic_open+0x36a/0xd70 [lustre] [ 6432.616743] lookup_open+0x356/0xb40 [ 6432.617872] do_last+0x313/0xfc0 [ 6432.618769] ? nd_jump_root+0xe5/0x160 [ 6432.619797] ? path_init+0x437/0x520 [ 6432.620770] path_openat+0xf7/0x500 [ 6432.621797] ? do_raw_spin_unlock+0x75/0x190 [ 6432.623238] do_filp_open+0x99/0x140 [ 6432.624514] ? getname_flags+0x6e/0x330 [ 6432.628443] ? __check_object_size+0xff/0x256 [ 6432.633571] ? do_raw_spin_unlock+0x75/0x190 [ 6432.641529] ? _raw_spin_unlock+0x12/0x30 [ 6432.647726] do_sys_openat2+0x2b4/0x410 [ 6432.651721] do_sys_open+0x73/0xa0 [ 6432.654275] __x64_sys_openat+0x24/0x30 [ 6432.655956] do_syscall_64+0xc1/0x440 [ 6432.658573] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6432.661148] RIP: 0033:0x7f53425a5332 [ 6432.662664] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6432.684521] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6432.692195] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6432.694203] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6432.703580] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6432.715490] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6432.717461] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6440.052883] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 6440.061432] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6440.093351] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6440.101357] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6440.104173] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6440.107931] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6440.109729] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6440.115399] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6440.118247] RAX: 000000000000745f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6440.120498] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6440.123276] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6440.126137] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6440.129258] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6440.131958] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6440.135013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6440.137279] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6440.139594] Call Trace: [ 6440.140681] [ 6440.141412] ? show_regs.cold.9+0x22/0x2f [ 6440.143099] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6440.144961] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6440.146583] ? watchdog+0x40/0x40 [ 6440.147953] ? hrtimer_interrupt+0x128/0x2c0 [ 6440.149377] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6440.151130] ? apic_timer_interrupt+0xf/0x20 [ 6440.152729] [ 6440.153536] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6440.155932] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6440.158016] ? newidle_balance.isra.93+0x207/0x580 [ 6440.159422] do_raw_spin_lock+0x177/0x190 [ 6440.161076] _raw_spin_lock+0x21/0x30 [ 6440.162450] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6440.164517] process_one_work+0x2c8/0x700 [ 6440.166274] worker_thread+0x296/0x6e0 [ 6440.168061] ? rescuer_thread+0x570/0x570 [ 6440.169568] kthread+0x1d1/0x200 [ 6440.170825] ? set_kthread_struct+0x70/0x70 [ 6440.172388] ret_from_fork+0x1f/0x30 [ 6446.049082] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3808s! [ 6446.052894] Showing busy workqueues and worker pools: [ 6446.054953] workqueue events: flags=0x0 [ 6446.056314] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6446.063771] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6446.072084] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6446.074386] pending: cpu_latency_work [ptlrpc] [ 6446.079127] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6446.082004] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6446.089292] workqueue events_long: flags=0x0 [ 6446.091013] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6446.098546] pending: lu_objects_flush [obdclass] [ 6446.103505] workqueue events_unbound: flags=0x2 [ 6446.105611] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6446.111070] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6446.114258] workqueue events_power_efficient: flags=0x80 [ 6446.116171] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6446.119344] pending: check_lifetime, do_cache_clean [ 6446.121948] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6446.125368] pending: neigh_periodic_work, neigh_periodic_work [ 6446.128150] workqueue mm_percpu_wq: flags=0x8 [ 6446.130215] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6446.133526] pending: vmstat_update, lru_add_drain_per_cpu [ 6446.135272] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6446.137360] pending: vmstat_update, lru_add_drain_per_cpu [ 6446.143794] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6446.146618] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6446.149587] workqueue xprtiod: flags=0xa [ 6446.151292] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6446.155927] pending: xprt_autoclose, xs_error_handle [ 6446.158295] workqueue ipv6_addrconf: flags=0x40008 [ 6446.161376] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6446.164251] pending: addrconf_verify_work [ 6446.166322] workqueue knbd0-recv: flags=0x1a [ 6446.168295] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6446.173049] in-flight: 651:recv_work [ 6446.175854] workqueue ptlrpc_pinger: flags=0x2 [ 6446.177373] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6446.182470] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6446.185748] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3809s workers=4 idle: 15979 31824 8 [ 6446.189022] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6436s workers=2 idle: 654 [ 6460.055229] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 6460.058011] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 6460.063786] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 6460.063802] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6460.063885] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6460.063891] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6460.063894] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6460.063910] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6460.063914] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6460.063918] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6460.063921] RDX: 0000000000000fa6 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6460.063923] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6460.063926] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6460.063928] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6460.063932] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6460.063935] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6460.063937] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6460.063946] Call Trace: [ 6460.063951] [ 6460.063957] ? show_regs.cold.9+0x22/0x2f [ 6460.063967] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6460.063974] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6460.063978] ? watchdog+0x40/0x40 [ 6460.063983] ? hrtimer_interrupt+0x128/0x2c0 [ 6460.063988] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6460.063996] ? apic_timer_interrupt+0xf/0x20 [ 6460.064000] [ 6460.064003] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6460.064008] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6460.064013] do_raw_spin_lock+0x177/0x190 [ 6460.064018] _raw_spin_lock+0x21/0x30 [ 6460.064024] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6460.064308] obd_mod_free+0x46/0x140 [ptlrpc] [ 6460.064532] mdc_commit_open+0x91/0xb0 [mdc] [ 6460.064571] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6460.064771] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6460.064971] after_reply+0xe2c/0x1780 [ptlrpc] [ 6460.065169] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6460.065359] ? __next_timer_interrupt+0x160/0x160 [ 6460.065370] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6460.065591] ? do_wait_intr+0xf0/0xf0 [ 6460.065600] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6460.065798] kthread+0x1d1/0x200 [ 6460.065804] ? set_kthread_struct+0x70/0x70 [ 6460.065808] ret_from_fork+0x1f/0x30 [ 6460.070107] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) [ 6460.073907] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6460.074045] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6460.074053] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6460.074056] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6460.074070] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6460.074075] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6460.074079] RAX: 0000000000003adf RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6460.074082] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6460.074084] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6460.074086] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6460.074089] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6460.074093] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6460.074096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6460.074098] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6460.074106] Call Trace: [ 6460.074111] [ 6460.074117] ? show_regs.cold.9+0x22/0x2f [ 6460.074131] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6460.074137] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6460.074140] ? watchdog+0x40/0x40 [ 6460.074145] ? hrtimer_interrupt+0x128/0x2c0 [ 6460.074148] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6460.074156] ? apic_timer_interrupt+0xf/0x20 [ 6460.074160] [ 6460.074163] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6460.074167] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6460.074173] do_raw_spin_lock+0x177/0x190 [ 6460.074239] _raw_spin_lock+0x21/0x30 [ 6460.074246] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6460.074661] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6460.074877] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6460.075113] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6460.075399] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6460.075895] ? woken_wake_function+0x30/0x30 [ 6460.075906] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6460.076140] kthread+0x1d1/0x200 [ 6460.076145] ? set_kthread_struct+0x70/0x70 [ 6460.076149] ret_from_fork+0x1f/0x30 [ 6460.334803] ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6460.347233] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6460.351387] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6460.355557] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6460.358514] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6460.366254] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6460.369365] RAX: 000000000000464b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6460.372977] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6460.379369] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6460.386167] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6460.392837] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6460.398299] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6460.405961] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6460.413769] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6460.417213] Call Trace: [ 6460.418257] [ 6460.419285] ? show_regs.cold.9+0x22/0x2f [ 6460.421112] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6460.423207] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6460.425308] ? watchdog+0x40/0x40 [ 6460.426879] ? hrtimer_interrupt+0x128/0x2c0 [ 6460.428986] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6460.430725] ? apic_timer_interrupt+0xf/0x20 [ 6460.431903] [ 6460.432710] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6460.434393] do_raw_spin_lock+0x177/0x190 [ 6460.435716] _raw_spin_lock+0x21/0x30 [ 6460.437067] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6460.439342] ? do_raw_spin_unlock+0x75/0x190 [ 6460.441052] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6460.443113] ? do_wait_intr+0xf0/0xf0 [ 6460.444702] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6460.446990] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6460.449269] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6460.451998] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6460.453959] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6460.456194] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6460.458549] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6460.460659] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6460.463117] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6460.465702] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6460.468040] lmv_intent_open+0x338/0x1150 [lmv] [ 6460.469950] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6460.471904] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6460.474121] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6460.475902] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6460.477874] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6460.479845] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6460.481462] ? do_raw_spin_unlock+0x75/0x190 [ 6460.482928] ? _raw_spin_unlock+0x12/0x30 [ 6460.484234] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6460.486282] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6460.488265] ll_atomic_open+0x36a/0xd70 [lustre] [ 6460.490202] lookup_open+0x356/0xb40 [ 6460.491635] do_last+0x313/0xfc0 [ 6460.492848] ? nd_jump_root+0xe5/0x160 [ 6460.494320] ? path_init+0x437/0x520 [ 6460.496699] path_openat+0xf7/0x500 [ 6460.498089] ? do_raw_spin_unlock+0x75/0x190 [ 6460.500214] do_filp_open+0x99/0x140 [ 6460.501879] ? getname_flags+0x6e/0x330 [ 6460.503312] ? __check_object_size+0xff/0x256 [ 6460.505856] ? do_raw_spin_unlock+0x75/0x190 [ 6460.508962] ? _raw_spin_unlock+0x12/0x30 [ 6460.510928] do_sys_openat2+0x2b4/0x410 [ 6460.512891] do_sys_open+0x73/0xa0 [ 6460.514458] __x64_sys_openat+0x24/0x30 [ 6460.516633] do_syscall_64+0xc1/0x440 [ 6460.518165] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6460.520568] RIP: 0033:0x7f53425a5332 [ 6460.522052] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6460.537843] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6460.542324] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6460.546244] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6460.552874] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6460.560354] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6460.568042] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6472.052069] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 6472.055991] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6472.094357] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6472.101123] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6472.105898] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6472.111750] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6472.115789] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6472.139635] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 6472.144680] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6472.153462] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6472.158687] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6472.164812] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6472.171133] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6472.177651] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6472.183004] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6472.189229] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6472.198432] Call Trace: [ 6472.199929] [ 6472.202168] ? show_regs.cold.9+0x22/0x2f [ 6472.205818] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6472.208193] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6472.212812] ? watchdog+0x40/0x40 [ 6472.219128] ? hrtimer_interrupt+0x128/0x2c0 [ 6472.226185] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6472.234689] ? apic_timer_interrupt+0xf/0x20 [ 6472.239418] [ 6472.241156] ? native_safe_halt+0x2e/0x50 [ 6472.243303] kvm_wait+0xcd/0xe0 [ 6472.244918] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6472.247453] ? newidle_balance.isra.93+0x207/0x580 [ 6472.249824] do_raw_spin_lock+0x177/0x190 [ 6472.253070] _raw_spin_lock+0x21/0x30 [ 6472.257179] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6472.263104] process_one_work+0x2c8/0x700 [ 6472.267081] worker_thread+0x296/0x6e0 [ 6472.268309] ? rescuer_thread+0x570/0x570 [ 6472.273856] kthread+0x1d1/0x200 [ 6472.276988] ? set_kthread_struct+0x70/0x70 [ 6472.281143] ret_from_fork+0x1f/0x30 [ 6474.534283] rcu: INFO: rcu_sched self-detected stall on CPU [ 6474.543490] rcu: 2-....: (2355116 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 6474.555734] (t=4380108 jiffies g=274185 q=64799) [ 6474.561873] rcu: rcu_sched kthread starved for 3838474 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 6474.575060] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6474.583647] rcu: RCU grace-period kthread stack dump: [ 6474.591452] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6474.604230] Call Trace: [ 6474.604836] __schedule+0x351/0xcb0 [ 6474.605811] schedule+0xc0/0x180 [ 6474.614552] schedule_timeout+0xb4/0x190 [ 6474.618800] ? do_raw_spin_unlock+0x75/0x190 [ 6474.621116] ? __next_timer_interrupt+0x160/0x160 [ 6474.626601] ? prepare_to_swait_event+0xf3/0x140 [ 6474.631023] rcu_gp_kthread+0x8dd/0x15b0 [ 6474.633653] ? force_qs_rnp+0x2c0/0x2c0 [ 6474.635539] kthread+0x1d1/0x200 [ 6474.636458] ? set_kthread_struct+0x70/0x70 [ 6474.637648] ret_from_fork+0x1f/0x30 [ 6474.645394] rcu: Stack dump where RCU GP kthread last ran: [ 6474.647176] Sending NMI from CPU 2 to CPUs 0: [ 6474.650151] NMI backtrace for cpu 0 [ 6474.652537] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6474.652539] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6474.652541] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6474.652544] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6474.652546] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6474.652548] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 [ 6474.652549] RAX: 000000000000707b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6474.652551] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6474.652552] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6474.652554] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6474.652555] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6474.652557] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6474.652559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6474.652560] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6474.652561] Call Trace: [ 6474.652562] [ 6474.652563] ? show_regs.cold.9+0x22/0x2f [ 6474.652564] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6474.652565] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6474.652566] ? nmi_handle+0xbd/0x210 [ 6474.652566] ? default_do_nmi+0x4b/0x1b0 [ 6474.652568] ? do_nmi+0x280/0x380 [ 6474.652569] ? end_repeat_nmi+0x16/0x1f [ 6474.652570] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6474.652571] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6474.652573] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6474.652574] [ 6474.652575] ? newidle_balance.isra.93+0x207/0x580 [ 6474.652576] do_raw_spin_lock+0x177/0x190 [ 6474.652577] _raw_spin_lock+0x21/0x30 [ 6474.652578] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6474.652579] process_one_work+0x2c8/0x700 [ 6474.652580] worker_thread+0x296/0x6e0 [ 6474.652581] ? rescuer_thread+0x570/0x570 [ 6474.652582] kthread+0x1d1/0x200 [ 6474.652584] ? set_kthread_struct+0x70/0x70 [ 6474.652585] ret_from_fork+0x1f/0x30 [ 6474.656338] Sending NMI from CPU 2 to CPUs 1: [ 6474.792672] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 6474.794226] NMI backtrace for cpu 2 [ 6474.803494] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6474.813805] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6474.829423] Call Trace: [ 6474.830161] [ 6474.830835] ? dump_stack+0xbb/0x10e [ 6474.831954] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6474.838973] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6474.845390] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6474.847748] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6474.850277] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6474.851930] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6474.859744] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6474.865371] ? update_process_times+0x6d/0xc0 [ 6474.867493] ? tick_sched_handle.isra.18+0x27/0x90 [ 6474.870873] ? tick_sched_timer+0x53/0xb0 [ 6474.872982] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6474.877471] ? tick_sched_handle.isra.18+0x90/0x90 [ 6474.883791] ? hrtimer_interrupt+0x128/0x2c0 [ 6474.885947] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6474.895823] ? apic_timer_interrupt+0xf/0x20 [ 6474.903609] [ 6474.908877] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6474.914978] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6474.921294] ? do_raw_spin_lock+0x177/0x190 [ 6474.924318] ? _raw_spin_lock+0x21/0x30 [ 6474.926865] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6474.928984] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6474.932853] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6474.935255] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6474.938786] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6474.941131] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6474.944337] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6474.947488] ? __next_timer_interrupt+0x160/0x160 [ 6474.949727] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6474.952245] ? do_wait_intr+0xf0/0xf0 [ 6474.956341] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6474.958772] ? kthread+0x1d1/0x200 [ 6474.960702] ? set_kthread_struct+0x70/0x70 [ 6474.962463] ? ret_from_fork+0x1f/0x30 [ 6476.256273] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3839s! [ 6476.261851] Showing busy workqueues and worker pools: [ 6476.264066] workqueue events: flags=0x0 [ 6476.265933] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6476.269099] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6476.276895] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6476.282816] pending: cpu_latency_work [ptlrpc] [ 6476.284992] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6476.286787] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6476.295945] workqueue events_long: flags=0x0 [ 6476.300742] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6476.309226] pending: lu_objects_flush [obdclass] [ 6476.311267] workqueue events_unbound: flags=0x2 [ 6476.312479] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6476.319205] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6476.323513] workqueue events_power_efficient: flags=0x80 [ 6476.325406] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6476.336240] pending: check_lifetime, do_cache_clean [ 6476.338094] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6476.343621] pending: neigh_periodic_work, neigh_periodic_work [ 6476.345861] workqueue mm_percpu_wq: flags=0x8 [ 6476.347027] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6476.348608] pending: vmstat_update, lru_add_drain_per_cpu [ 6476.351008] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6476.353096] pending: vmstat_update, lru_add_drain_per_cpu [ 6476.355039] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6476.357092] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6476.359455] workqueue xprtiod: flags=0xa [ 6476.360562] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6476.362898] pending: xprt_autoclose, xs_error_handle [ 6476.369587] workqueue ipv6_addrconf: flags=0x40008 [ 6476.375776] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6476.377946] pending: addrconf_verify_work [ 6476.379530] workqueue knbd0-recv: flags=0x1a [ 6476.387801] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6476.393980] in-flight: 651:recv_work [ 6476.399573] workqueue ptlrpc_pinger: flags=0x2 [ 6476.401832] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6476.407211] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6476.409210] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3839s workers=4 idle: 15979 31824 8 [ 6476.414482] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6466s workers=2 idle: 654 [ 6488.058029] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 6488.062216] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6488.085202] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6488.090202] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6488.093691] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6488.095907] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6488.103556] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6488.105465] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6488.107245] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6488.108862] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6488.111720] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6488.114514] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6488.117395] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6488.120578] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6488.122867] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6488.125791] Call Trace: [ 6488.126748] [ 6488.127499] ? show_regs.cold.9+0x22/0x2f [ 6488.128908] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6488.130596] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6488.132491] ? watchdog+0x40/0x40 [ 6488.133736] ? hrtimer_interrupt+0x128/0x2c0 [ 6488.135572] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6488.138857] ? apic_timer_interrupt+0xf/0x20 [ 6488.140863] [ 6488.141665] ? native_safe_halt+0x2e/0x50 [ 6488.143058] kvm_wait+0xcd/0xe0 [ 6488.143730] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6488.145221] do_raw_spin_lock+0x177/0x190 [ 6488.146339] _raw_spin_lock+0x21/0x30 [ 6488.147617] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6488.149565] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6488.151338] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6488.153404] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6488.155485] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6488.157588] ? woken_wake_function+0x30/0x30 [ 6488.158883] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6488.161148] kthread+0x1d1/0x200 [ 6488.162207] ? set_kthread_struct+0x70/0x70 [ 6488.163499] ret_from_fork+0x1f/0x30 [ 6500.056124] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 6500.059218] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 6500.060307] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6500.060381] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6500.062499] Modules linked in: [ 6500.095283] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6500.095294] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6500.095310] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6500.095315] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6500.095321] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6500.095324] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6500.095326] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6500.095328] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6500.095330] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6500.095333] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6500.095336] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6500.095338] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6500.095347] Call Trace: [ 6500.095352] [ 6500.095358] ? show_regs.cold.9+0x22/0x2f [ 6500.095368] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6500.095910] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6500.095915] ? watchdog+0x40/0x40 [ 6500.095920] ? hrtimer_interrupt+0x128/0x2c0 [ 6500.095924] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6500.095931] ? apic_timer_interrupt+0xf/0x20 [ 6500.095936] [ 6500.095939] ? native_safe_halt+0x2e/0x50 [ 6500.095944] kvm_wait+0xcd/0xe0 [ 6500.095951] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6500.095961] do_raw_spin_lock+0x177/0x190 [ 6500.095966] _raw_spin_lock+0x21/0x30 [ 6500.095970] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6500.101322] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6500.101392] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6500.101398] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6500.101401] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6500.101414] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6500.101419] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6500.101424] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6500.101427] RDX: 0000000000007f94 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6500.101429] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6500.101431] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6500.101433] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6500.101436] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6500.101438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6500.101440] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6500.101524] Call Trace: [ 6500.101530] [ 6500.101536] ? show_regs.cold.9+0x22/0x2f [ 6500.101545] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6500.101551] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6500.101556] ? watchdog+0x40/0x40 [ 6500.101561] ? hrtimer_interrupt+0x128/0x2c0 [ 6500.101565] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6500.101573] ? apic_timer_interrupt+0xf/0x20 [ 6500.101578] [ 6500.101581] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6500.101585] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6500.101592] do_raw_spin_lock+0x177/0x190 [ 6500.101596] _raw_spin_lock+0x21/0x30 [ 6500.101601] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6500.101933] obd_mod_free+0x46/0x140 [ptlrpc] [ 6500.103131] mdc_commit_open+0x91/0xb0 [mdc] [ 6500.103165] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6500.103535] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6500.103792] after_reply+0xe2c/0x1780 [ptlrpc] [ 6500.103994] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6500.104331] ? __next_timer_interrupt+0x160/0x160 [ 6500.104342] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6500.104777] ? do_wait_intr+0xf0/0xf0 [ 6500.104786] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6500.104993] kthread+0x1d1/0x200 [ 6500.104999] ? set_kthread_struct+0x70/0x70 [ 6500.105003] ret_from_fork+0x1f/0x30 [ 6500.426586] ? do_raw_spin_unlock+0x75/0x190 [ 6500.430396] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6500.435028] ? do_wait_intr+0xf0/0xf0 [ 6500.436370] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6500.439348] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6500.444188] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6500.448268] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6500.451220] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6500.454985] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6500.459961] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6500.463004] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6500.466563] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6500.471428] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6500.476260] lmv_intent_open+0x338/0x1150 [lmv] [ 6500.482603] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6500.486074] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6500.492034] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6500.499607] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6500.501433] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6500.510945] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6500.517314] ? do_raw_spin_unlock+0x75/0x190 [ 6500.524314] ? _raw_spin_unlock+0x12/0x30 [ 6500.527555] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6500.529693] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6500.534180] ll_atomic_open+0x36a/0xd70 [lustre] [ 6500.540353] lookup_open+0x356/0xb40 [ 6500.545894] do_last+0x313/0xfc0 [ 6500.546955] ? nd_jump_root+0xe5/0x160 [ 6500.551217] ? path_init+0x437/0x520 [ 6500.552484] path_openat+0xf7/0x500 [ 6500.558687] ? do_raw_spin_unlock+0x75/0x190 [ 6500.562653] do_filp_open+0x99/0x140 [ 6500.569295] ? getname_flags+0x6e/0x330 [ 6500.572064] ? __check_object_size+0xff/0x256 [ 6500.573670] ? do_raw_spin_unlock+0x75/0x190 [ 6500.581227] ? _raw_spin_unlock+0x12/0x30 [ 6500.584327] do_sys_openat2+0x2b4/0x410 [ 6500.589767] do_sys_open+0x73/0xa0 [ 6500.590903] __x64_sys_openat+0x24/0x30 [ 6500.593418] do_syscall_64+0xc1/0x440 [ 6500.598752] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6500.601423] RIP: 0033:0x7f53425a5332 [ 6500.604894] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6500.619161] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6500.624262] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6500.627402] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6500.631013] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6500.634587] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6500.640628] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6504.053287] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 6504.060266] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6504.088991] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6504.100978] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6504.109548] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6504.115204] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6504.118599] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6504.136689] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6504.139247] RAX: 0000000000001289 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6504.151398] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6504.155911] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6504.160921] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6504.168737] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6504.180053] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6504.182355] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6504.189253] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6504.193568] Call Trace: [ 6504.196126] [ 6504.197237] ? show_regs.cold.9+0x22/0x2f [ 6504.202883] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6504.207246] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6504.209431] ? watchdog+0x40/0x40 [ 6504.214547] ? hrtimer_interrupt+0x128/0x2c0 [ 6504.220202] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6504.224278] ? apic_timer_interrupt+0xf/0x20 [ 6504.233487] [ 6504.234204] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6504.241502] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6504.250722] ? newidle_balance.isra.93+0x207/0x580 [ 6504.254670] do_raw_spin_lock+0x177/0x190 [ 6504.256561] _raw_spin_lock+0x21/0x30 [ 6504.258877] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6504.264520] process_one_work+0x2c8/0x700 [ 6504.277914] worker_thread+0x296/0x6e0 [ 6504.280565] ? rescuer_thread+0x570/0x570 [ 6504.284509] kthread+0x1d1/0x200 [ 6504.293435] ? set_kthread_struct+0x70/0x70 [ 6504.298515] ret_from_fork+0x1f/0x30 [ 6506.467826] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3869s! [ 6506.473735] Showing busy workqueues and worker pools: [ 6506.477610] workqueue events: flags=0x0 [ 6506.481650] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6506.489762] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6506.497659] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6506.503821] pending: cpu_latency_work [ptlrpc] [ 6506.510265] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6506.514287] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6506.518283] workqueue events_long: flags=0x0 [ 6506.520112] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6506.523015] pending: lu_objects_flush [obdclass] [ 6506.525661] workqueue events_unbound: flags=0x2 [ 6506.527230] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6506.529831] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6506.532711] workqueue events_power_efficient: flags=0x80 [ 6506.534716] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6506.538319] pending: check_lifetime, do_cache_clean [ 6506.540422] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6506.544317] pending: neigh_periodic_work, neigh_periodic_work [ 6506.546779] workqueue mm_percpu_wq: flags=0x8 [ 6506.548690] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6506.551913] pending: vmstat_update, lru_add_drain_per_cpu [ 6506.554303] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6506.556586] pending: vmstat_update, lru_add_drain_per_cpu [ 6506.558457] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6506.561253] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6506.563567] workqueue xprtiod: flags=0xa [ 6506.565150] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6506.567456] pending: xprt_autoclose, xs_error_handle [ 6506.569437] workqueue ipv6_addrconf: flags=0x40008 [ 6506.571730] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6506.574184] pending: addrconf_verify_work [ 6506.576019] workqueue knbd0-recv: flags=0x1a [ 6506.577473] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6506.579558] in-flight: 651:recv_work [ 6506.581674] workqueue ptlrpc_pinger: flags=0x2 [ 6506.583489] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6506.586029] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6506.588724] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3869s workers=4 idle: 15979 31824 8 [ 6506.591877] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6497s workers=2 idle: 654 [ 6516.063842] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 6516.073294] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6516.107763] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6516.118241] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6516.125204] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6516.130738] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6516.142916] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6516.149781] RAX: 0000000000005a8c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6516.157261] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6516.161687] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6516.167815] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6516.173348] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6516.175712] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6516.181021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6516.195143] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6516.203673] Call Trace: [ 6516.205899] [ 6516.209314] ? show_regs.cold.9+0x22/0x2f [ 6516.211817] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6516.214659] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6516.216914] ? watchdog+0x40/0x40 [ 6516.219184] ? hrtimer_interrupt+0x128/0x2c0 [ 6516.222314] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6516.225073] ? apic_timer_interrupt+0xf/0x20 [ 6516.229705] [ 6516.232038] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6516.235974] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6516.239419] do_raw_spin_lock+0x177/0x190 [ 6516.241555] _raw_spin_lock+0x21/0x30 [ 6516.245756] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6516.251843] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6516.258009] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6516.270218] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6516.277954] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6516.282192] ? woken_wake_function+0x30/0x30 [ 6516.285555] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6516.290308] kthread+0x1d1/0x200 [ 6516.292231] ? set_kthread_struct+0x70/0x70 [ 6516.296922] ret_from_fork+0x1f/0x30 [ 6528.055011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 6528.058900] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 6528.059514] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6528.059589] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6528.059596] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6528.059599] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6528.059614] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6528.059619] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6528.059624] RAX: 00000000000024d0 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6528.059627] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6528.059629] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6528.059632] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6528.059635] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6528.059638] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6528.059641] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6528.059644] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6528.059654] Call Trace: [ 6528.059660] [ 6528.059667] ? show_regs.cold.9+0x22/0x2f [ 6528.059676] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6528.059684] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6528.059688] ? watchdog+0x40/0x40 [ 6528.059693] ? hrtimer_interrupt+0x128/0x2c0 [ 6528.059697] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6528.059862] ? apic_timer_interrupt+0xf/0x20 [ 6528.059867] [ 6528.059870] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6528.059878] do_raw_spin_lock+0x177/0x190 [ 6528.059884] _raw_spin_lock+0x21/0x30 [ 6528.059889] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6528.064104] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6528.064187] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6528.064195] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6528.064198] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6528.064235] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6528.064239] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6528.064244] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6528.064246] RDX: 0000000000007e80 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6528.064249] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6528.064251] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6528.064254] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6528.064257] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6528.064260] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6528.064263] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6528.064395] Call Trace: [ 6528.064401] [ 6528.064405] ? show_regs.cold.9+0x22/0x2f [ 6528.064415] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6528.064421] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6528.064426] ? watchdog+0x40/0x40 [ 6528.064430] ? hrtimer_interrupt+0x128/0x2c0 [ 6528.064435] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6528.064443] ? apic_timer_interrupt+0xf/0x20 [ 6528.064448] [ 6528.064452] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6528.064470] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6528.064476] do_raw_spin_lock+0x177/0x190 [ 6528.064481] _raw_spin_lock+0x21/0x30 [ 6528.064487] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6528.064781] obd_mod_free+0x46/0x140 [ptlrpc] [ 6528.064983] mdc_commit_open+0x91/0xb0 [mdc] [ 6528.065028] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6528.065246] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6528.065475] after_reply+0xe2c/0x1780 [ptlrpc] [ 6528.065676] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6528.066266] ? __next_timer_interrupt+0x160/0x160 [ 6528.066287] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6528.066563] ? do_wait_intr+0xf0/0xf0 [ 6528.066571] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6528.066911] kthread+0x1d1/0x200 [ 6528.066918] ? set_kthread_struct+0x70/0x70 [ 6528.066921] ret_from_fork+0x1f/0x30 [ 6528.318549] ? do_raw_spin_unlock+0x75/0x190 [ 6528.320254] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6528.322622] ? do_wait_intr+0xf0/0xf0 [ 6528.324205] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6528.327183] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6528.329535] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6528.332618] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6528.335356] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6528.338013] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6528.340021] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6528.342505] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6528.345496] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6528.348601] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6528.351873] lmv_intent_open+0x338/0x1150 [lmv] [ 6528.354452] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6528.357285] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6528.360280] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6528.362556] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6528.364427] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6528.366815] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6528.369333] ? do_raw_spin_unlock+0x75/0x190 [ 6528.371662] ? _raw_spin_unlock+0x12/0x30 [ 6528.374519] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6528.377629] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6528.380139] ll_atomic_open+0x36a/0xd70 [lustre] [ 6528.383022] lookup_open+0x356/0xb40 [ 6528.384795] do_last+0x313/0xfc0 [ 6528.386372] ? nd_jump_root+0xe5/0x160 [ 6528.388198] ? path_init+0x437/0x520 [ 6528.390098] path_openat+0xf7/0x500 [ 6528.391038] ? do_raw_spin_unlock+0x75/0x190 [ 6528.392849] do_filp_open+0x99/0x140 [ 6528.394606] ? getname_flags+0x6e/0x330 [ 6528.396895] ? __check_object_size+0xff/0x256 [ 6528.398914] ? do_raw_spin_unlock+0x75/0x190 [ 6528.401474] ? _raw_spin_unlock+0x12/0x30 [ 6528.403481] do_sys_openat2+0x2b4/0x410 [ 6528.405519] do_sys_open+0x73/0xa0 [ 6528.407464] __x64_sys_openat+0x24/0x30 [ 6528.409778] do_syscall_64+0xc1/0x440 [ 6528.412604] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6528.416400] RIP: 0033:0x7f53425a5332 [ 6528.418386] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6528.429029] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6528.433197] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6528.436818] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6528.440041] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6528.443410] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6528.446814] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6532.052128] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 6532.054835] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6532.093145] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6532.103368] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6532.117520] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6532.129820] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6532.137857] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6532.168791] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6532.179759] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6532.184910] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6532.188553] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6532.193345] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6532.197732] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6532.201895] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6532.207393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6532.209385] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6532.217829] Call Trace: [ 6532.220315] [ 6532.223252] ? show_regs.cold.9+0x22/0x2f [ 6532.227892] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6532.231399] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6532.233789] ? watchdog+0x40/0x40 [ 6532.235552] ? hrtimer_interrupt+0x128/0x2c0 [ 6532.238545] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6532.241524] ? apic_timer_interrupt+0xf/0x20 [ 6532.244565] [ 6532.245974] ? native_safe_halt+0x2e/0x50 [ 6532.248698] kvm_wait+0xcd/0xe0 [ 6532.251094] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6532.252947] ? newidle_balance.isra.93+0x207/0x580 [ 6532.255908] do_raw_spin_lock+0x177/0x190 [ 6532.260323] _raw_spin_lock+0x21/0x30 [ 6532.261728] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6532.270020] process_one_work+0x2c8/0x700 [ 6532.271578] worker_thread+0x296/0x6e0 [ 6532.272922] ? rescuer_thread+0x570/0x570 [ 6532.284920] kthread+0x1d1/0x200 [ 6532.285987] ? set_kthread_struct+0x70/0x70 [ 6532.295783] ret_from_fork+0x1f/0x30 [ 6536.674703] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3899s! [ 6536.679957] Showing busy workqueues and worker pools: [ 6536.683126] workqueue events: flags=0x0 [ 6536.685014] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6536.689220] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6536.702746] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6536.705462] pending: cpu_latency_work [ptlrpc] [ 6536.708681] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6536.714506] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6536.724367] workqueue events_long: flags=0x0 [ 6536.725888] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6536.732002] pending: lu_objects_flush [obdclass] [ 6536.735293] workqueue events_unbound: flags=0x2 [ 6536.736788] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6536.743747] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6536.747684] workqueue events_power_efficient: flags=0x80 [ 6536.750851] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6536.759986] pending: check_lifetime, do_cache_clean [ 6536.769422] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6536.774788] pending: neigh_periodic_work, neigh_periodic_work [ 6536.779590] workqueue mm_percpu_wq: flags=0x8 [ 6536.785645] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6536.797041] pending: vmstat_update, lru_add_drain_per_cpu [ 6536.804992] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6536.810668] pending: vmstat_update, lru_add_drain_per_cpu [ 6536.818372] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6536.826368] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6536.830307] workqueue xprtiod: flags=0xa [ 6536.832201] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6536.833686] pending: xprt_autoclose, xs_error_handle [ 6536.843138] workqueue ipv6_addrconf: flags=0x40008 [ 6536.853354] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6536.856734] pending: addrconf_verify_work [ 6536.863726] workqueue knbd0-recv: flags=0x1a [ 6536.864925] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6536.867759] in-flight: 651:recv_work [ 6536.870437] workqueue ptlrpc_pinger: flags=0x2 [ 6536.873694] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6536.875449] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6536.878495] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3899s workers=4 idle: 15979 31824 8 [ 6536.887851] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6527s workers=2 idle: 654 [ 6544.058442] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 6544.061087] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6544.088708] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6544.096486] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6544.103772] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6544.108462] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6544.118459] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6544.123781] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6544.129166] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6544.134597] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6544.138922] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6544.146606] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6544.149438] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6544.152443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6544.155677] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6544.158750] Call Trace: [ 6544.159662] [ 6544.161290] ? show_regs.cold.9+0x22/0x2f [ 6544.163152] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6544.164816] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6544.166844] ? watchdog+0x40/0x40 [ 6544.168905] ? hrtimer_interrupt+0x128/0x2c0 [ 6544.172744] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6544.176702] ? apic_timer_interrupt+0xf/0x20 [ 6544.178159] [ 6544.178857] ? native_safe_halt+0x2e/0x50 [ 6544.185497] kvm_wait+0xcd/0xe0 [ 6544.188087] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6544.189898] do_raw_spin_lock+0x177/0x190 [ 6544.194409] _raw_spin_lock+0x21/0x30 [ 6544.196090] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6544.198609] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6544.200627] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6544.202889] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6544.205599] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6544.208052] ? woken_wake_function+0x30/0x30 [ 6544.209577] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6544.212179] kthread+0x1d1/0x200 [ 6544.213387] ? set_kthread_struct+0x70/0x70 [ 6544.215287] ret_from_fork+0x1f/0x30 [ 6556.056839] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 6556.057013] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 6556.060846] Modules linked in: [ 6556.063056] Modules linked in: [ 6556.063118] lustre(O) [ 6556.064414] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6556.064483] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6556.064491] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6556.064493] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6556.064507] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6556.064511] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6556.064515] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6556.064518] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6556.064520] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6556.064522] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6556.064524] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6556.064527] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6556.064529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6556.064531] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6556.064539] Call Trace: [ 6556.064545] [ 6556.064549] ? show_regs.cold.9+0x22/0x2f [ 6556.064558] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6556.064564] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6556.064568] ? watchdog+0x40/0x40 [ 6556.064572] ? hrtimer_interrupt+0x128/0x2c0 [ 6556.064576] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6556.064581] ? apic_timer_interrupt+0xf/0x20 [ 6556.064586] [ 6556.064588] ? native_safe_halt+0x2e/0x50 [ 6556.064592] kvm_wait+0xcd/0xe0 [ 6556.064598] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6556.064605] do_raw_spin_lock+0x177/0x190 [ 6556.064610] _raw_spin_lock+0x21/0x30 [ 6556.064615] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6556.066527] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6556.066597] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6556.066603] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6556.066606] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6556.066616] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6556.066620] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6556.066624] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6556.066626] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6556.066628] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6556.066630] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6556.066632] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6556.066635] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6556.066638] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6556.066640] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6556.066649] Call Trace: [ 6556.066654] [ 6556.066657] ? show_regs.cold.9+0x22/0x2f [ 6556.066666] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6556.066672] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6556.066676] ? watchdog+0x40/0x40 [ 6556.066679] ? hrtimer_interrupt+0x128/0x2c0 [ 6556.066683] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6556.066689] ? apic_timer_interrupt+0xf/0x20 [ 6556.066693] [ 6556.066695] ? native_safe_halt+0x2e/0x50 [ 6556.066699] kvm_wait+0xcd/0xe0 [ 6556.066706] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6556.066713] do_raw_spin_lock+0x177/0x190 [ 6556.066717] _raw_spin_lock+0x21/0x30 [ 6556.066722] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6556.069422] obd_mod_free+0x46/0x140 [ptlrpc] [ 6556.069666] mdc_commit_open+0x91/0xb0 [mdc] [ 6556.069704] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6556.090761] ? do_raw_spin_unlock+0x75/0x190 [ 6556.090782] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6556.091181] ? do_wait_intr+0xf0/0xf0 [ 6556.091190] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6556.091382] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6556.091599] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6556.091823] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6556.092119] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6556.092158] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6556.092188] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6556.092216] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6556.092304] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6556.092528] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6556.092557] lmv_intent_open+0x338/0x1150 [lmv] [ 6556.092587] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6556.092698] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6556.092967] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6556.092984] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6556.096720] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6556.104249] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6556.105554] after_reply+0xe2c/0x1780 [ptlrpc] [ 6556.117538] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6556.117684] ? do_raw_spin_unlock+0x75/0x190 [ 6556.117694] ? _raw_spin_unlock+0x12/0x30 [ 6556.117702] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6556.117813] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6556.117840] ll_atomic_open+0x36a/0xd70 [lustre] [ 6556.118052] lookup_open+0x356/0xb40 [ 6556.118064] do_last+0x313/0xfc0 [ 6556.118070] ? nd_jump_root+0xe5/0x160 [ 6556.118075] ? path_init+0x437/0x520 [ 6556.118080] path_openat+0xf7/0x500 [ 6556.118086] ? do_raw_spin_unlock+0x75/0x190 [ 6556.118091] do_filp_open+0x99/0x140 [ 6556.118098] ? getname_flags+0x6e/0x330 [ 6556.118102] ? __check_object_size+0xff/0x256 [ 6556.118216] ? do_raw_spin_unlock+0x75/0x190 [ 6556.118221] ? _raw_spin_unlock+0x12/0x30 [ 6556.118226] do_sys_openat2+0x2b4/0x410 [ 6556.118236] do_sys_open+0x73/0xa0 [ 6556.118241] __x64_sys_openat+0x24/0x30 [ 6556.118249] do_syscall_64+0xc1/0x440 [ 6556.118255] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6556.118261] RIP: 0033:0x7f53425a5332 [ 6556.118273] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6556.118280] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6556.118285] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6556.118288] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6556.118290] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6556.118292] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6556.118295] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6556.309007] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6556.310872] ? __next_timer_interrupt+0x160/0x160 [ 6556.312136] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6556.313584] ? do_wait_intr+0xf0/0xf0 [ 6556.314682] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6556.316340] kthread+0x1d1/0x200 [ 6556.317423] ? set_kthread_struct+0x70/0x70 [ 6556.318785] ret_from_fork+0x1f/0x30 [ 6564.055539] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 6564.063053] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6564.089798] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6564.096073] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6564.099864] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6564.112139] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6564.115208] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6564.122108] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6564.126318] RAX: 0000000000001979 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6564.129464] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6564.132226] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6564.137178] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6564.140100] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6564.147532] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6564.149712] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6564.153248] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6564.155972] Call Trace: [ 6564.157781] [ 6564.158242] ? show_regs.cold.9+0x22/0x2f [ 6564.159652] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6564.161267] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6564.162942] ? watchdog+0x40/0x40 [ 6564.164348] ? hrtimer_interrupt+0x128/0x2c0 [ 6564.165965] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6564.167532] ? apic_timer_interrupt+0xf/0x20 [ 6564.169388] [ 6564.170059] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6564.171876] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6564.174798] ? newidle_balance.isra.93+0x207/0x580 [ 6564.176258] do_raw_spin_lock+0x177/0x190 [ 6564.177857] _raw_spin_lock+0x21/0x30 [ 6564.184507] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6564.187367] process_one_work+0x2c8/0x700 [ 6564.189279] worker_thread+0x296/0x6e0 [ 6564.194131] ? rescuer_thread+0x570/0x570 [ 6564.195986] kthread+0x1d1/0x200 [ 6564.197979] ? set_kthread_struct+0x70/0x70 [ 6564.208031] ret_from_fork+0x1f/0x30 [ 6567.399558] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3930s! [ 6567.407029] Showing busy workqueues and worker pools: [ 6567.411584] workqueue events: flags=0x0 [ 6567.415360] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6567.424646] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6567.434170] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6567.440826] pending: cpu_latency_work [ptlrpc] [ 6567.444342] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6567.449314] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6567.458535] workqueue events_long: flags=0x0 [ 6567.460120] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6567.464405] pending: lu_objects_flush [obdclass] [ 6567.468009] workqueue events_unbound: flags=0x2 [ 6567.470350] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6567.473515] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6567.476315] workqueue events_power_efficient: flags=0x80 [ 6567.478914] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6567.482357] pending: check_lifetime, do_cache_clean [ 6567.486056] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6567.491470] pending: neigh_periodic_work, neigh_periodic_work [ 6567.494684] workqueue mm_percpu_wq: flags=0x8 [ 6567.497095] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6567.500543] pending: vmstat_update, lru_add_drain_per_cpu [ 6567.503666] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6567.507884] pending: vmstat_update, lru_add_drain_per_cpu [ 6567.510821] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6567.513895] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6567.518052] workqueue xprtiod: flags=0xa [ 6567.520946] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6567.526116] pending: xprt_autoclose, xs_error_handle [ 6567.530339] workqueue ipv6_addrconf: flags=0x40008 [ 6567.532675] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6567.535225] pending: addrconf_verify_work [ 6567.536633] workqueue knbd0-recv: flags=0x1a [ 6567.538773] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6567.540866] in-flight: 651:recv_work [ 6567.542280] workqueue ptlrpc_pinger: flags=0x2 [ 6567.543779] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6567.549674] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6567.552819] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3930s workers=4 idle: 15979 31824 8 [ 6567.555088] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6558s workers=2 idle: 654 [ 6572.058705] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 6572.066381] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6572.101533] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6572.106857] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6572.110425] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6572.112976] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6572.122261] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6572.129325] RAX: 0000000000007f09 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6572.135351] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6572.139284] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6572.143612] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6572.147057] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6572.150808] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6572.155233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6572.158184] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6572.162017] Call Trace: [ 6572.163189] [ 6572.164789] ? show_regs.cold.9+0x22/0x2f [ 6572.167900] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6572.170139] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6572.172266] ? watchdog+0x40/0x40 [ 6572.173873] ? hrtimer_interrupt+0x128/0x2c0 [ 6572.175838] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6572.177870] ? apic_timer_interrupt+0xf/0x20 [ 6572.180124] [ 6572.181045] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6572.183201] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6572.185685] do_raw_spin_lock+0x177/0x190 [ 6572.187794] _raw_spin_lock+0x21/0x30 [ 6572.189286] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6572.192315] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6572.195077] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6572.197843] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6572.200313] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6572.203207] ? woken_wake_function+0x30/0x30 [ 6572.205133] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6572.208005] kthread+0x1d1/0x200 [ 6572.209311] ? set_kthread_struct+0x70/0x70 [ 6572.211216] ret_from_fork+0x1f/0x30 [ 6584.055023] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 6584.057085] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 6584.057571] Modules linked in: [ 6584.066205] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6584.066321] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6584.066328] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6584.066331] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6584.066344] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6584.066349] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6584.066354] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6584.066357] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6584.066359] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6584.066361] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6584.066364] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6584.066367] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6584.066371] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6584.066373] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6584.066381] Call Trace: [ 6584.066386] [ 6584.066392] ? show_regs.cold.9+0x22/0x2f [ 6584.066402] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6584.066408] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6584.067847] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6584.067914] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6584.067920] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6584.067922] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6584.106628] ? watchdog+0x40/0x40 [ 6584.106648] ? hrtimer_interrupt+0x128/0x2c0 [ 6584.106654] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6584.106662] ? apic_timer_interrupt+0xf/0x20 [ 6584.106667] [ 6584.106672] ? native_safe_halt+0x2e/0x50 [ 6584.106676] kvm_wait+0xcd/0xe0 [ 6584.106684] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6584.106692] do_raw_spin_lock+0x177/0x190 [ 6584.106697] _raw_spin_lock+0x21/0x30 [ 6584.106704] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6584.107306] obd_mod_free+0x46/0x140 [ptlrpc] [ 6584.107583] mdc_commit_open+0x91/0xb0 [mdc] [ 6584.107620] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6584.116541] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6584.116554] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6584.116561] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6584.116564] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6584.116566] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6584.116568] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6584.117033] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6584.127459] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6584.127723] after_reply+0xe2c/0x1780 [ptlrpc] [ 6584.127920] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6584.128120] ? __next_timer_interrupt+0x160/0x160 [ 6584.128129] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6584.128371] ? do_wait_intr+0xf0/0xf0 [ 6584.128380] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6584.128591] kthread+0x1d1/0x200 [ 6584.128597] ? set_kthread_struct+0x70/0x70 [ 6584.128602] ret_from_fork+0x1f/0x30 [ 6584.404388] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6584.412180] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6584.415509] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6584.423629] Call Trace: [ 6584.426745] [ 6584.427703] ? show_regs.cold.9+0x22/0x2f [ 6584.430542] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6584.435566] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6584.440193] ? watchdog+0x40/0x40 [ 6584.443193] ? hrtimer_interrupt+0x128/0x2c0 [ 6584.445191] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6584.447669] ? apic_timer_interrupt+0xf/0x20 [ 6584.451451] [ 6584.453413] ? native_safe_halt+0x2e/0x50 [ 6584.455838] kvm_wait+0xcd/0xe0 [ 6584.458303] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6584.460878] do_raw_spin_lock+0x177/0x190 [ 6584.462665] _raw_spin_lock+0x21/0x30 [ 6584.464629] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6584.468115] ? do_raw_spin_unlock+0x75/0x190 [ 6584.469562] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6584.472009] ? do_wait_intr+0xf0/0xf0 [ 6584.474918] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6584.484612] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6584.491656] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6584.498208] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6584.502595] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6584.507563] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6584.512673] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6584.514291] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6584.526551] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6584.536591] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6584.547858] lmv_intent_open+0x338/0x1150 [lmv] [ 6584.560696] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6584.571924] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6584.581806] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6584.592801] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6584.601698] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6584.618871] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6584.625897] ? do_raw_spin_unlock+0x75/0x190 [ 6584.630756] ? _raw_spin_unlock+0x12/0x30 [ 6584.638826] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6584.642811] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6584.646917] ll_atomic_open+0x36a/0xd70 [lustre] [ 6584.653058] lookup_open+0x356/0xb40 [ 6584.657845] do_last+0x313/0xfc0 [ 6584.661570] ? nd_jump_root+0xe5/0x160 [ 6584.665351] ? path_init+0x437/0x520 [ 6584.668262] path_openat+0xf7/0x500 [ 6584.672728] ? do_raw_spin_unlock+0x75/0x190 [ 6584.676042] do_filp_open+0x99/0x140 [ 6584.677050] ? getname_flags+0x6e/0x330 [ 6584.677990] ? __check_object_size+0xff/0x256 [ 6584.687965] ? do_raw_spin_unlock+0x75/0x190 [ 6584.692586] ? _raw_spin_unlock+0x12/0x30 [ 6584.698072] do_sys_openat2+0x2b4/0x410 [ 6584.701207] do_sys_open+0x73/0xa0 [ 6584.707165] __x64_sys_openat+0x24/0x30 [ 6584.708303] do_syscall_64+0xc1/0x440 [ 6584.709317] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6584.719174] RIP: 0033:0x7f53425a5332 [ 6584.723663] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6584.749981] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6584.758339] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6584.774186] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6584.780664] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6584.789523] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6584.802907] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6592.055164] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 6592.064089] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6592.113731] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6592.131761] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6592.140425] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6592.147769] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6592.162948] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6592.191671] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6592.216651] RAX: 0000000000006e89 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6592.219054] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6592.228810] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6592.252776] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6592.268733] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6592.276969] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6592.286743] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6592.293156] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6592.298026] Call Trace: [ 6592.300718] [ 6592.301614] ? show_regs.cold.9+0x22/0x2f [ 6592.303769] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6592.308907] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6592.316081] ? watchdog+0x40/0x40 [ 6592.319993] ? hrtimer_interrupt+0x128/0x2c0 [ 6592.331782] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6592.343511] ? apic_timer_interrupt+0xf/0x20 [ 6592.347668] [ 6592.353345] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6592.355992] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6592.412652] ? newidle_balance.isra.93+0x207/0x580 [ 6592.415624] do_raw_spin_lock+0x177/0x190 [ 6592.420857] _raw_spin_lock+0x21/0x30 [ 6592.423475] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6592.426752] process_one_work+0x2c8/0x700 [ 6592.428450] worker_thread+0x296/0x6e0 [ 6592.430233] ? rescuer_thread+0x570/0x570 [ 6592.431742] kthread+0x1d1/0x200 [ 6592.433115] ? set_kthread_struct+0x70/0x70 [ 6592.434663] ret_from_fork+0x1f/0x30 [ 6597.605092] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3960s! [ 6597.611152] Showing busy workqueues and worker pools: [ 6597.622880] workqueue events: flags=0x0 [ 6597.633938] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6597.653811] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6597.668602] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6597.679138] pending: cpu_latency_work [ptlrpc] [ 6597.683224] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6597.689158] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6597.700403] workqueue events_long: flags=0x0 [ 6597.706110] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6597.717191] pending: lu_objects_flush [obdclass] [ 6597.719394] workqueue events_unbound: flags=0x2 [ 6597.722643] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6597.733291] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6597.737386] workqueue events_power_efficient: flags=0x80 [ 6597.742421] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6597.750875] pending: check_lifetime, do_cache_clean [ 6597.757654] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6597.762446] pending: neigh_periodic_work, neigh_periodic_work [ 6597.767683] workqueue mm_percpu_wq: flags=0x8 [ 6597.772951] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6597.781100] pending: vmstat_update, lru_add_drain_per_cpu [ 6597.787425] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6597.792812] pending: vmstat_update, lru_add_drain_per_cpu [ 6597.794949] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6597.804845] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6597.812935] workqueue xprtiod: flags=0xa [ 6597.821345] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6597.828481] pending: xprt_autoclose, xs_error_handle [ 6597.831525] workqueue ipv6_addrconf: flags=0x40008 [ 6597.837164] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6597.846577] pending: addrconf_verify_work [ 6597.849903] workqueue knbd0-recv: flags=0x1a [ 6597.853606] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6597.857637] in-flight: 651:recv_work [ 6597.859262] workqueue ptlrpc_pinger: flags=0x2 [ 6597.861594] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6597.868565] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6597.872734] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3960s workers=4 idle: 15979 31824 8 [ 6597.877783] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6588s workers=2 idle: 654 [ 6600.060650] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 6600.068306] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6600.123181] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6600.141706] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6600.147933] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6600.151854] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6600.168105] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6600.171448] RAX: 000000000000620d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6600.174492] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6600.177906] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6600.181535] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6600.185220] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6600.191200] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6600.198856] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6600.205133] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6600.212626] Call Trace: [ 6600.224045] [ 6600.227680] ? show_regs.cold.9+0x22/0x2f [ 6600.235843] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6600.241717] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6600.248310] ? watchdog+0x40/0x40 [ 6600.250203] ? hrtimer_interrupt+0x128/0x2c0 [ 6600.264859] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6600.266498] ? apic_timer_interrupt+0xf/0x20 [ 6600.272879] [ 6600.274633] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6600.291374] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6600.300895] do_raw_spin_lock+0x177/0x190 [ 6600.306792] _raw_spin_lock+0x21/0x30 [ 6600.311026] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6600.316340] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6600.325646] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6600.333918] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6600.340168] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6600.343993] ? woken_wake_function+0x30/0x30 [ 6600.350917] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6600.358401] kthread+0x1d1/0x200 [ 6600.361994] ? set_kthread_struct+0x70/0x70 [ 6600.365870] ret_from_fork+0x1f/0x30 [ 6612.055252] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 6612.057114] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 6612.060903] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) [ 6612.063461] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6612.063537] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6612.063545] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6612.063547] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6612.063563] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6612.063567] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6612.063572] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6612.063575] RDX: 0000000000005554 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6612.063577] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6612.063579] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6612.063582] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6612.063584] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6612.063588] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6612.063590] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6612.063599] Call Trace: [ 6612.063604] [ 6612.063610] ? show_regs.cold.9+0x22/0x2f [ 6612.063619] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6612.063626] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6612.063630] ? watchdog+0x40/0x40 [ 6612.063634] ? hrtimer_interrupt+0x128/0x2c0 [ 6612.063637] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6612.063646] ? apic_timer_interrupt+0xf/0x20 [ 6612.063651] [ 6612.063654] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6612.063659] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6612.063664] do_raw_spin_lock+0x177/0x190 [ 6612.063669] _raw_spin_lock+0x21/0x30 [ 6612.063675] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6612.063913] obd_mod_free+0x46/0x140 [ptlrpc] [ 6612.064095] mdc_commit_open+0x91/0xb0 [mdc] [ 6612.064125] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6612.064248] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6612.064371] after_reply+0xe2c/0x1780 [ptlrpc] [ 6612.064495] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6612.064687] ? __next_timer_interrupt+0x160/0x160 [ 6612.064695] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6612.068480] lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6612.069115] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6612.069123] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6612.069126] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6612.069140] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6612.069144] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6612.069149] RAX: 00000000000026af RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6612.069152] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6612.069154] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6612.069157] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6612.069159] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6612.069163] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6612.069166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6612.069168] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6612.069178] Call Trace: [ 6612.069183] [ 6612.069189] ? show_regs.cold.9+0x22/0x2f [ 6612.069199] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6612.069205] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6612.069210] ? watchdog+0x40/0x40 [ 6612.069215] ? hrtimer_interrupt+0x128/0x2c0 [ 6612.069219] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6612.069228] ? apic_timer_interrupt+0xf/0x20 [ 6612.069232] [ 6612.069235] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6612.069242] do_raw_spin_lock+0x177/0x190 [ 6612.069247] _raw_spin_lock+0x21/0x30 [ 6612.069253] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6612.099651] ? do_wait_intr+0xf0/0xf0 [ 6612.099676] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6612.111718] ? do_raw_spin_unlock+0x75/0x190 [ 6612.112221] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6612.112490] ? do_wait_intr+0xf0/0xf0 [ 6612.112501] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6612.112717] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6612.112949] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6612.117359] kthread+0x1d1/0x200 [ 6612.117376] ? set_kthread_struct+0x70/0x70 [ 6612.117380] ret_from_fork+0x1f/0x30 [ 6612.488236] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6612.495860] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6612.497201] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6612.501587] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6612.505993] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6612.508198] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6612.510190] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6612.512327] lmv_intent_open+0x338/0x1150 [lmv] [ 6612.515634] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6612.520401] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6612.522822] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6612.524155] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6612.527413] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6612.534000] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6612.535286] ? do_raw_spin_unlock+0x75/0x190 [ 6612.537573] ? _raw_spin_unlock+0x12/0x30 [ 6612.538896] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6612.546228] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6612.558111] ll_atomic_open+0x36a/0xd70 [lustre] [ 6612.565457] lookup_open+0x356/0xb40 [ 6612.580816] do_last+0x313/0xfc0 [ 6612.582084] ? nd_jump_root+0xe5/0x160 [ 6612.584743] ? path_init+0x437/0x520 [ 6612.585997] path_openat+0xf7/0x500 [ 6612.591724] ? do_raw_spin_unlock+0x75/0x190 [ 6612.595560] do_filp_open+0x99/0x140 [ 6612.599819] ? getname_flags+0x6e/0x330 [ 6612.601227] ? __check_object_size+0xff/0x256 [ 6612.602732] ? do_raw_spin_unlock+0x75/0x190 [ 6612.612010] ? _raw_spin_unlock+0x12/0x30 [ 6612.613389] do_sys_openat2+0x2b4/0x410 [ 6612.617373] do_sys_open+0x73/0xa0 [ 6612.622723] __x64_sys_openat+0x24/0x30 [ 6612.629091] do_syscall_64+0xc1/0x440 [ 6612.630420] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6612.639169] RIP: 0033:0x7f53425a5332 [ 6612.640665] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6612.666089] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6612.670374] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6612.679271] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6612.686527] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6612.703766] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6612.708087] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6624.056035] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 6624.063203] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6624.113316] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6624.121741] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6624.131138] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6624.140814] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6624.143626] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6624.160457] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6624.171048] RAX: 000000000000651c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6624.176468] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6624.180959] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6624.185797] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6624.189475] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6624.198677] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6624.204830] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6624.212128] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6624.217361] Call Trace: [ 6624.218937] [ 6624.220281] ? show_regs.cold.9+0x22/0x2f [ 6624.226935] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6624.231386] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6624.233781] ? watchdog+0x40/0x40 [ 6624.235364] ? hrtimer_interrupt+0x128/0x2c0 [ 6624.242142] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6624.250038] ? apic_timer_interrupt+0xf/0x20 [ 6624.257448] [ 6624.259116] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6624.264372] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6624.270352] ? newidle_balance.isra.93+0x207/0x580 [ 6624.272899] do_raw_spin_lock+0x177/0x190 [ 6624.280353] _raw_spin_lock+0x21/0x30 [ 6624.282744] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6624.285741] process_one_work+0x2c8/0x700 [ 6624.289895] worker_thread+0x296/0x6e0 [ 6624.298026] ? rescuer_thread+0x570/0x570 [ 6624.299987] kthread+0x1d1/0x200 [ 6624.303034] ? set_kthread_struct+0x70/0x70 [ 6624.308798] ret_from_fork+0x1f/0x30 [ 6628.058010] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 6628.062361] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6628.104122] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6628.115947] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6628.125036] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6628.137487] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6628.170374] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6628.177217] RAX: 0000000000003321 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6628.181737] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6628.186350] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6628.195145] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6628.200404] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6628.204171] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6628.211646] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6628.214458] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6628.220044] Call Trace: [ 6628.221768] [ 6628.222862] ? show_regs.cold.9+0x22/0x2f [ 6628.224920] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6628.226773] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6628.228775] ? watchdog+0x40/0x40 [ 6628.231138] ? hrtimer_interrupt+0x128/0x2c0 [ 6628.234016] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6628.238079] ? apic_timer_interrupt+0xf/0x20 [ 6628.239869] [ 6628.240623] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6628.242845] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6628.244864] do_raw_spin_lock+0x177/0x190 [ 6628.246383] _raw_spin_lock+0x21/0x30 [ 6628.247804] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6628.250705] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6628.252774] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6628.254644] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6628.256193] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6628.258187] ? woken_wake_function+0x30/0x30 [ 6628.259377] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6628.261331] kthread+0x1d1/0x200 [ 6628.262324] ? set_kthread_struct+0x70/0x70 [ 6628.263416] ret_from_fork+0x1f/0x30 [ 6628.328270] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 3991s! [ 6628.338168] Showing busy workqueues and worker pools: [ 6628.343099] workqueue events: flags=0x0 [ 6628.346296] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6628.353844] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6628.363997] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6628.368898] pending: cpu_latency_work [ptlrpc] [ 6628.371996] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6628.375304] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6628.384044] workqueue events_long: flags=0x0 [ 6628.388918] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6628.397346] pending: lu_objects_flush [obdclass] [ 6628.404971] workqueue events_unbound: flags=0x2 [ 6628.409155] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6628.411421] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6628.420979] workqueue events_power_efficient: flags=0x80 [ 6628.431078] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6628.438766] pending: check_lifetime, do_cache_clean [ 6628.441452] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6628.447322] pending: neigh_periodic_work, neigh_periodic_work [ 6628.450216] workqueue mm_percpu_wq: flags=0x8 [ 6628.457295] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6628.464971] pending: vmstat_update, lru_add_drain_per_cpu [ 6628.468338] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6628.472419] pending: vmstat_update, lru_add_drain_per_cpu [ 6628.475883] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6628.484514] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6628.490512] workqueue xprtiod: flags=0xa [ 6628.495098] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6628.503379] pending: xprt_autoclose, xs_error_handle [ 6628.510028] workqueue ipv6_addrconf: flags=0x40008 [ 6628.518182] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6628.525505] pending: addrconf_verify_work [ 6628.530753] workqueue knbd0-recv: flags=0x1a [ 6628.535321] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6628.546409] in-flight: 651:recv_work [ 6628.550389] workqueue ptlrpc_pinger: flags=0x2 [ 6628.553970] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6628.558317] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6628.565810] pool 8: cpus=0-3 flags=0x4 nice=0 hung=3991s workers=4 idle: 15979 31824 8 [ 6628.570541] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6619s workers=2 idle: 654 [ 6640.055028] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 6640.057104] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 6640.060483] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6640.060582] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6640.060589] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6640.060592] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6640.060607] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6640.060613] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 [ 6640.064977] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6640.065141] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6640.065148] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6640.065151] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6640.065165] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6640.065169] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6640.065174] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6640.065177] RDX: 000000000000286c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6640.065179] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6640.065182] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6640.065184] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6640.065187] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6640.065191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6640.065193] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6640.065202] Call Trace: [ 6640.065207] [ 6640.065214] ? show_regs.cold.9+0x22/0x2f [ 6640.065223] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6640.065230] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6640.065234] ? watchdog+0x40/0x40 [ 6640.065238] ? hrtimer_interrupt+0x128/0x2c0 [ 6640.065242] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6640.065251] ? apic_timer_interrupt+0xf/0x20 [ 6640.065269] [ 6640.065273] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6640.065278] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6640.065283] do_raw_spin_lock+0x177/0x190 [ 6640.065288] _raw_spin_lock+0x21/0x30 [ 6640.065294] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6640.065681] obd_mod_free+0x46/0x140 [ptlrpc] [ 6640.065900] mdc_commit_open+0x91/0xb0 [mdc] [ 6640.065938] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6640.066223] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6640.066425] after_reply+0xe2c/0x1780 [ptlrpc] [ 6640.114862] ORIG_RAX: ffffffffffffff13 [ 6640.119607] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6640.119899] ? __next_timer_interrupt+0x160/0x160 [ 6640.119910] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6640.120412] ? do_wait_intr+0xf0/0xf0 [ 6640.120431] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6640.120636] kthread+0x1d1/0x200 [ 6640.120641] ? set_kthread_struct+0x70/0x70 [ 6640.120645] ret_from_fork+0x1f/0x30 [ 6640.277574] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6640.280202] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6640.283561] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6640.286564] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6640.290756] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6640.294198] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6640.297208] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6640.299345] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6640.301911] Call Trace: [ 6640.302910] [ 6640.303586] ? show_regs.cold.9+0x22/0x2f [ 6640.304772] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6640.306604] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6640.307890] ? watchdog+0x40/0x40 [ 6640.308941] ? hrtimer_interrupt+0x128/0x2c0 [ 6640.312227] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6640.314638] ? apic_timer_interrupt+0xf/0x20 [ 6640.316442] [ 6640.317348] ? native_safe_halt+0x2e/0x50 [ 6640.319423] kvm_wait+0xcd/0xe0 [ 6640.320522] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6640.322917] do_raw_spin_lock+0x177/0x190 [ 6640.324965] _raw_spin_lock+0x21/0x30 [ 6640.326495] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6640.329478] ? do_raw_spin_unlock+0x75/0x190 [ 6640.331935] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6640.334995] ? do_wait_intr+0xf0/0xf0 [ 6640.336616] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6640.339557] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6640.342664] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6640.344555] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6640.349504] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6640.353405] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6640.355320] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6640.357847] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6640.360690] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6640.365130] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6640.367468] lmv_intent_open+0x338/0x1150 [lmv] [ 6640.370692] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6640.373313] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6640.374622] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6640.376611] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6640.377961] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6640.382153] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6640.384953] ? do_raw_spin_unlock+0x75/0x190 [ 6640.386121] ? _raw_spin_unlock+0x12/0x30 [ 6640.389709] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6640.391563] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6640.394079] ll_atomic_open+0x36a/0xd70 [lustre] [ 6640.396133] lookup_open+0x356/0xb40 [ 6640.397530] do_last+0x313/0xfc0 [ 6640.399229] ? nd_jump_root+0xe5/0x160 [ 6640.400219] ? path_init+0x437/0x520 [ 6640.401899] path_openat+0xf7/0x500 [ 6640.405559] ? do_raw_spin_unlock+0x75/0x190 [ 6640.408120] do_filp_open+0x99/0x140 [ 6640.409071] ? getname_flags+0x6e/0x330 [ 6640.410111] ? __check_object_size+0xff/0x256 [ 6640.411207] ? do_raw_spin_unlock+0x75/0x190 [ 6640.414459] ? _raw_spin_unlock+0x12/0x30 [ 6640.416153] do_sys_openat2+0x2b4/0x410 [ 6640.417938] do_sys_open+0x73/0xa0 [ 6640.419676] __x64_sys_openat+0x24/0x30 [ 6640.421093] do_syscall_64+0xc1/0x440 [ 6640.422612] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6640.426096] RIP: 0033:0x7f53425a5332 [ 6640.428028] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6640.437372] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6640.440999] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6640.445033] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6640.448933] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6640.452888] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6640.456448] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6654.537024] rcu: INFO: rcu_sched self-detected stall on CPU [ 6654.540823] rcu: 2-....: (2464352 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 6654.547334] (t=4560111 jiffies g=274185 q=64833) [ 6654.549910] rcu: rcu_sched kthread starved for 4018477 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 6654.556481] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6654.561804] rcu: RCU grace-period kthread stack dump: [ 6654.565067] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6654.570751] Call Trace: [ 6654.572187] __schedule+0x351/0xcb0 [ 6654.574313] schedule+0xc0/0x180 [ 6654.576172] schedule_timeout+0xb4/0x190 [ 6654.578595] ? do_raw_spin_unlock+0x75/0x190 [ 6654.581360] ? __next_timer_interrupt+0x160/0x160 [ 6654.583809] ? prepare_to_swait_event+0xf3/0x140 [ 6654.587315] rcu_gp_kthread+0x8dd/0x15b0 [ 6654.589054] ? force_qs_rnp+0x2c0/0x2c0 [ 6654.590833] kthread+0x1d1/0x200 [ 6654.592847] ? set_kthread_struct+0x70/0x70 [ 6654.594897] ret_from_fork+0x1f/0x30 [ 6654.596769] rcu: Stack dump where RCU GP kthread last ran: [ 6654.600324] Sending NMI from CPU 2 to CPUs 1: [ 6654.610727] NMI backtrace for cpu 1 [ 6654.610735] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6654.610737] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6654.610739] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6654.610740] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6654.610742] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [ 6654.610744] RAX: 0000000000003cb2 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6654.610746] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6654.610747] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6654.610748] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6654.610749] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6654.610750] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6654.610751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6654.610752] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6654.610754] Call Trace: [ 6654.610755] [ 6654.610755] ? show_regs.cold.9+0x22/0x2f [ 6654.610756] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6654.610758] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6654.610759] ? nmi_handle+0xbd/0x210 [ 6654.610760] ? default_do_nmi+0x4b/0x1b0 [ 6654.610761] ? do_nmi+0x280/0x380 [ 6654.610762] ? end_repeat_nmi+0x16/0x1f [ 6654.610763] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6654.610764] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6654.610766] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6654.610767] [ 6654.610768] do_raw_spin_lock+0x177/0x190 [ 6654.610769] _raw_spin_lock+0x21/0x30 [ 6654.610770] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6654.610771] ? do_raw_spin_unlock+0x75/0x190 [ 6654.610773] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6654.610774] ? do_wait_intr+0xf0/0xf0 [ 6654.610775] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6654.610777] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6654.610778] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6654.610779] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6654.610781] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6654.610782] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6654.610783] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6654.610784] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6654.610785] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6654.610786] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6654.610787] lmv_intent_open+0x338/0x1150 [lmv] [ 6654.610788] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6654.610790] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6654.610791] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6654.610792] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6654.610793] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6654.610794] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6654.610795] ? do_raw_spin_unlock+0x75/0x190 [ 6654.610796] ? _raw_spin_unlock+0x12/0x30 [ 6654.610797] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6654.610798] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6654.610799] ll_atomic_open+0x36a/0xd70 [lustre] [ 6654.610800] lookup_open+0x356/0xb40 [ 6654.610801] do_last+0x313/0xfc0 [ 6654.610802] ? nd_jump_root+0xe5/0x160 [ 6654.610803] ? path_init+0x437/0x520 [ 6654.610804] path_openat+0xf7/0x500 [ 6654.610805] ? do_raw_spin_unlock+0x75/0x190 [ 6654.610806] do_filp_open+0x99/0x140 [ 6654.610807] ? getname_flags+0x6e/0x330 [ 6654.610808] ? __check_object_size+0xff/0x256 [ 6654.610809] ? do_raw_spin_unlock+0x75/0x190 [ 6654.610810] ? _raw_spin_unlock+0x12/0x30 [ 6654.610811] do_sys_openat2+0x2b4/0x410 [ 6654.610812] do_sys_open+0x73/0xa0 [ 6654.610813] __x64_sys_openat+0x24/0x30 [ 6654.610814] do_syscall_64+0xc1/0x440 [ 6654.610815] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6654.610816] RIP: 0033:0x7f53425a5332 [ 6654.610817] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6654.610819] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6654.610821] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6654.610823] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6654.610824] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6654.610825] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6654.610827] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6654.611736] Sending NMI from CPU 2 to CPUs 1: [ 6654.822798] NMI backtrace for cpu 1 [ 6654.822806] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6654.822808] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6654.822809] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6654.822811] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6654.822813] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [ 6654.822816] RAX: 0000000000007e2a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6654.822817] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6654.822819] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6654.822820] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6654.822822] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6654.822823] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6654.822825] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6654.822826] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6654.822827] Call Trace: [ 6654.822828] [ 6654.822829] ? show_regs.cold.9+0x22/0x2f [ 6654.822830] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6654.822832] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6654.822833] ? nmi_handle+0xbd/0x210 [ 6654.822834] ? default_do_nmi+0x4b/0x1b0 [ 6654.822835] ? do_nmi+0x280/0x380 [ 6654.822836] ? end_repeat_nmi+0x16/0x1f [ 6654.822837] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6654.822838] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6654.822840] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6654.822841] [ 6654.822842] do_raw_spin_lock+0x177/0x190 [ 6654.822843] _raw_spin_lock+0x21/0x30 [ 6654.822844] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6654.822845] ? do_raw_spin_unlock+0x75/0x190 [ 6654.822846] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6654.822848] ? do_wait_intr+0xf0/0xf0 [ 6654.822849] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6654.822850] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6654.822851] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6654.822853] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6654.822854] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6654.822855] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6654.822856] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6654.822858] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6654.822859] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6654.822860] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6654.822862] lmv_intent_open+0x338/0x1150 [lmv] [ 6654.822863] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6654.822864] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6654.822865] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6654.822866] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6654.822867] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6654.822868] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6654.822869] ? do_raw_spin_unlock+0x75/0x190 [ 6654.822870] ? _raw_spin_unlock+0x12/0x30 [ 6654.822871] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6654.822872] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6654.822873] ll_atomic_open+0x36a/0xd70 [lustre] [ 6654.822874] lookup_open+0x356/0xb40 [ 6654.822875] do_last+0x313/0xfc0 [ 6654.822876] ? nd_jump_root+0xe5/0x160 [ 6654.822877] ? path_init+0x437/0x520 [ 6654.822878] path_openat+0xf7/0x500 [ 6654.822879] ? do_raw_spin_unlock+0x75/0x190 [ 6654.822880] do_filp_open+0x99/0x140 [ 6654.822881] ? getname_flags+0x6e/0x330 [ 6654.822882] ? __check_object_size+0xff/0x256 [ 6654.822883] ? do_raw_spin_unlock+0x75/0x190 [ 6654.822884] ? _raw_spin_unlock+0x12/0x30 [ 6654.822885] do_sys_openat2+0x2b4/0x410 [ 6654.822886] do_sys_open+0x73/0xa0 [ 6654.822887] __x64_sys_openat+0x24/0x30 [ 6654.822888] do_syscall_64+0xc1/0x440 [ 6654.822889] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6654.822890] RIP: 0033:0x7f53425a5332 [ 6654.822891] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6654.822893] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6654.822895] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6654.822896] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6654.822898] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6654.822899] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6654.822900] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6654.823131] NMI backtrace for cpu 2 [ 6655.008381] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6655.012407] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6655.015544] Call Trace: [ 6655.016460] [ 6655.017248] ? dump_stack+0xbb/0x10e [ 6655.018722] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6655.020905] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6655.022679] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6655.024297] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6655.026216] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6655.028160] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6655.030014] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6655.031977] ? update_process_times+0x6d/0xc0 [ 6655.033833] ? tick_sched_handle.isra.18+0x27/0x90 [ 6655.035967] ? tick_sched_timer+0x53/0xb0 [ 6655.037601] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6655.039451] ? tick_sched_handle.isra.18+0x90/0x90 [ 6655.041310] ? hrtimer_interrupt+0x128/0x2c0 [ 6655.043066] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6655.045218] ? apic_timer_interrupt+0xf/0x20 [ 6655.047382] [ 6655.048319] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6655.050527] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6655.052582] ? do_raw_spin_lock+0x177/0x190 [ 6655.054197] ? _raw_spin_lock+0x21/0x30 [ 6655.055865] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6655.058298] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6655.060479] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6655.062268] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6655.065033] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6655.067525] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6655.069647] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6655.072089] ? __next_timer_interrupt+0x160/0x160 [ 6655.074033] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6655.076133] ? do_wait_intr+0xf0/0xf0 [ 6655.077693] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6655.080219] ? kthread+0x1d1/0x200 [ 6655.081427] ? set_kthread_struct+0x70/0x70 [ 6655.083212] ? ret_from_fork+0x1f/0x30 [ 6656.052831] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 6656.058899] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 6656.065440] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6656.065793] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6656.065802] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6656.065809] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6656.066654] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6656.066799] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6656.066803] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6656.066808] RAX: 000000000000520f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6656.066811] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6656.066813] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6656.066815] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6656.066817] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6656.066821] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6656.066824] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6656.066826] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6656.066834] Call Trace: [ 6656.066840] [ 6656.066845] ? show_regs.cold.9+0x22/0x2f [ 6656.066855] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6656.066861] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6656.066866] ? watchdog+0x40/0x40 [ 6656.066871] ? hrtimer_interrupt+0x128/0x2c0 [ 6656.066875] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6656.066884] ? apic_timer_interrupt+0xf/0x20 [ 6656.066889] [ 6656.066892] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6656.066897] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6656.066901] ? newidle_balance.isra.93+0x207/0x580 [ 6656.066907] do_raw_spin_lock+0x177/0x190 [ 6656.066912] _raw_spin_lock+0x21/0x30 [ 6656.066917] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6656.082604] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6656.082698] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6656.082706] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6656.082713] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6656.082731] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6656.082736] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6656.082740] RAX: 0000000000007992 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6656.082742] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6656.082745] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6656.082746] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6656.082749] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6656.082752] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6656.082755] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6656.082758] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6656.082765] Call Trace: [ 6656.082772] [ 6656.082779] ? show_regs.cold.9+0x22/0x2f [ 6656.082797] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6656.082804] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6656.082809] ? watchdog+0x40/0x40 [ 6656.082814] ? hrtimer_interrupt+0x128/0x2c0 [ 6656.082817] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6656.082831] ? apic_timer_interrupt+0xf/0x20 [ 6656.082837] [ 6656.082840] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6656.082845] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6656.082851] do_raw_spin_lock+0x177/0x190 [ 6656.082856] _raw_spin_lock+0x21/0x30 [ 6656.082860] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6656.083315] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6656.114303] process_one_work+0x2c8/0x700 [ 6656.114324] worker_thread+0x296/0x6e0 [ 6656.114330] ? rescuer_thread+0x570/0x570 [ 6656.114334] kthread+0x1d1/0x200 [ 6656.114338] ? set_kthread_struct+0x70/0x70 [ 6656.114343] ret_from_fork+0x1f/0x30 [ 6656.406512] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6656.410269] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6656.420239] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6656.427876] ? woken_wake_function+0x30/0x30 [ 6656.430565] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6656.434957] kthread+0x1d1/0x200 [ 6656.437421] ? set_kthread_struct+0x70/0x70 [ 6656.447155] ret_from_fork+0x1f/0x30 [ 6659.039133] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4021s! [ 6659.043416] Showing busy workqueues and worker pools: [ 6659.045852] workqueue events: flags=0x0 [ 6659.047844] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6659.051169] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6659.056484] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6659.059834] pending: cpu_latency_work [ptlrpc] [ 6659.062581] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6659.065736] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6659.070058] workqueue events_long: flags=0x0 [ 6659.071436] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6659.073796] pending: lu_objects_flush [obdclass] [ 6659.076652] workqueue events_unbound: flags=0x2 [ 6659.078798] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6659.081863] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6659.084508] workqueue events_power_efficient: flags=0x80 [ 6659.086904] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6659.089747] pending: check_lifetime, do_cache_clean [ 6659.092405] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6659.095772] pending: neigh_periodic_work, neigh_periodic_work [ 6659.098633] workqueue mm_percpu_wq: flags=0x8 [ 6659.100203] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6659.103068] pending: vmstat_update, lru_add_drain_per_cpu [ 6659.105241] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6659.109138] pending: vmstat_update, lru_add_drain_per_cpu [ 6659.111833] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6659.114818] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6659.118137] workqueue xprtiod: flags=0xa [ 6659.120232] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6659.123828] pending: xprt_autoclose, xs_error_handle [ 6659.126505] workqueue ipv6_addrconf: flags=0x40008 [ 6659.128794] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6659.132586] pending: addrconf_verify_work [ 6659.134788] workqueue knbd0-recv: flags=0x1a [ 6659.137277] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6659.140653] in-flight: 651:recv_work [ 6659.142608] workqueue ptlrpc_pinger: flags=0x2 [ 6659.145107] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6659.148414] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6659.151771] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4022s workers=4 idle: 15979 31824 8 [ 6659.160123] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6649s workers=2 idle: 654 [ 6680.055437] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 6680.062613] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6680.064516] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 6680.085930] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6680.085945] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6680.085949] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6680.085963] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6680.085966] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6680.085972] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6680.085975] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6680.085977] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6680.085979] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6680.085981] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6680.085984] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6680.085988] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6680.085990] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6680.085999] Call Trace: [ 6680.086005] [ 6680.086010] ? show_regs.cold.9+0x22/0x2f [ 6680.086020] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6680.086028] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6680.086033] ? watchdog+0x40/0x40 [ 6680.086037] ? hrtimer_interrupt+0x128/0x2c0 [ 6680.086041] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6680.086048] ? apic_timer_interrupt+0xf/0x20 [ 6680.086053] [ 6680.086057] ? native_safe_halt+0x2e/0x50 [ 6680.086061] kvm_wait+0xcd/0xe0 [ 6680.086069] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6680.086079] do_raw_spin_lock+0x177/0x190 [ 6680.086084] _raw_spin_lock+0x21/0x30 [ 6680.086089] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6680.086353] ? do_raw_spin_unlock+0x75/0x190 [ 6680.086358] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6680.086873] ? do_wait_intr+0xf0/0xf0 [ 6680.086882] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6680.094991] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6680.095078] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6680.095086] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6680.095090] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6680.095104] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6680.095108] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6680.095113] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6680.095116] RDX: 000000000000482a RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6680.095118] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6680.095121] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6680.095123] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6680.095126] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6680.095129] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6680.095131] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6680.095138] Call Trace: [ 6680.095143] [ 6680.095149] ? show_regs.cold.9+0x22/0x2f [ 6680.095158] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6680.095164] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6680.095169] ? watchdog+0x40/0x40 [ 6680.095173] ? hrtimer_interrupt+0x128/0x2c0 [ 6680.095178] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6680.095186] ? apic_timer_interrupt+0xf/0x20 [ 6680.095191] [ 6680.095193] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6680.095198] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6680.095202] do_raw_spin_lock+0x177/0x190 [ 6680.095206] _raw_spin_lock+0x21/0x30 [ 6680.095212] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6680.095480] obd_mod_free+0x46/0x140 [ptlrpc] [ 6680.107108] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6680.107399] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6680.115150] mdc_commit_open+0x91/0xb0 [mdc] [ 6680.115197] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6680.123149] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6680.123278] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6680.123320] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6680.123349] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6680.123375] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6680.123460] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6680.123950] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6680.123981] lmv_intent_open+0x338/0x1150 [lmv] [ 6680.124013] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6680.124108] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6680.124188] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6680.124208] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6680.124290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6680.124373] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6680.124460] ? do_raw_spin_unlock+0x75/0x190 [ 6680.124469] ? _raw_spin_unlock+0x12/0x30 [ 6680.124476] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6680.126062] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6680.126087] ll_atomic_open+0x36a/0xd70 [lustre] [ 6680.126195] lookup_open+0x356/0xb40 [ 6680.126205] do_last+0x313/0xfc0 [ 6680.126211] ? nd_jump_root+0xe5/0x160 [ 6680.126216] ? path_init+0x437/0x520 [ 6680.126222] path_openat+0xf7/0x500 [ 6680.126228] ? do_raw_spin_unlock+0x75/0x190 [ 6680.126235] do_filp_open+0x99/0x140 [ 6680.126242] ? getname_flags+0x6e/0x330 [ 6680.126247] ? __check_object_size+0xff/0x256 [ 6680.126254] ? do_raw_spin_unlock+0x75/0x190 [ 6680.126258] ? _raw_spin_unlock+0x12/0x30 [ 6680.126264] do_sys_openat2+0x2b4/0x410 [ 6680.126269] do_sys_open+0x73/0xa0 [ 6680.126274] __x64_sys_openat+0x24/0x30 [ 6680.126278] do_syscall_64+0xc1/0x440 [ 6680.126287] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6680.126293] RIP: 0033:0x7f53425a5332 [ 6680.126301] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6680.126305] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6680.126311] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6680.126314] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6680.126316] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6680.126319] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6680.126321] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6680.714607] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6680.717010] after_reply+0xe2c/0x1780 [ptlrpc] [ 6680.719033] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6680.722114] ? __next_timer_interrupt+0x160/0x160 [ 6680.724651] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6680.726963] ? do_wait_intr+0xf0/0xf0 [ 6680.728528] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6680.731256] kthread+0x1d1/0x200 [ 6680.732583] ? set_kthread_struct+0x70/0x70 [ 6680.734752] ret_from_fork+0x1f/0x30 [ 6684.052610] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 6684.057013] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6684.058535] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 6684.103952] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6684.103968] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6684.103975] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6684.106334] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6684.106348] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6684.106353] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6684.106357] RAX: 000000000000749d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6684.106360] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6684.106362] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6684.106364] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6684.106367] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6684.106370] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6684.106373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6684.106375] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6684.106472] Call Trace: [ 6684.106538] [ 6684.106545] ? show_regs.cold.9+0x22/0x2f [ 6684.106555] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6684.106562] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6684.106567] ? watchdog+0x40/0x40 [ 6684.106573] ? hrtimer_interrupt+0x128/0x2c0 [ 6684.106577] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6684.106585] ? apic_timer_interrupt+0xf/0x20 [ 6684.106590] [ 6684.106594] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6684.106598] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6684.106603] ? newidle_balance.isra.93+0x207/0x580 [ 6684.106609] do_raw_spin_lock+0x177/0x190 [ 6684.106614] _raw_spin_lock+0x21/0x30 [ 6684.106619] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6684.106846] process_one_work+0x2c8/0x700 [ 6684.106853] worker_thread+0x296/0x6e0 [ 6684.106858] ? rescuer_thread+0x570/0x570 [ 6684.106862] kthread+0x1d1/0x200 [ 6684.106867] ? set_kthread_struct+0x70/0x70 [ 6684.106871] ret_from_fork+0x1f/0x30 [ 6684.245051] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6684.280057] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6684.287250] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6684.296250] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6684.299333] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6684.314710] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6684.320619] RAX: 0000000000007289 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6684.329840] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6684.337632] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6684.344335] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6684.347403] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6684.351353] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6684.357175] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6684.370693] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6684.375999] Call Trace: [ 6684.378112] [ 6684.380093] ? show_regs.cold.9+0x22/0x2f [ 6684.386719] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6684.388314] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6684.396340] ? watchdog+0x40/0x40 [ 6684.397815] ? hrtimer_interrupt+0x128/0x2c0 [ 6684.400162] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6684.402712] ? apic_timer_interrupt+0xf/0x20 [ 6684.407947] [ 6684.408878] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6684.416395] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6684.418240] do_raw_spin_lock+0x177/0x190 [ 6684.422076] _raw_spin_lock+0x21/0x30 [ 6684.424470] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6684.427748] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6684.438767] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6684.440728] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6684.455745] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6684.461016] ? woken_wake_function+0x30/0x30 [ 6684.468782] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6684.476535] kthread+0x1d1/0x200 [ 6684.477907] ? set_kthread_struct+0x70/0x70 [ 6684.481399] ret_from_fork+0x1f/0x30 [ 6689.247238] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4052s! [ 6689.251716] Showing busy workqueues and worker pools: [ 6689.256214] workqueue events: flags=0x0 [ 6689.258362] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6689.262935] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6689.270254] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6689.274287] pending: cpu_latency_work [ptlrpc] [ 6689.276801] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6689.280469] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6689.290511] workqueue events_long: flags=0x0 [ 6689.296482] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6689.300995] pending: lu_objects_flush [obdclass] [ 6689.305638] workqueue events_unbound: flags=0x2 [ 6689.307321] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6689.312532] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6689.321839] workqueue events_power_efficient: flags=0x80 [ 6689.330342] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6689.337797] pending: check_lifetime, do_cache_clean [ 6689.342831] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6689.350083] pending: neigh_periodic_work, neigh_periodic_work [ 6689.355326] workqueue mm_percpu_wq: flags=0x8 [ 6689.358528] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6689.368523] pending: vmstat_update, lru_add_drain_per_cpu [ 6689.375175] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6689.382808] pending: vmstat_update, lru_add_drain_per_cpu [ 6689.386769] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6689.395815] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6689.404806] workqueue xprtiod: flags=0xa [ 6689.409794] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6689.419458] pending: xprt_autoclose, xs_error_handle [ 6689.421551] workqueue ipv6_addrconf: flags=0x40008 [ 6689.430924] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6689.437077] pending: addrconf_verify_work [ 6689.441779] workqueue knbd0-recv: flags=0x1a [ 6689.450026] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6689.456957] in-flight: 651:recv_work [ 6689.461026] workqueue ptlrpc_pinger: flags=0x2 [ 6689.471852] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6689.483593] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6689.490327] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4052s workers=4 idle: 15979 31824 8 [ 6689.512042] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6679s workers=2 idle: 654 [ 6708.055659] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 6708.058808] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 6708.058870] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6708.058965] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6708.058974] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6708.058977] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6708.058989] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6708.058997] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6708.059001] RAX: 000000000000722b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6708.059006] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6708.059009] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6708.059012] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6708.059014] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6708.059018] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6708.059024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6708.059026] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6708.059034] Call Trace: [ 6708.059039] [ 6708.059045] ? show_regs.cold.9+0x22/0x2f [ 6708.059054] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6708.059076] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6708.059084] ? watchdog+0x40/0x40 [ 6708.059091] ? hrtimer_interrupt+0x128/0x2c0 [ 6708.059095] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6708.059106] ? apic_timer_interrupt+0xf/0x20 [ 6708.059111] [ 6708.059114] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6708.059124] do_raw_spin_lock+0x177/0x190 [ 6708.059132] _raw_spin_lock+0x21/0x30 [ 6708.059137] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6708.059566] ? do_raw_spin_unlock+0x75/0x190 [ 6708.059572] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6708.060037] ? do_wait_intr+0xf0/0xf0 [ 6708.060046] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6708.060447] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6708.060645] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6708.067836] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) [ 6708.088794] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6708.088966] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6708.088996] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6708.089016] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6708.089044] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6708.089202] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6708.089573] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6708.089600] lmv_intent_open+0x338/0x1150 [lmv] [ 6708.089626] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6708.089723] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6708.090369] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6708.090394] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6708.090480] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6708.090659] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6708.090721] ? do_raw_spin_unlock+0x75/0x190 [ 6708.090727] ? _raw_spin_unlock+0x12/0x30 [ 6708.090733] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6708.090753] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6708.090766] ll_atomic_open+0x36a/0xd70 [lustre] [ 6708.090837] lookup_open+0x356/0xb40 [ 6708.090844] do_last+0x313/0xfc0 [ 6708.090848] ? nd_jump_root+0xe5/0x160 [ 6708.090851] ? path_init+0x437/0x520 [ 6708.090855] path_openat+0xf7/0x500 [ 6708.090858] ? do_raw_spin_unlock+0x75/0x190 [ 6708.090862] do_filp_open+0x99/0x140 [ 6708.090866] ? getname_flags+0x6e/0x330 [ 6708.090869] ? __check_object_size+0xff/0x256 [ 6708.090873] ? do_raw_spin_unlock+0x75/0x190 [ 6708.090876] ? _raw_spin_unlock+0x12/0x30 [ 6708.090879] do_sys_openat2+0x2b4/0x410 [ 6708.090881] do_sys_open+0x73/0xa0 [ 6708.090884] __x64_sys_openat+0x24/0x30 [ 6708.090886] do_syscall_64+0xc1/0x440 [ 6708.090891] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6708.090895] RIP: 0033:0x7f53425a5332 [ 6708.090901] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6708.090904] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6708.090907] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6708.090909] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6708.090910] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6708.090912] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6708.090913] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6708.617715] osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6708.658440] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6708.673844] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6708.682900] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6708.688389] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6708.714039] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6708.721454] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6708.730331] RDX: 0000000000001d0b RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6708.742327] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6708.751766] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6708.762940] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6708.772795] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6708.787203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6708.795490] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6708.806530] Call Trace: [ 6708.810107] [ 6708.813525] ? show_regs.cold.9+0x22/0x2f [ 6708.818977] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6708.825759] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6708.830477] ? watchdog+0x40/0x40 [ 6708.835472] ? hrtimer_interrupt+0x128/0x2c0 [ 6708.845427] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6708.851580] ? apic_timer_interrupt+0xf/0x20 [ 6708.856406] [ 6708.858359] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6708.866538] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6708.873445] do_raw_spin_lock+0x177/0x190 [ 6708.876978] _raw_spin_lock+0x21/0x30 [ 6708.882874] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6708.895508] obd_mod_free+0x46/0x140 [ptlrpc] [ 6708.902537] mdc_commit_open+0x91/0xb0 [mdc] [ 6708.903810] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6708.912895] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6708.921449] after_reply+0xe2c/0x1780 [ptlrpc] [ 6708.931242] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6708.933216] ? __next_timer_interrupt+0x160/0x160 [ 6708.945007] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6708.957799] ? do_wait_intr+0xf0/0xf0 [ 6708.964018] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6708.974815] kthread+0x1d1/0x200 [ 6708.975835] ? set_kthread_struct+0x70/0x70 [ 6708.976904] ret_from_fork+0x1f/0x30 [ 6712.058018] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 6712.063409] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6712.093850] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6712.097902] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6712.101848] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6712.104206] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6712.112250] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 6712.115138] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6712.117914] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6712.120650] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6712.123438] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6712.126522] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6712.129072] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6712.132949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6712.135364] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6712.139175] Call Trace: [ 6712.140091] [ 6712.140721] ? show_regs.cold.9+0x22/0x2f [ 6712.142598] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6712.144438] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6712.146500] ? watchdog+0x40/0x40 [ 6712.147742] ? hrtimer_interrupt+0x128/0x2c0 [ 6712.150033] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6712.152282] ? apic_timer_interrupt+0xf/0x20 [ 6712.153914] [ 6712.155246] ? native_safe_halt+0x2e/0x50 [ 6712.156789] kvm_wait+0xcd/0xe0 [ 6712.158274] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6712.160306] do_raw_spin_lock+0x177/0x190 [ 6712.162076] _raw_spin_lock+0x21/0x30 [ 6712.163564] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6712.166423] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6712.168736] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6712.171348] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6712.174592] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6712.176727] ? woken_wake_function+0x30/0x30 [ 6712.178678] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6712.181757] kthread+0x1d1/0x200 [ 6712.183089] ? set_kthread_struct+0x70/0x70 [ 6712.184715] ret_from_fork+0x1f/0x30 [ 6716.061596] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 6716.068184] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6716.139316] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6716.157858] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6716.166391] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6716.175086] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6716.185211] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6716.216302] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6716.221614] RAX: 0000000000006418 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6716.228198] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6716.231617] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6716.237305] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6716.241710] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6716.249389] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6716.258672] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6716.264432] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6716.271781] Call Trace: [ 6716.273666] [ 6716.275322] ? show_regs.cold.9+0x22/0x2f [ 6716.280711] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6716.284072] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6716.287841] ? watchdog+0x40/0x40 [ 6716.288825] ? hrtimer_interrupt+0x128/0x2c0 [ 6716.289982] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6716.293291] ? apic_timer_interrupt+0xf/0x20 [ 6716.295755] [ 6716.299352] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6716.304889] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6716.309179] ? newidle_balance.isra.93+0x207/0x580 [ 6716.310773] do_raw_spin_lock+0x177/0x190 [ 6716.316080] _raw_spin_lock+0x21/0x30 [ 6716.319552] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6716.322065] process_one_work+0x2c8/0x700 [ 6716.325235] worker_thread+0x296/0x6e0 [ 6716.326786] ? rescuer_thread+0x570/0x570 [ 6716.328664] kthread+0x1d1/0x200 [ 6716.329900] ? set_kthread_struct+0x70/0x70 [ 6716.331624] ret_from_fork+0x1f/0x30 [ 6719.969586] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4082s! [ 6719.972978] Showing busy workqueues and worker pools: [ 6719.980249] workqueue events: flags=0x0 [ 6719.984324] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6719.992314] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6720.006090] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6720.014252] pending: cpu_latency_work [ptlrpc] [ 6720.022842] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6720.027248] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6720.033593] workqueue events_long: flags=0x0 [ 6720.036144] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6720.038643] pending: lu_objects_flush [obdclass] [ 6720.043170] workqueue events_unbound: flags=0x2 [ 6720.051140] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6720.056115] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6720.058649] workqueue events_power_efficient: flags=0x80 [ 6720.061546] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6720.068984] pending: check_lifetime, do_cache_clean [ 6720.072695] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6720.079761] pending: neigh_periodic_work, neigh_periodic_work [ 6720.085220] workqueue mm_percpu_wq: flags=0x8 [ 6720.086692] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6720.092350] pending: vmstat_update, lru_add_drain_per_cpu [ 6720.095555] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6720.099721] pending: vmstat_update, lru_add_drain_per_cpu [ 6720.101384] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6720.105431] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6720.111392] workqueue xprtiod: flags=0xa [ 6720.112773] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6720.115761] pending: xprt_autoclose, xs_error_handle [ 6720.119548] workqueue ipv6_addrconf: flags=0x40008 [ 6720.122732] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6720.126486] pending: addrconf_verify_work [ 6720.129269] workqueue knbd0-recv: flags=0x1a [ 6720.131315] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6720.135214] in-flight: 651:recv_work [ 6720.138366] workqueue ptlrpc_pinger: flags=0x2 [ 6720.141326] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6720.144508] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6720.146287] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4083s workers=4 idle: 15979 31824 8 [ 6720.151108] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6710s workers=2 idle: 654 [ 6736.055305] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 6736.057023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 6736.073278] Modules linked in: [ 6736.076293] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6736.076384] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6736.076392] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6736.076395] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6736.076410] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6736.076414] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6736.076419] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6736.076421] RDX: 000000000000083f RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6736.076424] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6736.076426] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6736.076428] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6736.076431] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6736.076434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6736.076436] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6736.076445] Call Trace: [ 6736.076450] [ 6736.076456] ? show_regs.cold.9+0x22/0x2f [ 6736.076466] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6736.076472] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6736.076476] ? watchdog+0x40/0x40 [ 6736.076481] ? hrtimer_interrupt+0x128/0x2c0 [ 6736.076485] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6736.076494] ? apic_timer_interrupt+0xf/0x20 [ 6736.076499] [ 6736.076502] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6736.076507] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6736.076512] do_raw_spin_lock+0x177/0x190 [ 6736.076517] _raw_spin_lock+0x21/0x30 [ 6736.076524] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6736.081112] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6736.081194] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6736.081202] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6736.081206] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6736.081221] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6736.081225] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6736.081230] RAX: 00000000000073b4 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6736.081232] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6736.081235] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6736.081237] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6736.081239] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6736.081242] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6736.081246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6736.081248] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6736.081322] Call Trace: [ 6736.081329] [ 6736.081335] ? show_regs.cold.9+0x22/0x2f [ 6736.081346] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6736.081353] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6736.081358] ? watchdog+0x40/0x40 [ 6736.081362] ? hrtimer_interrupt+0x128/0x2c0 [ 6736.081366] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6736.081375] ? apic_timer_interrupt+0xf/0x20 [ 6736.081381] [ 6736.081384] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6736.081392] do_raw_spin_lock+0x177/0x190 [ 6736.081397] _raw_spin_lock+0x21/0x30 [ 6736.081402] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6736.082324] ? do_raw_spin_unlock+0x75/0x190 [ 6736.082335] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6736.099299] obd_mod_free+0x46/0x140 [ptlrpc] [ 6736.099750] mdc_commit_open+0x91/0xb0 [mdc] [ 6736.099789] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6736.100027] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6736.100216] after_reply+0xe2c/0x1780 [ptlrpc] [ 6736.100406] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6736.100661] ? __next_timer_interrupt+0x160/0x160 [ 6736.100671] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6736.100930] ? do_wait_intr+0xf0/0xf0 [ 6736.100940] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6736.101139] kthread+0x1d1/0x200 [ 6736.101145] ? set_kthread_struct+0x70/0x70 [ 6736.101150] ret_from_fork+0x1f/0x30 [ 6736.533252] ? do_wait_intr+0xf0/0xf0 [ 6736.535094] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6736.543495] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6736.554299] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6736.562841] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6736.565048] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6736.570895] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6736.581029] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6736.585115] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6736.590427] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6736.592942] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6736.595290] lmv_intent_open+0x338/0x1150 [lmv] [ 6736.597185] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6736.601435] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6736.605991] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6736.615920] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6736.621724] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6736.629408] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6736.631019] ? do_raw_spin_unlock+0x75/0x190 [ 6736.632491] ? _raw_spin_unlock+0x12/0x30 [ 6736.636226] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6736.646602] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6736.653934] ll_atomic_open+0x36a/0xd70 [lustre] [ 6736.661293] lookup_open+0x356/0xb40 [ 6736.669079] do_last+0x313/0xfc0 [ 6736.670681] ? nd_jump_root+0xe5/0x160 [ 6736.677422] ? path_init+0x437/0x520 [ 6736.688926] path_openat+0xf7/0x500 [ 6736.690058] ? do_raw_spin_unlock+0x75/0x190 [ 6736.691472] do_filp_open+0x99/0x140 [ 6736.698231] ? getname_flags+0x6e/0x330 [ 6736.701879] ? __check_object_size+0xff/0x256 [ 6736.710033] ? do_raw_spin_unlock+0x75/0x190 [ 6736.716732] ? _raw_spin_unlock+0x12/0x30 [ 6736.718532] do_sys_openat2+0x2b4/0x410 [ 6736.722675] do_sys_open+0x73/0xa0 [ 6736.725072] __x64_sys_openat+0x24/0x30 [ 6736.726476] do_syscall_64+0xc1/0x440 [ 6736.729511] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6736.736133] RIP: 0033:0x7f53425a5332 [ 6736.743091] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6736.770179] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6736.774948] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6736.787998] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6736.798270] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6736.807975] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6736.813312] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6740.059887] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 6740.068450] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6740.133184] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6740.146748] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6740.162860] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6740.167349] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6740.190454] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6740.201249] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6740.220243] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6740.234176] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6740.242391] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6740.249225] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6740.262841] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6740.270327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6740.282287] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6740.289420] Call Trace: [ 6740.293810] [ 6740.294512] ? show_regs.cold.9+0x22/0x2f [ 6740.300097] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6740.305681] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6740.311409] ? watchdog+0x40/0x40 [ 6740.312591] ? hrtimer_interrupt+0x128/0x2c0 [ 6740.317971] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6740.319123] ? apic_timer_interrupt+0xf/0x20 [ 6740.320089] [ 6740.322453] ? native_safe_halt+0x2e/0x50 [ 6740.326414] kvm_wait+0xcd/0xe0 [ 6740.328467] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6740.333383] do_raw_spin_lock+0x177/0x190 [ 6740.334588] _raw_spin_lock+0x21/0x30 [ 6740.343047] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6740.346251] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6740.349731] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6740.352308] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6740.359348] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6740.368724] ? woken_wake_function+0x30/0x30 [ 6740.374559] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6740.383461] kthread+0x1d1/0x200 [ 6740.391138] ? set_kthread_struct+0x70/0x70 [ 6740.397963] ret_from_fork+0x1f/0x30 [ 6748.054452] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 6748.059623] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6748.093567] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6748.097768] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6748.102616] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6748.106795] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6748.111638] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6748.120019] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6748.124094] RAX: 0000000000001a09 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6748.130940] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6748.144728] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6748.146997] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6748.154776] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6748.157887] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6748.166938] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6748.169457] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6748.173965] Call Trace: [ 6748.176803] [ 6748.177293] ? show_regs.cold.9+0x22/0x2f [ 6748.181870] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6748.186969] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6748.189008] ? watchdog+0x40/0x40 [ 6748.195338] ? hrtimer_interrupt+0x128/0x2c0 [ 6748.196742] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6748.202878] ? apic_timer_interrupt+0xf/0x20 [ 6748.204829] [ 6748.207491] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6748.210621] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6748.215287] ? newidle_balance.isra.93+0x207/0x580 [ 6748.221115] do_raw_spin_lock+0x177/0x190 [ 6748.222724] _raw_spin_lock+0x21/0x30 [ 6748.223930] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6748.238810] process_one_work+0x2c8/0x700 [ 6748.241315] worker_thread+0x296/0x6e0 [ 6748.242567] ? rescuer_thread+0x570/0x570 [ 6748.243982] kthread+0x1d1/0x200 [ 6748.250239] ? set_kthread_struct+0x70/0x70 [ 6748.253019] ret_from_fork+0x1f/0x30 [ 6750.179117] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4113s! [ 6750.187093] Showing busy workqueues and worker pools: [ 6750.196066] workqueue events: flags=0x0 [ 6750.199674] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6750.207246] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6750.219152] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6750.227054] pending: cpu_latency_work [ptlrpc] [ 6750.235452] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6750.242359] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6750.246994] workqueue events_long: flags=0x0 [ 6750.252174] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6750.254663] pending: lu_objects_flush [obdclass] [ 6750.263562] workqueue events_unbound: flags=0x2 [ 6750.267567] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6750.279018] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6750.287100] workqueue events_power_efficient: flags=0x80 [ 6750.296606] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6750.300725] pending: check_lifetime, do_cache_clean [ 6750.303222] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6750.305966] pending: neigh_periodic_work, neigh_periodic_work [ 6750.308827] workqueue mm_percpu_wq: flags=0x8 [ 6750.310979] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6750.318519] pending: vmstat_update, lru_add_drain_per_cpu [ 6750.320990] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6750.327278] pending: vmstat_update, lru_add_drain_per_cpu [ 6750.329552] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6750.338201] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6750.340154] workqueue xprtiod: flags=0xa [ 6750.346173] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6750.353590] pending: xprt_autoclose, xs_error_handle [ 6750.359792] workqueue ipv6_addrconf: flags=0x40008 [ 6750.363119] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6750.374502] pending: addrconf_verify_work [ 6750.376023] workqueue knbd0-recv: flags=0x1a [ 6750.390647] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6750.398981] in-flight: 651:recv_work [ 6750.404300] workqueue ptlrpc_pinger: flags=0x2 [ 6750.408109] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6750.409975] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6750.419102] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4113s workers=4 idle: 15979 31824 8 [ 6750.427017] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6740s workers=2 idle: 654 [ 6764.055077] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 6764.057897] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 6764.061948] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6764.062033] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6764.062041] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6764.062044] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6764.062058] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6764.062063] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6764.062068] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6764.062070] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6764.062072] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6764.062076] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6764.063351] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6764.063367] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6764.063370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6764.063371] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6764.063379] Call Trace: [ 6764.063385] [ 6764.063392] ? show_regs.cold.9+0x22/0x2f [ 6764.063403] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6764.063409] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6764.063413] ? watchdog+0x40/0x40 [ 6764.063417] ? hrtimer_interrupt+0x128/0x2c0 [ 6764.063421] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6764.063429] ? apic_timer_interrupt+0xf/0x20 [ 6764.063433] [ 6764.063437] ? native_safe_halt+0x2e/0x50 [ 6764.063440] kvm_wait+0xcd/0xe0 [ 6764.063448] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6764.063457] do_raw_spin_lock+0x177/0x190 [ 6764.063461] _raw_spin_lock+0x21/0x30 [ 6764.063465] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6764.063732] ? do_raw_spin_unlock+0x75/0x190 [ 6764.063738] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6764.063957] ? do_wait_intr+0xf0/0xf0 [ 6764.063966] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6764.064410] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6764.064628] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6764.064839] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6764.065162] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6764.065200] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6764.065232] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6764.065255] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6764.065337] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6764.065905] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6764.066006] lmv_intent_open+0x338/0x1150 [lmv] [ 6764.066161] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6764.066252] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6764.066337] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6764.066433] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6764.066518] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6764.066602] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6764.066684] ? do_raw_spin_unlock+0x75/0x190 [ 6764.066692] ? _raw_spin_unlock+0x12/0x30 [ 6764.066699] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6764.066727] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6764.066747] ll_atomic_open+0x36a/0xd70 [lustre] [ 6764.066832] lookup_open+0x356/0xb40 [ 6764.066842] do_last+0x313/0xfc0 [ 6764.066847] ? nd_jump_root+0xe5/0x160 [ 6764.066852] ? path_init+0x437/0x520 [ 6764.066856] path_openat+0xf7/0x500 [ 6764.066861] ? do_raw_spin_unlock+0x75/0x190 [ 6764.066866] do_filp_open+0x99/0x140 [ 6764.066873] ? getname_flags+0x6e/0x330 [ 6764.066877] ? __check_object_size+0xff/0x256 [ 6764.066885] ? do_raw_spin_unlock+0x75/0x190 [ 6764.066889] ? _raw_spin_unlock+0x12/0x30 [ 6764.066894] do_sys_openat2+0x2b4/0x410 [ 6764.066900] do_sys_open+0x73/0xa0 [ 6764.066904] __x64_sys_openat+0x24/0x30 [ 6764.066908] do_syscall_64+0xc1/0x440 [ 6764.066915] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6764.066921] RIP: 0033:0x7f53425a5332 [ 6764.066929] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6764.066933] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6764.067006] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6764.067009] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6764.067011] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6764.067013] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6764.067016] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6764.511316] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6764.556526] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6764.564840] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6764.572681] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6764.580547] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6764.602128] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6764.606302] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6764.622521] RDX: 0000000000001061 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6764.634824] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6764.645709] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6764.653727] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6764.663289] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6764.673143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6764.676721] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6764.686833] Call Trace: [ 6764.689220] [ 6764.689988] ? show_regs.cold.9+0x22/0x2f [ 6764.693973] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6764.696882] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6764.701339] ? watchdog+0x40/0x40 [ 6764.703128] ? hrtimer_interrupt+0x128/0x2c0 [ 6764.708242] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6764.710918] ? apic_timer_interrupt+0xf/0x20 [ 6764.717284] [ 6764.719808] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6764.727682] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6764.736252] do_raw_spin_lock+0x177/0x190 [ 6764.743332] _raw_spin_lock+0x21/0x30 [ 6764.746088] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6764.756893] obd_mod_free+0x46/0x140 [ptlrpc] [ 6764.761337] mdc_commit_open+0x91/0xb0 [mdc] [ 6764.764107] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6764.776464] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6764.782328] after_reply+0xe2c/0x1780 [ptlrpc] [ 6764.786698] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6764.789856] ? __next_timer_interrupt+0x160/0x160 [ 6764.792754] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6764.796795] ? do_wait_intr+0xf0/0xf0 [ 6764.800983] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6764.804249] kthread+0x1d1/0x200 [ 6764.807449] ? set_kthread_struct+0x70/0x70 [ 6764.809835] ret_from_fork+0x1f/0x30 [ 6768.058269] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 6768.065132] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6768.086789] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6768.092445] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6768.096575] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6768.099688] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6768.114711] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6768.118522] RAX: 0000000000007f88 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6768.121779] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6768.125438] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6768.128828] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6768.134485] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6768.139968] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6768.143461] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6768.145829] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6768.148596] Call Trace: [ 6768.149509] [ 6768.150214] ? show_regs.cold.9+0x22/0x2f [ 6768.151790] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6768.154097] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6768.156350] ? watchdog+0x40/0x40 [ 6768.158245] ? hrtimer_interrupt+0x128/0x2c0 [ 6768.163614] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6768.165537] ? apic_timer_interrupt+0xf/0x20 [ 6768.167740] [ 6768.168673] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6768.170788] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6768.172536] do_raw_spin_lock+0x177/0x190 [ 6768.173930] _raw_spin_lock+0x21/0x30 [ 6768.175993] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6768.179124] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6768.181744] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6768.183872] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6768.185957] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6768.187983] ? woken_wake_function+0x30/0x30 [ 6768.189560] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6768.191885] kthread+0x1d1/0x200 [ 6768.192999] ? set_kthread_struct+0x70/0x70 [ 6768.195160] ret_from_fork+0x1f/0x30 [ 6776.052062] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 6776.060760] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6776.085888] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6776.091011] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6776.095009] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6776.099544] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6776.106443] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6776.121741] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 6776.131271] RAX: 000000000000492a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6776.136740] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6776.152355] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6776.160663] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6776.163802] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6776.172571] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6776.181055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6776.184466] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6776.191792] Call Trace: [ 6776.193540] [ 6776.194610] ? show_regs.cold.9+0x22/0x2f [ 6776.200558] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6776.202905] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6776.206287] ? watchdog+0x40/0x40 [ 6776.208093] ? hrtimer_interrupt+0x128/0x2c0 [ 6776.214245] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6776.216391] ? apic_timer_interrupt+0xf/0x20 [ 6776.219264] [ 6776.221987] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6776.225470] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6776.229420] ? newidle_balance.isra.93+0x207/0x580 [ 6776.233191] do_raw_spin_lock+0x177/0x190 [ 6776.235328] _raw_spin_lock+0x21/0x30 [ 6776.237810] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6776.240963] process_one_work+0x2c8/0x700 [ 6776.243390] worker_thread+0x296/0x6e0 [ 6776.245135] ? rescuer_thread+0x570/0x570 [ 6776.247204] kthread+0x1d1/0x200 [ 6776.248806] ? set_kthread_struct+0x70/0x70 [ 6776.250287] ret_from_fork+0x1f/0x30 [ 6780.896021] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4143s! [ 6780.899704] Showing busy workqueues and worker pools: [ 6780.909968] workqueue events: flags=0x0 [ 6780.920953] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6780.936054] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6780.955103] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6780.967153] pending: cpu_latency_work [ptlrpc] [ 6780.975261] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6780.992197] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6780.998871] workqueue events_long: flags=0x0 [ 6781.005140] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6781.014499] pending: lu_objects_flush [obdclass] [ 6781.017488] workqueue events_unbound: flags=0x2 [ 6781.021709] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6781.025523] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6781.030360] workqueue events_power_efficient: flags=0x80 [ 6781.034410] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6781.038865] pending: check_lifetime, do_cache_clean [ 6781.045221] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6781.048379] pending: neigh_periodic_work, neigh_periodic_work [ 6781.056509] workqueue mm_percpu_wq: flags=0x8 [ 6781.067277] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6781.071144] pending: vmstat_update, lru_add_drain_per_cpu [ 6781.072672] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6781.074424] pending: vmstat_update, lru_add_drain_per_cpu [ 6781.077191] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6781.083693] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6781.090735] workqueue xprtiod: flags=0xa [ 6781.093547] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6781.100839] pending: xprt_autoclose, xs_error_handle [ 6781.103458] workqueue ipv6_addrconf: flags=0x40008 [ 6781.105962] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6781.113438] pending: addrconf_verify_work [ 6781.117947] workqueue knbd0-recv: flags=0x1a [ 6781.120097] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6781.124265] in-flight: 651:recv_work [ 6781.127019] workqueue ptlrpc_pinger: flags=0x2 [ 6781.129489] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6781.132734] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6781.134769] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4144s workers=4 idle: 15979 31824 8 [ 6781.149862] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6771s workers=2 idle: 654 [ 6792.055067] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 6792.058642] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6792.060113] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 6792.080893] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6792.080907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6792.080910] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6792.080983] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6792.080988] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6792.080993] RAX: 0000000000004d6d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6792.080996] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6792.080999] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6792.081001] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6792.081004] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6792.081008] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6792.081011] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6792.081014] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6792.081022] Call Trace: [ 6792.081028] [ 6792.081043] ? show_regs.cold.9+0x22/0x2f [ 6792.081054] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6792.081059] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6792.081064] ? watchdog+0x40/0x40 [ 6792.081068] ? hrtimer_interrupt+0x128/0x2c0 [ 6792.081182] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6792.081192] ? apic_timer_interrupt+0xf/0x20 [ 6792.081197] [ 6792.081201] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6792.081209] do_raw_spin_lock+0x177/0x190 [ 6792.081215] _raw_spin_lock+0x21/0x30 [ 6792.081220] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6792.081499] ? do_raw_spin_unlock+0x75/0x190 [ 6792.081506] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6792.081834] ? do_wait_intr+0xf0/0xf0 [ 6792.081844] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6792.088569] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6792.088712] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6792.088720] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6792.088723] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6792.088736] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6792.088740] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6792.088745] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6792.088747] RDX: 00000000000059df RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6792.088750] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6792.088753] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6792.088755] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6792.088759] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6792.088762] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6792.088765] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6792.088774] Call Trace: [ 6792.088779] [ 6792.088785] ? show_regs.cold.9+0x22/0x2f [ 6792.088794] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6792.088800] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6792.088805] ? watchdog+0x40/0x40 [ 6792.088810] ? hrtimer_interrupt+0x128/0x2c0 [ 6792.088815] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6792.088822] ? apic_timer_interrupt+0xf/0x20 [ 6792.088827] [ 6792.088831] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6792.088836] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6792.088841] do_raw_spin_lock+0x177/0x190 [ 6792.088846] _raw_spin_lock+0x21/0x30 [ 6792.088851] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6792.093322] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6792.093615] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6792.093828] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6792.093951] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6792.093995] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6792.094026] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6792.094055] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6792.094142] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6792.094381] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6792.094411] lmv_intent_open+0x338/0x1150 [lmv] [ 6792.094439] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6792.094524] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6792.094631] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6792.094655] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6792.094743] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6792.094834] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6792.094926] ? do_raw_spin_unlock+0x75/0x190 [ 6792.094934] ? _raw_spin_unlock+0x12/0x30 [ 6792.094941] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6792.094969] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6792.094991] ll_atomic_open+0x36a/0xd70 [lustre] [ 6792.095082] lookup_open+0x356/0xb40 [ 6792.095092] do_last+0x313/0xfc0 [ 6792.095097] ? nd_jump_root+0xe5/0x160 [ 6792.095102] ? path_init+0x437/0x520 [ 6792.095107] path_openat+0xf7/0x500 [ 6792.095112] ? do_raw_spin_unlock+0x75/0x190 [ 6792.095117] do_filp_open+0x99/0x140 [ 6792.095124] ? getname_flags+0x6e/0x330 [ 6792.095129] ? __check_object_size+0xff/0x256 [ 6792.095137] ? do_raw_spin_unlock+0x75/0x190 [ 6792.095141] ? _raw_spin_unlock+0x12/0x30 [ 6792.095146] do_sys_openat2+0x2b4/0x410 [ 6792.095151] do_sys_open+0x73/0xa0 [ 6792.095155] __x64_sys_openat+0x24/0x30 [ 6792.095159] do_syscall_64+0xc1/0x440 [ 6792.095165] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6792.095172] RIP: 0033:0x7f53425a5332 [ 6792.095180] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6792.095185] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6792.095190] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6792.095193] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6792.095196] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6792.095198] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6792.095201] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6792.329240] obd_mod_free+0x46/0x140 [ptlrpc] [ 6792.330911] mdc_commit_open+0x91/0xb0 [mdc] [ 6792.332284] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6792.334533] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6792.336951] after_reply+0xe2c/0x1780 [ptlrpc] [ 6792.338932] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6792.341012] ? __next_timer_interrupt+0x160/0x160 [ 6792.342873] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6792.344885] ? do_wait_intr+0xf0/0xf0 [ 6792.347415] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6792.349473] kthread+0x1d1/0x200 [ 6792.351000] ? set_kthread_struct+0x70/0x70 [ 6792.352403] ret_from_fork+0x1f/0x30 [ 6796.058478] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 6796.062122] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6796.098569] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6796.103065] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6796.111198] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6796.115922] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6796.127175] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6796.132113] RAX: 0000000000005e77 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6796.138599] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6796.142886] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6796.154243] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6796.160767] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6796.165001] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6796.173508] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6796.174847] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6796.182581] Call Trace: [ 6796.184690] [ 6796.186904] ? show_regs.cold.9+0x22/0x2f [ 6796.191903] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6796.196451] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6796.201029] ? watchdog+0x40/0x40 [ 6796.209341] ? hrtimer_interrupt+0x128/0x2c0 [ 6796.212860] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6796.223253] ? apic_timer_interrupt+0xf/0x20 [ 6796.227708] [ 6796.233412] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6796.239968] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6796.245059] do_raw_spin_lock+0x177/0x190 [ 6796.251421] _raw_spin_lock+0x21/0x30 [ 6796.254497] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6796.261170] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6796.269033] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6796.275453] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6796.282710] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6796.292707] ? woken_wake_function+0x30/0x30 [ 6796.294748] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6796.303593] kthread+0x1d1/0x200 [ 6796.308604] ? set_kthread_struct+0x70/0x70 [ 6796.312972] ret_from_fork+0x1f/0x30 [ 6808.056102] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 6808.063962] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6808.109901] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6808.125659] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6808.142437] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6808.157077] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6808.166607] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6808.198430] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6808.207350] RAX: 0000000000006d84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6808.214750] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6808.230972] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6808.234788] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6808.236616] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6808.258684] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6808.264865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6808.275490] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6808.284891] Call Trace: [ 6808.292915] [ 6808.293668] ? show_regs.cold.9+0x22/0x2f [ 6808.294981] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6808.297253] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6808.299662] ? watchdog+0x40/0x40 [ 6808.303739] ? hrtimer_interrupt+0x128/0x2c0 [ 6808.308706] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6808.316224] ? apic_timer_interrupt+0xf/0x20 [ 6808.324841] [ 6808.330835] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6808.336056] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6808.345723] ? newidle_balance.isra.93+0x207/0x580 [ 6808.351942] do_raw_spin_lock+0x177/0x190 [ 6808.356033] _raw_spin_lock+0x21/0x30 [ 6808.362846] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6808.368401] process_one_work+0x2c8/0x700 [ 6808.374158] worker_thread+0x296/0x6e0 [ 6808.375506] ? rescuer_thread+0x570/0x570 [ 6808.392593] kthread+0x1d1/0x200 [ 6808.399821] ? set_kthread_struct+0x70/0x70 [ 6808.404898] ret_from_fork+0x1f/0x30 [ 6811.616480] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4174s! [ 6811.618393] Showing busy workqueues and worker pools: [ 6811.624286] workqueue events: flags=0x0 [ 6811.628140] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6811.631160] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6811.636716] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6811.641043] pending: cpu_latency_work [ptlrpc] [ 6811.643117] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6811.645689] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6811.658919] workqueue events_long: flags=0x0 [ 6811.664518] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6811.667957] pending: lu_objects_flush [obdclass] [ 6811.678846] workqueue events_unbound: flags=0x2 [ 6811.686027] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6811.695104] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6811.702056] workqueue events_power_efficient: flags=0x80 [ 6811.705911] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6811.717389] pending: check_lifetime, do_cache_clean [ 6811.724972] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6811.736401] pending: neigh_periodic_work, neigh_periodic_work [ 6811.743639] workqueue mm_percpu_wq: flags=0x8 [ 6811.745883] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6811.749982] pending: vmstat_update, lru_add_drain_per_cpu [ 6811.752470] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6811.759943] pending: vmstat_update, lru_add_drain_per_cpu [ 6811.763124] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6811.773277] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6811.786111] workqueue xprtiod: flags=0xa [ 6811.791078] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6811.803455] pending: xprt_autoclose, xs_error_handle [ 6811.812876] workqueue ipv6_addrconf: flags=0x40008 [ 6811.820972] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6811.830961] pending: addrconf_verify_work [ 6811.835158] workqueue knbd0-recv: flags=0x1a [ 6811.844018] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6811.857714] in-flight: 651:recv_work [ 6811.866018] workqueue ptlrpc_pinger: flags=0x2 [ 6811.871729] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6811.879557] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6811.892965] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4174s workers=4 idle: 15979 31824 8 [ 6811.901784] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6802s workers=2 idle: 654 [ 6820.056626] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 6820.057076] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 6820.060558] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6820.060641] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6820.060648] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6820.060651] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6820.060666] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6820.060671] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6820.060676] RAX: 000000000000650f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6820.060678] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6820.060681] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6820.060683] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6820.060686] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6820.060689] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6820.060692] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6820.060694] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6820.060703] Call Trace: [ 6820.060709] [ 6820.060716] ? show_regs.cold.9+0x22/0x2f [ 6820.060724] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6820.060731] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6820.060735] ? watchdog+0x40/0x40 [ 6820.060740] ? hrtimer_interrupt+0x128/0x2c0 [ 6820.060744] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6820.060752] ? apic_timer_interrupt+0xf/0x20 [ 6820.060757] [ 6820.060760] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6820.060766] do_raw_spin_lock+0x177/0x190 [ 6820.060771] _raw_spin_lock+0x21/0x30 [ 6820.060776] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6820.061045] ? do_raw_spin_unlock+0x75/0x190 [ 6820.061051] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6820.061280] ? do_wait_intr+0xf0/0xf0 [ 6820.061290] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6820.061987] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6820.062706] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6820.062944] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6820.063305] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6820.063352] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6820.063380] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6820.063409] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6820.063507] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6820.063733] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6820.063762] lmv_intent_open+0x338/0x1150 [lmv] [ 6820.063793] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6820.063879] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6820.063962] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6820.063987] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6820.064092] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6820.064393] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6820.064482] ? do_raw_spin_unlock+0x75/0x190 [ 6820.064489] ? _raw_spin_unlock+0x12/0x30 [ 6820.064496] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6820.064524] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6820.064545] ll_atomic_open+0x36a/0xd70 [lustre] [ 6820.064636] lookup_open+0x356/0xb40 [ 6820.064645] do_last+0x313/0xfc0 [ 6820.064650] ? nd_jump_root+0xe5/0x160 [ 6820.064655] ? path_init+0x437/0x520 [ 6820.064661] path_openat+0xf7/0x500 [ 6820.064666] ? do_raw_spin_unlock+0x75/0x190 [ 6820.064671] do_filp_open+0x99/0x140 [ 6820.064677] ? getname_flags+0x6e/0x330 [ 6820.064682] ? __check_object_size+0xff/0x256 [ 6820.064689] ? do_raw_spin_unlock+0x75/0x190 [ 6820.064693] ? _raw_spin_unlock+0x12/0x30 [ 6820.064699] do_sys_openat2+0x2b4/0x410 [ 6820.064703] do_sys_open+0x73/0xa0 [ 6820.064708] __x64_sys_openat+0x24/0x30 [ 6820.064711] do_syscall_64+0xc1/0x440 [ 6820.064718] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6820.064723] RIP: 0033:0x7f53425a5332 [ 6820.064731] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6820.064735] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6820.064741] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6820.064743] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6820.064746] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6820.064748] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6820.064751] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6820.303200] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6820.322262] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6820.327055] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6820.333602] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6820.336448] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6820.344828] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6820.348265] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6820.351401] RDX: 0000000000003edb RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6820.354070] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6820.357601] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6820.360639] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6820.363716] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6820.367466] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6820.369935] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6820.373209] Call Trace: [ 6820.374229] [ 6820.374892] ? show_regs.cold.9+0x22/0x2f [ 6820.376737] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6820.379751] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6820.385556] ? watchdog+0x40/0x40 [ 6820.386918] ? hrtimer_interrupt+0x128/0x2c0 [ 6820.389574] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6820.391679] ? apic_timer_interrupt+0xf/0x20 [ 6820.393029] [ 6820.393936] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6820.396411] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6820.398583] do_raw_spin_lock+0x177/0x190 [ 6820.400204] _raw_spin_lock+0x21/0x30 [ 6820.401778] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6820.403786] obd_mod_free+0x46/0x140 [ptlrpc] [ 6820.405803] mdc_commit_open+0x91/0xb0 [mdc] [ 6820.407961] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6820.410268] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6820.413268] after_reply+0xe2c/0x1780 [ptlrpc] [ 6820.415677] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6820.417829] ? __next_timer_interrupt+0x160/0x160 [ 6820.419875] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6820.421832] ? do_wait_intr+0xf0/0xf0 [ 6820.423251] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6820.425547] kthread+0x1d1/0x200 [ 6820.426858] ? set_kthread_struct+0x70/0x70 [ 6820.428778] ret_from_fork+0x1f/0x30 [ 6824.077038] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 6824.091488] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6824.184471] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6824.211520] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6824.235681] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6824.244710] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6824.263745] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6824.297938] RAX: 0000000000006c8a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6824.306926] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6824.339547] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6824.348677] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6824.368904] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6824.389132] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6824.391933] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6824.421928] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6824.431939] Call Trace: [ 6824.432679] [ 6824.433267] ? show_regs.cold.9+0x22/0x2f [ 6824.446778] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6824.452136] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6824.461169] ? watchdog+0x40/0x40 [ 6824.466207] ? hrtimer_interrupt+0x128/0x2c0 [ 6824.475035] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6824.486923] ? apic_timer_interrupt+0xf/0x20 [ 6824.494905] [ 6824.510693] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6824.516599] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6824.531691] do_raw_spin_lock+0x177/0x190 [ 6824.533148] _raw_spin_lock+0x21/0x30 [ 6824.534395] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6824.562523] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6824.583030] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6824.603558] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6824.616102] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6824.628813] ? woken_wake_function+0x30/0x30 [ 6824.650553] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6824.652759] kthread+0x1d1/0x200 [ 6824.668813] ? set_kthread_struct+0x70/0x70 [ 6824.670311] ret_from_fork+0x1f/0x30 [ 6834.549049] rcu: INFO: rcu_sched self-detected stall on CPU [ 6834.565373] rcu: 2-....: (2559864 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 6834.593527] (t=4740123 jiffies g=274185 q=64897) [ 6834.621233] rcu: rcu_sched kthread starved for 4198489 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 6834.634176] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 6834.648810] rcu: RCU grace-period kthread stack dump: [ 6834.653210] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 6834.678657] Call Trace: [ 6834.682287] __schedule+0x351/0xcb0 [ 6834.693551] schedule+0xc0/0x180 [ 6834.695532] schedule_timeout+0xb4/0x190 [ 6834.701400] ? do_raw_spin_unlock+0x75/0x190 [ 6834.702527] ? __next_timer_interrupt+0x160/0x160 [ 6834.703766] ? prepare_to_swait_event+0xf3/0x140 [ 6834.715812] rcu_gp_kthread+0x8dd/0x15b0 [ 6834.717220] ? force_qs_rnp+0x2c0/0x2c0 [ 6834.718511] kthread+0x1d1/0x200 [ 6834.730395] ? set_kthread_struct+0x70/0x70 [ 6834.731513] ret_from_fork+0x1f/0x30 [ 6834.732472] rcu: Stack dump where RCU GP kthread last ran: [ 6834.737334] Sending NMI from CPU 2 to CPUs 1: [ 6834.742147] NMI backtrace for cpu 1 [ 6834.742154] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6834.742156] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6834.742157] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6834.742159] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6834.742161] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [ 6834.742163] RAX: 0000000000000096 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6834.742165] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6834.742166] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6834.742168] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6834.742170] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6834.742171] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6834.742173] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6834.742174] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6834.742175] Call Trace: [ 6834.742176] [ 6834.742177] ? show_regs.cold.9+0x22/0x2f [ 6834.742178] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6834.742180] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6834.742181] ? nmi_handle+0xbd/0x210 [ 6834.742182] ? default_do_nmi+0x4b/0x1b0 [ 6834.742183] ? do_nmi+0x280/0x380 [ 6834.742184] ? end_repeat_nmi+0x16/0x1f [ 6834.742185] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6834.742186] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6834.742188] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6834.742189] [ 6834.742190] do_raw_spin_lock+0x177/0x190 [ 6834.742191] _raw_spin_lock+0x21/0x30 [ 6834.742192] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6834.742193] ? do_raw_spin_unlock+0x75/0x190 [ 6834.742194] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6834.742195] ? do_wait_intr+0xf0/0xf0 [ 6834.742196] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6834.742198] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6834.742199] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6834.742200] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6834.742201] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6834.742203] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6834.742204] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6834.742205] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6834.742207] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6834.742208] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6834.742209] lmv_intent_open+0x338/0x1150 [lmv] [ 6834.742210] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6834.742212] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6834.742214] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6834.742215] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6834.742216] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6834.742217] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6834.742218] ? do_raw_spin_unlock+0x75/0x190 [ 6834.742220] ? _raw_spin_unlock+0x12/0x30 [ 6834.742221] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6834.742223] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6834.742224] ll_atomic_open+0x36a/0xd70 [lustre] [ 6834.742225] lookup_open+0x356/0xb40 [ 6834.742226] do_last+0x313/0xfc0 [ 6834.742228] ? nd_jump_root+0xe5/0x160 [ 6834.742228] ? path_init+0x437/0x520 [ 6834.742229] path_openat+0xf7/0x500 [ 6834.742230] ? do_raw_spin_unlock+0x75/0x190 [ 6834.742231] do_filp_open+0x99/0x140 [ 6834.742233] ? getname_flags+0x6e/0x330 [ 6834.742234] ? __check_object_size+0xff/0x256 [ 6834.742235] ? do_raw_spin_unlock+0x75/0x190 [ 6834.742236] ? _raw_spin_unlock+0x12/0x30 [ 6834.742238] do_sys_openat2+0x2b4/0x410 [ 6834.742239] do_sys_open+0x73/0xa0 [ 6834.742240] __x64_sys_openat+0x24/0x30 [ 6834.742241] do_syscall_64+0xc1/0x440 [ 6834.742242] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6834.742243] RIP: 0033:0x7f53425a5332 [ 6834.742245] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6834.742247] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6834.742250] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6834.742252] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6834.742253] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6834.742255] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6834.742257] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6834.745546] Sending NMI from CPU 2 to CPUs 1: [ 6835.198593] INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 23.217 msecs [ 6835.200204] NMI backtrace for cpu 1 [ 6835.200212] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6835.200214] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6835.200216] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6835.200217] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6835.200220] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [ 6835.200222] RAX: 0000000000006f1c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6835.200224] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6835.200225] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6835.200226] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6835.200228] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6835.200230] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6835.200231] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6835.200233] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6835.200234] Call Trace: [ 6835.200236] [ 6835.200237] ? show_regs.cold.9+0x22/0x2f [ 6835.200238] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 6835.200239] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 6835.200240] ? nmi_handle+0xbd/0x210 [ 6835.200252] ? default_do_nmi+0x4b/0x1b0 [ 6835.200253] ? do_nmi+0x280/0x380 [ 6835.200254] ? end_repeat_nmi+0x16/0x1f [ 6835.200255] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6835.200256] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6835.200258] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6835.200259] [ 6835.200260] do_raw_spin_lock+0x177/0x190 [ 6835.200261] _raw_spin_lock+0x21/0x30 [ 6835.200262] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6835.200264] ? do_raw_spin_unlock+0x75/0x190 [ 6835.200265] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6835.200266] ? do_wait_intr+0xf0/0xf0 [ 6835.200267] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6835.200269] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6835.200270] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6835.200271] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6835.200273] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6835.200274] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6835.200276] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6835.200277] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6835.200278] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6835.200280] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6835.200281] lmv_intent_open+0x338/0x1150 [lmv] [ 6835.200282] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6835.200284] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6835.200285] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6835.200286] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6835.200287] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6835.200288] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6835.200289] ? do_raw_spin_unlock+0x75/0x190 [ 6835.200290] ? _raw_spin_unlock+0x12/0x30 [ 6835.200291] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6835.200293] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6835.200294] ll_atomic_open+0x36a/0xd70 [lustre] [ 6835.200295] lookup_open+0x356/0xb40 [ 6835.200296] do_last+0x313/0xfc0 [ 6835.200297] ? nd_jump_root+0xe5/0x160 [ 6835.200299] ? path_init+0x437/0x520 [ 6835.200300] path_openat+0xf7/0x500 [ 6835.200302] ? do_raw_spin_unlock+0x75/0x190 [ 6835.200303] do_filp_open+0x99/0x140 [ 6835.200304] ? getname_flags+0x6e/0x330 [ 6835.200305] ? __check_object_size+0xff/0x256 [ 6835.200306] ? do_raw_spin_unlock+0x75/0x190 [ 6835.200307] ? _raw_spin_unlock+0x12/0x30 [ 6835.200308] do_sys_openat2+0x2b4/0x410 [ 6835.200309] do_sys_open+0x73/0xa0 [ 6835.200310] __x64_sys_openat+0x24/0x30 [ 6835.200311] do_syscall_64+0xc1/0x440 [ 6835.200313] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6835.200314] RIP: 0033:0x7f53425a5332 [ 6835.200316] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6835.200319] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6835.200322] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6835.200323] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6835.200324] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6835.200326] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6835.200327] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6835.200331] NMI backtrace for cpu 2 [ 6835.427859] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6835.434482] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6835.438877] Call Trace: [ 6835.439743] [ 6835.446629] ? dump_stack+0xbb/0x10e [ 6835.450004] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 6835.455665] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 6835.458404] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 6835.462683] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 6835.465388] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 6835.467751] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 6835.470478] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 6835.473156] ? update_process_times+0x6d/0xc0 [ 6835.474811] ? tick_sched_handle.isra.18+0x27/0x90 [ 6835.479991] ? tick_sched_timer+0x53/0xb0 [ 6835.484503] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6835.487349] ? tick_sched_handle.isra.18+0x90/0x90 [ 6835.493439] ? hrtimer_interrupt+0x128/0x2c0 [ 6835.500350] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6835.506452] ? apic_timer_interrupt+0xf/0x20 [ 6835.510745] [ 6835.513792] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6835.522568] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6835.524858] ? do_raw_spin_lock+0x177/0x190 [ 6835.526965] ? _raw_spin_lock+0x21/0x30 [ 6835.528691] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6835.530269] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 6835.534530] ? mdc_commit_open+0x91/0xb0 [mdc] [ 6835.536400] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6835.540589] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6835.547606] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 6835.550628] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6835.556537] ? __next_timer_interrupt+0x160/0x160 [ 6835.558889] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6835.563861] ? do_wait_intr+0xf0/0xf0 [ 6835.564815] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6835.577055] ? kthread+0x1d1/0x200 [ 6835.585674] ? set_kthread_struct+0x70/0x70 [ 6835.588622] ? ret_from_fork+0x1f/0x30 [ 6836.052732] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 6836.056206] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6836.100702] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6836.111206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6836.118207] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6836.125385] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6836.135469] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6836.150516] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6836.156551] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6836.163391] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6836.166524] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6836.169510] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6836.174976] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6836.178901] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6836.182180] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6836.185319] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6836.189509] Call Trace: [ 6836.195898] [ 6836.196672] ? show_regs.cold.9+0x22/0x2f [ 6836.199466] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6836.204761] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6836.206352] ? watchdog+0x40/0x40 [ 6836.207518] ? hrtimer_interrupt+0x128/0x2c0 [ 6836.211711] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6836.219298] ? apic_timer_interrupt+0xf/0x20 [ 6836.221537] [ 6836.222509] ? native_safe_halt+0x2e/0x50 [ 6836.227716] kvm_wait+0xcd/0xe0 [ 6836.231217] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6836.237766] ? newidle_balance.isra.93+0x207/0x580 [ 6836.243541] do_raw_spin_lock+0x177/0x190 [ 6836.245882] _raw_spin_lock+0x21/0x30 [ 6836.252278] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6836.255837] process_one_work+0x2c8/0x700 [ 6836.259589] worker_thread+0x296/0x6e0 [ 6836.261350] ? rescuer_thread+0x570/0x570 [ 6836.266602] kthread+0x1d1/0x200 [ 6836.268438] ? set_kthread_struct+0x70/0x70 [ 6836.269573] ret_from_fork+0x1f/0x30 [ 6842.338375] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4205s! [ 6842.343597] Showing busy workqueues and worker pools: [ 6842.349265] workqueue events: flags=0x0 [ 6842.351270] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6842.355497] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6842.371058] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6842.377561] pending: cpu_latency_work [ptlrpc] [ 6842.380672] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6842.384175] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6842.394397] workqueue events_long: flags=0x0 [ 6842.397236] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6842.405480] pending: lu_objects_flush [obdclass] [ 6842.408364] workqueue events_unbound: flags=0x2 [ 6842.409792] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6842.415035] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6842.417617] workqueue events_power_efficient: flags=0x80 [ 6842.421677] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6842.430104] pending: check_lifetime, do_cache_clean [ 6842.432694] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6842.437136] pending: neigh_periodic_work, neigh_periodic_work [ 6842.440274] workqueue mm_percpu_wq: flags=0x8 [ 6842.442346] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6842.454304] pending: vmstat_update, lru_add_drain_per_cpu [ 6842.466213] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6842.470081] pending: vmstat_update, lru_add_drain_per_cpu [ 6842.473051] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6842.484517] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6842.493386] workqueue xprtiod: flags=0xa [ 6842.498179] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6842.509500] pending: xprt_autoclose, xs_error_handle [ 6842.514821] workqueue ipv6_addrconf: flags=0x40008 [ 6842.520465] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6842.525141] pending: addrconf_verify_work [ 6842.532548] workqueue knbd0-recv: flags=0x1a [ 6842.534104] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6842.538203] in-flight: 651:recv_work [ 6842.541533] workqueue ptlrpc_pinger: flags=0x2 [ 6842.548041] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6842.557972] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6842.563781] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4205s workers=4 idle: 15979 31824 8 [ 6842.572415] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6833s workers=2 idle: 654 [ 6852.058619] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 6852.066348] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6852.090847] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6852.099427] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6852.105957] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6852.107831] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6852.113634] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6852.116589] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6852.119161] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6852.122046] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6852.124435] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6852.127158] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6852.129938] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6852.136666] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6852.139851] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6852.149211] Call Trace: [ 6852.152489] [ 6852.154732] ? show_regs.cold.9+0x22/0x2f [ 6852.156286] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6852.157792] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6852.159287] ? watchdog+0x40/0x40 [ 6852.162303] ? hrtimer_interrupt+0x128/0x2c0 [ 6852.163296] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6852.166098] ? apic_timer_interrupt+0xf/0x20 [ 6852.168302] [ 6852.172187] ? native_safe_halt+0x2e/0x50 [ 6852.176996] kvm_wait+0xcd/0xe0 [ 6852.178880] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6852.183488] do_raw_spin_lock+0x177/0x190 [ 6852.186491] _raw_spin_lock+0x21/0x30 [ 6852.187663] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6852.203230] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6852.208614] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6852.219141] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6852.226116] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6852.233461] ? woken_wake_function+0x30/0x30 [ 6852.235565] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6852.240786] kthread+0x1d1/0x200 [ 6852.242002] ? set_kthread_struct+0x70/0x70 [ 6852.243510] ret_from_fork+0x1f/0x30 [ 6860.055018] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 6860.057899] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6860.068836] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 6860.075987] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6860.076003] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6860.076006] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6860.076020] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6860.076025] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6860.076030] RAX: 0000000000006c1a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6860.076032] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6860.076035] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6860.076037] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6860.076039] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6860.076042] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6860.076045] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6860.076048] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6860.076057] Call Trace: [ 6860.076062] [ 6860.076068] ? show_regs.cold.9+0x22/0x2f [ 6860.076078] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6860.080572] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6860.080652] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6860.080660] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6860.080662] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6860.080677] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6860.080681] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6860.080686] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6860.080688] RDX: 0000000000007f0b RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6860.080691] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6860.080693] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6860.080696] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6860.080699] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6860.080702] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6860.080704] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6860.080792] Call Trace: [ 6860.080799] [ 6860.080816] ? show_regs.cold.9+0x22/0x2f [ 6860.080828] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6860.080834] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6860.080838] ? watchdog+0x40/0x40 [ 6860.080843] ? hrtimer_interrupt+0x128/0x2c0 [ 6860.080847] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6860.080917] ? apic_timer_interrupt+0xf/0x20 [ 6860.080922] [ 6860.080926] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6860.080931] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6860.080936] do_raw_spin_lock+0x177/0x190 [ 6860.081000] _raw_spin_lock+0x21/0x30 [ 6860.081007] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6860.081608] obd_mod_free+0x46/0x140 [ptlrpc] [ 6860.081841] mdc_commit_open+0x91/0xb0 [mdc] [ 6860.082605] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6860.088848] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6860.088870] ? watchdog+0x40/0x40 [ 6860.088880] ? hrtimer_interrupt+0x128/0x2c0 [ 6860.088884] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6860.088891] ? apic_timer_interrupt+0xf/0x20 [ 6860.088896] [ 6860.088904] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6860.088915] do_raw_spin_lock+0x177/0x190 [ 6860.088925] _raw_spin_lock+0x21/0x30 [ 6860.088932] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6860.089291] ? do_raw_spin_unlock+0x75/0x190 [ 6860.089301] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6860.089707] ? do_wait_intr+0xf0/0xf0 [ 6860.089782] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6860.089977] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6860.090239] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6860.090558] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6860.090758] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6860.090807] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6860.090842] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6860.090872] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6860.090966] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6860.091361] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6860.091465] lmv_intent_open+0x338/0x1150 [lmv] [ 6860.091562] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6860.091664] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6860.091757] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6860.091779] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6860.091861] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6860.092085] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6860.092178] ? do_raw_spin_unlock+0x75/0x190 [ 6860.092189] ? _raw_spin_unlock+0x12/0x30 [ 6860.092197] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6860.092232] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6860.097776] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6860.100094] ll_atomic_open+0x36a/0xd70 [lustre] [ 6860.111556] after_reply+0xe2c/0x1780 [ptlrpc] [ 6860.112040] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6860.112269] ? __next_timer_interrupt+0x160/0x160 [ 6860.112279] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6860.112846] ? do_wait_intr+0xf0/0xf0 [ 6860.112857] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6860.113219] kthread+0x1d1/0x200 [ 6860.113231] ? set_kthread_struct+0x70/0x70 [ 6860.113236] ret_from_fork+0x1f/0x30 [ 6860.441993] lookup_open+0x356/0xb40 [ 6860.443950] do_last+0x313/0xfc0 [ 6860.448998] ? nd_jump_root+0xe5/0x160 [ 6860.455135] ? path_init+0x437/0x520 [ 6860.458799] path_openat+0xf7/0x500 [ 6860.461157] ? do_raw_spin_unlock+0x75/0x190 [ 6860.464482] do_filp_open+0x99/0x140 [ 6860.466444] ? getname_flags+0x6e/0x330 [ 6860.470128] ? __check_object_size+0xff/0x256 [ 6860.475913] ? do_raw_spin_unlock+0x75/0x190 [ 6860.480028] ? _raw_spin_unlock+0x12/0x30 [ 6860.484858] do_sys_openat2+0x2b4/0x410 [ 6860.487416] do_sys_open+0x73/0xa0 [ 6860.492316] __x64_sys_openat+0x24/0x30 [ 6860.495560] do_syscall_64+0xc1/0x440 [ 6860.500291] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6860.504620] RIP: 0033:0x7f53425a5332 [ 6860.508375] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6860.525237] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6860.530932] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6860.535898] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6860.540297] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6860.543043] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6860.547984] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6868.053319] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 6868.064421] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6868.121880] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6868.135372] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6868.145052] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6868.151216] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6868.157583] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6868.167860] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6868.171614] RAX: 0000000000007e5b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6868.174328] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6868.181228] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6868.184820] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6868.190954] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6868.195596] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6868.202266] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6868.203807] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6868.209451] Call Trace: [ 6868.211001] [ 6868.211683] ? show_regs.cold.9+0x22/0x2f [ 6868.212962] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6868.219815] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6868.220978] ? watchdog+0x40/0x40 [ 6868.221869] ? hrtimer_interrupt+0x128/0x2c0 [ 6868.233183] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6868.234789] ? apic_timer_interrupt+0xf/0x20 [ 6868.243318] [ 6868.243993] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6868.248386] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6868.253993] ? newidle_balance.isra.93+0x207/0x580 [ 6868.257634] do_raw_spin_lock+0x177/0x190 [ 6868.260760] _raw_spin_lock+0x21/0x30 [ 6868.267197] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6868.268885] process_one_work+0x2c8/0x700 [ 6868.269939] worker_thread+0x296/0x6e0 [ 6868.277697] ? rescuer_thread+0x570/0x570 [ 6868.279036] kthread+0x1d1/0x200 [ 6868.279943] ? set_kthread_struct+0x70/0x70 [ 6868.282521] ret_from_fork+0x1f/0x30 [ 6873.059963] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4235s! [ 6873.071627] Showing busy workqueues and worker pools: [ 6873.076352] workqueue events: flags=0x0 [ 6873.078608] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6873.086103] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6873.099537] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6873.104641] pending: cpu_latency_work [ptlrpc] [ 6873.115130] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6873.118638] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6873.126673] workqueue events_long: flags=0x0 [ 6873.128666] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6873.135934] pending: lu_objects_flush [obdclass] [ 6873.140840] workqueue events_unbound: flags=0x2 [ 6873.142837] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6873.152064] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6873.157703] workqueue events_power_efficient: flags=0x80 [ 6873.161697] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6873.165979] pending: check_lifetime, do_cache_clean [ 6873.168940] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6873.172689] pending: neigh_periodic_work, neigh_periodic_work [ 6873.175885] workqueue mm_percpu_wq: flags=0x8 [ 6873.180969] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6873.183021] pending: vmstat_update, lru_add_drain_per_cpu [ 6873.185453] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6873.188114] pending: vmstat_update, lru_add_drain_per_cpu [ 6873.190355] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6873.192715] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6873.195350] workqueue xprtiod: flags=0xa [ 6873.196823] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6873.199623] pending: xprt_autoclose, xs_error_handle [ 6873.201700] workqueue ipv6_addrconf: flags=0x40008 [ 6873.203554] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6873.206317] pending: addrconf_verify_work [ 6873.208972] workqueue knbd0-recv: flags=0x1a [ 6873.210344] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6873.218812] in-flight: 651:recv_work [ 6873.221123] workqueue ptlrpc_pinger: flags=0x2 [ 6873.224685] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6873.230046] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6873.236110] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4236s workers=4 idle: 15979 31824 8 [ 6873.243552] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6863s workers=2 idle: 654 [ 6880.058025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 6880.063015] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6880.109443] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6880.118816] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6880.130379] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6880.137061] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6880.166917] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6880.177555] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6880.187709] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6880.196277] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6880.200505] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6880.206526] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6880.217334] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6880.220332] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6880.224618] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6880.242931] Call Trace: [ 6880.249885] [ 6880.250580] ? show_regs.cold.9+0x22/0x2f [ 6880.251875] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6880.254535] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6880.258601] ? watchdog+0x40/0x40 [ 6880.263169] ? hrtimer_interrupt+0x128/0x2c0 [ 6880.265996] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6880.269743] ? apic_timer_interrupt+0xf/0x20 [ 6880.271202] [ 6880.271938] ? native_safe_halt+0x2e/0x50 [ 6880.277271] kvm_wait+0xcd/0xe0 [ 6880.279465] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6880.285456] do_raw_spin_lock+0x177/0x190 [ 6880.290645] _raw_spin_lock+0x21/0x30 [ 6880.298188] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6880.307790] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6880.315312] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6880.320534] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6880.324552] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6880.328424] ? woken_wake_function+0x30/0x30 [ 6880.336368] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6880.339905] kthread+0x1d1/0x200 [ 6880.342257] ? set_kthread_struct+0x70/0x70 [ 6880.344526] ret_from_fork+0x1f/0x30 [ 6888.056512] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 6888.060520] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 6888.069605] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6888.069684] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6888.069691] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6888.069695] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6888.070340] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6888.070345] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6888.070350] RAX: 0000000000000e18 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6888.070353] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6888.070355] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6888.070358] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6888.070360] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6888.070364] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6888.070368] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6888.070370] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6888.070379] Call Trace: [ 6888.070385] [ 6888.070391] ? show_regs.cold.9+0x22/0x2f [ 6888.070400] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6888.070407] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6888.070412] ? watchdog+0x40/0x40 [ 6888.070417] ? hrtimer_interrupt+0x128/0x2c0 [ 6888.070421] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6888.070430] ? apic_timer_interrupt+0xf/0x20 [ 6888.070434] [ 6888.070438] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6888.070446] do_raw_spin_lock+0x177/0x190 [ 6888.070452] _raw_spin_lock+0x21/0x30 [ 6888.070457] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6888.078153] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6888.078232] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6888.078239] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6888.078242] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6888.078256] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6888.078261] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6888.078265] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6888.078268] RDX: 0000000000007a2a RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6888.078270] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6888.078272] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6888.078275] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6888.078278] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6888.078281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6888.078284] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6888.078293] Call Trace: [ 6888.078299] [ 6888.078305] ? show_regs.cold.9+0x22/0x2f [ 6888.078315] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6888.078322] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6888.078326] ? watchdog+0x40/0x40 [ 6888.078330] ? hrtimer_interrupt+0x128/0x2c0 [ 6888.078334] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6888.078342] ? apic_timer_interrupt+0xf/0x20 [ 6888.078347] [ 6888.078349] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6888.078354] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6888.078358] do_raw_spin_lock+0x177/0x190 [ 6888.078363] _raw_spin_lock+0x21/0x30 [ 6888.078370] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6888.078872] obd_mod_free+0x46/0x140 [ptlrpc] [ 6888.121959] ? do_raw_spin_unlock+0x75/0x190 [ 6888.127100] mdc_commit_open+0x91/0xb0 [mdc] [ 6888.127668] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6888.127976] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6888.128652] after_reply+0xe2c/0x1780 [ptlrpc] [ 6888.128869] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6888.129317] ? __next_timer_interrupt+0x160/0x160 [ 6888.129328] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6888.129535] ? do_wait_intr+0xf0/0xf0 [ 6888.129545] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6888.129753] kthread+0x1d1/0x200 [ 6888.129760] ? set_kthread_struct+0x70/0x70 [ 6888.129764] ret_from_fork+0x1f/0x30 [ 6888.473751] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6888.476226] ? do_wait_intr+0xf0/0xf0 [ 6888.480580] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6888.486983] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6888.491884] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6888.497680] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6888.501606] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6888.504127] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6888.506241] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6888.508479] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6888.510417] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6888.513064] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6888.515764] lmv_intent_open+0x338/0x1150 [lmv] [ 6888.517411] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6888.519885] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6888.522566] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6888.524501] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6888.526684] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6888.529286] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6888.531975] ? do_raw_spin_unlock+0x75/0x190 [ 6888.533443] ? _raw_spin_unlock+0x12/0x30 [ 6888.535111] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6888.537126] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6888.539274] ll_atomic_open+0x36a/0xd70 [lustre] [ 6888.541121] lookup_open+0x356/0xb40 [ 6888.542317] do_last+0x313/0xfc0 [ 6888.543674] ? nd_jump_root+0xe5/0x160 [ 6888.545772] ? path_init+0x437/0x520 [ 6888.552098] path_openat+0xf7/0x500 [ 6888.553675] ? do_raw_spin_unlock+0x75/0x190 [ 6888.559276] do_filp_open+0x99/0x140 [ 6888.564677] ? getname_flags+0x6e/0x330 [ 6888.569221] ? __check_object_size+0xff/0x256 [ 6888.571445] ? do_raw_spin_unlock+0x75/0x190 [ 6888.573828] ? _raw_spin_unlock+0x12/0x30 [ 6888.575551] do_sys_openat2+0x2b4/0x410 [ 6888.579023] do_sys_open+0x73/0xa0 [ 6888.584354] __x64_sys_openat+0x24/0x30 [ 6888.589738] do_syscall_64+0xc1/0x440 [ 6888.595982] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6888.601804] RIP: 0033:0x7f53425a5332 [ 6888.603448] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6888.624099] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6888.629856] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6888.638418] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6888.643834] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6888.647662] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6888.655451] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6900.052028] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 6900.059100] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6900.099686] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6900.111363] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6900.116249] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6900.119991] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6900.123680] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6900.133754] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6900.137181] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6900.140248] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6900.143869] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6900.147672] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6900.151948] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6900.155840] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6900.160469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6900.163121] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6900.166353] Call Trace: [ 6900.167685] [ 6900.169166] ? show_regs.cold.9+0x22/0x2f [ 6900.171317] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6900.174137] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6900.176520] ? watchdog+0x40/0x40 [ 6900.178407] ? hrtimer_interrupt+0x128/0x2c0 [ 6900.180516] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6900.183228] ? apic_timer_interrupt+0xf/0x20 [ 6900.185258] [ 6900.186345] ? native_safe_halt+0x2e/0x50 [ 6900.188338] kvm_wait+0xcd/0xe0 [ 6900.189884] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6900.192571] ? newidle_balance.isra.93+0x207/0x580 [ 6900.195653] do_raw_spin_lock+0x177/0x190 [ 6900.200599] _raw_spin_lock+0x21/0x30 [ 6900.202404] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6900.205412] process_one_work+0x2c8/0x700 [ 6900.207361] worker_thread+0x296/0x6e0 [ 6900.209654] ? rescuer_thread+0x570/0x570 [ 6900.211786] kthread+0x1d1/0x200 [ 6900.213448] ? set_kthread_struct+0x70/0x70 [ 6900.215648] ret_from_fork+0x1f/0x30 [ 6903.263114] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4266s! [ 6903.269229] Showing busy workqueues and worker pools: [ 6903.272536] workqueue events: flags=0x0 [ 6903.276101] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6903.279377] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6903.286250] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6903.288605] pending: cpu_latency_work [ptlrpc] [ 6903.290888] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6903.295163] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6903.301530] workqueue events_long: flags=0x0 [ 6903.303550] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6903.306102] pending: lu_objects_flush [obdclass] [ 6903.309617] workqueue events_unbound: flags=0x2 [ 6903.312821] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6903.316580] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6903.320127] workqueue events_power_efficient: flags=0x80 [ 6903.323659] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6903.327550] pending: check_lifetime, do_cache_clean [ 6903.330679] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6903.334859] pending: neigh_periodic_work, neigh_periodic_work [ 6903.338053] workqueue mm_percpu_wq: flags=0x8 [ 6903.345519] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6903.351100] pending: vmstat_update, lru_add_drain_per_cpu [ 6903.355605] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6903.359962] pending: vmstat_update, lru_add_drain_per_cpu [ 6903.362468] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6903.366006] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6903.368468] workqueue xprtiod: flags=0xa [ 6903.369916] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6903.372629] pending: xprt_autoclose, xs_error_handle [ 6903.374781] workqueue ipv6_addrconf: flags=0x40008 [ 6903.376517] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6903.379046] pending: addrconf_verify_work [ 6903.380983] workqueue knbd0-recv: flags=0x1a [ 6903.383021] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6903.385746] in-flight: 651:recv_work [ 6903.387202] workqueue ptlrpc_pinger: flags=0x2 [ 6903.389236] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6903.391696] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6903.394095] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4266s workers=4 idle: 15979 31824 8 [ 6903.397108] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6893s workers=2 idle: 654 [ 6908.058700] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 6908.061725] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6908.115100] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6908.120484] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6908.124093] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6908.126825] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6908.134110] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6908.136983] RAX: 0000000000007f8f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6908.142089] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6908.146806] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6908.152026] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6908.155855] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6908.159043] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6908.164034] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6908.172211] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6908.175734] Call Trace: [ 6908.177701] [ 6908.178298] ? show_regs.cold.9+0x22/0x2f [ 6908.179351] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6908.181863] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6908.184747] ? watchdog+0x40/0x40 [ 6908.186072] ? hrtimer_interrupt+0x128/0x2c0 [ 6908.188861] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6908.191992] ? apic_timer_interrupt+0xf/0x20 [ 6908.194663] [ 6908.195741] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6908.197809] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6908.200248] do_raw_spin_lock+0x177/0x190 [ 6908.201808] _raw_spin_lock+0x21/0x30 [ 6908.205814] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6908.208936] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6908.214197] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6908.216772] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6908.222039] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6908.227742] ? woken_wake_function+0x30/0x30 [ 6908.232688] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6908.234768] kthread+0x1d1/0x200 [ 6908.239709] ? set_kthread_struct+0x70/0x70 [ 6908.240760] ret_from_fork+0x1f/0x30 [ 6916.057190] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 6916.060905] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 6916.070169] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6916.070252] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6916.070261] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6916.070263] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6916.070279] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6916.070284] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6916.070289] RAX: 0000000000005f8f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6916.070291] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6916.070294] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6916.070372] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6916.070375] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6916.070379] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6916.070382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6916.070384] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6916.070393] Call Trace: [ 6916.070398] [ 6916.070405] ? show_regs.cold.9+0x22/0x2f [ 6916.070540] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6916.070547] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6916.070551] ? watchdog+0x40/0x40 [ 6916.070556] ? hrtimer_interrupt+0x128/0x2c0 [ 6916.070560] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6916.070567] ? apic_timer_interrupt+0xf/0x20 [ 6916.070571] [ 6916.070574] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6916.070580] do_raw_spin_lock+0x177/0x190 [ 6916.070585] _raw_spin_lock+0x21/0x30 [ 6916.070591] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6916.071698] ? do_raw_spin_unlock+0x75/0x190 [ 6916.071705] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6916.072045] ? do_wait_intr+0xf0/0xf0 [ 6916.072054] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6916.072250] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6916.072649] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6916.072996] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6916.073112] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6916.073152] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6916.073182] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6916.073212] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6916.073372] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6916.073685] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6916.073964] lmv_intent_open+0x338/0x1150 [lmv] [ 6916.073996] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6916.074086] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6916.074168] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6916.074190] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6916.074276] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6916.074453] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6916.074534] ? do_raw_spin_unlock+0x75/0x190 [ 6916.074540] ? _raw_spin_unlock+0x12/0x30 [ 6916.074548] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6916.074575] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6916.074594] ll_atomic_open+0x36a/0xd70 [lustre] [ 6916.074681] lookup_open+0x356/0xb40 [ 6916.074691] do_last+0x313/0xfc0 [ 6916.074695] ? nd_jump_root+0xe5/0x160 [ 6916.074700] ? path_init+0x437/0x520 [ 6916.074704] path_openat+0xf7/0x500 [ 6916.074708] ? do_raw_spin_unlock+0x75/0x190 [ 6916.074712] do_filp_open+0x99/0x140 [ 6916.074718] ? getname_flags+0x6e/0x330 [ 6916.074854] ? __check_object_size+0xff/0x256 [ 6916.074862] ? do_raw_spin_unlock+0x75/0x190 [ 6916.074865] ? _raw_spin_unlock+0x12/0x30 [ 6916.074870] do_sys_openat2+0x2b4/0x410 [ 6916.074875] do_sys_open+0x73/0xa0 [ 6916.074879] __x64_sys_openat+0x24/0x30 [ 6916.074883] do_syscall_64+0xc1/0x440 [ 6916.074888] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6916.074894] RIP: 0033:0x7f53425a5332 [ 6916.074947] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6916.074953] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6916.074957] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6916.074960] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6916.074962] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6916.074964] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6916.074966] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6916.503548] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6916.546042] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6916.553827] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6916.561278] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6916.565010] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6916.587598] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6916.596204] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6916.603872] RDX: 000000000000547c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6916.612051] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6916.617574] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6916.625655] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6916.636910] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6916.642276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6916.648393] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6916.652730] Call Trace: [ 6916.653762] [ 6916.656866] ? show_regs.cold.9+0x22/0x2f [ 6916.658365] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6916.660330] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6916.662129] ? watchdog+0x40/0x40 [ 6916.669270] ? hrtimer_interrupt+0x128/0x2c0 [ 6916.674934] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6916.680725] ? apic_timer_interrupt+0xf/0x20 [ 6916.686316] [ 6916.687627] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6916.693826] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6916.696270] do_raw_spin_lock+0x177/0x190 [ 6916.697851] _raw_spin_lock+0x21/0x30 [ 6916.699190] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6916.701221] obd_mod_free+0x46/0x140 [ptlrpc] [ 6916.703053] mdc_commit_open+0x91/0xb0 [mdc] [ 6916.704715] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6916.706726] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6916.708969] after_reply+0xe2c/0x1780 [ptlrpc] [ 6916.710882] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6916.712892] ? __next_timer_interrupt+0x160/0x160 [ 6916.714368] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6916.716026] ? do_wait_intr+0xf0/0xf0 [ 6916.717289] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6916.719345] kthread+0x1d1/0x200 [ 6916.720641] ? set_kthread_struct+0x70/0x70 [ 6916.721975] ret_from_fork+0x1f/0x30 [ 6928.052098] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 6928.056739] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6928.104685] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6928.110461] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6928.114873] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6928.118572] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6928.121600] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6928.136284] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6928.140765] RAX: 0000000000004fa2 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6928.142883] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6928.145346] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6928.148671] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6928.150846] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6928.154009] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6928.163925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6928.169728] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6928.179293] Call Trace: [ 6928.183556] [ 6928.184269] ? show_regs.cold.9+0x22/0x2f [ 6928.188685] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6928.191012] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6928.195047] ? watchdog+0x40/0x40 [ 6928.196738] ? hrtimer_interrupt+0x128/0x2c0 [ 6928.200909] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6928.205356] ? apic_timer_interrupt+0xf/0x20 [ 6928.208012] [ 6928.209114] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6928.213787] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6928.220217] ? newidle_balance.isra.93+0x207/0x580 [ 6928.223973] do_raw_spin_lock+0x177/0x190 [ 6928.227271] _raw_spin_lock+0x21/0x30 [ 6928.232493] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6928.240991] process_one_work+0x2c8/0x700 [ 6928.248021] worker_thread+0x296/0x6e0 [ 6928.250085] ? rescuer_thread+0x570/0x570 [ 6928.255996] kthread+0x1d1/0x200 [ 6928.257550] ? set_kthread_struct+0x70/0x70 [ 6928.258682] ret_from_fork+0x1f/0x30 [ 6933.473843] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4296s! [ 6933.483084] Showing busy workqueues and worker pools: [ 6933.487648] workqueue events: flags=0x0 [ 6933.496911] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6933.508932] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6933.529747] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6933.536186] pending: cpu_latency_work [ptlrpc] [ 6933.538093] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6933.544938] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6933.549974] workqueue events_long: flags=0x0 [ 6933.552302] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6933.561899] pending: lu_objects_flush [obdclass] [ 6933.565388] workqueue events_unbound: flags=0x2 [ 6933.569594] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6933.576137] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6933.583323] workqueue events_power_efficient: flags=0x80 [ 6933.585654] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6933.595298] pending: check_lifetime, do_cache_clean [ 6933.604244] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6933.606737] pending: neigh_periodic_work, neigh_periodic_work [ 6933.621056] workqueue mm_percpu_wq: flags=0x8 [ 6933.629065] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6933.637305] pending: vmstat_update, lru_add_drain_per_cpu [ 6933.641178] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6933.644313] pending: vmstat_update, lru_add_drain_per_cpu [ 6933.647952] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6933.654001] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6933.660410] workqueue xprtiod: flags=0xa [ 6933.662827] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6933.670081] pending: xprt_autoclose, xs_error_handle [ 6933.674823] workqueue ipv6_addrconf: flags=0x40008 [ 6933.682246] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6933.690248] pending: addrconf_verify_work [ 6933.700502] workqueue knbd0-recv: flags=0x1a [ 6933.705095] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6933.730618] in-flight: 651:recv_work [ 6933.743014] workqueue ptlrpc_pinger: flags=0x2 [ 6933.744651] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6933.750891] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6933.759210] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4296s workers=4 idle: 15979 31824 8 [ 6933.771153] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6924s workers=2 idle: 654 [ 6936.058081] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 6936.063857] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6936.093977] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6936.108955] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6936.122669] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6936.129869] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6936.151142] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6936.159226] RAX: 0000000000003a92 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6936.161647] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6936.164953] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6936.170264] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6936.174229] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6936.179556] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6936.185707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6936.190577] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6936.192914] Call Trace: [ 6936.197628] [ 6936.200175] ? show_regs.cold.9+0x22/0x2f [ 6936.202871] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6936.206576] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6936.210783] ? watchdog+0x40/0x40 [ 6936.214410] ? hrtimer_interrupt+0x128/0x2c0 [ 6936.220917] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6936.222991] ? apic_timer_interrupt+0xf/0x20 [ 6936.226069] [ 6936.226883] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6936.235703] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6936.237485] do_raw_spin_lock+0x177/0x190 [ 6936.240804] _raw_spin_lock+0x21/0x30 [ 6936.242195] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6936.246816] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6936.250869] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6936.254142] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6936.256419] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6936.264347] ? woken_wake_function+0x30/0x30 [ 6936.265638] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6936.270026] kthread+0x1d1/0x200 [ 6936.271328] ? set_kthread_struct+0x70/0x70 [ 6936.277601] ret_from_fork+0x1f/0x30 [ 6944.056967] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 6944.058661] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 6944.063116] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6944.063196] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6944.069551] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6944.069625] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6944.069633] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6944.069635] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6944.069650] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 6944.069655] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6944.069659] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6944.069662] RDX: 0000000000006cc0 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6944.069665] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6944.069667] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6944.069669] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6944.069673] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6944.069675] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6944.069677] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6944.069685] Call Trace: [ 6944.069691] [ 6944.069698] ? show_regs.cold.9+0x22/0x2f [ 6944.069707] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6944.069712] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6944.069716] ? watchdog+0x40/0x40 [ 6944.069720] ? hrtimer_interrupt+0x128/0x2c0 [ 6944.069724] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6944.069732] ? apic_timer_interrupt+0xf/0x20 [ 6944.069737] [ 6944.069741] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 6944.069746] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6944.069751] do_raw_spin_lock+0x177/0x190 [ 6944.069755] _raw_spin_lock+0x21/0x30 [ 6944.069762] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6944.070689] obd_mod_free+0x46/0x140 [ptlrpc] [ 6944.117060] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6944.121526] mdc_commit_open+0x91/0xb0 [mdc] [ 6944.121577] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6944.149101] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6944.149124] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6944.149129] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6944.149135] RAX: 0000000000000c84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6944.149137] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6944.149139] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6944.149142] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6944.149145] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6944.149148] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6944.149151] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6944.149154] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6944.149162] Call Trace: [ 6944.149167] [ 6944.149173] ? show_regs.cold.9+0x22/0x2f [ 6944.149181] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6944.149510] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6944.149514] ? watchdog+0x40/0x40 [ 6944.149519] ? hrtimer_interrupt+0x128/0x2c0 [ 6944.149523] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6944.149530] ? apic_timer_interrupt+0xf/0x20 [ 6944.149535] [ 6944.149538] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6944.149546] do_raw_spin_lock+0x177/0x190 [ 6944.149551] _raw_spin_lock+0x21/0x30 [ 6944.149555] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6944.149978] ? do_raw_spin_unlock+0x75/0x190 [ 6944.149986] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6944.150190] ? do_wait_intr+0xf0/0xf0 [ 6944.150253] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6944.150452] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6944.154769] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6944.155090] after_reply+0xe2c/0x1780 [ptlrpc] [ 6944.155510] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6944.155713] ? __next_timer_interrupt+0x160/0x160 [ 6944.155723] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6944.156005] ? do_wait_intr+0xf0/0xf0 [ 6944.156014] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6944.157433] kthread+0x1d1/0x200 [ 6944.157441] ? set_kthread_struct+0x70/0x70 [ 6944.157445] ret_from_fork+0x1f/0x30 [ 6944.473386] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6944.476671] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6944.481534] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6944.484786] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6944.486907] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6944.488954] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6944.491556] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6944.494210] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6944.497126] lmv_intent_open+0x338/0x1150 [lmv] [ 6944.499353] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6944.501781] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6944.504523] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6944.506277] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6944.508876] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6944.511272] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6944.516705] ? do_raw_spin_unlock+0x75/0x190 [ 6944.528958] ? _raw_spin_unlock+0x12/0x30 [ 6944.533605] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6944.543265] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6944.550751] ll_atomic_open+0x36a/0xd70 [lustre] [ 6944.561257] lookup_open+0x356/0xb40 [ 6944.565361] do_last+0x313/0xfc0 [ 6944.570247] ? nd_jump_root+0xe5/0x160 [ 6944.572524] ? path_init+0x437/0x520 [ 6944.574619] path_openat+0xf7/0x500 [ 6944.580025] ? do_raw_spin_unlock+0x75/0x190 [ 6944.583862] do_filp_open+0x99/0x140 [ 6944.586691] ? getname_flags+0x6e/0x330 [ 6944.588188] ? __check_object_size+0xff/0x256 [ 6944.589850] ? do_raw_spin_unlock+0x75/0x190 [ 6944.592384] ? _raw_spin_unlock+0x12/0x30 [ 6944.593682] do_sys_openat2+0x2b4/0x410 [ 6944.595570] do_sys_open+0x73/0xa0 [ 6944.597188] __x64_sys_openat+0x24/0x30 [ 6944.598661] do_syscall_64+0xc1/0x440 [ 6944.600518] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6944.603733] RIP: 0033:0x7f53425a5332 [ 6944.605604] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6944.618677] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6944.621466] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6944.625646] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6944.629083] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6944.632507] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6944.636555] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6960.068744] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 6960.077839] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6960.127825] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6960.139020] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6960.143391] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6960.155402] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6960.168119] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6960.201322] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6960.209001] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6960.212219] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6960.215068] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6960.218803] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6960.222227] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6960.226482] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6960.233559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6960.246383] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6960.253154] Call Trace: [ 6960.257400] [ 6960.258255] ? show_regs.cold.9+0x22/0x2f [ 6960.260450] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6960.264056] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6960.266485] ? watchdog+0x40/0x40 [ 6960.275994] ? hrtimer_interrupt+0x128/0x2c0 [ 6960.283991] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6960.290181] ? apic_timer_interrupt+0xf/0x20 [ 6960.299172] [ 6960.302728] ? native_safe_halt+0x2e/0x50 [ 6960.306283] kvm_wait+0xcd/0xe0 [ 6960.308432] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6960.313092] ? newidle_balance.isra.93+0x207/0x580 [ 6960.322336] do_raw_spin_lock+0x177/0x190 [ 6960.327949] _raw_spin_lock+0x21/0x30 [ 6960.333504] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6960.341299] process_one_work+0x2c8/0x700 [ 6960.346516] worker_thread+0x296/0x6e0 [ 6960.350525] ? rescuer_thread+0x570/0x570 [ 6960.353874] kthread+0x1d1/0x200 [ 6960.354898] ? set_kthread_struct+0x70/0x70 [ 6960.360336] ret_from_fork+0x1f/0x30 [ 6964.058020] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 6964.061442] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6964.106802] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6964.115300] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6964.129243] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6964.131939] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6964.151025] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6964.160221] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6964.165675] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6964.171422] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6964.176204] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6964.184298] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6964.193410] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6964.193745] BUG: workqueue lockup - pool [ 6964.200831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6964.200843] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6964.200852] Call Trace: [ 6964.200859] [ 6964.200865] ? show_regs.cold.9+0x22/0x2f [ 6964.200878] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6964.200885] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6964.200890] ? watchdog+0x40/0x40 [ 6964.200895] ? hrtimer_interrupt+0x128/0x2c0 [ 6964.200899] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6964.200907] ? apic_timer_interrupt+0xf/0x20 [ 6964.200912] [ 6964.200916] ? native_safe_halt+0x2e/0x50 [ 6964.200920] kvm_wait+0xcd/0xe0 [ 6964.200927] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6964.200937] do_raw_spin_lock+0x177/0x190 [ 6964.200942] _raw_spin_lock+0x21/0x30 [ 6964.200947] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6964.216353] cpus=0-3 flags=0x4 nice=0 stuck for 4327s! [ 6964.216373] Showing busy workqueues and worker pools: [ 6964.216375] workqueue events: flags=0x0 [ 6964.216379] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6964.216387] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6964.217385] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6964.217397] pending: cpu_latency_work [ptlrpc] [ 6964.224331] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6964.224853] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6964.225072] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6964.225282] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6964.225724] ? woken_wake_function+0x30/0x30 [ 6964.225748] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6964.225973] kthread+0x1d1/0x200 [ 6964.225980] ? set_kthread_struct+0x70/0x70 [ 6964.225984] ret_from_fork+0x1f/0x30 [ 6964.409400] [ 6964.410164] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6964.413126] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6964.418358] workqueue events_long: flags=0x0 [ 6964.419813] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6964.431684] pending: lu_objects_flush [obdclass] [ 6964.438006] workqueue events_unbound: flags=0x2 [ 6964.446662] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6964.448286] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6964.458823] workqueue events_power_efficient: flags=0x80 [ 6964.465347] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6964.476046] pending: check_lifetime, do_cache_clean [ 6964.480086] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6964.485879] pending: neigh_periodic_work, neigh_periodic_work [ 6964.494510] workqueue mm_percpu_wq: flags=0x8 [ 6964.495574] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6964.498414] pending: vmstat_update, lru_add_drain_per_cpu [ 6964.499629] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6964.500977] pending: vmstat_update, lru_add_drain_per_cpu [ 6964.505048] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6964.507712] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6964.509878] workqueue xprtiod: flags=0xa [ 6964.511195] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6964.512858] pending: xprt_autoclose, xs_error_handle [ 6964.515498] workqueue ipv6_addrconf: flags=0x40008 [ 6964.516667] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6964.518816] pending: addrconf_verify_work [ 6964.519993] workqueue knbd0-recv: flags=0x1a [ 6964.521057] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6964.522978] in-flight: 651:recv_work [ 6964.524079] workqueue ptlrpc_pinger: flags=0x2 [ 6964.525197] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6964.526970] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6964.528669] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4327s workers=4 idle: 15979 31824 8 [ 6964.530615] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6954s workers=2 idle: 654 [ 6972.057075] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 6972.060216] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6972.060682] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 6972.081953] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6972.087841] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac [ 6972.093522] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6972.093533] RIP: 0010:native_safe_halt+0x2e/0x50 [ 6972.093548] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 6972.093552] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6972.093557] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 6972.093559] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 6972.093561] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 6972.093563] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 6972.093566] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 6972.093569] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 6972.093572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6972.093574] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 6972.093584] Call Trace: [ 6972.093590] [ 6972.093595] ? show_regs.cold.9+0x22/0x2f [ 6972.093605] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6972.093610] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6972.093614] ? watchdog+0x40/0x40 [ 6972.093619] ? hrtimer_interrupt+0x128/0x2c0 [ 6972.093622] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6972.093629] ? apic_timer_interrupt+0xf/0x20 [ 6972.093635] [ 6972.093638] ? native_safe_halt+0x2e/0x50 [ 6972.093642] kvm_wait+0xcd/0xe0 [ 6972.093649] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 6972.093655] do_raw_spin_lock+0x177/0x190 [ 6972.093660] _raw_spin_lock+0x21/0x30 [ 6972.093666] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 6972.132271] rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6972.132307] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6972.132314] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6972.132317] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6972.132330] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6972.132334] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6972.132338] RAX: 0000000000006b3d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6972.132340] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 6972.132343] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6972.132345] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 6972.132347] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 6972.132350] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 6972.132352] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6972.132354] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 6972.132363] Call Trace: [ 6972.132368] [ 6972.132373] ? show_regs.cold.9+0x22/0x2f [ 6972.132382] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6972.132387] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6972.132392] ? watchdog+0x40/0x40 [ 6972.132396] ? hrtimer_interrupt+0x128/0x2c0 [ 6972.132400] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6972.132408] ? apic_timer_interrupt+0xf/0x20 [ 6972.132412] [ 6972.132415] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6972.132422] do_raw_spin_lock+0x177/0x190 [ 6972.132427] _raw_spin_lock+0x21/0x30 [ 6972.132444] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 6972.133731] ? do_raw_spin_unlock+0x75/0x190 [ 6972.133739] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 6972.133949] ? do_wait_intr+0xf0/0xf0 [ 6972.133958] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 6972.134156] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 6972.134387] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6972.141505] obd_mod_free+0x46/0x140 [ptlrpc] [ 6972.142086] mdc_commit_open+0x91/0xb0 [mdc] [ 6972.142126] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 6972.142402] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 6972.142628] after_reply+0xe2c/0x1780 [ptlrpc] [ 6972.142825] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 6972.143030] ? __next_timer_interrupt+0x160/0x160 [ 6972.143040] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 6972.143249] ? do_wait_intr+0xf0/0xf0 [ 6972.143258] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 6972.143856] kthread+0x1d1/0x200 [ 6972.143863] ? set_kthread_struct+0x70/0x70 [ 6972.143867] ret_from_fork+0x1f/0x30 [ 6972.547123] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6972.549059] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6972.553973] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 6972.556389] mdc_intent_lock+0x353/0x7e0 [mdc] [ 6972.558375] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6972.565665] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 6972.570579] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 6972.574324] lmv_intent_open+0x338/0x1150 [lmv] [ 6972.581044] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6972.585340] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6972.589923] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 6972.594243] ll_intent_lock+0xc5/0xaa0 [lustre] [ 6972.598848] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 6972.602720] ll_lookup_it+0xe0e/0x2270 [lustre] [ 6972.604603] ? do_raw_spin_unlock+0x75/0x190 [ 6972.605959] ? _raw_spin_unlock+0x12/0x30 [ 6972.610578] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 6972.613405] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 6972.615905] ll_atomic_open+0x36a/0xd70 [lustre] [ 6972.621595] lookup_open+0x356/0xb40 [ 6972.624037] do_last+0x313/0xfc0 [ 6972.626363] ? nd_jump_root+0xe5/0x160 [ 6972.628232] ? path_init+0x437/0x520 [ 6972.633334] path_openat+0xf7/0x500 [ 6972.641838] ? do_raw_spin_unlock+0x75/0x190 [ 6972.648773] do_filp_open+0x99/0x140 [ 6972.649948] ? getname_flags+0x6e/0x330 [ 6972.655874] ? __check_object_size+0xff/0x256 [ 6972.658284] ? do_raw_spin_unlock+0x75/0x190 [ 6972.663124] ? _raw_spin_unlock+0x12/0x30 [ 6972.666794] do_sys_openat2+0x2b4/0x410 [ 6972.669201] do_sys_open+0x73/0xa0 [ 6972.673502] __x64_sys_openat+0x24/0x30 [ 6972.675065] do_syscall_64+0xc1/0x440 [ 6972.677347] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 6972.679667] RIP: 0033:0x7f53425a5332 [ 6972.685419] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 6972.697619] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 6972.700349] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 6972.712646] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 6972.716727] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 6972.720470] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 6972.724248] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 6992.056710] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 6992.058289] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 6992.062746] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6992.062840] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6992.062848] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6992.062854] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 6992.063142] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6992.063152] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6992.063155] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 6992.063159] RAX: 00000000000065a1 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6992.063162] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 6992.063164] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 6992.063166] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 6992.063167] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 6992.063170] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 6992.063173] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6992.063175] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 6992.063184] Call Trace: [ 6992.063189] [ 6992.063194] ? show_regs.cold.9+0x22/0x2f [ 6992.063203] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6992.063210] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6992.063215] ? watchdog+0x40/0x40 [ 6992.063219] ? hrtimer_interrupt+0x128/0x2c0 [ 6992.063223] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6992.063231] ? apic_timer_interrupt+0xf/0x20 [ 6992.063235] [ 6992.063239] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6992.063243] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6992.063247] ? newidle_balance.isra.93+0x207/0x580 [ 6992.063252] do_raw_spin_lock+0x177/0x190 [ 6992.063256] _raw_spin_lock+0x21/0x30 [ 6992.063261] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 6992.063472] process_one_work+0x2c8/0x700 [ 6992.063479] worker_thread+0x296/0x6e0 [ 6992.063483] ? rescuer_thread+0x570/0x570 [ 6992.063486] kthread+0x1d1/0x200 [ 6992.063490] ? set_kthread_struct+0x70/0x70 [ 6992.063494] ret_from_fork+0x1f/0x30 [ 6992.253971] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 6992.272488] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 6992.276579] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 6992.280874] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6992.283748] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 6992.295230] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 6992.299538] RAX: 00000000000078c0 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 6992.306873] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 6992.318818] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 6992.329998] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 6992.337473] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 6992.345789] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 6992.351428] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6992.356936] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 6992.363813] Call Trace: [ 6992.366066] [ 6992.367431] ? show_regs.cold.9+0x22/0x2f [ 6992.369317] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 6992.374042] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 6992.376506] ? watchdog+0x40/0x40 [ 6992.378512] ? hrtimer_interrupt+0x128/0x2c0 [ 6992.380936] ? smp_apic_timer_interrupt+0xa6/0x270 [ 6992.383112] ? apic_timer_interrupt+0xf/0x20 [ 6992.385684] [ 6992.386955] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 6992.389385] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 6992.396385] do_raw_spin_lock+0x177/0x190 [ 6992.403265] _raw_spin_lock+0x21/0x30 [ 6992.407281] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 6992.414855] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 6992.420913] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 6992.426292] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 6992.435275] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 6992.440895] ? woken_wake_function+0x30/0x30 [ 6992.443909] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 6992.453101] kthread+0x1d1/0x200 [ 6992.456022] ? set_kthread_struct+0x70/0x70 [ 6992.458966] ret_from_fork+0x1f/0x30 [ 6994.916901] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4357s! [ 6994.924672] Showing busy workqueues and worker pools: [ 6994.926645] workqueue events: flags=0x0 [ 6994.932823] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 6994.939203] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 6994.952384] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6994.958922] pending: cpu_latency_work [ptlrpc] [ 6994.965470] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 6994.972340] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 6994.977887] workqueue events_long: flags=0x0 [ 6994.981718] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 6994.988490] pending: lu_objects_flush [obdclass] [ 6994.991381] workqueue events_unbound: flags=0x2 [ 6994.994487] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 6994.999909] pending: flush_memcg_stats_dwork, idle_cull_fn [ 6995.005888] workqueue events_power_efficient: flags=0x80 [ 6995.007697] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6995.014967] pending: check_lifetime, do_cache_clean [ 6995.020710] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6995.027588] pending: neigh_periodic_work, neigh_periodic_work [ 6995.032432] workqueue mm_percpu_wq: flags=0x8 [ 6995.036109] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6995.041950] pending: vmstat_update, lru_add_drain_per_cpu [ 6995.047447] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 6995.050707] pending: vmstat_update, lru_add_drain_per_cpu [ 6995.056166] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 6995.059827] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 6995.062885] workqueue xprtiod: flags=0xa [ 6995.067118] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 6995.070690] pending: xprt_autoclose, xs_error_handle [ 6995.075310] workqueue ipv6_addrconf: flags=0x40008 [ 6995.079602] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 6995.084765] pending: addrconf_verify_work [ 6995.089905] workqueue knbd0-recv: flags=0x1a [ 6995.091822] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 6995.097925] in-flight: 651:recv_work [ 6995.103714] workqueue ptlrpc_pinger: flags=0x2 [ 6995.108260] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 6995.117569] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 6995.121523] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4358s workers=4 idle: 15979 31824 8 [ 6995.129929] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=6985s workers=2 idle: 654 [ 7000.057014] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 7000.061494] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7000.070138] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 7000.086902] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7000.086916] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7000.086920] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7000.086935] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7000.087298] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7000.087304] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7000.087307] RDX: 000000000000427a RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7000.087309] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7000.087312] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7000.087314] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7000.087318] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7000.087333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7000.087336] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7000.087344] Call Trace: [ 7000.087350] [ 7000.087355] ? show_regs.cold.9+0x22/0x2f [ 7000.087367] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7000.087374] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7000.087378] ? watchdog+0x40/0x40 [ 7000.087383] ? hrtimer_interrupt+0x128/0x2c0 [ 7000.087388] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7000.087397] ? apic_timer_interrupt+0xf/0x20 [ 7000.087416] [ 7000.087419] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7000.087425] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7000.087430] do_raw_spin_lock+0x177/0x190 [ 7000.087435] _raw_spin_lock+0x21/0x30 [ 7000.087443] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7000.087736] obd_mod_free+0x46/0x140 [ptlrpc] [ 7000.087966] mdc_commit_open+0x91/0xb0 [mdc] [ 7000.088005] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7000.088457] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7000.088664] after_reply+0xe2c/0x1780 [ptlrpc] [ 7000.088856] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7000.089290] ? __next_timer_interrupt+0x160/0x160 [ 7000.089300] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7000.089566] ? do_wait_intr+0xf0/0xf0 [ 7000.089574] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7000.089780] kthread+0x1d1/0x200 [ 7000.089787] ? set_kthread_struct+0x70/0x70 [ 7000.089791] ret_from_fork+0x1f/0x30 [ 7000.236503] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7000.257801] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7000.262906] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7000.267232] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7000.270451] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7000.278799] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7000.283861] RAX: 0000000000003405 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7000.288902] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7000.291704] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7000.295084] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7000.299662] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7000.302819] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7000.307303] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7000.310256] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7000.314121] Call Trace: [ 7000.315656] [ 7000.316729] ? show_regs.cold.9+0x22/0x2f [ 7000.318748] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7000.321676] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7000.324771] ? watchdog+0x40/0x40 [ 7000.326067] ? hrtimer_interrupt+0x128/0x2c0 [ 7000.328264] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7000.330967] ? apic_timer_interrupt+0xf/0x20 [ 7000.333682] [ 7000.334765] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7000.337826] do_raw_spin_lock+0x177/0x190 [ 7000.339965] _raw_spin_lock+0x21/0x30 [ 7000.342252] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7000.345640] ? do_raw_spin_unlock+0x75/0x190 [ 7000.348114] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7000.351895] ? do_wait_intr+0xf0/0xf0 [ 7000.353744] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7000.356809] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7000.360259] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7000.363606] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7000.366993] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7000.370570] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7000.373915] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7000.376604] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7000.380418] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7000.383791] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7000.387140] lmv_intent_open+0x338/0x1150 [lmv] [ 7000.390304] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7000.393799] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7000.397670] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7000.400360] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7000.403693] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7000.407542] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7000.410618] ? do_raw_spin_unlock+0x75/0x190 [ 7000.413272] ? _raw_spin_unlock+0x12/0x30 [ 7000.415841] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7000.418912] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7000.421819] ll_atomic_open+0x36a/0xd70 [lustre] [ 7000.426806] lookup_open+0x356/0xb40 [ 7000.428874] do_last+0x313/0xfc0 [ 7000.430619] ? nd_jump_root+0xe5/0x160 [ 7000.433193] ? path_init+0x437/0x520 [ 7000.435426] path_openat+0xf7/0x500 [ 7000.437636] ? do_raw_spin_unlock+0x75/0x190 [ 7000.440371] do_filp_open+0x99/0x140 [ 7000.442345] ? getname_flags+0x6e/0x330 [ 7000.444843] ? __check_object_size+0xff/0x256 [ 7000.447490] ? do_raw_spin_unlock+0x75/0x190 [ 7000.450629] ? _raw_spin_unlock+0x12/0x30 [ 7000.453090] do_sys_openat2+0x2b4/0x410 [ 7000.456805] do_sys_open+0x73/0xa0 [ 7000.459766] __x64_sys_openat+0x24/0x30 [ 7000.461803] do_syscall_64+0xc1/0x440 [ 7000.463230] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7000.466487] RIP: 0033:0x7f53425a5332 [ 7000.468222] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7000.478864] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7000.482587] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7000.485631] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7000.488809] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7000.492678] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7000.496492] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7014.556111] rcu: INFO: rcu_sched self-detected stall on CPU [ 7014.561490] rcu: 2-....: (2658509 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 7014.569646] (t=4920130 jiffies g=274185 q=64950) [ 7014.574600] rcu: rcu_sched kthread starved for 4378496 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 7014.579796] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7014.589395] rcu: RCU grace-period kthread stack dump: [ 7014.593518] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7014.601670] Call Trace: [ 7014.602667] __schedule+0x351/0xcb0 [ 7014.609444] schedule+0xc0/0x180 [ 7014.613642] schedule_timeout+0xb4/0x190 [ 7014.621607] ? do_raw_spin_unlock+0x75/0x190 [ 7014.626058] ? __next_timer_interrupt+0x160/0x160 [ 7014.631678] ? prepare_to_swait_event+0xf3/0x140 [ 7014.638019] rcu_gp_kthread+0x8dd/0x15b0 [ 7014.642337] ? force_qs_rnp+0x2c0/0x2c0 [ 7014.644142] kthread+0x1d1/0x200 [ 7014.645516] ? set_kthread_struct+0x70/0x70 [ 7014.647523] ret_from_fork+0x1f/0x30 [ 7014.650631] rcu: Stack dump where RCU GP kthread last ran: [ 7014.653961] NMI backtrace for cpu 2 [ 7014.655624] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7014.664917] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7014.672188] Call Trace: [ 7014.673406] [ 7014.674490] ? dump_stack+0xbb/0x10e [ 7014.676108] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7014.682390] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7014.686066] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7014.690814] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7014.695629] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7014.702654] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 7014.705786] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 7014.707548] ? update_process_times+0x6d/0xc0 [ 7014.715494] ? tick_sched_handle.isra.18+0x27/0x90 [ 7014.719448] ? tick_sched_timer+0x53/0xb0 [ 7014.722483] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7014.724763] ? tick_sched_handle.isra.18+0x90/0x90 [ 7014.731884] ? hrtimer_interrupt+0x128/0x2c0 [ 7014.736473] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7014.740637] ? apic_timer_interrupt+0xf/0x20 [ 7014.744640] [ 7014.750582] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7014.760428] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7014.763787] ? do_raw_spin_lock+0x177/0x190 [ 7014.768917] ? _raw_spin_lock+0x21/0x30 [ 7014.772790] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7014.775613] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7014.781956] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7014.786420] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7014.788371] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7014.791942] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7014.796044] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7014.799393] ? __next_timer_interrupt+0x160/0x160 [ 7014.801642] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7014.807680] ? do_wait_intr+0xf0/0xf0 [ 7014.810248] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7014.811946] ? kthread+0x1d1/0x200 [ 7014.814069] ? set_kthread_struct+0x70/0x70 [ 7014.817126] ? ret_from_fork+0x1f/0x30 [ 7014.818538] Sending NMI from CPU 2 to CPUs 1: [ 7014.834763] NMI backtrace for cpu 1 [ 7014.834769] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7014.834772] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7014.834774] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7014.834777] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7014.834779] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [ 7014.834782] RAX: 000000000000489a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7014.834784] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7014.834785] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7014.834786] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7014.834788] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7014.834789] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7014.834791] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7014.834792] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7014.834793] Call Trace: [ 7014.834794] [ 7014.834795] ? show_regs.cold.9+0x22/0x2f [ 7014.834796] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7014.834797] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7014.834798] ? nmi_handle+0xbd/0x210 [ 7014.834799] ? default_do_nmi+0x4b/0x1b0 [ 7014.834800] ? do_nmi+0x280/0x380 [ 7014.834801] ? end_repeat_nmi+0x16/0x1f [ 7014.834802] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7014.834803] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7014.834805] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7014.834806] [ 7014.834807] do_raw_spin_lock+0x177/0x190 [ 7014.834808] _raw_spin_lock+0x21/0x30 [ 7014.834809] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7014.834810] ? do_raw_spin_unlock+0x75/0x190 [ 7014.834811] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7014.834812] ? do_wait_intr+0xf0/0xf0 [ 7014.834813] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7014.834814] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7014.834884] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7014.834887] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7014.834888] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7014.834890] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7014.834891] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7014.834892] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7014.834893] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7014.834895] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7014.834896] lmv_intent_open+0x338/0x1150 [lmv] [ 7014.834897] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7014.834898] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7014.834900] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7014.834901] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7014.834902] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7014.834903] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7014.834904] ? do_raw_spin_unlock+0x75/0x190 [ 7014.834905] ? _raw_spin_unlock+0x12/0x30 [ 7014.834906] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7014.834908] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7014.834909] ll_atomic_open+0x36a/0xd70 [lustre] [ 7014.834910] lookup_open+0x356/0xb40 [ 7014.834911] do_last+0x313/0xfc0 [ 7014.834912] ? nd_jump_root+0xe5/0x160 [ 7014.834913] ? path_init+0x437/0x520 [ 7014.834914] path_openat+0xf7/0x500 [ 7014.834915] ? do_raw_spin_unlock+0x75/0x190 [ 7014.834916] do_filp_open+0x99/0x140 [ 7014.834917] ? getname_flags+0x6e/0x330 [ 7014.834918] ? __check_object_size+0xff/0x256 [ 7014.834919] ? do_raw_spin_unlock+0x75/0x190 [ 7014.834920] ? _raw_spin_unlock+0x12/0x30 [ 7014.834921] do_sys_openat2+0x2b4/0x410 [ 7014.834922] do_sys_open+0x73/0xa0 [ 7014.834923] __x64_sys_openat+0x24/0x30 [ 7014.834924] do_syscall_64+0xc1/0x440 [ 7014.834925] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7014.834926] RIP: 0033:0x7f53425a5332 [ 7014.834927] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7014.834929] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7014.834932] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7014.834933] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7014.834935] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7014.834936] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7014.834937] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7014.834944] NMI backtrace for cpu 2 [ 7015.004644] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7015.008417] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7015.011138] Call Trace: [ 7015.012072] [ 7015.012628] ? dump_stack+0xbb/0x10e [ 7015.013975] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7015.016411] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7015.019275] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7015.021573] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7015.023819] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7015.025591] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7015.027192] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7015.029537] ? update_process_times+0x6d/0xc0 [ 7015.031189] ? tick_sched_handle.isra.18+0x27/0x90 [ 7015.033435] ? tick_sched_timer+0x53/0xb0 [ 7015.035345] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7015.037289] ? tick_sched_handle.isra.18+0x90/0x90 [ 7015.039469] ? hrtimer_interrupt+0x128/0x2c0 [ 7015.040951] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7015.042499] ? apic_timer_interrupt+0xf/0x20 [ 7015.044042] [ 7015.044906] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7015.046986] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7015.049744] ? do_raw_spin_lock+0x177/0x190 [ 7015.051509] ? _raw_spin_lock+0x21/0x30 [ 7015.053029] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7015.054952] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7015.057502] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7015.059165] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7015.061529] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7015.063807] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7015.066185] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7015.068951] ? __next_timer_interrupt+0x160/0x160 [ 7015.071377] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7015.073468] ? do_wait_intr+0xf0/0xf0 [ 7015.075613] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7015.077608] ? kthread+0x1d1/0x200 [ 7015.078946] ? set_kthread_struct+0x70/0x70 [ 7015.080430] ? ret_from_fork+0x1f/0x30 [ 7020.054271] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 7020.058061] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 7020.064623] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) [ 7020.067871] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7020.067969] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7020.068048] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7020.068052] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7020.068065] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7020.068069] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7020.068074] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7020.068077] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7020.068079] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7020.068081] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7020.068083] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7020.068086] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7020.068089] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7020.068091] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7020.068100] Call Trace: [ 7020.068104] [ 7020.068109] ? show_regs.cold.9+0x22/0x2f [ 7020.068118] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7020.068124] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7020.068129] ? watchdog+0x40/0x40 [ 7020.068133] ? hrtimer_interrupt+0x128/0x2c0 [ 7020.068137] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7020.068144] ? apic_timer_interrupt+0xf/0x20 [ 7020.068149] [ 7020.068151] ? native_safe_halt+0x2e/0x50 [ 7020.068156] kvm_wait+0xcd/0xe0 [ 7020.068163] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7020.068171] do_raw_spin_lock+0x177/0x190 [ 7020.068175] _raw_spin_lock+0x21/0x30 [ 7020.068180] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7020.068866] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7020.069133] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7020.069740] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7020.070492] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7020.071118] ? woken_wake_function+0x30/0x30 [ 7020.071127] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7020.071493] kthread+0x1d1/0x200 [ 7020.071503] ? set_kthread_struct+0x70/0x70 [ 7020.071507] ret_from_fork+0x1f/0x30 [ 7020.247085] osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7020.265194] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7020.269235] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7020.272114] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7020.274427] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7020.276373] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7020.282228] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7020.285054] RAX: 000000000000168d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7020.289539] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7020.292252] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7020.294647] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7020.297367] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7020.299888] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7020.303689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7020.306114] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7020.308954] Call Trace: [ 7020.309928] [ 7020.310709] ? show_regs.cold.9+0x22/0x2f [ 7020.312232] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7020.314249] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7020.315937] ? watchdog+0x40/0x40 [ 7020.317193] ? hrtimer_interrupt+0x128/0x2c0 [ 7020.318779] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7020.320928] ? apic_timer_interrupt+0xf/0x20 [ 7020.322998] [ 7020.323822] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7020.326225] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7020.328835] ? newidle_balance.isra.93+0x207/0x580 [ 7020.331025] do_raw_spin_lock+0x177/0x190 [ 7020.332903] _raw_spin_lock+0x21/0x30 [ 7020.334375] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7020.336760] process_one_work+0x2c8/0x700 [ 7020.338843] worker_thread+0x296/0x6e0 [ 7020.340277] ? rescuer_thread+0x570/0x570 [ 7020.341751] kthread+0x1d1/0x200 [ 7020.343096] ? set_kthread_struct+0x70/0x70 [ 7020.344978] ret_from_fork+0x1f/0x30 [ 7025.632023] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4388s! [ 7025.636239] Showing busy workqueues and worker pools: [ 7025.638957] workqueue events: flags=0x0 [ 7025.641029] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7025.645680] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7025.652376] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7025.656299] pending: cpu_latency_work [ptlrpc] [ 7025.659048] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7025.662737] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7025.667753] workqueue events_long: flags=0x0 [ 7025.670062] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7025.673885] pending: lu_objects_flush [obdclass] [ 7025.677081] workqueue events_unbound: flags=0x2 [ 7025.679898] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7025.683748] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7025.686993] workqueue events_power_efficient: flags=0x80 [ 7025.690442] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7025.696055] pending: check_lifetime, do_cache_clean [ 7025.699258] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7025.703461] pending: neigh_periodic_work, neigh_periodic_work [ 7025.706880] workqueue mm_percpu_wq: flags=0x8 [ 7025.709624] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7025.713905] pending: vmstat_update, lru_add_drain_per_cpu [ 7025.717069] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7025.721869] pending: vmstat_update, lru_add_drain_per_cpu [ 7025.725179] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7025.729316] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7025.733149] workqueue xprtiod: flags=0xa [ 7025.735024] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7025.738476] pending: xprt_autoclose, xs_error_handle [ 7025.741523] workqueue ipv6_addrconf: flags=0x40008 [ 7025.744016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7025.747915] pending: addrconf_verify_work [ 7025.749745] workqueue knbd0-recv: flags=0x1a [ 7025.751880] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7025.755046] in-flight: 651:recv_work [ 7025.757141] workqueue ptlrpc_pinger: flags=0x2 [ 7025.759611] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7025.762921] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7025.765978] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4388s workers=4 idle: 15979 31824 8 [ 7025.770238] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7016s workers=2 idle: 654 [ 7040.057022] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7040.057596] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7040.058721] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7040.058844] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7040.058850] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7040.058853] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7040.058867] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7040.058871] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7040.058874] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7040.058876] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7040.058878] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7040.058879] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7040.058881] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7040.058884] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7040.058886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7040.058888] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7040.058895] Call Trace: [ 7040.058900] [ 7040.058907] ? show_regs.cold.9+0x22/0x2f [ 7040.058917] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7040.058922] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7040.058925] ? watchdog+0x40/0x40 [ 7040.058928] ? hrtimer_interrupt+0x128/0x2c0 [ 7040.058930] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7040.058936] ? apic_timer_interrupt+0xf/0x20 [ 7040.058940] [ 7040.058941] ? native_safe_halt+0x2e/0x50 [ 7040.058944] kvm_wait+0xcd/0xe0 [ 7040.058951] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7040.058957] do_raw_spin_lock+0x177/0x190 [ 7040.058960] _raw_spin_lock+0x21/0x30 [ 7040.058964] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7040.059193] obd_mod_free+0x46/0x140 [ptlrpc] [ 7040.059329] mdc_commit_open+0x91/0xb0 [mdc] [ 7040.059363] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7040.059485] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7040.059609] after_reply+0xe2c/0x1780 [ptlrpc] [ 7040.059822] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7040.059977] ? __next_timer_interrupt+0x160/0x160 [ 7040.059987] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7040.060128] ? do_wait_intr+0xf0/0xf0 [ 7040.060135] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7040.060283] kthread+0x1d1/0x200 [ 7040.060289] ? set_kthread_struct+0x70/0x70 [ 7040.060291] ret_from_fork+0x1f/0x30 [ 7040.415517] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7040.486029] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7040.498095] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7040.507842] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7040.515999] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7040.537840] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7040.555199] RAX: 0000000000007a38 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7040.561944] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7040.574016] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7040.578793] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7040.599339] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7040.607951] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7040.620728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7040.631750] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7040.637315] Call Trace: [ 7040.653947] [ 7040.654615] ? show_regs.cold.9+0x22/0x2f [ 7040.656036] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7040.678097] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7040.684915] ? watchdog+0x40/0x40 [ 7040.693802] ? hrtimer_interrupt+0x128/0x2c0 [ 7040.705541] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7040.711802] ? apic_timer_interrupt+0xf/0x20 [ 7040.720289] [ 7040.721015] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7040.733651] do_raw_spin_lock+0x177/0x190 [ 7040.735050] _raw_spin_lock+0x21/0x30 [ 7040.742629] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7040.753009] ? do_raw_spin_unlock+0x75/0x190 [ 7040.758498] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7040.771876] ? do_wait_intr+0xf0/0xf0 [ 7040.778664] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7040.788960] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7040.794930] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7040.807769] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7040.816771] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7040.826347] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7040.835605] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7040.842066] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7040.851573] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7040.862595] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7040.877068] lmv_intent_open+0x338/0x1150 [lmv] [ 7040.881699] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7040.891765] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7040.903338] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7040.908395] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7040.917759] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7040.926836] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7040.932610] ? do_raw_spin_unlock+0x75/0x190 [ 7040.946524] ? _raw_spin_unlock+0x12/0x30 [ 7040.960986] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7040.973845] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7040.978370] ll_atomic_open+0x36a/0xd70 [lustre] [ 7040.981295] lookup_open+0x356/0xb40 [ 7040.984341] do_last+0x313/0xfc0 [ 7040.987597] ? nd_jump_root+0xe5/0x160 [ 7040.989446] ? path_init+0x437/0x520 [ 7040.992209] path_openat+0xf7/0x500 [ 7040.995960] ? do_raw_spin_unlock+0x75/0x190 [ 7041.004267] do_filp_open+0x99/0x140 [ 7041.005516] ? getname_flags+0x6e/0x330 [ 7041.024396] ? __check_object_size+0xff/0x256 [ 7041.033645] ? do_raw_spin_unlock+0x75/0x190 [ 7041.041990] ? _raw_spin_unlock+0x12/0x30 [ 7041.050153] do_sys_openat2+0x2b4/0x410 [ 7041.056736] do_sys_open+0x73/0xa0 [ 7041.058329] __x64_sys_openat+0x24/0x30 [ 7041.060826] do_syscall_64+0xc1/0x440 [ 7041.067507] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7041.069309] RIP: 0033:0x7f53425a5332 [ 7041.081260] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7041.093408] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7041.105462] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7041.115772] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7041.129154] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7041.135313] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7041.141797] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7048.062758] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 7048.064734] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7048.096227] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7048.107985] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7048.117525] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7048.125572] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7048.142681] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7048.145678] RAX: 0000000000007612 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7048.152610] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7048.154858] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7048.157717] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7048.169602] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7048.183557] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7048.185755] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7048.197377] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7048.199342] Call Trace: [ 7048.200003] [ 7048.211294] ? show_regs.cold.9+0x22/0x2f [ 7048.213594] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7048.220155] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7048.223718] ? watchdog+0x40/0x40 [ 7048.225538] ? hrtimer_interrupt+0x128/0x2c0 [ 7048.226839] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7048.229032] ? apic_timer_interrupt+0xf/0x20 [ 7048.234431] [ 7048.234856] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7048.240519] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7048.252851] do_raw_spin_lock+0x177/0x190 [ 7048.255487] _raw_spin_lock+0x21/0x30 [ 7048.256436] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7048.259598] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7048.264773] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7048.270788] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7048.277152] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7048.281823] ? woken_wake_function+0x30/0x30 [ 7048.283605] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7048.285800] kthread+0x1d1/0x200 [ 7048.286997] ? set_kthread_struct+0x70/0x70 [ 7048.288595] ret_from_fork+0x1f/0x30 [ 7052.056119] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 7052.064341] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7052.147557] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7052.170447] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7052.193416] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7052.199497] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7052.214354] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7052.241275] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7052.247997] RAX: 0000000000002e0f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7052.260761] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7052.266326] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7052.274403] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7052.277495] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7052.289414] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7052.301442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7052.306439] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7052.312518] Call Trace: [ 7052.320371] [ 7052.326970] ? show_regs.cold.9+0x22/0x2f [ 7052.339387] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7052.346849] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7052.360545] ? watchdog+0x40/0x40 [ 7052.365883] ? hrtimer_interrupt+0x128/0x2c0 [ 7052.374207] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7052.378543] ? apic_timer_interrupt+0xf/0x20 [ 7052.395260] [ 7052.400454] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7052.412287] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7052.429192] ? newidle_balance.isra.93+0x207/0x580 [ 7052.440471] do_raw_spin_lock+0x177/0x190 [ 7052.454870] _raw_spin_lock+0x21/0x30 [ 7052.463278] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7052.468428] process_one_work+0x2c8/0x700 [ 7052.469645] worker_thread+0x296/0x6e0 [ 7052.479487] ? rescuer_thread+0x570/0x570 [ 7052.492619] kthread+0x1d1/0x200 [ 7052.499036] ? set_kthread_struct+0x70/0x70 [ 7052.507651] ret_from_fork+0x1f/0x30 [ 7055.841842] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4418s! [ 7055.855199] Showing busy workqueues and worker pools: [ 7055.859407] workqueue events: flags=0x0 [ 7055.865296] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7055.869852] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7055.894256] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7055.906114] pending: cpu_latency_work [ptlrpc] [ 7055.916486] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7055.932738] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7055.953105] workqueue events_long: flags=0x0 [ 7055.957971] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7055.966130] pending: lu_objects_flush [obdclass] [ 7055.973876] workqueue events_unbound: flags=0x2 [ 7055.985789] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7055.993384] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7056.001646] workqueue events_power_efficient: flags=0x80 [ 7056.007552] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7056.021181] pending: check_lifetime, do_cache_clean [ 7056.027909] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7056.041992] pending: neigh_periodic_work, neigh_periodic_work [ 7056.050643] workqueue mm_percpu_wq: flags=0x8 [ 7056.055183] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7056.065663] pending: vmstat_update, lru_add_drain_per_cpu [ 7056.072544] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7056.079544] pending: vmstat_update, lru_add_drain_per_cpu [ 7056.083698] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7056.090374] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7056.096666] workqueue xprtiod: flags=0xa [ 7056.098077] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7056.101615] pending: xprt_autoclose, xs_error_handle [ 7056.104221] workqueue ipv6_addrconf: flags=0x40008 [ 7056.106867] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7056.109628] pending: addrconf_verify_work [ 7056.111648] workqueue knbd0-recv: flags=0x1a [ 7056.113873] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7056.118701] in-flight: 651:recv_work [ 7056.121246] workqueue ptlrpc_pinger: flags=0x2 [ 7056.123576] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7056.126339] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7056.129868] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4419s workers=4 idle: 15979 31824 8 [ 7056.135369] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7046s workers=2 idle: 654 [ 7068.055059] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7068.057985] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7068.058675] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7068.075713] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7068.075726] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7068.075728] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7068.075741] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7068.075744] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7068.075761] RAX: 00000000000053c9 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7068.075764] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7068.075766] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7068.075767] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7068.075770] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7068.075773] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7068.075776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7068.075778] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7068.075787] Call Trace: [ 7068.075792] [ 7068.075798] ? show_regs.cold.9+0x22/0x2f [ 7068.075808] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7068.075814] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7068.075818] ? watchdog+0x40/0x40 [ 7068.075822] ? hrtimer_interrupt+0x128/0x2c0 [ 7068.075825] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7068.075832] ? apic_timer_interrupt+0xf/0x20 [ 7068.075836] [ 7068.075839] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7068.075846] do_raw_spin_lock+0x177/0x190 [ 7068.075850] _raw_spin_lock+0x21/0x30 [ 7068.075854] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7068.077212] ? do_raw_spin_unlock+0x75/0x190 [ 7068.077221] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7068.077428] ? do_wait_intr+0xf0/0xf0 [ 7068.077438] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7068.077631] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7068.077844] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7068.078348] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7068.078462] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7068.078503] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7068.078528] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7068.078552] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7068.078631] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7068.078859] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7068.078885] lmv_intent_open+0x338/0x1150 [lmv] [ 7068.078912] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7068.079003] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7068.079235] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7068.079252] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7068.079308] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7068.079385] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7068.079440] ? do_raw_spin_unlock+0x75/0x190 [ 7068.079446] ? _raw_spin_unlock+0x12/0x30 [ 7068.079453] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7068.079478] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7068.079499] ll_atomic_open+0x36a/0xd70 [lustre] [ 7068.079583] lookup_open+0x356/0xb40 [ 7068.079592] do_last+0x313/0xfc0 [ 7068.079598] ? nd_jump_root+0xe5/0x160 [ 7068.079602] ? path_init+0x437/0x520 [ 7068.079607] path_openat+0xf7/0x500 [ 7068.079612] ? do_raw_spin_unlock+0x75/0x190 [ 7068.079617] do_filp_open+0x99/0x140 [ 7068.079624] ? getname_flags+0x6e/0x330 [ 7068.079628] ? __check_object_size+0xff/0x256 [ 7068.079636] ? do_raw_spin_unlock+0x75/0x190 [ 7068.079640] ? _raw_spin_unlock+0x12/0x30 [ 7068.079644] do_sys_openat2+0x2b4/0x410 [ 7068.079649] do_sys_open+0x73/0xa0 [ 7068.079653] __x64_sys_openat+0x24/0x30 [ 7068.079657] do_syscall_64+0xc1/0x440 [ 7068.079664] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7068.079670] RIP: 0033:0x7f53425a5332 [ 7068.079677] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7068.079682] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7068.079687] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7068.079690] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7068.079693] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7068.079695] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7068.079698] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7068.433134] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7068.474378] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7068.492728] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7068.510258] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7068.524262] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7068.545218] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7068.549237] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7068.561138] RDX: 0000000000006f66 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7068.569888] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7068.577215] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7068.585114] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7068.590173] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7068.592792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7068.596975] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7068.600437] Call Trace: [ 7068.601961] [ 7068.602894] ? show_regs.cold.9+0x22/0x2f [ 7068.607627] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7068.609132] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7068.614102] ? watchdog+0x40/0x40 [ 7068.616131] ? hrtimer_interrupt+0x128/0x2c0 [ 7068.619358] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7068.624657] ? apic_timer_interrupt+0xf/0x20 [ 7068.626700] [ 7068.627392] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7068.629876] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7068.631863] do_raw_spin_lock+0x177/0x190 [ 7068.640241] _raw_spin_lock+0x21/0x30 [ 7068.642286] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7068.645932] obd_mod_free+0x46/0x140 [ptlrpc] [ 7068.648139] mdc_commit_open+0x91/0xb0 [mdc] [ 7068.659083] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7068.661198] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7068.663725] after_reply+0xe2c/0x1780 [ptlrpc] [ 7068.665631] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7068.671415] ? __next_timer_interrupt+0x160/0x160 [ 7068.677707] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7068.679171] ? do_wait_intr+0xf0/0xf0 [ 7068.684317] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7068.688053] kthread+0x1d1/0x200 [ 7068.689009] ? set_kthread_struct+0x70/0x70 [ 7068.690455] ret_from_fork+0x1f/0x30 [ 7076.059718] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 7076.068127] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7076.106428] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7076.113134] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7076.117694] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7076.124570] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7076.141411] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7076.146893] RAX: 0000000000007587 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7076.150858] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7076.155099] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7076.159076] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7076.165042] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7076.171102] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7076.181675] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7076.183600] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7076.199234] Call Trace: [ 7076.200584] [ 7076.202546] ? show_regs.cold.9+0x22/0x2f [ 7076.204055] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7076.210072] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7076.213371] ? watchdog+0x40/0x40 [ 7076.215225] ? hrtimer_interrupt+0x128/0x2c0 [ 7076.221150] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7076.224302] ? apic_timer_interrupt+0xf/0x20 [ 7076.226690] [ 7076.228078] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7076.232107] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7076.236339] do_raw_spin_lock+0x177/0x190 [ 7076.241165] _raw_spin_lock+0x21/0x30 [ 7076.245507] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7076.255214] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7076.260300] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7076.264117] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7076.266784] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7076.274671] ? woken_wake_function+0x30/0x30 [ 7076.282296] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7076.292131] kthread+0x1d1/0x200 [ 7076.293384] ? set_kthread_struct+0x70/0x70 [ 7076.300238] ret_from_fork+0x1f/0x30 [ 7084.061289] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 7084.077699] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7084.152024] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7084.173057] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7084.192379] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7084.203351] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7084.208129] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7084.223107] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7084.233760] RAX: 0000000000007e98 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7084.240600] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7084.245447] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7084.253423] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7084.262924] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7084.270301] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7084.276023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7084.279893] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7084.286913] Call Trace: [ 7084.291471] [ 7084.294370] ? show_regs.cold.9+0x22/0x2f [ 7084.299696] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7084.304592] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7084.312551] ? watchdog+0x40/0x40 [ 7084.315554] ? hrtimer_interrupt+0x128/0x2c0 [ 7084.317021] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7084.329367] ? apic_timer_interrupt+0xf/0x20 [ 7084.338950] [ 7084.345917] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7084.356995] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7084.358844] ? newidle_balance.isra.93+0x207/0x580 [ 7084.365447] do_raw_spin_lock+0x177/0x190 [ 7084.369724] _raw_spin_lock+0x21/0x30 [ 7084.371377] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7084.373454] process_one_work+0x2c8/0x700 [ 7084.374927] worker_thread+0x296/0x6e0 [ 7084.376334] ? rescuer_thread+0x570/0x570 [ 7084.377679] kthread+0x1d1/0x200 [ 7084.380015] ? set_kthread_struct+0x70/0x70 [ 7084.381633] ret_from_fork+0x1f/0x30 [ 7086.565581] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4449s! [ 7086.573567] Showing busy workqueues and worker pools: [ 7086.582771] workqueue events: flags=0x0 [ 7086.590731] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7086.599723] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7086.614180] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7086.617764] pending: cpu_latency_work [ptlrpc] [ 7086.622736] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7086.631369] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7086.639809] workqueue events_long: flags=0x0 [ 7086.642393] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7086.646850] pending: lu_objects_flush [obdclass] [ 7086.649747] workqueue events_unbound: flags=0x2 [ 7086.651733] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7086.655029] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7086.657738] workqueue events_power_efficient: flags=0x80 [ 7086.659877] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7086.662820] pending: check_lifetime, do_cache_clean [ 7086.669331] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7086.676902] pending: neigh_periodic_work, neigh_periodic_work [ 7086.682016] workqueue mm_percpu_wq: flags=0x8 [ 7086.688259] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7086.693668] pending: vmstat_update, lru_add_drain_per_cpu [ 7086.697647] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7086.704324] pending: vmstat_update, lru_add_drain_per_cpu [ 7086.710779] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7086.715771] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7086.724114] workqueue xprtiod: flags=0xa [ 7086.725824] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7086.732574] pending: xprt_autoclose, xs_error_handle [ 7086.738800] workqueue ipv6_addrconf: flags=0x40008 [ 7086.742741] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7086.748651] pending: addrconf_verify_work [ 7086.753648] workqueue knbd0-recv: flags=0x1a [ 7086.757175] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7086.766312] in-flight: 651:recv_work [ 7086.772961] workqueue ptlrpc_pinger: flags=0x2 [ 7086.780372] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7086.791216] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7086.797952] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4449s workers=4 idle: 15979 31824 8 [ 7086.807981] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7077s workers=2 idle: 654 [ 7096.055013] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7096.057507] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7096.058722] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7096.058789] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7096.058795] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7096.058798] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7096.058811] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7096.058816] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7096.058820] RAX: 0000000000004cab RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7096.058831] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7096.058834] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7096.058836] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7096.058838] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7096.058842] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7096.058845] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7096.058848] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7096.058856] Call Trace: [ 7096.058861] [ 7096.058866] ? show_regs.cold.9+0x22/0x2f [ 7096.058876] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7096.058893] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7096.058899] ? watchdog+0x40/0x40 [ 7096.058903] ? hrtimer_interrupt+0x128/0x2c0 [ 7096.058907] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7096.058915] ? apic_timer_interrupt+0xf/0x20 [ 7096.058919] [ 7096.058922] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7096.058929] do_raw_spin_lock+0x177/0x190 [ 7096.058934] _raw_spin_lock+0x21/0x30 [ 7096.058938] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7096.059245] ? do_raw_spin_unlock+0x75/0x190 [ 7096.059252] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7096.059440] ? do_wait_intr+0xf0/0xf0 [ 7096.059447] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7096.059580] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7096.059751] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7096.059909] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7096.060002] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7096.060031] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7096.060050] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7096.060067] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7096.060122] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7096.066564] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata [ 7096.118742] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7096.126394] dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7096.126418] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7096.126426] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7096.126429] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7096.126445] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7096.126449] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7096.126453] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7096.126456] RDX: 0000000000004027 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7096.126458] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7096.126460] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7096.126462] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7096.126465] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7096.126468] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7096.126470] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7096.126479] Call Trace: [ 7096.126484] [ 7096.126489] ? show_regs.cold.9+0x22/0x2f [ 7096.126499] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7096.126505] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7096.126515] ? watchdog+0x40/0x40 [ 7096.126519] ? hrtimer_interrupt+0x128/0x2c0 [ 7096.126523] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7096.126531] ? apic_timer_interrupt+0xf/0x20 [ 7096.126536] [ 7096.126538] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7096.126543] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7096.126551] do_raw_spin_lock+0x177/0x190 [ 7096.126556] _raw_spin_lock+0x21/0x30 [ 7096.126566] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7096.133547] lmv_intent_open+0x338/0x1150 [lmv] [ 7096.136744] obd_mod_free+0x46/0x140 [ptlrpc] [ 7096.147306] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7096.151418] mdc_commit_open+0x91/0xb0 [mdc] [ 7096.160082] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7096.164092] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7096.174153] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7096.174210] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7096.179011] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7096.179558] after_reply+0xe2c/0x1780 [ptlrpc] [ 7096.179791] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7096.180863] ? __next_timer_interrupt+0x160/0x160 [ 7096.180875] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7096.190859] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7096.190984] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7096.191077] ? do_raw_spin_unlock+0x75/0x190 [ 7096.191085] ? _raw_spin_unlock+0x12/0x30 [ 7096.191093] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7096.191125] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7096.191160] ll_atomic_open+0x36a/0xd70 [lustre] [ 7096.191244] lookup_open+0x356/0xb40 [ 7096.191264] do_last+0x313/0xfc0 [ 7096.195179] ? do_wait_intr+0xf0/0xf0 [ 7096.195323] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7096.195593] kthread+0x1d1/0x200 [ 7096.195600] ? set_kthread_struct+0x70/0x70 [ 7096.195604] ret_from_fork+0x1f/0x30 [ 7096.411974] ? nd_jump_root+0xe5/0x160 [ 7096.413768] ? path_init+0x437/0x520 [ 7096.416337] path_openat+0xf7/0x500 [ 7096.417781] ? do_raw_spin_unlock+0x75/0x190 [ 7096.419914] do_filp_open+0x99/0x140 [ 7096.421700] ? getname_flags+0x6e/0x330 [ 7096.423380] ? __check_object_size+0xff/0x256 [ 7096.425482] ? do_raw_spin_unlock+0x75/0x190 [ 7096.427601] ? _raw_spin_unlock+0x12/0x30 [ 7096.429401] do_sys_openat2+0x2b4/0x410 [ 7096.430795] do_sys_open+0x73/0xa0 [ 7096.432233] __x64_sys_openat+0x24/0x30 [ 7096.433742] do_syscall_64+0xc1/0x440 [ 7096.435371] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7096.438712] RIP: 0033:0x7f53425a5332 [ 7096.441638] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7096.452698] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7096.456589] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7096.459863] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7096.463402] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7096.466906] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7096.470904] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7104.063252] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 7104.071001] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7104.109029] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7104.117628] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7104.126036] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7104.128716] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7104.137999] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7104.150624] RAX: 00000000000064f9 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7104.154028] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7104.165985] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7104.172934] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7104.178508] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7104.181944] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7104.191107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7104.195766] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7104.200338] Call Trace: [ 7104.203884] [ 7104.204694] ? show_regs.cold.9+0x22/0x2f [ 7104.206609] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7104.208876] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7104.211221] ? watchdog+0x40/0x40 [ 7104.212963] ? hrtimer_interrupt+0x128/0x2c0 [ 7104.214402] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7104.216266] ? apic_timer_interrupt+0xf/0x20 [ 7104.217944] [ 7104.218982] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7104.223033] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7104.227677] do_raw_spin_lock+0x177/0x190 [ 7104.232395] _raw_spin_lock+0x21/0x30 [ 7104.234039] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7104.240923] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7104.252955] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7104.255012] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7104.267045] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7104.269095] ? woken_wake_function+0x30/0x30 [ 7104.273723] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7104.281794] kthread+0x1d1/0x200 [ 7104.283091] ? set_kthread_struct+0x70/0x70 [ 7104.288269] ret_from_fork+0x1f/0x30 [ 7112.054017] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 7112.058316] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7112.099614] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7112.106344] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7112.109586] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7112.111514] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7112.113413] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7112.120218] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7112.124318] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7112.127363] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7112.133771] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7112.136906] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7112.145602] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7112.151216] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7112.161199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7112.164120] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7112.169821] Call Trace: [ 7112.170619] [ 7112.175215] ? show_regs.cold.9+0x22/0x2f [ 7112.177630] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7112.180060] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7112.184946] ? watchdog+0x40/0x40 [ 7112.187282] ? hrtimer_interrupt+0x128/0x2c0 [ 7112.189534] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7112.191833] ? apic_timer_interrupt+0xf/0x20 [ 7112.193829] [ 7112.194750] ? native_safe_halt+0x2e/0x50 [ 7112.196923] kvm_wait+0xcd/0xe0 [ 7112.200298] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7112.205031] ? newidle_balance.isra.93+0x207/0x580 [ 7112.211658] do_raw_spin_lock+0x177/0x190 [ 7112.215047] _raw_spin_lock+0x21/0x30 [ 7112.218440] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7112.223270] process_one_work+0x2c8/0x700 [ 7112.227489] worker_thread+0x296/0x6e0 [ 7112.230893] ? rescuer_thread+0x570/0x570 [ 7112.234441] kthread+0x1d1/0x200 [ 7112.241740] ? set_kthread_struct+0x70/0x70 [ 7112.246899] ret_from_fork+0x1f/0x30 [ 7117.290138] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4480s! [ 7117.294163] Showing busy workqueues and worker pools: [ 7117.296764] workqueue events: flags=0x0 [ 7117.298584] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7117.302219] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7117.309282] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7117.312184] pending: cpu_latency_work [ptlrpc] [ 7117.317943] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7117.322004] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7117.326584] workqueue events_long: flags=0x0 [ 7117.328925] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7117.332112] pending: lu_objects_flush [obdclass] [ 7117.337428] workqueue events_unbound: flags=0x2 [ 7117.339768] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7117.344409] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7117.347724] workqueue events_power_efficient: flags=0x80 [ 7117.349849] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7117.353325] pending: check_lifetime, do_cache_clean [ 7117.355649] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7117.358746] pending: neigh_periodic_work, neigh_periodic_work [ 7117.360807] workqueue mm_percpu_wq: flags=0x8 [ 7117.362657] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7117.365484] pending: vmstat_update, lru_add_drain_per_cpu [ 7117.367893] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7117.370129] pending: vmstat_update, lru_add_drain_per_cpu [ 7117.372519] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7117.375418] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7117.378302] workqueue xprtiod: flags=0xa [ 7117.379523] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7117.381753] pending: xprt_autoclose, xs_error_handle [ 7117.384104] workqueue ipv6_addrconf: flags=0x40008 [ 7117.388322] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7117.391369] pending: addrconf_verify_work [ 7117.393479] workqueue knbd0-recv: flags=0x1a [ 7117.395337] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7117.397512] in-flight: 651:recv_work [ 7117.399375] workqueue ptlrpc_pinger: flags=0x2 [ 7117.401342] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7117.403592] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7117.406321] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4480s workers=4 idle: 15979 31824 8 [ 7117.409181] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7107s workers=2 idle: 654 [ 7124.055164] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7124.059249] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7124.059347] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr [ 7124.067935] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7124.068012] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7124.068019] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7124.068022] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7124.068037] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7124.068041] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7124.068046] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7124.068049] RDX: 0000000000007f08 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7124.068051] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7124.068054] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7124.068056] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7124.068059] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7124.068063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7124.068065] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7124.068075] Call Trace: [ 7124.068081] [ 7124.068087] ? show_regs.cold.9+0x22/0x2f [ 7124.068097] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7124.068105] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7124.068110] ? watchdog+0x40/0x40 [ 7124.068114] ? hrtimer_interrupt+0x128/0x2c0 [ 7124.068118] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7124.068126] ? apic_timer_interrupt+0xf/0x20 [ 7124.068130] [ 7124.068133] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7124.068139] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7124.068144] do_raw_spin_lock+0x177/0x190 [ 7124.068149] _raw_spin_lock+0x21/0x30 [ 7124.068156] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7124.083474] intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7124.083512] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7124.083520] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7124.083523] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7124.083536] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7124.083540] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7124.083615] RAX: 0000000000006087 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7124.083618] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7124.083621] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7124.083623] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7124.083625] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7124.083629] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7124.083632] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7124.083634] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7124.083642] Call Trace: [ 7124.083647] [ 7124.083653] ? show_regs.cold.9+0x22/0x2f [ 7124.083664] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7124.083670] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7124.083674] ? watchdog+0x40/0x40 [ 7124.083678] ? hrtimer_interrupt+0x128/0x2c0 [ 7124.083682] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7124.083692] ? apic_timer_interrupt+0xf/0x20 [ 7124.083697] [ 7124.083700] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7124.083708] do_raw_spin_lock+0x177/0x190 [ 7124.083713] _raw_spin_lock+0x21/0x30 [ 7124.083718] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7124.084184] ? do_raw_spin_unlock+0x75/0x190 [ 7124.084191] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7124.084388] ? do_wait_intr+0xf0/0xf0 [ 7124.084476] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7124.084885] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7124.085336] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7124.085781] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7124.085890] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7124.085932] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7124.085961] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7124.086054] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7124.086211] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7124.086502] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7124.086533] lmv_intent_open+0x338/0x1150 [lmv] [ 7124.086705] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7124.086808] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7124.134835] obd_mod_free+0x46/0x140 [ptlrpc] [ 7124.135097] mdc_commit_open+0x91/0xb0 [mdc] [ 7124.135134] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7124.135308] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7124.135502] after_reply+0xe2c/0x1780 [ptlrpc] [ 7124.135732] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7124.135973] ? __next_timer_interrupt+0x160/0x160 [ 7124.135984] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7124.136184] ? do_wait_intr+0xf0/0xf0 [ 7124.136193] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7124.136464] kthread+0x1d1/0x200 [ 7124.136470] ? set_kthread_struct+0x70/0x70 [ 7124.136474] ret_from_fork+0x1f/0x30 [ 7124.449298] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7124.451210] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7124.453436] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7124.456063] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7124.458108] ? do_raw_spin_unlock+0x75/0x190 [ 7124.459670] ? _raw_spin_unlock+0x12/0x30 [ 7124.461336] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7124.463652] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7124.466425] ll_atomic_open+0x36a/0xd70 [lustre] [ 7124.468450] lookup_open+0x356/0xb40 [ 7124.469752] do_last+0x313/0xfc0 [ 7124.471486] ? nd_jump_root+0xe5/0x160 [ 7124.472571] ? path_init+0x437/0x520 [ 7124.473513] path_openat+0xf7/0x500 [ 7124.474436] ? do_raw_spin_unlock+0x75/0x190 [ 7124.475639] do_filp_open+0x99/0x140 [ 7124.476628] ? getname_flags+0x6e/0x330 [ 7124.477687] ? __check_object_size+0xff/0x256 [ 7124.479073] ? do_raw_spin_unlock+0x75/0x190 [ 7124.480086] ? _raw_spin_unlock+0x12/0x30 [ 7124.481052] do_sys_openat2+0x2b4/0x410 [ 7124.482078] do_sys_open+0x73/0xa0 [ 7124.482953] __x64_sys_openat+0x24/0x30 [ 7124.483871] do_syscall_64+0xc1/0x440 [ 7124.484733] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7124.486238] RIP: 0033:0x7f53425a5332 [ 7124.487157] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7124.491993] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7124.494718] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7124.497702] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7124.499522] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7124.501334] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7124.502962] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7132.058013] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 7132.062369] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7132.085890] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7132.091368] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7132.096445] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7132.105050] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7132.116239] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7132.121472] RAX: 00000000000030bb RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7132.125606] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7132.129435] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7132.131901] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7132.137133] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7132.140528] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7132.145188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7132.147966] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7132.152833] Call Trace: [ 7132.154159] [ 7132.155018] ? show_regs.cold.9+0x22/0x2f [ 7132.157100] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7132.159276] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7132.162274] ? watchdog+0x40/0x40 [ 7132.164019] ? hrtimer_interrupt+0x128/0x2c0 [ 7132.166249] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7132.168224] ? apic_timer_interrupt+0xf/0x20 [ 7132.170382] [ 7132.171315] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7132.175057] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7132.178084] do_raw_spin_lock+0x177/0x190 [ 7132.179987] _raw_spin_lock+0x21/0x30 [ 7132.181894] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7132.185275] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7132.188195] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7132.191130] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7132.193953] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7132.196538] ? woken_wake_function+0x30/0x30 [ 7132.199231] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7132.202592] kthread+0x1d1/0x200 [ 7132.204374] ? set_kthread_struct+0x70/0x70 [ 7132.206422] ret_from_fork+0x1f/0x30 [ 7144.052220] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 7144.057577] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7144.094697] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7144.099395] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7144.103579] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7144.111024] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7144.117907] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7144.131091] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7144.140549] RAX: 0000000000001a90 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7144.142574] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7144.152754] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7144.157315] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7144.162634] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7144.168237] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7144.176988] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7144.182774] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7144.198924] Call Trace: [ 7144.200493] [ 7144.201221] ? show_regs.cold.9+0x22/0x2f [ 7144.208762] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7144.211588] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7144.213845] ? watchdog+0x40/0x40 [ 7144.219210] ? hrtimer_interrupt+0x128/0x2c0 [ 7144.221562] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7144.225027] ? apic_timer_interrupt+0xf/0x20 [ 7144.227011] [ 7144.227583] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7144.233776] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7144.235949] ? newidle_balance.isra.93+0x207/0x580 [ 7144.241777] do_raw_spin_lock+0x177/0x190 [ 7144.242985] _raw_spin_lock+0x21/0x30 [ 7144.248698] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7144.252707] process_one_work+0x2c8/0x700 [ 7144.254720] worker_thread+0x296/0x6e0 [ 7144.256345] ? rescuer_thread+0x570/0x570 [ 7144.258675] kthread+0x1d1/0x200 [ 7144.264303] ? set_kthread_struct+0x70/0x70 [ 7144.266487] ret_from_fork+0x1f/0x30 [ 7147.488048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4510s! [ 7147.490863] Showing busy workqueues and worker pools: [ 7147.492134] workqueue events: flags=0x0 [ 7147.493968] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7147.498210] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7147.503970] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7147.507550] pending: cpu_latency_work [ptlrpc] [ 7147.510704] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7147.516539] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7147.522789] workqueue events_long: flags=0x0 [ 7147.527374] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7147.539938] pending: lu_objects_flush [obdclass] [ 7147.555986] workqueue events_unbound: flags=0x2 [ 7147.563622] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7147.570479] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7147.578244] workqueue events_power_efficient: flags=0x80 [ 7147.581701] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7147.585900] pending: check_lifetime, do_cache_clean [ 7147.587385] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7147.589332] pending: neigh_periodic_work, neigh_periodic_work [ 7147.597268] workqueue mm_percpu_wq: flags=0x8 [ 7147.602315] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7147.606366] pending: vmstat_update, lru_add_drain_per_cpu [ 7147.612115] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7147.617258] pending: vmstat_update, lru_add_drain_per_cpu [ 7147.620020] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7147.622489] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7147.625067] workqueue xprtiod: flags=0xa [ 7147.626527] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7147.630275] pending: xprt_autoclose, xs_error_handle [ 7147.633782] workqueue ipv6_addrconf: flags=0x40008 [ 7147.635965] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7147.640107] pending: addrconf_verify_work [ 7147.642012] workqueue knbd0-recv: flags=0x1a [ 7147.643792] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7147.648672] in-flight: 651:recv_work [ 7147.650254] workqueue ptlrpc_pinger: flags=0x2 [ 7147.656464] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7147.659473] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7147.662599] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4510s workers=4 idle: 15979 31824 8 [ 7147.667198] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7138s workers=2 idle: 654 [ 7152.055090] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7152.057239] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7152.058899] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7152.058977] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7152.058984] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7152.058987] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7152.059002] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7152.059006] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7152.059011] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7152.059131] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7152.059134] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7152.059136] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7152.059139] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7152.059142] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7152.059145] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7152.059148] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7152.059156] Call Trace: [ 7152.059161] [ 7152.059167] ? show_regs.cold.9+0x22/0x2f [ 7152.059177] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7152.070505] Modules linked in: lustre(O) [ 7152.115889] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7152.115912] ? watchdog+0x40/0x40 [ 7152.115920] ? hrtimer_interrupt+0x128/0x2c0 [ 7152.115924] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7152.115932] ? apic_timer_interrupt+0xf/0x20 [ 7152.115938] [ 7152.115943] ? native_safe_halt+0x2e/0x50 [ 7152.115947] kvm_wait+0xcd/0xe0 [ 7152.115957] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7152.115967] do_raw_spin_lock+0x177/0x190 [ 7152.115972] _raw_spin_lock+0x21/0x30 [ 7152.115976] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7152.116241] ? do_raw_spin_unlock+0x75/0x190 [ 7152.116247] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7152.116542] ? do_wait_intr+0xf0/0xf0 [ 7152.116552] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7152.116759] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7152.116984] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7152.125022] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7152.125101] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7152.125108] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7152.125111] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7152.125124] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7152.125128] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7152.125132] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7152.125135] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7152.125137] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7152.125139] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7152.125142] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7152.125145] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7152.125147] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7152.125149] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7152.125158] Call Trace: [ 7152.125164] [ 7152.125170] ? show_regs.cold.9+0x22/0x2f [ 7152.125180] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7152.125186] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7152.125191] ? watchdog+0x40/0x40 [ 7152.125195] ? hrtimer_interrupt+0x128/0x2c0 [ 7152.125200] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7152.125207] ? apic_timer_interrupt+0xf/0x20 [ 7152.125212] [ 7152.125215] ? native_safe_halt+0x2e/0x50 [ 7152.125219] kvm_wait+0xcd/0xe0 [ 7152.125226] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7152.125234] do_raw_spin_lock+0x177/0x190 [ 7152.125238] _raw_spin_lock+0x21/0x30 [ 7152.125244] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7152.125528] obd_mod_free+0x46/0x140 [ptlrpc] [ 7152.125737] mdc_commit_open+0x91/0xb0 [mdc] [ 7152.125776] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7152.125979] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7152.126182] after_reply+0xe2c/0x1780 [ptlrpc] [ 7152.127219] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7152.140583] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7152.143254] ? __next_timer_interrupt+0x160/0x160 [ 7152.143335] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7152.166331] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7152.171083] ? do_wait_intr+0xf0/0xf0 [ 7152.176697] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7152.176753] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7152.181314] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7152.185045] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7152.185167] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7152.199231] kthread+0x1d1/0x200 [ 7152.202557] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7152.207405] ? set_kthread_struct+0x70/0x70 [ 7152.207425] ret_from_fork+0x1f/0x30 [ 7152.384854] lmv_intent_open+0x338/0x1150 [lmv] [ 7152.386978] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7152.389470] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7152.392754] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7152.395265] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7152.400150] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7152.402460] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7152.404068] ? do_raw_spin_unlock+0x75/0x190 [ 7152.406033] ? _raw_spin_unlock+0x12/0x30 [ 7152.407576] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7152.413389] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7152.416238] ll_atomic_open+0x36a/0xd70 [lustre] [ 7152.417774] lookup_open+0x356/0xb40 [ 7152.419070] do_last+0x313/0xfc0 [ 7152.420578] ? nd_jump_root+0xe5/0x160 [ 7152.421711] ? path_init+0x437/0x520 [ 7152.422687] path_openat+0xf7/0x500 [ 7152.423650] ? do_raw_spin_unlock+0x75/0x190 [ 7152.424917] do_filp_open+0x99/0x140 [ 7152.425986] ? getname_flags+0x6e/0x330 [ 7152.427226] ? __check_object_size+0xff/0x256 [ 7152.428527] ? do_raw_spin_unlock+0x75/0x190 [ 7152.429656] ? _raw_spin_unlock+0x12/0x30 [ 7152.431002] do_sys_openat2+0x2b4/0x410 [ 7152.432651] do_sys_open+0x73/0xa0 [ 7152.433699] __x64_sys_openat+0x24/0x30 [ 7152.434715] do_syscall_64+0xc1/0x440 [ 7152.435633] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7152.437257] RIP: 0033:0x7f53425a5332 [ 7152.438411] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7152.444589] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7152.446698] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7152.448883] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7152.450934] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7152.453258] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7152.455094] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7160.065287] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 7160.077597] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7160.133403] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7160.145511] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7160.158224] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30a/0x6d0 [ 7160.161949] Code: 03 00 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 [ 7160.178668] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7160.188899] RAX: 0000000000003cc1 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7160.195604] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7160.206417] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7160.214160] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7160.225425] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7160.234209] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7160.243283] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7160.257214] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7160.262047] Call Trace: [ 7160.264750] [ 7160.265735] ? show_regs.cold.9+0x22/0x2f [ 7160.277262] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7160.285665] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7160.294534] ? watchdog+0x40/0x40 [ 7160.306028] ? hrtimer_interrupt+0x128/0x2c0 [ 7160.311784] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7160.320277] ? apic_timer_interrupt+0xf/0x20 [ 7160.324580] [ 7160.329439] ? __pv_queued_spin_lock_slowpath+0x30a/0x6d0 [ 7160.333729] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7160.338353] do_raw_spin_lock+0x177/0x190 [ 7160.340813] _raw_spin_lock+0x21/0x30 [ 7160.344519] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7160.351714] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7160.358658] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7160.363338] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7160.368599] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7160.376576] ? woken_wake_function+0x30/0x30 [ 7160.378444] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7160.382029] kthread+0x1d1/0x200 [ 7160.384870] ? set_kthread_struct+0x70/0x70 [ 7160.387215] ret_from_fork+0x1f/0x30 [ 7172.057573] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 7172.063749] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7172.106152] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7172.121333] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7172.134202] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7172.144128] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7172.146864] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7172.167600] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7172.173896] RAX: 00000000000001a6 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7172.179764] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7172.185822] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7172.195309] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7172.204750] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7172.209948] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7172.220945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7172.228381] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7172.230688] Call Trace: [ 7172.231604] [ 7172.233379] ? show_regs.cold.9+0x22/0x2f [ 7172.237354] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7172.248283] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7172.249871] ? watchdog+0x40/0x40 [ 7172.252568] ? hrtimer_interrupt+0x128/0x2c0 [ 7172.255549] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7172.263190] ? apic_timer_interrupt+0xf/0x20 [ 7172.270828] [ 7172.275284] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7172.284244] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7172.288445] ? newidle_balance.isra.93+0x207/0x580 [ 7172.291854] do_raw_spin_lock+0x177/0x190 [ 7172.297798] _raw_spin_lock+0x21/0x30 [ 7172.299446] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7172.307624] process_one_work+0x2c8/0x700 [ 7172.317296] worker_thread+0x296/0x6e0 [ 7172.319733] ? rescuer_thread+0x570/0x570 [ 7172.325526] kthread+0x1d1/0x200 [ 7172.330711] ? set_kthread_struct+0x70/0x70 [ 7172.333851] ret_from_fork+0x1f/0x30 [ 7177.705105] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4540s! [ 7177.723017] Showing busy workqueues and worker pools: [ 7177.726725] workqueue events: flags=0x0 [ 7177.730441] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7177.735968] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7177.761706] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7177.772477] pending: cpu_latency_work [ptlrpc] [ 7177.776183] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7177.781580] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7177.793049] workqueue events_long: flags=0x0 [ 7177.797035] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7177.801749] pending: lu_objects_flush [obdclass] [ 7177.809664] workqueue events_unbound: flags=0x2 [ 7177.816752] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7177.821144] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7177.830452] workqueue events_power_efficient: flags=0x80 [ 7177.837644] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7177.841362] pending: check_lifetime, do_cache_clean [ 7177.844459] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7177.848592] pending: neigh_periodic_work, neigh_periodic_work [ 7177.852529] workqueue mm_percpu_wq: flags=0x8 [ 7177.856404] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7177.859608] pending: vmstat_update, lru_add_drain_per_cpu [ 7177.865049] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7177.868975] pending: vmstat_update, lru_add_drain_per_cpu [ 7177.871477] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7177.875931] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7177.879133] workqueue xprtiod: flags=0xa [ 7177.883270] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7177.886151] pending: xprt_autoclose, xs_error_handle [ 7177.891532] workqueue ipv6_addrconf: flags=0x40008 [ 7177.894831] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7177.898118] pending: addrconf_verify_work [ 7177.905750] workqueue knbd0-recv: flags=0x1a [ 7177.907104] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7177.912460] in-flight: 651:recv_work [ 7177.914229] workqueue ptlrpc_pinger: flags=0x2 [ 7177.918319] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7177.920450] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7177.923436] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4540s workers=4 idle: 15979 31824 8 [ 7177.929374] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7168s workers=2 idle: 654 [ 7180.055053] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7180.060261] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel [ 7180.060987] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7180.060389] ghash_clmulni_intel [ 7180.092425] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) [ 7180.095867] ata_piix [ 7180.096729] libcfs(O) [ 7180.112675] serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7180.112705] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7180.112715] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7180.112718] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7180.112734] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7180.112739] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7180.112744] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7180.112747] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7180.112749] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7180.112751] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7180.112753] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7180.112757] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7180.112760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7180.112762] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7180.112772] Call Trace: [ 7180.112778] [ 7180.112798] ? show_regs.cold.9+0x22/0x2f [ 7180.112817] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7180.112824] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7180.112828] ? watchdog+0x40/0x40 [ 7180.112832] ? hrtimer_interrupt+0x128/0x2c0 [ 7180.112836] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7180.112844] ? apic_timer_interrupt+0xf/0x20 [ 7180.112849] [ 7180.112852] ? native_safe_halt+0x2e/0x50 [ 7180.112856] kvm_wait+0xcd/0xe0 [ 7180.112864] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7180.112874] do_raw_spin_lock+0x177/0x190 [ 7180.112879] _raw_spin_lock+0x21/0x30 [ 7180.112884] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7180.113187] ? do_raw_spin_unlock+0x75/0x190 [ 7180.113194] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7180.113390] ? do_wait_intr+0xf0/0xf0 [ 7180.113399] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7180.113597] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7180.113817] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7180.114028] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7180.114153] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7180.114194] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7180.114223] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7180.114250] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7180.114336] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7180.114551] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7180.114579] lmv_intent_open+0x338/0x1150 [lmv] [ 7180.114611] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7180.115302] rpcsec_gss_krb5 [ 7180.117528] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7180.117645] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7180.117673] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7180.117759] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7180.118070] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7180.118152] ? do_raw_spin_unlock+0x75/0x190 [ 7180.118162] ? _raw_spin_unlock+0x12/0x30 [ 7180.118169] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7180.118196] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7180.118214] ll_atomic_open+0x36a/0xd70 [lustre] [ 7180.118296] lookup_open+0x356/0xb40 [ 7180.118306] do_last+0x313/0xfc0 [ 7180.118311] ? nd_jump_root+0xe5/0x160 [ 7180.118315] ? path_init+0x437/0x520 [ 7180.118320] path_openat+0xf7/0x500 [ 7180.118325] ? do_raw_spin_unlock+0x75/0x190 [ 7180.118330] do_filp_open+0x99/0x140 [ 7180.118399] ? getname_flags+0x6e/0x330 [ 7180.118405] ? __check_object_size+0xff/0x256 [ 7180.118412] ? do_raw_spin_unlock+0x75/0x190 [ 7180.118416] ? _raw_spin_unlock+0x12/0x30 [ 7180.118420] do_sys_openat2+0x2b4/0x410 [ 7180.118424] do_sys_open+0x73/0xa0 [ 7180.118428] __x64_sys_openat+0x24/0x30 [ 7180.118431] do_syscall_64+0xc1/0x440 [ 7180.118438] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7180.118444] RIP: 0033:0x7f53425a5332 [ 7180.118452] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7180.118455] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7180.118461] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7180.118464] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7180.118466] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7180.118468] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7180.118470] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7180.455926] auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7180.466221] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7180.474557] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7180.482078] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7180.484282] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7180.490239] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7180.492635] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7180.495186] RDX: 0000000000007f85 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7180.499282] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7180.501741] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7180.506250] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7180.512208] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7180.517919] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7180.520144] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7180.522318] Call Trace: [ 7180.523098] [ 7180.523699] ? show_regs.cold.9+0x22/0x2f [ 7180.525676] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7180.527732] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7180.529273] ? watchdog+0x40/0x40 [ 7180.530535] ? hrtimer_interrupt+0x128/0x2c0 [ 7180.532142] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7180.534495] ? apic_timer_interrupt+0xf/0x20 [ 7180.536224] [ 7180.537071] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7180.539161] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7180.540608] do_raw_spin_lock+0x177/0x190 [ 7180.542201] _raw_spin_lock+0x21/0x30 [ 7180.543171] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7180.544750] obd_mod_free+0x46/0x140 [ptlrpc] [ 7180.546164] mdc_commit_open+0x91/0xb0 [mdc] [ 7180.547375] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7180.549304] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7180.551135] after_reply+0xe2c/0x1780 [ptlrpc] [ 7180.553652] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7180.562769] ? __next_timer_interrupt+0x160/0x160 [ 7180.564947] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7180.567334] ? do_wait_intr+0xf0/0xf0 [ 7180.569068] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7180.574327] kthread+0x1d1/0x200 [ 7180.575891] ? set_kthread_struct+0x70/0x70 [ 7180.577862] ret_from_fork+0x1f/0x30 [ 7188.069167] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 7188.075588] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7188.119049] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7188.131961] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7188.139060] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7188.144058] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7188.158090] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7188.165164] RAX: 0000000000001e97 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7188.170987] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7188.174390] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7188.183282] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7188.192801] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7188.197306] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7188.201257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7188.204456] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7188.208809] Call Trace: [ 7188.210909] [ 7188.213081] ? show_regs.cold.9+0x22/0x2f [ 7188.215104] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7188.217977] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7188.222062] ? watchdog+0x40/0x40 [ 7188.224246] ? hrtimer_interrupt+0x128/0x2c0 [ 7188.227291] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7188.234379] ? apic_timer_interrupt+0xf/0x20 [ 7188.238355] [ 7188.239319] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7188.243609] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7188.246804] do_raw_spin_lock+0x177/0x190 [ 7188.250292] _raw_spin_lock+0x21/0x30 [ 7188.252399] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7188.261435] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7188.268154] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7188.271679] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7188.278182] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7188.288524] ? woken_wake_function+0x30/0x30 [ 7188.291379] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7188.297555] kthread+0x1d1/0x200 [ 7188.302452] ? set_kthread_struct+0x70/0x70 [ 7188.306979] ret_from_fork+0x1f/0x30 [ 7194.560568] rcu: INFO: rcu_sched self-detected stall on CPU [ 7194.567413] rcu: 2-....: (2754702 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 7194.573610] (t=5100134 jiffies g=274185 q=65029) [ 7194.575925] rcu: rcu_sched kthread starved for 4558500 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 7194.582434] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7194.586291] rcu: RCU grace-period kthread stack dump: [ 7194.589327] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7194.594548] Call Trace: [ 7194.595549] __schedule+0x351/0xcb0 [ 7194.598993] schedule+0xc0/0x180 [ 7194.600330] schedule_timeout+0xb4/0x190 [ 7194.602334] ? do_raw_spin_unlock+0x75/0x190 [ 7194.604919] ? __next_timer_interrupt+0x160/0x160 [ 7194.607095] ? prepare_to_swait_event+0xf3/0x140 [ 7194.609138] rcu_gp_kthread+0x8dd/0x15b0 [ 7194.611050] ? force_qs_rnp+0x2c0/0x2c0 [ 7194.613493] kthread+0x1d1/0x200 [ 7194.615395] ? set_kthread_struct+0x70/0x70 [ 7194.617442] ret_from_fork+0x1f/0x30 [ 7194.619402] rcu: Stack dump where RCU GP kthread last ran: [ 7194.622321] Sending NMI from CPU 2 to CPUs 1: [ 7194.628847] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 7194.629811] Sending NMI from CPU 2 to CPUs 1: [ 7194.645426] NMI backtrace for cpu 1 [ 7194.645434] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7194.645436] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7194.645438] RIP: 0010:apic_timer_interrupt+0x0/0x20 [ 7194.645440] Code: 04 24 48 89 54 24 08 48 89 ef e8 7b ee 2b ff 48 89 c7 48 8b 54 24 08 48 8b 04 24 48 83 c4 18 ff e7 c3 90 90 90 90 90 90 90 90 <68> 13 ff ff ff e8 86 f0 ff ff e8 d1 12 00 00 e9 7b f1 ff ff 66 66 [ 7194.645442] RSP: 0018:ffffbb53815034d8 EFLAGS: 00000002 [ 7194.645444] RAX: 0000000000007254 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7194.645446] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7194.645447] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7194.645449] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7194.645450] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7194.645451] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7194.645452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7194.645453] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7194.645454] Call Trace: [ 7194.645455] RIP: 0010:__pv_queued_spin_lock_slowpath+0x320/0x6d0 [ 7194.645457] Code: 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 <48> 83 05 18 d4 dc 02 01 eb e0 48 83 05 06 d4 dc 02 01 48 83 05 be [ 7194.645460] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ? do_raw_spin_lock+0x177/0x190 [ 7194.645461] ? _raw_spin_lock+0x21/0x30 [ 7194.645463] ? ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7194.645464] ? do_raw_spin_unlock+0x75/0x190 [ 7194.645465] ? ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7194.645467] ? do_wait_intr+0xf0/0xf0 [ 7194.645468] ? ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7194.645469] ? ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7194.645471] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7194.645472] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7194.645474] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7194.645475] ? mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7194.645477] ? mdc_intent_lock+0x353/0x7e0 [mdc] [ 7194.645478] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7194.645479] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7194.645481] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7194.645482] ? lmv_intent_open+0x338/0x1150 [lmv] [ 7194.645483] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7194.645484] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7194.645485] ? lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7194.645486] ? ll_intent_lock+0xc5/0xaa0 [lustre] [ 7194.645487] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7194.645488] ? ll_lookup_it+0xe0e/0x2270 [lustre] [ 7194.645489] ? do_raw_spin_unlock+0x75/0x190 [ 7194.645490] ? _raw_spin_unlock+0x12/0x30 [ 7194.645491] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7194.645492] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7194.645493] ? ll_atomic_open+0x36a/0xd70 [lustre] [ 7194.645493] ? lookup_open+0x356/0xb40 [ 7194.645494] ? do_last+0x313/0xfc0 [ 7194.645495] ? nd_jump_root+0xe5/0x160 [ 7194.645496] ? path_init+0x437/0x520 [ 7194.645497] ? path_openat+0xf7/0x500 [ 7194.645498] ? do_raw_spin_unlock+0x75/0x190 [ 7194.645499] ? do_filp_open+0x99/0x140 [ 7194.645500] ? getname_flags+0x6e/0x330 [ 7194.645501] ? __check_object_size+0xff/0x256 [ 7194.645502] ? do_raw_spin_unlock+0x75/0x190 [ 7194.645502] ? _raw_spin_unlock+0x12/0x30 [ 7194.645504] ? do_sys_openat2+0x2b4/0x410 [ 7194.645505] ? do_sys_open+0x73/0xa0 [ 7194.645506] ? __x64_sys_openat+0x24/0x30 [ 7194.645507] ? do_syscall_64+0xc1/0x440 [ 7194.645508] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7194.645514] NMI backtrace for cpu 2 [ 7194.810059] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7194.814326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7194.817517] Call Trace: [ 7194.818382] [ 7194.819169] ? dump_stack+0xbb/0x10e [ 7194.820427] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7194.822267] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7194.824236] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7194.826137] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7194.827994] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7194.829896] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7194.831366] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7194.833178] ? update_process_times+0x6d/0xc0 [ 7194.834937] ? tick_sched_handle.isra.18+0x27/0x90 [ 7194.836553] ? tick_sched_timer+0x53/0xb0 [ 7194.838185] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7194.839941] ? tick_sched_handle.isra.18+0x90/0x90 [ 7194.841511] ? hrtimer_interrupt+0x128/0x2c0 [ 7194.843114] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7194.844986] ? apic_timer_interrupt+0xf/0x20 [ 7194.846396] [ 7194.847299] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7194.849876] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7194.851933] ? do_raw_spin_lock+0x177/0x190 [ 7194.853414] ? _raw_spin_lock+0x21/0x30 [ 7194.855026] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7194.857799] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7194.859954] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7194.862194] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7194.864322] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7194.867397] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7194.870098] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7194.873214] ? __next_timer_interrupt+0x160/0x160 [ 7194.875902] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7194.878332] ? do_wait_intr+0xf0/0xf0 [ 7194.880248] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7194.884280] ? kthread+0x1d1/0x200 [ 7194.886402] ? set_kthread_struct+0x70/0x70 [ 7194.888853] ? ret_from_fork+0x1f/0x30 [ 7204.061849] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 7204.070230] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7204.125666] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7204.135500] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7204.146035] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7204.154214] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7204.163102] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7204.176535] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7204.180255] RAX: 0000000000006a83 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7204.184041] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7204.188346] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7204.191734] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7204.195583] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7204.199192] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7204.202415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7204.204282] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7204.216865] Call Trace: [ 7204.217743] [ 7204.218464] ? show_regs.cold.9+0x22/0x2f [ 7204.229529] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7204.233751] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7204.242284] ? watchdog+0x40/0x40 [ 7204.247278] ? hrtimer_interrupt+0x128/0x2c0 [ 7204.251737] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7204.258510] ? apic_timer_interrupt+0xf/0x20 [ 7204.264551] [ 7204.267368] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7204.273391] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7204.279392] ? newidle_balance.isra.93+0x207/0x580 [ 7204.285216] do_raw_spin_lock+0x177/0x190 [ 7204.290796] _raw_spin_lock+0x21/0x30 [ 7204.295202] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7204.302279] process_one_work+0x2c8/0x700 [ 7204.304021] worker_thread+0x296/0x6e0 [ 7204.305219] ? rescuer_thread+0x570/0x570 [ 7204.308178] kthread+0x1d1/0x200 [ 7204.310641] ? set_kthread_struct+0x70/0x70 [ 7204.315672] ret_from_fork+0x1f/0x30 [ 7208.415185] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4571s! [ 7208.422747] Showing busy workqueues and worker pools: [ 7208.425724] workqueue events: flags=0x0 [ 7208.427431] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7208.430730] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7208.435968] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7208.437949] pending: cpu_latency_work [ptlrpc] [ 7208.441271] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7208.452488] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7208.456677] workqueue events_long: flags=0x0 [ 7208.458390] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7208.461875] pending: lu_objects_flush [obdclass] [ 7208.464034] workqueue events_unbound: flags=0x2 [ 7208.465615] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7208.471485] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7208.474008] workqueue events_power_efficient: flags=0x80 [ 7208.476614] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7208.479630] pending: check_lifetime, do_cache_clean [ 7208.483659] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7208.485950] pending: neigh_periodic_work, neigh_periodic_work [ 7208.488274] workqueue mm_percpu_wq: flags=0x8 [ 7208.490186] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7208.492789] pending: vmstat_update, lru_add_drain_per_cpu [ 7208.495252] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7208.497939] pending: vmstat_update, lru_add_drain_per_cpu [ 7208.500579] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7208.503047] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7208.505183] workqueue xprtiod: flags=0xa [ 7208.506721] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7208.509026] pending: xprt_autoclose, xs_error_handle [ 7208.511221] workqueue ipv6_addrconf: flags=0x40008 [ 7208.513189] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7208.515838] pending: addrconf_verify_work [ 7208.517890] workqueue knbd0-recv: flags=0x1a [ 7208.519461] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7208.522218] in-flight: 651:recv_work [ 7208.523911] workqueue ptlrpc_pinger: flags=0x2 [ 7208.525766] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7208.528215] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7208.530529] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4571s workers=4 idle: 15979 31824 8 [ 7208.534034] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7198s workers=2 idle: 654 [ 7216.066957] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 7216.072961] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7216.136247] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7216.143866] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7216.155970] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7216.163643] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7216.187437] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7216.197762] RAX: 0000000000007683 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7216.209103] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7216.224002] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7216.236812] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7216.245969] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7216.255134] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7216.275792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7216.288751] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7216.291238] Call Trace: [ 7216.292082] [ 7216.307837] ? show_regs.cold.9+0x22/0x2f [ 7216.309326] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7216.328568] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7216.332299] ? watchdog+0x40/0x40 [ 7216.342636] ? hrtimer_interrupt+0x128/0x2c0 [ 7216.348958] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7216.355204] ? apic_timer_interrupt+0xf/0x20 [ 7216.363028] [ 7216.370061] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7216.375879] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7216.385941] do_raw_spin_lock+0x177/0x190 [ 7216.390724] _raw_spin_lock+0x21/0x30 [ 7216.396952] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7216.410784] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7216.418951] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7216.426384] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7216.434766] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7216.446631] ? woken_wake_function+0x30/0x30 [ 7216.448037] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7216.456578] kthread+0x1d1/0x200 [ 7216.463806] ? set_kthread_struct+0x70/0x70 [ 7216.466548] ret_from_fork+0x1f/0x30 [ 7220.055869] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 7220.057066] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 7220.070409] Modules linked in: lustre(O) [ 7220.073493] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7220.073549] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7220.073555] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7220.073558] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7220.073569] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7220.073572] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7220.073575] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7220.073576] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7220.073578] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7220.073579] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7220.073581] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7220.073583] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7220.073586] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7220.073588] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7220.073597] Call Trace: [ 7220.073603] [ 7220.073609] ? show_regs.cold.9+0x22/0x2f [ 7220.073619] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7220.073626] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7220.073630] ? watchdog+0x40/0x40 [ 7220.073634] ? hrtimer_interrupt+0x128/0x2c0 [ 7220.073639] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7220.073646] ? apic_timer_interrupt+0xf/0x20 [ 7220.073649] [ 7220.073651] ? native_safe_halt+0x2e/0x50 [ 7220.073654] kvm_wait+0xcd/0xe0 [ 7220.073659] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7220.073664] do_raw_spin_lock+0x177/0x190 [ 7220.073667] _raw_spin_lock+0x21/0x30 [ 7220.073671] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7220.073861] obd_mod_free+0x46/0x140 [ptlrpc] [ 7220.074620] mdc_commit_open+0x91/0xb0 [mdc] [ 7220.074657] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7220.074855] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7220.075063] after_reply+0xe2c/0x1780 [ptlrpc] [ 7220.075329] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7220.075520] ? __next_timer_interrupt+0x160/0x160 [ 7220.075529] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7220.075729] ? do_wait_intr+0xf0/0xf0 [ 7220.075738] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7220.075944] kthread+0x1d1/0x200 [ 7220.075950] ? set_kthread_struct+0x70/0x70 [ 7220.075954] ret_from_fork+0x1f/0x30 [ 7220.191204] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7220.208202] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7220.212742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7220.215916] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7220.218210] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7220.225703] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7220.228703] RAX: 000000000000453d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7220.230981] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7220.232932] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7220.235839] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7220.238813] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7220.241686] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7220.244931] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7220.247162] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7220.250001] Call Trace: [ 7220.250935] [ 7220.251623] ? show_regs.cold.9+0x22/0x2f [ 7220.253378] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7220.255550] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7220.257472] ? watchdog+0x40/0x40 [ 7220.258779] ? hrtimer_interrupt+0x128/0x2c0 [ 7220.260586] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7220.262439] ? apic_timer_interrupt+0xf/0x20 [ 7220.264249] [ 7220.265010] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7220.266876] do_raw_spin_lock+0x177/0x190 [ 7220.268675] _raw_spin_lock+0x21/0x30 [ 7220.270027] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7220.272495] ? do_raw_spin_unlock+0x75/0x190 [ 7220.274324] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7220.276510] ? do_wait_intr+0xf0/0xf0 [ 7220.277961] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7220.280240] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7220.282464] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7220.284926] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7220.287271] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7220.289481] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7220.291344] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7220.292970] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7220.295248] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7220.297511] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7220.299594] lmv_intent_open+0x338/0x1150 [lmv] [ 7220.300716] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7220.302606] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7220.304704] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7220.306500] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7220.307849] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7220.309792] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7220.311868] ? do_raw_spin_unlock+0x75/0x190 [ 7220.313685] ? _raw_spin_unlock+0x12/0x30 [ 7220.315995] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7220.318260] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7220.320321] ll_atomic_open+0x36a/0xd70 [lustre] [ 7220.322266] lookup_open+0x356/0xb40 [ 7220.323589] do_last+0x313/0xfc0 [ 7220.324965] ? nd_jump_root+0xe5/0x160 [ 7220.326568] ? path_init+0x437/0x520 [ 7220.327851] path_openat+0xf7/0x500 [ 7220.329362] ? do_raw_spin_unlock+0x75/0x190 [ 7220.330872] do_filp_open+0x99/0x140 [ 7220.332436] ? getname_flags+0x6e/0x330 [ 7220.333927] ? __check_object_size+0xff/0x256 [ 7220.335740] ? do_raw_spin_unlock+0x75/0x190 [ 7220.337505] ? _raw_spin_unlock+0x12/0x30 [ 7220.339257] do_sys_openat2+0x2b4/0x410 [ 7220.340711] do_sys_open+0x73/0xa0 [ 7220.342049] __x64_sys_openat+0x24/0x30 [ 7220.343642] do_syscall_64+0xc1/0x440 [ 7220.345055] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7220.347642] RIP: 0033:0x7f53425a5332 [ 7220.349323] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7220.356704] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7220.359736] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7220.362441] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7220.365195] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7220.368094] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7220.370539] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7236.054865] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 7236.061903] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7236.127864] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7236.139515] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7236.152321] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7236.158631] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7236.163265] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7236.181957] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7236.184494] RAX: 0000000000000ec7 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7236.191668] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7236.199584] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7236.210315] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7236.224094] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7236.237751] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7236.252094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7236.257114] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7236.260291] Call Trace: [ 7236.264571] [ 7236.265684] ? show_regs.cold.9+0x22/0x2f [ 7236.271490] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7236.283483] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7236.296837] ? watchdog+0x40/0x40 [ 7236.298109] ? hrtimer_interrupt+0x128/0x2c0 [ 7236.306810] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7236.308614] ? apic_timer_interrupt+0xf/0x20 [ 7236.318497] [ 7236.320369] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7236.326594] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7236.333962] ? newidle_balance.isra.93+0x207/0x580 [ 7236.336026] do_raw_spin_lock+0x177/0x190 [ 7236.341787] _raw_spin_lock+0x21/0x30 [ 7236.343580] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7236.347590] process_one_work+0x2c8/0x700 [ 7236.349755] worker_thread+0x296/0x6e0 [ 7236.351720] ? rescuer_thread+0x570/0x570 [ 7236.353740] kthread+0x1d1/0x200 [ 7236.359457] ? set_kthread_struct+0x70/0x70 [ 7236.361396] ret_from_fork+0x1f/0x30 [ 7238.623988] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4601s! [ 7238.627161] Showing busy workqueues and worker pools: [ 7238.629631] workqueue events: flags=0x0 [ 7238.631082] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7238.633950] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7238.639896] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7238.643288] pending: cpu_latency_work [ptlrpc] [ 7238.645964] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7238.648179] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7238.653072] workqueue events_long: flags=0x0 [ 7238.655161] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7238.658151] pending: lu_objects_flush [obdclass] [ 7238.661147] workqueue events_unbound: flags=0x2 [ 7238.663412] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7238.666678] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7238.669600] workqueue events_power_efficient: flags=0x80 [ 7238.672641] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7238.677130] pending: check_lifetime, do_cache_clean [ 7238.681287] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7238.686267] pending: neigh_periodic_work, neigh_periodic_work [ 7238.692625] workqueue mm_percpu_wq: flags=0x8 [ 7238.696102] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7238.700575] pending: vmstat_update, lru_add_drain_per_cpu [ 7238.704414] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7238.709716] pending: vmstat_update, lru_add_drain_per_cpu [ 7238.714129] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7238.718235] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7238.723120] workqueue xprtiod: flags=0xa [ 7238.725454] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7238.729124] pending: xprt_autoclose, xs_error_handle [ 7238.732310] workqueue ipv6_addrconf: flags=0x40008 [ 7238.735437] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7238.739944] pending: addrconf_verify_work [ 7238.742552] workqueue knbd0-recv: flags=0x1a [ 7238.744429] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7238.746427] in-flight: 651:recv_work [ 7238.748129] workqueue ptlrpc_pinger: flags=0x2 [ 7238.750220] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7238.753209] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7238.756124] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4601s workers=4 idle: 15979 31824 8 [ 7238.759363] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7229s workers=2 idle: 654 [ 7244.059309] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 7244.066395] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7244.108605] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7244.117413] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7244.128038] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7244.134954] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7244.163847] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7244.170287] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7244.180518] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7244.199010] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7244.202611] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7244.210080] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7244.217205] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7244.222262] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7244.227600] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7244.234354] Call Trace: [ 7244.234969] [ 7244.235464] ? show_regs.cold.9+0x22/0x2f [ 7244.241072] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7244.246010] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7244.249890] ? watchdog+0x40/0x40 [ 7244.251148] ? hrtimer_interrupt+0x128/0x2c0 [ 7244.260991] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7244.274471] ? apic_timer_interrupt+0xf/0x20 [ 7244.293986] [ 7244.294690] ? native_safe_halt+0x2e/0x50 [ 7244.297190] kvm_wait+0xcd/0xe0 [ 7244.298588] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7244.306388] do_raw_spin_lock+0x177/0x190 [ 7244.310010] _raw_spin_lock+0x21/0x30 [ 7244.311601] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7244.316244] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7244.319506] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7244.325798] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7244.334874] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7244.336769] ? woken_wake_function+0x30/0x30 [ 7244.340720] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7244.345213] kthread+0x1d1/0x200 [ 7244.346396] ? set_kthread_struct+0x70/0x70 [ 7244.348380] ret_from_fork+0x1f/0x30 [ 7248.061330] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 7248.062611] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 7248.064571] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7248.064663] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7248.064672] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7248.064675] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7248.064691] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7248.064695] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7248.064700] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7248.064702] RDX: 000000000000764a RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7248.064704] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7248.064706] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7248.064709] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7248.064711] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7248.064714] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7248.064716] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7248.064726] Call Trace: [ 7248.064731] [ 7248.064738] ? show_regs.cold.9+0x22/0x2f [ 7248.064747] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7248.064753] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7248.064757] ? watchdog+0x40/0x40 [ 7248.064762] ? hrtimer_interrupt+0x128/0x2c0 [ 7248.064766] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7248.064773] ? apic_timer_interrupt+0xf/0x20 [ 7248.064778] [ 7248.064781] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7248.064786] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7248.064791] do_raw_spin_lock+0x177/0x190 [ 7248.064796] _raw_spin_lock+0x21/0x30 [ 7248.064803] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7248.065070] obd_mod_free+0x46/0x140 [ptlrpc] [ 7248.065266] mdc_commit_open+0x91/0xb0 [mdc] [ 7248.065307] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7248.065496] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7248.066690] after_reply+0xe2c/0x1780 [ptlrpc] [ 7248.066914] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7248.067111] ? __next_timer_interrupt+0x160/0x160 [ 7248.067121] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7248.067326] ? do_wait_intr+0xf0/0xf0 [ 7248.067336] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7248.067537] kthread+0x1d1/0x200 [ 7248.067543] ? set_kthread_struct+0x70/0x70 [ 7248.067547] ret_from_fork+0x1f/0x30 [ 7248.542869] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7248.620219] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7248.636975] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7248.659251] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7248.667780] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7248.711524] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7248.720075] RAX: 000000000000720e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7248.726552] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7248.744470] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7248.751610] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7248.759499] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7248.764354] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7248.776556] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7248.788249] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7248.799351] Call Trace: [ 7248.800457] [ 7248.803383] ? show_regs.cold.9+0x22/0x2f [ 7248.807638] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7248.809212] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7248.814540] ? watchdog+0x40/0x40 [ 7248.821404] ? hrtimer_interrupt+0x128/0x2c0 [ 7248.823570] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7248.828843] ? apic_timer_interrupt+0xf/0x20 [ 7248.831502] [ 7248.835930] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7248.843593] do_raw_spin_lock+0x177/0x190 [ 7248.845078] _raw_spin_lock+0x21/0x30 [ 7248.855275] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7248.869347] ? do_raw_spin_unlock+0x75/0x190 [ 7248.874668] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7248.878154] ? do_wait_intr+0xf0/0xf0 [ 7248.879289] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7248.890609] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7248.892111] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7248.915599] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7248.917432] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7248.920649] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7248.922238] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7248.937142] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7248.939084] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7248.960434] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7248.970444] lmv_intent_open+0x338/0x1150 [lmv] [ 7248.972187] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7248.983871] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7248.997049] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7249.002316] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7249.019075] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7249.025387] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7249.033405] ? do_raw_spin_unlock+0x75/0x190 [ 7249.043358] ? _raw_spin_unlock+0x12/0x30 [ 7249.045812] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7249.055030] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7249.067484] ll_atomic_open+0x36a/0xd70 [lustre] [ 7249.074901] lookup_open+0x356/0xb40 [ 7249.078193] do_last+0x313/0xfc0 [ 7249.089988] ? nd_jump_root+0xe5/0x160 [ 7249.093319] ? path_init+0x437/0x520 [ 7249.103676] path_openat+0xf7/0x500 [ 7249.104933] ? do_raw_spin_unlock+0x75/0x190 [ 7249.110259] do_filp_open+0x99/0x140 [ 7249.117181] ? getname_flags+0x6e/0x330 [ 7249.121936] ? __check_object_size+0xff/0x256 [ 7249.125960] ? do_raw_spin_unlock+0x75/0x190 [ 7249.127282] ? _raw_spin_unlock+0x12/0x30 [ 7249.136139] do_sys_openat2+0x2b4/0x410 [ 7249.139477] do_sys_open+0x73/0xa0 [ 7249.141077] __x64_sys_openat+0x24/0x30 [ 7249.143268] do_syscall_64+0xc1/0x440 [ 7249.148900] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7249.154340] RIP: 0033:0x7f53425a5332 [ 7249.167157] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7249.193314] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7249.195547] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7249.211113] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7249.219246] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7249.231666] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7249.237434] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7264.052057] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 7264.055696] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7264.075306] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7264.079483] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7264.082993] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7264.085163] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7264.086904] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7264.094386] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7264.098050] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7264.101490] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7264.104433] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7264.107157] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7264.110278] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7264.113441] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7264.116899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7264.119306] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7264.122415] Call Trace: [ 7264.123424] [ 7264.124162] ? show_regs.cold.9+0x22/0x2f [ 7264.125627] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7264.127446] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7264.128856] ? watchdog+0x40/0x40 [ 7264.130067] ? hrtimer_interrupt+0x128/0x2c0 [ 7264.132013] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7264.133981] ? apic_timer_interrupt+0xf/0x20 [ 7264.135928] [ 7264.137059] ? native_safe_halt+0x2e/0x50 [ 7264.138876] kvm_wait+0xcd/0xe0 [ 7264.140097] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7264.142114] ? newidle_balance.isra.93+0x207/0x580 [ 7264.143882] do_raw_spin_lock+0x177/0x190 [ 7264.145235] _raw_spin_lock+0x21/0x30 [ 7264.147131] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7264.149524] process_one_work+0x2c8/0x700 [ 7264.151678] worker_thread+0x296/0x6e0 [ 7264.153273] ? rescuer_thread+0x570/0x570 [ 7264.154766] kthread+0x1d1/0x200 [ 7264.156833] ? set_kthread_struct+0x70/0x70 [ 7264.162975] ret_from_fork+0x1f/0x30 [ 7268.834863] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4631s! [ 7268.842028] Showing busy workqueues and worker pools: [ 7268.845944] workqueue events: flags=0x0 [ 7268.849124] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7268.853647] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7268.862169] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7268.868147] pending: cpu_latency_work [ptlrpc] [ 7268.872939] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7268.878582] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7268.886053] workqueue events_long: flags=0x0 [ 7268.888547] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7268.894557] pending: lu_objects_flush [obdclass] [ 7268.898788] workqueue events_unbound: flags=0x2 [ 7268.902924] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7268.907132] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7268.912527] workqueue events_power_efficient: flags=0x80 [ 7268.915350] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7268.921101] pending: check_lifetime, do_cache_clean [ 7268.925654] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7268.929780] pending: neigh_periodic_work, neigh_periodic_work [ 7268.935014] workqueue mm_percpu_wq: flags=0x8 [ 7268.938044] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7268.942977] pending: vmstat_update, lru_add_drain_per_cpu [ 7268.946986] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7268.950874] pending: vmstat_update, lru_add_drain_per_cpu [ 7268.956024] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7268.960945] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7268.966018] workqueue xprtiod: flags=0xa [ 7268.971919] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7268.976978] pending: xprt_autoclose, xs_error_handle [ 7268.981648] workqueue ipv6_addrconf: flags=0x40008 [ 7268.989106] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7268.996762] pending: addrconf_verify_work [ 7269.000585] workqueue knbd0-recv: flags=0x1a [ 7269.003013] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7269.005609] in-flight: 651:recv_work [ 7269.009714] workqueue ptlrpc_pinger: flags=0x2 [ 7269.011559] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7269.019450] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7269.027540] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4631s workers=4 idle: 15979 31824 8 [ 7269.034064] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7259s workers=2 idle: 654 [ 7272.058600] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 7272.061433] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7272.090048] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7272.096079] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7272.101675] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7272.104515] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7272.112828] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7272.118136] RAX: 0000000000001f67 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7272.123487] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7272.133211] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7272.136823] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7272.140683] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7272.144076] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7272.147969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7272.150913] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7272.153059] Call Trace: [ 7272.153851] [ 7272.154580] ? show_regs.cold.9+0x22/0x2f [ 7272.156299] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7272.158572] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7272.161111] ? watchdog+0x40/0x40 [ 7272.162823] ? hrtimer_interrupt+0x128/0x2c0 [ 7272.164978] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7272.167657] ? apic_timer_interrupt+0xf/0x20 [ 7272.169522] [ 7272.170571] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7272.173501] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7272.176847] do_raw_spin_lock+0x177/0x190 [ 7272.178823] _raw_spin_lock+0x21/0x30 [ 7272.180677] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7272.183854] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7272.188905] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7272.193541] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7272.197648] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7272.202032] ? woken_wake_function+0x30/0x30 [ 7272.207466] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7272.209796] kthread+0x1d1/0x200 [ 7272.215184] ? set_kthread_struct+0x70/0x70 [ 7272.224786] ret_from_fork+0x1f/0x30 [ 7276.058438] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7276.058460] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7276.064800] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7276.064937] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7276.064945] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7276.064948] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7276.064962] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7276.064966] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7276.064971] RAX: 0000000000001d83 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7276.064973] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7276.064975] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7276.064977] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7276.064980] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7276.064983] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7276.064986] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7276.064989] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7276.064998] Call Trace: [ 7276.065004] [ 7276.065010] ? show_regs.cold.9+0x22/0x2f [ 7276.065020] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7276.065026] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7276.065031] ? watchdog+0x40/0x40 [ 7276.065036] ? hrtimer_interrupt+0x128/0x2c0 [ 7276.065041] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7276.065051] ? apic_timer_interrupt+0xf/0x20 [ 7276.065056] [ 7276.065060] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7276.065066] do_raw_spin_lock+0x177/0x190 [ 7276.065071] _raw_spin_lock+0x21/0x30 [ 7276.065076] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7276.066013] ? do_raw_spin_unlock+0x75/0x190 [ 7276.066020] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7276.066445] ? do_wait_intr+0xf0/0xf0 [ 7276.066454] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7276.073595] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7276.073690] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7276.073697] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7276.073700] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7276.073921] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7276.073927] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7276.073932] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7276.073933] RDX: 0000000000004e56 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7276.073935] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7276.073936] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7276.073938] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7276.073940] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7276.073942] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7276.073944] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7276.073951] Call Trace: [ 7276.073957] [ 7276.073963] ? show_regs.cold.9+0x22/0x2f [ 7276.073974] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7276.073980] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7276.073985] ? watchdog+0x40/0x40 [ 7276.073989] ? hrtimer_interrupt+0x128/0x2c0 [ 7276.073993] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7276.074004] ? apic_timer_interrupt+0xf/0x20 [ 7276.074007] [ 7276.074009] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7276.074015] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7276.074020] do_raw_spin_lock+0x177/0x190 [ 7276.074025] _raw_spin_lock+0x21/0x30 [ 7276.074032] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7276.074277] obd_mod_free+0x46/0x140 [ptlrpc] [ 7276.074402] mdc_commit_open+0x91/0xb0 [mdc] [ 7276.074432] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7276.074617] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7276.074976] after_reply+0xe2c/0x1780 [ptlrpc] [ 7276.075174] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7276.075300] ? __next_timer_interrupt+0x160/0x160 [ 7276.075307] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7276.075440] ? do_wait_intr+0xf0/0xf0 [ 7276.075447] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7276.075644] kthread+0x1d1/0x200 [ 7276.075649] ? set_kthread_struct+0x70/0x70 [ 7276.075652] ret_from_fork+0x1f/0x30 [ 7276.514642] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7276.518691] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7276.523741] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7276.526876] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7276.530522] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7276.532231] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7276.534332] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7276.536728] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7276.541802] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7276.546700] lmv_intent_open+0x338/0x1150 [lmv] [ 7276.549715] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7276.553398] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7276.556384] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7276.558899] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7276.561742] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7276.566493] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7276.570160] ? do_raw_spin_unlock+0x75/0x190 [ 7276.572881] ? _raw_spin_unlock+0x12/0x30 [ 7276.576567] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7276.579859] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7276.582858] ll_atomic_open+0x36a/0xd70 [lustre] [ 7276.586721] lookup_open+0x356/0xb40 [ 7276.588840] do_last+0x313/0xfc0 [ 7276.590876] ? nd_jump_root+0xe5/0x160 [ 7276.593197] ? path_init+0x437/0x520 [ 7276.595587] path_openat+0xf7/0x500 [ 7276.597703] ? do_raw_spin_unlock+0x75/0x190 [ 7276.600179] do_filp_open+0x99/0x140 [ 7276.601811] ? getname_flags+0x6e/0x330 [ 7276.603766] ? __check_object_size+0xff/0x256 [ 7276.606041] ? do_raw_spin_unlock+0x75/0x190 [ 7276.608768] ? _raw_spin_unlock+0x12/0x30 [ 7276.611004] do_sys_openat2+0x2b4/0x410 [ 7276.613584] do_sys_open+0x73/0xa0 [ 7276.615317] __x64_sys_openat+0x24/0x30 [ 7276.617720] do_syscall_64+0xc1/0x440 [ 7276.619643] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7276.622716] RIP: 0033:0x7f53425a5332 [ 7276.624688] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7276.634845] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7276.638924] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7276.643613] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7276.647197] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7276.650884] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7276.654502] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7296.052033] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 7296.055594] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7296.096047] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7296.107435] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7296.111783] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7296.114627] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7296.116230] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7296.129292] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7296.141227] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7296.143488] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7296.154628] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7296.163539] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7296.175472] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7296.189625] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7296.206591] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7296.210665] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7296.214960] Call Trace: [ 7296.215726] [ 7296.221474] ? show_regs.cold.9+0x22/0x2f [ 7296.224025] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7296.226304] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7296.227766] ? watchdog+0x40/0x40 [ 7296.233012] ? hrtimer_interrupt+0x128/0x2c0 [ 7296.234506] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7296.237879] ? apic_timer_interrupt+0xf/0x20 [ 7296.251183] [ 7296.252926] ? native_safe_halt+0x2e/0x50 [ 7296.254493] kvm_wait+0xcd/0xe0 [ 7296.257036] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7296.262333] ? newidle_balance.isra.93+0x207/0x580 [ 7296.264670] do_raw_spin_lock+0x177/0x190 [ 7296.267303] _raw_spin_lock+0x21/0x30 [ 7296.270220] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7296.273275] process_one_work+0x2c8/0x700 [ 7296.275866] worker_thread+0x296/0x6e0 [ 7296.277695] ? rescuer_thread+0x570/0x570 [ 7296.279275] kthread+0x1d1/0x200 [ 7296.281511] ? set_kthread_struct+0x70/0x70 [ 7296.289923] ret_from_fork+0x1f/0x30 [ 7299.551281] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4662s! [ 7299.559122] Showing busy workqueues and worker pools: [ 7299.562732] workqueue events: flags=0x0 [ 7299.565192] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7299.569417] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7299.574799] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7299.578495] pending: cpu_latency_work [ptlrpc] [ 7299.581856] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7299.584843] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7299.591121] workqueue events_long: flags=0x0 [ 7299.592857] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7299.598168] pending: lu_objects_flush [obdclass] [ 7299.601441] workqueue events_unbound: flags=0x2 [ 7299.604300] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7299.608775] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7299.612603] workqueue events_power_efficient: flags=0x80 [ 7299.615893] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7299.620160] pending: check_lifetime, do_cache_clean [ 7299.622388] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7299.624463] pending: neigh_periodic_work, neigh_periodic_work [ 7299.626678] workqueue mm_percpu_wq: flags=0x8 [ 7299.628385] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7299.630516] pending: vmstat_update, lru_add_drain_per_cpu [ 7299.632734] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7299.635268] pending: vmstat_update, lru_add_drain_per_cpu [ 7299.637216] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7299.639659] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7299.642405] workqueue xprtiod: flags=0xa [ 7299.643832] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7299.646275] pending: xprt_autoclose, xs_error_handle [ 7299.648356] workqueue ipv6_addrconf: flags=0x40008 [ 7299.650211] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7299.652590] pending: addrconf_verify_work [ 7299.654385] workqueue knbd0-recv: flags=0x1a [ 7299.656228] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7299.658618] in-flight: 651:recv_work [ 7299.660215] workqueue ptlrpc_pinger: flags=0x2 [ 7299.661961] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7299.665019] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7299.668027] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4662s workers=4 idle: 15979 31824 8 [ 7299.671335] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7290s workers=2 idle: 654 [ 7300.061027] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 7300.065018] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7300.098488] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7300.104249] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7300.113998] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7300.116424] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7300.129505] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7300.133306] RAX: 000000000000670b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7300.136931] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7300.139873] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7300.146202] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7300.152840] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7300.156533] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7300.159629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7300.165903] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7300.172028] Call Trace: [ 7300.173071] [ 7300.174848] ? show_regs.cold.9+0x22/0x2f [ 7300.176440] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7300.179361] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7300.185379] ? watchdog+0x40/0x40 [ 7300.187076] ? hrtimer_interrupt+0x128/0x2c0 [ 7300.192413] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7300.198097] ? apic_timer_interrupt+0xf/0x20 [ 7300.201151] [ 7300.202567] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7300.210918] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7300.215630] do_raw_spin_lock+0x177/0x190 [ 7300.217549] _raw_spin_lock+0x21/0x30 [ 7300.223931] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7300.230330] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7300.232510] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7300.236091] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7300.240872] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7300.246605] ? woken_wake_function+0x30/0x30 [ 7300.248843] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7300.251231] kthread+0x1d1/0x200 [ 7300.252739] ? set_kthread_struct+0x70/0x70 [ 7300.254567] ret_from_fork+0x1f/0x30 [ 7304.055059] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7304.057016] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7304.058518] Modules linked in: [ 7304.062766] Modules linked in: [ 7304.062815] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7304.064144] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7304.064150] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7304.064153] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7304.064167] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7304.064172] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7304.064176] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7304.064179] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7304.064181] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7304.064182] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7304.064184] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7304.064187] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7304.064190] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7304.064192] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7304.064200] Call Trace: [ 7304.064206] [ 7304.064211] ? show_regs.cold.9+0x22/0x2f [ 7304.064220] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7304.064226] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7304.064230] ? watchdog+0x40/0x40 [ 7304.064234] ? hrtimer_interrupt+0x128/0x2c0 [ 7304.064238] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7304.064245] ? apic_timer_interrupt+0xf/0x20 [ 7304.064249] [ 7304.064252] ? native_safe_halt+0x2e/0x50 [ 7304.064256] kvm_wait+0xcd/0xe0 [ 7304.065224] lustre(O) [ 7304.084694] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7304.084722] do_raw_spin_lock+0x177/0x190 [ 7304.084727] _raw_spin_lock+0x21/0x30 [ 7304.084735] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7304.085065] ? do_raw_spin_unlock+0x75/0x190 [ 7304.085072] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7304.089402] osp(O) [ 7304.094010] ? do_wait_intr+0xf0/0xf0 [ 7304.096123] ofd(O) [ 7304.106539] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7304.109468] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) [ 7304.115345] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7304.118201] fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7304.118259] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7304.118266] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7304.118269] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7304.118282] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7304.118286] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7304.118291] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7304.118293] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7304.118295] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7304.118297] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7304.118300] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7304.118303] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7304.118306] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7304.118309] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7304.118318] Call Trace: [ 7304.121331] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7304.124119] [ 7304.127960] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7304.131201] ? show_regs.cold.9+0x22/0x2f [ 7304.133653] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7304.136029] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7304.137241] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7304.138062] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7304.140016] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7304.142315] ? watchdog+0x40/0x40 [ 7304.144238] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7304.144373] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7304.145073] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7304.145130] lmv_intent_open+0x338/0x1150 [lmv] [ 7304.145260] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7304.145911] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7304.146015] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7304.146047] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7304.146143] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7304.151521] ? hrtimer_interrupt+0x128/0x2c0 [ 7304.153485] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7304.159757] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7304.161585] ? do_raw_spin_unlock+0x75/0x190 [ 7304.161666] ? _raw_spin_unlock+0x12/0x30 [ 7304.161676] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7304.161710] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7304.161732] ll_atomic_open+0x36a/0xd70 [lustre] [ 7304.161846] lookup_open+0x356/0xb40 [ 7304.161856] do_last+0x313/0xfc0 [ 7304.161861] ? nd_jump_root+0xe5/0x160 [ 7304.161865] ? path_init+0x437/0x520 [ 7304.161869] path_openat+0xf7/0x500 [ 7304.161874] ? do_raw_spin_unlock+0x75/0x190 [ 7304.161878] do_filp_open+0x99/0x140 [ 7304.161884] ? getname_flags+0x6e/0x330 [ 7304.161888] ? __check_object_size+0xff/0x256 [ 7304.161893] ? do_raw_spin_unlock+0x75/0x190 [ 7304.161897] ? _raw_spin_unlock+0x12/0x30 [ 7304.161900] do_sys_openat2+0x2b4/0x410 [ 7304.161904] do_sys_open+0x73/0xa0 [ 7304.161907] __x64_sys_openat+0x24/0x30 [ 7304.161910] do_syscall_64+0xc1/0x440 [ 7304.161916] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7304.161921] RIP: 0033:0x7f53425a5332 [ 7304.161930] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7304.161934] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7304.161939] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7304.161941] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7304.161943] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7304.161945] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7304.161947] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7304.486749] ? apic_timer_interrupt+0xf/0x20 [ 7304.491130] [ 7304.493330] ? native_safe_halt+0x2e/0x50 [ 7304.496397] kvm_wait+0xcd/0xe0 [ 7304.497787] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7304.500382] do_raw_spin_lock+0x177/0x190 [ 7304.504497] _raw_spin_lock+0x21/0x30 [ 7304.506715] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7304.509215] obd_mod_free+0x46/0x140 [ptlrpc] [ 7304.512909] mdc_commit_open+0x91/0xb0 [mdc] [ 7304.516454] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7304.522315] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7304.524217] after_reply+0xe2c/0x1780 [ptlrpc] [ 7304.531891] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7304.533519] ? __next_timer_interrupt+0x160/0x160 [ 7304.540081] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7304.544374] ? do_wait_intr+0xf0/0xf0 [ 7304.549943] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7304.552190] kthread+0x1d1/0x200 [ 7304.553323] ? set_kthread_struct+0x70/0x70 [ 7304.556186] ret_from_fork+0x1f/0x30 [ 7324.052331] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 7324.056006] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7324.075256] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7324.079801] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7324.083391] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7324.086430] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7324.088317] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7324.096611] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7324.100416] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7324.103588] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7324.106847] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7324.110225] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7324.112905] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7324.115512] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7324.118772] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7324.121206] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7324.124243] Call Trace: [ 7324.125312] [ 7324.126477] ? show_regs.cold.9+0x22/0x2f [ 7324.128352] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7324.129775] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7324.131693] ? watchdog+0x40/0x40 [ 7324.133196] ? hrtimer_interrupt+0x128/0x2c0 [ 7324.136498] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7324.138161] ? apic_timer_interrupt+0xf/0x20 [ 7324.139949] [ 7324.140639] ? native_safe_halt+0x2e/0x50 [ 7324.143114] kvm_wait+0xcd/0xe0 [ 7324.144279] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7324.146073] ? newidle_balance.isra.93+0x207/0x580 [ 7324.147568] do_raw_spin_lock+0x177/0x190 [ 7324.149325] _raw_spin_lock+0x21/0x30 [ 7324.150530] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7324.152462] process_one_work+0x2c8/0x700 [ 7324.154238] worker_thread+0x296/0x6e0 [ 7324.155505] ? rescuer_thread+0x570/0x570 [ 7324.156823] kthread+0x1d1/0x200 [ 7324.157861] ? set_kthread_struct+0x70/0x70 [ 7324.159422] ret_from_fork+0x1f/0x30 [ 7328.063502] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 7328.069079] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7328.085382] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7328.089104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7328.091807] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7328.094166] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7328.100966] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7328.102941] RAX: 0000000000000c08 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7328.105148] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7328.107555] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7328.110094] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7328.113054] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7328.116815] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7328.121048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7328.123838] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7328.126833] Call Trace: [ 7328.128483] [ 7328.129510] ? show_regs.cold.9+0x22/0x2f [ 7328.131176] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7328.133952] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7328.136145] ? watchdog+0x40/0x40 [ 7328.137753] ? hrtimer_interrupt+0x128/0x2c0 [ 7328.139737] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7328.142844] ? apic_timer_interrupt+0xf/0x20 [ 7328.147163] [ 7328.148423] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7328.152036] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7328.156082] do_raw_spin_lock+0x177/0x190 [ 7328.158141] _raw_spin_lock+0x21/0x30 [ 7328.160100] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7328.163577] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7328.166427] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7328.169685] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7328.173023] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7328.176085] ? woken_wake_function+0x30/0x30 [ 7328.178410] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7328.182137] kthread+0x1d1/0x200 [ 7328.183721] ? set_kthread_struct+0x70/0x70 [ 7328.186057] ret_from_fork+0x1f/0x30 [ 7329.764356] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4692s! [ 7329.770458] Showing busy workqueues and worker pools: [ 7329.773031] workqueue events: flags=0x0 [ 7329.775798] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7329.781971] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7329.799015] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7329.808042] pending: cpu_latency_work [ptlrpc] [ 7329.811092] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7329.817876] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7329.823136] workqueue events_long: flags=0x0 [ 7329.829036] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7329.834569] pending: lu_objects_flush [obdclass] [ 7329.841304] workqueue events_unbound: flags=0x2 [ 7329.847013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7329.857716] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7329.864127] workqueue events_power_efficient: flags=0x80 [ 7329.866927] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7329.871086] pending: check_lifetime, do_cache_clean [ 7329.877541] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7329.882040] pending: neigh_periodic_work, neigh_periodic_work [ 7329.884760] workqueue mm_percpu_wq: flags=0x8 [ 7329.890691] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7329.899854] pending: vmstat_update, lru_add_drain_per_cpu [ 7329.902935] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7329.907922] pending: vmstat_update, lru_add_drain_per_cpu [ 7329.912391] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7329.916741] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7329.921404] workqueue xprtiod: flags=0xa [ 7329.923374] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7329.929723] pending: xprt_autoclose, xs_error_handle [ 7329.933415] workqueue ipv6_addrconf: flags=0x40008 [ 7329.938494] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7329.940804] pending: addrconf_verify_work [ 7329.953732] workqueue knbd0-recv: flags=0x1a [ 7329.957781] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7329.966818] in-flight: 651:recv_work [ 7329.968879] workqueue ptlrpc_pinger: flags=0x2 [ 7329.971909] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7329.979455] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7329.983865] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4692s workers=4 idle: 15979 31824 8 [ 7329.988395] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7320s workers=2 idle: 654 [ 7332.057994] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7332.064178] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7332.065819] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7332.065897] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7332.065905] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7332.065907] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7332.065922] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7332.065926] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7332.065931] RAX: 0000000000005a0d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7332.065933] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7332.065935] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7332.065938] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7332.065940] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7332.065943] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7332.065946] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7332.065949] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7332.065958] Call Trace: [ 7332.065964] [ 7332.065970] ? show_regs.cold.9+0x22/0x2f [ 7332.065978] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7332.065984] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7332.065989] ? watchdog+0x40/0x40 [ 7332.065994] ? hrtimer_interrupt+0x128/0x2c0 [ 7332.065998] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7332.066007] ? apic_timer_interrupt+0xf/0x20 [ 7332.066012] [ 7332.066015] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7332.066022] do_raw_spin_lock+0x177/0x190 [ 7332.066026] _raw_spin_lock+0x21/0x30 [ 7332.066031] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7332.069911] ? do_raw_spin_unlock+0x75/0x190 [ 7332.069925] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7332.070363] ? do_wait_intr+0xf0/0xf0 [ 7332.070372] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7332.070640] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7332.070876] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7332.071087] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7332.071730] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7332.071776] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7332.071808] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7332.071837] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7332.071923] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7332.072744] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7332.072782] lmv_intent_open+0x338/0x1150 [lmv] [ 7332.072814] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7332.072915] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7332.072997] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7332.073021] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7332.073107] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7332.073725] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7332.073812] ? do_raw_spin_unlock+0x75/0x190 [ 7332.073822] ? _raw_spin_unlock+0x12/0x30 [ 7332.073830] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7332.073859] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7332.073881] ll_atomic_open+0x36a/0xd70 [lustre] [ 7332.073967] lookup_open+0x356/0xb40 [ 7332.073978] do_last+0x313/0xfc0 [ 7332.073983] ? nd_jump_root+0xe5/0x160 [ 7332.073988] ? path_init+0x437/0x520 [ 7332.073994] path_openat+0xf7/0x500 [ 7332.073999] ? do_raw_spin_unlock+0x75/0x190 [ 7332.074004] do_filp_open+0x99/0x140 [ 7332.074011] ? getname_flags+0x6e/0x330 [ 7332.074016] ? __check_object_size+0xff/0x256 [ 7332.074025] ? do_raw_spin_unlock+0x75/0x190 [ 7332.074029] ? _raw_spin_unlock+0x12/0x30 [ 7332.074033] do_sys_openat2+0x2b4/0x410 [ 7332.074038] do_sys_open+0x73/0xa0 [ 7332.074043] __x64_sys_openat+0x24/0x30 [ 7332.074047] do_syscall_64+0xc1/0x440 [ 7332.074054] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7332.074061] RIP: 0033:0x7f53425a5332 [ 7332.074115] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7332.074187] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7332.074192] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7332.074195] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7332.074197] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7332.074199] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7332.074202] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7332.670309] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7332.728870] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7332.739994] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7332.753541] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7332.762333] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7332.786636] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7332.792376] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7332.798936] RDX: 0000000000003e83 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7332.808082] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7332.812804] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7332.823992] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7332.828892] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7332.837714] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7332.840012] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7332.846498] Call Trace: [ 7332.851649] [ 7332.852626] ? show_regs.cold.9+0x22/0x2f [ 7332.855900] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7332.860087] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7332.869499] ? watchdog+0x40/0x40 [ 7332.871385] ? hrtimer_interrupt+0x128/0x2c0 [ 7332.879005] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7332.881059] ? apic_timer_interrupt+0xf/0x20 [ 7332.886790] [ 7332.887650] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7332.894919] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7332.900382] do_raw_spin_lock+0x177/0x190 [ 7332.902624] _raw_spin_lock+0x21/0x30 [ 7332.903600] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7332.907740] obd_mod_free+0x46/0x140 [ptlrpc] [ 7332.913812] mdc_commit_open+0x91/0xb0 [mdc] [ 7332.915886] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7332.921360] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7332.927774] after_reply+0xe2c/0x1780 [ptlrpc] [ 7332.931438] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7332.935742] ? __next_timer_interrupt+0x160/0x160 [ 7332.939773] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7332.946966] ? do_wait_intr+0xf0/0xf0 [ 7332.948257] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7332.954268] kthread+0x1d1/0x200 [ 7332.955432] ? set_kthread_struct+0x70/0x70 [ 7332.958274] ret_from_fork+0x1f/0x30 [ 7356.053774] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 7356.057722] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7356.058757] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 7356.117555] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7356.117570] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7356.117577] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7356.122673] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7356.123900] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7356.123909] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7356.123912] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7356.123927] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7356.123932] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7356.123936] RAX: 00000000000078a6 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7356.123939] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7356.123942] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7356.123944] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7356.123947] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7356.123950] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7356.123953] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7356.123955] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7356.123964] Call Trace: [ 7356.123970] [ 7356.123977] ? show_regs.cold.9+0x22/0x2f [ 7356.123987] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7356.123994] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7356.123999] ? watchdog+0x40/0x40 [ 7356.124003] ? hrtimer_interrupt+0x128/0x2c0 [ 7356.124007] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7356.124016] ? apic_timer_interrupt+0xf/0x20 [ 7356.124021] [ 7356.124025] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7356.124030] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7356.124036] do_raw_spin_lock+0x177/0x190 [ 7356.124041] _raw_spin_lock+0x21/0x30 [ 7356.124046] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7356.124328] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7356.124522] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7356.125909] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7356.126151] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7356.126387] ? woken_wake_function+0x30/0x30 [ 7356.126397] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7356.126631] kthread+0x1d1/0x200 [ 7356.126638] ? set_kthread_struct+0x70/0x70 [ 7356.126642] ret_from_fork+0x1f/0x30 [ 7356.488366] [ 7356.489231] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7356.503036] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7356.525113] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7356.528051] RAX: 000000000000298b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7356.534727] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7356.544646] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7356.550670] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7356.553948] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7356.556967] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7356.560683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7356.563123] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7356.568342] Call Trace: [ 7356.569568] [ 7356.570488] ? show_regs.cold.9+0x22/0x2f [ 7356.572507] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7356.575280] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7356.577337] ? watchdog+0x40/0x40 [ 7356.579085] ? hrtimer_interrupt+0x128/0x2c0 [ 7356.581268] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7356.587549] ? apic_timer_interrupt+0xf/0x20 [ 7356.594003] [ 7356.597364] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7356.602950] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7356.608097] ? newidle_balance.isra.93+0x207/0x580 [ 7356.611554] do_raw_spin_lock+0x177/0x190 [ 7356.617180] _raw_spin_lock+0x21/0x30 [ 7356.619105] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7356.624259] process_one_work+0x2c8/0x700 [ 7356.628372] worker_thread+0x296/0x6e0 [ 7356.633912] ? rescuer_thread+0x570/0x570 [ 7356.638269] kthread+0x1d1/0x200 [ 7356.643540] ? set_kthread_struct+0x70/0x70 [ 7356.646421] ret_from_fork+0x1f/0x30 [ 7360.055083] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7360.058682] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7360.063501] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7360.063582] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7360.063588] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7360.063590] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7360.063601] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7360.063605] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7360.063609] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7360.063611] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7360.063614] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7360.063616] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7360.063618] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7360.063621] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7360.066170] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7360.066247] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7360.066254] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7360.066257] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7360.066270] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7360.066274] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7360.066278] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7360.066281] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7360.066285] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7360.097545] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7360.104259] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7360.104270] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7360.104273] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7360.104277] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7360.104279] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7360.104288] Call Trace: [ 7360.104294] [ 7360.104299] ? show_regs.cold.9+0x22/0x2f [ 7360.104313] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7360.104319] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7360.104324] ? watchdog+0x40/0x40 [ 7360.104328] ? hrtimer_interrupt+0x128/0x2c0 [ 7360.104332] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7360.104397] ? apic_timer_interrupt+0xf/0x20 [ 7360.104404] [ 7360.104407] ? native_safe_halt+0x2e/0x50 [ 7360.104411] kvm_wait+0xcd/0xe0 [ 7360.104418] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7360.104425] do_raw_spin_lock+0x177/0x190 [ 7360.104430] _raw_spin_lock+0x21/0x30 [ 7360.104437] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7360.113412] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7360.113429] Call Trace: [ 7360.113437] [ 7360.113442] ? show_regs.cold.9+0x22/0x2f [ 7360.113456] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7360.113462] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7360.113467] ? watchdog+0x40/0x40 [ 7360.113471] ? hrtimer_interrupt+0x128/0x2c0 [ 7360.113476] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7360.113483] ? apic_timer_interrupt+0xf/0x20 [ 7360.113488] [ 7360.113491] ? native_safe_halt+0x2e/0x50 [ 7360.113495] kvm_wait+0xcd/0xe0 [ 7360.113502] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7360.113512] do_raw_spin_lock+0x177/0x190 [ 7360.113516] _raw_spin_lock+0x21/0x30 [ 7360.113520] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7360.115935] obd_mod_free+0x46/0x140 [ptlrpc] [ 7360.138983] ? do_raw_spin_unlock+0x75/0x190 [ 7360.139003] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7360.139290] ? do_wait_intr+0xf0/0xf0 [ 7360.139300] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7360.139604] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7360.139882] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7360.140091] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7360.140205] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7360.140242] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7360.140273] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7360.140304] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7360.140391] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7360.143441] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7360.143481] lmv_intent_open+0x338/0x1150 [lmv] [ 7360.143519] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7360.149562] mdc_commit_open+0x91/0xb0 [mdc] [ 7360.153951] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7360.154336] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7360.154369] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7360.154559] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7360.159432] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7360.163888] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7360.164130] ? do_raw_spin_unlock+0x75/0x190 [ 7360.164138] ? _raw_spin_unlock+0x12/0x30 [ 7360.164145] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7360.164239] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7360.164261] ll_atomic_open+0x36a/0xd70 [lustre] [ 7360.164348] lookup_open+0x356/0xb40 [ 7360.164359] do_last+0x313/0xfc0 [ 7360.164364] ? nd_jump_root+0xe5/0x160 [ 7360.164368] ? path_init+0x437/0x520 [ 7360.164373] path_openat+0xf7/0x500 [ 7360.164378] ? do_raw_spin_unlock+0x75/0x190 [ 7360.164382] do_filp_open+0x99/0x140 [ 7360.164389] ? getname_flags+0x6e/0x330 [ 7360.164394] ? __check_object_size+0xff/0x256 [ 7360.164402] ? do_raw_spin_unlock+0x75/0x190 [ 7360.164407] ? _raw_spin_unlock+0x12/0x30 [ 7360.164411] do_sys_openat2+0x2b4/0x410 [ 7360.164416] do_sys_open+0x73/0xa0 [ 7360.164420] __x64_sys_openat+0x24/0x30 [ 7360.164424] do_syscall_64+0xc1/0x440 [ 7360.164430] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7360.164436] RIP: 0033:0x7f53425a5332 [ 7360.164443] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7360.164447] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7360.164452] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7360.164455] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7360.164457] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7360.164459] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7360.164461] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7360.685970] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7360.691270] after_reply+0xe2c/0x1780 [ptlrpc] [ 7360.695981] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7360.700592] ? __next_timer_interrupt+0x160/0x160 [ 7360.704318] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7360.707127] ? do_wait_intr+0xf0/0xf0 [ 7360.709239] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7360.713877] kthread+0x1d1/0x200 [ 7360.716295] ? set_kthread_struct+0x70/0x70 [ 7360.720084] ret_from_fork+0x1f/0x30 [ 7360.725061] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4723s! [ 7360.730130] Showing busy workqueues and worker pools: [ 7360.733097] workqueue events: flags=0x0 [ 7360.735180] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7360.740991] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7360.751100] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7360.756015] pending: cpu_latency_work [ptlrpc] [ 7360.760229] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7360.765391] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7360.774978] workqueue events_long: flags=0x0 [ 7360.780021] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7360.786001] pending: lu_objects_flush [obdclass] [ 7360.790944] workqueue events_unbound: flags=0x2 [ 7360.792541] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7360.795482] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7360.800331] workqueue events_power_efficient: flags=0x80 [ 7360.802919] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7360.808039] pending: check_lifetime, do_cache_clean [ 7360.816476] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7360.821814] pending: neigh_periodic_work, neigh_periodic_work [ 7360.825274] workqueue mm_percpu_wq: flags=0x8 [ 7360.827912] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7360.831645] pending: vmstat_update, lru_add_drain_per_cpu [ 7360.834300] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7360.839064] pending: vmstat_update, lru_add_drain_per_cpu [ 7360.843066] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7360.847196] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7360.854040] workqueue xprtiod: flags=0xa [ 7360.856917] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7360.860635] pending: xprt_autoclose, xs_error_handle [ 7360.865847] workqueue ipv6_addrconf: flags=0x40008 [ 7360.868914] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7360.875086] pending: addrconf_verify_work [ 7360.878831] workqueue knbd0-recv: flags=0x1a [ 7360.882730] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7360.889693] in-flight: 651:recv_work [ 7360.894956] workqueue ptlrpc_pinger: flags=0x2 [ 7360.897358] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7360.900816] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7360.906903] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4723s workers=4 idle: 15979 31824 8 [ 7360.913533] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7351s workers=2 idle: 654 [ 7374.565060] rcu: INFO: rcu_sched self-detected stall on CPU [ 7374.567860] rcu: 2-....: (2857572 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 7374.574292] (t=5280139 jiffies g=274185 q=65087) [ 7374.577076] rcu: rcu_sched kthread starved for 4738505 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 7374.582897] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7374.588533] rcu: RCU grace-period kthread stack dump: [ 7374.590216] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7374.593039] Call Trace: [ 7374.597870] __schedule+0x351/0xcb0 [ 7374.599209] schedule+0xc0/0x180 [ 7374.601161] schedule_timeout+0xb4/0x190 [ 7374.604668] ? do_raw_spin_unlock+0x75/0x190 [ 7374.607796] ? __next_timer_interrupt+0x160/0x160 [ 7374.610622] ? prepare_to_swait_event+0xf3/0x140 [ 7374.611994] rcu_gp_kthread+0x8dd/0x15b0 [ 7374.613866] ? force_qs_rnp+0x2c0/0x2c0 [ 7374.615230] kthread+0x1d1/0x200 [ 7374.616392] ? set_kthread_struct+0x70/0x70 [ 7374.621816] ret_from_fork+0x1f/0x30 [ 7374.624331] rcu: Stack dump where RCU GP kthread last ran: [ 7374.628107] Sending NMI from CPU 2 to CPUs 0: [ 7374.636228] NMI backtrace for cpu 0 [ 7374.636235] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7374.636237] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7374.636238] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7374.636240] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7374.636242] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7374.636244] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 [ 7374.636245] RAX: 00000000000079d1 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7374.636247] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7374.636248] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7374.636249] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7374.636250] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7374.636252] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7374.636253] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7374.636254] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7374.636255] Call Trace: [ 7374.636256] [ 7374.636257] ? show_regs.cold.9+0x22/0x2f [ 7374.636258] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7374.636259] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7374.636259] ? nmi_handle+0xbd/0x210 [ 7374.636261] ? default_do_nmi+0x4b/0x1b0 [ 7374.636261] ? do_nmi+0x280/0x380 [ 7374.636262] ? end_repeat_nmi+0x16/0x1f [ 7374.636263] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7374.636264] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7374.636266] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7374.636266] [ 7374.636267] ? newidle_balance.isra.93+0x207/0x580 [ 7374.636268] do_raw_spin_lock+0x177/0x190 [ 7374.636269] _raw_spin_lock+0x21/0x30 [ 7374.636270] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7374.636271] process_one_work+0x2c8/0x700 [ 7374.636272] worker_thread+0x296/0x6e0 [ 7374.636273] ? rescuer_thread+0x570/0x570 [ 7374.636274] kthread+0x1d1/0x200 [ 7374.636275] ? set_kthread_struct+0x70/0x70 [ 7374.636276] ret_from_fork+0x1f/0x30 [ 7374.637071] Sending NMI from CPU 2 to CPUs 1: [ 7374.759652] NMI backtrace for cpu 1 [ 7374.759661] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7374.759664] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7374.759665] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7374.759667] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7374.759669] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [ 7374.759672] RAX: 0000000000003686 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7374.759674] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7374.759675] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7374.759677] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7374.759679] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7374.759680] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7374.759681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7374.759683] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7374.759684] Call Trace: [ 7374.759685] [ 7374.759685] ? show_regs.cold.9+0x22/0x2f [ 7374.759686] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7374.759687] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7374.759688] ? nmi_handle+0xbd/0x210 [ 7374.759689] ? default_do_nmi+0x4b/0x1b0 [ 7374.759690] ? do_nmi+0x280/0x380 [ 7374.759691] ? end_repeat_nmi+0x16/0x1f [ 7374.759692] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7374.759693] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7374.759694] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7374.759695] [ 7374.759696] do_raw_spin_lock+0x177/0x190 [ 7374.759697] _raw_spin_lock+0x21/0x30 [ 7374.759698] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7374.759699] ? do_raw_spin_unlock+0x75/0x190 [ 7374.759700] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7374.759701] ? do_wait_intr+0xf0/0xf0 [ 7374.759703] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7374.759704] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7374.759705] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7374.759706] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7374.759707] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7374.759709] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7374.759710] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7374.759711] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7374.759712] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7374.759714] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7374.759715] lmv_intent_open+0x338/0x1150 [lmv] [ 7374.759716] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7374.759717] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7374.759719] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7374.759720] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7374.759721] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7374.759723] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7374.759724] ? do_raw_spin_unlock+0x75/0x190 [ 7374.759725] ? _raw_spin_unlock+0x12/0x30 [ 7374.759726] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7374.759727] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7374.759728] ll_atomic_open+0x36a/0xd70 [lustre] [ 7374.759729] lookup_open+0x356/0xb40 [ 7374.759730] do_last+0x313/0xfc0 [ 7374.759731] ? nd_jump_root+0xe5/0x160 [ 7374.759732] ? path_init+0x437/0x520 [ 7374.759733] path_openat+0xf7/0x500 [ 7374.759734] ? do_raw_spin_unlock+0x75/0x190 [ 7374.759735] do_filp_open+0x99/0x140 [ 7374.759736] ? getname_flags+0x6e/0x330 [ 7374.759737] ? __check_object_size+0xff/0x256 [ 7374.759738] ? do_raw_spin_unlock+0x75/0x190 [ 7374.759739] ? _raw_spin_unlock+0x12/0x30 [ 7374.759740] do_sys_openat2+0x2b4/0x410 [ 7374.759741] do_sys_open+0x73/0xa0 [ 7374.759742] __x64_sys_openat+0x24/0x30 [ 7374.759743] do_syscall_64+0xc1/0x440 [ 7374.759744] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7374.759745] RIP: 0033:0x7f53425a5332 [ 7374.759746] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7374.759749] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7374.759751] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7374.759753] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7374.759754] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7374.759756] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7374.759757] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7374.760470] NMI backtrace for cpu 2 [ 7375.135871] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7375.149904] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7375.159742] Call Trace: [ 7375.163760] [ 7375.169762] ? dump_stack+0xbb/0x10e [ 7375.172630] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7375.178972] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7375.183205] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7375.190636] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7375.199079] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7375.201575] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7375.203098] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7375.210520] ? update_process_times+0x6d/0xc0 [ 7375.218027] ? tick_sched_handle.isra.18+0x27/0x90 [ 7375.226306] ? tick_sched_timer+0x53/0xb0 [ 7375.235855] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7375.238402] ? tick_sched_handle.isra.18+0x90/0x90 [ 7375.243701] ? hrtimer_interrupt+0x128/0x2c0 [ 7375.252457] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7375.262240] ? apic_timer_interrupt+0xf/0x20 [ 7375.272059] [ 7375.276241] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7375.288883] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7375.301129] ? do_raw_spin_lock+0x177/0x190 [ 7375.302733] ? _raw_spin_lock+0x21/0x30 [ 7375.314240] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7375.319822] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7375.330785] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7375.342139] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7375.352174] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7375.356300] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7375.357879] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7375.377082] ? __next_timer_interrupt+0x160/0x160 [ 7375.380854] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7375.388895] ? do_wait_intr+0xf0/0xf0 [ 7375.394633] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7375.398811] ? kthread+0x1d1/0x200 [ 7375.402732] ? set_kthread_struct+0x70/0x70 [ 7375.414745] ? ret_from_fork+0x1f/0x30 [ 7384.058580] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 7384.082623] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7384.175443] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7384.182514] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7384.187959] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7384.192318] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7384.202741] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7384.208953] RAX: 0000000000006e87 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7384.212873] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7384.220499] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7384.228070] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7384.242825] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7384.250195] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7384.272136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7384.285781] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7384.306897] Call Trace: [ 7384.308930] [ 7384.312759] ? show_regs.cold.9+0x22/0x2f [ 7384.326826] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7384.328503] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7384.336505] ? watchdog+0x40/0x40 [ 7384.344390] ? hrtimer_interrupt+0x128/0x2c0 [ 7384.352440] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7384.354114] ? apic_timer_interrupt+0xf/0x20 [ 7384.372943] [ 7384.373969] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7384.381803] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7384.390810] do_raw_spin_lock+0x177/0x190 [ 7384.396028] _raw_spin_lock+0x21/0x30 [ 7384.397432] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7384.403020] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7384.411459] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7384.422011] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7384.428030] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7384.440103] ? woken_wake_function+0x30/0x30 [ 7384.443601] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7384.446762] kthread+0x1d1/0x200 [ 7384.448039] ? set_kthread_struct+0x70/0x70 [ 7384.459417] ret_from_fork+0x1f/0x30 [ 7391.205615] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4754s! [ 7391.213044] Showing busy workqueues and worker pools: [ 7391.220413] workqueue events: flags=0x0 [ 7391.226584] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7391.233579] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7391.245194] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7391.259451] pending: cpu_latency_work [ptlrpc] [ 7391.262908] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7391.266673] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7391.272133] workqueue events_long: flags=0x0 [ 7391.283101] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7391.287454] pending: lu_objects_flush [obdclass] [ 7391.293453] workqueue events_unbound: flags=0x2 [ 7391.295938] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7391.300971] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7391.303012] workqueue events_power_efficient: flags=0x80 [ 7391.305881] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7391.309802] pending: check_lifetime, do_cache_clean [ 7391.312122] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7391.315552] pending: neigh_periodic_work, neigh_periodic_work [ 7391.318460] workqueue mm_percpu_wq: flags=0x8 [ 7391.320539] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7391.323874] pending: vmstat_update, lru_add_drain_per_cpu [ 7391.326000] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7391.329319] pending: vmstat_update, lru_add_drain_per_cpu [ 7391.331336] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7391.333513] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7391.335614] workqueue xprtiod: flags=0xa [ 7391.337537] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7391.340535] pending: xprt_autoclose, xs_error_handle [ 7391.345413] workqueue ipv6_addrconf: flags=0x40008 [ 7391.346869] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7391.352459] pending: addrconf_verify_work [ 7391.357621] workqueue knbd0-recv: flags=0x1a [ 7391.359021] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7391.363557] in-flight: 651:recv_work [ 7391.365851] workqueue ptlrpc_pinger: flags=0x2 [ 7391.371779] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7391.375296] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7391.382013] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4754s workers=4 idle: 15979 31824 8 [ 7391.396292] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7381s workers=2 idle: 654 [ 7400.060016] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7400.060492] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7400.067340] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7400.067438] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7400.067445] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7400.067447] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7400.067462] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7400.067466] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7400.067471] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7400.067473] RDX: 0000000000001ceb RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7400.067475] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7400.067478] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7400.067481] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7400.067484] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7400.067487] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7400.067489] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7400.067499] Call Trace: [ 7400.067506] [ 7400.067513] ? show_regs.cold.9+0x22/0x2f [ 7400.067523] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7400.067530] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7400.067534] ? watchdog+0x40/0x40 [ 7400.067539] ? hrtimer_interrupt+0x128/0x2c0 [ 7400.067544] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7400.067553] ? apic_timer_interrupt+0xf/0x20 [ 7400.067557] [ 7400.067561] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7400.067566] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7400.067570] do_raw_spin_lock+0x177/0x190 [ 7400.067575] _raw_spin_lock+0x21/0x30 [ 7400.067581] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7400.067862] obd_mod_free+0x46/0x140 [ptlrpc] [ 7400.068071] mdc_commit_open+0x91/0xb0 [mdc] [ 7400.068109] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7400.068321] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7400.068535] after_reply+0xe2c/0x1780 [ptlrpc] [ 7400.068741] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7400.069400] ? __next_timer_interrupt+0x160/0x160 [ 7400.069410] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7400.069681] ? do_wait_intr+0xf0/0xf0 [ 7400.069690] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7400.069907] kthread+0x1d1/0x200 [ 7400.069913] ? set_kthread_struct+0x70/0x70 [ 7400.069918] ret_from_fork+0x1f/0x30 [ 7400.378317] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7400.421897] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7400.429200] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7400.441060] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7400.445149] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7400.459207] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7400.464182] RAX: 0000000000007c5a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7400.467502] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7400.473392] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7400.483783] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7400.486718] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7400.504399] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7400.516610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7400.518624] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7400.524140] Call Trace: [ 7400.525392] [ 7400.526248] ? show_regs.cold.9+0x22/0x2f [ 7400.532156] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7400.534351] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7400.536501] ? watchdog+0x40/0x40 [ 7400.537878] ? hrtimer_interrupt+0x128/0x2c0 [ 7400.539518] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7400.541512] ? apic_timer_interrupt+0xf/0x20 [ 7400.543279] [ 7400.544081] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7400.545752] do_raw_spin_lock+0x177/0x190 [ 7400.551336] _raw_spin_lock+0x21/0x30 [ 7400.554145] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7400.557559] ? do_raw_spin_unlock+0x75/0x190 [ 7400.559620] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7400.562574] ? do_wait_intr+0xf0/0xf0 [ 7400.564036] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7400.566388] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7400.569312] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7400.572683] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7400.575142] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7400.577387] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7400.579424] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7400.581394] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7400.583582] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7400.586200] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7400.588429] lmv_intent_open+0x338/0x1150 [lmv] [ 7400.591104] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7400.593385] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7400.597086] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7400.602012] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7400.603610] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7400.606496] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7400.610416] ? do_raw_spin_unlock+0x75/0x190 [ 7400.613929] ? _raw_spin_unlock+0x12/0x30 [ 7400.618454] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7400.621752] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7400.626107] ll_atomic_open+0x36a/0xd70 [lustre] [ 7400.628243] lookup_open+0x356/0xb40 [ 7400.630290] do_last+0x313/0xfc0 [ 7400.632379] ? nd_jump_root+0xe5/0x160 [ 7400.641521] ? path_init+0x437/0x520 [ 7400.644589] path_openat+0xf7/0x500 [ 7400.652421] ? do_raw_spin_unlock+0x75/0x190 [ 7400.659349] do_filp_open+0x99/0x140 [ 7400.663803] ? getname_flags+0x6e/0x330 [ 7400.667631] ? __check_object_size+0xff/0x256 [ 7400.670163] ? do_raw_spin_unlock+0x75/0x190 [ 7400.676367] ? _raw_spin_unlock+0x12/0x30 [ 7400.677676] do_sys_openat2+0x2b4/0x410 [ 7400.682992] do_sys_open+0x73/0xa0 [ 7400.684692] __x64_sys_openat+0x24/0x30 [ 7400.690533] do_syscall_64+0xc1/0x440 [ 7400.692180] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7400.700820] RIP: 0033:0x7f53425a5332 [ 7400.702434] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7400.717459] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7400.720801] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7400.727042] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7400.733342] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7400.741480] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7400.753792] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7412.058785] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 7412.068104] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7412.107577] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7412.119356] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7412.128018] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7412.129919] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7412.143769] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7412.151190] RAX: 0000000000006a2d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7412.154393] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7412.158526] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7412.161705] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7412.175535] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7412.180967] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7412.193627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7412.199239] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7412.207540] Call Trace: [ 7412.210260] [ 7412.211291] ? show_regs.cold.9+0x22/0x2f [ 7412.213947] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7412.220048] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7412.223948] ? watchdog+0x40/0x40 [ 7412.225334] ? hrtimer_interrupt+0x128/0x2c0 [ 7412.230067] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7412.232054] ? apic_timer_interrupt+0xf/0x20 [ 7412.236997] [ 7412.242057] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7412.245893] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7412.247252] do_raw_spin_lock+0x177/0x190 [ 7412.249529] _raw_spin_lock+0x21/0x30 [ 7412.254978] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7412.259556] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7412.261971] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7412.265806] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7412.268465] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7412.274632] ? woken_wake_function+0x30/0x30 [ 7412.277423] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7412.282886] kthread+0x1d1/0x200 [ 7412.285582] ? set_kthread_struct+0x70/0x70 [ 7412.287377] ret_from_fork+0x1f/0x30 [ 7416.056474] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 7416.063154] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7416.112405] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7416.125750] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7416.134220] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7416.141764] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7416.143932] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7416.180139] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7416.183255] RAX: 0000000000007976 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7416.189028] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7416.194298] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7416.208831] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7416.214011] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7416.218388] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7416.226168] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7416.232734] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7416.239830] Call Trace: [ 7416.244117] [ 7416.246419] ? show_regs.cold.9+0x22/0x2f [ 7416.248920] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7416.251416] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7416.256561] ? watchdog+0x40/0x40 [ 7416.259201] ? hrtimer_interrupt+0x128/0x2c0 [ 7416.261842] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7416.263583] ? apic_timer_interrupt+0xf/0x20 [ 7416.268329] [ 7416.269614] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7416.276315] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7416.281028] ? newidle_balance.isra.93+0x207/0x580 [ 7416.283851] do_raw_spin_lock+0x177/0x190 [ 7416.286253] _raw_spin_lock+0x21/0x30 [ 7416.289971] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7416.296547] process_one_work+0x2c8/0x700 [ 7416.301320] worker_thread+0x296/0x6e0 [ 7416.305976] ? rescuer_thread+0x570/0x570 [ 7416.307810] kthread+0x1d1/0x200 [ 7416.312389] ? set_kthread_struct+0x70/0x70 [ 7416.314868] ret_from_fork+0x1f/0x30 [ 7421.411905] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4784s! [ 7421.421792] Showing busy workqueues and worker pools: [ 7421.424629] workqueue events: flags=0x0 [ 7421.431439] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7421.441782] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7421.453580] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7421.464768] pending: cpu_latency_work [ptlrpc] [ 7421.467710] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7421.476142] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7421.487864] workqueue events_long: flags=0x0 [ 7421.493346] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7421.498151] pending: lu_objects_flush [obdclass] [ 7421.500168] workqueue events_unbound: flags=0x2 [ 7421.510675] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7421.519718] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7421.521700] workqueue events_power_efficient: flags=0x80 [ 7421.536694] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7421.544144] pending: check_lifetime, do_cache_clean [ 7421.556480] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7421.566003] pending: neigh_periodic_work, neigh_periodic_work [ 7421.579316] workqueue mm_percpu_wq: flags=0x8 [ 7421.580864] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7421.597176] pending: vmstat_update, lru_add_drain_per_cpu [ 7421.608407] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7421.622373] pending: vmstat_update, lru_add_drain_per_cpu [ 7421.630233] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7421.638277] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7421.651167] workqueue xprtiod: flags=0xa [ 7421.656635] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7421.669352] pending: xprt_autoclose, xs_error_handle [ 7421.674334] workqueue ipv6_addrconf: flags=0x40008 [ 7421.680182] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7421.689041] pending: addrconf_verify_work [ 7421.692504] workqueue knbd0-recv: flags=0x1a [ 7421.697287] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7421.703093] in-flight: 651:recv_work [ 7421.704978] workqueue ptlrpc_pinger: flags=0x2 [ 7421.706835] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7421.709385] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7421.711972] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4784s workers=4 idle: 15979 31824 8 [ 7421.715166] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7412s workers=2 idle: 654 [ 7428.060716] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7428.061042] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7428.063319] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7428.063393] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7428.063400] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7428.063403] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7428.063417] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7428.063422] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7428.063426] RAX: 000000000000548d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7428.063428] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7428.063431] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7428.063433] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7428.063435] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7428.063449] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7428.063453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7428.063455] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7428.133131] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7428.133207] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7428.133216] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7428.133219] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7428.133235] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7428.133240] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7428.133244] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7428.133247] RDX: 000000000000488d RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7428.133249] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7428.133251] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7428.133253] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7428.133257] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7428.133259] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7428.133261] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7428.133270] Call Trace: [ 7428.133277] [ 7428.133283] ? show_regs.cold.9+0x22/0x2f [ 7428.133293] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7428.133300] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7428.133304] ? watchdog+0x40/0x40 [ 7428.133308] ? hrtimer_interrupt+0x128/0x2c0 [ 7428.133313] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7428.133320] ? apic_timer_interrupt+0xf/0x20 [ 7428.133326] [ 7428.133329] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7428.133334] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7428.133338] do_raw_spin_lock+0x177/0x190 [ 7428.133342] _raw_spin_lock+0x21/0x30 [ 7428.133348] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7428.133987] obd_mod_free+0x46/0x140 [ptlrpc] [ 7428.134182] mdc_commit_open+0x91/0xb0 [mdc] [ 7428.134223] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7428.134416] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7428.134795] after_reply+0xe2c/0x1780 [ptlrpc] [ 7428.134999] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7428.135196] ? __next_timer_interrupt+0x160/0x160 [ 7428.135206] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7428.135405] ? do_wait_intr+0xf0/0xf0 [ 7428.135415] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7428.135872] kthread+0x1d1/0x200 [ 7428.135880] ? set_kthread_struct+0x70/0x70 [ 7428.135884] ret_from_fork+0x1f/0x30 [ 7428.442531] Call Trace: [ 7428.443606] [ 7428.445203] ? show_regs.cold.9+0x22/0x2f [ 7428.447403] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7428.449542] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7428.454007] ? watchdog+0x40/0x40 [ 7428.456300] ? hrtimer_interrupt+0x128/0x2c0 [ 7428.461169] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7428.470896] ? apic_timer_interrupt+0xf/0x20 [ 7428.472223] [ 7428.482839] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7428.486181] do_raw_spin_lock+0x177/0x190 [ 7428.489149] _raw_spin_lock+0x21/0x30 [ 7428.495698] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7428.498416] ? do_raw_spin_unlock+0x75/0x190 [ 7428.501426] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7428.504148] ? do_wait_intr+0xf0/0xf0 [ 7428.509126] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7428.510920] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7428.513258] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7428.518104] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7428.522948] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7428.527839] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7428.531382] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7428.532922] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7428.538602] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7428.542033] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7428.543740] lmv_intent_open+0x338/0x1150 [lmv] [ 7428.555173] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7428.556872] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7428.563588] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7428.565276] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7428.575825] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7428.586020] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7428.587793] ? do_raw_spin_unlock+0x75/0x190 [ 7428.592862] ? _raw_spin_unlock+0x12/0x30 [ 7428.596187] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7428.601953] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7428.605726] ll_atomic_open+0x36a/0xd70 [lustre] [ 7428.613764] lookup_open+0x356/0xb40 [ 7428.615879] do_last+0x313/0xfc0 [ 7428.617250] ? nd_jump_root+0xe5/0x160 [ 7428.619068] ? path_init+0x437/0x520 [ 7428.620746] path_openat+0xf7/0x500 [ 7428.628659] ? do_raw_spin_unlock+0x75/0x190 [ 7428.637112] do_filp_open+0x99/0x140 [ 7428.642360] ? getname_flags+0x6e/0x330 [ 7428.644368] ? __check_object_size+0xff/0x256 [ 7428.645894] ? do_raw_spin_unlock+0x75/0x190 [ 7428.653318] ? _raw_spin_unlock+0x12/0x30 [ 7428.655332] do_sys_openat2+0x2b4/0x410 [ 7428.661518] do_sys_open+0x73/0xa0 [ 7428.662681] __x64_sys_openat+0x24/0x30 [ 7428.670609] do_syscall_64+0xc1/0x440 [ 7428.676870] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7428.685300] RIP: 0033:0x7f53425a5332 [ 7428.686777] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7428.712796] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7428.721918] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7428.726936] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7428.735240] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7428.743276] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7428.756379] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7440.060913] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 7440.067222] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7440.106650] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7440.114482] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7440.127678] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7440.132074] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7440.141402] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7440.145746] RAX: 0000000000007e06 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7440.149857] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7440.155481] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7440.157744] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7440.162148] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7440.166904] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7440.173470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7440.180404] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7440.193573] Call Trace: [ 7440.194959] [ 7440.197280] ? show_regs.cold.9+0x22/0x2f [ 7440.202067] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7440.205842] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7440.211622] ? watchdog+0x40/0x40 [ 7440.212803] ? hrtimer_interrupt+0x128/0x2c0 [ 7440.217712] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7440.222881] ? apic_timer_interrupt+0xf/0x20 [ 7440.225140] [ 7440.225927] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7440.230690] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7440.239572] do_raw_spin_lock+0x177/0x190 [ 7440.243456] _raw_spin_lock+0x21/0x30 [ 7440.245051] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7440.255416] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7440.258139] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7440.262832] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7440.272022] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7440.282775] ? woken_wake_function+0x30/0x30 [ 7440.285772] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7440.289922] kthread+0x1d1/0x200 [ 7440.293160] ? set_kthread_struct+0x70/0x70 [ 7440.298086] ret_from_fork+0x1f/0x30 [ 7448.052017] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 7448.055545] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7448.073549] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7448.078016] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7448.081825] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7448.084950] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7448.087818] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7448.095839] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7448.098755] RAX: 0000000000006377 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7448.101499] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7448.103982] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7448.106784] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7448.109036] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7448.111769] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7448.114844] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7448.116880] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7448.119716] Call Trace: [ 7448.120556] [ 7448.121735] ? show_regs.cold.9+0x22/0x2f [ 7448.123619] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7448.125544] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7448.127234] ? watchdog+0x40/0x40 [ 7448.128292] ? hrtimer_interrupt+0x128/0x2c0 [ 7448.129974] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7448.131761] ? apic_timer_interrupt+0xf/0x20 [ 7448.133361] [ 7448.134228] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7448.137974] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7448.148164] ? newidle_balance.isra.93+0x207/0x580 [ 7448.150914] do_raw_spin_lock+0x177/0x190 [ 7448.156717] _raw_spin_lock+0x21/0x30 [ 7448.158746] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7448.164543] process_one_work+0x2c8/0x700 [ 7448.168730] worker_thread+0x296/0x6e0 [ 7448.171877] ? rescuer_thread+0x570/0x570 [ 7448.173342] kthread+0x1d1/0x200 [ 7448.175244] ? set_kthread_struct+0x70/0x70 [ 7448.176797] ret_from_fork+0x1f/0x30 [ 7452.130085] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4815s! [ 7452.135818] Showing busy workqueues and worker pools: [ 7452.139772] workqueue events: flags=0x0 [ 7452.143719] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7452.151148] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7452.165095] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7452.176292] pending: cpu_latency_work [ptlrpc] [ 7452.183405] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7452.188855] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7452.202857] workqueue events_long: flags=0x0 [ 7452.207051] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7452.215014] pending: lu_objects_flush [obdclass] [ 7452.220272] workqueue events_unbound: flags=0x2 [ 7452.226708] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7452.233802] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7452.237326] workqueue events_power_efficient: flags=0x80 [ 7452.238970] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7452.243692] pending: check_lifetime, do_cache_clean [ 7452.248644] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7452.250907] pending: neigh_periodic_work, neigh_periodic_work [ 7452.252873] workqueue mm_percpu_wq: flags=0x8 [ 7452.254785] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7452.257527] pending: vmstat_update, lru_add_drain_per_cpu [ 7452.262100] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7452.265820] pending: vmstat_update, lru_add_drain_per_cpu [ 7452.268722] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7452.272969] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7452.276647] workqueue xprtiod: flags=0xa [ 7452.279024] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7452.282906] pending: xprt_autoclose, xs_error_handle [ 7452.285325] workqueue ipv6_addrconf: flags=0x40008 [ 7452.287906] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7452.291919] pending: addrconf_verify_work [ 7452.295144] workqueue knbd0-recv: flags=0x1a [ 7452.297539] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7452.304281] in-flight: 651:recv_work [ 7452.306226] workqueue ptlrpc_pinger: flags=0x2 [ 7452.313889] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7452.319565] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7452.322652] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4815s workers=4 idle: 15979 31824 8 [ 7452.330559] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7442s workers=2 idle: 654 [ 7456.055340] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7456.057008] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7456.058788] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7456.058868] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7456.058877] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7456.058880] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7456.058896] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7456.058901] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7456.058906] RAX: 00000000000037df RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7456.058909] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7456.058912] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7456.058914] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7456.058916] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7456.058920] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7456.058923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7456.058926] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7456.058935] Call Trace: [ 7456.058941] [ 7456.058947] ? show_regs.cold.9+0x22/0x2f [ 7456.058975] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7456.058982] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7456.058986] ? watchdog+0x40/0x40 [ 7456.058990] ? hrtimer_interrupt+0x128/0x2c0 [ 7456.059085] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7456.059096] ? apic_timer_interrupt+0xf/0x20 [ 7456.059100] [ 7456.059104] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7456.059164] do_raw_spin_lock+0x177/0x190 [ 7456.059171] _raw_spin_lock+0x21/0x30 [ 7456.059181] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7456.059509] ? do_raw_spin_unlock+0x75/0x190 [ 7456.059516] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7456.059735] ? do_wait_intr+0xf0/0xf0 [ 7456.059745] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7456.060018] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7456.060287] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7456.060538] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7456.060651] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7456.060694] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7456.060726] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7456.060760] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7456.060892] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7456.061204] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7456.061250] lmv_intent_open+0x338/0x1150 [lmv] [ 7456.061281] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7456.061453] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7456.061547] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7456.061575] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7456.061665] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7456.061766] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7456.061856] ? do_raw_spin_unlock+0x75/0x190 [ 7456.061863] ? _raw_spin_unlock+0x12/0x30 [ 7456.061876] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7456.061903] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7456.061924] ll_atomic_open+0x36a/0xd70 [lustre] [ 7456.062029] lookup_open+0x356/0xb40 [ 7456.062041] do_last+0x313/0xfc0 [ 7456.062047] ? nd_jump_root+0xe5/0x160 [ 7456.062052] ? path_init+0x437/0x520 [ 7456.062057] path_openat+0xf7/0x500 [ 7456.062062] ? do_raw_spin_unlock+0x75/0x190 [ 7456.062067] do_filp_open+0x99/0x140 [ 7456.062074] ? getname_flags+0x6e/0x330 [ 7456.062079] ? __check_object_size+0xff/0x256 [ 7456.062087] ? do_raw_spin_unlock+0x75/0x190 [ 7456.062091] ? _raw_spin_unlock+0x12/0x30 [ 7456.062096] do_sys_openat2+0x2b4/0x410 [ 7456.062101] do_sys_open+0x73/0xa0 [ 7456.062105] __x64_sys_openat+0x24/0x30 [ 7456.062109] do_syscall_64+0xc1/0x440 [ 7456.062117] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7456.062122] RIP: 0033:0x7f53425a5332 [ 7456.062136] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7456.062140] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7456.062145] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7456.062148] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7456.062150] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7456.062152] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7456.062155] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7456.621552] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7456.643366] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7456.649624] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7456.657550] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7456.669261] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7456.690447] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7456.698560] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7456.703713] RDX: 0000000000005941 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7456.706631] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7456.710766] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7456.716277] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7456.719574] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7456.723712] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7456.726843] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7456.730834] Call Trace: [ 7456.732259] [ 7456.733412] ? show_regs.cold.9+0x22/0x2f [ 7456.735418] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7456.737369] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7456.738786] ? watchdog+0x40/0x40 [ 7456.740663] ? hrtimer_interrupt+0x128/0x2c0 [ 7456.742831] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7456.744999] ? apic_timer_interrupt+0xf/0x20 [ 7456.747613] [ 7456.748697] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7456.751274] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7456.753536] do_raw_spin_lock+0x177/0x190 [ 7456.755296] _raw_spin_lock+0x21/0x30 [ 7456.756743] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7456.759584] obd_mod_free+0x46/0x140 [ptlrpc] [ 7456.761857] mdc_commit_open+0x91/0xb0 [mdc] [ 7456.766092] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7456.768209] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7456.771305] after_reply+0xe2c/0x1780 [ptlrpc] [ 7456.773486] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7456.776372] ? __next_timer_interrupt+0x160/0x160 [ 7456.778299] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7456.780539] ? do_wait_intr+0xf0/0xf0 [ 7456.782343] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7456.784725] kthread+0x1d1/0x200 [ 7456.785848] ? set_kthread_struct+0x70/0x70 [ 7456.788435] ret_from_fork+0x1f/0x30 [ 7468.062766] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 7468.076406] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7468.130006] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7468.132109] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7468.148389] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7468.158327] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7468.186494] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7468.195288] RAX: 0000000000007e86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7468.207244] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7468.221303] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7468.229289] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7468.236690] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7468.246839] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7468.252130] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7468.254640] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7468.264119] Call Trace: [ 7468.265395] [ 7468.266251] ? show_regs.cold.9+0x22/0x2f [ 7468.269671] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7468.275243] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7468.279634] ? watchdog+0x40/0x40 [ 7468.283572] ? hrtimer_interrupt+0x128/0x2c0 [ 7468.288874] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7468.293372] ? apic_timer_interrupt+0xf/0x20 [ 7468.297102] [ 7468.299449] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7468.301495] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7468.303666] do_raw_spin_lock+0x177/0x190 [ 7468.305273] _raw_spin_lock+0x21/0x30 [ 7468.310346] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7468.314435] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7468.326033] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7468.335420] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7468.343359] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7468.348431] ? woken_wake_function+0x30/0x30 [ 7468.349885] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7468.353501] kthread+0x1d1/0x200 [ 7468.356304] ? set_kthread_struct+0x70/0x70 [ 7468.358939] ret_from_fork+0x1f/0x30 [ 7480.060268] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 7480.066353] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7480.117705] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7480.127297] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7480.139777] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7480.144211] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7480.150923] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7480.171082] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7480.185129] RAX: 0000000000007b8f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7480.195972] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7480.204497] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7480.213471] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7480.220873] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7480.224958] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7480.230860] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7480.238061] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7480.248327] Call Trace: [ 7480.249237] [ 7480.250020] ? show_regs.cold.9+0x22/0x2f [ 7480.258347] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7480.263838] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7480.268027] ? watchdog+0x40/0x40 [ 7480.269264] ? hrtimer_interrupt+0x128/0x2c0 [ 7480.275751] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7480.291247] ? apic_timer_interrupt+0xf/0x20 [ 7480.292826] [ 7480.296992] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7480.300675] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7480.306088] ? newidle_balance.isra.93+0x207/0x580 [ 7480.310442] do_raw_spin_lock+0x177/0x190 [ 7480.316212] _raw_spin_lock+0x21/0x30 [ 7480.320649] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7480.323794] process_one_work+0x2c8/0x700 [ 7480.325346] worker_thread+0x296/0x6e0 [ 7480.338373] ? rescuer_thread+0x570/0x570 [ 7480.348419] kthread+0x1d1/0x200 [ 7480.351206] ? set_kthread_struct+0x70/0x70 [ 7480.357356] ret_from_fork+0x1f/0x30 [ 7482.852727] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4845s! [ 7482.867122] Showing busy workqueues and worker pools: [ 7482.870553] workqueue events: flags=0x0 [ 7482.871916] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7482.883364] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7482.901656] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7482.907443] pending: cpu_latency_work [ptlrpc] [ 7482.910943] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7482.921347] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7482.932229] workqueue events_long: flags=0x0 [ 7482.938785] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7482.954368] pending: lu_objects_flush [obdclass] [ 7482.958457] workqueue events_unbound: flags=0x2 [ 7482.964562] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7482.973024] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7482.985159] workqueue events_power_efficient: flags=0x80 [ 7482.988030] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7482.995378] pending: check_lifetime, do_cache_clean [ 7483.000341] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7483.007907] pending: neigh_periodic_work, neigh_periodic_work [ 7483.011431] workqueue mm_percpu_wq: flags=0x8 [ 7483.013584] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7483.015324] pending: vmstat_update, lru_add_drain_per_cpu [ 7483.021349] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7483.029846] pending: vmstat_update, lru_add_drain_per_cpu [ 7483.038055] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7483.050714] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7483.059551] workqueue xprtiod: flags=0xa [ 7483.060888] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7483.072426] pending: xprt_autoclose, xs_error_handle [ 7483.081561] workqueue ipv6_addrconf: flags=0x40008 [ 7483.085824] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7483.089233] pending: addrconf_verify_work [ 7483.096584] workqueue knbd0-recv: flags=0x1a [ 7483.098260] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7483.104868] in-flight: 651:recv_work [ 7483.111851] workqueue ptlrpc_pinger: flags=0x2 [ 7483.120865] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7483.125133] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7483.129442] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4846s workers=4 idle: 15979 31824 8 [ 7483.140445] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7473s workers=2 idle: 654 [ 7484.055759] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 7484.061102] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 7484.072311] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7484.072601] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7484.072608] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7484.072611] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7484.072741] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7484.072745] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7484.072750] RAX: 0000000000007f87 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7484.072753] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7484.072755] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7484.072758] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7484.072760] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7484.072764] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7484.072767] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7484.072769] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7484.072778] Call Trace: [ 7484.072783] [ 7484.072789] ? show_regs.cold.9+0x22/0x2f [ 7484.072796] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7484.072803] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7484.072807] ? watchdog+0x40/0x40 [ 7484.072811] ? hrtimer_interrupt+0x128/0x2c0 [ 7484.072815] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7484.072823] ? apic_timer_interrupt+0xf/0x20 [ 7484.072828] [ 7484.072831] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7484.072837] do_raw_spin_lock+0x177/0x190 [ 7484.072899] _raw_spin_lock+0x21/0x30 [ 7484.072905] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7484.082335] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7484.082410] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7484.082417] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7484.082419] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7484.082432] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7484.082436] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7484.082440] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7484.082442] RDX: 0000000000007bb4 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7484.082444] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7484.082446] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7484.082448] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7484.082450] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7484.082453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7484.082456] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7484.082474] Call Trace: [ 7484.082481] [ 7484.082487] ? show_regs.cold.9+0x22/0x2f [ 7484.082499] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7484.082505] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7484.082510] ? watchdog+0x40/0x40 [ 7484.082515] ? hrtimer_interrupt+0x128/0x2c0 [ 7484.082519] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7484.082527] ? apic_timer_interrupt+0xf/0x20 [ 7484.082531] [ 7484.082533] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7484.082537] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7484.082542] do_raw_spin_lock+0x177/0x190 [ 7484.082546] _raw_spin_lock+0x21/0x30 [ 7484.082553] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7484.082891] obd_mod_free+0x46/0x140 [ptlrpc] [ 7484.083092] mdc_commit_open+0x91/0xb0 [mdc] [ 7484.083129] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7484.083331] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7484.084210] after_reply+0xe2c/0x1780 [ptlrpc] [ 7484.084407] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7484.084728] ? __next_timer_interrupt+0x160/0x160 [ 7484.084738] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7484.084935] ? do_wait_intr+0xf0/0xf0 [ 7484.084945] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7484.086162] kthread+0x1d1/0x200 [ 7484.086170] ? set_kthread_struct+0x70/0x70 [ 7484.086175] ret_from_fork+0x1f/0x30 [ 7484.640109] ? do_raw_spin_unlock+0x75/0x190 [ 7484.642590] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7484.645519] ? do_wait_intr+0xf0/0xf0 [ 7484.647558] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7484.652296] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7484.657420] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7484.661319] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7484.665737] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7484.669700] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7484.672769] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7484.675009] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7484.679334] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7484.682853] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7484.685984] lmv_intent_open+0x338/0x1150 [lmv] [ 7484.688200] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7484.690175] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7484.692318] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7484.694090] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7484.696363] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7484.698354] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7484.700246] ? do_raw_spin_unlock+0x75/0x190 [ 7484.702721] ? _raw_spin_unlock+0x12/0x30 [ 7484.704911] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7484.708919] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7484.715808] ll_atomic_open+0x36a/0xd70 [lustre] [ 7484.720655] lookup_open+0x356/0xb40 [ 7484.722551] do_last+0x313/0xfc0 [ 7484.723283] ? nd_jump_root+0xe5/0x160 [ 7484.724333] ? path_init+0x437/0x520 [ 7484.726423] path_openat+0xf7/0x500 [ 7484.727724] ? do_raw_spin_unlock+0x75/0x190 [ 7484.730667] do_filp_open+0x99/0x140 [ 7484.733096] ? getname_flags+0x6e/0x330 [ 7484.737345] ? __check_object_size+0xff/0x256 [ 7484.741132] ? do_raw_spin_unlock+0x75/0x190 [ 7484.744639] ? _raw_spin_unlock+0x12/0x30 [ 7484.752097] do_sys_openat2+0x2b4/0x410 [ 7484.754152] do_sys_open+0x73/0xa0 [ 7484.758096] __x64_sys_openat+0x24/0x30 [ 7484.760671] do_syscall_64+0xc1/0x440 [ 7484.763855] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7484.772320] RIP: 0033:0x7f53425a5332 [ 7484.774693] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7484.805231] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7484.818291] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7484.832113] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7484.838675] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7484.847324] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7484.855532] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7496.061238] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 7496.069356] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7496.124506] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7496.130477] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7496.136637] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7496.139347] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7496.162095] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7496.167661] RAX: 0000000000007f0b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7496.181590] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7496.187537] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7496.190267] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7496.194944] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7496.197744] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7496.200786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7496.203011] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7496.205735] Call Trace: [ 7496.206771] [ 7496.207433] ? show_regs.cold.9+0x22/0x2f [ 7496.208636] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7496.210389] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7496.212058] ? watchdog+0x40/0x40 [ 7496.213430] ? hrtimer_interrupt+0x128/0x2c0 [ 7496.214915] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7496.216824] ? apic_timer_interrupt+0xf/0x20 [ 7496.218724] [ 7496.219572] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7496.221595] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7496.223320] do_raw_spin_lock+0x177/0x190 [ 7496.224379] _raw_spin_lock+0x21/0x30 [ 7496.225565] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7496.228435] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7496.231973] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7496.234611] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7496.237138] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7496.239826] ? woken_wake_function+0x30/0x30 [ 7496.241447] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7496.249494] kthread+0x1d1/0x200 [ 7496.250562] ? set_kthread_struct+0x70/0x70 [ 7496.251856] ret_from_fork+0x1f/0x30 [ 7508.053123] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 7508.057541] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7508.085376] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7508.094662] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7508.100631] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7508.105271] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7508.118370] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7508.145104] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7508.157622] RAX: 000000000000793e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7508.161651] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7508.174794] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7508.178764] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7508.186283] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7508.193334] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7508.198810] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7508.203701] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7508.210073] Call Trace: [ 7508.210850] [ 7508.211531] ? show_regs.cold.9+0x22/0x2f [ 7508.220755] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7508.222339] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7508.230566] ? watchdog+0x40/0x40 [ 7508.234786] ? hrtimer_interrupt+0x128/0x2c0 [ 7508.237967] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7508.241324] ? apic_timer_interrupt+0xf/0x20 [ 7508.243440] [ 7508.245139] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7508.247951] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7508.256296] ? newidle_balance.isra.93+0x207/0x580 [ 7508.263443] do_raw_spin_lock+0x177/0x190 [ 7508.267777] _raw_spin_lock+0x21/0x30 [ 7508.272608] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7508.279853] process_one_work+0x2c8/0x700 [ 7508.281281] worker_thread+0x296/0x6e0 [ 7508.299010] ? rescuer_thread+0x570/0x570 [ 7508.300371] kthread+0x1d1/0x200 [ 7508.301507] ? set_kthread_struct+0x70/0x70 [ 7508.320867] ret_from_fork+0x1f/0x30 [ 7512.057294] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 7512.063285] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7512.067670] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 7512.106750] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7512.106766] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7512.106770] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7512.106786] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7512.106790] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7512.106796] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7512.106799] RDX: 00000000000044c1 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7512.106801] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7512.106803] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7512.106806] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7512.106809] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7512.106812] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7512.106815] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7512.106824] Call Trace: [ 7512.106830] [ 7512.106837] ? show_regs.cold.9+0x22/0x2f [ 7512.106848] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7512.106855] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7512.106860] ? watchdog+0x40/0x40 [ 7512.106864] ? hrtimer_interrupt+0x128/0x2c0 [ 7512.106868] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7512.106877] ? apic_timer_interrupt+0xf/0x20 [ 7512.106882] [ 7512.106885] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7512.106890] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7512.106895] do_raw_spin_lock+0x177/0x190 [ 7512.106899] _raw_spin_lock+0x21/0x30 [ 7512.106907] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7512.109743] obd_mod_free+0x46/0x140 [ptlrpc] [ 7512.109945] mdc_commit_open+0x91/0xb0 [mdc] [ 7512.110211] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7512.110410] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7512.110624] after_reply+0xe2c/0x1780 [ptlrpc] [ 7512.110823] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7512.111215] ? __next_timer_interrupt+0x160/0x160 [ 7512.111225] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7512.111426] ? do_wait_intr+0xf0/0xf0 [ 7512.111435] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7512.111655] kthread+0x1d1/0x200 [ 7512.111661] ? set_kthread_struct+0x70/0x70 [ 7512.111665] ret_from_fork+0x1f/0x30 [ 7512.364401] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7512.384486] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7512.389897] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7512.400139] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7512.406273] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7512.423705] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7512.429052] RAX: 0000000000007a08 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7512.441247] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7512.451746] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7512.462790] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7512.470955] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7512.473372] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7512.479949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7512.485015] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7512.494381] Call Trace: [ 7512.498615] [ 7512.502711] ? show_regs.cold.9+0x22/0x2f [ 7512.506778] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7512.510157] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7512.515834] ? watchdog+0x40/0x40 [ 7512.519941] ? hrtimer_interrupt+0x128/0x2c0 [ 7512.523761] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7512.531277] ? apic_timer_interrupt+0xf/0x20 [ 7512.537440] [ 7512.538879] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7512.542786] do_raw_spin_lock+0x177/0x190 [ 7512.550628] _raw_spin_lock+0x21/0x30 [ 7512.556182] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7512.559256] ? do_raw_spin_unlock+0x75/0x190 [ 7512.563879] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7512.572003] ? do_wait_intr+0xf0/0xf0 [ 7512.577463] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7512.584767] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7512.591899] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7512.597799] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7512.603768] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7512.611747] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7512.620742] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7512.627864] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7512.640756] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7512.645005] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7512.648522] lmv_intent_open+0x338/0x1150 [lmv] [ 7512.653352] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7512.655482] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7512.660513] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7512.663576] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7512.670438] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7512.676936] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7512.681573] ? do_raw_spin_unlock+0x75/0x190 [ 7512.684315] ? _raw_spin_unlock+0x12/0x30 [ 7512.688771] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7512.691459] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7512.700981] ll_atomic_open+0x36a/0xd70 [lustre] [ 7512.705178] lookup_open+0x356/0xb40 [ 7512.710039] do_last+0x313/0xfc0 [ 7512.712389] ? nd_jump_root+0xe5/0x160 [ 7512.715392] ? path_init+0x437/0x520 [ 7512.718692] path_openat+0xf7/0x500 [ 7512.724937] ? do_raw_spin_unlock+0x75/0x190 [ 7512.727924] do_filp_open+0x99/0x140 [ 7512.734106] ? getname_flags+0x6e/0x330 [ 7512.737634] ? __check_object_size+0xff/0x256 [ 7512.741752] ? do_raw_spin_unlock+0x75/0x190 [ 7512.751906] ? _raw_spin_unlock+0x12/0x30 [ 7512.754842] do_sys_openat2+0x2b4/0x410 [ 7512.761797] do_sys_open+0x73/0xa0 [ 7512.769463] __x64_sys_openat+0x24/0x30 [ 7512.771932] do_syscall_64+0xc1/0x440 [ 7512.778269] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7512.783197] RIP: 0033:0x7f53425a5332 [ 7512.784467] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7512.815244] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7512.820151] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7512.828632] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7512.835354] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7512.841428] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7512.844970] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7513.567231] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4876s! [ 7513.579229] Showing busy workqueues and worker pools: [ 7513.584249] workqueue events: flags=0x0 [ 7513.587321] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7513.591106] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7513.599820] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7513.604776] pending: cpu_latency_work [ptlrpc] [ 7513.613828] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7513.623067] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7513.630103] workqueue events_long: flags=0x0 [ 7513.631693] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7513.640365] pending: lu_objects_flush [obdclass] [ 7513.646642] workqueue events_unbound: flags=0x2 [ 7513.648925] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7513.655602] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7513.660797] workqueue events_power_efficient: flags=0x80 [ 7513.665445] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7513.676845] pending: check_lifetime, do_cache_clean [ 7513.684870] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7513.689882] pending: neigh_periodic_work, neigh_periodic_work [ 7513.693803] workqueue mm_percpu_wq: flags=0x8 [ 7513.703695] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7513.711128] pending: vmstat_update, lru_add_drain_per_cpu [ 7513.715322] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7513.721179] pending: vmstat_update, lru_add_drain_per_cpu [ 7513.724930] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7513.728877] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7513.736159] workqueue xprtiod: flags=0xa [ 7513.739245] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7513.748361] pending: xprt_autoclose, xs_error_handle [ 7513.755116] workqueue ipv6_addrconf: flags=0x40008 [ 7513.758765] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7513.763017] pending: addrconf_verify_work [ 7513.765861] workqueue knbd0-recv: flags=0x1a [ 7513.768714] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7513.773877] in-flight: 651:recv_work [ 7513.777990] workqueue ptlrpc_pinger: flags=0x2 [ 7513.785960] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7513.795163] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7513.799606] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4876s workers=4 idle: 15979 31824 8 [ 7513.803473] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7504s workers=2 idle: 654 [ 7524.061369] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 7524.065642] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7524.110554] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7524.122561] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7524.129576] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7524.134496] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7524.152394] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7524.160373] RAX: 0000000000007f84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7524.163328] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7524.167692] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7524.171952] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7524.175348] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7524.182037] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7524.189458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7524.191329] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7524.199442] Call Trace: [ 7524.200818] [ 7524.202150] ? show_regs.cold.9+0x22/0x2f [ 7524.204504] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7524.207256] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7524.211208] ? watchdog+0x40/0x40 [ 7524.212753] ? hrtimer_interrupt+0x128/0x2c0 [ 7524.215030] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7524.220081] ? apic_timer_interrupt+0xf/0x20 [ 7524.225964] [ 7524.226685] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7524.228525] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7524.234432] do_raw_spin_lock+0x177/0x190 [ 7524.239930] _raw_spin_lock+0x21/0x30 [ 7524.241091] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7524.245529] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7524.248529] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7524.253270] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7524.258546] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7524.260583] ? woken_wake_function+0x30/0x30 [ 7524.264296] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7524.270335] kthread+0x1d1/0x200 [ 7524.271434] ? set_kthread_struct+0x70/0x70 [ 7524.274716] ret_from_fork+0x1f/0x30 [ 7540.052029] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 7540.054228] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7540.055032] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 7540.058872] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 7540.058884] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7540.058957] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7540.058963] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7540.058966] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7540.058979] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7540.058984] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7540.058989] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7540.058991] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7540.058994] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7540.058996] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7540.058998] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7540.059002] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7540.059005] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7540.059007] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7540.059017] Call Trace: [ 7540.059023] [ 7540.059028] ? show_regs.cold.9+0x22/0x2f [ 7540.059039] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7540.059045] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7540.059049] ? watchdog+0x40/0x40 [ 7540.059054] ? hrtimer_interrupt+0x128/0x2c0 [ 7540.059058] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7540.059065] ? apic_timer_interrupt+0xf/0x20 [ 7540.059070] [ 7540.059073] ? native_safe_halt+0x2e/0x50 [ 7540.059077] kvm_wait+0xcd/0xe0 [ 7540.059084] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7540.059092] do_raw_spin_lock+0x177/0x190 [ 7540.059097] _raw_spin_lock+0x21/0x30 [ 7540.059103] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7540.059370] obd_mod_free+0x46/0x140 [ptlrpc] [ 7540.059581] mdc_commit_open+0x91/0xb0 [mdc] [ 7540.059615] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7540.059829] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7540.060037] after_reply+0xe2c/0x1780 [ptlrpc] [ 7540.060243] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7540.060447] ? __next_timer_interrupt+0x160/0x160 [ 7540.060456] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7540.060685] ? do_wait_intr+0xf0/0xf0 [ 7540.060694] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7540.060903] kthread+0x1d1/0x200 [ 7540.060909] ? set_kthread_struct+0x70/0x70 [ 7540.060913] ret_from_fork+0x1f/0x30 [ 7540.118630] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7540.122527] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7540.122607] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7540.122616] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7540.122619] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7540.122632] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7540.122637] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7540.122642] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7540.122644] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7540.122646] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7540.122648] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7540.122650] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7540.122653] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7540.122656] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7540.122658] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7540.122666] Call Trace: [ 7540.122671] [ 7540.122676] ? show_regs.cold.9+0x22/0x2f [ 7540.122684] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7540.122690] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7540.122693] ? watchdog+0x40/0x40 [ 7540.122697] ? hrtimer_interrupt+0x128/0x2c0 [ 7540.122700] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7540.122707] ? apic_timer_interrupt+0xf/0x20 [ 7540.122711] [ 7540.122714] ? native_safe_halt+0x2e/0x50 [ 7540.122717] kvm_wait+0xcd/0xe0 [ 7540.122724] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7540.122732] do_raw_spin_lock+0x177/0x190 [ 7540.122736] _raw_spin_lock+0x21/0x30 [ 7540.122740] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7540.122998] ? do_raw_spin_unlock+0x75/0x190 [ 7540.123004] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7540.123202] ? do_wait_intr+0xf0/0xf0 [ 7540.123211] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7540.123420] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7540.140468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7540.140617] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7540.140890] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7540.140900] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7540.140904] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7540.140909] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7540.140911] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7540.140913] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7540.140915] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7540.140917] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7540.140920] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7540.213430] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7540.217399] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7540.219228] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7540.219271] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7540.219302] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7540.219332] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7540.219420] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7540.235334] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7540.235346] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7540.235355] Call Trace: [ 7540.235363] [ 7540.235368] ? show_regs.cold.9+0x22/0x2f [ 7540.235379] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7540.235386] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7540.235390] ? watchdog+0x40/0x40 [ 7540.235394] ? hrtimer_interrupt+0x128/0x2c0 [ 7540.235398] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7540.235405] ? apic_timer_interrupt+0xf/0x20 [ 7540.235410] [ 7540.235413] ? native_safe_halt+0x2e/0x50 [ 7540.235417] kvm_wait+0xcd/0xe0 [ 7540.235424] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7540.235431] ? newidle_balance.isra.93+0x207/0x580 [ 7540.235438] do_raw_spin_lock+0x177/0x190 [ 7540.235443] _raw_spin_lock+0x21/0x30 [ 7540.235448] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7540.235733] process_one_work+0x2c8/0x700 [ 7540.235740] worker_thread+0x296/0x6e0 [ 7540.235745] ? rescuer_thread+0x570/0x570 [ 7540.235749] kthread+0x1d1/0x200 [ 7540.235753] ? set_kthread_struct+0x70/0x70 [ 7540.235756] ret_from_fork+0x1f/0x30 [ 7541.070384] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7541.077068] lmv_intent_open+0x338/0x1150 [lmv] [ 7541.079381] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7541.082979] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7541.085980] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7541.087419] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7541.093162] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7541.106816] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7541.118775] ? do_raw_spin_unlock+0x75/0x190 [ 7541.125910] ? _raw_spin_unlock+0x12/0x30 [ 7541.133895] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7541.145135] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7541.152634] ll_atomic_open+0x36a/0xd70 [lustre] [ 7541.156748] lookup_open+0x356/0xb40 [ 7541.163251] do_last+0x313/0xfc0 [ 7541.168340] ? nd_jump_root+0xe5/0x160 [ 7541.171502] ? path_init+0x437/0x520 [ 7541.172657] path_openat+0xf7/0x500 [ 7541.174923] ? do_raw_spin_unlock+0x75/0x190 [ 7541.176423] do_filp_open+0x99/0x140 [ 7541.177540] ? getname_flags+0x6e/0x330 [ 7541.178967] ? __check_object_size+0xff/0x256 [ 7541.180670] ? do_raw_spin_unlock+0x75/0x190 [ 7541.182130] ? _raw_spin_unlock+0x12/0x30 [ 7541.183691] do_sys_openat2+0x2b4/0x410 [ 7541.185339] do_sys_open+0x73/0xa0 [ 7541.186586] __x64_sys_openat+0x24/0x30 [ 7541.187913] do_syscall_64+0xc1/0x440 [ 7541.193447] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7541.201568] RIP: 0033:0x7f53425a5332 [ 7541.204977] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7541.225960] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7541.230345] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7541.235292] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7541.237931] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7541.240852] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7541.244307] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7544.292156] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4907s! [ 7544.310577] Showing busy workqueues and worker pools: [ 7544.320175] workqueue events: flags=0x0 [ 7544.331926] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7544.338837] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7544.363538] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7544.382304] pending: cpu_latency_work [ptlrpc] [ 7544.389606] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7544.403163] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7544.414840] workqueue events_long: flags=0x0 [ 7544.417789] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7544.421960] pending: lu_objects_flush [obdclass] [ 7544.427877] workqueue events_unbound: flags=0x2 [ 7544.433132] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7544.435264] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7544.444100] workqueue events_power_efficient: flags=0x80 [ 7544.448550] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7544.452377] pending: check_lifetime, do_cache_clean [ 7544.459319] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7544.463730] pending: neigh_periodic_work, neigh_periodic_work [ 7544.466786] workqueue mm_percpu_wq: flags=0x8 [ 7544.468925] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7544.475223] pending: vmstat_update, lru_add_drain_per_cpu [ 7544.478330] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7544.482248] pending: vmstat_update, lru_add_drain_per_cpu [ 7544.489828] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7544.502358] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7544.509752] workqueue xprtiod: flags=0xa [ 7544.513690] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7544.519054] pending: xprt_autoclose, xs_error_handle [ 7544.521707] workqueue ipv6_addrconf: flags=0x40008 [ 7544.524304] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7544.530694] pending: addrconf_verify_work [ 7544.533383] workqueue knbd0-recv: flags=0x1a [ 7544.536289] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7544.542386] in-flight: 651:recv_work [ 7544.544297] workqueue ptlrpc_pinger: flags=0x2 [ 7544.546127] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7544.552094] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7544.562199] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4907s workers=4 idle: 15979 31824 8 [ 7544.576083] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7535s workers=2 idle: 654 [ 7552.061944] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 7552.066543] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7552.106768] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7552.115660] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7552.124250] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7552.126237] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7552.156493] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7552.177654] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7552.190624] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7552.197667] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7552.211158] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7552.219599] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7552.231503] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7552.247542] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7552.258103] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7552.267042] Call Trace: [ 7552.268034] [ 7552.270035] ? show_regs.cold.9+0x22/0x2f [ 7552.271929] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7552.277910] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7552.281814] ? watchdog+0x40/0x40 [ 7552.283672] ? hrtimer_interrupt+0x128/0x2c0 [ 7552.287789] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7552.291024] ? apic_timer_interrupt+0xf/0x20 [ 7552.292460] [ 7552.293123] ? native_safe_halt+0x2e/0x50 [ 7552.295707] kvm_wait+0xcd/0xe0 [ 7552.297237] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7552.303218] do_raw_spin_lock+0x177/0x190 [ 7552.313100] _raw_spin_lock+0x21/0x30 [ 7552.315102] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7552.324854] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7552.327105] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7552.335815] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7552.350732] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7552.360399] ? woken_wake_function+0x30/0x30 [ 7552.361733] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7552.371514] kthread+0x1d1/0x200 [ 7552.372930] ? set_kthread_struct+0x70/0x70 [ 7552.376341] ret_from_fork+0x1f/0x30 [ 7554.570667] rcu: INFO: rcu_sched self-detected stall on CPU [ 7554.575896] rcu: 2-....: (2949517 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 7554.585754] (t=5460144 jiffies g=274185 q=65136) [ 7554.587772] rcu: rcu_sched kthread starved for 4918510 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 7554.592910] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7554.596927] rcu: RCU grace-period kthread stack dump: [ 7554.598983] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7554.604921] Call Trace: [ 7554.605755] __schedule+0x351/0xcb0 [ 7554.607011] schedule+0xc0/0x180 [ 7554.609947] schedule_timeout+0xb4/0x190 [ 7554.614381] ? do_raw_spin_unlock+0x75/0x190 [ 7554.625902] ? __next_timer_interrupt+0x160/0x160 [ 7554.633692] ? prepare_to_swait_event+0xf3/0x140 [ 7554.637474] rcu_gp_kthread+0x8dd/0x15b0 [ 7554.641750] ? force_qs_rnp+0x2c0/0x2c0 [ 7554.651247] kthread+0x1d1/0x200 [ 7554.652417] ? set_kthread_struct+0x70/0x70 [ 7554.659519] ret_from_fork+0x1f/0x30 [ 7554.660746] rcu: Stack dump where RCU GP kthread last ran: [ 7554.670098] Sending NMI from CPU 2 to CPUs 0: [ 7554.672281] NMI backtrace for cpu 0 [ 7554.672287] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7554.672289] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7554.672290] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7554.672293] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7554.672294] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7554.672296] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 [ 7554.672298] RAX: 000000000000340a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7554.672300] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7554.672301] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7554.672302] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7554.672303] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7554.672305] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7554.672377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7554.672380] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7554.672381] Call Trace: [ 7554.672382] [ 7554.672383] ? show_regs.cold.9+0x22/0x2f [ 7554.672384] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7554.672385] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7554.672386] ? nmi_handle+0xbd/0x210 [ 7554.672388] ? default_do_nmi+0x4b/0x1b0 [ 7554.672389] ? do_nmi+0x280/0x380 [ 7554.672390] ? end_repeat_nmi+0x16/0x1f [ 7554.672391] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7554.672392] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7554.672393] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7554.672394] [ 7554.672395] ? newidle_balance.isra.93+0x207/0x580 [ 7554.672396] do_raw_spin_lock+0x177/0x190 [ 7554.672398] _raw_spin_lock+0x21/0x30 [ 7554.672399] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7554.672400] process_one_work+0x2c8/0x700 [ 7554.672401] worker_thread+0x296/0x6e0 [ 7554.672402] ? rescuer_thread+0x570/0x570 [ 7554.672403] kthread+0x1d1/0x200 [ 7554.672404] ? set_kthread_struct+0x70/0x70 [ 7554.672405] ret_from_fork+0x1f/0x30 [ 7554.677276] Sending NMI from CPU 2 to CPUs 1: [ 7554.895620] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 7554.896678] NMI backtrace for cpu 2 [ 7554.903379] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7554.908729] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7554.914867] Call Trace: [ 7554.917283] [ 7554.917989] ? dump_stack+0xbb/0x10e [ 7554.919794] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7554.921971] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7554.924464] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7554.926449] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7554.929646] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7554.932909] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7554.934788] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7554.941966] ? update_process_times+0x6d/0xc0 [ 7554.947992] ? tick_sched_handle.isra.18+0x27/0x90 [ 7554.951141] ? tick_sched_timer+0x53/0xb0 [ 7554.953199] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7554.956630] ? tick_sched_handle.isra.18+0x90/0x90 [ 7554.959401] ? hrtimer_interrupt+0x128/0x2c0 [ 7554.961955] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7554.970160] ? apic_timer_interrupt+0xf/0x20 [ 7554.971849] [ 7554.972652] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7554.978600] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7554.983274] ? do_raw_spin_lock+0x177/0x190 [ 7554.985072] ? _raw_spin_lock+0x21/0x30 [ 7554.987162] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7554.990093] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7554.992366] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7554.994354] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7554.999675] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7555.007754] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7555.016094] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7555.036902] ? __next_timer_interrupt+0x160/0x160 [ 7555.049585] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7555.059517] ? do_wait_intr+0xf0/0xf0 [ 7555.070212] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7555.075122] ? kthread+0x1d1/0x200 [ 7555.083822] ? set_kthread_struct+0x70/0x70 [ 7555.088471] ? ret_from_fork+0x1f/0x30 [ 7575.012218] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4937s! [ 7575.021087] Showing busy workqueues and worker pools: [ 7575.030237] workqueue events: flags=0x0 [ 7575.034167] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7575.040482] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7575.052905] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7575.055800] pending: cpu_latency_work [ptlrpc] [ 7575.058636] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7575.068226] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7575.071755] workqueue events_long: flags=0x0 [ 7575.073842] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7575.077608] pending: lu_objects_flush [obdclass] [ 7575.081548] workqueue events_unbound: flags=0x2 [ 7575.083343] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7575.085841] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7575.087834] workqueue events_power_efficient: flags=0x80 [ 7575.089855] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7575.094021] pending: check_lifetime, do_cache_clean [ 7575.095753] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7575.099659] pending: neigh_periodic_work, neigh_periodic_work [ 7575.102627] workqueue mm_percpu_wq: flags=0x8 [ 7575.108810] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7575.111958] pending: vmstat_update, lru_add_drain_per_cpu [ 7575.113957] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7575.116489] pending: vmstat_update, lru_add_drain_per_cpu [ 7575.119056] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7575.122130] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7575.124495] workqueue xprtiod: flags=0xa [ 7575.131119] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7575.138781] pending: xprt_autoclose, xs_error_handle [ 7575.146049] workqueue ipv6_addrconf: flags=0x40008 [ 7575.149975] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7575.156314] pending: addrconf_verify_work [ 7575.159859] workqueue knbd0-recv: flags=0x1a [ 7575.163825] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7575.168678] in-flight: 651:recv_work [ 7575.170792] workqueue ptlrpc_pinger: flags=0x2 [ 7575.185178] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7575.186796] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7575.195396] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4938s workers=4 idle: 15979 31824 8 [ 7575.204692] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7565s workers=2 idle: 654 [ 7580.055860] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7580.058951] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7580.060223] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 7580.060232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7580.060296] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7580.060301] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7580.060304] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7580.060319] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7580.060323] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7580.060328] RAX: 0000000000007615 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7580.060330] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7580.060332] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7580.060334] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7580.060337] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7580.060339] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7580.060342] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7580.060344] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7580.060353] Call Trace: [ 7580.060359] [ 7580.060365] ? show_regs.cold.9+0x22/0x2f [ 7580.060374] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7580.060381] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7580.060385] ? watchdog+0x40/0x40 [ 7580.060389] ? hrtimer_interrupt+0x128/0x2c0 [ 7580.060394] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7580.060402] ? apic_timer_interrupt+0xf/0x20 [ 7580.060406] [ 7580.060409] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7580.060415] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7580.060421] do_raw_spin_lock+0x177/0x190 [ 7580.060426] _raw_spin_lock+0x21/0x30 [ 7580.060493] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7580.060550] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7580.060555] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7580.060557] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7580.060563] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7580.060567] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7580.060571] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7580.060574] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7580.060576] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7580.060578] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7580.060581] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7580.060584] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7580.060587] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7580.060589] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7580.060430] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7580.060598] Call Trace: [ 7580.060603] [ 7580.060606] ? show_regs.cold.9+0x22/0x2f [ 7580.060613] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7580.060617] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7580.060621] ? watchdog+0x40/0x40 [ 7580.060625] ? hrtimer_interrupt+0x128/0x2c0 [ 7580.060629] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7580.060634] ? apic_timer_interrupt+0xf/0x20 [ 7580.060639] [ 7580.060642] ? native_safe_halt+0x2e/0x50 [ 7580.060646] kvm_wait+0xcd/0xe0 [ 7580.060654] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7580.060660] do_raw_spin_lock+0x177/0x190 [ 7580.060665] _raw_spin_lock+0x21/0x30 [ 7580.060750] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7580.060669] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7580.061422] ? do_raw_spin_unlock+0x75/0x190 [ 7580.061430] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7580.061709] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7580.062458] ? do_wait_intr+0xf0/0xf0 [ 7580.062468] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7580.061954] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7580.063246] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7580.063729] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7580.068144] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7580.068259] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7580.068301] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7580.068333] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7580.068359] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7580.068445] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7580.068889] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7580.068921] lmv_intent_open+0x338/0x1150 [lmv] [ 7580.068955] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7580.069051] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7580.069134] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7580.069158] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7580.069252] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7580.069339] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7580.069427] ? do_raw_spin_unlock+0x75/0x190 [ 7580.069442] ? _raw_spin_unlock+0x12/0x30 [ 7580.069450] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7580.069477] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7580.069497] ll_atomic_open+0x36a/0xd70 [lustre] [ 7580.069597] lookup_open+0x356/0xb40 [ 7580.069608] do_last+0x313/0xfc0 [ 7580.069614] ? nd_jump_root+0xe5/0x160 [ 7580.069619] ? path_init+0x437/0x520 [ 7580.069624] path_openat+0xf7/0x500 [ 7580.069629] ? do_raw_spin_unlock+0x75/0x190 [ 7580.069633] do_filp_open+0x99/0x140 [ 7580.069640] ? getname_flags+0x6e/0x330 [ 7580.069645] ? __check_object_size+0xff/0x256 [ 7580.069652] ? do_raw_spin_unlock+0x75/0x190 [ 7580.069656] ? _raw_spin_unlock+0x12/0x30 [ 7580.069661] do_sys_openat2+0x2b4/0x410 [ 7580.069665] do_sys_open+0x73/0xa0 [ 7580.069669] __x64_sys_openat+0x24/0x30 [ 7580.069673] do_syscall_64+0xc1/0x440 [ 7580.069680] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7580.069687] RIP: 0033:0x7f53425a5332 [ 7580.069695] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7580.069703] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7580.069708] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7580.069711] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7580.069713] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7580.069715] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7580.069718] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7580.070727] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7580.070989] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7580.070995] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7580.070998] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7580.071009] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7580.071014] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7580.071019] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7580.071021] RDX: 000000000000503e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7580.071023] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7580.071025] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7580.071028] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7580.071030] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7580.071033] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7580.071036] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7580.071044] Call Trace: [ 7580.071049] [ 7580.071053] ? show_regs.cold.9+0x22/0x2f [ 7580.071062] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7580.071074] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7580.071078] ? watchdog+0x40/0x40 [ 7580.071082] ? hrtimer_interrupt+0x128/0x2c0 [ 7580.071085] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7580.071091] ? apic_timer_interrupt+0xf/0x20 [ 7580.071095] [ 7580.071097] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7580.071102] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7580.071106] do_raw_spin_lock+0x177/0x190 [ 7580.071110] _raw_spin_lock+0x21/0x30 [ 7580.071115] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7580.071385] obd_mod_free+0x46/0x140 [ptlrpc] [ 7580.071640] mdc_commit_open+0x91/0xb0 [mdc] [ 7580.071675] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7580.072020] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7580.072235] after_reply+0xe2c/0x1780 [ptlrpc] [ 7580.072440] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7580.072653] ? __next_timer_interrupt+0x160/0x160 [ 7580.072663] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7580.073018] ? do_wait_intr+0xf0/0xf0 [ 7580.073028] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7580.073240] kthread+0x1d1/0x200 [ 7580.073246] ? set_kthread_struct+0x70/0x70 [ 7580.073249] ret_from_fork+0x1f/0x30 [ 7580.942521] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7580.944627] ? woken_wake_function+0x30/0x30 [ 7580.946637] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7580.951332] kthread+0x1d1/0x200 [ 7580.952341] ? set_kthread_struct+0x70/0x70 [ 7580.954652] ret_from_fork+0x1f/0x30 [ 7600.052012] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 7600.056686] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7600.086826] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7600.093274] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7600.097952] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7600.101343] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7600.105659] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7600.123275] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7600.128619] RAX: 0000000000000607 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7600.143826] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7600.155318] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7600.163967] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7600.168281] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7600.174507] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7600.179273] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7600.181667] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7600.185269] Call Trace: [ 7600.187651] [ 7600.188884] ? show_regs.cold.9+0x22/0x2f [ 7600.190669] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7600.192642] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7600.194833] ? watchdog+0x40/0x40 [ 7600.196269] ? hrtimer_interrupt+0x128/0x2c0 [ 7600.198294] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7600.200595] ? apic_timer_interrupt+0xf/0x20 [ 7600.202567] [ 7600.203404] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7600.205730] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7600.208922] ? newidle_balance.isra.93+0x207/0x580 [ 7600.211598] do_raw_spin_lock+0x177/0x190 [ 7600.213566] _raw_spin_lock+0x21/0x30 [ 7600.215519] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7600.218287] process_one_work+0x2c8/0x700 [ 7600.220580] worker_thread+0x296/0x6e0 [ 7600.223408] ? rescuer_thread+0x570/0x570 [ 7600.225797] kthread+0x1d1/0x200 [ 7600.230622] ? set_kthread_struct+0x70/0x70 [ 7600.231997] ret_from_fork+0x1f/0x30 [ 7605.220858] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4968s! [ 7605.233970] Showing busy workqueues and worker pools: [ 7605.245133] workqueue events: flags=0x0 [ 7605.246316] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7605.258778] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7605.275800] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7605.280032] pending: cpu_latency_work [ptlrpc] [ 7605.289814] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7605.297498] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7605.302188] workqueue events_long: flags=0x0 [ 7605.307698] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7605.310734] pending: lu_objects_flush [obdclass] [ 7605.314087] workqueue events_unbound: flags=0x2 [ 7605.316493] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7605.323021] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7605.327222] workqueue events_power_efficient: flags=0x80 [ 7605.330011] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7605.338199] pending: check_lifetime, do_cache_clean [ 7605.341101] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7605.348597] pending: neigh_periodic_work, neigh_periodic_work [ 7605.353126] workqueue mm_percpu_wq: flags=0x8 [ 7605.354609] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7605.363873] pending: vmstat_update, lru_add_drain_per_cpu [ 7605.369631] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7605.371938] pending: vmstat_update, lru_add_drain_per_cpu [ 7605.383866] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7605.388229] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7605.391616] workqueue xprtiod: flags=0xa [ 7605.393038] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7605.401420] pending: xprt_autoclose, xs_error_handle [ 7605.409087] workqueue ipv6_addrconf: flags=0x40008 [ 7605.411252] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7605.417734] pending: addrconf_verify_work [ 7605.428094] workqueue knbd0-recv: flags=0x1a [ 7605.433821] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7605.439320] in-flight: 651:recv_work [ 7605.447718] workqueue ptlrpc_pinger: flags=0x2 [ 7605.449756] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7605.454912] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7605.458623] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4968s workers=4 idle: 15979 31824 8 [ 7605.466159] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7595s workers=2 idle: 654 [ 7608.057828] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7608.058538] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 7608.058582] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7608.058589] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7608.058656] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7608.058661] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7608.058664] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7608.058676] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7608.058680] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7608.058684] RAX: 0000000000007708 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7608.058687] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7608.058689] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7608.058691] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7608.058693] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7608.058696] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7608.058699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7608.058701] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7608.058710] Call Trace: [ 7608.058716] [ 7608.058721] ? show_regs.cold.9+0x22/0x2f [ 7608.058730] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7608.058738] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7608.058742] ? watchdog+0x40/0x40 [ 7608.058746] ? hrtimer_interrupt+0x128/0x2c0 [ 7608.058750] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7608.058758] ? apic_timer_interrupt+0xf/0x20 [ 7608.058763] [ 7608.058766] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7608.058773] do_raw_spin_lock+0x177/0x190 [ 7608.058778] _raw_spin_lock+0x21/0x30 [ 7608.058783] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7608.060326] ? do_raw_spin_unlock+0x75/0x190 [ 7608.060334] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7608.060538] ? do_wait_intr+0xf0/0xf0 [ 7608.060546] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7608.060784] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7608.061161] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7608.061392] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7608.061547] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7608.061637] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7608.061667] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7608.061695] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7608.061805] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7608.062072] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7608.062101] lmv_intent_open+0x338/0x1150 [lmv] [ 7608.062131] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7608.062219] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7608.062317] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7608.062337] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7608.062421] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7608.062500] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7608.062579] ? do_raw_spin_unlock+0x75/0x190 [ 7608.062585] ? _raw_spin_unlock+0x12/0x30 [ 7608.062591] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7608.062616] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7608.062634] ll_atomic_open+0x36a/0xd70 [lustre] [ 7608.062714] lookup_open+0x356/0xb40 [ 7608.062723] do_last+0x313/0xfc0 [ 7608.062728] ? nd_jump_root+0xe5/0x160 [ 7608.062732] ? path_init+0x437/0x520 [ 7608.062737] path_openat+0xf7/0x500 [ 7608.062742] ? do_raw_spin_unlock+0x75/0x190 [ 7608.062747] do_filp_open+0x99/0x140 [ 7608.062754] ? getname_flags+0x6e/0x330 [ 7608.062759] ? __check_object_size+0xff/0x256 [ 7608.062765] ? do_raw_spin_unlock+0x75/0x190 [ 7608.062769] ? _raw_spin_unlock+0x12/0x30 [ 7608.062773] do_sys_openat2+0x2b4/0x410 [ 7608.062778] do_sys_open+0x73/0xa0 [ 7608.062782] __x64_sys_openat+0x24/0x30 [ 7608.062786] do_syscall_64+0xc1/0x440 [ 7608.062793] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7608.062798] RIP: 0033:0x7f53425a5332 [ 7608.062805] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7608.062810] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7608.062815] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7608.062817] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7608.062819] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7608.062821] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7608.062823] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7608.073229] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7608.073471] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7608.073478] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7608.073482] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7608.073496] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7608.073500] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7608.073504] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7608.073506] RDX: 0000000000000478 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7608.074690] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7608.074694] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7608.074697] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7608.074700] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7608.074703] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7608.074706] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7608.074714] Call Trace: [ 7608.074720] [ 7608.074726] ? show_regs.cold.9+0x22/0x2f [ 7608.074736] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7608.074743] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7608.074747] ? watchdog+0x40/0x40 [ 7608.074752] ? hrtimer_interrupt+0x128/0x2c0 [ 7608.074757] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7608.074765] ? apic_timer_interrupt+0xf/0x20 [ 7608.074770] [ 7608.074773] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7608.074779] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7608.074784] do_raw_spin_lock+0x177/0x190 [ 7608.074788] _raw_spin_lock+0x21/0x30 [ 7608.074795] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7608.080906] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7608.080990] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7608.080998] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7608.081983] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7608.081997] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7608.091251] obd_mod_free+0x46/0x140 [ptlrpc] [ 7608.096014] mdc_commit_open+0x91/0xb0 [mdc] [ 7608.096053] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7608.096261] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7608.132917] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7608.132931] RAX: 0000000000005c22 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7608.132933] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7608.132936] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7608.132938] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7608.132940] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7608.132943] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7608.132946] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7608.132948] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7608.132957] Call Trace: [ 7608.132963] [ 7608.132969] ? show_regs.cold.9+0x22/0x2f [ 7608.132983] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7608.132989] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7608.132994] ? watchdog+0x40/0x40 [ 7608.133192] ? hrtimer_interrupt+0x128/0x2c0 [ 7608.133198] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7608.133207] ? apic_timer_interrupt+0xf/0x20 [ 7608.133211] [ 7608.133215] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7608.133222] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7608.133229] do_raw_spin_lock+0x177/0x190 [ 7608.133234] _raw_spin_lock+0x21/0x30 [ 7608.133239] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7608.133568] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7608.138844] after_reply+0xe2c/0x1780 [ptlrpc] [ 7608.139150] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7608.139467] ? __next_timer_interrupt+0x160/0x160 [ 7608.139476] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7608.139752] ? do_wait_intr+0xf0/0xf0 [ 7608.139762] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7608.139964] kthread+0x1d1/0x200 [ 7608.139971] ? set_kthread_struct+0x70/0x70 [ 7608.139975] ret_from_fork+0x1f/0x30 [ 7608.852024] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7608.858799] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7608.862861] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7608.872947] ? woken_wake_function+0x30/0x30 [ 7608.877540] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7608.888816] kthread+0x1d1/0x200 [ 7608.894853] ? set_kthread_struct+0x70/0x70 [ 7608.899184] ret_from_fork+0x1f/0x30 [ 7632.052030] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 7632.054823] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7632.104384] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7632.116740] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7632.125828] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7632.130259] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7632.134220] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7632.152854] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7632.162337] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7632.169292] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7632.175976] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7632.178720] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7632.187505] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7632.191765] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7632.204778] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7632.209456] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7632.218148] Call Trace: [ 7632.222345] [ 7632.225371] ? show_regs.cold.9+0x22/0x2f [ 7632.231084] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7632.233395] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7632.238389] ? watchdog+0x40/0x40 [ 7632.243096] ? hrtimer_interrupt+0x128/0x2c0 [ 7632.244464] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7632.246199] ? apic_timer_interrupt+0xf/0x20 [ 7632.250398] [ 7632.251472] ? native_safe_halt+0x2e/0x50 [ 7632.254738] kvm_wait+0xcd/0xe0 [ 7632.259151] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7632.262349] ? newidle_balance.isra.93+0x207/0x580 [ 7632.265262] do_raw_spin_lock+0x177/0x190 [ 7632.267225] _raw_spin_lock+0x21/0x30 [ 7632.270009] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7632.272201] process_one_work+0x2c8/0x700 [ 7632.274006] worker_thread+0x296/0x6e0 [ 7632.275605] ? rescuer_thread+0x570/0x570 [ 7632.277161] kthread+0x1d1/0x200 [ 7632.278398] ? set_kthread_struct+0x70/0x70 [ 7632.280426] ret_from_fork+0x1f/0x30 [ 7635.937034] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 4998s! [ 7635.942545] Showing busy workqueues and worker pools: [ 7635.949781] workqueue events: flags=0x0 [ 7635.953637] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7635.960245] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7635.966587] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7635.977383] pending: cpu_latency_work [ptlrpc] [ 7635.979772] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7635.992529] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7635.996505] workqueue events_long: flags=0x0 [ 7636.003305] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7636.008901] pending: lu_objects_flush [obdclass] [ 7636.011474] workqueue events_unbound: flags=0x2 [ 7636.014889] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7636.019236] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7636.024521] workqueue events_power_efficient: flags=0x80 [ 7636.026480] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7636.035716] pending: check_lifetime, do_cache_clean [ 7636.037977] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7636.044289] pending: neigh_periodic_work, neigh_periodic_work [ 7636.052487] workqueue mm_percpu_wq: flags=0x8 [ 7636.053657] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7636.055408] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7636.059568] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 7636.059581] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7636.059649] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7636.059654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7636.059657] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7636.059673] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7636.059678] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7636.059682] RAX: 00000000000027fa RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7636.059685] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7636.059687] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7636.059690] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7636.059698] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7636.059702] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7636.059708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7636.059711] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7636.059722] Call Trace: [ 7636.059728] [ 7636.059733] ? show_regs.cold.9+0x22/0x2f [ 7636.059743] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7636.059755] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7636.059760] ? watchdog+0x40/0x40 [ 7636.059764] ? hrtimer_interrupt+0x128/0x2c0 [ 7636.059768] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7636.059778] ? apic_timer_interrupt+0xf/0x20 [ 7636.059782] [ 7636.059785] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7636.059791] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7636.059796] do_raw_spin_lock+0x177/0x190 [ 7636.059818] _raw_spin_lock+0x21/0x30 [ 7636.059944] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7636.059949] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7636.060009] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7636.060014] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7636.060016] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7636.060024] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7636.060028] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7636.060032] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7636.060035] RDX: 0000000000006ea5 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7636.060037] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7636.060040] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7636.060042] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7636.060045] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7636.060048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7636.060050] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7636.060058] Call Trace: [ 7636.060061] [ 7636.060064] ? show_regs.cold.9+0x22/0x2f [ 7636.060071] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7636.060076] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7636.060080] ? watchdog+0x40/0x40 [ 7636.060085] ? hrtimer_interrupt+0x128/0x2c0 [ 7636.060089] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7636.060095] ? apic_timer_interrupt+0xf/0x20 [ 7636.060099] [ 7636.060101] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7636.060107] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7636.060111] do_raw_spin_lock+0x177/0x190 [ 7636.060116] _raw_spin_lock+0x21/0x30 [ 7636.059824] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7636.060121] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7636.060574] obd_mod_free+0x46/0x140 [ptlrpc] [ 7636.060768] mdc_commit_open+0x91/0xb0 [mdc] [ 7636.060815] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7636.061103] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7636.061505] after_reply+0xe2c/0x1780 [ptlrpc] [ 7636.061940] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7636.062143] ? __next_timer_interrupt+0x160/0x160 [ 7636.062152] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7636.062362] ? do_wait_intr+0xf0/0xf0 [ 7636.062372] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7636.062564] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7636.062757] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7636.063400] kthread+0x1d1/0x200 [ 7636.063406] ? set_kthread_struct+0x70/0x70 [ 7636.063410] ret_from_fork+0x1f/0x30 [ 7636.063612] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7636.063915] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7636.064190] ? woken_wake_function+0x30/0x30 [ 7636.064198] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7636.064615] kthread+0x1d1/0x200 [ 7636.064621] ? set_kthread_struct+0x70/0x70 [ 7636.064624] ret_from_fork+0x1f/0x30 [ 7636.065057] pending: vmstat_update, lru_add_drain_per_cpu [ 7636.065092] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7636.065104] pending: vmstat_update, lru_add_drain_per_cpu [ 7636.065112] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7636.065119] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7636.065144] workqueue xprtiod: flags=0xa [ 7636.065147] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7636.065154] pending: xprt_autoclose, xs_error_handle [ 7636.065599] workqueue ipv6_addrconf: flags=0x40008 [ 7636.065603] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7636.065612] pending: addrconf_verify_work [ 7636.065623] workqueue knbd0-recv: flags=0x1a [ 7636.065625] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7636.065643] in-flight: 651:recv_work [ 7636.065654] workqueue ptlrpc_pinger: flags=0x2 [ 7636.065656] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7636.065661] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7636.065951] pool 8: cpus=0-3 flags=0x4 nice=0 hung=4998s workers=4 idle: 15979 31824 8 [ 7636.065963] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7626s workers=2 idle: 654 [ 7636.745550] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7636.765210] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7636.770529] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7636.774708] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7636.777287] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7636.785467] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7636.798743] RAX: 0000000000007c05 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7636.810609] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7636.813552] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7636.817531] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7636.824588] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7636.827598] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7636.831677] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7636.834110] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7636.838451] Call Trace: [ 7636.839414] [ 7636.840191] ? show_regs.cold.9+0x22/0x2f [ 7636.842337] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7636.846726] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7636.849767] ? watchdog+0x40/0x40 [ 7636.851411] ? hrtimer_interrupt+0x128/0x2c0 [ 7636.853409] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7636.860303] ? apic_timer_interrupt+0xf/0x20 [ 7636.864632] [ 7636.865181] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7636.866708] do_raw_spin_lock+0x177/0x190 [ 7636.885386] _raw_spin_lock+0x21/0x30 [ 7636.886772] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7636.896560] ? do_raw_spin_unlock+0x75/0x190 [ 7636.903293] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7636.913115] ? do_wait_intr+0xf0/0xf0 [ 7636.914639] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7636.919707] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7636.923199] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7636.927910] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7636.931571] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7636.933947] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7636.935772] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7636.937501] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7636.939340] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7636.942643] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7636.945123] lmv_intent_open+0x338/0x1150 [lmv] [ 7636.947290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7636.949663] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7636.952172] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7636.954088] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7636.955752] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7636.958032] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7636.959299] ? do_raw_spin_unlock+0x75/0x190 [ 7636.962278] ? _raw_spin_unlock+0x12/0x30 [ 7636.964309] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7636.965977] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7636.967630] ll_atomic_open+0x36a/0xd70 [lustre] [ 7636.969598] lookup_open+0x356/0xb40 [ 7636.971178] do_last+0x313/0xfc0 [ 7636.974077] ? nd_jump_root+0xe5/0x160 [ 7636.976689] ? path_init+0x437/0x520 [ 7636.979539] path_openat+0xf7/0x500 [ 7636.981550] ? do_raw_spin_unlock+0x75/0x190 [ 7636.983221] do_filp_open+0x99/0x140 [ 7636.984506] ? getname_flags+0x6e/0x330 [ 7636.985710] ? __check_object_size+0xff/0x256 [ 7636.987264] ? do_raw_spin_unlock+0x75/0x190 [ 7636.989301] ? _raw_spin_unlock+0x12/0x30 [ 7636.991151] do_sys_openat2+0x2b4/0x410 [ 7636.992573] do_sys_open+0x73/0xa0 [ 7636.993734] __x64_sys_openat+0x24/0x30 [ 7636.995084] do_syscall_64+0xc1/0x440 [ 7636.998243] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7637.000126] RIP: 0033:0x7f53425a5332 [ 7637.001515] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7637.010355] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7637.016703] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7637.026289] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7637.033056] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7637.039452] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7637.042711] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7664.055946] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7664.056182] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 7664.059510] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7664.059523] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7664.059674] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7664.059680] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7664.059683] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7664.059708] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7664.059712] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7664.059717] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7664.059719] RDX: 00000000000024c4 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7664.059722] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7664.059724] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7664.059727] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7664.059730] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7664.059734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7664.059737] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7664.059745] Call Trace: [ 7664.059759] [ 7664.059765] ? show_regs.cold.9+0x22/0x2f [ 7664.059774] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7664.059781] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7664.059786] ? watchdog+0x40/0x40 [ 7664.059790] ? hrtimer_interrupt+0x128/0x2c0 [ 7664.059795] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7664.059803] ? apic_timer_interrupt+0xf/0x20 [ 7664.059808] [ 7664.059811] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7664.059817] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7664.059821] do_raw_spin_lock+0x177/0x190 [ 7664.059826] _raw_spin_lock+0x21/0x30 [ 7664.059833] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7664.060108] obd_mod_free+0x46/0x140 [ptlrpc] [ 7664.060344] mdc_commit_open+0x91/0xb0 [mdc] [ 7664.060378] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7664.061833] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7664.062031] after_reply+0xe2c/0x1780 [ptlrpc] [ 7664.062627] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 7664.062634] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7664.062702] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7664.062707] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7664.062709] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7664.062719] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7664.062785] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7664.062790] RAX: 0000000000007224 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7664.062793] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7664.062795] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7664.062797] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7664.062799] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7664.062802] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7664.062805] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7664.062807] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7664.062815] Call Trace: [ 7664.062819] [ 7664.062822] ? show_regs.cold.9+0x22/0x2f [ 7664.062832] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7664.062838] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7664.062842] ? watchdog+0x40/0x40 [ 7664.062847] ? hrtimer_interrupt+0x128/0x2c0 [ 7664.062852] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7664.062860] ? apic_timer_interrupt+0xf/0x20 [ 7664.062864] [ 7664.062867] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7664.062872] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7664.062878] do_raw_spin_lock+0x177/0x190 [ 7664.062883] _raw_spin_lock+0x21/0x30 [ 7664.062888] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7664.063333] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7664.064042] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7664.063671] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7664.065036] ? __next_timer_interrupt+0x160/0x160 [ 7664.065348] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7664.065775] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7664.066004] ? woken_wake_function+0x30/0x30 [ 7664.066025] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7664.066963] kthread+0x1d1/0x200 [ 7664.066971] ? set_kthread_struct+0x70/0x70 [ 7664.066975] ret_from_fork+0x1f/0x30 [ 7664.065046] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7664.067811] ? do_wait_intr+0xf0/0xf0 [ 7664.067927] Modules linked in: lustre(O) osp(O) [ 7664.067820] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7664.067937] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7664.067996] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7664.068001] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7664.068004] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7664.068013] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7664.068017] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7664.068021] RAX: 000000000000740b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7664.068024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7664.068026] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7664.068028] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7664.068030] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7664.068033] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7664.068036] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7664.068042] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7664.068041] kthread+0x1d1/0x200 [ 7664.068051] ? set_kthread_struct+0x70/0x70 [ 7664.068054] Call Trace: [ 7664.068057] ret_from_fork+0x1f/0x30 [ 7664.068061] [ 7664.068071] ? show_regs.cold.9+0x22/0x2f [ 7664.068081] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7664.068089] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7664.068094] ? watchdog+0x40/0x40 [ 7664.068099] ? hrtimer_interrupt+0x128/0x2c0 [ 7664.068103] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7664.068108] ? apic_timer_interrupt+0xf/0x20 [ 7664.068113] [ 7664.068115] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7664.068121] do_raw_spin_lock+0x177/0x190 [ 7664.068125] _raw_spin_lock+0x21/0x30 [ 7664.068130] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7664.079851] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7664.079931] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7664.079939] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7664.079947] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7664.081782] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7664.081796] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7664.081800] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7664.081805] RAX: 0000000000007609 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7664.081807] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7664.081809] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7664.081811] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7664.081813] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7664.081816] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7664.081818] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7664.081820] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7664.081830] Call Trace: [ 7664.081835] [ 7664.081840] ? show_regs.cold.9+0x22/0x2f [ 7664.081848] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7664.081855] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7664.081859] ? watchdog+0x40/0x40 [ 7664.081863] ? hrtimer_interrupt+0x128/0x2c0 [ 7664.081867] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7664.081875] ? apic_timer_interrupt+0xf/0x20 [ 7664.081880] [ 7664.081883] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7664.081888] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7664.081892] ? newidle_balance.isra.93+0x207/0x580 [ 7664.081897] do_raw_spin_lock+0x177/0x190 [ 7664.081902] _raw_spin_lock+0x21/0x30 [ 7664.081907] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7664.109240] ? do_raw_spin_unlock+0x75/0x190 [ 7664.114848] process_one_work+0x2c8/0x700 [ 7664.114870] worker_thread+0x296/0x6e0 [ 7664.114875] ? rescuer_thread+0x570/0x570 [ 7664.114880] kthread+0x1d1/0x200 [ 7664.114884] ? set_kthread_struct+0x70/0x70 [ 7664.114889] ret_from_fork+0x1f/0x30 [ 7664.770094] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7664.772530] ? do_wait_intr+0xf0/0xf0 [ 7664.773808] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7664.775963] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7664.778308] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7664.781091] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7664.783239] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7664.788399] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7664.790188] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7664.795279] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7664.803808] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7664.806142] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7664.812582] lmv_intent_open+0x338/0x1150 [lmv] [ 7664.814874] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7664.816758] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7664.818495] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7664.819730] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7664.821569] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7664.823856] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7664.825602] ? do_raw_spin_unlock+0x75/0x190 [ 7664.826931] ? _raw_spin_unlock+0x12/0x30 [ 7664.828226] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7664.829858] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7664.831694] ll_atomic_open+0x36a/0xd70 [lustre] [ 7664.833764] lookup_open+0x356/0xb40 [ 7664.835114] do_last+0x313/0xfc0 [ 7664.836565] ? nd_jump_root+0xe5/0x160 [ 7664.838013] ? path_init+0x437/0x520 [ 7664.839346] path_openat+0xf7/0x500 [ 7664.840889] ? do_raw_spin_unlock+0x75/0x190 [ 7664.842484] do_filp_open+0x99/0x140 [ 7664.849199] ? getname_flags+0x6e/0x330 [ 7664.851060] ? __check_object_size+0xff/0x256 [ 7664.856755] ? do_raw_spin_unlock+0x75/0x190 [ 7664.860590] ? _raw_spin_unlock+0x12/0x30 [ 7664.864961] do_sys_openat2+0x2b4/0x410 [ 7664.872140] do_sys_open+0x73/0xa0 [ 7664.873467] __x64_sys_openat+0x24/0x30 [ 7664.874515] do_syscall_64+0xc1/0x440 [ 7664.875600] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7664.881785] RIP: 0033:0x7f53425a5332 [ 7664.883595] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7664.898557] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7664.905541] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7664.910952] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7664.917274] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7664.926629] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7664.934342] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7666.146111] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5029s! [ 7666.149029] Showing busy workqueues and worker pools: [ 7666.153435] workqueue events: flags=0x0 [ 7666.156205] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7666.161158] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7666.170112] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7666.175804] pending: cpu_latency_work [ptlrpc] [ 7666.190755] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7666.207765] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7666.220931] workqueue events_long: flags=0x0 [ 7666.224534] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7666.236686] pending: lu_objects_flush [obdclass] [ 7666.242756] workqueue events_unbound: flags=0x2 [ 7666.256292] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7666.264864] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7666.280937] workqueue events_power_efficient: flags=0x80 [ 7666.287089] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7666.290083] pending: check_lifetime, do_cache_clean [ 7666.295162] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7666.301185] pending: neigh_periodic_work, neigh_periodic_work [ 7666.311951] workqueue mm_percpu_wq: flags=0x8 [ 7666.316207] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7666.321637] pending: vmstat_update, lru_add_drain_per_cpu [ 7666.325070] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7666.331810] pending: vmstat_update, lru_add_drain_per_cpu [ 7666.336897] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7666.343730] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7666.352632] workqueue xprtiod: flags=0xa [ 7666.356131] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7666.360137] pending: xprt_autoclose, xs_error_handle [ 7666.366137] workqueue ipv6_addrconf: flags=0x40008 [ 7666.371050] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7666.376968] pending: addrconf_verify_work [ 7666.379483] workqueue knbd0-recv: flags=0x1a [ 7666.381203] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7666.385916] in-flight: 651:recv_work [ 7666.389181] workqueue ptlrpc_pinger: flags=0x2 [ 7666.394830] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7666.397925] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7666.402553] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5029s workers=4 idle: 15979 31824 8 [ 7666.410008] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7656s workers=2 idle: 654 [ 7692.053196] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 7692.055026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7692.058107] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7692.058182] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7692.058189] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7692.058196] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7692.058469] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7692.058481] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7692.058486] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7692.058491] RAX: 0000000000007a86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7692.058494] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7692.058496] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7692.058499] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7692.058501] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7692.058505] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7692.058508] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7692.058511] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7692.058519] Call Trace: [ 7692.058526] [ 7692.058531] ? show_regs.cold.9+0x22/0x2f [ 7692.058541] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7692.058548] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7692.058553] ? watchdog+0x40/0x40 [ 7692.058557] ? hrtimer_interrupt+0x128/0x2c0 [ 7692.058562] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7692.058570] ? apic_timer_interrupt+0xf/0x20 [ 7692.058575] [ 7692.058579] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7692.058584] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7692.058588] ? newidle_balance.isra.93+0x207/0x580 [ 7692.058594] do_raw_spin_lock+0x177/0x190 [ 7692.058874] _raw_spin_lock+0x21/0x30 [ 7692.058881] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7692.059089] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 7692.059096] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [ 7692.059102] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7692.059108] mdt(O) [ 7692.059111] Modules linked in: [ 7692.059113] mdd(O) [ 7692.059115] lustre(O) [ 7692.059117] mgs(O) [ 7692.059121] osp(O) [ 7692.059122] lquota(O) [ 7692.059127] ofd(O) [ 7692.059128] lfsck(O) [ 7692.059132] lod(O) [ 7692.059135] obdecho(O) [ 7692.059137] mdt(O) [ 7692.059139] mgc(O) [ 7692.059143] mdd(O) [ 7692.059144] mdc(O) [ 7692.059148] mgs(O) [ 7692.059150] lov(O) [ 7692.059154] lquota(O) [ 7692.059155] osc(O) [ 7692.059160] lfsck(O) [ 7692.059161] lmv(O) [ 7692.059165] obdecho(O) [ 7692.059166] ec(O) [ 7692.059170] mgc(O) [ 7692.059172] fid(O) [ 7692.059174] mdc(O) [ 7692.059177] fld(O) [ 7692.059179] lov(O) [ 7692.059181] ptlrpc_gss(O) [ 7692.059184] osc(O) [ 7692.059186] ptlrpc(O) [ 7692.059188] lmv(O) [ 7692.059190] obdclass(O) [ 7692.059193] ec(O) [ 7692.059195] ksocklnd(O) [ 7692.059197] fid(O) [ 7692.059200] lnet(O) [ 7692.059202] fld(O) [ 7692.059206] libcfs(O) [ 7692.059207] ptlrpc_gss(O) [ 7692.059210] rpcsec_gss_krb5 [ 7692.059212] ptlrpc(O) [ 7692.059214] auth_rpcgss [ 7692.059217] obdclass(O) [ 7692.059218] nfsv4 [ 7692.059222] ksocklnd(O) [ 7692.059224] dns_resolver [ 7692.059228] lnet(O) [ 7692.059230] intel_rapl_msr [ 7692.059234] libcfs(O) [ 7692.059235] intel_rapl_common [ 7692.059239] rpcsec_gss_krb5 [ 7692.059240] sb_edac [ 7692.059244] auth_rpcgss [ 7692.059245] rapl [ 7692.059249] nfsv4 [ 7692.059251] i2c_piix4 [ 7692.059255] dns_resolver [ 7692.059256] pcspkr [ 7692.059260] intel_rapl_msr [ 7692.059262] squashfs [ 7692.059266] intel_rapl_common [ 7692.059267] ata_generic [ 7692.059271] sb_edac [ 7692.059273] crct10dif_pclmul [ 7692.059275] rapl [ 7692.059278] crc32_pclmul [ 7692.059280] i2c_piix4 [ 7692.059284] crc32c_intel [ 7692.059285] pcspkr [ 7692.059289] ghash_clmulni_intel [ 7692.059290] squashfs [ 7692.059295] ata_piix [ 7692.059295] ata_generic crct10dif_pclmul [ 7692.059301] serio_raw [ 7692.059305] crc32_pclmul [ 7692.059307] libata [ 7692.059309] crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7692.059314] dm_mirror [ 7692.059327] [ 7692.059329] dm_region_hash [ 7692.059331] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7692.059334] dm_log dm_mod [ 7692.059341] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7692.059342] sha512_ssse3 [ 7692.059347] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7692.059355] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7692.059356] sha512_generic [ 7692.059362] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 [ 7692.059363] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7692.059368] ORIG_RAX: ffffffffffffff13 [ 7692.059373] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7692.059374] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7692.059380] RDX: 00000000000007bc RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7692.059378] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7692.059384] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7692.059391] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7692.059390] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7692.059395] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7692.059399] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 [ 7692.059402] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7692.059406] ORIG_RAX: ffffffffffffff13 [ 7692.059410] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7692.059411] RAX: 0000000000003875 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7692.059416] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7692.059417] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7692.059426] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7692.059430] Call Trace: [ 7692.059440] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7692.059443] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7692.059447] [ 7692.059448] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7692.059452] ? show_regs.cold.9+0x22/0x2f [ 7692.059454] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7692.059463] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7692.059468] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7692.059473] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7692.059479] Call Trace: [ 7692.059480] ? watchdog+0x40/0x40 [ 7692.059485] [ 7692.059490] ? hrtimer_interrupt+0x128/0x2c0 [ 7692.059491] ? show_regs.cold.9+0x22/0x2f [ 7692.059500] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7692.059503] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7692.059513] ? apic_timer_interrupt+0xf/0x20 [ 7692.059514] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7692.059524] [ 7692.059525] ? watchdog+0x40/0x40 [ 7692.059531] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7692.059539] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7692.059544] do_raw_spin_lock+0x177/0x190 [ 7692.059549] _raw_spin_lock+0x21/0x30 [ 7692.059719] ? hrtimer_interrupt+0x128/0x2c0 [ 7692.059725] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7692.059731] ? apic_timer_interrupt+0xf/0x20 [ 7692.059736] [ 7692.059740] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7692.059745] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7692.059751] do_raw_spin_lock+0x177/0x190 [ 7692.059756] _raw_spin_lock+0x21/0x30 [ 7692.059761] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7692.060002] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7692.059554] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7692.060208] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7692.060489] obd_mod_free+0x46/0x140 [ptlrpc] [ 7692.061068] mdc_commit_open+0x91/0xb0 [mdc] [ 7692.061106] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7692.061517] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7692.062012] after_reply+0xe2c/0x1780 [ptlrpc] [ 7692.062431] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7692.061651] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7692.062955] ? __next_timer_interrupt+0x160/0x160 [ 7692.063045] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7692.062965] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7692.063521] ? do_wait_intr+0xf0/0xf0 [ 7692.063736] ? woken_wake_function+0x30/0x30 [ 7692.063743] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7692.064047] kthread+0x1d1/0x200 [ 7692.064054] ? set_kthread_struct+0x70/0x70 [ 7692.064057] ret_from_fork+0x1f/0x30 [ 7692.063531] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7692.064202] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7692.064272] kthread+0x1d1/0x200 [ 7692.064273] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7692.064283] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7692.064282] ? set_kthread_struct+0x70/0x70 [ 7692.064287] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7692.064292] ret_from_fork+0x1f/0x30 [ 7692.064300] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7692.064307] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7692.064312] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7692.064315] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7692.064317] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7692.064319] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7692.064322] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7692.064326] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7692.064329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7692.064331] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7692.064341] Call Trace: [ 7692.064345] [ 7692.064349] ? show_regs.cold.9+0x22/0x2f [ 7692.064357] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7692.064362] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7692.064366] ? watchdog+0x40/0x40 [ 7692.064370] ? hrtimer_interrupt+0x128/0x2c0 [ 7692.064374] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7692.064379] ? apic_timer_interrupt+0xf/0x20 [ 7692.064384] [ 7692.064387] ? native_safe_halt+0x2e/0x50 [ 7692.064392] kvm_wait+0xcd/0xe0 [ 7692.064398] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7692.064405] do_raw_spin_lock+0x177/0x190 [ 7692.064410] _raw_spin_lock+0x21/0x30 [ 7692.064414] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7692.064618] ? do_raw_spin_unlock+0x75/0x190 [ 7692.064623] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7692.065282] ? do_wait_intr+0xf0/0xf0 [ 7692.065290] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7692.065496] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7692.065740] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7692.067063] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7692.094182] process_one_work+0x2c8/0x700 [ 7692.094202] worker_thread+0x296/0x6e0 [ 7692.094207] ? rescuer_thread+0x570/0x570 [ 7692.094211] kthread+0x1d1/0x200 [ 7692.094215] ? set_kthread_struct+0x70/0x70 [ 7692.094219] ret_from_fork+0x1f/0x30 [ 7693.222343] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7693.225790] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7693.228772] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7693.231623] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7693.235485] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7693.239400] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7693.242722] lmv_intent_open+0x338/0x1150 [lmv] [ 7693.245213] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7693.248549] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7693.253121] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7693.255977] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7693.265130] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7693.268331] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7693.271512] ? do_raw_spin_unlock+0x75/0x190 [ 7693.273363] ? _raw_spin_unlock+0x12/0x30 [ 7693.278783] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7693.280677] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7693.289450] ll_atomic_open+0x36a/0xd70 [lustre] [ 7693.299604] lookup_open+0x356/0xb40 [ 7693.301572] do_last+0x313/0xfc0 [ 7693.303552] ? nd_jump_root+0xe5/0x160 [ 7693.305849] ? path_init+0x437/0x520 [ 7693.307155] path_openat+0xf7/0x500 [ 7693.311718] ? do_raw_spin_unlock+0x75/0x190 [ 7693.314758] do_filp_open+0x99/0x140 [ 7693.317641] ? getname_flags+0x6e/0x330 [ 7693.321542] ? __check_object_size+0xff/0x256 [ 7693.325223] ? do_raw_spin_unlock+0x75/0x190 [ 7693.327831] ? _raw_spin_unlock+0x12/0x30 [ 7693.330259] do_sys_openat2+0x2b4/0x410 [ 7693.332379] do_sys_open+0x73/0xa0 [ 7693.334239] __x64_sys_openat+0x24/0x30 [ 7693.337513] do_syscall_64+0xc1/0x440 [ 7693.339473] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7693.343590] RIP: 0033:0x7f53425a5332 [ 7693.347902] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7693.362433] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7693.366334] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7693.372518] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7693.380542] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7693.387242] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7693.390555] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7696.864047] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5059s! [ 7696.866725] Showing busy workqueues and worker pools: [ 7696.869338] workqueue events: flags=0x0 [ 7696.875548] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7696.886982] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7696.901174] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7696.906546] pending: cpu_latency_work [ptlrpc] [ 7696.915495] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7696.925582] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7696.941502] workqueue events_long: flags=0x0 [ 7696.954411] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7696.970368] pending: lu_objects_flush [obdclass] [ 7696.975682] workqueue events_unbound: flags=0x2 [ 7696.981620] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7696.991239] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7697.003978] workqueue events_power_efficient: flags=0x80 [ 7697.008990] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7697.019482] pending: check_lifetime, do_cache_clean [ 7697.027333] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7697.037275] pending: neigh_periodic_work, neigh_periodic_work [ 7697.046090] workqueue mm_percpu_wq: flags=0x8 [ 7697.047491] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7697.058197] pending: vmstat_update, lru_add_drain_per_cpu [ 7697.062123] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7697.074981] pending: vmstat_update, lru_add_drain_per_cpu [ 7697.083192] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7697.094472] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7697.101328] workqueue xprtiod: flags=0xa [ 7697.109184] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7697.116344] pending: xprt_autoclose, xs_error_handle [ 7697.122756] workqueue ipv6_addrconf: flags=0x40008 [ 7697.124410] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7697.134389] pending: addrconf_verify_work [ 7697.141384] workqueue knbd0-recv: flags=0x1a [ 7697.144482] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7697.155370] in-flight: 651:recv_work [ 7697.160437] workqueue ptlrpc_pinger: flags=0x2 [ 7697.164585] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7697.173107] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7697.175288] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5060s workers=4 idle: 15979 31824 8 [ 7697.192108] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7687s workers=2 idle: 654 [ 7720.056210] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7720.060949] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7720.061317] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7720.061588] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7720.061594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7720.061597] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7720.061610] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7720.061615] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7720.061619] RAX: 0000000000003e44 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7720.061622] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7720.061624] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7720.061626] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7720.061628] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7720.061632] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7720.061635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7720.061638] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7720.061646] Call Trace: [ 7720.061651] [ 7720.061656] ? show_regs.cold.9+0x22/0x2f [ 7720.061666] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7720.061672] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7720.061677] ? watchdog+0x40/0x40 [ 7720.061682] ? hrtimer_interrupt+0x128/0x2c0 [ 7720.061686] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7720.061694] ? apic_timer_interrupt+0xf/0x20 [ 7720.061699] [ 7720.061702] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7720.061710] do_raw_spin_lock+0x177/0x190 [ 7720.061716] _raw_spin_lock+0x21/0x30 [ 7720.061721] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7720.061975] ? do_raw_spin_unlock+0x75/0x190 [ 7720.061981] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7720.062306] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 7720.062313] Modules linked in: [ 7720.062310] ? do_wait_intr+0xf0/0xf0 [ 7720.062315] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7720.062393] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7720.062399] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7720.062401] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7720.062411] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7720.062416] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7720.062420] RAX: 000000000000746e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7720.062423] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7720.062426] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7720.062428] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7720.062324] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7720.062884] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7720.063246] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7720.063251] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7720.063254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7720.063256] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7720.063263] Call Trace: [ 7720.063267] [ 7720.063270] ? show_regs.cold.9+0x22/0x2f [ 7720.063279] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7720.063284] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7720.063288] ? watchdog+0x40/0x40 [ 7720.063292] ? hrtimer_interrupt+0x128/0x2c0 [ 7720.063296] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7720.063302] ? apic_timer_interrupt+0xf/0x20 [ 7720.063307] [ 7720.063310] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7720.063315] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7720.063321] do_raw_spin_lock+0x177/0x190 [ 7720.063326] _raw_spin_lock+0x21/0x30 [ 7720.063097] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7720.063342] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7720.063898] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7720.064653] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7720.064765] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7720.064800] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7720.064829] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7720.064855] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7720.064938] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7720.064128] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7720.070050] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7720.070279] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7720.075840] ? woken_wake_function+0x30/0x30 [ 7720.075854] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7720.076084] kthread+0x1d1/0x200 [ 7720.076090] ? set_kthread_struct+0x70/0x70 [ 7720.076094] ret_from_fork+0x1f/0x30 [ 7720.077552] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7720.077631] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7720.077638] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7720.077641] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7720.077654] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7720.077658] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7720.077663] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7720.077666] RDX: 0000000000003230 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7720.077668] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7720.077670] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7720.077673] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7720.077676] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7720.077679] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7720.077681] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7720.077690] Call Trace: [ 7720.077694] [ 7720.077698] ? show_regs.cold.9+0x22/0x2f [ 7720.077707] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7720.077714] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7720.077719] ? watchdog+0x40/0x40 [ 7720.077723] ? hrtimer_interrupt+0x128/0x2c0 [ 7720.077728] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7720.077736] ? apic_timer_interrupt+0xf/0x20 [ 7720.077741] [ 7720.077743] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7720.077748] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7720.077753] do_raw_spin_lock+0x177/0x190 [ 7720.077758] _raw_spin_lock+0x21/0x30 [ 7720.077764] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7720.078088] obd_mod_free+0x46/0x140 [ptlrpc] [ 7720.078291] mdc_commit_open+0x91/0xb0 [mdc] [ 7720.110820] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7720.110876] lmv_intent_open+0x338/0x1150 [lmv] [ 7720.110913] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7720.111024] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7720.111213] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7720.111249] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7720.111390] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7720.111865] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7720.111948] ? do_raw_spin_unlock+0x75/0x190 [ 7720.111957] ? _raw_spin_unlock+0x12/0x30 [ 7720.111965] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7720.111988] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7720.112008] ll_atomic_open+0x36a/0xd70 [lustre] [ 7720.112089] lookup_open+0x356/0xb40 [ 7720.112099] do_last+0x313/0xfc0 [ 7720.112390] ? nd_jump_root+0xe5/0x160 [ 7720.112455] ? path_init+0x437/0x520 [ 7720.112460] path_openat+0xf7/0x500 [ 7720.112465] ? do_raw_spin_unlock+0x75/0x190 [ 7720.112470] do_filp_open+0x99/0x140 [ 7720.112477] ? getname_flags+0x6e/0x330 [ 7720.112482] ? __check_object_size+0xff/0x256 [ 7720.112488] ? do_raw_spin_unlock+0x75/0x190 [ 7720.112492] ? _raw_spin_unlock+0x12/0x30 [ 7720.112497] do_sys_openat2+0x2b4/0x410 [ 7720.112502] do_sys_open+0x73/0xa0 [ 7720.112506] __x64_sys_openat+0x24/0x30 [ 7720.112580] do_syscall_64+0xc1/0x440 [ 7720.112586] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7720.112592] RIP: 0033:0x7f53425a5332 [ 7720.112600] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7720.112603] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7720.112608] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7720.112610] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7720.112612] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7720.112614] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7720.112617] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7721.253349] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7721.255219] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7721.262985] after_reply+0xe2c/0x1780 [ptlrpc] [ 7721.266179] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7721.271768] ? __next_timer_interrupt+0x160/0x160 [ 7721.288782] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7721.303615] ? do_wait_intr+0xf0/0xf0 [ 7721.306061] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7721.323754] kthread+0x1d1/0x200 [ 7721.324999] ? set_kthread_struct+0x70/0x70 [ 7721.341968] ret_from_fork+0x1f/0x30 [ 7724.057402] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 7724.073997] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7724.149359] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7724.173601] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7724.185325] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7724.197106] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7724.201906] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7724.238000] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7724.252475] RAX: 0000000000005588 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7724.264973] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7724.280541] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7724.291693] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7724.296788] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7724.301010] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7724.305724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7724.310167] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7724.315005] Call Trace: [ 7724.323196] [ 7724.325724] ? show_regs.cold.9+0x22/0x2f [ 7724.329504] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7724.332186] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7724.334761] ? watchdog+0x40/0x40 [ 7724.338236] ? hrtimer_interrupt+0x128/0x2c0 [ 7724.340680] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7724.343698] ? apic_timer_interrupt+0xf/0x20 [ 7724.346161] [ 7724.347558] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7724.359261] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7724.364934] ? newidle_balance.isra.93+0x207/0x580 [ 7724.371983] do_raw_spin_lock+0x177/0x190 [ 7724.375224] _raw_spin_lock+0x21/0x30 [ 7724.377671] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7724.381512] process_one_work+0x2c8/0x700 [ 7724.382876] worker_thread+0x296/0x6e0 [ 7724.385888] ? rescuer_thread+0x570/0x570 [ 7724.389452] kthread+0x1d1/0x200 [ 7724.399178] ? set_kthread_struct+0x70/0x70 [ 7724.403697] ret_from_fork+0x1f/0x30 [ 7727.590824] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5090s! [ 7727.596009] Showing busy workqueues and worker pools: [ 7727.608571] workqueue events: flags=0x0 [ 7727.609977] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7727.624685] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7727.642564] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7727.645227] pending: cpu_latency_work [ptlrpc] [ 7727.654516] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7727.664448] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7727.682375] workqueue events_long: flags=0x0 [ 7727.686087] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7727.699854] pending: lu_objects_flush [obdclass] [ 7727.708540] workqueue events_unbound: flags=0x2 [ 7727.717957] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7727.731549] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7727.746787] workqueue events_power_efficient: flags=0x80 [ 7727.748768] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7727.754904] pending: check_lifetime, do_cache_clean [ 7727.758934] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7727.765865] pending: neigh_periodic_work, neigh_periodic_work [ 7727.769612] workqueue mm_percpu_wq: flags=0x8 [ 7727.775873] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7727.779766] pending: vmstat_update, lru_add_drain_per_cpu [ 7727.784023] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7727.787322] pending: vmstat_update, lru_add_drain_per_cpu [ 7727.791655] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7727.795869] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7727.798903] workqueue xprtiod: flags=0xa [ 7727.800340] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7727.805029] pending: xprt_autoclose, xs_error_handle [ 7727.807833] workqueue ipv6_addrconf: flags=0x40008 [ 7727.810516] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7727.815106] pending: addrconf_verify_work [ 7727.816521] workqueue knbd0-recv: flags=0x1a [ 7727.820354] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7727.825611] in-flight: 651:recv_work [ 7727.828440] workqueue ptlrpc_pinger: flags=0x2 [ 7727.834561] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7727.841487] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7727.852877] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5090s workers=4 idle: 15979 31824 8 [ 7727.861597] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7718s workers=2 idle: 654 [ 7734.575050] rcu: INFO: rcu_sched self-detected stall on CPU [ 7734.577527] rcu: 2-....: (3042704 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 7734.593498] (t=5640149 jiffies g=274185 q=65253) [ 7734.598406] rcu: rcu_sched kthread starved for 5098515 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 7734.608563] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7734.623582] rcu: RCU grace-period kthread stack dump: [ 7734.627991] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7734.638979] Call Trace: [ 7734.646392] __schedule+0x351/0xcb0 [ 7734.647793] schedule+0xc0/0x180 [ 7734.648971] schedule_timeout+0xb4/0x190 [ 7734.654347] ? do_raw_spin_unlock+0x75/0x190 [ 7734.660230] ? __next_timer_interrupt+0x160/0x160 [ 7734.664732] ? prepare_to_swait_event+0xf3/0x140 [ 7734.671917] rcu_gp_kthread+0x8dd/0x15b0 [ 7734.679505] ? force_qs_rnp+0x2c0/0x2c0 [ 7734.682213] kthread+0x1d1/0x200 [ 7734.689193] ? set_kthread_struct+0x70/0x70 [ 7734.691670] ret_from_fork+0x1f/0x30 [ 7734.698371] rcu: Stack dump where RCU GP kthread last ran: [ 7734.702786] Sending NMI from CPU 2 to CPUs 0: [ 7734.717932] NMI backtrace for cpu 0 [ 7734.717940] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7734.717942] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7734.717943] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7734.717960] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7734.717962] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7734.717964] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 [ 7734.717966] RAX: 0000000000004a91 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7734.717968] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7734.717969] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7734.717971] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7734.717972] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7734.717981] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7734.717983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7734.717984] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7734.717985] Call Trace: [ 7734.717987] [ 7734.717987] ? show_regs.cold.9+0x22/0x2f [ 7734.717989] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7734.717990] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7734.717991] ? nmi_handle+0xbd/0x210 [ 7734.717992] ? default_do_nmi+0x4b/0x1b0 [ 7734.717992] ? do_nmi+0x280/0x380 [ 7734.717993] ? end_repeat_nmi+0x16/0x1f [ 7734.717995] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7734.717996] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7734.717997] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7734.717998] [ 7734.717999] ? newidle_balance.isra.93+0x207/0x580 [ 7734.718000] do_raw_spin_lock+0x177/0x190 [ 7734.718001] _raw_spin_lock+0x21/0x30 [ 7734.718002] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7734.718003] process_one_work+0x2c8/0x700 [ 7734.718004] worker_thread+0x296/0x6e0 [ 7734.718006] ? rescuer_thread+0x570/0x570 [ 7734.718007] kthread+0x1d1/0x200 [ 7734.718008] ? set_kthread_struct+0x70/0x70 [ 7734.718009] ret_from_fork+0x1f/0x30 [ 7734.721461] Sending NMI from CPU 2 to CPUs 1: [ 7735.024927] NMI backtrace for cpu 1 [ 7735.024935] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7735.024938] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7735.024939] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7735.024941] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7735.024943] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [ 7735.024946] RAX: 0000000000002699 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7735.024947] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7735.024949] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7735.024950] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7735.024952] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7735.024953] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7735.024955] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7735.024956] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7735.024958] Call Trace: [ 7735.024958] [ 7735.024959] ? show_regs.cold.9+0x22/0x2f [ 7735.024960] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7735.024961] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7735.024962] ? nmi_handle+0xbd/0x210 [ 7735.024963] ? default_do_nmi+0x4b/0x1b0 [ 7735.024964] ? do_nmi+0x280/0x380 [ 7735.024965] ? end_repeat_nmi+0x16/0x1f [ 7735.024967] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7735.024968] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7735.024969] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7735.024970] [ 7735.024971] do_raw_spin_lock+0x177/0x190 [ 7735.024972] _raw_spin_lock+0x21/0x30 [ 7735.024974] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7735.024975] ? do_raw_spin_unlock+0x75/0x190 [ 7735.024976] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7735.024977] ? do_wait_intr+0xf0/0xf0 [ 7735.024978] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7735.024979] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7735.024981] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7735.024982] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7735.024983] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7735.024985] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7735.024986] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7735.024988] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7735.024989] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7735.024990] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7735.024991] lmv_intent_open+0x338/0x1150 [lmv] [ 7735.024993] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7735.024994] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7735.024995] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7735.024997] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7735.024998] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7735.024999] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7735.025001] ? do_raw_spin_unlock+0x75/0x190 [ 7735.025002] ? _raw_spin_unlock+0x12/0x30 [ 7735.025003] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7735.025004] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7735.025006] ll_atomic_open+0x36a/0xd70 [lustre] [ 7735.025007] lookup_open+0x356/0xb40 [ 7735.025008] do_last+0x313/0xfc0 [ 7735.025009] ? nd_jump_root+0xe5/0x160 [ 7735.025010] ? path_init+0x437/0x520 [ 7735.025011] path_openat+0xf7/0x500 [ 7735.025012] ? do_raw_spin_unlock+0x75/0x190 [ 7735.025013] do_filp_open+0x99/0x140 [ 7735.025014] ? getname_flags+0x6e/0x330 [ 7735.025014] ? __check_object_size+0xff/0x256 [ 7735.025015] ? do_raw_spin_unlock+0x75/0x190 [ 7735.025016] ? _raw_spin_unlock+0x12/0x30 [ 7735.025017] do_sys_openat2+0x2b4/0x410 [ 7735.025018] do_sys_open+0x73/0xa0 [ 7735.025019] __x64_sys_openat+0x24/0x30 [ 7735.025020] do_syscall_64+0xc1/0x440 [ 7735.025021] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7735.025022] RIP: 0033:0x7f53425a5332 [ 7735.025023] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7735.025025] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7735.025028] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7735.025029] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7735.025031] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7735.025032] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7735.025034] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7735.025826] NMI backtrace for cpu 2 [ 7735.207017] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7735.211037] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7735.214345] Call Trace: [ 7735.215295] [ 7735.215986] ? dump_stack+0xbb/0x10e [ 7735.217284] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7735.219025] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7735.220806] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7735.222564] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7735.224528] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7735.226305] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7735.227790] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7735.229468] ? update_process_times+0x6d/0xc0 [ 7735.230921] ? tick_sched_handle.isra.18+0x27/0x90 [ 7735.232627] ? tick_sched_timer+0x53/0xb0 [ 7735.234033] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7735.235631] ? tick_sched_handle.isra.18+0x90/0x90 [ 7735.237421] ? hrtimer_interrupt+0x128/0x2c0 [ 7735.239012] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7735.240736] ? apic_timer_interrupt+0xf/0x20 [ 7735.242344] [ 7735.243037] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7735.244956] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7735.246978] ? do_raw_spin_lock+0x177/0x190 [ 7735.248608] ? _raw_spin_lock+0x21/0x30 [ 7735.249803] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7735.251703] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7735.253621] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7735.255328] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7735.257359] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7735.259546] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7735.261373] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7735.263435] ? __next_timer_interrupt+0x160/0x160 [ 7735.265138] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7735.266887] ? do_wait_intr+0xf0/0xf0 [ 7735.268265] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7735.270414] ? kthread+0x1d1/0x200 [ 7735.271633] ? set_kthread_struct+0x70/0x70 [ 7735.273162] ? ret_from_fork+0x1f/0x30 [ 7748.068918] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 7748.080563] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7748.140772] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7748.153343] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7748.168397] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7748.176177] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7748.195488] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7748.205226] RAX: 0000000000006086 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7748.209962] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7748.217195] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7748.223987] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7748.230307] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7748.237875] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7748.246548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7748.251621] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7748.259351] Call Trace: [ 7748.261593] [ 7748.264586] ? show_regs.cold.9+0x22/0x2f [ 7748.265988] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7748.275336] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7748.282448] ? watchdog+0x40/0x40 [ 7748.287585] ? hrtimer_interrupt+0x128/0x2c0 [ 7748.294703] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7748.302610] ? apic_timer_interrupt+0xf/0x20 [ 7748.310512] [ 7748.312462] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7748.320679] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7748.328690] do_raw_spin_lock+0x177/0x190 [ 7748.332992] _raw_spin_lock+0x21/0x30 [ 7748.334166] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7748.340670] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7748.348407] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7748.354247] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7748.360720] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7748.369192] ? woken_wake_function+0x30/0x30 [ 7748.375674] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7748.386879] kthread+0x1d1/0x200 [ 7748.390198] ? set_kthread_struct+0x70/0x70 [ 7748.397169] ret_from_fork+0x1f/0x30 [ 7758.303756] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5121s! [ 7758.307471] Showing busy workqueues and worker pools: [ 7758.311643] workqueue events: flags=0x0 [ 7758.318147] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7758.324500] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7758.339184] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7758.343222] pending: cpu_latency_work [ptlrpc] [ 7758.348364] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7758.352557] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7758.357610] workqueue events_long: flags=0x0 [ 7758.362366] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7758.367900] pending: lu_objects_flush [obdclass] [ 7758.372451] workqueue events_unbound: flags=0x2 [ 7758.376918] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7758.381086] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7758.386114] workqueue events_power_efficient: flags=0x80 [ 7758.390124] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7758.394834] pending: check_lifetime, do_cache_clean [ 7758.399365] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7758.403777] pending: neigh_periodic_work, neigh_periodic_work [ 7758.407249] workqueue mm_percpu_wq: flags=0x8 [ 7758.409280] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7758.415139] pending: vmstat_update, lru_add_drain_per_cpu [ 7758.418344] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7758.424305] pending: vmstat_update, lru_add_drain_per_cpu [ 7758.427561] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7758.433157] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7758.437602] workqueue xprtiod: flags=0xa [ 7758.442471] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7758.448821] pending: xprt_autoclose, xs_error_handle [ 7758.453912] workqueue ipv6_addrconf: flags=0x40008 [ 7758.455979] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7758.461055] pending: addrconf_verify_work [ 7758.467491] workqueue knbd0-recv: flags=0x1a [ 7758.471096] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7758.476312] in-flight: 651:recv_work [ 7758.480960] workqueue ptlrpc_pinger: flags=0x2 [ 7758.484568] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7758.490991] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7758.499482] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5121s workers=4 idle: 15979 31824 8 [ 7758.503477] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7748s workers=2 idle: 654 [ 7760.060021] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 7760.060793] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 7760.069623] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7760.069720] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7760.069728] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7760.069731] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7760.069749] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7760.069754] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7760.069758] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7760.069761] RDX: 0000000000007e8c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7760.069763] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7760.069766] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7760.069768] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7760.069772] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7760.069775] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7760.069777] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7760.069786] Call Trace: [ 7760.069792] [ 7760.069798] ? show_regs.cold.9+0x22/0x2f [ 7760.069807] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7760.069814] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7760.069818] ? watchdog+0x40/0x40 [ 7760.069822] ? hrtimer_interrupt+0x128/0x2c0 [ 7760.069825] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7760.069833] ? apic_timer_interrupt+0xf/0x20 [ 7760.069851] [ 7760.069855] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7760.069860] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7760.069865] do_raw_spin_lock+0x177/0x190 [ 7760.069870] _raw_spin_lock+0x21/0x30 [ 7760.069876] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7760.070160] obd_mod_free+0x46/0x140 [ptlrpc] [ 7760.070403] mdc_commit_open+0x91/0xb0 [mdc] [ 7760.070440] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7760.070737] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7760.071011] after_reply+0xe2c/0x1780 [ptlrpc] [ 7760.071354] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7760.080780] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [ 7760.106768] ? __next_timer_interrupt+0x160/0x160 [ 7760.106794] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7760.107093] ? do_wait_intr+0xf0/0xf0 [ 7760.107102] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7760.107294] kthread+0x1d1/0x200 [ 7760.107300] ? set_kthread_struct+0x70/0x70 [ 7760.107304] ret_from_fork+0x1f/0x30 [ 7760.327447] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7760.345097] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7760.349379] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7760.352249] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7760.353774] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7760.361329] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7760.367692] RAX: 0000000000003406 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7760.370128] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7760.374342] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7760.379582] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7760.383640] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7760.387480] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7760.391431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7760.394580] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7760.398179] Call Trace: [ 7760.399568] [ 7760.400968] ? show_regs.cold.9+0x22/0x2f [ 7760.402975] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7760.405033] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7760.406989] ? watchdog+0x40/0x40 [ 7760.408837] ? hrtimer_interrupt+0x128/0x2c0 [ 7760.410960] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7760.412885] ? apic_timer_interrupt+0xf/0x20 [ 7760.415157] [ 7760.416139] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7760.418570] do_raw_spin_lock+0x177/0x190 [ 7760.419961] _raw_spin_lock+0x21/0x30 [ 7760.421414] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7760.424096] ? do_raw_spin_unlock+0x75/0x190 [ 7760.425916] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7760.428478] ? do_wait_intr+0xf0/0xf0 [ 7760.430107] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7760.432647] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7760.434653] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7760.437323] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7760.440993] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7760.442443] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7760.446554] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7760.447791] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7760.450219] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7760.455088] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7760.458527] lmv_intent_open+0x338/0x1150 [lmv] [ 7760.461177] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7760.463391] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7760.466606] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7760.469381] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7760.471014] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7760.473436] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7760.476292] ? do_raw_spin_unlock+0x75/0x190 [ 7760.479509] ? _raw_spin_unlock+0x12/0x30 [ 7760.480699] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7760.481891] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7760.483407] ll_atomic_open+0x36a/0xd70 [lustre] [ 7760.484943] lookup_open+0x356/0xb40 [ 7760.486031] do_last+0x313/0xfc0 [ 7760.488332] ? nd_jump_root+0xe5/0x160 [ 7760.490004] ? path_init+0x437/0x520 [ 7760.491441] path_openat+0xf7/0x500 [ 7760.493422] ? do_raw_spin_unlock+0x75/0x190 [ 7760.496569] do_filp_open+0x99/0x140 [ 7760.498761] ? getname_flags+0x6e/0x330 [ 7760.500679] ? __check_object_size+0xff/0x256 [ 7760.502585] ? do_raw_spin_unlock+0x75/0x190 [ 7760.503918] ? _raw_spin_unlock+0x12/0x30 [ 7760.505511] do_sys_openat2+0x2b4/0x410 [ 7760.507064] do_sys_open+0x73/0xa0 [ 7760.508431] __x64_sys_openat+0x24/0x30 [ 7760.510004] do_syscall_64+0xc1/0x440 [ 7760.511383] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7760.513374] RIP: 0033:0x7f53425a5332 [ 7760.514747] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7760.521581] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7760.524348] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7760.527534] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7760.529201] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7760.531798] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7760.534154] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7776.058055] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 7776.065850] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7776.115937] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7776.123187] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7776.133429] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7776.140615] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7776.162561] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7776.173005] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7776.179967] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7776.184772] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7776.189495] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7776.192860] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7776.198884] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7776.203316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7776.208129] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7776.210995] Call Trace: [ 7776.214095] [ 7776.215685] ? show_regs.cold.9+0x22/0x2f [ 7776.217751] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7776.219248] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7776.222604] ? watchdog+0x40/0x40 [ 7776.223919] ? hrtimer_interrupt+0x128/0x2c0 [ 7776.226152] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7776.228203] ? apic_timer_interrupt+0xf/0x20 [ 7776.229835] [ 7776.230672] ? native_safe_halt+0x2e/0x50 [ 7776.232683] kvm_wait+0xcd/0xe0 [ 7776.233777] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7776.238938] do_raw_spin_lock+0x177/0x190 [ 7776.240729] _raw_spin_lock+0x21/0x30 [ 7776.244240] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7776.250046] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7776.255643] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7776.259276] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7776.264243] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7776.272049] ? woken_wake_function+0x30/0x30 [ 7776.277485] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7776.280044] kthread+0x1d1/0x200 [ 7776.281082] ? set_kthread_struct+0x70/0x70 [ 7776.284793] ret_from_fork+0x1f/0x30 [ 7784.055620] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 7784.067762] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7784.121482] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7784.129454] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7784.141082] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7784.144782] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7784.149470] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7784.171736] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7784.177467] RAX: 0000000000007e93 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7784.180356] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7784.188678] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7784.193906] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7784.198324] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7784.202729] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7784.208449] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7784.212006] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7784.219294] Call Trace: [ 7784.221101] [ 7784.223522] ? show_regs.cold.9+0x22/0x2f [ 7784.226653] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7784.230971] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7784.234377] ? watchdog+0x40/0x40 [ 7784.237695] ? hrtimer_interrupt+0x128/0x2c0 [ 7784.247909] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7784.251785] ? apic_timer_interrupt+0xf/0x20 [ 7784.255700] [ 7784.257696] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7784.262072] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7784.264807] ? newidle_balance.isra.93+0x207/0x580 [ 7784.266127] do_raw_spin_lock+0x177/0x190 [ 7784.269430] _raw_spin_lock+0x21/0x30 [ 7784.273599] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7784.280091] process_one_work+0x2c8/0x700 [ 7784.284620] worker_thread+0x296/0x6e0 [ 7784.286466] ? rescuer_thread+0x570/0x570 [ 7784.292849] kthread+0x1d1/0x200 [ 7784.301717] ? set_kthread_struct+0x70/0x70 [ 7784.303668] ret_from_fork+0x1f/0x30 [ 7788.055026] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 7788.059072] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 7788.060671] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7788.060744] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7788.061118] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7788.061121] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7788.061148] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7788.061153] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7788.061158] RAX: 0000000000007188 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7788.061160] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7788.061163] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7788.061165] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7788.061168] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7788.061170] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7788.061173] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7788.061175] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7788.061183] Call Trace: [ 7788.061188] [ 7788.061194] ? show_regs.cold.9+0x22/0x2f [ 7788.061306] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7788.061324] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7788.061329] ? watchdog+0x40/0x40 [ 7788.066415] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7788.066492] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7788.066499] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7788.066502] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7788.066517] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7788.066521] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7788.066525] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7788.066527] RDX: 0000000000006cbb RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7788.066529] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7788.066531] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7788.066533] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7788.066536] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7788.066539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7788.066541] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7788.066549] Call Trace: [ 7788.066555] [ 7788.066560] ? show_regs.cold.9+0x22/0x2f [ 7788.066569] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7788.066575] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7788.066579] ? watchdog+0x40/0x40 [ 7788.066584] ? hrtimer_interrupt+0x128/0x2c0 [ 7788.066588] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7788.066598] ? apic_timer_interrupt+0xf/0x20 [ 7788.066602] [ 7788.066605] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7788.066610] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7788.066615] do_raw_spin_lock+0x177/0x190 [ 7788.066620] _raw_spin_lock+0x21/0x30 [ 7788.066627] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7788.066907] obd_mod_free+0x46/0x140 [ptlrpc] [ 7788.103691] ? hrtimer_interrupt+0x128/0x2c0 [ 7788.103710] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7788.103719] ? apic_timer_interrupt+0xf/0x20 [ 7788.103724] [ 7788.103729] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7788.103740] do_raw_spin_lock+0x177/0x190 [ 7788.103746] _raw_spin_lock+0x21/0x30 [ 7788.103750] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7788.104033] ? do_raw_spin_unlock+0x75/0x190 [ 7788.104039] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7788.114454] mdc_commit_open+0x91/0xb0 [mdc] [ 7788.123017] ? do_wait_intr+0xf0/0xf0 [ 7788.123042] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7788.125217] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7788.135069] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7788.135729] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7788.135947] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7788.136068] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7788.136106] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7788.136150] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7788.136180] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7788.136264] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7788.136567] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7788.136598] lmv_intent_open+0x338/0x1150 [lmv] [ 7788.136629] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7788.136719] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7788.136809] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7788.136832] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7788.136920] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7788.137006] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7788.137095] ? do_raw_spin_unlock+0x75/0x190 [ 7788.137103] ? _raw_spin_unlock+0x12/0x30 [ 7788.137110] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7788.137153] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7788.137172] ll_atomic_open+0x36a/0xd70 [lustre] [ 7788.137265] lookup_open+0x356/0xb40 [ 7788.137275] do_last+0x313/0xfc0 [ 7788.137281] ? nd_jump_root+0xe5/0x160 [ 7788.137286] ? path_init+0x437/0x520 [ 7788.137291] path_openat+0xf7/0x500 [ 7788.137297] ? do_raw_spin_unlock+0x75/0x190 [ 7788.137302] do_filp_open+0x99/0x140 [ 7788.137308] ? getname_flags+0x6e/0x330 [ 7788.137313] ? __check_object_size+0xff/0x256 [ 7788.137321] ? do_raw_spin_unlock+0x75/0x190 [ 7788.137325] ? _raw_spin_unlock+0x12/0x30 [ 7788.137330] do_sys_openat2+0x2b4/0x410 [ 7788.137334] do_sys_open+0x73/0xa0 [ 7788.137339] __x64_sys_openat+0x24/0x30 [ 7788.140771] do_syscall_64+0xc1/0x440 [ 7788.140786] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7788.140794] RIP: 0033:0x7f53425a5332 [ 7788.140803] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7788.140808] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7788.140813] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7788.140816] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7788.140818] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7788.140820] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7788.140822] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7788.656653] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7788.663432] after_reply+0xe2c/0x1780 [ptlrpc] [ 7788.668979] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7788.673580] ? __next_timer_interrupt+0x160/0x160 [ 7788.678071] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7788.682089] ? do_wait_intr+0xf0/0xf0 [ 7788.683415] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7788.693115] kthread+0x1d1/0x200 [ 7788.699160] ? set_kthread_struct+0x70/0x70 [ 7788.701230] ret_from_fork+0x1f/0x30 [ 7788.714708] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5151s! [ 7788.719633] Showing busy workqueues and worker pools: [ 7788.723965] workqueue events: flags=0x0 [ 7788.726919] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7788.733492] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7788.742855] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7788.747805] pending: cpu_latency_work [ptlrpc] [ 7788.754425] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7788.756890] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7788.765591] workqueue events_long: flags=0x0 [ 7788.769617] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7788.774795] pending: lu_objects_flush [obdclass] [ 7788.779813] workqueue events_unbound: flags=0x2 [ 7788.784843] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7788.793867] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7788.799790] workqueue events_power_efficient: flags=0x80 [ 7788.804938] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7788.810558] pending: check_lifetime, do_cache_clean [ 7788.813467] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7788.819649] pending: neigh_periodic_work, neigh_periodic_work [ 7788.824779] workqueue mm_percpu_wq: flags=0x8 [ 7788.829904] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7788.833206] pending: vmstat_update, lru_add_drain_per_cpu [ 7788.836204] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7788.843247] pending: vmstat_update, lru_add_drain_per_cpu [ 7788.851442] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7788.853458] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7788.858652] workqueue xprtiod: flags=0xa [ 7788.863905] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7788.867024] pending: xprt_autoclose, xs_error_handle [ 7788.876849] workqueue ipv6_addrconf: flags=0x40008 [ 7788.881750] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7788.888533] pending: addrconf_verify_work [ 7788.893634] workqueue knbd0-recv: flags=0x1a [ 7788.895573] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7788.900271] in-flight: 651:recv_work [ 7788.901867] workqueue ptlrpc_pinger: flags=0x2 [ 7788.909616] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7788.914885] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7788.923034] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5151s workers=4 idle: 15979 31824 8 [ 7788.926463] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7779s workers=2 idle: 654 [ 7804.058029] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 7804.063185] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7804.084719] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7804.088716] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7804.092021] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7804.093747] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7804.100405] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7804.103308] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7804.106211] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7804.109034] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7804.111239] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7804.113812] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7804.116647] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7804.120301] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7804.122210] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7804.124429] Call Trace: [ 7804.125459] [ 7804.126358] ? show_regs.cold.9+0x22/0x2f [ 7804.127970] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7804.129851] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7804.131666] ? watchdog+0x40/0x40 [ 7804.133023] ? hrtimer_interrupt+0x128/0x2c0 [ 7804.134797] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7804.136298] ? apic_timer_interrupt+0xf/0x20 [ 7804.137602] [ 7804.138490] ? native_safe_halt+0x2e/0x50 [ 7804.140152] kvm_wait+0xcd/0xe0 [ 7804.141453] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7804.143395] do_raw_spin_lock+0x177/0x190 [ 7804.145068] _raw_spin_lock+0x21/0x30 [ 7804.146484] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7804.148651] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7804.151009] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7804.153245] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7804.155983] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7804.158428] ? woken_wake_function+0x30/0x30 [ 7804.160035] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7804.162035] kthread+0x1d1/0x200 [ 7804.162796] ? set_kthread_struct+0x70/0x70 [ 7804.163829] ret_from_fork+0x1f/0x30 [ 7816.053727] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 7816.055020] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 7816.057350] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 7816.057362] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7816.057768] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7816.057775] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7816.057778] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7816.062739] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7816.062753] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7816.062765] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7816.062768] RDX: 000000000000778f RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7816.062770] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7816.062773] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7816.062775] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7816.062779] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7816.062782] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7816.062784] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7816.062800] Call Trace: [ 7816.062806] [ 7816.062812] ? show_regs.cold.9+0x22/0x2f [ 7816.062824] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7816.062831] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7816.062836] ? watchdog+0x40/0x40 [ 7816.062841] ? hrtimer_interrupt+0x128/0x2c0 [ 7816.062846] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7816.062854] ? apic_timer_interrupt+0xf/0x20 [ 7816.062859] [ 7816.062862] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7816.062868] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7816.062874] do_raw_spin_lock+0x177/0x190 [ 7816.062879] _raw_spin_lock+0x21/0x30 [ 7816.062886] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7816.063402] Modules linked in: lustre(O) osp(O) [ 7816.065566] obd_mod_free+0x46/0x140 [ptlrpc] [ 7816.065839] mdc_commit_open+0x91/0xb0 [mdc] [ 7816.065879] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7816.066075] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7816.066277] after_reply+0xe2c/0x1780 [ptlrpc] [ 7816.066741] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7816.066948] ? __next_timer_interrupt+0x160/0x160 [ 7816.066958] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7816.067167] ? do_wait_intr+0xf0/0xf0 [ 7816.067176] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7816.067761] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7816.067845] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7816.067852] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7816.067854] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7816.067986] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7816.067991] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7816.067996] RAX: 0000000000004c07 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7816.067999] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7816.068001] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7816.068004] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7816.068006] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7816.068010] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7816.068013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7816.068016] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7816.068025] Call Trace: [ 7816.068030] [ 7816.068033] ? show_regs.cold.9+0x22/0x2f [ 7816.068042] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7816.068048] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7816.068052] ? watchdog+0x40/0x40 [ 7816.068055] ? hrtimer_interrupt+0x128/0x2c0 [ 7816.068059] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7816.068067] ? apic_timer_interrupt+0xf/0x20 [ 7816.068071] [ 7816.068074] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7816.068080] do_raw_spin_lock+0x177/0x190 [ 7816.068085] _raw_spin_lock+0x21/0x30 [ 7816.068161] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7816.068768] kthread+0x1d1/0x200 [ 7816.068778] ? set_kthread_struct+0x70/0x70 [ 7816.068782] ret_from_fork+0x1f/0x30 [ 7816.078235] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7816.078310] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7816.078317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7816.078324] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7816.126532] ? do_raw_spin_unlock+0x75/0x190 [ 7816.126553] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7816.134148] [ 7816.134157] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7816.134175] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7816.134179] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7816.134184] RAX: 0000000000001da3 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7816.134186] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7816.134189] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7816.134191] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7816.134194] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7816.134197] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7816.134200] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7816.134202] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7816.134211] Call Trace: [ 7816.134216] [ 7816.134222] ? show_regs.cold.9+0x22/0x2f [ 7816.134231] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7816.134236] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7816.134241] ? watchdog+0x40/0x40 [ 7816.134245] ? hrtimer_interrupt+0x128/0x2c0 [ 7816.134248] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7816.134256] ? apic_timer_interrupt+0xf/0x20 [ 7816.134261] [ 7816.134264] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7816.134269] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7816.134273] ? newidle_balance.isra.93+0x207/0x580 [ 7816.134280] do_raw_spin_lock+0x177/0x190 [ 7816.134285] _raw_spin_lock+0x21/0x30 [ 7816.134289] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7816.138468] ? do_wait_intr+0xf0/0xf0 [ 7816.138487] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7816.147030] process_one_work+0x2c8/0x700 [ 7816.147050] worker_thread+0x296/0x6e0 [ 7816.147056] ? rescuer_thread+0x570/0x570 [ 7816.147061] kthread+0x1d1/0x200 [ 7816.147065] ? set_kthread_struct+0x70/0x70 [ 7816.147069] ret_from_fork+0x1f/0x30 [ 7816.641740] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7816.648724] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7816.654249] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7816.657622] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7816.663569] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7816.668865] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7816.677218] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7816.682407] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7816.685719] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7816.692124] lmv_intent_open+0x338/0x1150 [lmv] [ 7816.694270] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7816.696065] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7816.702053] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7816.704404] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7816.706469] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7816.709345] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7816.711471] ? do_raw_spin_unlock+0x75/0x190 [ 7816.713363] ? _raw_spin_unlock+0x12/0x30 [ 7816.716186] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7816.718807] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7816.721244] ll_atomic_open+0x36a/0xd70 [lustre] [ 7816.723559] lookup_open+0x356/0xb40 [ 7816.725215] do_last+0x313/0xfc0 [ 7816.726784] ? nd_jump_root+0xe5/0x160 [ 7816.728353] ? path_init+0x437/0x520 [ 7816.729969] path_openat+0xf7/0x500 [ 7816.731465] ? do_raw_spin_unlock+0x75/0x190 [ 7816.732891] do_filp_open+0x99/0x140 [ 7816.735572] ? getname_flags+0x6e/0x330 [ 7816.741348] ? __check_object_size+0xff/0x256 [ 7816.742897] ? do_raw_spin_unlock+0x75/0x190 [ 7816.748316] ? _raw_spin_unlock+0x12/0x30 [ 7816.750387] do_sys_openat2+0x2b4/0x410 [ 7816.752172] do_sys_open+0x73/0xa0 [ 7816.757356] __x64_sys_openat+0x24/0x30 [ 7816.760112] do_syscall_64+0xc1/0x440 [ 7816.762187] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7816.767752] RIP: 0033:0x7f53425a5332 [ 7816.770052] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7816.791258] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7816.804815] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7816.811158] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7816.816847] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7816.819275] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7816.827469] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7819.233029] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5182s! [ 7819.237048] Showing busy workqueues and worker pools: [ 7819.239446] workqueue events: flags=0x0 [ 7819.241299] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7819.244841] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7819.251200] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7819.254937] pending: cpu_latency_work [ptlrpc] [ 7819.257775] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7819.261099] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7819.264201] workqueue events_long: flags=0x0 [ 7819.265450] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7819.269132] pending: lu_objects_flush [obdclass] [ 7819.271534] workqueue events_unbound: flags=0x2 [ 7819.273567] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7819.275506] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7819.278463] workqueue events_power_efficient: flags=0x80 [ 7819.281135] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7819.284007] pending: check_lifetime, do_cache_clean [ 7819.286075] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7819.289383] pending: neigh_periodic_work, neigh_periodic_work [ 7819.292088] workqueue mm_percpu_wq: flags=0x8 [ 7819.294665] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7819.298163] pending: vmstat_update, lru_add_drain_per_cpu [ 7819.300448] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7819.303480] pending: vmstat_update, lru_add_drain_per_cpu [ 7819.306383] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7819.309222] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7819.312865] workqueue xprtiod: flags=0xa [ 7819.314544] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7819.317177] pending: xprt_autoclose, xs_error_handle [ 7819.320465] workqueue ipv6_addrconf: flags=0x40008 [ 7819.323299] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7819.326472] pending: addrconf_verify_work [ 7819.328195] workqueue knbd0-recv: flags=0x1a [ 7819.330136] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7819.333032] in-flight: 651:recv_work [ 7819.334680] workqueue ptlrpc_pinger: flags=0x2 [ 7819.336472] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7819.339431] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7819.342210] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5182s workers=4 idle: 15979 31824 8 [ 7819.345677] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7809s workers=2 idle: 654 [ 7832.058172] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 7832.062528] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7832.107212] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7832.122970] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7832.145560] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7832.159408] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7832.174697] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7832.178847] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7832.184561] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7832.189229] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7832.193100] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7832.196303] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7832.200100] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7832.208475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7832.210471] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7832.216228] Call Trace: [ 7832.217724] [ 7832.222327] ? show_regs.cold.9+0x22/0x2f [ 7832.226430] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7832.230910] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7832.232256] ? watchdog+0x40/0x40 [ 7832.235589] ? hrtimer_interrupt+0x128/0x2c0 [ 7832.239030] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7832.243516] ? apic_timer_interrupt+0xf/0x20 [ 7832.247498] [ 7832.249940] ? native_safe_halt+0x2e/0x50 [ 7832.254074] kvm_wait+0xcd/0xe0 [ 7832.256858] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7832.260935] do_raw_spin_lock+0x177/0x190 [ 7832.266155] _raw_spin_lock+0x21/0x30 [ 7832.274293] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7832.283330] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7832.291801] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7832.304172] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7832.309757] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7832.316090] ? woken_wake_function+0x30/0x30 [ 7832.324591] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7832.333504] kthread+0x1d1/0x200 [ 7832.336513] ? set_kthread_struct+0x70/0x70 [ 7832.343584] ret_from_fork+0x1f/0x30 [ 7844.055081] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 7844.057640] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 7844.062027] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7844.063000] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7844.063008] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7844.063014] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 7844.063012] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7844.063022] Modules linked in: lustre(O) osp(O) [ 7844.063030] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7844.063033] ofd(O) lod(O) mdt(O) [ 7844.063040] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [ 7844.063043] mdd(O) [ 7844.063048] ORIG_RAX: ffffffffffffff13 [ 7844.063049] mgs(O) lquota(O) [ 7844.063054] RAX: 0000000000007e88 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7844.063058] lfsck(O) [ 7844.063061] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7844.063063] obdecho(O) mgc(O) [ 7844.063069] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7844.063070] mdc(O) [ 7844.063077] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7844.063078] lov(O) osc(O) [ 7844.063084] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7844.063086] lmv(O) ec(O) [ 7844.063091] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7844.063093] fid(O) fld(O) [ 7844.063098] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7844.063102] ptlrpc_gss(O) [ 7844.063106] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7844.063107] ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7844.063146] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7844.063151] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7844.063155] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7844.063448] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7844.063455] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7844.063458] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7844.063461] RAX: 00000000000017e5 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7844.063463] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7844.063465] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7844.063467] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7844.063469] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7844.063472] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7844.063475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7844.063477] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7844.063486] Call Trace: [ 7844.063492] [ 7844.063498] ? show_regs.cold.9+0x22/0x2f [ 7844.063509] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7844.063515] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7844.063519] ? watchdog+0x40/0x40 [ 7844.063528] ? hrtimer_interrupt+0x128/0x2c0 [ 7844.063532] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7844.063539] ? apic_timer_interrupt+0xf/0x20 [ 7844.063544] [ 7844.063547] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7844.063551] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7844.063555] ? newidle_balance.isra.93+0x207/0x580 [ 7844.063560] do_raw_spin_lock+0x177/0x190 [ 7844.063564] _raw_spin_lock+0x21/0x30 [ 7844.063569] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7844.063843] Call Trace: [ 7844.063847] [ 7844.063850] ? show_regs.cold.9+0x22/0x2f [ 7844.063857] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7844.063862] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7844.063866] ? watchdog+0x40/0x40 [ 7844.063870] ? hrtimer_interrupt+0x128/0x2c0 [ 7844.063875] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7844.063881] ? apic_timer_interrupt+0xf/0x20 [ 7844.063886] [ 7844.063889] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7844.063895] do_raw_spin_lock+0x177/0x190 [ 7844.063900] _raw_spin_lock+0x21/0x30 [ 7844.063904] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7844.065285] ? do_raw_spin_unlock+0x75/0x190 [ 7844.065300] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7844.066007] ? do_wait_intr+0xf0/0xf0 [ 7844.066017] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7844.066983] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7844.068145] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7844.068419] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7844.068565] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7844.068609] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7844.068637] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7844.068661] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7844.068742] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7844.068959] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7844.068988] lmv_intent_open+0x338/0x1150 [lmv] [ 7844.069025] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7844.069111] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7844.069205] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7844.069225] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7844.069308] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7844.069389] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7844.069471] ? do_raw_spin_unlock+0x75/0x190 [ 7844.069479] ? _raw_spin_unlock+0x12/0x30 [ 7844.069487] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7844.069514] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7844.069534] ll_atomic_open+0x36a/0xd70 [lustre] [ 7844.069621] lookup_open+0x356/0xb40 [ 7844.069631] do_last+0x313/0xfc0 [ 7844.069636] ? nd_jump_root+0xe5/0x160 [ 7844.069641] ? path_init+0x437/0x520 [ 7844.069646] path_openat+0xf7/0x500 [ 7844.069650] ? do_raw_spin_unlock+0x75/0x190 [ 7844.069655] do_filp_open+0x99/0x140 [ 7844.069662] ? getname_flags+0x6e/0x330 [ 7844.069668] ? __check_object_size+0xff/0x256 [ 7844.069675] ? do_raw_spin_unlock+0x75/0x190 [ 7844.069679] ? _raw_spin_unlock+0x12/0x30 [ 7844.069683] do_sys_openat2+0x2b4/0x410 [ 7844.069687] do_sys_open+0x73/0xa0 [ 7844.069691] __x64_sys_openat+0x24/0x30 [ 7844.069694] do_syscall_64+0xc1/0x440 [ 7844.069702] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7844.069708] RIP: 0033:0x7f53425a5332 [ 7844.069760] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7844.069766] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7844.069772] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7844.069774] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7844.069777] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7844.069779] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7844.069781] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7844.070026] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7844.070096] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7844.070103] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7844.070105] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7844.070113] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7844.070118] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7844.070122] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7844.070125] RDX: 0000000000007242 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7844.070127] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7844.070130] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7844.070132] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7844.070135] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7844.070138] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7844.070140] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7844.070150] Call Trace: [ 7844.070155] [ 7844.070168] ? show_regs.cold.9+0x22/0x2f [ 7844.070179] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7844.070186] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7844.070190] ? watchdog+0x40/0x40 [ 7844.070195] ? hrtimer_interrupt+0x128/0x2c0 [ 7844.070199] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7844.070205] ? apic_timer_interrupt+0xf/0x20 [ 7844.070210] [ 7844.070213] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7844.070218] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7844.070223] do_raw_spin_lock+0x177/0x190 [ 7844.070228] _raw_spin_lock+0x21/0x30 [ 7844.070233] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7844.070468] obd_mod_free+0x46/0x140 [ptlrpc] [ 7844.070737] mdc_commit_open+0x91/0xb0 [mdc] [ 7844.070774] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7844.070973] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7844.071201] after_reply+0xe2c/0x1780 [ptlrpc] [ 7844.071413] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7844.072778] ? __next_timer_interrupt+0x160/0x160 [ 7844.072788] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7844.072992] ? do_wait_intr+0xf0/0xf0 [ 7844.073001] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7844.121685] process_one_work+0x2c8/0x700 [ 7844.121709] worker_thread+0x296/0x6e0 [ 7844.121714] ? rescuer_thread+0x570/0x570 [ 7844.121719] kthread+0x1d1/0x200 [ 7844.121724] ? set_kthread_struct+0x70/0x70 [ 7844.121728] ret_from_fork+0x1f/0x30 [ 7844.952226] kthread+0x1d1/0x200 [ 7844.955961] ? set_kthread_struct+0x70/0x70 [ 7844.958980] ret_from_fork+0x1f/0x30 [ 7849.443589] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5212s! [ 7849.457217] Showing busy workqueues and worker pools: [ 7849.460023] workqueue events: flags=0x0 [ 7849.463365] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7849.467088] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7849.474582] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7849.479313] pending: cpu_latency_work [ptlrpc] [ 7849.482023] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7849.485012] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7849.493888] workqueue events_long: flags=0x0 [ 7849.497536] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7849.500704] pending: lu_objects_flush [obdclass] [ 7849.508806] workqueue events_unbound: flags=0x2 [ 7849.511365] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7849.514946] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7849.518762] workqueue events_power_efficient: flags=0x80 [ 7849.526803] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7849.532992] pending: check_lifetime, do_cache_clean [ 7849.535826] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7849.538864] pending: neigh_periodic_work, neigh_periodic_work [ 7849.541766] workqueue mm_percpu_wq: flags=0x8 [ 7849.543365] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7849.546195] pending: vmstat_update, lru_add_drain_per_cpu [ 7849.548529] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7849.555777] pending: vmstat_update, lru_add_drain_per_cpu [ 7849.560386] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7849.562985] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7849.565491] workqueue xprtiod: flags=0xa [ 7849.566708] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7849.575357] pending: xprt_autoclose, xs_error_handle [ 7849.580655] workqueue ipv6_addrconf: flags=0x40008 [ 7849.584224] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7849.591675] pending: addrconf_verify_work [ 7849.593095] workqueue knbd0-recv: flags=0x1a [ 7849.594575] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7849.597648] in-flight: 651:recv_work [ 7849.603669] workqueue ptlrpc_pinger: flags=0x2 [ 7849.605771] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7849.610413] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7849.612818] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5212s workers=4 idle: 15979 31824 8 [ 7849.617583] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7840s workers=2 idle: 654 [ 7860.059405] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 7860.063594] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7860.086824] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7860.093613] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7860.097933] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7860.101212] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7860.110368] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7860.114738] RAX: 0000000000006609 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7860.121589] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7860.125209] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7860.133099] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7860.138733] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7860.146050] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7860.155558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7860.162361] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7860.165384] Call Trace: [ 7860.166334] [ 7860.169123] ? show_regs.cold.9+0x22/0x2f [ 7860.173988] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7860.179235] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7860.181728] ? watchdog+0x40/0x40 [ 7860.183639] ? hrtimer_interrupt+0x128/0x2c0 [ 7860.185068] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7860.187162] ? apic_timer_interrupt+0xf/0x20 [ 7860.189167] [ 7860.191218] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7860.193845] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7860.196317] do_raw_spin_lock+0x177/0x190 [ 7860.199030] _raw_spin_lock+0x21/0x30 [ 7860.200541] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7860.207690] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7860.211183] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7860.214209] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7860.216389] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7860.218803] ? woken_wake_function+0x30/0x30 [ 7860.220679] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7860.223225] kthread+0x1d1/0x200 [ 7860.224619] ? set_kthread_struct+0x70/0x70 [ 7860.226623] ret_from_fork+0x1f/0x30 [ 7872.055083] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7872.057053] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7872.061936] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7872.062213] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7872.062221] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7872.062225] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7872.062238] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7872.062242] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7872.062248] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7872.062250] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7872.062252] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7872.062255] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7872.062257] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7872.062260] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7872.062263] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7872.062265] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7872.062275] Call Trace: [ 7872.062281] [ 7872.062286] ? show_regs.cold.9+0x22/0x2f [ 7872.062296] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7872.062303] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7872.062307] ? watchdog+0x40/0x40 [ 7872.062313] ? hrtimer_interrupt+0x128/0x2c0 [ 7872.067615] Modules linked in: lustre(O) [ 7872.108607] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7872.117391] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7872.117464] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7872.117470] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7872.117473] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7872.117486] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7872.117490] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7872.117495] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7872.117497] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7872.117499] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7872.117501] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7872.117503] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7872.117506] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7872.117509] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7872.117511] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7872.117520] Call Trace: [ 7872.117525] [ 7872.117531] ? show_regs.cold.9+0x22/0x2f [ 7872.117540] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7872.117546] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7872.117550] ? watchdog+0x40/0x40 [ 7872.117554] ? hrtimer_interrupt+0x128/0x2c0 [ 7872.117558] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7872.117564] ? apic_timer_interrupt+0xf/0x20 [ 7872.117569] [ 7872.117572] ? native_safe_halt+0x2e/0x50 [ 7872.117575] kvm_wait+0xcd/0xe0 [ 7872.117582] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7872.117589] do_raw_spin_lock+0x177/0x190 [ 7872.117594] _raw_spin_lock+0x21/0x30 [ 7872.117599] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7872.117881] obd_mod_free+0x46/0x140 [ptlrpc] [ 7872.118390] mdc_commit_open+0x91/0xb0 [mdc] [ 7872.118430] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7872.118693] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7872.118888] after_reply+0xe2c/0x1780 [ptlrpc] [ 7872.119081] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7872.119273] ? __next_timer_interrupt+0x160/0x160 [ 7872.119282] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7872.119479] ? do_wait_intr+0xf0/0xf0 [ 7872.119489] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7872.130274] ? apic_timer_interrupt+0xf/0x20 [ 7872.132528] kthread+0x1d1/0x200 [ 7872.144177] [ 7872.146388] ? set_kthread_struct+0x70/0x70 [ 7872.153918] ? native_safe_halt+0x2e/0x50 [ 7872.153938] kvm_wait+0xcd/0xe0 [ 7872.153947] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7872.153957] do_raw_spin_lock+0x177/0x190 [ 7872.153961] _raw_spin_lock+0x21/0x30 [ 7872.153966] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7872.162534] ret_from_fork+0x1f/0x30 [ 7872.360152] ? do_raw_spin_unlock+0x75/0x190 [ 7872.362500] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7872.365205] ? do_wait_intr+0xf0/0xf0 [ 7872.366640] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7872.369292] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7872.372643] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7872.377066] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7872.379067] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7872.383117] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7872.386069] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7872.388449] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7872.390606] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7872.393551] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7872.396950] lmv_intent_open+0x338/0x1150 [lmv] [ 7872.401113] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7872.403383] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7872.405304] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7872.407645] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7872.409579] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7872.412304] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7872.415094] ? do_raw_spin_unlock+0x75/0x190 [ 7872.416813] ? _raw_spin_unlock+0x12/0x30 [ 7872.418146] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7872.421609] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7872.425021] ll_atomic_open+0x36a/0xd70 [lustre] [ 7872.428090] lookup_open+0x356/0xb40 [ 7872.429582] do_last+0x313/0xfc0 [ 7872.430905] ? nd_jump_root+0xe5/0x160 [ 7872.432328] ? path_init+0x437/0x520 [ 7872.434571] path_openat+0xf7/0x500 [ 7872.436554] ? do_raw_spin_unlock+0x75/0x190 [ 7872.440089] do_filp_open+0x99/0x140 [ 7872.442285] ? getname_flags+0x6e/0x330 [ 7872.443498] ? __check_object_size+0xff/0x256 [ 7872.445081] ? do_raw_spin_unlock+0x75/0x190 [ 7872.446799] ? _raw_spin_unlock+0x12/0x30 [ 7872.448991] do_sys_openat2+0x2b4/0x410 [ 7872.451439] do_sys_open+0x73/0xa0 [ 7872.453448] __x64_sys_openat+0x24/0x30 [ 7872.456280] do_syscall_64+0xc1/0x440 [ 7872.457802] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7872.461105] RIP: 0033:0x7f53425a5332 [ 7872.462622] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7872.472997] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7872.476963] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7872.480578] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7872.483547] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7872.487152] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7872.491283] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7876.052024] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 7876.055186] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7876.092727] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7876.101092] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7876.106301] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7876.111138] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7876.114189] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7876.127481] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7876.132111] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7876.136147] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7876.139772] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7876.142553] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7876.146064] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7876.149039] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7876.152853] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7876.155228] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7876.159781] Call Trace: [ 7876.161070] [ 7876.162101] ? show_regs.cold.9+0x22/0x2f [ 7876.164022] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7876.166311] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7876.170088] ? watchdog+0x40/0x40 [ 7876.172965] ? hrtimer_interrupt+0x128/0x2c0 [ 7876.174485] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7876.179302] ? apic_timer_interrupt+0xf/0x20 [ 7876.182242] [ 7876.184189] ? native_safe_halt+0x2e/0x50 [ 7876.188996] kvm_wait+0xcd/0xe0 [ 7876.191096] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7876.194243] ? newidle_balance.isra.93+0x207/0x580 [ 7876.197119] do_raw_spin_lock+0x177/0x190 [ 7876.199253] _raw_spin_lock+0x21/0x30 [ 7876.205090] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7876.215419] process_one_work+0x2c8/0x700 [ 7876.223589] worker_thread+0x296/0x6e0 [ 7876.227304] ? rescuer_thread+0x570/0x570 [ 7876.230110] kthread+0x1d1/0x200 [ 7876.231447] ? set_kthread_struct+0x70/0x70 [ 7876.234456] ret_from_fork+0x1f/0x30 [ 7879.648091] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5242s! [ 7879.651379] Showing busy workqueues and worker pools: [ 7879.654290] workqueue events: flags=0x0 [ 7879.657268] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7879.661204] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7879.668170] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7879.673164] pending: cpu_latency_work [ptlrpc] [ 7879.676988] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7879.683990] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7879.689662] workqueue events_long: flags=0x0 [ 7879.691923] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7879.695554] pending: lu_objects_flush [obdclass] [ 7879.698937] workqueue events_unbound: flags=0x2 [ 7879.702329] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7879.706648] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7879.709992] workqueue events_power_efficient: flags=0x80 [ 7879.712708] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7879.716421] pending: check_lifetime, do_cache_clean [ 7879.719027] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7879.723234] pending: neigh_periodic_work, neigh_periodic_work [ 7879.726794] workqueue mm_percpu_wq: flags=0x8 [ 7879.729629] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7879.733063] pending: vmstat_update, lru_add_drain_per_cpu [ 7879.736748] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7879.740171] pending: vmstat_update, lru_add_drain_per_cpu [ 7879.744565] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7879.748462] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7879.751445] workqueue xprtiod: flags=0xa [ 7879.753778] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7879.757034] pending: xprt_autoclose, xs_error_handle [ 7879.760010] workqueue ipv6_addrconf: flags=0x40008 [ 7879.762411] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7879.765143] pending: addrconf_verify_work [ 7879.767179] workqueue knbd0-recv: flags=0x1a [ 7879.769144] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7879.771900] in-flight: 651:recv_work [ 7879.773824] workqueue ptlrpc_pinger: flags=0x2 [ 7879.775852] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7879.778695] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7879.782174] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5242s workers=4 idle: 15979 31824 8 [ 7879.785749] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7870s workers=2 idle: 654 [ 7888.058072] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 7888.066786] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7888.109165] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7888.117684] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7888.122729] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7888.127767] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7888.141967] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7888.151824] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7888.154376] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7888.156626] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7888.159535] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7888.165976] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7888.169296] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7888.171455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7888.173201] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7888.175671] Call Trace: [ 7888.176618] [ 7888.177126] ? show_regs.cold.9+0x22/0x2f [ 7888.178624] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7888.179993] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7888.181473] ? watchdog+0x40/0x40 [ 7888.182451] ? hrtimer_interrupt+0x128/0x2c0 [ 7888.183695] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7888.185166] ? apic_timer_interrupt+0xf/0x20 [ 7888.186546] [ 7888.187087] ? native_safe_halt+0x2e/0x50 [ 7888.188908] kvm_wait+0xcd/0xe0 [ 7888.190148] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7888.191842] do_raw_spin_lock+0x177/0x190 [ 7888.193140] _raw_spin_lock+0x21/0x30 [ 7888.194329] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7888.196719] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7888.198693] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7888.200631] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7888.202798] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7888.204786] ? woken_wake_function+0x30/0x30 [ 7888.206165] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7888.208602] kthread+0x1d1/0x200 [ 7888.209976] ? set_kthread_struct+0x70/0x70 [ 7888.211540] ret_from_fork+0x1f/0x30 [ 7900.055535] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7900.058761] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7900.063328] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7900.063780] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7900.063787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7900.063790] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7900.063803] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7900.063808] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7900.063812] RAX: 0000000000005884 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7900.063815] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7900.063817] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7900.063819] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7900.063822] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7900.063826] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7900.063829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7900.063831] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7900.063840] Call Trace: [ 7900.063845] [ 7900.063850] ? show_regs.cold.9+0x22/0x2f [ 7900.063859] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7900.063866] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7900.063870] ? watchdog+0x40/0x40 [ 7900.063874] ? hrtimer_interrupt+0x128/0x2c0 [ 7900.063878] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7900.063884] ? apic_timer_interrupt+0xf/0x20 [ 7900.063889] [ 7900.063891] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7900.063899] do_raw_spin_lock+0x177/0x190 [ 7900.063903] _raw_spin_lock+0x21/0x30 [ 7900.063908] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7900.070815] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7900.070899] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7900.070907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7900.070910] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7900.070924] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7900.070929] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7900.070934] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7900.070936] RDX: 00000000000017a1 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7900.070938] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7900.070941] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7900.070944] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7900.070947] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7900.070950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7900.070952] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7900.070961] Call Trace: [ 7900.070967] [ 7900.070972] ? show_regs.cold.9+0x22/0x2f [ 7900.070984] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7900.070990] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7900.070995] ? watchdog+0x40/0x40 [ 7900.070999] ? hrtimer_interrupt+0x128/0x2c0 [ 7900.071003] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7900.071011] ? apic_timer_interrupt+0xf/0x20 [ 7900.071016] [ 7900.071019] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7900.071024] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7900.071029] do_raw_spin_lock+0x177/0x190 [ 7900.071033] _raw_spin_lock+0x21/0x30 [ 7900.071039] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7900.071333] obd_mod_free+0x46/0x140 [ptlrpc] [ 7900.071522] mdc_commit_open+0x91/0xb0 [mdc] [ 7900.071560] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7900.071755] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7900.071951] after_reply+0xe2c/0x1780 [ptlrpc] [ 7900.072149] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7900.072356] ? __next_timer_interrupt+0x160/0x160 [ 7900.072365] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7900.072580] ? do_wait_intr+0xf0/0xf0 [ 7900.072589] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7900.072809] kthread+0x1d1/0x200 [ 7900.072815] ? set_kthread_struct+0x70/0x70 [ 7900.072818] ret_from_fork+0x1f/0x30 [ 7900.485816] ? do_raw_spin_unlock+0x75/0x190 [ 7900.491485] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7900.493698] ? do_wait_intr+0xf0/0xf0 [ 7900.495733] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7900.498280] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7900.500660] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7900.503304] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7900.505547] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7900.508385] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7900.510314] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7900.512218] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7900.514522] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7900.519470] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7900.523660] lmv_intent_open+0x338/0x1150 [lmv] [ 7900.527381] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7900.534545] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7900.536732] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7900.541013] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7900.548048] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7900.551464] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7900.554225] ? do_raw_spin_unlock+0x75/0x190 [ 7900.555890] ? _raw_spin_unlock+0x12/0x30 [ 7900.558040] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7900.560220] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7900.566924] ll_atomic_open+0x36a/0xd70 [lustre] [ 7900.569209] lookup_open+0x356/0xb40 [ 7900.572929] do_last+0x313/0xfc0 [ 7900.574834] ? nd_jump_root+0xe5/0x160 [ 7900.576822] ? path_init+0x437/0x520 [ 7900.579988] path_openat+0xf7/0x500 [ 7900.581592] ? do_raw_spin_unlock+0x75/0x190 [ 7900.585371] do_filp_open+0x99/0x140 [ 7900.587037] ? getname_flags+0x6e/0x330 [ 7900.588775] ? __check_object_size+0xff/0x256 [ 7900.591700] ? do_raw_spin_unlock+0x75/0x190 [ 7900.593748] ? _raw_spin_unlock+0x12/0x30 [ 7900.595675] do_sys_openat2+0x2b4/0x410 [ 7900.598962] do_sys_open+0x73/0xa0 [ 7900.600516] __x64_sys_openat+0x24/0x30 [ 7900.602404] do_syscall_64+0xc1/0x440 [ 7900.603886] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7900.606354] RIP: 0033:0x7f53425a5332 [ 7900.608956] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7900.619689] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7900.623482] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7900.626645] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7900.629734] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7900.633000] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7900.636688] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7904.052579] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 7904.059516] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7904.090772] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7904.104662] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7904.114903] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7904.122917] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7904.127173] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7904.146656] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7904.151360] RAX: 0000000000007f0b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7904.156957] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7904.164671] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7904.174001] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7904.179159] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7904.187100] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7904.193437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7904.200428] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7904.211325] Call Trace: [ 7904.214398] [ 7904.215192] ? show_regs.cold.9+0x22/0x2f [ 7904.216636] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7904.222629] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7904.226495] ? watchdog+0x40/0x40 [ 7904.229548] ? hrtimer_interrupt+0x128/0x2c0 [ 7904.231350] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7904.234365] ? apic_timer_interrupt+0xf/0x20 [ 7904.238230] [ 7904.239219] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7904.241076] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7904.243010] ? newidle_balance.isra.93+0x207/0x580 [ 7904.245364] do_raw_spin_lock+0x177/0x190 [ 7904.249706] _raw_spin_lock+0x21/0x30 [ 7904.251700] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7904.257497] process_one_work+0x2c8/0x700 [ 7904.259660] worker_thread+0x296/0x6e0 [ 7904.261403] ? rescuer_thread+0x570/0x570 [ 7904.263390] kthread+0x1d1/0x200 [ 7904.264890] ? set_kthread_struct+0x70/0x70 [ 7904.267765] ret_from_fork+0x1f/0x30 [ 7909.859644] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5272s! [ 7909.867189] Showing busy workqueues and worker pools: [ 7909.869644] workqueue events: flags=0x0 [ 7909.872546] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7909.878528] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7909.892693] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7909.894952] pending: cpu_latency_work [ptlrpc] [ 7909.897947] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7909.903362] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7909.908633] workqueue events_long: flags=0x0 [ 7909.914320] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7909.922556] pending: lu_objects_flush [obdclass] [ 7909.927537] workqueue events_unbound: flags=0x2 [ 7909.930496] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7909.937824] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7909.945500] workqueue events_power_efficient: flags=0x80 [ 7909.953893] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7909.961559] pending: check_lifetime, do_cache_clean [ 7909.967400] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7909.975143] pending: neigh_periodic_work, neigh_periodic_work [ 7909.979787] workqueue mm_percpu_wq: flags=0x8 [ 7909.983907] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7909.987771] pending: vmstat_update, lru_add_drain_per_cpu [ 7909.995974] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7910.000458] pending: vmstat_update, lru_add_drain_per_cpu [ 7910.005118] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7910.009669] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7910.014071] workqueue xprtiod: flags=0xa [ 7910.017208] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7910.021082] pending: xprt_autoclose, xs_error_handle [ 7910.024342] workqueue ipv6_addrconf: flags=0x40008 [ 7910.026812] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7910.030648] pending: addrconf_verify_work [ 7910.034563] workqueue knbd0-recv: flags=0x1a [ 7910.041667] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7910.048395] in-flight: 651:recv_work [ 7910.059449] workqueue ptlrpc_pinger: flags=0x2 [ 7910.066250] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7910.070128] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7910.072423] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5272s workers=4 idle: 15979 31824 8 [ 7910.075478] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7900s workers=2 idle: 654 [ 7914.580773] rcu: INFO: rcu_sched self-detected stall on CPU [ 7914.583993] rcu: 2-....: (3147371 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 7914.590807] (t=5820154 jiffies g=274185 q=65285) [ 7914.593724] rcu: rcu_sched kthread starved for 5278520 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 7914.599609] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 7914.603738] rcu: RCU grace-period kthread stack dump: [ 7914.607237] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 7914.613087] Call Trace: [ 7914.614412] __schedule+0x351/0xcb0 [ 7914.616428] schedule+0xc0/0x180 [ 7914.618039] schedule_timeout+0xb4/0x190 [ 7914.620300] ? do_raw_spin_unlock+0x75/0x190 [ 7914.622342] ? __next_timer_interrupt+0x160/0x160 [ 7914.624445] ? prepare_to_swait_event+0xf3/0x140 [ 7914.626906] rcu_gp_kthread+0x8dd/0x15b0 [ 7914.630304] ? force_qs_rnp+0x2c0/0x2c0 [ 7914.632279] kthread+0x1d1/0x200 [ 7914.633886] ? set_kthread_struct+0x70/0x70 [ 7914.635957] ret_from_fork+0x1f/0x30 [ 7914.637479] rcu: Stack dump where RCU GP kthread last ran: [ 7914.640335] Sending NMI from CPU 2 to CPUs 0: [ 7914.643893] NMI backtrace for cpu 0 [ 7914.643900] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7914.643903] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7914.643904] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7914.643906] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7914.643908] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7914.643910] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 [ 7914.643912] RAX: 000000000000180f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7914.643914] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7914.643915] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7914.643916] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7914.643918] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7914.643919] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7914.643921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7914.643935] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7914.643936] Call Trace: [ 7914.643937] [ 7914.643938] ? show_regs.cold.9+0x22/0x2f [ 7914.643939] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7914.643941] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7914.643942] ? nmi_handle+0xbd/0x210 [ 7914.643943] ? default_do_nmi+0x4b/0x1b0 [ 7914.643945] ? do_nmi+0x280/0x380 [ 7914.643946] ? end_repeat_nmi+0x16/0x1f [ 7914.643947] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7914.643948] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7914.643950] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7914.643951] [ 7914.643952] ? newidle_balance.isra.93+0x207/0x580 [ 7914.643953] do_raw_spin_lock+0x177/0x190 [ 7914.643954] _raw_spin_lock+0x21/0x30 [ 7914.643955] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7914.643956] process_one_work+0x2c8/0x700 [ 7914.643957] worker_thread+0x296/0x6e0 [ 7914.643958] ? rescuer_thread+0x570/0x570 [ 7914.643960] kthread+0x1d1/0x200 [ 7914.643960] ? set_kthread_struct+0x70/0x70 [ 7914.643962] ret_from_fork+0x1f/0x30 [ 7914.644355] Sending NMI from CPU 2 to CPUs 1: [ 7914.756620] NMI backtrace for cpu 1 [ 7914.756628] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7914.756630] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7914.756632] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7914.756634] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7914.756636] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [ 7914.756639] RAX: 0000000000005844 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7914.756640] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7914.756641] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7914.756643] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7914.756644] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7914.756646] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7914.756647] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7914.756648] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7914.756649] Call Trace: [ 7914.756650] [ 7914.756651] ? show_regs.cold.9+0x22/0x2f [ 7914.756652] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 7914.756653] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 7914.756654] ? nmi_handle+0xbd/0x210 [ 7914.756655] ? default_do_nmi+0x4b/0x1b0 [ 7914.756656] ? do_nmi+0x280/0x380 [ 7914.756657] ? end_repeat_nmi+0x16/0x1f [ 7914.756658] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7914.756659] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7914.756660] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7914.756661] [ 7914.756662] do_raw_spin_lock+0x177/0x190 [ 7914.756663] _raw_spin_lock+0x21/0x30 [ 7914.756664] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7914.756665] ? do_raw_spin_unlock+0x75/0x190 [ 7914.756666] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7914.756667] ? do_wait_intr+0xf0/0xf0 [ 7914.756668] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7914.756669] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7914.756670] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7914.756671] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7914.756672] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7914.756673] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7914.756674] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7914.756675] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7914.756676] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7914.756677] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7914.756678] lmv_intent_open+0x338/0x1150 [lmv] [ 7914.756679] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7914.756680] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7914.756681] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7914.756682] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7914.756683] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7914.756684] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7914.756685] ? do_raw_spin_unlock+0x75/0x190 [ 7914.756686] ? _raw_spin_unlock+0x12/0x30 [ 7914.756687] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7914.756688] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7914.756689] ll_atomic_open+0x36a/0xd70 [lustre] [ 7914.756690] lookup_open+0x356/0xb40 [ 7914.756691] do_last+0x313/0xfc0 [ 7914.756691] ? nd_jump_root+0xe5/0x160 [ 7914.756692] ? path_init+0x437/0x520 [ 7914.756693] path_openat+0xf7/0x500 [ 7914.756694] ? do_raw_spin_unlock+0x75/0x190 [ 7914.756695] do_filp_open+0x99/0x140 [ 7914.756696] ? getname_flags+0x6e/0x330 [ 7914.756697] ? __check_object_size+0xff/0x256 [ 7914.756698] ? do_raw_spin_unlock+0x75/0x190 [ 7914.756699] ? _raw_spin_unlock+0x12/0x30 [ 7914.756699] do_sys_openat2+0x2b4/0x410 [ 7914.756700] do_sys_open+0x73/0xa0 [ 7914.756701] __x64_sys_openat+0x24/0x30 [ 7914.756702] do_syscall_64+0xc1/0x440 [ 7914.756703] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7914.756704] RIP: 0033:0x7f53425a5332 [ 7914.756705] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7914.756707] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7914.756709] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7914.756710] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7914.756711] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7914.756712] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7914.756713] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7914.757591] NMI backtrace for cpu 2 [ 7915.041300] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7915.045054] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7915.047821] Call Trace: [ 7915.048749] [ 7915.049509] ? dump_stack+0xbb/0x10e [ 7915.050661] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 7915.052516] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 7915.053980] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 7915.055513] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 7915.056927] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 7915.058510] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 7915.059666] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 7915.061336] ? update_process_times+0x6d/0xc0 [ 7915.062613] ? tick_sched_handle.isra.18+0x27/0x90 [ 7915.064026] ? tick_sched_timer+0x53/0xb0 [ 7915.065030] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7915.066508] ? tick_sched_handle.isra.18+0x90/0x90 [ 7915.069478] ? hrtimer_interrupt+0x128/0x2c0 [ 7915.070827] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7915.072328] ? apic_timer_interrupt+0xf/0x20 [ 7915.073548] [ 7915.074252] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7915.075852] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7915.078052] ? do_raw_spin_lock+0x177/0x190 [ 7915.079346] ? _raw_spin_lock+0x21/0x30 [ 7915.080545] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7915.083071] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 7915.085236] ? mdc_commit_open+0x91/0xb0 [mdc] [ 7915.086826] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7915.088699] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7915.090655] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 7915.092309] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7915.094028] ? __next_timer_interrupt+0x160/0x160 [ 7915.095498] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7915.097249] ? do_wait_intr+0xf0/0xf0 [ 7915.098519] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7915.100519] ? kthread+0x1d1/0x200 [ 7915.102530] ? set_kthread_struct+0x70/0x70 [ 7915.104230] ? ret_from_fork+0x1f/0x30 [ 7916.058316] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 7916.072052] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7916.124427] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7916.139159] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7916.147529] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7916.154473] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7916.181739] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7916.191465] RAX: 0000000000005e8a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7916.200797] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7916.208486] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7916.211762] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7916.214054] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7916.218211] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7916.222449] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7916.226112] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7916.233297] Call Trace: [ 7916.234104] [ 7916.234756] ? show_regs.cold.9+0x22/0x2f [ 7916.239409] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7916.242637] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7916.250532] ? watchdog+0x40/0x40 [ 7916.254472] ? hrtimer_interrupt+0x128/0x2c0 [ 7916.255920] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7916.259002] ? apic_timer_interrupt+0xf/0x20 [ 7916.261075] [ 7916.262763] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7916.265339] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7916.267555] do_raw_spin_lock+0x177/0x190 [ 7916.269625] _raw_spin_lock+0x21/0x30 [ 7916.277489] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7916.282097] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7916.285825] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7916.291111] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7916.294404] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7916.297470] ? woken_wake_function+0x30/0x30 [ 7916.299529] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7916.305813] kthread+0x1d1/0x200 [ 7916.307517] ? set_kthread_struct+0x70/0x70 [ 7916.311398] ret_from_fork+0x1f/0x30 [ 7940.052951] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 7940.055016] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7940.057017] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7940.057037] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7940.057116] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7940.057122] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7940.057125] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7940.057138] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7940.057143] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7940.057148] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7940.057150] RDX: 00000000000011bd RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7940.057153] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7940.057155] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7940.057158] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7940.057162] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7940.057165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7940.057167] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7940.057176] Call Trace: [ 7940.057181] [ 7940.057188] ? show_regs.cold.9+0x22/0x2f [ 7940.057196] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7940.057202] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7940.057206] ? watchdog+0x40/0x40 [ 7940.057210] ? hrtimer_interrupt+0x128/0x2c0 [ 7940.057215] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7940.057223] ? apic_timer_interrupt+0xf/0x20 [ 7940.057228] [ 7940.057231] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7940.057236] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7940.057241] do_raw_spin_lock+0x177/0x190 [ 7940.057245] _raw_spin_lock+0x21/0x30 [ 7940.057251] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7940.057443] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7940.057515] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7940.057521] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7940.057792] obd_mod_free+0x46/0x140 [ptlrpc] [ 7940.058274] mdc_commit_open+0x91/0xb0 [mdc] [ 7940.058317] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7940.058563] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7940.061426] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7940.061516] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7940.061524] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7940.061527] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7940.061540] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7940.061544] RSP: 0018:ffffbb5381503500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 7940.061549] RAX: 0000000000000d89 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7940.061551] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7940.061553] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7940.061555] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7940.061557] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7940.061560] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7940.061563] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7940.061566] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7940.061574] Call Trace: [ 7940.061579] [ 7940.061585] ? show_regs.cold.9+0x22/0x2f [ 7940.061595] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7940.061603] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7940.061607] ? watchdog+0x40/0x40 [ 7940.061611] ? hrtimer_interrupt+0x128/0x2c0 [ 7940.061615] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7940.061625] ? apic_timer_interrupt+0xf/0x20 [ 7940.061630] [ 7940.061633] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7940.061639] do_raw_spin_lock+0x177/0x190 [ 7940.061644] _raw_spin_lock+0x21/0x30 [ 7940.057526] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7940.061649] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7940.062193] ? do_raw_spin_unlock+0x75/0x190 [ 7940.062199] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7940.072558] ? do_wait_intr+0xf0/0xf0 [ 7940.072576] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7940.073124] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7940.073348] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7940.073587] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7940.074259] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7940.074298] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7940.074328] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7940.074355] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7940.074450] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7940.074727] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7940.074759] lmv_intent_open+0x338/0x1150 [lmv] [ 7940.074793] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7940.074966] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7940.075052] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7940.075076] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7940.075167] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7940.075256] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7940.075345] ? do_raw_spin_unlock+0x75/0x190 [ 7940.075352] ? _raw_spin_unlock+0x12/0x30 [ 7940.075359] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7940.075384] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7940.075403] ll_atomic_open+0x36a/0xd70 [lustre] [ 7940.075502] lookup_open+0x356/0xb40 [ 7940.075510] do_last+0x313/0xfc0 [ 7940.075515] ? nd_jump_root+0xe5/0x160 [ 7940.075519] ? path_init+0x437/0x520 [ 7940.075523] path_openat+0xf7/0x500 [ 7940.075528] ? do_raw_spin_unlock+0x75/0x190 [ 7940.075533] do_filp_open+0x99/0x140 [ 7940.075539] ? getname_flags+0x6e/0x330 [ 7940.075543] ? __check_object_size+0xff/0x256 [ 7940.075551] ? do_raw_spin_unlock+0x75/0x190 [ 7940.075554] ? _raw_spin_unlock+0x12/0x30 [ 7940.075559] do_sys_openat2+0x2b4/0x410 [ 7940.075563] do_sys_open+0x73/0xa0 [ 7940.075567] __x64_sys_openat+0x24/0x30 [ 7940.075570] do_syscall_64+0xc1/0x440 [ 7940.075578] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7940.075584] RIP: 0033:0x7f53425a5332 [ 7940.075592] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7940.075595] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7940.075600] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7940.075603] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7940.075605] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7940.075607] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7940.075608] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7940.078330] after_reply+0xe2c/0x1780 [ptlrpc] [ 7940.078621] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7940.078816] ? __next_timer_interrupt+0x160/0x160 [ 7940.078826] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7940.079031] ? do_wait_intr+0xf0/0xf0 [ 7940.079040] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7940.079238] kthread+0x1d1/0x200 [ 7940.079245] ? set_kthread_struct+0x70/0x70 [ 7940.079249] ret_from_fork+0x1f/0x30 [ 7940.886334] [ 7940.889210] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7940.895163] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7940.912290] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7940.919883] RAX: 0000000000003522 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7940.922401] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7940.935518] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7940.947198] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7940.951837] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7940.964441] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7940.977883] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7940.981108] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7940.987028] Call Trace: [ 7940.989219] [ 7940.990687] ? show_regs.cold.9+0x22/0x2f [ 7940.998537] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7941.003729] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7941.008510] ? watchdog+0x40/0x40 [ 7941.009683] ? hrtimer_interrupt+0x128/0x2c0 [ 7941.021227] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7941.028778] ? apic_timer_interrupt+0xf/0x20 [ 7941.031764] [ 7941.034234] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7941.037057] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7941.042602] ? newidle_balance.isra.93+0x207/0x580 [ 7941.045298] do_raw_spin_lock+0x177/0x190 [ 7941.047323] _raw_spin_lock+0x21/0x30 [ 7941.050338] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7941.053280] process_one_work+0x2c8/0x700 [ 7941.056257] worker_thread+0x296/0x6e0 [ 7941.058407] ? rescuer_thread+0x570/0x570 [ 7941.060544] kthread+0x1d1/0x200 [ 7941.062256] ? set_kthread_struct+0x70/0x70 [ 7941.064765] ret_from_fork+0x1f/0x30 [ 7941.066857] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5303s! [ 7941.075932] Showing busy workqueues and worker pools: [ 7941.077711] workqueue events: flags=0x0 [ 7941.079082] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7941.086120] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7941.091997] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7941.095304] pending: cpu_latency_work [ptlrpc] [ 7941.098380] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7941.101928] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7941.105849] workqueue events_long: flags=0x0 [ 7941.107748] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7941.114248] pending: lu_objects_flush [obdclass] [ 7941.122608] workqueue events_unbound: flags=0x2 [ 7941.125386] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7941.135054] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7941.139849] workqueue events_power_efficient: flags=0x80 [ 7941.143139] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7941.161885] pending: check_lifetime, do_cache_clean [ 7941.167617] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7941.174072] pending: neigh_periodic_work, neigh_periodic_work [ 7941.178025] workqueue mm_percpu_wq: flags=0x8 [ 7941.181896] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7941.185900] pending: vmstat_update, lru_add_drain_per_cpu [ 7941.189182] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7941.197134] pending: vmstat_update, lru_add_drain_per_cpu [ 7941.200269] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7941.204998] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7941.210759] workqueue xprtiod: flags=0xa [ 7941.216383] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7941.224082] pending: xprt_autoclose, xs_error_handle [ 7941.228383] workqueue ipv6_addrconf: flags=0x40008 [ 7941.234803] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7941.241844] pending: addrconf_verify_work [ 7941.243408] workqueue knbd0-recv: flags=0x1a [ 7941.255289] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7941.259267] in-flight: 651:recv_work [ 7941.263320] workqueue ptlrpc_pinger: flags=0x2 [ 7941.268076] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7941.273073] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7941.278957] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5304s workers=4 idle: 15979 31824 8 [ 7941.287353] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7931s workers=2 idle: 654 [ 7944.060485] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 7944.067658] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7944.088244] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7944.093684] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7944.099196] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7944.102895] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7944.118287] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7944.125669] RAX: 0000000000002248 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7944.130582] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7944.138187] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7944.146844] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7944.151711] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7944.158397] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7944.161873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7944.164896] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7944.168627] Call Trace: [ 7944.169795] [ 7944.170827] ? show_regs.cold.9+0x22/0x2f [ 7944.172934] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7944.175511] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7944.177629] ? watchdog+0x40/0x40 [ 7944.179323] ? hrtimer_interrupt+0x128/0x2c0 [ 7944.181613] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7944.184326] ? apic_timer_interrupt+0xf/0x20 [ 7944.186381] [ 7944.187374] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7944.190281] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7944.192799] do_raw_spin_lock+0x177/0x190 [ 7944.194874] _raw_spin_lock+0x21/0x30 [ 7944.196802] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7944.203815] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7944.208305] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7944.211893] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7944.217707] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7944.220729] ? woken_wake_function+0x30/0x30 [ 7944.223383] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7944.226807] kthread+0x1d1/0x200 [ 7944.229104] ? set_kthread_struct+0x70/0x70 [ 7944.231635] ret_from_fork+0x1f/0x30 [ 7968.053354] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 7968.056154] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7968.057087] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7968.057096] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7968.057184] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7968.057190] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7968.057193] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7968.057206] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7968.057211] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7968.057215] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7968.057218] RDX: 0000000000003816 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7968.057220] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7968.057223] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7968.057225] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7968.057228] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7968.057231] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7968.057234] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7968.057242] Call Trace: [ 7968.057248] [ 7968.057254] ? show_regs.cold.9+0x22/0x2f [ 7968.057267] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7968.057274] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7968.057278] ? watchdog+0x40/0x40 [ 7968.057283] ? hrtimer_interrupt+0x128/0x2c0 [ 7968.057290] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7968.057298] ? apic_timer_interrupt+0xf/0x20 [ 7968.057303] [ 7968.057306] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7968.057310] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7968.057314] do_raw_spin_lock+0x177/0x190 [ 7968.057318] _raw_spin_lock+0x21/0x30 [ 7968.057324] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7968.057598] obd_mod_free+0x46/0x140 [ptlrpc] [ 7968.057781] mdc_commit_open+0x91/0xb0 [mdc] [ 7968.057817] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7968.058379] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7968.058654] after_reply+0xe2c/0x1780 [ptlrpc] [ 7968.058890] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7968.059181] ? __next_timer_interrupt+0x160/0x160 [ 7968.059191] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7968.059482] ? do_wait_intr+0xf0/0xf0 [ 7968.059491] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7968.059717] kthread+0x1d1/0x200 [ 7968.059723] ? set_kthread_struct+0x70/0x70 [ 7968.059727] ret_from_fork+0x1f/0x30 [ 7968.061229] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7968.061305] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7968.061312] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7968.061318] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7968.061810] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7968.061821] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7968.066506] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7968.066580] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7968.066587] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7968.066589] RIP: 0010:native_safe_halt+0x2e/0x50 [ 7968.066601] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 7968.066605] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7968.066609] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7968.066825] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7968.066828] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7968.066830] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7968.066832] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7968.066835] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7968.066838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7968.066840] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7968.066849] Call Trace: [ 7968.066855] [ 7968.066861] ? show_regs.cold.9+0x22/0x2f [ 7968.066872] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7968.066879] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7968.066883] ? watchdog+0x40/0x40 [ 7968.066888] ? hrtimer_interrupt+0x128/0x2c0 [ 7968.066892] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7968.066900] ? apic_timer_interrupt+0xf/0x20 [ 7968.066904] [ 7968.066907] ? native_safe_halt+0x2e/0x50 [ 7968.066912] kvm_wait+0xcd/0xe0 [ 7968.066919] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7968.066928] do_raw_spin_lock+0x177/0x190 [ 7968.066932] _raw_spin_lock+0x21/0x30 [ 7968.066937] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7968.067217] ? do_raw_spin_unlock+0x75/0x190 [ 7968.067224] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7968.067738] ? do_wait_intr+0xf0/0xf0 [ 7968.067747] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7968.072787] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7968.072802] RAX: 0000000000006588 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7968.072805] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7968.072807] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7968.072809] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7968.072812] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7968.072815] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7968.072818] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7968.072821] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7968.072829] Call Trace: [ 7968.072837] [ 7968.072842] ? show_regs.cold.9+0x22/0x2f [ 7968.072856] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7968.072933] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7968.072939] ? watchdog+0x40/0x40 [ 7968.072944] ? hrtimer_interrupt+0x128/0x2c0 [ 7968.072948] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7968.072957] ? apic_timer_interrupt+0xf/0x20 [ 7968.072961] [ 7968.072964] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7968.072971] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7968.072974] ? newidle_balance.isra.93+0x207/0x580 [ 7968.072982] do_raw_spin_lock+0x177/0x190 [ 7968.072998] _raw_spin_lock+0x21/0x30 [ 7968.073003] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7968.117525] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7968.126708] process_one_work+0x2c8/0x700 [ 7968.126729] worker_thread+0x296/0x6e0 [ 7968.126734] ? rescuer_thread+0x570/0x570 [ 7968.126738] kthread+0x1d1/0x200 [ 7968.126743] ? set_kthread_struct+0x70/0x70 [ 7968.126746] ret_from_fork+0x1f/0x30 [ 7968.751785] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7968.754713] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7968.758404] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7968.761561] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7968.763860] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7968.765920] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7968.769341] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7968.772139] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7968.774912] lmv_intent_open+0x338/0x1150 [lmv] [ 7968.777572] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7968.781709] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7968.784595] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7968.788918] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7968.791485] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7968.793015] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7968.794741] ? do_raw_spin_unlock+0x75/0x190 [ 7968.799710] ? _raw_spin_unlock+0x12/0x30 [ 7968.804262] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7968.809266] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7968.817510] ll_atomic_open+0x36a/0xd70 [lustre] [ 7968.821663] lookup_open+0x356/0xb40 [ 7968.826703] do_last+0x313/0xfc0 [ 7968.831294] ? nd_jump_root+0xe5/0x160 [ 7968.836469] ? path_init+0x437/0x520 [ 7968.838641] path_openat+0xf7/0x500 [ 7968.841709] ? do_raw_spin_unlock+0x75/0x190 [ 7968.846604] do_filp_open+0x99/0x140 [ 7968.852198] ? getname_flags+0x6e/0x330 [ 7968.853914] ? __check_object_size+0xff/0x256 [ 7968.855445] ? do_raw_spin_unlock+0x75/0x190 [ 7968.858908] ? _raw_spin_unlock+0x12/0x30 [ 7968.860722] do_sys_openat2+0x2b4/0x410 [ 7968.863558] do_sys_open+0x73/0xa0 [ 7968.865599] __x64_sys_openat+0x24/0x30 [ 7968.867637] do_syscall_64+0xc1/0x440 [ 7968.869551] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7968.874699] RIP: 0033:0x7f53425a5332 [ 7968.881460] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7968.899676] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7968.909282] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7968.918196] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7968.924967] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7968.932560] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7968.938364] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7971.301929] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5334s! [ 7971.314241] Showing busy workqueues and worker pools: [ 7971.317646] workqueue events: flags=0x0 [ 7971.322274] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 7971.324640] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 7971.338783] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7971.344746] pending: cpu_latency_work [ptlrpc] [ 7971.350810] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 7971.356478] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 7971.362710] workqueue events_long: flags=0x0 [ 7971.365561] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 7971.368804] pending: lu_objects_flush [obdclass] [ 7971.371740] workqueue events_unbound: flags=0x2 [ 7971.373403] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 7971.374906] pending: flush_memcg_stats_dwork, idle_cull_fn [ 7971.377427] workqueue events_power_efficient: flags=0x80 [ 7971.380288] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7971.383424] pending: check_lifetime, do_cache_clean [ 7971.385786] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7971.390751] pending: neigh_periodic_work, neigh_periodic_work [ 7971.395355] workqueue mm_percpu_wq: flags=0x8 [ 7971.397383] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7971.406400] pending: vmstat_update, lru_add_drain_per_cpu [ 7971.414445] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 7971.420345] pending: vmstat_update, lru_add_drain_per_cpu [ 7971.423725] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 7971.432660] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 7971.437185] workqueue xprtiod: flags=0xa [ 7971.438223] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 7971.442407] pending: xprt_autoclose, xs_error_handle [ 7971.447465] workqueue ipv6_addrconf: flags=0x40008 [ 7971.450802] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 7971.455794] pending: addrconf_verify_work [ 7971.464421] workqueue knbd0-recv: flags=0x1a [ 7971.474442] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 7971.477798] in-flight: 651:recv_work [ 7971.479838] workqueue ptlrpc_pinger: flags=0x2 [ 7971.481988] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 7971.489581] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 7971.500468] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5334s workers=4 idle: 15979 31824 8 [ 7971.508548] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7961s workers=2 idle: 654 [ 7972.058292] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 7972.062522] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7972.089094] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7972.096350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7972.100176] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7972.105683] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7972.120097] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 7972.130775] RAX: 0000000000006d0f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7972.135565] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 7972.137721] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7972.142701] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 7972.148575] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 7972.151223] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 7972.154768] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7972.157191] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 7972.163484] Call Trace: [ 7972.164508] [ 7972.167424] ? show_regs.cold.9+0x22/0x2f [ 7972.168846] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7972.177793] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7972.180863] ? watchdog+0x40/0x40 [ 7972.184323] ? hrtimer_interrupt+0x128/0x2c0 [ 7972.191382] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7972.195885] ? apic_timer_interrupt+0xf/0x20 [ 7972.198153] [ 7972.198876] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7972.202631] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7972.208398] do_raw_spin_lock+0x177/0x190 [ 7972.212771] _raw_spin_lock+0x21/0x30 [ 7972.219545] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 7972.222503] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 7972.225956] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 7972.229756] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 7972.235984] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 7972.238515] ? woken_wake_function+0x30/0x30 [ 7972.239868] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 7972.242800] kthread+0x1d1/0x200 [ 7972.243872] ? set_kthread_struct+0x70/0x70 [ 7972.245127] ret_from_fork+0x1f/0x30 [ 7996.052397] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 7996.055037] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 7996.058617] Modules linked in: lustre(O) [ 7996.061255] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 7996.061269] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7996.061340] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7996.061345] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7996.061348] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7996.061364] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 7996.061369] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7996.061374] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 7996.061376] RDX: 0000000000006a64 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 7996.061379] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 7996.061382] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 7996.061384] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 7996.061388] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 7996.062449] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7996.062453] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 7996.062461] Call Trace: [ 7996.062467] [ 7996.062474] ? show_regs.cold.9+0x22/0x2f [ 7996.062486] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7996.062492] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7996.062507] ? watchdog+0x40/0x40 [ 7996.062512] ? hrtimer_interrupt+0x128/0x2c0 [ 7996.062517] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7996.062525] ? apic_timer_interrupt+0xf/0x20 [ 7996.062530] [ 7996.062532] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 7996.062539] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 7996.062545] do_raw_spin_lock+0x177/0x190 [ 7996.062550] _raw_spin_lock+0x21/0x30 [ 7996.062558] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 7996.062830] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7996.062901] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7996.062907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7996.062909] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7996.062918] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7996.062921] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 7996.062843] obd_mod_free+0x46/0x140 [ptlrpc] [ 7996.063147] RAX: 0000000000006117 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7996.063150] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 7996.063152] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 7996.063154] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 7996.063157] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 7996.063160] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 7996.063163] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7996.063166] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 7996.063176] Call Trace: [ 7996.063180] [ 7996.063183] ? show_regs.cold.9+0x22/0x2f [ 7996.063191] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7996.063196] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7996.063199] ? watchdog+0x40/0x40 [ 7996.063203] ? hrtimer_interrupt+0x128/0x2c0 [ 7996.063207] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7996.063212] ? apic_timer_interrupt+0xf/0x20 [ 7996.063216] [ 7996.063219] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7996.063226] do_raw_spin_lock+0x177/0x190 [ 7996.063231] _raw_spin_lock+0x21/0x30 [ 7996.063493] mdc_commit_open+0x91/0xb0 [mdc] [ 7996.063543] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 7996.063235] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 7996.064146] ? do_raw_spin_unlock+0x75/0x190 [ 7996.064153] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 7996.064357] ? do_wait_intr+0xf0/0xf0 [ 7996.064366] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 7996.065675] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 7996.065911] after_reply+0xe2c/0x1780 [ptlrpc] [ 7996.066662] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 7996.067070] ? __next_timer_interrupt+0x160/0x160 [ 7996.067111] osp(O) [ 7996.067083] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 7996.067449] ? do_wait_intr+0xf0/0xf0 [ 7996.067459] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 7996.067671] kthread+0x1d1/0x200 [ 7996.067677] ? set_kthread_struct+0x70/0x70 [ 7996.067681] ret_from_fork+0x1f/0x30 [ 7996.070614] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 7996.070905] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7996.071203] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7996.071324] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7996.071366] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 7996.071396] mdc_intent_lock+0x353/0x7e0 [mdc] [ 7996.071422] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7996.071565] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 7996.071783] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 7996.071812] lmv_intent_open+0x338/0x1150 [lmv] [ 7996.071843] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7996.071929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7996.072016] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 7996.072039] ll_intent_lock+0xc5/0xaa0 [lustre] [ 7996.072130] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 7996.072219] ll_lookup_it+0xe0e/0x2270 [lustre] [ 7996.072306] ? do_raw_spin_unlock+0x75/0x190 [ 7996.072314] ? _raw_spin_unlock+0x12/0x30 [ 7996.072320] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 7996.072348] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 7996.072369] ll_atomic_open+0x36a/0xd70 [lustre] [ 7996.072458] lookup_open+0x356/0xb40 [ 7996.072467] do_last+0x313/0xfc0 [ 7996.072473] ? nd_jump_root+0xe5/0x160 [ 7996.072478] ? path_init+0x437/0x520 [ 7996.072482] path_openat+0xf7/0x500 [ 7996.072487] ? do_raw_spin_unlock+0x75/0x190 [ 7996.072492] do_filp_open+0x99/0x140 [ 7996.072661] ? getname_flags+0x6e/0x330 [ 7996.072667] ? __check_object_size+0xff/0x256 [ 7996.072674] ? do_raw_spin_unlock+0x75/0x190 [ 7996.072678] ? _raw_spin_unlock+0x12/0x30 [ 7996.072683] do_sys_openat2+0x2b4/0x410 [ 7996.072688] do_sys_open+0x73/0xa0 [ 7996.072692] __x64_sys_openat+0x24/0x30 [ 7996.072696] do_syscall_64+0xc1/0x440 [ 7996.072707] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 7996.072714] RIP: 0033:0x7f53425a5332 [ 7996.072722] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 7996.072726] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 7996.072731] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 7996.072734] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 7996.072736] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 7996.072738] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 7996.072741] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 7996.844043] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 7996.886976] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 7996.897272] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 7996.903466] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 7996.907079] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7996.911203] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 7996.920013] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 7996.923730] RAX: 0000000000002d8a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 7996.927190] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 7996.931643] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 7996.935860] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 7996.948183] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 7996.952951] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 7996.961681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7996.970093] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 7996.974457] Call Trace: [ 7996.977160] [ 7996.980453] ? show_regs.cold.9+0x22/0x2f [ 7996.985611] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 7996.987310] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 7996.994725] ? watchdog+0x40/0x40 [ 7996.996644] ? hrtimer_interrupt+0x128/0x2c0 [ 7996.997993] ? smp_apic_timer_interrupt+0xa6/0x270 [ 7997.006019] ? apic_timer_interrupt+0xf/0x20 [ 7997.012626] [ 7997.013583] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 7997.022126] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 7997.024033] ? newidle_balance.isra.93+0x207/0x580 [ 7997.031073] do_raw_spin_lock+0x177/0x190 [ 7997.040034] _raw_spin_lock+0x21/0x30 [ 7997.043932] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 7997.052874] process_one_work+0x2c8/0x700 [ 7997.056433] worker_thread+0x296/0x6e0 [ 7997.069256] ? rescuer_thread+0x570/0x570 [ 7997.070798] kthread+0x1d1/0x200 [ 7997.074660] ? set_kthread_struct+0x70/0x70 [ 7997.075851] ret_from_fork+0x1f/0x30 [ 8000.058816] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 8000.068778] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8000.120306] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8000.128473] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8000.137602] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8000.141831] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8000.165496] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8000.168170] RAX: 0000000000007d18 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8000.171269] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8000.174563] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8000.177741] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8000.181267] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8000.184696] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8000.188603] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8000.191318] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8000.194702] Call Trace: [ 8000.195581] [ 8000.196267] ? show_regs.cold.9+0x22/0x2f [ 8000.201919] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8000.205381] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8000.214751] ? watchdog+0x40/0x40 [ 8000.218922] ? hrtimer_interrupt+0x128/0x2c0 [ 8000.224390] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8000.232391] ? apic_timer_interrupt+0xf/0x20 [ 8000.235254] [ 8000.248259] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8000.254289] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8000.262304] do_raw_spin_lock+0x177/0x190 [ 8000.265420] _raw_spin_lock+0x21/0x30 [ 8000.270551] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8000.281437] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8000.282890] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8000.291384] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8000.302040] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8000.308803] ? woken_wake_function+0x30/0x30 [ 8000.312786] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8000.322278] kthread+0x1d1/0x200 [ 8000.326124] ? set_kthread_struct+0x70/0x70 [ 8000.328341] ret_from_fork+0x1f/0x30 [ 8002.015139] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5364s! [ 8002.020607] Showing busy workqueues and worker pools: [ 8002.023729] workqueue events: flags=0x0 [ 8002.025720] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8002.028917] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8002.034324] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8002.037260] pending: cpu_latency_work [ptlrpc] [ 8002.039471] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8002.041647] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8002.044839] workqueue events_long: flags=0x0 [ 8002.046139] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8002.048526] pending: lu_objects_flush [obdclass] [ 8002.051149] workqueue events_unbound: flags=0x2 [ 8002.054213] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8002.056370] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8002.058690] workqueue events_power_efficient: flags=0x80 [ 8002.061933] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8002.065313] pending: check_lifetime, do_cache_clean [ 8002.067975] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8002.070755] pending: neigh_periodic_work, neigh_periodic_work [ 8002.073744] workqueue mm_percpu_wq: flags=0x8 [ 8002.075593] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8002.078265] pending: vmstat_update, lru_add_drain_per_cpu [ 8002.080357] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8002.082956] pending: vmstat_update, lru_add_drain_per_cpu [ 8002.085247] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8002.088475] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8002.091282] workqueue xprtiod: flags=0xa [ 8002.092853] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8002.095564] pending: xprt_autoclose, xs_error_handle [ 8002.097707] workqueue ipv6_addrconf: flags=0x40008 [ 8002.099652] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8002.106213] pending: addrconf_verify_work [ 8002.107652] workqueue knbd0-recv: flags=0x1a [ 8002.117713] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8002.130171] in-flight: 651:recv_work [ 8002.131590] workqueue ptlrpc_pinger: flags=0x2 [ 8002.133101] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8002.145489] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8002.153872] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5365s workers=4 idle: 15979 31824 8 [ 8002.160911] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=7992s workers=2 idle: 654 [ 8024.055367] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 8024.057816] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8024.059545] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 8024.078450] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8024.086402] Modules linked in: [ 8024.090404] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8024.090414] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8024.090427] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8024.090431] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8024.090437] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8024.090439] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8024.090441] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8024.090443] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8024.090446] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8024.090449] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8024.090452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8024.090454] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8024.090462] Call Trace: [ 8024.090467] [ 8024.090472] ? show_regs.cold.9+0x22/0x2f [ 8024.090482] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8024.090488] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8024.090493] ? watchdog+0x40/0x40 [ 8024.090498] ? hrtimer_interrupt+0x128/0x2c0 [ 8024.090502] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8024.090509] ? apic_timer_interrupt+0xf/0x20 [ 8024.090514] [ 8024.090517] ? native_safe_halt+0x2e/0x50 [ 8024.090521] kvm_wait+0xcd/0xe0 [ 8024.090529] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8024.090539] do_raw_spin_lock+0x177/0x190 [ 8024.090544] _raw_spin_lock+0x21/0x30 [ 8024.090548] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8024.091078] ? do_raw_spin_unlock+0x75/0x190 [ 8024.091085] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8024.091479] ? do_wait_intr+0xf0/0xf0 [ 8024.091489] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8024.091706] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8024.091943] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8024.092612] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8024.092730] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8024.092827] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8024.092858] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8024.092888] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8024.092971] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8024.093277] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8024.093308] lmv_intent_open+0x338/0x1150 [lmv] [ 8024.093341] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8024.093434] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8024.093527] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8024.093550] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8024.093643] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8024.093733] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8024.095662] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8024.095731] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8024.095737] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8024.095739] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8024.095806] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8024.095810] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8024.095814] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8024.095817] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8024.095819] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8024.095822] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8024.095824] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8024.095828] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8024.095832] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8024.095834] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8024.095842] Call Trace: [ 8024.095847] [ 8024.095851] ? show_regs.cold.9+0x22/0x2f [ 8024.095862] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8024.095869] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8024.095873] ? watchdog+0x40/0x40 [ 8024.095877] ? hrtimer_interrupt+0x128/0x2c0 [ 8024.095882] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8024.095888] ? apic_timer_interrupt+0xf/0x20 [ 8024.095893] [ 8024.095897] ? native_safe_halt+0x2e/0x50 [ 8024.095901] kvm_wait+0xcd/0xe0 [ 8024.095909] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8024.095916] do_raw_spin_lock+0x177/0x190 [ 8024.095921] _raw_spin_lock+0x21/0x30 [ 8024.095926] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8024.102272] ? do_raw_spin_unlock+0x75/0x190 [ 8024.102290] ? _raw_spin_unlock+0x12/0x30 [ 8024.102297] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8024.102325] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8024.102347] ll_atomic_open+0x36a/0xd70 [lustre] [ 8024.102462] lookup_open+0x356/0xb40 [ 8024.102472] do_last+0x313/0xfc0 [ 8024.102477] ? nd_jump_root+0xe5/0x160 [ 8024.102482] ? path_init+0x437/0x520 [ 8024.102487] path_openat+0xf7/0x500 [ 8024.102492] ? do_raw_spin_unlock+0x75/0x190 [ 8024.102496] do_filp_open+0x99/0x140 [ 8024.102503] ? getname_flags+0x6e/0x330 [ 8024.102507] ? __check_object_size+0xff/0x256 [ 8024.102514] ? do_raw_spin_unlock+0x75/0x190 [ 8024.102518] ? _raw_spin_unlock+0x12/0x30 [ 8024.102524] do_sys_openat2+0x2b4/0x410 [ 8024.102528] do_sys_open+0x73/0xa0 [ 8024.102533] __x64_sys_openat+0x24/0x30 [ 8024.102537] do_syscall_64+0xc1/0x440 [ 8024.102544] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8024.102550] RIP: 0033:0x7f53425a5332 [ 8024.102558] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8024.102563] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8024.102569] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8024.102571] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8024.102573] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8024.102575] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8024.102577] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8024.411608] obd_mod_free+0x46/0x140 [ptlrpc] [ 8024.413188] mdc_commit_open+0x91/0xb0 [mdc] [ 8024.415595] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8024.418347] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8024.421681] after_reply+0xe2c/0x1780 [ptlrpc] [ 8024.425346] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8024.427434] ? __next_timer_interrupt+0x160/0x160 [ 8024.429379] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8024.431241] ? do_wait_intr+0xf0/0xf0 [ 8024.432571] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8024.434695] kthread+0x1d1/0x200 [ 8024.435945] ? set_kthread_struct+0x70/0x70 [ 8024.437405] ret_from_fork+0x1f/0x30 [ 8028.052522] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8028.059864] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8028.062227] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 8028.116449] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8028.116465] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8028.116472] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8028.116761] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8028.116772] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8028.116776] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8028.116781] RAX: 0000000000003d95 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8028.116784] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8028.116785] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8028.116788] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8028.116790] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8028.116793] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8028.116796] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8028.116798] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8028.116808] Call Trace: [ 8028.116819] [ 8028.116826] ? show_regs.cold.9+0x22/0x2f [ 8028.116835] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8028.116842] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8028.116846] ? watchdog+0x40/0x40 [ 8028.116850] ? hrtimer_interrupt+0x128/0x2c0 [ 8028.116854] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8028.116863] ? apic_timer_interrupt+0xf/0x20 [ 8028.116868] [ 8028.116870] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8028.116875] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8028.116879] ? newidle_balance.isra.93+0x207/0x580 [ 8028.116885] do_raw_spin_lock+0x177/0x190 [ 8028.116890] _raw_spin_lock+0x21/0x30 [ 8028.116895] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8028.117975] process_one_work+0x2c8/0x700 [ 8028.117984] worker_thread+0x296/0x6e0 [ 8028.117988] ? rescuer_thread+0x570/0x570 [ 8028.117992] kthread+0x1d1/0x200 [ 8028.117997] ? set_kthread_struct+0x70/0x70 [ 8028.118000] ret_from_fork+0x1f/0x30 [ 8028.203177] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8028.226496] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8028.232643] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8028.238341] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8028.243014] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8028.251621] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8028.260816] RAX: 0000000000001c0c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8028.264069] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8028.268447] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8028.271663] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8028.274783] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8028.277930] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8028.281452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8028.283751] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8028.286256] Call Trace: [ 8028.287143] [ 8028.287855] ? show_regs.cold.9+0x22/0x2f [ 8028.289227] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8028.291449] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8028.293280] ? watchdog+0x40/0x40 [ 8028.294743] ? hrtimer_interrupt+0x128/0x2c0 [ 8028.296622] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8028.298710] ? apic_timer_interrupt+0xf/0x20 [ 8028.300284] [ 8028.301062] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8028.303903] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8028.306096] do_raw_spin_lock+0x177/0x190 [ 8028.307869] _raw_spin_lock+0x21/0x30 [ 8028.309686] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8028.312426] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8028.314851] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8028.317241] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8028.319822] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8028.322463] ? woken_wake_function+0x30/0x30 [ 8028.324703] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8028.327467] kthread+0x1d1/0x200 [ 8028.328835] ? set_kthread_struct+0x70/0x70 [ 8028.330739] ret_from_fork+0x1f/0x30 [ 8032.224037] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5395s! [ 8032.226380] Showing busy workqueues and worker pools: [ 8032.228012] workqueue events: flags=0x0 [ 8032.229370] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8032.231446] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8032.235728] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8032.239651] pending: cpu_latency_work [ptlrpc] [ 8032.243243] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8032.247022] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8032.250907] workqueue events_long: flags=0x0 [ 8032.252442] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8032.256399] pending: lu_objects_flush [obdclass] [ 8032.259623] workqueue events_unbound: flags=0x2 [ 8032.263955] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8032.269715] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8032.276379] workqueue events_power_efficient: flags=0x80 [ 8032.280969] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8032.285683] pending: check_lifetime, do_cache_clean [ 8032.289228] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8032.295342] pending: neigh_periodic_work, neigh_periodic_work [ 8032.298343] workqueue mm_percpu_wq: flags=0x8 [ 8032.301984] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8032.312500] pending: vmstat_update, lru_add_drain_per_cpu [ 8032.314818] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8032.327800] pending: vmstat_update, lru_add_drain_per_cpu [ 8032.331211] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8032.333630] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8032.337224] workqueue xprtiod: flags=0xa [ 8032.338754] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8032.341849] pending: xprt_autoclose, xs_error_handle [ 8032.343707] workqueue ipv6_addrconf: flags=0x40008 [ 8032.345726] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8032.349068] pending: addrconf_verify_work [ 8032.350614] workqueue knbd0-recv: flags=0x1a [ 8032.352440] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8032.354756] in-flight: 651:recv_work [ 8032.356731] workqueue ptlrpc_pinger: flags=0x2 [ 8032.358392] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8032.361531] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8032.364321] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5395s workers=4 idle: 15979 31824 8 [ 8032.368269] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8022s workers=2 idle: 654 [ 8052.055075] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 8052.057017] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 8052.057651] Modules linked in: [ 8052.059419] Modules linked in: [ 8052.059444] lustre(O) [ 8052.060270] lustre(O) [ 8052.061625] osp(O) [ 8052.062193] osp(O) [ 8052.063251] ofd(O) [ 8052.063888] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) [ 8052.064757] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8052.064853] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8052.064859] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8052.064861] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8052.064892] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8052.064896] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8052.064899] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8052.064901] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8052.064903] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8052.064904] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8052.064907] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8052.064910] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8052.064912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8052.064914] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8052.064920] Call Trace: [ 8052.064931] [ 8052.064936] ? show_regs.cold.9+0x22/0x2f [ 8052.064956] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8052.064962] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8052.064966] ? watchdog+0x40/0x40 [ 8052.064969] ? hrtimer_interrupt+0x128/0x2c0 [ 8052.064971] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8052.064977] ? apic_timer_interrupt+0xf/0x20 [ 8052.064981] [ 8052.064983] ? native_safe_halt+0x2e/0x50 [ 8052.064985] kvm_wait+0xcd/0xe0 [ 8052.064992] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8052.065000] do_raw_spin_lock+0x177/0x190 [ 8052.065003] _raw_spin_lock+0x21/0x30 [ 8052.065006] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8052.065311] ? do_raw_spin_unlock+0x75/0x190 [ 8052.065316] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8052.065507] ? do_wait_intr+0xf0/0xf0 [ 8052.065704] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8052.065844] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8052.066739] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8052.066774] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8052.066779] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8052.066780] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8052.066789] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8052.066791] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8052.066795] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8052.066796] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8052.066798] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8052.066800] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8052.066801] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8052.066804] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8052.066806] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8052.066807] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8052.066813] Call Trace: [ 8052.066815] [ 8052.066817] ? show_regs.cold.9+0x22/0x2f [ 8052.066823] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8052.066828] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8052.066830] ? watchdog+0x40/0x40 [ 8052.066833] ? hrtimer_interrupt+0x128/0x2c0 [ 8052.066835] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8052.066840] ? apic_timer_interrupt+0xf/0x20 [ 8052.066844] [ 8052.066846] ? native_safe_halt+0x2e/0x50 [ 8052.066848] kvm_wait+0xcd/0xe0 [ 8052.066853] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8052.066858] do_raw_spin_lock+0x177/0x190 [ 8052.066860] _raw_spin_lock+0x21/0x30 [ 8052.066863] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8052.067011] obd_mod_free+0x46/0x140 [ptlrpc] [ 8052.067176] mdc_commit_open+0x91/0xb0 [mdc] [ 8052.067211] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8052.067407] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8052.067773] after_reply+0xe2c/0x1780 [ptlrpc] [ 8052.068014] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8052.068194] ? __next_timer_interrupt+0x160/0x160 [ 8052.068201] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8052.068334] ? do_wait_intr+0xf0/0xf0 [ 8052.068340] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8052.068470] kthread+0x1d1/0x200 [ 8052.068474] ? set_kthread_struct+0x70/0x70 [ 8052.068477] ret_from_fork+0x1f/0x30 [ 8052.212867] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8052.214762] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8052.216361] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8052.218028] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8052.219752] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8052.221244] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8052.222694] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8052.224762] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8052.226320] lmv_intent_open+0x338/0x1150 [lmv] [ 8052.227898] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8052.230590] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8052.233171] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8052.237496] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8052.240209] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8052.242750] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8052.244269] ? do_raw_spin_unlock+0x75/0x190 [ 8052.248111] ? _raw_spin_unlock+0x12/0x30 [ 8052.250490] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8052.253147] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8052.255789] ll_atomic_open+0x36a/0xd70 [lustre] [ 8052.257794] lookup_open+0x356/0xb40 [ 8052.259070] do_last+0x313/0xfc0 [ 8052.263060] ? nd_jump_root+0xe5/0x160 [ 8052.264325] ? path_init+0x437/0x520 [ 8052.265487] path_openat+0xf7/0x500 [ 8052.266644] ? do_raw_spin_unlock+0x75/0x190 [ 8052.268061] do_filp_open+0x99/0x140 [ 8052.269208] ? getname_flags+0x6e/0x330 [ 8052.270811] ? __check_object_size+0xff/0x256 [ 8052.271925] ? do_raw_spin_unlock+0x75/0x190 [ 8052.274381] ? _raw_spin_unlock+0x12/0x30 [ 8052.277278] do_sys_openat2+0x2b4/0x410 [ 8052.279145] do_sys_open+0x73/0xa0 [ 8052.280006] __x64_sys_openat+0x24/0x30 [ 8052.281466] do_syscall_64+0xc1/0x440 [ 8052.287062] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8052.291867] RIP: 0033:0x7f53425a5332 [ 8052.295676] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8052.303315] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8052.315196] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8052.322306] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8052.324834] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8052.326934] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8052.329854] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8056.063387] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 8056.068685] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8056.108563] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8056.115236] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8056.123300] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 8056.127897] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 8056.139122] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8056.146190] RAX: 0000000000001c9f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8056.148405] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8056.150997] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8056.153326] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8056.156122] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8056.159420] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8056.162715] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8056.164132] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8056.166433] Call Trace: [ 8056.167352] [ 8056.168015] ? show_regs.cold.9+0x22/0x2f [ 8056.170369] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8056.173951] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8056.175120] ? watchdog+0x40/0x40 [ 8056.175957] ? hrtimer_interrupt+0x128/0x2c0 [ 8056.180423] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8056.184016] ? apic_timer_interrupt+0xf/0x20 [ 8056.187171] [ 8056.187825] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 8056.189972] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8056.191886] do_raw_spin_lock+0x177/0x190 [ 8056.193384] _raw_spin_lock+0x21/0x30 [ 8056.196720] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8056.200058] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8056.201712] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8056.204405] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8056.210021] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8056.218154] ? woken_wake_function+0x30/0x30 [ 8056.221695] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8056.224015] kthread+0x1d1/0x200 [ 8056.225193] ? set_kthread_struct+0x70/0x70 [ 8056.226951] ret_from_fork+0x1f/0x30 [ 8060.052104] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8060.058781] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8060.096956] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8060.110104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8060.115960] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8060.123677] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8060.130738] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8060.147164] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8060.150281] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8060.158141] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8060.166372] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8060.175955] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8060.181124] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8060.189239] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8060.194460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8060.207347] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8060.214633] Call Trace: [ 8060.217291] [ 8060.219912] ? show_regs.cold.9+0x22/0x2f [ 8060.224072] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8060.235188] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8060.241693] ? watchdog+0x40/0x40 [ 8060.243482] ? hrtimer_interrupt+0x128/0x2c0 [ 8060.249301] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8060.254187] ? apic_timer_interrupt+0xf/0x20 [ 8060.261718] [ 8060.265661] ? native_safe_halt+0x2e/0x50 [ 8060.270690] kvm_wait+0xcd/0xe0 [ 8060.276188] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8060.284050] ? newidle_balance.isra.93+0x207/0x580 [ 8060.287175] do_raw_spin_lock+0x177/0x190 [ 8060.297725] _raw_spin_lock+0x21/0x30 [ 8060.300324] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8060.310514] process_one_work+0x2c8/0x700 [ 8060.317518] worker_thread+0x296/0x6e0 [ 8060.325288] ? rescuer_thread+0x570/0x570 [ 8060.332062] kthread+0x1d1/0x200 [ 8060.338687] ? set_kthread_struct+0x70/0x70 [ 8060.343893] ret_from_fork+0x1f/0x30 [ 8062.441448] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5425s! [ 8062.453322] Showing busy workqueues and worker pools: [ 8062.461283] workqueue events: flags=0x0 [ 8062.467108] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8062.472463] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8062.496849] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8062.504184] pending: cpu_latency_work [ptlrpc] [ 8062.511540] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8062.519150] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8062.531015] workqueue events_long: flags=0x0 [ 8062.536099] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8062.551324] pending: lu_objects_flush [obdclass] [ 8062.556175] workqueue events_unbound: flags=0x2 [ 8062.560489] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8062.564698] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8062.568822] workqueue events_power_efficient: flags=0x80 [ 8062.571918] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8062.575295] pending: check_lifetime, do_cache_clean [ 8062.577708] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8062.581853] pending: neigh_periodic_work, neigh_periodic_work [ 8062.585806] workqueue mm_percpu_wq: flags=0x8 [ 8062.588129] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8062.591539] pending: vmstat_update, lru_add_drain_per_cpu [ 8062.593609] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8062.596380] pending: vmstat_update, lru_add_drain_per_cpu [ 8062.600892] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8062.607417] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8062.613022] workqueue xprtiod: flags=0xa [ 8062.617012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8062.618892] pending: xprt_autoclose, xs_error_handle [ 8062.630750] workqueue ipv6_addrconf: flags=0x40008 [ 8062.632780] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8062.637650] pending: addrconf_verify_work [ 8062.639442] workqueue knbd0-recv: flags=0x1a [ 8062.641101] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8062.644233] in-flight: 651:recv_work [ 8062.645947] workqueue ptlrpc_pinger: flags=0x2 [ 8062.647699] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8062.650639] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8062.654110] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5425s workers=4 idle: 15979 31824 8 [ 8062.657455] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8053s workers=2 idle: 654 [ 8080.057012] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 8080.058986] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8080.059456] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 8080.075331] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8080.075348] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8080.075351] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8080.075366] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8080.075371] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8080.075376] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8080.075379] RDX: 0000000000001568 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8080.075381] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8080.075384] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8080.075387] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8080.075390] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8080.075393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8080.075396] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8080.075404] Call Trace: [ 8080.075409] [ 8080.075414] ? show_regs.cold.9+0x22/0x2f [ 8080.075424] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8080.075430] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8080.075435] ? watchdog+0x40/0x40 [ 8080.075440] ? hrtimer_interrupt+0x128/0x2c0 [ 8080.075443] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8080.075453] ? apic_timer_interrupt+0xf/0x20 [ 8080.075457] [ 8080.075460] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8080.075466] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8080.075470] do_raw_spin_lock+0x177/0x190 [ 8080.075475] _raw_spin_lock+0x21/0x30 [ 8080.075482] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8080.075900] obd_mod_free+0x46/0x140 [ptlrpc] [ 8080.076184] mdc_commit_open+0x91/0xb0 [mdc] [ 8080.076226] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8080.076407] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8080.086101] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix [ 8080.090486] after_reply+0xe2c/0x1780 [ptlrpc] [ 8080.090776] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8080.091072] ? __next_timer_interrupt+0x160/0x160 [ 8080.091083] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8080.091293] ? do_wait_intr+0xf0/0xf0 [ 8080.091302] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8080.091504] kthread+0x1d1/0x200 [ 8080.091509] ? set_kthread_struct+0x70/0x70 [ 8080.091513] ret_from_fork+0x1f/0x30 [ 8080.306680] serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8080.310222] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8080.321227] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8080.327906] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8080.334753] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8080.346273] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8080.350462] RAX: 0000000000007509 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8080.355836] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8080.363544] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8080.366566] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8080.372692] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8080.380749] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8080.388753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8080.398980] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8080.407126] Call Trace: [ 8080.408661] [ 8080.410669] ? show_regs.cold.9+0x22/0x2f [ 8080.415646] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8080.418231] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8080.420518] ? watchdog+0x40/0x40 [ 8080.421766] ? hrtimer_interrupt+0x128/0x2c0 [ 8080.426541] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8080.429518] ? apic_timer_interrupt+0xf/0x20 [ 8080.430922] [ 8080.431634] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8080.437687] do_raw_spin_lock+0x177/0x190 [ 8080.438871] _raw_spin_lock+0x21/0x30 [ 8080.445036] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8080.446663] ? do_raw_spin_unlock+0x75/0x190 [ 8080.450944] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8080.455372] ? do_wait_intr+0xf0/0xf0 [ 8080.456678] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8080.461642] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8080.466952] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8080.471273] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8080.475420] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8080.482108] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8080.486303] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8080.489836] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8080.496862] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8080.499139] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8080.505738] lmv_intent_open+0x338/0x1150 [lmv] [ 8080.510949] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8080.515670] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8080.524088] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8080.525628] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8080.533690] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8080.535105] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8080.538960] ? do_raw_spin_unlock+0x75/0x190 [ 8080.544851] ? _raw_spin_unlock+0x12/0x30 [ 8080.548966] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8080.550819] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8080.556813] ll_atomic_open+0x36a/0xd70 [lustre] [ 8080.559407] lookup_open+0x356/0xb40 [ 8080.564282] do_last+0x313/0xfc0 [ 8080.569566] ? nd_jump_root+0xe5/0x160 [ 8080.577837] ? path_init+0x437/0x520 [ 8080.580900] path_openat+0xf7/0x500 [ 8080.587680] ? do_raw_spin_unlock+0x75/0x190 [ 8080.589256] do_filp_open+0x99/0x140 [ 8080.590515] ? getname_flags+0x6e/0x330 [ 8080.600314] ? __check_object_size+0xff/0x256 [ 8080.608228] ? do_raw_spin_unlock+0x75/0x190 [ 8080.612688] ? _raw_spin_unlock+0x12/0x30 [ 8080.622451] do_sys_openat2+0x2b4/0x410 [ 8080.623903] do_sys_open+0x73/0xa0 [ 8080.626609] __x64_sys_openat+0x24/0x30 [ 8080.629525] do_syscall_64+0xc1/0x440 [ 8080.635330] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8080.646773] RIP: 0033:0x7f53425a5332 [ 8080.650335] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8080.661423] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8080.667787] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8080.670930] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8080.673873] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8080.676649] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8080.679755] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8084.060978] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 8084.070243] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8084.117239] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8084.127359] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8084.135427] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8084.139917] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8084.150158] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8084.153485] RAX: 0000000000002109 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8084.156930] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8084.161417] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8084.168373] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8084.172558] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8084.177843] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8084.183569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8084.189558] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8084.192482] Call Trace: [ 8084.193922] [ 8084.194880] ? show_regs.cold.9+0x22/0x2f [ 8084.196868] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8084.198812] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8084.203634] ? watchdog+0x40/0x40 [ 8084.205953] ? hrtimer_interrupt+0x128/0x2c0 [ 8084.210814] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8084.217705] ? apic_timer_interrupt+0xf/0x20 [ 8084.223902] [ 8084.224818] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8084.228403] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8084.231758] do_raw_spin_lock+0x177/0x190 [ 8084.235368] _raw_spin_lock+0x21/0x30 [ 8084.239037] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8084.248233] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8084.255794] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8084.261718] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8084.270419] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8084.279502] ? woken_wake_function+0x30/0x30 [ 8084.281923] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8084.285433] kthread+0x1d1/0x200 [ 8084.289474] ? set_kthread_struct+0x70/0x70 [ 8084.293296] ret_from_fork+0x1f/0x30 [ 8088.052076] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8088.055563] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8088.095824] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8088.106638] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8088.111540] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8088.114416] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8088.118631] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8088.137531] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8088.142457] RAX: 0000000000002a2b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8088.145234] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8088.152006] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8088.155797] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8088.164237] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8088.169085] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8088.174214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8088.180251] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8088.183479] Call Trace: [ 8088.184442] [ 8088.189563] ? show_regs.cold.9+0x22/0x2f [ 8088.191285] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8088.196230] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8088.200695] ? watchdog+0x40/0x40 [ 8088.206577] ? hrtimer_interrupt+0x128/0x2c0 [ 8088.210380] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8088.218630] ? apic_timer_interrupt+0xf/0x20 [ 8088.220628] [ 8088.221251] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8088.229526] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8088.237690] ? newidle_balance.isra.93+0x207/0x580 [ 8088.241111] do_raw_spin_lock+0x177/0x190 [ 8088.244581] _raw_spin_lock+0x21/0x30 [ 8088.247421] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8088.252494] process_one_work+0x2c8/0x700 [ 8088.256309] worker_thread+0x296/0x6e0 [ 8088.258531] ? rescuer_thread+0x570/0x570 [ 8088.260236] kthread+0x1d1/0x200 [ 8088.261164] ? set_kthread_struct+0x70/0x70 [ 8088.262780] ret_from_fork+0x1f/0x30 [ 8093.155202] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5456s! [ 8093.159531] Showing busy workqueues and worker pools: [ 8093.162269] workqueue events: flags=0x0 [ 8093.165766] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8093.169569] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8093.179364] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8093.183105] pending: cpu_latency_work [ptlrpc] [ 8093.186548] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8093.189752] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8093.196311] workqueue events_long: flags=0x0 [ 8093.199215] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8093.207727] pending: lu_objects_flush [obdclass] [ 8093.209606] workqueue events_unbound: flags=0x2 [ 8093.213162] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8093.218511] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8093.222527] workqueue events_power_efficient: flags=0x80 [ 8093.225425] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8093.231005] pending: check_lifetime, do_cache_clean [ 8093.235774] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8093.239689] pending: neigh_periodic_work, neigh_periodic_work [ 8093.246733] workqueue mm_percpu_wq: flags=0x8 [ 8093.250161] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8093.257756] pending: vmstat_update, lru_add_drain_per_cpu [ 8093.260727] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8093.267470] pending: vmstat_update, lru_add_drain_per_cpu [ 8093.271080] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8093.277034] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8093.280285] workqueue xprtiod: flags=0xa [ 8093.289946] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8093.293051] pending: xprt_autoclose, xs_error_handle [ 8093.296695] workqueue ipv6_addrconf: flags=0x40008 [ 8093.300092] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8093.306128] pending: addrconf_verify_work [ 8093.308620] workqueue knbd0-recv: flags=0x1a [ 8093.311293] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8093.317438] in-flight: 651:recv_work [ 8093.319310] workqueue ptlrpc_pinger: flags=0x2 [ 8093.321220] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8093.325257] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8093.328163] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5456s workers=4 idle: 15979 31824 8 [ 8093.331945] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8083s workers=2 idle: 654 [ 8094.583421] rcu: INFO: rcu_sched self-detected stall on CPU [ 8094.586546] rcu: 2-....: (3250408 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 8094.595488] (t=6000157 jiffies g=274185 q=65304) [ 8094.597195] rcu: rcu_sched kthread starved for 5458523 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 8094.611347] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8094.617636] rcu: RCU grace-period kthread stack dump: [ 8094.624330] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8094.631871] Call Trace: [ 8094.633048] __schedule+0x351/0xcb0 [ 8094.639677] schedule+0xc0/0x180 [ 8094.641552] schedule_timeout+0xb4/0x190 [ 8094.650787] ? do_raw_spin_unlock+0x75/0x190 [ 8094.654993] ? __next_timer_interrupt+0x160/0x160 [ 8094.660584] ? prepare_to_swait_event+0xf3/0x140 [ 8094.664798] rcu_gp_kthread+0x8dd/0x15b0 [ 8094.669021] ? force_qs_rnp+0x2c0/0x2c0 [ 8094.674947] kthread+0x1d1/0x200 [ 8094.678729] ? set_kthread_struct+0x70/0x70 [ 8094.682257] ret_from_fork+0x1f/0x30 [ 8094.683458] rcu: Stack dump where RCU GP kthread last ran: [ 8094.688756] NMI backtrace for cpu 2 [ 8094.692866] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8094.701460] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8094.710567] Call Trace: [ 8094.711352] [ 8094.712910] ? dump_stack+0xbb/0x10e [ 8094.715303] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8094.718636] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8094.726718] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8094.730655] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8094.734258] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8094.736559] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 8094.743095] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 8094.747376] ? update_process_times+0x6d/0xc0 [ 8094.750603] ? tick_sched_handle.isra.18+0x27/0x90 [ 8094.753283] ? tick_sched_timer+0x53/0xb0 [ 8094.762902] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8094.766637] ? tick_sched_handle.isra.18+0x90/0x90 [ 8094.771227] ? hrtimer_interrupt+0x128/0x2c0 [ 8094.774984] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8094.779263] ? apic_timer_interrupt+0xf/0x20 [ 8094.783265] [ 8094.784642] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8094.787954] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8094.795764] ? do_raw_spin_lock+0x177/0x190 [ 8094.798303] ? _raw_spin_lock+0x21/0x30 [ 8094.801251] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8094.804237] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8094.807193] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8094.811986] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8094.815251] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8094.827026] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8094.839988] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8094.842690] ? __next_timer_interrupt+0x160/0x160 [ 8094.847327] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8094.852195] ? do_wait_intr+0xf0/0xf0 [ 8094.857988] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8094.859879] ? kthread+0x1d1/0x200 [ 8094.863635] ? set_kthread_struct+0x70/0x70 [ 8094.868846] ? ret_from_fork+0x1f/0x30 [ 8094.871713] Sending NMI from CPU 2 to CPUs 1: [ 8094.899263] NMI backtrace for cpu 1 [ 8094.899270] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8094.899272] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8094.899274] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8094.899276] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8094.899278] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [ 8094.899282] RAX: 0000000000003e94 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8094.899284] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8094.899285] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8094.899286] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8094.899288] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8094.899289] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8094.899290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8094.899292] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8094.899293] Call Trace: [ 8094.899294] [ 8094.899295] ? show_regs.cold.9+0x22/0x2f [ 8094.899296] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8094.899297] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8094.899298] ? nmi_handle+0xbd/0x210 [ 8094.899299] ? default_do_nmi+0x4b/0x1b0 [ 8094.899300] ? do_nmi+0x280/0x380 [ 8094.899302] ? end_repeat_nmi+0x16/0x1f [ 8094.899303] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8094.899304] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8094.899305] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8094.899306] [ 8094.899307] do_raw_spin_lock+0x177/0x190 [ 8094.899308] _raw_spin_lock+0x21/0x30 [ 8094.899309] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8094.899310] ? do_raw_spin_unlock+0x75/0x190 [ 8094.899311] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8094.899313] ? do_wait_intr+0xf0/0xf0 [ 8094.899314] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8094.899315] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8094.899316] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8094.899317] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8094.899318] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8094.899319] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8094.899320] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8094.899321] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8094.899323] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8094.899324] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8094.899325] lmv_intent_open+0x338/0x1150 [lmv] [ 8094.899326] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8094.899327] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8094.899328] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8094.899329] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8094.899330] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8094.899331] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8094.899333] ? do_raw_spin_unlock+0x75/0x190 [ 8094.899334] ? _raw_spin_unlock+0x12/0x30 [ 8094.899335] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8094.899336] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8094.899337] ll_atomic_open+0x36a/0xd70 [lustre] [ 8094.899338] lookup_open+0x356/0xb40 [ 8094.899339] do_last+0x313/0xfc0 [ 8094.899340] ? nd_jump_root+0xe5/0x160 [ 8094.899341] ? path_init+0x437/0x520 [ 8094.899342] path_openat+0xf7/0x500 [ 8094.899343] ? do_raw_spin_unlock+0x75/0x190 [ 8094.899344] do_filp_open+0x99/0x140 [ 8094.899345] ? getname_flags+0x6e/0x330 [ 8094.899346] ? __check_object_size+0xff/0x256 [ 8094.899347] ? do_raw_spin_unlock+0x75/0x190 [ 8094.899348] ? _raw_spin_unlock+0x12/0x30 [ 8094.899349] do_sys_openat2+0x2b4/0x410 [ 8094.899350] do_sys_open+0x73/0xa0 [ 8094.899351] __x64_sys_openat+0x24/0x30 [ 8094.899352] do_syscall_64+0xc1/0x440 [ 8094.899353] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8094.899354] RIP: 0033:0x7f53425a5332 [ 8094.899356] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8094.899358] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8094.899360] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8094.899361] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8094.899363] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8094.899364] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8094.899365] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8094.902161] NMI backtrace for cpu 2 [ 8095.348643] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8095.375490] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8095.385333] Call Trace: [ 8095.387613] [ 8095.389215] ? dump_stack+0xbb/0x10e [ 8095.394017] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8095.401077] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8095.407127] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8095.414880] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8095.427005] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8095.430539] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8095.431699] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8095.435698] ? update_process_times+0x6d/0xc0 [ 8095.443025] ? tick_sched_handle.isra.18+0x27/0x90 [ 8095.446982] ? tick_sched_timer+0x53/0xb0 [ 8095.449945] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8095.463774] ? tick_sched_handle.isra.18+0x90/0x90 [ 8095.468054] ? hrtimer_interrupt+0x128/0x2c0 [ 8095.472613] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8095.475893] ? apic_timer_interrupt+0xf/0x20 [ 8095.477388] [ 8095.481486] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8095.487984] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8095.491998] ? do_raw_spin_lock+0x177/0x190 [ 8095.495694] ? _raw_spin_lock+0x21/0x30 [ 8095.498323] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8095.508487] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8095.519695] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8095.521928] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8095.530316] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8095.539692] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8095.547444] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8095.553238] ? __next_timer_interrupt+0x160/0x160 [ 8095.556292] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8095.565149] ? do_wait_intr+0xf0/0xf0 [ 8095.567560] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8095.575208] ? kthread+0x1d1/0x200 [ 8095.577452] ? set_kthread_struct+0x70/0x70 [ 8095.579752] ? ret_from_fork+0x1f/0x30 [ 8112.063011] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 8112.070938] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8112.105131] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8112.114657] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8112.122897] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8112.127892] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8112.136862] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8112.140804] RAX: 00000000000065a3 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8112.145199] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8112.150209] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8112.154796] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8112.159674] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8112.164310] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8112.168900] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8112.173831] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8112.177404] Call Trace: [ 8112.178877] [ 8112.181164] ? show_regs.cold.9+0x22/0x2f [ 8112.183194] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8112.186193] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8112.188796] ? watchdog+0x40/0x40 [ 8112.190438] ? hrtimer_interrupt+0x128/0x2c0 [ 8112.193268] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8112.195834] ? apic_timer_interrupt+0xf/0x20 [ 8112.198945] [ 8112.199681] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8112.201864] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8112.205629] do_raw_spin_lock+0x177/0x190 [ 8112.208874] _raw_spin_lock+0x21/0x30 [ 8112.210715] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8112.215687] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8112.219338] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8112.222769] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8112.226734] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8112.229031] ? woken_wake_function+0x30/0x30 [ 8112.231384] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8112.237687] kthread+0x1d1/0x200 [ 8112.241243] ? set_kthread_struct+0x70/0x70 [ 8112.244687] ret_from_fork+0x1f/0x30 [ 8120.054479] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8120.058341] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 8120.058388] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 8120.058395] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8120.058463] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8120.058468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8120.058471] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8120.058482] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8120.058486] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8120.058490] RAX: 0000000000007e94 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8120.058492] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8120.058494] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8120.058497] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8120.058499] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8120.058502] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8120.058505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8120.058507] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8120.058515] Call Trace: [ 8120.058519] [ 8120.058524] ? show_regs.cold.9+0x22/0x2f [ 8120.058532] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8120.058538] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8120.058542] ? watchdog+0x40/0x40 [ 8120.058546] ? hrtimer_interrupt+0x128/0x2c0 [ 8120.058550] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8120.058558] ? apic_timer_interrupt+0xf/0x20 [ 8120.058563] [ 8120.058566] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8120.058569] Modules linked in: lustre(O) [ 8120.058574] do_raw_spin_lock+0x177/0x190 [ 8120.058578] osp(O) ofd(O) lod(O) mdt(O) [ 8120.058585] _raw_spin_lock+0x21/0x30 [ 8120.058590] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8120.058649] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8120.058654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8120.058659] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8120.058928] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8120.058934] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8120.058938] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8120.058942] RAX: 00000000000045d7 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8120.058945] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8120.058947] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8120.058949] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8120.058952] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8120.058955] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8120.058958] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8120.058960] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8120.058968] Call Trace: [ 8120.058970] [ 8120.058974] ? show_regs.cold.9+0x22/0x2f [ 8120.058981] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8120.058987] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8120.058990] ? watchdog+0x40/0x40 [ 8120.058995] ? hrtimer_interrupt+0x128/0x2c0 [ 8120.058999] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8120.059005] ? apic_timer_interrupt+0xf/0x20 [ 8120.059009] [ 8120.059012] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8120.059017] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8120.059021] ? newidle_balance.isra.93+0x207/0x580 [ 8120.059027] do_raw_spin_lock+0x177/0x190 [ 8120.059032] _raw_spin_lock+0x21/0x30 [ 8120.059037] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8120.059262] process_one_work+0x2c8/0x700 [ 8120.059270] worker_thread+0x296/0x6e0 [ 8120.059275] ? rescuer_thread+0x570/0x570 [ 8120.059278] kthread+0x1d1/0x200 [ 8120.059283] ? set_kthread_struct+0x70/0x70 [ 8120.059287] ret_from_fork+0x1f/0x30 [ 8120.058597] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8120.076936] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8120.077028] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8120.077036] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8120.077039] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8120.077054] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8120.077058] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8120.077063] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8120.077066] RDX: 0000000000007f8c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8120.077069] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8120.077072] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8120.077074] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8120.077078] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8120.077081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8120.077084] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8120.077093] Call Trace: [ 8120.077099] [ 8120.077104] ? show_regs.cold.9+0x22/0x2f [ 8120.077113] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8120.077119] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8120.077124] ? watchdog+0x40/0x40 [ 8120.077130] ? hrtimer_interrupt+0x128/0x2c0 [ 8120.130032] ? do_raw_spin_unlock+0x75/0x190 [ 8120.130055] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8120.131655] ? do_wait_intr+0xf0/0xf0 [ 8120.131680] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8120.132003] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8120.143461] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8120.143483] ? apic_timer_interrupt+0xf/0x20 [ 8120.143488] [ 8120.143493] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8120.143500] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8120.143505] do_raw_spin_lock+0x177/0x190 [ 8120.143512] _raw_spin_lock+0x21/0x30 [ 8120.143520] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8120.152361] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8120.153535] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8120.159064] obd_mod_free+0x46/0x140 [ptlrpc] [ 8120.169732] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8120.169966] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8120.169999] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8120.170032] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8120.170148] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8120.170499] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8120.170539] lmv_intent_open+0x338/0x1150 [lmv] [ 8120.170568] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8120.170652] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8120.170731] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8120.178532] mdc_commit_open+0x91/0xb0 [mdc] [ 8120.178584] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8120.178976] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8120.179189] after_reply+0xe2c/0x1780 [ptlrpc] [ 8120.179514] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8120.179799] ? __next_timer_interrupt+0x160/0x160 [ 8120.179810] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8120.180030] ? do_wait_intr+0xf0/0xf0 [ 8120.180040] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8120.180258] kthread+0x1d1/0x200 [ 8120.180264] ? set_kthread_struct+0x70/0x70 [ 8120.180269] ret_from_fork+0x1f/0x30 [ 8120.681725] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8120.685718] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8120.700002] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8120.701717] ? do_raw_spin_unlock+0x75/0x190 [ 8120.708033] ? _raw_spin_unlock+0x12/0x30 [ 8120.711114] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8120.714115] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8120.717725] ll_atomic_open+0x36a/0xd70 [lustre] [ 8120.719997] lookup_open+0x356/0xb40 [ 8120.723257] do_last+0x313/0xfc0 [ 8120.726375] ? nd_jump_root+0xe5/0x160 [ 8120.727702] ? path_init+0x437/0x520 [ 8120.735523] path_openat+0xf7/0x500 [ 8120.738570] ? do_raw_spin_unlock+0x75/0x190 [ 8120.745713] do_filp_open+0x99/0x140 [ 8120.748485] ? getname_flags+0x6e/0x330 [ 8120.752824] ? __check_object_size+0xff/0x256 [ 8120.755591] ? do_raw_spin_unlock+0x75/0x190 [ 8120.758590] ? _raw_spin_unlock+0x12/0x30 [ 8120.760938] do_sys_openat2+0x2b4/0x410 [ 8120.763491] do_sys_open+0x73/0xa0 [ 8120.764630] __x64_sys_openat+0x24/0x30 [ 8120.766792] do_syscall_64+0xc1/0x440 [ 8120.769642] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8120.772174] RIP: 0033:0x7f53425a5332 [ 8120.773530] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8120.784953] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8120.791183] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8120.797095] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8120.800577] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8120.803696] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8120.812255] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8123.360150] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5486s! [ 8123.363500] Showing busy workqueues and worker pools: [ 8123.365396] workqueue events: flags=0x0 [ 8123.368595] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8123.372260] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8123.376937] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8123.380014] pending: cpu_latency_work [ptlrpc] [ 8123.382247] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8123.385165] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8123.389986] workqueue events_long: flags=0x0 [ 8123.391575] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8123.394215] pending: lu_objects_flush [obdclass] [ 8123.396575] workqueue events_unbound: flags=0x2 [ 8123.398207] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8123.400735] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8123.403365] workqueue events_power_efficient: flags=0x80 [ 8123.405136] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8123.407435] pending: check_lifetime, do_cache_clean [ 8123.409139] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8123.411715] pending: neigh_periodic_work, neigh_periodic_work [ 8123.414197] workqueue mm_percpu_wq: flags=0x8 [ 8123.416011] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8123.419143] pending: vmstat_update, lru_add_drain_per_cpu [ 8123.421619] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8123.424485] pending: vmstat_update, lru_add_drain_per_cpu [ 8123.427171] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8123.430034] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8123.432716] workqueue xprtiod: flags=0xa [ 8123.434476] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8123.437655] pending: xprt_autoclose, xs_error_handle [ 8123.440160] workqueue ipv6_addrconf: flags=0x40008 [ 8123.441740] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8123.444618] pending: addrconf_verify_work [ 8123.446557] workqueue knbd0-recv: flags=0x1a [ 8123.449528] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8123.457056] in-flight: 651:recv_work [ 8123.458547] workqueue ptlrpc_pinger: flags=0x2 [ 8123.466048] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8123.471500] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8123.473964] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5486s workers=4 idle: 15979 31824 8 [ 8123.480648] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8113s workers=2 idle: 654 [ 8140.066517] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 8140.074934] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8140.118927] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8140.135912] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8140.154998] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8140.159522] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8140.179626] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8140.186778] RAX: 0000000000003b90 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8140.197402] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8140.205101] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8140.216674] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8140.219001] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8140.228868] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8140.243749] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8140.249928] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8140.260046] Call Trace: [ 8140.263631] [ 8140.265440] ? show_regs.cold.9+0x22/0x2f [ 8140.269886] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8140.276661] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8140.288113] ? watchdog+0x40/0x40 [ 8140.292960] ? hrtimer_interrupt+0x128/0x2c0 [ 8140.299513] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8140.311240] ? apic_timer_interrupt+0xf/0x20 [ 8140.312574] [ 8140.316224] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8140.320183] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8140.326812] do_raw_spin_lock+0x177/0x190 [ 8140.331713] _raw_spin_lock+0x21/0x30 [ 8140.344258] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8140.346147] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8140.360713] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8140.377480] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8140.379506] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8140.390650] ? woken_wake_function+0x30/0x30 [ 8140.393767] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8140.407376] kthread+0x1d1/0x200 [ 8140.408344] ? set_kthread_struct+0x70/0x70 [ 8140.420435] ret_from_fork+0x1f/0x30 [ 8148.052023] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8148.055032] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 8148.055419] Modules linked in: [ 8148.058696] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 8148.058709] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8148.058792] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8148.058798] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8148.058801] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8148.058813] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8148.058818] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8148.058822] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8148.058824] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8148.058827] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8148.058829] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8148.058831] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8148.058834] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8148.058846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8148.058848] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8148.058857] Call Trace: [ 8148.058867] [ 8148.058872] ? show_regs.cold.9+0x22/0x2f [ 8148.058882] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8148.058888] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8148.058893] ? watchdog+0x40/0x40 [ 8148.058896] ? hrtimer_interrupt+0x128/0x2c0 [ 8148.058901] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8148.058908] ? apic_timer_interrupt+0xf/0x20 [ 8148.058912] [ 8148.058915] ? native_safe_halt+0x2e/0x50 [ 8148.058919] kvm_wait+0xcd/0xe0 [ 8148.058926] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8148.058934] do_raw_spin_lock+0x177/0x190 [ 8148.058938] _raw_spin_lock+0x21/0x30 [ 8148.058943] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8148.059231] obd_mod_free+0x46/0x140 [ptlrpc] [ 8148.059424] mdc_commit_open+0x91/0xb0 [mdc] [ 8148.059546] Modules linked in: lustre(O) osp(O) [ 8148.059459] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8148.060235] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8148.061002] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror [ 8148.060428] after_reply+0xe2c/0x1780 [ptlrpc] [ 8148.062001] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8148.062204] ? __next_timer_interrupt+0x160/0x160 [ 8148.062213] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8148.062458] ? do_wait_intr+0xf0/0xf0 [ 8148.062470] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8148.062685] kthread+0x1d1/0x200 [ 8148.062690] ? set_kthread_struct+0x70/0x70 [ 8148.062695] ret_from_fork+0x1f/0x30 [ 8148.063172] ofd(O) lod(O) [ 8148.081774] dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8148.081794] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8148.081802] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8148.081809] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8148.086488] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata [ 8148.090143] [ 8148.090152] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8148.090168] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8148.090172] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8148.090176] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8148.090178] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8148.090181] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8148.090183] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8148.090185] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8148.090189] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8148.090191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8148.090193] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8148.090201] Call Trace: [ 8148.090207] [ 8148.090212] ? show_regs.cold.9+0x22/0x2f [ 8148.090224] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8148.090230] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8148.090234] ? watchdog+0x40/0x40 [ 8148.090239] ? hrtimer_interrupt+0x128/0x2c0 [ 8148.090243] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8148.090249] ? apic_timer_interrupt+0xf/0x20 [ 8148.090254] [ 8148.090256] ? native_safe_halt+0x2e/0x50 [ 8148.090261] kvm_wait+0xcd/0xe0 [ 8148.090270] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8148.101423] dm_mirror [ 8148.109792] ? newidle_balance.isra.93+0x207/0x580 [ 8148.109817] do_raw_spin_lock+0x177/0x190 [ 8148.109824] _raw_spin_lock+0x21/0x30 [ 8148.109830] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8148.118349] dm_region_hash [ 8148.121988] process_one_work+0x2c8/0x700 [ 8148.129200] dm_log [ 8148.132362] worker_thread+0x296/0x6e0 [ 8148.143320] dm_mod [ 8148.146389] ? rescuer_thread+0x570/0x570 [ 8148.146411] kthread+0x1d1/0x200 [ 8148.146416] ? set_kthread_struct+0x70/0x70 [ 8148.146421] ret_from_fork+0x1f/0x30 [ 8148.357500] sha512_ssse3 sha512_generic [ 8148.358598] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8148.363345] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8148.366799] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8148.369290] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8148.378613] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8148.382327] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8148.386979] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8148.390321] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8148.393636] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8148.396812] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8148.400120] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8148.404547] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8148.406297] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8148.410332] Call Trace: [ 8148.413226] [ 8148.414147] ? show_regs.cold.9+0x22/0x2f [ 8148.416127] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8148.419377] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8148.421328] ? watchdog+0x40/0x40 [ 8148.422624] ? hrtimer_interrupt+0x128/0x2c0 [ 8148.425403] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8148.427302] ? apic_timer_interrupt+0xf/0x20 [ 8148.428779] [ 8148.429486] ? native_safe_halt+0x2e/0x50 [ 8148.430659] kvm_wait+0xcd/0xe0 [ 8148.431787] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8148.433688] do_raw_spin_lock+0x177/0x190 [ 8148.435557] _raw_spin_lock+0x21/0x30 [ 8148.437287] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8148.439608] ? do_raw_spin_unlock+0x75/0x190 [ 8148.441281] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8148.444287] ? do_wait_intr+0xf0/0xf0 [ 8148.445622] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8148.449722] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8148.453731] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8148.459667] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8148.463653] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8148.466689] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8148.470232] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8148.475026] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8148.477432] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8148.481150] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8148.484669] lmv_intent_open+0x338/0x1150 [lmv] [ 8148.488388] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8148.491565] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8148.494343] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8148.496779] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8148.499272] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8148.500600] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8148.501737] ? do_raw_spin_unlock+0x75/0x190 [ 8148.505127] ? _raw_spin_unlock+0x12/0x30 [ 8148.507762] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8148.510771] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8148.513254] ll_atomic_open+0x36a/0xd70 [lustre] [ 8148.515548] lookup_open+0x356/0xb40 [ 8148.517536] do_last+0x313/0xfc0 [ 8148.518770] ? nd_jump_root+0xe5/0x160 [ 8148.521430] ? path_init+0x437/0x520 [ 8148.524356] path_openat+0xf7/0x500 [ 8148.525697] ? do_raw_spin_unlock+0x75/0x190 [ 8148.527756] do_filp_open+0x99/0x140 [ 8148.531697] ? getname_flags+0x6e/0x330 [ 8148.533678] ? __check_object_size+0xff/0x256 [ 8148.536377] ? do_raw_spin_unlock+0x75/0x190 [ 8148.538118] ? _raw_spin_unlock+0x12/0x30 [ 8148.539147] do_sys_openat2+0x2b4/0x410 [ 8148.541316] do_sys_open+0x73/0xa0 [ 8148.543206] __x64_sys_openat+0x24/0x30 [ 8148.545209] do_syscall_64+0xc1/0x440 [ 8148.546238] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8148.547539] RIP: 0033:0x7f53425a5332 [ 8148.548662] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8148.556599] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8148.559237] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8148.561732] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8148.565271] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8148.567939] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8148.570448] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8153.573336] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5516s! [ 8153.584383] Showing busy workqueues and worker pools: [ 8153.592130] workqueue events: flags=0x0 [ 8153.599021] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8153.617311] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8153.632053] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8153.639354] pending: cpu_latency_work [ptlrpc] [ 8153.646286] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8153.651226] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8153.665457] workqueue events_long: flags=0x0 [ 8153.671058] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8153.682307] pending: lu_objects_flush [obdclass] [ 8153.690094] workqueue events_unbound: flags=0x2 [ 8153.694907] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8153.701213] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8153.711013] workqueue events_power_efficient: flags=0x80 [ 8153.718300] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8153.725440] pending: check_lifetime, do_cache_clean [ 8153.732559] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8153.738304] pending: neigh_periodic_work, neigh_periodic_work [ 8153.745489] workqueue mm_percpu_wq: flags=0x8 [ 8153.748013] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8153.757049] pending: vmstat_update, lru_add_drain_per_cpu [ 8153.766223] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8153.773377] pending: vmstat_update, lru_add_drain_per_cpu [ 8153.775366] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8153.780982] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8153.789693] workqueue xprtiod: flags=0xa [ 8153.793082] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8153.797769] pending: xprt_autoclose, xs_error_handle [ 8153.803968] workqueue ipv6_addrconf: flags=0x40008 [ 8153.816273] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8153.826434] pending: addrconf_verify_work [ 8153.829214] workqueue knbd0-recv: flags=0x1a [ 8153.832283] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8153.838135] in-flight: 651:recv_work [ 8153.844119] workqueue ptlrpc_pinger: flags=0x2 [ 8153.850265] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8153.857771] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8153.867286] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5516s workers=4 idle: 15979 31824 8 [ 8153.878728] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8144s workers=2 idle: 654 [ 8168.064182] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 8168.072909] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8168.126685] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8168.137204] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8168.149971] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8168.159839] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8168.214301] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8168.223362] RAX: 0000000000007f2b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8168.225302] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8168.232815] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8168.241894] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8168.243794] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8168.252106] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8168.258142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8168.264808] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8168.267322] Call Trace: [ 8168.269809] [ 8168.270360] ? show_regs.cold.9+0x22/0x2f [ 8168.277057] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8168.283237] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8168.288716] ? watchdog+0x40/0x40 [ 8168.292142] ? hrtimer_interrupt+0x128/0x2c0 [ 8168.302047] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8168.303361] ? apic_timer_interrupt+0xf/0x20 [ 8168.318405] [ 8168.319120] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8168.327685] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8168.331779] do_raw_spin_lock+0x177/0x190 [ 8168.337939] _raw_spin_lock+0x21/0x30 [ 8168.341028] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8168.344221] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8168.356434] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8168.365351] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8168.375845] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8168.378719] ? woken_wake_function+0x30/0x30 [ 8168.380033] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8168.387959] kthread+0x1d1/0x200 [ 8168.388817] ? set_kthread_struct+0x70/0x70 [ 8168.398735] ret_from_fork+0x1f/0x30 [ 8176.056299] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 8176.058152] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 8176.061984] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8176.062058] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8176.062065] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8176.062067] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8176.062080] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8176.062084] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8176.062088] RAX: 0000000000007e8e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8176.062091] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8176.062093] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8176.062095] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8176.062097] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8176.062101] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8176.062103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8176.062105] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8176.062114] Call Trace: [ 8176.062121] [ 8176.062126] ? show_regs.cold.9+0x22/0x2f [ 8176.062135] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8176.062141] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8176.062146] ? watchdog+0x40/0x40 [ 8176.062151] ? hrtimer_interrupt+0x128/0x2c0 [ 8176.062155] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8176.062163] ? apic_timer_interrupt+0xf/0x20 [ 8176.062168] [ 8176.062170] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8176.062177] do_raw_spin_lock+0x177/0x190 [ 8176.062182] _raw_spin_lock+0x21/0x30 [ 8176.062187] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8176.064425] ? do_raw_spin_unlock+0x75/0x190 [ 8176.064434] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8176.064675] ? do_wait_intr+0xf0/0xf0 [ 8176.064685] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8176.075743] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8176.075827] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8176.075834] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8176.075837] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8176.075852] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8176.075856] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8176.075861] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8176.075864] RDX: 0000000000006ea7 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8176.075866] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8176.075869] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8176.075871] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8176.075875] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8176.075879] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8176.075881] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8176.075891] Call Trace: [ 8176.075897] [ 8176.075903] ? show_regs.cold.9+0x22/0x2f [ 8176.075925] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8176.075932] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8176.075937] ? watchdog+0x40/0x40 [ 8176.075942] ? hrtimer_interrupt+0x128/0x2c0 [ 8176.075946] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8176.075955] ? apic_timer_interrupt+0xf/0x20 [ 8176.075960] [ 8176.075963] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8176.075969] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8176.077426] do_raw_spin_lock+0x177/0x190 [ 8176.077445] _raw_spin_lock+0x21/0x30 [ 8176.077453] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8176.077734] obd_mod_free+0x46/0x140 [ptlrpc] [ 8176.077941] mdc_commit_open+0x91/0xb0 [mdc] [ 8176.079715] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8176.079983] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8176.080200] after_reply+0xe2c/0x1780 [ptlrpc] [ 8176.080655] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8176.080871] ? __next_timer_interrupt+0x160/0x160 [ 8176.080882] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8176.082302] ? do_wait_intr+0xf0/0xf0 [ 8176.082314] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8176.082590] kthread+0x1d1/0x200 [ 8176.082595] ? set_kthread_struct+0x70/0x70 [ 8176.082599] ret_from_fork+0x1f/0x30 [ 8176.583428] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8176.591560] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8176.603948] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8176.611167] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8176.620694] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8176.623813] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8176.628227] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8176.634573] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8176.645081] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8176.652495] lmv_intent_open+0x338/0x1150 [lmv] [ 8176.657299] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8176.660484] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8176.667009] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8176.677576] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8176.679060] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8176.683390] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8176.686140] ? do_raw_spin_unlock+0x75/0x190 [ 8176.691665] ? _raw_spin_unlock+0x12/0x30 [ 8176.693891] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8176.705148] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8176.712249] ll_atomic_open+0x36a/0xd70 [lustre] [ 8176.715851] lookup_open+0x356/0xb40 [ 8176.717493] do_last+0x313/0xfc0 [ 8176.721296] ? nd_jump_root+0xe5/0x160 [ 8176.723032] ? path_init+0x437/0x520 [ 8176.725539] path_openat+0xf7/0x500 [ 8176.729729] ? do_raw_spin_unlock+0x75/0x190 [ 8176.740263] do_filp_open+0x99/0x140 [ 8176.743866] ? getname_flags+0x6e/0x330 [ 8176.747293] ? __check_object_size+0xff/0x256 [ 8176.748806] ? do_raw_spin_unlock+0x75/0x190 [ 8176.753674] ? _raw_spin_unlock+0x12/0x30 [ 8176.755921] do_sys_openat2+0x2b4/0x410 [ 8176.757872] do_sys_open+0x73/0xa0 [ 8176.759787] __x64_sys_openat+0x24/0x30 [ 8176.767465] do_syscall_64+0xc1/0x440 [ 8176.769518] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8176.774757] RIP: 0033:0x7f53425a5332 [ 8176.778906] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8176.797924] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8176.802274] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8176.805761] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8176.822055] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8176.826805] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8176.829841] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8180.052915] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 8180.059400] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8180.092181] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8180.102134] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8180.109119] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8180.116217] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8180.122314] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8180.151056] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8180.156942] RAX: 0000000000006885 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8180.162222] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8180.174872] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8180.188995] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8180.192954] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8180.199985] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8180.206027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8180.213152] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8180.223217] Call Trace: [ 8180.224135] [ 8180.224874] ? show_regs.cold.9+0x22/0x2f [ 8180.226153] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8180.239676] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8180.249032] ? watchdog+0x40/0x40 [ 8180.251927] ? hrtimer_interrupt+0x128/0x2c0 [ 8180.259990] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8180.265694] ? apic_timer_interrupt+0xf/0x20 [ 8180.267260] [ 8180.273612] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8180.281601] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8180.284773] ? newidle_balance.isra.93+0x207/0x580 [ 8180.287983] do_raw_spin_lock+0x177/0x190 [ 8180.291237] _raw_spin_lock+0x21/0x30 [ 8180.296356] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8180.302555] process_one_work+0x2c8/0x700 [ 8180.306364] worker_thread+0x296/0x6e0 [ 8180.308535] ? rescuer_thread+0x570/0x570 [ 8180.312195] kthread+0x1d1/0x200 [ 8180.315656] ? set_kthread_struct+0x70/0x70 [ 8180.318505] ret_from_fork+0x1f/0x30 [ 8184.288408] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5547s! [ 8184.299946] Showing busy workqueues and worker pools: [ 8184.306323] workqueue events: flags=0x0 [ 8184.307737] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8184.320395] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8184.339695] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8184.347170] pending: cpu_latency_work [ptlrpc] [ 8184.351396] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8184.355182] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8184.364675] workqueue events_long: flags=0x0 [ 8184.368612] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8184.376193] pending: lu_objects_flush [obdclass] [ 8184.381075] workqueue events_unbound: flags=0x2 [ 8184.384069] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8184.386960] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8184.390112] workqueue events_power_efficient: flags=0x80 [ 8184.397433] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8184.399962] pending: check_lifetime, do_cache_clean [ 8184.407032] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8184.418463] pending: neigh_periodic_work, neigh_periodic_work [ 8184.429148] workqueue mm_percpu_wq: flags=0x8 [ 8184.435529] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8184.440940] pending: vmstat_update, lru_add_drain_per_cpu [ 8184.444146] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8184.448175] pending: vmstat_update, lru_add_drain_per_cpu [ 8184.454363] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8184.461111] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8184.466110] workqueue xprtiod: flags=0xa [ 8184.468310] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8184.472174] pending: xprt_autoclose, xs_error_handle [ 8184.474895] workqueue ipv6_addrconf: flags=0x40008 [ 8184.481124] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8184.485042] pending: addrconf_verify_work [ 8184.487766] workqueue knbd0-recv: flags=0x1a [ 8184.491815] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8184.497291] in-flight: 651:recv_work [ 8184.502291] workqueue ptlrpc_pinger: flags=0x2 [ 8184.506313] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8184.513157] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8184.519135] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5547s workers=4 idle: 15979 31824 8 [ 8184.528476] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8174s workers=2 idle: 654 [ 8196.058394] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 8196.065832] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8196.107739] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8196.115481] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8196.120381] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8196.121905] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8196.141853] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8196.149936] RAX: 0000000000005a7b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8196.155446] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8196.159721] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8196.166742] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8196.173873] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8196.181230] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8196.186977] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8196.191739] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8196.197279] Call Trace: [ 8196.201405] [ 8196.202273] ? show_regs.cold.9+0x22/0x2f [ 8196.204176] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8196.209017] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8196.210664] ? watchdog+0x40/0x40 [ 8196.212191] ? hrtimer_interrupt+0x128/0x2c0 [ 8196.213979] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8196.223804] ? apic_timer_interrupt+0xf/0x20 [ 8196.225360] [ 8196.234337] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8196.243047] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8196.248020] do_raw_spin_lock+0x177/0x190 [ 8196.249557] _raw_spin_lock+0x21/0x30 [ 8196.252537] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8196.257122] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8196.260530] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8196.267474] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8196.272134] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8196.275314] ? woken_wake_function+0x30/0x30 [ 8196.277289] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8196.280382] kthread+0x1d1/0x200 [ 8196.283688] ? set_kthread_struct+0x70/0x70 [ 8196.286140] ret_from_fork+0x1f/0x30 [ 8204.055014] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 8204.057759] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 8204.059792] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8204.059879] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8204.059887] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8204.059889] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8204.059905] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8204.059909] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8204.059914] RAX: 0000000000001096 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8204.059916] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8204.059918] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8204.059920] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8204.059923] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8204.059926] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8204.059930] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8204.059932] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8204.059940] Call Trace: [ 8204.059946] [ 8204.059952] ? show_regs.cold.9+0x22/0x2f [ 8204.059962] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8204.059968] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8204.059972] ? watchdog+0x40/0x40 [ 8204.059976] ? hrtimer_interrupt+0x128/0x2c0 [ 8204.059980] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8204.059988] ? apic_timer_interrupt+0xf/0x20 [ 8204.059992] [ 8204.059995] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8204.060002] do_raw_spin_lock+0x177/0x190 [ 8204.060007] _raw_spin_lock+0x21/0x30 [ 8204.060011] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8204.060888] ? do_raw_spin_unlock+0x75/0x190 [ 8204.060895] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8204.062732] ? do_wait_intr+0xf0/0xf0 [ 8204.062745] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8204.066499] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8204.066577] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8204.066584] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8204.066587] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8204.066602] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8204.066607] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8204.066612] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8204.066614] RDX: 0000000000001783 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8204.066617] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8204.066620] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8204.066623] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8204.066626] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8204.066629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8204.066632] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8204.066641] Call Trace: [ 8204.066648] [ 8204.066654] ? show_regs.cold.9+0x22/0x2f [ 8204.066665] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8204.066672] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8204.066677] ? watchdog+0x40/0x40 [ 8204.066681] ? hrtimer_interrupt+0x128/0x2c0 [ 8204.066686] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8204.066695] ? apic_timer_interrupt+0xf/0x20 [ 8204.066700] [ 8204.066704] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8204.066709] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8204.066714] do_raw_spin_lock+0x177/0x190 [ 8204.066719] _raw_spin_lock+0x21/0x30 [ 8204.066725] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8204.067255] obd_mod_free+0x46/0x140 [ptlrpc] [ 8204.067470] mdc_commit_open+0x91/0xb0 [mdc] [ 8204.067508] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8204.067725] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8204.068045] after_reply+0xe2c/0x1780 [ptlrpc] [ 8204.068251] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8204.068515] ? __next_timer_interrupt+0x160/0x160 [ 8204.068525] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8204.068749] ? do_wait_intr+0xf0/0xf0 [ 8204.068759] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8204.096832] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8204.097482] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8204.102273] kthread+0x1d1/0x200 [ 8204.102294] ? set_kthread_struct+0x70/0x70 [ 8204.102299] ret_from_fork+0x1f/0x30 [ 8204.332586] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8204.334841] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8204.336817] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8204.339401] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8204.344137] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8204.347621] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8204.350101] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8204.354751] lmv_intent_open+0x338/0x1150 [lmv] [ 8204.357310] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8204.358829] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8204.366059] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8204.370385] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8204.377749] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8204.386753] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8204.390731] ? do_raw_spin_unlock+0x75/0x190 [ 8204.392210] ? _raw_spin_unlock+0x12/0x30 [ 8204.400538] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8204.405717] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8204.410336] ll_atomic_open+0x36a/0xd70 [lustre] [ 8204.413061] lookup_open+0x356/0xb40 [ 8204.415525] do_last+0x313/0xfc0 [ 8204.416801] ? nd_jump_root+0xe5/0x160 [ 8204.419969] ? path_init+0x437/0x520 [ 8204.421758] path_openat+0xf7/0x500 [ 8204.426300] ? do_raw_spin_unlock+0x75/0x190 [ 8204.429281] do_filp_open+0x99/0x140 [ 8204.431608] ? getname_flags+0x6e/0x330 [ 8204.436148] ? __check_object_size+0xff/0x256 [ 8204.442093] ? do_raw_spin_unlock+0x75/0x190 [ 8204.448484] ? _raw_spin_unlock+0x12/0x30 [ 8204.452647] do_sys_openat2+0x2b4/0x410 [ 8204.456818] do_sys_open+0x73/0xa0 [ 8204.460831] __x64_sys_openat+0x24/0x30 [ 8204.468206] do_syscall_64+0xc1/0x440 [ 8204.474331] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8204.480428] RIP: 0033:0x7f53425a5332 [ 8204.482057] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8204.492777] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8204.495747] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8204.502403] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8204.508763] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8204.516175] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8204.519472] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8212.052071] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 8212.061100] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8212.099665] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8212.104695] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8212.110053] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8212.112624] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8212.118026] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8212.139776] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8212.144448] RAX: 0000000000005118 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8212.153626] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8212.159403] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8212.166303] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8212.170407] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8212.174129] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8212.177617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8212.180225] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8212.183411] Call Trace: [ 8212.184707] [ 8212.185261] ? show_regs.cold.9+0x22/0x2f [ 8212.186398] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8212.189187] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8212.191074] ? watchdog+0x40/0x40 [ 8212.192336] ? hrtimer_interrupt+0x128/0x2c0 [ 8212.194312] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8212.196081] ? apic_timer_interrupt+0xf/0x20 [ 8212.197666] [ 8212.198521] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8212.200543] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8212.202551] ? newidle_balance.isra.93+0x207/0x580 [ 8212.208198] do_raw_spin_lock+0x177/0x190 [ 8212.210087] _raw_spin_lock+0x21/0x30 [ 8212.214165] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8212.216297] process_one_work+0x2c8/0x700 [ 8212.217694] worker_thread+0x296/0x6e0 [ 8212.219643] ? rescuer_thread+0x570/0x570 [ 8212.221902] kthread+0x1d1/0x200 [ 8212.226573] ? set_kthread_struct+0x70/0x70 [ 8212.228070] ret_from_fork+0x1f/0x30 [ 8215.008022] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5577s! [ 8215.010586] Showing busy workqueues and worker pools: [ 8215.012117] workqueue events: flags=0x0 [ 8215.014226] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8215.016561] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8215.022826] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8215.027990] pending: cpu_latency_work [ptlrpc] [ 8215.035851] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8215.038932] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8215.042561] workqueue events_long: flags=0x0 [ 8215.043998] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8215.049295] pending: lu_objects_flush [obdclass] [ 8215.053146] workqueue events_unbound: flags=0x2 [ 8215.055380] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8215.058059] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8215.060396] workqueue events_power_efficient: flags=0x80 [ 8215.064853] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8215.067370] pending: check_lifetime, do_cache_clean [ 8215.072742] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8215.075114] pending: neigh_periodic_work, neigh_periodic_work [ 8215.077575] workqueue mm_percpu_wq: flags=0x8 [ 8215.087179] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8215.090457] pending: vmstat_update, lru_add_drain_per_cpu [ 8215.093622] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8215.097799] pending: vmstat_update, lru_add_drain_per_cpu [ 8215.103394] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8215.110213] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8215.112662] workqueue xprtiod: flags=0xa [ 8215.114412] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8215.116749] pending: xprt_autoclose, xs_error_handle [ 8215.123013] workqueue ipv6_addrconf: flags=0x40008 [ 8215.127115] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8215.132996] pending: addrconf_verify_work [ 8215.134853] workqueue knbd0-recv: flags=0x1a [ 8215.136777] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8215.140116] in-flight: 651:recv_work [ 8215.141749] workqueue ptlrpc_pinger: flags=0x2 [ 8215.143799] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8215.147549] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8215.151536] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5578s workers=4 idle: 15979 31824 8 [ 8215.158556] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8205s workers=2 idle: 654 [ 8224.058029] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 8224.063594] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8224.117292] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8224.132998] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8224.150820] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8224.152484] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8224.168881] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8224.182969] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8224.194422] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8224.204332] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8224.209850] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8224.216658] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8224.225732] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8224.236255] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8224.255991] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8224.262312] Call Trace: [ 8224.265489] [ 8224.266572] ? show_regs.cold.9+0x22/0x2f [ 8224.275386] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8224.281067] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8224.282485] ? watchdog+0x40/0x40 [ 8224.290219] ? hrtimer_interrupt+0x128/0x2c0 [ 8224.292382] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8224.295899] ? apic_timer_interrupt+0xf/0x20 [ 8224.297337] [ 8224.302486] ? native_safe_halt+0x2e/0x50 [ 8224.304405] kvm_wait+0xcd/0xe0 [ 8224.310785] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8224.314346] do_raw_spin_lock+0x177/0x190 [ 8224.320331] _raw_spin_lock+0x21/0x30 [ 8224.328318] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8224.334440] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8224.341404] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8224.344578] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8224.347166] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8224.352971] ? woken_wake_function+0x30/0x30 [ 8224.355199] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8224.365377] kthread+0x1d1/0x200 [ 8224.372157] ? set_kthread_struct+0x70/0x70 [ 8224.378057] ret_from_fork+0x1f/0x30 [ 8232.056248] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 8232.057653] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 8232.067164] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8232.067243] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8232.067250] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8232.067253] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8232.067267] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8232.067272] RSP: 0018:ffffbb5381503500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8232.067277] RAX: 0000000000007c8a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8232.067279] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8232.067282] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8232.067284] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8232.067287] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8232.067290] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8232.067293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8232.067295] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8232.067303] Call Trace: [ 8232.067309] [ 8232.067315] ? show_regs.cold.9+0x22/0x2f [ 8232.067324] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8232.067330] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8232.067334] ? watchdog+0x40/0x40 [ 8232.067338] ? hrtimer_interrupt+0x128/0x2c0 [ 8232.067342] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8232.067350] ? apic_timer_interrupt+0xf/0x20 [ 8232.067355] [ 8232.067358] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8232.067365] do_raw_spin_lock+0x177/0x190 [ 8232.069394] _raw_spin_lock+0x21/0x30 [ 8232.069402] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8232.078312] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8232.078409] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8232.078417] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8232.078420] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8232.078433] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8232.078437] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8232.078442] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8232.078444] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8232.078447] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8232.078449] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8232.078452] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8232.078455] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8232.078458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8232.078461] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8232.078469] Call Trace: [ 8232.078476] [ 8232.078482] ? show_regs.cold.9+0x22/0x2f [ 8232.078492] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8232.078499] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8232.078504] ? watchdog+0x40/0x40 [ 8232.078508] ? hrtimer_interrupt+0x128/0x2c0 [ 8232.078512] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8232.078519] ? apic_timer_interrupt+0xf/0x20 [ 8232.078524] [ 8232.078528] ? native_safe_halt+0x2e/0x50 [ 8232.078533] kvm_wait+0xcd/0xe0 [ 8232.078542] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8232.078549] do_raw_spin_lock+0x177/0x190 [ 8232.078553] _raw_spin_lock+0x21/0x30 [ 8232.078559] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8232.079284] obd_mod_free+0x46/0x140 [ptlrpc] [ 8232.079513] mdc_commit_open+0x91/0xb0 [mdc] [ 8232.079559] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8232.080336] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8232.121148] ? do_raw_spin_unlock+0x75/0x190 [ 8232.121169] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8232.123918] ? do_wait_intr+0xf0/0xf0 [ 8232.123930] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8232.124146] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8232.126164] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8232.128600] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8232.128710] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8232.128750] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8232.128800] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8232.130972] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8232.131084] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8232.133994] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8232.134035] lmv_intent_open+0x338/0x1150 [lmv] [ 8232.134065] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8232.134176] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8232.134274] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8232.134297] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8232.146566] after_reply+0xe2c/0x1780 [ptlrpc] [ 8232.147311] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8232.147578] ? __next_timer_interrupt+0x160/0x160 [ 8232.147588] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8232.147917] ? do_wait_intr+0xf0/0xf0 [ 8232.147927] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8232.148209] kthread+0x1d1/0x200 [ 8232.148217] ? set_kthread_struct+0x70/0x70 [ 8232.148221] ret_from_fork+0x1f/0x30 [ 8232.657323] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8232.661818] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8232.664010] ? do_raw_spin_unlock+0x75/0x190 [ 8232.668997] ? _raw_spin_unlock+0x12/0x30 [ 8232.672397] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8232.675777] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8232.681126] ll_atomic_open+0x36a/0xd70 [lustre] [ 8232.685307] lookup_open+0x356/0xb40 [ 8232.687423] do_last+0x313/0xfc0 [ 8232.689001] ? nd_jump_root+0xe5/0x160 [ 8232.694445] ? path_init+0x437/0x520 [ 8232.696708] path_openat+0xf7/0x500 [ 8232.698686] ? do_raw_spin_unlock+0x75/0x190 [ 8232.707683] do_filp_open+0x99/0x140 [ 8232.709490] ? getname_flags+0x6e/0x330 [ 8232.717431] ? __check_object_size+0xff/0x256 [ 8232.724680] ? do_raw_spin_unlock+0x75/0x190 [ 8232.732578] ? _raw_spin_unlock+0x12/0x30 [ 8232.736290] do_sys_openat2+0x2b4/0x410 [ 8232.742951] do_sys_open+0x73/0xa0 [ 8232.747087] __x64_sys_openat+0x24/0x30 [ 8232.750396] do_syscall_64+0xc1/0x440 [ 8232.753716] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8232.756120] RIP: 0033:0x7f53425a5332 [ 8232.765840] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8232.778601] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8232.788461] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8232.798034] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8232.806000] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8232.811952] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8232.821479] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8240.052791] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 8240.057193] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8240.083482] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8240.090598] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8240.094940] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8240.098553] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8240.101187] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8240.109349] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8240.118722] RAX: 000000000000387b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8240.128644] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8240.133511] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8240.139478] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8240.146954] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8240.151461] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8240.160055] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8240.165363] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8240.173512] Call Trace: [ 8240.176891] [ 8240.177783] ? show_regs.cold.9+0x22/0x2f [ 8240.181828] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8240.184488] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8240.187746] ? watchdog+0x40/0x40 [ 8240.188838] ? hrtimer_interrupt+0x128/0x2c0 [ 8240.194060] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8240.199510] ? apic_timer_interrupt+0xf/0x20 [ 8240.202719] [ 8240.204199] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8240.210804] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8240.216264] ? newidle_balance.isra.93+0x207/0x580 [ 8240.219216] do_raw_spin_lock+0x177/0x190 [ 8240.222181] _raw_spin_lock+0x21/0x30 [ 8240.227446] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8240.230603] process_one_work+0x2c8/0x700 [ 8240.234607] worker_thread+0x296/0x6e0 [ 8240.235850] ? rescuer_thread+0x570/0x570 [ 8240.239604] kthread+0x1d1/0x200 [ 8240.240696] ? set_kthread_struct+0x70/0x70 [ 8240.243809] ret_from_fork+0x1f/0x30 [ 8245.218327] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5608s! [ 8245.221466] Showing busy workqueues and worker pools: [ 8245.224808] workqueue events: flags=0x0 [ 8245.228630] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8245.240464] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8245.249972] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8245.255013] pending: cpu_latency_work [ptlrpc] [ 8245.265868] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8245.272870] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8245.278975] workqueue events_long: flags=0x0 [ 8245.283242] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8245.292244] pending: lu_objects_flush [obdclass] [ 8245.295796] workqueue events_unbound: flags=0x2 [ 8245.303944] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8245.309409] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8245.313474] workqueue events_power_efficient: flags=0x80 [ 8245.321326] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8245.323697] pending: check_lifetime, do_cache_clean [ 8245.326623] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8245.333128] pending: neigh_periodic_work, neigh_periodic_work [ 8245.338666] workqueue mm_percpu_wq: flags=0x8 [ 8245.343094] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8245.350432] pending: vmstat_update, lru_add_drain_per_cpu [ 8245.352580] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8245.356585] pending: vmstat_update, lru_add_drain_per_cpu [ 8245.364460] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8245.369078] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8245.374865] workqueue xprtiod: flags=0xa [ 8245.378399] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8245.383829] pending: xprt_autoclose, xs_error_handle [ 8245.389305] workqueue ipv6_addrconf: flags=0x40008 [ 8245.392931] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8245.398953] pending: addrconf_verify_work [ 8245.402580] workqueue knbd0-recv: flags=0x1a [ 8245.415381] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8245.421130] in-flight: 651:recv_work [ 8245.428524] workqueue ptlrpc_pinger: flags=0x2 [ 8245.431668] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8245.433847] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8245.446010] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5608s workers=4 idle: 15979 31824 8 [ 8245.451056] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8235s workers=2 idle: 654 [ 8252.061256] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 8252.064844] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8252.101231] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8252.105904] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8252.109209] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8252.121845] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8252.133973] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8252.139782] RAX: 00000000000021f5 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8252.148538] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8252.152943] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8252.162576] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8252.167545] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8252.172617] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8252.178614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8252.184003] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8252.192981] Call Trace: [ 8252.196731] [ 8252.198201] ? show_regs.cold.9+0x22/0x2f [ 8252.200124] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8252.202937] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8252.208696] ? watchdog+0x40/0x40 [ 8252.213716] ? hrtimer_interrupt+0x128/0x2c0 [ 8252.217278] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8252.220229] ? apic_timer_interrupt+0xf/0x20 [ 8252.223464] [ 8252.226429] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8252.231952] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8252.236979] do_raw_spin_lock+0x177/0x190 [ 8252.240148] _raw_spin_lock+0x21/0x30 [ 8252.243181] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8252.248989] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8252.256916] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8252.260850] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8252.267845] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8252.278450] ? woken_wake_function+0x30/0x30 [ 8252.282177] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8252.286435] kthread+0x1d1/0x200 [ 8252.287406] ? set_kthread_struct+0x70/0x70 [ 8252.288600] ret_from_fork+0x1f/0x30 [ 8260.055970] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 8260.057283] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 8260.064578] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8260.064656] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8260.064663] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8260.064666] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8260.064679] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8260.064682] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8260.064686] RAX: 0000000000000c8c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8260.064689] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8260.064691] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8260.064693] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8260.064696] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8260.064699] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8260.064702] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8260.064705] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8260.064714] Call Trace: [ 8260.064719] [ 8260.064725] ? show_regs.cold.9+0x22/0x2f [ 8260.064734] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8260.064741] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8260.064745] ? watchdog+0x40/0x40 [ 8260.064750] ? hrtimer_interrupt+0x128/0x2c0 [ 8260.064755] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8260.064762] ? apic_timer_interrupt+0xf/0x20 [ 8260.068447] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8260.068536] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8260.068548] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8260.068552] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8260.068564] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8260.068569] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8260.068573] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8260.068576] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8260.068579] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8260.068581] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8260.068584] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8260.068591] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8260.068594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8260.068597] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8260.068604] Call Trace: [ 8260.068610] [ 8260.068615] ? show_regs.cold.9+0x22/0x2f [ 8260.068625] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8260.068632] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8260.068636] ? watchdog+0x40/0x40 [ 8260.068642] ? hrtimer_interrupt+0x128/0x2c0 [ 8260.068648] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8260.105402] [ 8260.112758] ? apic_timer_interrupt+0xf/0x20 [ 8260.112778] [ 8260.112784] ? native_safe_halt+0x2e/0x50 [ 8260.112789] kvm_wait+0xcd/0xe0 [ 8260.112798] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8260.112805] do_raw_spin_lock+0x177/0x190 [ 8260.112809] _raw_spin_lock+0x21/0x30 [ 8260.112816] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8260.113494] obd_mod_free+0x46/0x140 [ptlrpc] [ 8260.113844] mdc_commit_open+0x91/0xb0 [mdc] [ 8260.113899] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8260.115045] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8260.122121] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8260.122142] do_raw_spin_lock+0x177/0x190 [ 8260.122148] _raw_spin_lock+0x21/0x30 [ 8260.122154] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8260.122533] ? do_raw_spin_unlock+0x75/0x190 [ 8260.122545] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8260.122860] ? do_wait_intr+0xf0/0xf0 [ 8260.122870] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8260.123336] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8260.123595] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8260.124029] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8260.124227] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8260.124264] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8260.124293] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8260.124323] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8260.124414] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8260.124652] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8260.124683] lmv_intent_open+0x338/0x1150 [lmv] [ 8260.124844] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8260.128288] after_reply+0xe2c/0x1780 [ptlrpc] [ 8260.128571] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8260.128860] ? __next_timer_interrupt+0x160/0x160 [ 8260.128870] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8260.129219] ? do_wait_intr+0xf0/0xf0 [ 8260.129229] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8260.129449] kthread+0x1d1/0x200 [ 8260.129455] ? set_kthread_struct+0x70/0x70 [ 8260.129459] ret_from_fork+0x1f/0x30 [ 8260.427218] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8260.432092] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8260.438746] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8260.443888] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8260.450813] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8260.454707] ? do_raw_spin_unlock+0x75/0x190 [ 8260.458588] ? _raw_spin_unlock+0x12/0x30 [ 8260.462518] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8260.467312] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8260.469682] ll_atomic_open+0x36a/0xd70 [lustre] [ 8260.471322] lookup_open+0x356/0xb40 [ 8260.472781] do_last+0x313/0xfc0 [ 8260.474269] ? nd_jump_root+0xe5/0x160 [ 8260.475698] ? path_init+0x437/0x520 [ 8260.477313] path_openat+0xf7/0x500 [ 8260.478710] ? do_raw_spin_unlock+0x75/0x190 [ 8260.480601] do_filp_open+0x99/0x140 [ 8260.482447] ? getname_flags+0x6e/0x330 [ 8260.483986] ? __check_object_size+0xff/0x256 [ 8260.485809] ? do_raw_spin_unlock+0x75/0x190 [ 8260.487958] ? _raw_spin_unlock+0x12/0x30 [ 8260.489605] do_sys_openat2+0x2b4/0x410 [ 8260.491248] do_sys_open+0x73/0xa0 [ 8260.493301] __x64_sys_openat+0x24/0x30 [ 8260.494792] do_syscall_64+0xc1/0x440 [ 8260.496422] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8260.499168] RIP: 0033:0x7f53425a5332 [ 8260.500580] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8260.512004] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8260.518809] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8260.521800] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8260.526825] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8260.533034] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8260.535503] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8272.054877] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8272.066598] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8272.110379] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8272.117855] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8272.131626] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8272.139037] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8272.147145] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8272.167235] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8272.176178] RAX: 0000000000007e8b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8272.188918] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8272.199072] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8272.208525] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8272.218517] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8272.229384] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8272.235846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8272.242427] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8272.248651] Call Trace: [ 8272.252228] [ 8272.252889] ? show_regs.cold.9+0x22/0x2f [ 8272.254197] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8272.260596] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8272.267030] ? watchdog+0x40/0x40 [ 8272.273552] ? hrtimer_interrupt+0x128/0x2c0 [ 8272.280645] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8272.288839] ? apic_timer_interrupt+0xf/0x20 [ 8272.294421] [ 8272.299530] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8272.306460] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8272.313380] ? newidle_balance.isra.93+0x207/0x580 [ 8272.322090] do_raw_spin_lock+0x177/0x190 [ 8272.328006] _raw_spin_lock+0x21/0x30 [ 8272.330409] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8272.336913] process_one_work+0x2c8/0x700 [ 8272.338468] worker_thread+0x296/0x6e0 [ 8272.341428] ? rescuer_thread+0x570/0x570 [ 8272.345797] kthread+0x1d1/0x200 [ 8272.348315] ? set_kthread_struct+0x70/0x70 [ 8272.357339] ret_from_fork+0x1f/0x30 [ 8274.588489] rcu: INFO: rcu_sched self-detected stall on CPU [ 8274.593106] rcu: 2-....: (3350830 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 8274.599733] (t=6180162 jiffies g=274185 q=65369) [ 8274.606043] rcu: rcu_sched kthread starved for 5638528 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [ 8274.615935] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8274.624217] rcu: RCU grace-period kthread stack dump: [ 8274.626965] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8274.632544] Call Trace: [ 8274.634042] __schedule+0x351/0xcb0 [ 8274.636160] schedule+0xc0/0x180 [ 8274.642084] schedule_timeout+0xb4/0x190 [ 8274.643557] ? do_raw_spin_unlock+0x75/0x190 [ 8274.649530] ? __next_timer_interrupt+0x160/0x160 [ 8274.653332] ? prepare_to_swait_event+0xf3/0x140 [ 8274.656110] rcu_gp_kthread+0x8dd/0x15b0 [ 8274.661728] ? force_qs_rnp+0x2c0/0x2c0 [ 8274.668587] kthread+0x1d1/0x200 [ 8274.671519] ? set_kthread_struct+0x70/0x70 [ 8274.679507] ret_from_fork+0x1f/0x30 [ 8274.681790] rcu: Stack dump where RCU GP kthread last ran: [ 8274.687233] NMI backtrace for cpu 2 [ 8274.690600] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8274.700051] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8274.708074] Call Trace: [ 8274.714007] [ 8274.716072] ? dump_stack+0xbb/0x10e [ 8274.718984] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8274.723681] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8274.726724] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8274.729004] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8274.734505] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8274.741040] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [ 8274.747220] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [ 8274.752928] ? update_process_times+0x6d/0xc0 [ 8274.756142] ? tick_sched_handle.isra.18+0x27/0x90 [ 8274.759219] ? tick_sched_timer+0x53/0xb0 [ 8274.761337] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8274.763245] ? tick_sched_handle.isra.18+0x90/0x90 [ 8274.765350] ? hrtimer_interrupt+0x128/0x2c0 [ 8274.767978] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8274.774051] ? apic_timer_interrupt+0xf/0x20 [ 8274.777431] [ 8274.780787] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8274.783515] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8274.787602] ? do_raw_spin_lock+0x177/0x190 [ 8274.789229] ? _raw_spin_lock+0x21/0x30 [ 8274.792062] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8274.794544] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8274.797190] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8274.803899] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8274.807154] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8274.811398] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8274.815065] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8274.818555] ? __next_timer_interrupt+0x160/0x160 [ 8274.820568] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8274.823095] ? do_wait_intr+0xf0/0xf0 [ 8274.824470] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8274.827221] ? kthread+0x1d1/0x200 [ 8274.828501] ? set_kthread_struct+0x70/0x70 [ 8274.830084] ? ret_from_fork+0x1f/0x30 [ 8274.831415] Sending NMI from CPU 2 to CPUs 1: [ 8274.839713] NMI backtrace for cpu 1 [ 8274.839722] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8274.839724] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8274.839725] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8274.839727] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8274.839729] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [ 8274.839732] RAX: 0000000000000241 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8274.839733] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8274.839735] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8274.839736] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8274.839737] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8274.839738] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8274.839739] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8274.839740] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8274.839741] Call Trace: [ 8274.839742] [ 8274.839743] ? show_regs.cold.9+0x22/0x2f [ 8274.839744] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8274.839745] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8274.839746] ? nmi_handle+0xbd/0x210 [ 8274.839747] ? default_do_nmi+0x4b/0x1b0 [ 8274.839748] ? do_nmi+0x280/0x380 [ 8274.839749] ? end_repeat_nmi+0x16/0x1f [ 8274.839750] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8274.839751] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8274.839753] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8274.839754] [ 8274.839755] do_raw_spin_lock+0x177/0x190 [ 8274.839756] _raw_spin_lock+0x21/0x30 [ 8274.839757] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8274.839758] ? do_raw_spin_unlock+0x75/0x190 [ 8274.839759] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8274.839760] ? do_wait_intr+0xf0/0xf0 [ 8274.839761] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8274.839762] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8274.839763] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8274.839765] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8274.839766] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8274.839767] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8274.839768] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8274.839769] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8274.839770] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8274.839771] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8274.839773] lmv_intent_open+0x338/0x1150 [lmv] [ 8274.839774] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8274.839775] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8274.839776] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8274.839777] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8274.839778] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8274.839779] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8274.839780] ? do_raw_spin_unlock+0x75/0x190 [ 8274.839782] ? _raw_spin_unlock+0x12/0x30 [ 8274.839783] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8274.839784] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8274.839785] ll_atomic_open+0x36a/0xd70 [lustre] [ 8274.839786] lookup_open+0x356/0xb40 [ 8274.839787] do_last+0x313/0xfc0 [ 8274.839788] ? nd_jump_root+0xe5/0x160 [ 8274.839789] ? path_init+0x437/0x520 [ 8274.839791] path_openat+0xf7/0x500 [ 8274.839792] ? do_raw_spin_unlock+0x75/0x190 [ 8274.839793] do_filp_open+0x99/0x140 [ 8274.839794] ? getname_flags+0x6e/0x330 [ 8274.839795] ? __check_object_size+0xff/0x256 [ 8274.839796] ? do_raw_spin_unlock+0x75/0x190 [ 8274.839797] ? _raw_spin_unlock+0x12/0x30 [ 8274.839798] do_sys_openat2+0x2b4/0x410 [ 8274.839800] do_sys_open+0x73/0xa0 [ 8274.839927] __x64_sys_openat+0x24/0x30 [ 8274.839928] do_syscall_64+0xc1/0x440 [ 8274.839929] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8274.839930] RIP: 0033:0x7f53425a5332 [ 8274.839932] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8274.839934] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8274.839937] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8274.839939] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8274.839941] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8274.839942] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8274.839943] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8274.839953] NMI backtrace for cpu 2 [ 8275.184797] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8275.192084] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8275.199867] Call Trace: [ 8275.202884] [ 8275.204669] ? dump_stack+0xbb/0x10e [ 8275.207608] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8275.212202] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8275.218306] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8275.222954] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8275.226723] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8275.228814] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8275.232547] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8275.234947] ? update_process_times+0x6d/0xc0 [ 8275.236910] ? tick_sched_handle.isra.18+0x27/0x90 [ 8275.241526] ? tick_sched_timer+0x53/0xb0 [ 8275.244932] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8275.248307] ? tick_sched_handle.isra.18+0x90/0x90 [ 8275.252808] ? hrtimer_interrupt+0x128/0x2c0 [ 8275.254906] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8275.256869] ? apic_timer_interrupt+0xf/0x20 [ 8275.258725] [ 8275.259889] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8275.264743] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8275.269908] ? do_raw_spin_lock+0x177/0x190 [ 8275.275521] ? _raw_spin_lock+0x21/0x30 [ 8275.277909] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8275.280704] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8275.288520] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8275.294869] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8275.304593] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8275.307500] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8275.315946] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8275.321926] ? __next_timer_interrupt+0x160/0x160 [ 8275.328283] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8275.332880] ? do_wait_intr+0xf0/0xf0 [ 8275.334959] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8275.340308] ? kthread+0x1d1/0x200 [ 8275.342113] ? set_kthread_struct+0x70/0x70 [ 8275.343905] ? ret_from_fork+0x1f/0x30 [ 8275.948425] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5638s! [ 8275.954503] Showing busy workqueues and worker pools: [ 8275.962095] workqueue events: flags=0x0 [ 8275.963408] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8275.969428] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8275.979042] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8275.986435] pending: cpu_latency_work [ptlrpc] [ 8275.989171] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8275.991457] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8275.997028] workqueue events_long: flags=0x0 [ 8275.999016] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8276.006704] pending: lu_objects_flush [obdclass] [ 8276.008915] workqueue events_unbound: flags=0x2 [ 8276.014573] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8276.028730] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8276.032036] workqueue events_power_efficient: flags=0x80 [ 8276.036027] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8276.040718] pending: check_lifetime, do_cache_clean [ 8276.043029] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8276.047329] pending: neigh_periodic_work, neigh_periodic_work [ 8276.049717] workqueue mm_percpu_wq: flags=0x8 [ 8276.056580] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8276.060485] pending: vmstat_update, lru_add_drain_per_cpu [ 8276.062684] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8276.066031] pending: vmstat_update, lru_add_drain_per_cpu [ 8276.068477] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8276.071507] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8276.078479] workqueue xprtiod: flags=0xa [ 8276.080257] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8276.083201] pending: xprt_autoclose, xs_error_handle [ 8276.087715] workqueue ipv6_addrconf: flags=0x40008 [ 8276.089075] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8276.092767] pending: addrconf_verify_work [ 8276.094572] workqueue knbd0-recv: flags=0x1a [ 8276.096504] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8276.103219] in-flight: 651:recv_work [ 8276.107191] workqueue ptlrpc_pinger: flags=0x2 [ 8276.108889] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8276.121643] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8276.128241] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5639s workers=4 idle: 15979 31824 8 [ 8276.136551] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8266s workers=2 idle: 654 [ 8280.064097] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 8280.069242] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8280.115446] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8280.128274] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8280.138324] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8280.145453] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8280.165610] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8280.176211] RAX: 0000000000004d86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8280.181686] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8280.190837] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8280.200499] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8280.210305] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8280.215564] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8280.220251] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8280.231423] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8280.238047] Call Trace: [ 8280.241215] [ 8280.243088] ? show_regs.cold.9+0x22/0x2f [ 8280.245709] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8280.250148] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8280.253447] ? watchdog+0x40/0x40 [ 8280.255180] ? hrtimer_interrupt+0x128/0x2c0 [ 8280.258733] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8280.264361] ? apic_timer_interrupt+0xf/0x20 [ 8280.266643] [ 8280.267686] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8280.270190] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8280.272460] do_raw_spin_lock+0x177/0x190 [ 8280.274323] _raw_spin_lock+0x21/0x30 [ 8280.275815] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8280.279063] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8280.281753] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8280.284610] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8280.288137] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8280.291322] ? woken_wake_function+0x30/0x30 [ 8280.294801] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8280.301103] kthread+0x1d1/0x200 [ 8280.302428] ? set_kthread_struct+0x70/0x70 [ 8280.304809] ret_from_fork+0x1f/0x30 [ 8300.055418] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 8300.060887] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 8300.063473] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8300.063549] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8300.063557] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8300.063560] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8300.063573] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8300.063580] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8300.063584] RAX: 0000000000002c49 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8300.063587] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8300.063589] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8300.063591] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8300.063593] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8300.063596] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8300.063599] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8300.063601] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8300.063611] Call Trace: [ 8300.063616] [ 8300.063622] ? show_regs.cold.9+0x22/0x2f [ 8300.063631] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8300.063637] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8300.063641] ? watchdog+0x40/0x40 [ 8300.063645] ? hrtimer_interrupt+0x128/0x2c0 [ 8300.063649] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8300.063657] ? apic_timer_interrupt+0xf/0x20 [ 8300.063661] [ 8300.063664] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8300.063671] do_raw_spin_lock+0x177/0x190 [ 8300.063676] _raw_spin_lock+0x21/0x30 [ 8300.063681] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8300.063938] ? do_raw_spin_unlock+0x75/0x190 [ 8300.063944] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8300.064136] ? do_wait_intr+0xf0/0xf0 [ 8300.064144] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8300.066424] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8300.066638] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8300.066847] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8300.075410] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8300.076349] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8300.076499] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8300.076635] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8300.076789] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8300.076793] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8300.076798] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8300.076886] RDX: 0000000000007e9e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8300.076889] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8300.076891] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8300.076894] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8300.076897] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8300.076900] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8300.076903] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8300.076911] Call Trace: [ 8300.076916] [ 8300.076922] ? show_regs.cold.9+0x22/0x2f [ 8300.076931] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8300.076938] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8300.076942] ? watchdog+0x40/0x40 [ 8300.077572] ? hrtimer_interrupt+0x128/0x2c0 [ 8300.077583] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8300.077591] ? apic_timer_interrupt+0xf/0x20 [ 8300.077596] [ 8300.077600] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8300.077605] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8300.077610] do_raw_spin_lock+0x177/0x190 [ 8300.077615] _raw_spin_lock+0x21/0x30 [ 8300.077622] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8300.111803] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8300.111851] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8300.111881] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8300.111909] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8300.112029] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8300.112315] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8300.112344] lmv_intent_open+0x338/0x1150 [lmv] [ 8300.112375] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8300.112462] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8300.112544] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8300.112567] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8300.112652] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8300.112734] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8300.112813] ? do_raw_spin_unlock+0x75/0x190 [ 8300.112821] ? _raw_spin_unlock+0x12/0x30 [ 8300.112827] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8300.112855] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8300.112875] ll_atomic_open+0x36a/0xd70 [lustre] [ 8300.112957] lookup_open+0x356/0xb40 [ 8300.112965] do_last+0x313/0xfc0 [ 8300.112970] ? nd_jump_root+0xe5/0x160 [ 8300.112974] ? path_init+0x437/0x520 [ 8300.112979] path_openat+0xf7/0x500 [ 8300.112984] ? do_raw_spin_unlock+0x75/0x190 [ 8300.112989] do_filp_open+0x99/0x140 [ 8300.112996] ? getname_flags+0x6e/0x330 [ 8300.113000] ? __check_object_size+0xff/0x256 [ 8300.113007] ? do_raw_spin_unlock+0x75/0x190 [ 8300.113011] ? _raw_spin_unlock+0x12/0x30 [ 8300.113015] do_sys_openat2+0x2b4/0x410 [ 8300.113020] do_sys_open+0x73/0xa0 [ 8300.113025] __x64_sys_openat+0x24/0x30 [ 8300.113029] do_syscall_64+0xc1/0x440 [ 8300.113037] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8300.113044] RIP: 0033:0x7f53425a5332 [ 8300.113052] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8300.113056] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8300.113061] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8300.113063] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8300.113066] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8300.113068] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8300.113070] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8300.770462] obd_mod_free+0x46/0x140 [ptlrpc] [ 8300.780812] mdc_commit_open+0x91/0xb0 [mdc] [ 8300.782341] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8300.796090] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8300.805088] after_reply+0xe2c/0x1780 [ptlrpc] [ 8300.813978] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8300.822834] ? __next_timer_interrupt+0x160/0x160 [ 8300.830654] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8300.839806] ? do_wait_intr+0xf0/0xf0 [ 8300.844423] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8300.852600] kthread+0x1d1/0x200 [ 8300.854617] ? set_kthread_struct+0x70/0x70 [ 8300.858287] ret_from_fork+0x1f/0x30 [ 8304.052212] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8304.057288] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8304.099626] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8304.104496] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8304.111713] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8304.116422] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8304.119868] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8304.138939] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8304.143495] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8304.150638] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8304.156662] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8304.162288] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8304.168805] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8304.179044] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8304.186038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8304.190509] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8304.193800] Call Trace: [ 8304.195368] [ 8304.196433] ? show_regs.cold.9+0x22/0x2f [ 8304.198927] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8304.200871] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8304.204699] ? watchdog+0x40/0x40 [ 8304.208782] ? hrtimer_interrupt+0x128/0x2c0 [ 8304.211452] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8304.213878] ? apic_timer_interrupt+0xf/0x20 [ 8304.218199] [ 8304.220077] ? native_safe_halt+0x2e/0x50 [ 8304.221895] kvm_wait+0xcd/0xe0 [ 8304.223068] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8304.225761] ? newidle_balance.isra.93+0x207/0x580 [ 8304.230555] do_raw_spin_lock+0x177/0x190 [ 8304.236241] _raw_spin_lock+0x21/0x30 [ 8304.241017] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8304.245397] process_one_work+0x2c8/0x700 [ 8304.247202] worker_thread+0x296/0x6e0 [ 8304.249669] ? rescuer_thread+0x570/0x570 [ 8304.252171] kthread+0x1d1/0x200 [ 8304.254040] ? set_kthread_struct+0x70/0x70 [ 8304.259429] ret_from_fork+0x1f/0x30 [ 8306.143031] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5669s! [ 8306.145365] Showing busy workqueues and worker pools: [ 8306.150285] workqueue events: flags=0x0 [ 8306.155489] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8306.161633] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8306.170602] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8306.175303] pending: cpu_latency_work [ptlrpc] [ 8306.178339] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8306.183319] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8306.189541] workqueue events_long: flags=0x0 [ 8306.191047] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8306.195469] pending: lu_objects_flush [obdclass] [ 8306.215441] workqueue events_unbound: flags=0x2 [ 8306.224511] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8306.226767] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8306.245051] workqueue events_power_efficient: flags=0x80 [ 8306.247012] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8306.258368] pending: check_lifetime, do_cache_clean [ 8306.269644] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8306.276533] pending: neigh_periodic_work, neigh_periodic_work [ 8306.290262] workqueue mm_percpu_wq: flags=0x8 [ 8306.291731] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8306.304360] pending: vmstat_update, lru_add_drain_per_cpu [ 8306.314633] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8306.323358] pending: vmstat_update, lru_add_drain_per_cpu [ 8306.337871] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8306.341903] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8306.356788] workqueue xprtiod: flags=0xa [ 8306.366417] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8306.385929] pending: xprt_autoclose, xs_error_handle [ 8306.387939] workqueue ipv6_addrconf: flags=0x40008 [ 8306.405484] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8306.431888] pending: addrconf_verify_work [ 8306.433648] workqueue knbd0-recv: flags=0x1a [ 8306.442178] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8306.446547] in-flight: 651:recv_work [ 8306.453675] workqueue ptlrpc_pinger: flags=0x2 [ 8306.459327] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8306.468993] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8306.482912] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5669s workers=4 idle: 15979 31824 8 [ 8306.511628] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8296s workers=2 idle: 654 [ 8308.058008] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 8308.064427] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8308.088020] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8308.093267] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8308.107414] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8308.113514] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8308.130369] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8308.137944] RAX: 0000000000004cba RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8308.143410] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8308.153468] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8308.156632] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8308.167521] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8308.172187] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8308.179094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8308.181646] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8308.186433] Call Trace: [ 8308.191737] [ 8308.194917] ? show_regs.cold.9+0x22/0x2f [ 8308.196928] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8308.198712] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8308.200527] ? watchdog+0x40/0x40 [ 8308.202019] ? hrtimer_interrupt+0x128/0x2c0 [ 8308.203915] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8308.206414] ? apic_timer_interrupt+0xf/0x20 [ 8308.212563] [ 8308.214780] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8308.217210] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8308.220657] do_raw_spin_lock+0x177/0x190 [ 8308.222478] _raw_spin_lock+0x21/0x30 [ 8308.223967] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8308.239220] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8308.240980] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8308.247521] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8308.253517] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8308.255536] ? woken_wake_function+0x30/0x30 [ 8308.263184] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8308.266401] kthread+0x1d1/0x200 [ 8308.269090] ? set_kthread_struct+0x70/0x70 [ 8308.270391] ret_from_fork+0x1f/0x30 [ 8328.057409] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 8328.057958] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 8328.061250] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8328.061603] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8328.061611] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8328.061615] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8328.061630] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8328.061635] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8328.061639] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8328.061642] RDX: 000000000000711d RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8328.061644] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8328.061647] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8328.061650] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8328.061653] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8328.061656] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8328.061658] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8328.061668] Call Trace: [ 8328.061674] [ 8328.061679] ? show_regs.cold.9+0x22/0x2f [ 8328.061700] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8328.061707] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8328.061712] ? watchdog+0x40/0x40 [ 8328.061716] ? hrtimer_interrupt+0x128/0x2c0 [ 8328.061720] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8328.061730] ? apic_timer_interrupt+0xf/0x20 [ 8328.061734] [ 8328.061738] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8328.061742] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8328.061747] do_raw_spin_lock+0x177/0x190 [ 8328.061751] _raw_spin_lock+0x21/0x30 [ 8328.061757] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8328.070013] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8328.070507] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8328.070515] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8328.070519] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8328.070534] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8328.070539] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8328.070543] RAX: 0000000000001211 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8328.070546] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8328.070549] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8328.070551] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8328.070554] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8328.070558] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8328.070561] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8328.070564] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8328.070575] Call Trace: [ 8328.070582] [ 8328.070589] ? show_regs.cold.9+0x22/0x2f [ 8328.070601] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8328.070609] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8328.070615] ? watchdog+0x40/0x40 [ 8328.070619] ? hrtimer_interrupt+0x128/0x2c0 [ 8328.070623] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8328.070637] ? apic_timer_interrupt+0xf/0x20 [ 8328.070642] [ 8328.070645] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8328.070652] do_raw_spin_lock+0x177/0x190 [ 8328.070657] _raw_spin_lock+0x21/0x30 [ 8328.070662] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8328.072047] ? do_raw_spin_unlock+0x75/0x190 [ 8328.072055] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8328.073162] ? do_wait_intr+0xf0/0xf0 [ 8328.073172] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8328.073498] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8328.073733] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8328.074836] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8328.074960] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8328.075016] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8328.075049] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8328.075090] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8328.075186] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8328.075432] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8328.075464] lmv_intent_open+0x338/0x1150 [lmv] [ 8328.075498] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8328.075594] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8328.075696] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8328.075719] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8328.129733] obd_mod_free+0x46/0x140 [ptlrpc] [ 8328.130268] mdc_commit_open+0x91/0xb0 [mdc] [ 8328.130309] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8328.130514] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8328.130740] after_reply+0xe2c/0x1780 [ptlrpc] [ 8328.130946] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8328.131355] ? __next_timer_interrupt+0x160/0x160 [ 8328.131367] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8328.141869] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8328.142007] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8328.142110] ? do_raw_spin_unlock+0x75/0x190 [ 8328.142118] ? _raw_spin_unlock+0x12/0x30 [ 8328.142125] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8328.142153] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8328.142173] ll_atomic_open+0x36a/0xd70 [lustre] [ 8328.142259] lookup_open+0x356/0xb40 [ 8328.142268] do_last+0x313/0xfc0 [ 8328.142273] ? nd_jump_root+0xe5/0x160 [ 8328.142277] ? path_init+0x437/0x520 [ 8328.142280] path_openat+0xf7/0x500 [ 8328.142285] ? do_raw_spin_unlock+0x75/0x190 [ 8328.142290] do_filp_open+0x99/0x140 [ 8328.142657] ? getname_flags+0x6e/0x330 [ 8328.142664] ? __check_object_size+0xff/0x256 [ 8328.142671] ? do_raw_spin_unlock+0x75/0x190 [ 8328.142675] ? _raw_spin_unlock+0x12/0x30 [ 8328.142679] do_sys_openat2+0x2b4/0x410 [ 8328.142812] do_sys_open+0x73/0xa0 [ 8328.142819] __x64_sys_openat+0x24/0x30 [ 8328.142823] do_syscall_64+0xc1/0x440 [ 8328.142830] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8328.142836] RIP: 0033:0x7f53425a5332 [ 8328.142844] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8328.142848] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8328.142853] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8328.142856] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8328.142858] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8328.142860] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8328.142861] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8328.761159] ? do_wait_intr+0xf0/0xf0 [ 8328.763882] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8328.766544] kthread+0x1d1/0x200 [ 8328.771060] ? set_kthread_struct+0x70/0x70 [ 8328.774931] ret_from_fork+0x1f/0x30 [ 8332.054911] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8332.066880] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8332.127229] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8332.144907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8332.155359] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8332.162201] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8332.172371] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8332.190630] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8332.193841] RAX: 0000000000006284 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8332.197704] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8332.202006] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8332.205487] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8332.213000] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8332.218680] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8332.227381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8332.234353] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8332.240558] Call Trace: [ 8332.244770] [ 8332.245387] ? show_regs.cold.9+0x22/0x2f [ 8332.246436] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8332.251095] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8332.255767] ? watchdog+0x40/0x40 [ 8332.258550] ? hrtimer_interrupt+0x128/0x2c0 [ 8332.263171] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8332.266439] ? apic_timer_interrupt+0xf/0x20 [ 8332.281782] [ 8332.287008] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8332.290187] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8332.292826] ? newidle_balance.isra.93+0x207/0x580 [ 8332.295760] do_raw_spin_lock+0x177/0x190 [ 8332.298907] _raw_spin_lock+0x21/0x30 [ 8332.300337] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8332.305848] process_one_work+0x2c8/0x700 [ 8332.307915] worker_thread+0x296/0x6e0 [ 8332.312278] ? rescuer_thread+0x570/0x570 [ 8332.314683] kthread+0x1d1/0x200 [ 8332.316347] ? set_kthread_struct+0x70/0x70 [ 8332.318623] ret_from_fork+0x1f/0x30 [ 8336.064475] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 8336.068335] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8336.100247] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8336.113710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8336.122824] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8336.131826] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8336.145097] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8336.148948] RAX: 0000000000007f0d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8336.151245] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8336.158299] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8336.162955] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8336.166361] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8336.172607] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8336.181410] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8336.184902] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8336.187611] Call Trace: [ 8336.190395] [ 8336.191204] ? show_regs.cold.9+0x22/0x2f [ 8336.194100] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8336.198968] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8336.200202] ? watchdog+0x40/0x40 [ 8336.205420] ? hrtimer_interrupt+0x128/0x2c0 [ 8336.209696] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8336.213990] ? apic_timer_interrupt+0xf/0x20 [ 8336.215928] [ 8336.216818] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8336.221257] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8336.224240] do_raw_spin_lock+0x177/0x190 [ 8336.228120] _raw_spin_lock+0x21/0x30 [ 8336.229268] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8336.231647] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8336.233420] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8336.235942] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8336.238980] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8336.240704] ? woken_wake_function+0x30/0x30 [ 8336.242348] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8336.244458] kthread+0x1d1/0x200 [ 8336.245340] ? set_kthread_struct+0x70/0x70 [ 8336.246457] ret_from_fork+0x1f/0x30 [ 8336.873703] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5699s! [ 8336.877358] Showing busy workqueues and worker pools: [ 8336.879945] workqueue events: flags=0x0 [ 8336.881303] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8336.883782] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8336.888692] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8336.893041] pending: cpu_latency_work [ptlrpc] [ 8336.894906] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8336.897382] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8336.900550] workqueue events_long: flags=0x0 [ 8336.901920] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8336.904047] pending: lu_objects_flush [obdclass] [ 8336.905872] workqueue events_unbound: flags=0x2 [ 8336.907688] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8336.909932] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8336.912200] workqueue events_power_efficient: flags=0x80 [ 8336.913828] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8336.916048] pending: check_lifetime, do_cache_clean [ 8336.917830] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8336.920689] pending: neigh_periodic_work, neigh_periodic_work [ 8336.923752] workqueue mm_percpu_wq: flags=0x8 [ 8336.925667] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8336.932390] pending: vmstat_update, lru_add_drain_per_cpu [ 8336.936763] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8336.939570] pending: vmstat_update, lru_add_drain_per_cpu [ 8336.943423] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8336.946799] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8336.950363] workqueue xprtiod: flags=0xa [ 8336.953078] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8336.956548] pending: xprt_autoclose, xs_error_handle [ 8336.959478] workqueue ipv6_addrconf: flags=0x40008 [ 8336.961393] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8336.964435] pending: addrconf_verify_work [ 8336.965889] workqueue knbd0-recv: flags=0x1a [ 8336.967378] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8336.969469] in-flight: 651:recv_work [ 8336.971061] workqueue ptlrpc_pinger: flags=0x2 [ 8336.972872] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8336.974932] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8336.977591] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5699s workers=4 idle: 15979 31824 8 [ 8336.982199] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8327s workers=2 idle: 654 [ 8356.057020] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 8356.057901] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 8356.060321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8356.060479] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8356.060488] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8356.060491] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8356.060671] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8356.060675] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8356.060680] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8356.060683] RDX: 0000000000005783 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8356.060686] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8356.060688] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8356.060691] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8356.060694] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8356.060698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8356.060700] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8356.060709] Call Trace: [ 8356.060714] [ 8356.060720] ? show_regs.cold.9+0x22/0x2f [ 8356.060729] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8356.060736] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8356.060740] ? watchdog+0x40/0x40 [ 8356.060745] ? hrtimer_interrupt+0x128/0x2c0 [ 8356.060750] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8356.060757] ? apic_timer_interrupt+0xf/0x20 [ 8356.060763] [ 8356.060766] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8356.060771] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8356.060776] do_raw_spin_lock+0x177/0x190 [ 8356.060780] _raw_spin_lock+0x21/0x30 [ 8356.060787] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8356.061082] obd_mod_free+0x46/0x140 [ptlrpc] [ 8356.061295] mdc_commit_open+0x91/0xb0 [mdc] [ 8356.061337] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8356.061624] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8356.061850] after_reply+0xe2c/0x1780 [ptlrpc] [ 8356.062077] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8356.062277] ? __next_timer_interrupt+0x160/0x160 [ 8356.062285] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8356.062481] ? do_wait_intr+0xf0/0xf0 [ 8356.062490] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8356.062983] kthread+0x1d1/0x200 [ 8356.062988] ? set_kthread_struct+0x70/0x70 [ 8356.062992] ret_from_fork+0x1f/0x30 [ 8356.277770] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8356.315342] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8356.324615] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8356.339159] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8356.342698] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8356.349694] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8356.352128] RAX: 0000000000005e6d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8356.357356] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8356.361413] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8356.364980] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8356.367863] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8356.371092] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8356.374443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8356.376863] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8356.379812] Call Trace: [ 8356.380941] [ 8356.381761] ? show_regs.cold.9+0x22/0x2f [ 8356.383890] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8356.385611] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8356.389202] ? watchdog+0x40/0x40 [ 8356.394162] ? hrtimer_interrupt+0x128/0x2c0 [ 8356.397398] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8356.398996] ? apic_timer_interrupt+0xf/0x20 [ 8356.401058] [ 8356.405422] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8356.411839] do_raw_spin_lock+0x177/0x190 [ 8356.413852] _raw_spin_lock+0x21/0x30 [ 8356.421898] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8356.437335] ? do_raw_spin_unlock+0x75/0x190 [ 8356.439909] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8356.444385] ? do_wait_intr+0xf0/0xf0 [ 8356.447333] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8356.458405] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8356.463922] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8356.466164] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8356.470017] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8356.472681] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8356.475352] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8356.477912] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8356.480241] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8356.485059] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8356.488765] lmv_intent_open+0x338/0x1150 [lmv] [ 8356.491857] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8356.497082] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8356.501065] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8356.506043] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8356.509718] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8356.517760] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8356.521854] ? do_raw_spin_unlock+0x75/0x190 [ 8356.526423] ? _raw_spin_unlock+0x12/0x30 [ 8356.530357] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8356.535102] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8356.538361] ll_atomic_open+0x36a/0xd70 [lustre] [ 8356.543122] lookup_open+0x356/0xb40 [ 8356.548878] do_last+0x313/0xfc0 [ 8356.554753] ? nd_jump_root+0xe5/0x160 [ 8356.556951] ? path_init+0x437/0x520 [ 8356.562513] path_openat+0xf7/0x500 [ 8356.567355] ? do_raw_spin_unlock+0x75/0x190 [ 8356.571446] do_filp_open+0x99/0x140 [ 8356.573165] ? getname_flags+0x6e/0x330 [ 8356.577349] ? __check_object_size+0xff/0x256 [ 8356.580437] ? do_raw_spin_unlock+0x75/0x190 [ 8356.583080] ? _raw_spin_unlock+0x12/0x30 [ 8356.584935] do_sys_openat2+0x2b4/0x410 [ 8356.589089] do_sys_open+0x73/0xa0 [ 8356.597024] __x64_sys_openat+0x24/0x30 [ 8356.598404] do_syscall_64+0xc1/0x440 [ 8356.600946] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8356.604919] RIP: 0033:0x7f53425a5332 [ 8356.608013] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8356.624703] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8356.631843] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8356.634790] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8356.642166] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8356.646851] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8356.654929] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8364.052025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8364.058083] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 8364.059838] Modules linked in: [ 8364.067200] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8364.067277] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8364.067285] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8364.067288] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8364.067301] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8364.067307] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8364.067311] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8364.067314] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8364.067316] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8364.067318] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8364.067322] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8364.067325] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8364.067328] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8364.067330] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8364.067340] Call Trace: [ 8364.067346] [ 8364.067352] ? show_regs.cold.9+0x22/0x2f [ 8364.067362] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8364.067369] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8364.067374] ? watchdog+0x40/0x40 [ 8364.067377] ? hrtimer_interrupt+0x128/0x2c0 [ 8364.067382] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8364.067390] ? apic_timer_interrupt+0xf/0x20 [ 8364.067395] [ 8364.067398] ? native_safe_halt+0x2e/0x50 [ 8364.067402] kvm_wait+0xcd/0xe0 [ 8364.067410] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8364.067419] do_raw_spin_lock+0x177/0x190 [ 8364.067424] _raw_spin_lock+0x21/0x30 [ 8364.067429] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8364.068030] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8364.068295] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8364.068511] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8364.068792] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8364.069428] ? woken_wake_function+0x30/0x30 [ 8364.069438] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8364.070569] kthread+0x1d1/0x200 [ 8364.070577] ? set_kthread_struct+0x70/0x70 [ 8364.070582] ret_from_fork+0x1f/0x30 [ 8364.249997] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8364.284668] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8364.298616] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8364.308047] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8364.310362] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8364.322355] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8364.342577] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8364.347909] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8364.359900] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8364.364564] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8364.371884] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8364.384939] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8364.395760] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8364.409692] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8364.421039] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8364.427469] Call Trace: [ 8364.428259] [ 8364.428898] ? show_regs.cold.9+0x22/0x2f [ 8364.436677] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8364.439338] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8364.443160] ? watchdog+0x40/0x40 [ 8364.445136] ? hrtimer_interrupt+0x128/0x2c0 [ 8364.446942] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8364.449230] ? apic_timer_interrupt+0xf/0x20 [ 8364.451128] [ 8364.452130] ? native_safe_halt+0x2e/0x50 [ 8364.453959] kvm_wait+0xcd/0xe0 [ 8364.455459] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8364.457891] ? newidle_balance.isra.93+0x207/0x580 [ 8364.460206] do_raw_spin_lock+0x177/0x190 [ 8364.461763] _raw_spin_lock+0x21/0x30 [ 8364.463571] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8364.465979] process_one_work+0x2c8/0x700 [ 8364.467934] worker_thread+0x296/0x6e0 [ 8364.469471] ? rescuer_thread+0x570/0x570 [ 8364.470905] kthread+0x1d1/0x200 [ 8364.472183] ? set_kthread_struct+0x70/0x70 [ 8364.473743] ret_from_fork+0x1f/0x30 [ 8367.075559] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5729s! [ 8367.081776] Showing busy workqueues and worker pools: [ 8367.085270] workqueue events: flags=0x0 [ 8367.086915] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8367.091925] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8367.102499] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8367.106983] pending: cpu_latency_work [ptlrpc] [ 8367.113417] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8367.118634] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8367.130816] workqueue events_long: flags=0x0 [ 8367.137122] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8367.143066] pending: lu_objects_flush [obdclass] [ 8367.146898] workqueue events_unbound: flags=0x2 [ 8367.150731] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8367.154833] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8367.161808] workqueue events_power_efficient: flags=0x80 [ 8367.168242] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8367.174788] pending: check_lifetime, do_cache_clean [ 8367.179130] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8367.186503] pending: neigh_periodic_work, neigh_periodic_work [ 8367.189708] workqueue mm_percpu_wq: flags=0x8 [ 8367.192436] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8367.195906] pending: vmstat_update, lru_add_drain_per_cpu [ 8367.201795] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8367.207798] pending: vmstat_update, lru_add_drain_per_cpu [ 8367.212920] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8367.221661] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8367.229387] workqueue xprtiod: flags=0xa [ 8367.232107] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8367.235746] pending: xprt_autoclose, xs_error_handle [ 8367.238494] workqueue ipv6_addrconf: flags=0x40008 [ 8367.240785] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8367.244355] pending: addrconf_verify_work [ 8367.246295] workqueue knbd0-recv: flags=0x1a [ 8367.247952] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8367.253939] in-flight: 651:recv_work [ 8367.256868] workqueue ptlrpc_pinger: flags=0x2 [ 8367.262337] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8367.266914] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8367.272731] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5730s workers=4 idle: 15979 31824 8 [ 8367.280094] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8357s workers=2 idle: 654 [ 8384.055097] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 8384.065872] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 8384.068412] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8384.068488] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8384.068495] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8384.068498] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8384.068511] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8384.068516] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8384.068520] RAX: 0000000000007c0b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8384.068523] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8384.068525] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8384.068528] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8384.068531] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8384.068534] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8384.068537] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8384.077438] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8384.077515] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8384.077523] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8384.077525] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8384.077539] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8384.077544] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8384.077548] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8384.077551] RDX: 00000000000004be RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8384.077553] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8384.077555] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8384.077557] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8384.077559] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8384.077562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8384.077564] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8384.077573] Call Trace: [ 8384.077578] [ 8384.077585] ? show_regs.cold.9+0x22/0x2f [ 8384.077594] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8384.077599] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8384.077603] ? watchdog+0x40/0x40 [ 8384.077608] ? hrtimer_interrupt+0x128/0x2c0 [ 8384.077612] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8384.077620] ? apic_timer_interrupt+0xf/0x20 [ 8384.077626] [ 8384.077628] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8384.077633] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8384.077637] do_raw_spin_lock+0x177/0x190 [ 8384.077642] _raw_spin_lock+0x21/0x30 [ 8384.077647] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8384.077913] obd_mod_free+0x46/0x140 [ptlrpc] [ 8384.078110] mdc_commit_open+0x91/0xb0 [mdc] [ 8384.078148] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8384.078343] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8384.078531] after_reply+0xe2c/0x1780 [ptlrpc] [ 8384.087656] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8384.089093] ? __next_timer_interrupt+0x160/0x160 [ 8384.089104] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8384.094850] ? do_wait_intr+0xf0/0xf0 [ 8384.094864] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8384.167609] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8384.167624] Call Trace: [ 8384.167632] [ 8384.167637] ? show_regs.cold.9+0x22/0x2f [ 8384.167649] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8384.167656] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8384.167659] ? watchdog+0x40/0x40 [ 8384.167663] ? hrtimer_interrupt+0x128/0x2c0 [ 8384.167667] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8384.167676] ? apic_timer_interrupt+0xf/0x20 [ 8384.167681] [ 8384.167684] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8384.167694] do_raw_spin_lock+0x177/0x190 [ 8384.167699] _raw_spin_lock+0x21/0x30 [ 8384.167719] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8384.167996] ? do_raw_spin_unlock+0x75/0x190 [ 8384.168002] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8384.168613] ? do_wait_intr+0xf0/0xf0 [ 8384.168623] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8384.169013] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8384.169522] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8384.169946] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8384.170087] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8384.170209] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8384.170375] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8384.170402] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8384.170508] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8384.191026] kthread+0x1d1/0x200 [ 8384.191046] ? set_kthread_struct+0x70/0x70 [ 8384.191051] ret_from_fork+0x1f/0x30 [ 8385.082318] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8385.089437] lmv_intent_open+0x338/0x1150 [lmv] [ 8385.092593] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8385.103676] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8385.109377] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8385.117667] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8385.126661] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8385.136330] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8385.147279] ? do_raw_spin_unlock+0x75/0x190 [ 8385.158129] ? _raw_spin_unlock+0x12/0x30 [ 8385.162453] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8385.171572] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8385.177149] ll_atomic_open+0x36a/0xd70 [lustre] [ 8385.187791] lookup_open+0x356/0xb40 [ 8385.190153] do_last+0x313/0xfc0 [ 8385.191274] ? nd_jump_root+0xe5/0x160 [ 8385.193302] ? path_init+0x437/0x520 [ 8385.201653] path_openat+0xf7/0x500 [ 8385.208963] ? do_raw_spin_unlock+0x75/0x190 [ 8385.222509] do_filp_open+0x99/0x140 [ 8385.223843] ? getname_flags+0x6e/0x330 [ 8385.236970] ? __check_object_size+0xff/0x256 [ 8385.244622] ? do_raw_spin_unlock+0x75/0x190 [ 8385.257616] ? _raw_spin_unlock+0x12/0x30 [ 8385.267104] do_sys_openat2+0x2b4/0x410 [ 8385.273940] do_sys_open+0x73/0xa0 [ 8385.287217] __x64_sys_openat+0x24/0x30 [ 8385.290197] do_syscall_64+0xc1/0x440 [ 8385.296581] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8385.303995] RIP: 0033:0x7f53425a5332 [ 8385.307595] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8385.352833] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8385.366790] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8385.381527] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8385.399673] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8385.410984] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8385.415529] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8392.052112] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8392.057172] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8392.062759] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 8392.086832] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8392.086847] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8392.086854] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8392.087166] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8392.087177] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8392.087182] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8392.087186] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8392.087188] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8392.087191] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8392.087193] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8392.087195] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8392.087199] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8392.087203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8392.087205] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8392.087215] Call Trace: [ 8392.087220] [ 8392.087225] ? show_regs.cold.9+0x22/0x2f [ 8392.087235] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8392.087242] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8392.087246] ? watchdog+0x40/0x40 [ 8392.087250] ? hrtimer_interrupt+0x128/0x2c0 [ 8392.087254] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8392.087261] ? apic_timer_interrupt+0xf/0x20 [ 8392.087265] [ 8392.087268] ? native_safe_halt+0x2e/0x50 [ 8392.087271] kvm_wait+0xcd/0xe0 [ 8392.087279] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8392.087286] ? newidle_balance.isra.93+0x207/0x580 [ 8392.087292] do_raw_spin_lock+0x177/0x190 [ 8392.087296] _raw_spin_lock+0x21/0x30 [ 8392.087300] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8392.087507] process_one_work+0x2c8/0x700 [ 8392.087515] worker_thread+0x296/0x6e0 [ 8392.087520] ? rescuer_thread+0x570/0x570 [ 8392.087523] kthread+0x1d1/0x200 [ 8392.087527] ? set_kthread_struct+0x70/0x70 [ 8392.087531] ret_from_fork+0x1f/0x30 [ 8392.233376] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8392.258860] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8392.265729] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8392.270283] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8392.273410] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8392.285456] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8392.289285] RAX: 0000000000007f87 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8392.293392] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8392.297223] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8392.300846] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8392.304068] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8392.308121] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8392.313725] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8392.316638] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8392.320323] Call Trace: [ 8392.321479] [ 8392.322967] ? show_regs.cold.9+0x22/0x2f [ 8392.324897] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8392.327859] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8392.330359] ? watchdog+0x40/0x40 [ 8392.332545] ? hrtimer_interrupt+0x128/0x2c0 [ 8392.334697] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8392.337190] ? apic_timer_interrupt+0xf/0x20 [ 8392.339723] [ 8392.340922] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8392.343479] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8392.346395] do_raw_spin_lock+0x177/0x190 [ 8392.348244] _raw_spin_lock+0x21/0x30 [ 8392.352805] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8392.356906] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8392.359524] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8392.362491] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8392.366324] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8392.369008] ? woken_wake_function+0x30/0x30 [ 8392.370728] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8392.373708] kthread+0x1d1/0x200 [ 8392.375183] ? set_kthread_struct+0x70/0x70 [ 8392.377111] ret_from_fork+0x1f/0x30 [ 8397.792230] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5760s! [ 8397.798412] Showing busy workqueues and worker pools: [ 8397.803930] workqueue events: flags=0x0 [ 8397.805901] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8397.810316] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8397.820153] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8397.828175] pending: cpu_latency_work [ptlrpc] [ 8397.830250] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8397.846500] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8397.861323] workqueue events_long: flags=0x0 [ 8397.869288] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8397.883974] pending: lu_objects_flush [obdclass] [ 8397.890453] workqueue events_unbound: flags=0x2 [ 8397.896918] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8397.906054] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8397.908157] workqueue events_power_efficient: flags=0x80 [ 8397.925203] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8397.940971] pending: check_lifetime, do_cache_clean [ 8397.954355] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8397.965363] pending: neigh_periodic_work, neigh_periodic_work [ 8397.968821] workqueue mm_percpu_wq: flags=0x8 [ 8397.981432] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8397.995666] pending: vmstat_update, lru_add_drain_per_cpu [ 8398.002852] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8398.022064] pending: vmstat_update, lru_add_drain_per_cpu [ 8398.033601] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8398.046507] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8398.056314] workqueue xprtiod: flags=0xa [ 8398.063774] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8398.092149] pending: xprt_autoclose, xs_error_handle [ 8398.104395] workqueue ipv6_addrconf: flags=0x40008 [ 8398.106147] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8398.120957] pending: addrconf_verify_work [ 8398.136480] workqueue knbd0-recv: flags=0x1a [ 8398.138088] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8398.157452] in-flight: 651:recv_work [ 8398.165722] workqueue ptlrpc_pinger: flags=0x2 [ 8398.174079] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8398.184081] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8398.193137] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5761s workers=4 idle: 15979 31824 8 [ 8398.207222] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8388s workers=2 idle: 654 [ 8412.056826] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 8412.057948] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 8412.061884] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8412.061961] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8412.061969] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8412.061972] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8412.061985] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8412.061989] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8412.061994] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8412.061997] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8412.061999] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8412.062001] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8412.062004] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8412.062007] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8412.062010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8412.062012] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8412.062021] Call Trace: [ 8412.062027] [ 8412.062034] ? show_regs.cold.9+0x22/0x2f [ 8412.062045] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8412.062053] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8412.062057] ? watchdog+0x40/0x40 [ 8412.062062] ? hrtimer_interrupt+0x128/0x2c0 [ 8412.062066] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8412.062072] ? apic_timer_interrupt+0xf/0x20 [ 8412.062077] [ 8412.062079] ? native_safe_halt+0x2e/0x50 [ 8412.062083] kvm_wait+0xcd/0xe0 [ 8412.062091] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8412.062099] do_raw_spin_lock+0x177/0x190 [ 8412.062103] _raw_spin_lock+0x21/0x30 [ 8412.062107] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8412.062867] ? do_raw_spin_unlock+0x75/0x190 [ 8412.062875] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8412.063103] ? do_wait_intr+0xf0/0xf0 [ 8412.063113] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8412.068091] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8412.068170] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8412.068179] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8412.068182] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8412.068196] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8412.068200] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8412.068204] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8412.068207] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8412.068209] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8412.068211] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8412.068213] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8412.068228] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8412.068231] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8412.068234] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8412.068242] Call Trace: [ 8412.068248] [ 8412.068256] ? show_regs.cold.9+0x22/0x2f [ 8412.068267] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8412.068275] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8412.068280] ? watchdog+0x40/0x40 [ 8412.068284] ? hrtimer_interrupt+0x128/0x2c0 [ 8412.068288] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8412.068295] ? apic_timer_interrupt+0xf/0x20 [ 8412.068300] [ 8412.068304] ? native_safe_halt+0x2e/0x50 [ 8412.068307] kvm_wait+0xcd/0xe0 [ 8412.068314] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8412.068322] do_raw_spin_lock+0x177/0x190 [ 8412.068326] _raw_spin_lock+0x21/0x30 [ 8412.068331] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8412.068624] obd_mod_free+0x46/0x140 [ptlrpc] [ 8412.068819] mdc_commit_open+0x91/0xb0 [mdc] [ 8412.068857] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8412.116367] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8412.124974] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8412.126436] after_reply+0xe2c/0x1780 [ptlrpc] [ 8412.126737] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8412.127177] ? __next_timer_interrupt+0x160/0x160 [ 8412.127190] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8412.127952] ? do_wait_intr+0xf0/0xf0 [ 8412.127966] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8412.128175] kthread+0x1d1/0x200 [ 8412.128181] ? set_kthread_struct+0x70/0x70 [ 8412.128185] ret_from_fork+0x1f/0x30 [ 8412.486739] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8412.488494] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8412.490686] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8412.492088] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8412.493626] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8412.494897] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8412.496484] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8412.498356] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8412.499745] lmv_intent_open+0x338/0x1150 [lmv] [ 8412.500967] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8412.502547] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8412.504011] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8412.505517] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8412.506740] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8412.508423] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8412.509749] ? do_raw_spin_unlock+0x75/0x190 [ 8412.510988] ? _raw_spin_unlock+0x12/0x30 [ 8412.512185] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8412.513706] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8412.515379] ll_atomic_open+0x36a/0xd70 [lustre] [ 8412.517003] lookup_open+0x356/0xb40 [ 8412.517961] do_last+0x313/0xfc0 [ 8412.518802] ? nd_jump_root+0xe5/0x160 [ 8412.520182] ? path_init+0x437/0x520 [ 8412.521186] path_openat+0xf7/0x500 [ 8412.522183] ? do_raw_spin_unlock+0x75/0x190 [ 8412.523326] do_filp_open+0x99/0x140 [ 8412.524226] ? getname_flags+0x6e/0x330 [ 8412.525161] ? __check_object_size+0xff/0x256 [ 8412.526326] ? do_raw_spin_unlock+0x75/0x190 [ 8412.527400] ? _raw_spin_unlock+0x12/0x30 [ 8412.528424] do_sys_openat2+0x2b4/0x410 [ 8412.529396] do_sys_open+0x73/0xa0 [ 8412.530386] __x64_sys_openat+0x24/0x30 [ 8412.531418] do_syscall_64+0xc1/0x440 [ 8412.532398] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8412.533636] RIP: 0033:0x7f53425a5332 [ 8412.534538] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8412.539797] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8412.541693] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8412.543517] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8412.545319] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8412.547093] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8412.548778] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8420.058745] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 8420.066637] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8420.114349] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8420.129957] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8420.140345] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8420.147039] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8420.170894] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8420.175961] RAX: 0000000000005c65 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8420.193711] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8420.202641] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8420.210990] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8420.218226] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8420.220721] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8420.228803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8420.234921] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8420.245854] Call Trace: [ 8420.248943] [ 8420.249943] ? show_regs.cold.9+0x22/0x2f [ 8420.261980] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8420.264980] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8420.270811] ? watchdog+0x40/0x40 [ 8420.278245] ? hrtimer_interrupt+0x128/0x2c0 [ 8420.280919] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8420.289070] ? apic_timer_interrupt+0xf/0x20 [ 8420.301343] [ 8420.302232] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8420.310011] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8420.313013] do_raw_spin_lock+0x177/0x190 [ 8420.317160] _raw_spin_lock+0x21/0x30 [ 8420.319698] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8420.328211] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8420.331469] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8420.346556] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8420.350917] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8420.355774] ? woken_wake_function+0x30/0x30 [ 8420.359474] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8420.363706] kthread+0x1d1/0x200 [ 8420.369456] ? set_kthread_struct+0x70/0x70 [ 8420.370876] ret_from_fork+0x1f/0x30 [ 8424.054298] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8424.057868] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8424.103864] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8424.111434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8424.118230] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8424.128324] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8424.136852] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8424.149081] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8424.151683] RAX: 0000000000007420 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8424.159328] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8424.169047] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8424.173964] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8424.184770] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8424.187791] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8424.191488] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8424.194196] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8424.197292] Call Trace: [ 8424.198457] [ 8424.203677] ? show_regs.cold.9+0x22/0x2f [ 8424.205108] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8424.209155] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8424.210797] ? watchdog+0x40/0x40 [ 8424.219366] ? hrtimer_interrupt+0x128/0x2c0 [ 8424.221585] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8424.223626] ? apic_timer_interrupt+0xf/0x20 [ 8424.226675] [ 8424.227373] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8424.234443] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8424.237953] ? newidle_balance.isra.93+0x207/0x580 [ 8424.246258] do_raw_spin_lock+0x177/0x190 [ 8424.247879] _raw_spin_lock+0x21/0x30 [ 8424.249831] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8424.253323] process_one_work+0x2c8/0x700 [ 8424.264565] worker_thread+0x296/0x6e0 [ 8424.270279] ? rescuer_thread+0x570/0x570 [ 8424.271905] kthread+0x1d1/0x200 [ 8424.275801] ? set_kthread_struct+0x70/0x70 [ 8424.280722] ret_from_fork+0x1f/0x30 [ 8428.511779] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5791s! [ 8428.516613] Showing busy workqueues and worker pools: [ 8428.534863] workqueue events: flags=0x0 [ 8428.539024] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8428.550190] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8428.561201] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8428.573172] pending: cpu_latency_work [ptlrpc] [ 8428.586903] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8428.589113] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8428.603632] workqueue events_long: flags=0x0 [ 8428.608909] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8428.614406] pending: lu_objects_flush [obdclass] [ 8428.625557] workqueue events_unbound: flags=0x2 [ 8428.629539] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8428.631646] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8428.647440] workqueue events_power_efficient: flags=0x80 [ 8428.658259] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8428.665489] pending: check_lifetime, do_cache_clean [ 8428.668369] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8428.674441] pending: neigh_periodic_work, neigh_periodic_work [ 8428.678351] workqueue mm_percpu_wq: flags=0x8 [ 8428.682158] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8428.685433] pending: vmstat_update, lru_add_drain_per_cpu [ 8428.695744] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8428.703227] pending: vmstat_update, lru_add_drain_per_cpu [ 8428.712397] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8428.719910] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8428.727017] workqueue xprtiod: flags=0xa [ 8428.729289] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8428.740503] pending: xprt_autoclose, xs_error_handle [ 8428.746853] workqueue ipv6_addrconf: flags=0x40008 [ 8428.748473] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8428.753488] pending: addrconf_verify_work [ 8428.756134] workqueue knbd0-recv: flags=0x1a [ 8428.759769] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8428.765888] in-flight: 651:recv_work [ 8428.768361] workqueue ptlrpc_pinger: flags=0x2 [ 8428.769996] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8428.772391] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8428.777471] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5791s workers=4 idle: 15979 31824 8 [ 8428.789421] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8419s workers=2 idle: 654 [ 8440.057210] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 8440.062394] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8440.062893] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 8440.113153] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8440.113169] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8440.113172] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8440.113187] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8440.113191] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8440.113198] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8440.113200] RDX: 00000000000043c2 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8440.113202] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8440.113204] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8440.113206] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8440.113209] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8440.113212] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8440.113214] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8440.113223] Call Trace: [ 8440.113228] [ 8440.113234] ? show_regs.cold.9+0x22/0x2f [ 8440.113243] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8440.113249] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8440.113254] ? watchdog+0x40/0x40 [ 8440.113258] ? hrtimer_interrupt+0x128/0x2c0 [ 8440.113262] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8440.113271] ? apic_timer_interrupt+0xf/0x20 [ 8440.113276] [ 8440.113279] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8440.113283] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8440.113288] do_raw_spin_lock+0x177/0x190 [ 8440.113294] _raw_spin_lock+0x21/0x30 [ 8440.113301] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8440.113579] obd_mod_free+0x46/0x140 [ptlrpc] [ 8440.119667] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8440.119758] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8440.119766] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8440.119770] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8440.119783] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8440.119788] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8440.119793] RAX: 0000000000000a1a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8440.119795] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8440.119797] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8440.119800] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8440.119803] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8440.119806] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8440.119809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8440.119812] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8440.119821] Call Trace: [ 8440.119826] [ 8440.119833] ? show_regs.cold.9+0x22/0x2f [ 8440.119843] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8440.119851] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8440.119855] ? watchdog+0x40/0x40 [ 8440.119860] ? hrtimer_interrupt+0x128/0x2c0 [ 8440.125862] mdc_commit_open+0x91/0xb0 [mdc] [ 8440.125910] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8440.126255] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8440.126482] after_reply+0xe2c/0x1780 [ptlrpc] [ 8440.126903] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8440.127177] ? __next_timer_interrupt+0x160/0x160 [ 8440.127188] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8440.127389] ? do_wait_intr+0xf0/0xf0 [ 8440.142590] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8440.142614] ? apic_timer_interrupt+0xf/0x20 [ 8440.142619] [ 8440.142624] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8440.142635] do_raw_spin_lock+0x177/0x190 [ 8440.142642] _raw_spin_lock+0x21/0x30 [ 8440.142648] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8440.143378] ? do_raw_spin_unlock+0x75/0x190 [ 8440.143384] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8440.143591] ? do_wait_intr+0xf0/0xf0 [ 8440.143601] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8440.152277] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8440.152602] kthread+0x1d1/0x200 [ 8440.152611] ? set_kthread_struct+0x70/0x70 [ 8440.152615] ret_from_fork+0x1f/0x30 [ 8440.556076] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8440.558602] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8440.562096] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8440.564857] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8440.568530] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8440.571072] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8440.573874] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8440.576022] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8440.583114] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8440.589460] lmv_intent_open+0x338/0x1150 [lmv] [ 8440.592588] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8440.597894] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8440.601197] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8440.603400] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8440.606137] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8440.608534] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8440.611102] ? do_raw_spin_unlock+0x75/0x190 [ 8440.613101] ? _raw_spin_unlock+0x12/0x30 [ 8440.614886] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8440.617235] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8440.619641] ll_atomic_open+0x36a/0xd70 [lustre] [ 8440.621894] lookup_open+0x356/0xb40 [ 8440.623495] do_last+0x313/0xfc0 [ 8440.625072] ? nd_jump_root+0xe5/0x160 [ 8440.626622] ? path_init+0x437/0x520 [ 8440.628520] path_openat+0xf7/0x500 [ 8440.633277] ? do_raw_spin_unlock+0x75/0x190 [ 8440.636587] do_filp_open+0x99/0x140 [ 8440.638402] ? getname_flags+0x6e/0x330 [ 8440.641256] ? __check_object_size+0xff/0x256 [ 8440.648334] ? do_raw_spin_unlock+0x75/0x190 [ 8440.651185] ? _raw_spin_unlock+0x12/0x30 [ 8440.654140] do_sys_openat2+0x2b4/0x410 [ 8440.657071] do_sys_open+0x73/0xa0 [ 8440.659341] __x64_sys_openat+0x24/0x30 [ 8440.665735] do_syscall_64+0xc1/0x440 [ 8440.671464] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8440.676481] RIP: 0033:0x7f53425a5332 [ 8440.680318] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8440.692226] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8440.697417] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8440.701521] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8440.708588] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8440.711819] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8440.716199] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8448.060584] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 8448.074320] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8448.122478] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8448.130483] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8448.139289] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8448.144567] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8448.173921] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8448.178922] RAX: 0000000000007e84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8448.183431] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8448.187016] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8448.190454] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8448.194569] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8448.204469] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8448.212072] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8448.215760] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8448.221438] Call Trace: [ 8448.222976] [ 8448.224287] ? show_regs.cold.9+0x22/0x2f [ 8448.226662] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8448.229572] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8448.232314] ? watchdog+0x40/0x40 [ 8448.236232] ? hrtimer_interrupt+0x128/0x2c0 [ 8448.238118] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8448.241490] ? apic_timer_interrupt+0xf/0x20 [ 8448.243644] [ 8448.245571] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8448.250997] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8448.254533] do_raw_spin_lock+0x177/0x190 [ 8448.258246] _raw_spin_lock+0x21/0x30 [ 8448.259518] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8448.279020] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8448.287512] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8448.299428] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8448.306095] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8448.315952] ? woken_wake_function+0x30/0x30 [ 8448.320119] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8448.324042] kthread+0x1d1/0x200 [ 8448.325548] ? set_kthread_struct+0x70/0x70 [ 8448.329356] ret_from_fork+0x1f/0x30 [ 8454.591063] rcu: INFO: rcu_sched self-detected stall on CPU [ 8454.593284] rcu: 2-....: (3443631 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 8454.597057] (t=6360165 jiffies g=274185 q=65469) [ 8454.598931] rcu: rcu_sched kthread starved for 5818531 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 8454.603141] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8454.606802] rcu: RCU grace-period kthread stack dump: [ 8454.608953] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8454.614224] Call Trace: [ 8454.616014] __schedule+0x351/0xcb0 [ 8454.617336] schedule+0xc0/0x180 [ 8454.618770] schedule_timeout+0xb4/0x190 [ 8454.620783] ? do_raw_spin_unlock+0x75/0x190 [ 8454.623240] ? __next_timer_interrupt+0x160/0x160 [ 8454.627289] ? prepare_to_swait_event+0xf3/0x140 [ 8454.632278] rcu_gp_kthread+0x8dd/0x15b0 [ 8454.641275] ? force_qs_rnp+0x2c0/0x2c0 [ 8454.645797] kthread+0x1d1/0x200 [ 8454.649699] ? set_kthread_struct+0x70/0x70 [ 8454.654347] ret_from_fork+0x1f/0x30 [ 8454.658439] rcu: Stack dump where RCU GP kthread last ran: [ 8454.661970] Sending NMI from CPU 2 to CPUs 0: [ 8454.665047] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 8454.668441] Sending NMI from CPU 2 to CPUs 1: [ 8454.671208] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 8454.672626] NMI backtrace for cpu 2 [ 8454.683365] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8454.698226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8454.705129] Call Trace: [ 8454.706692] [ 8454.707908] ? dump_stack+0xbb/0x10e [ 8454.711131] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8454.718622] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8454.721923] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8454.730305] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8454.732713] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8454.740388] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8454.744127] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8454.748406] ? update_process_times+0x6d/0xc0 [ 8454.754552] ? tick_sched_handle.isra.18+0x27/0x90 [ 8454.760280] ? tick_sched_timer+0x53/0xb0 [ 8454.762348] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8454.770243] ? tick_sched_handle.isra.18+0x90/0x90 [ 8454.775376] ? hrtimer_interrupt+0x128/0x2c0 [ 8454.779973] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8454.784975] ? apic_timer_interrupt+0xf/0x20 [ 8454.788600] [ 8454.793776] ? native_safe_halt+0x2e/0x50 [ 8454.795758] ? kvm_wait+0xcd/0xe0 [ 8454.803201] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8454.811171] ? do_raw_spin_lock+0x177/0x190 [ 8454.815925] ? _raw_spin_lock+0x21/0x30 [ 8454.817246] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8454.821746] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8454.828397] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8454.830675] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8454.836945] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8454.845702] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8454.852751] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8454.857806] ? __next_timer_interrupt+0x160/0x160 [ 8454.860387] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8454.862762] ? do_wait_intr+0xf0/0xf0 [ 8454.864504] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8454.866967] ? kthread+0x1d1/0x200 [ 8454.868171] ? set_kthread_struct+0x70/0x70 [ 8454.869896] ? ret_from_fork+0x1f/0x30 [ 8459.238039] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5822s! [ 8459.243050] Showing busy workqueues and worker pools: [ 8459.245740] workqueue events: flags=0x0 [ 8459.247198] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8459.250683] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8459.261313] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8459.267093] pending: cpu_latency_work [ptlrpc] [ 8459.270125] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8459.274039] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8459.278364] workqueue events_long: flags=0x0 [ 8459.280508] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8459.282812] pending: lu_objects_flush [obdclass] [ 8459.285041] workqueue events_unbound: flags=0x2 [ 8459.286739] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8459.289698] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8459.292014] workqueue events_power_efficient: flags=0x80 [ 8459.294247] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8459.297343] pending: check_lifetime, do_cache_clean [ 8459.307894] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8459.316851] pending: neigh_periodic_work, neigh_periodic_work [ 8459.320300] workqueue mm_percpu_wq: flags=0x8 [ 8459.322616] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8459.328057] pending: vmstat_update, lru_add_drain_per_cpu [ 8459.331254] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8459.336948] pending: vmstat_update, lru_add_drain_per_cpu [ 8459.340044] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8459.347308] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8459.352836] workqueue xprtiod: flags=0xa [ 8459.354309] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8459.359433] pending: xprt_autoclose, xs_error_handle [ 8459.368373] workqueue ipv6_addrconf: flags=0x40008 [ 8459.370580] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8459.376925] pending: addrconf_verify_work [ 8459.378749] workqueue knbd0-recv: flags=0x1a [ 8459.380699] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8459.383636] in-flight: 651:recv_work [ 8459.385714] workqueue ptlrpc_pinger: flags=0x2 [ 8459.387757] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8459.393934] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8459.401932] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5822s workers=4 idle: 15979 31824 8 [ 8459.406754] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8449s workers=2 idle: 654 [ 8476.058027] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 8476.062451] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8476.097976] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8476.106356] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8476.113747] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8476.117398] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8476.131156] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8476.136991] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8476.143398] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8476.147393] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8476.153051] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8476.160001] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8476.169703] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8476.175524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8476.178617] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8476.182759] Call Trace: [ 8476.183790] [ 8476.184616] ? show_regs.cold.9+0x22/0x2f [ 8476.188784] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8476.191919] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8476.193994] ? watchdog+0x40/0x40 [ 8476.195963] ? hrtimer_interrupt+0x128/0x2c0 [ 8476.198122] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8476.199553] ? apic_timer_interrupt+0xf/0x20 [ 8476.204352] [ 8476.205381] ? native_safe_halt+0x2e/0x50 [ 8476.207378] kvm_wait+0xcd/0xe0 [ 8476.208919] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8476.210834] do_raw_spin_lock+0x177/0x190 [ 8476.212495] _raw_spin_lock+0x21/0x30 [ 8476.213900] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8476.220377] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8476.224721] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8476.229414] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8476.234873] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8476.239814] ? woken_wake_function+0x30/0x30 [ 8476.244086] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8476.247691] kthread+0x1d1/0x200 [ 8476.249489] ? set_kthread_struct+0x70/0x70 [ 8476.252688] ret_from_fork+0x1f/0x30 [ 8480.055773] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 8480.058744] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8480.064179] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 8480.094339] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8480.094358] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8480.094361] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8480.094376] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8480.094380] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8480.094386] RAX: 0000000000004422 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8480.094388] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8480.094391] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8480.094393] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8480.094395] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8480.094398] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8480.094401] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8480.094403] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8480.094412] Call Trace: [ 8480.094418] [ 8480.094426] ? show_regs.cold.9+0x22/0x2f [ 8480.094437] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8480.094445] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8480.094450] ? watchdog+0x40/0x40 [ 8480.094454] ? hrtimer_interrupt+0x128/0x2c0 [ 8480.094458] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8480.094466] ? apic_timer_interrupt+0xf/0x20 [ 8480.094471] [ 8480.094474] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8480.094483] do_raw_spin_lock+0x177/0x190 [ 8480.094488] _raw_spin_lock+0x21/0x30 [ 8480.094493] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8480.098121] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8480.098199] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8480.098208] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8480.098211] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8480.098225] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8480.098229] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8480.098233] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8480.098235] RDX: 0000000000007905 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8480.098238] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8480.098240] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8480.098242] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8480.098246] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8480.098248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8480.098251] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8480.098260] Call Trace: [ 8480.098265] [ 8480.098272] ? show_regs.cold.9+0x22/0x2f [ 8480.098283] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8480.098289] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8480.098293] ? watchdog+0x40/0x40 [ 8480.098297] ? hrtimer_interrupt+0x128/0x2c0 [ 8480.098301] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8480.098309] ? apic_timer_interrupt+0xf/0x20 [ 8480.098313] [ 8480.098316] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8480.098321] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8480.098326] do_raw_spin_lock+0x177/0x190 [ 8480.098330] _raw_spin_lock+0x21/0x30 [ 8480.098336] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8480.098629] obd_mod_free+0x46/0x140 [ptlrpc] [ 8480.098829] mdc_commit_open+0x91/0xb0 [mdc] [ 8480.098868] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8480.099313] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8480.099514] after_reply+0xe2c/0x1780 [ptlrpc] [ 8480.099708] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8480.107459] ? do_raw_spin_unlock+0x75/0x190 [ 8480.107484] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8480.107765] ? do_wait_intr+0xf0/0xf0 [ 8480.107774] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8480.107972] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8480.108215] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8480.108445] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8480.108566] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8480.108606] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8480.108636] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8480.108666] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8480.108756] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8480.125493] ? __next_timer_interrupt+0x160/0x160 [ 8480.125530] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8480.125837] ? do_wait_intr+0xf0/0xf0 [ 8480.125853] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8480.126103] kthread+0x1d1/0x200 [ 8480.126110] ? set_kthread_struct+0x70/0x70 [ 8480.126114] ret_from_fork+0x1f/0x30 [ 8480.516626] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8480.522476] lmv_intent_open+0x338/0x1150 [lmv] [ 8480.525865] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8480.531549] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8480.539172] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8480.542147] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8480.544448] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8480.556445] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8480.558237] ? do_raw_spin_unlock+0x75/0x190 [ 8480.559497] ? _raw_spin_unlock+0x12/0x30 [ 8480.566270] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8480.577543] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8480.579774] ll_atomic_open+0x36a/0xd70 [lustre] [ 8480.582576] lookup_open+0x356/0xb40 [ 8480.584981] do_last+0x313/0xfc0 [ 8480.589334] ? nd_jump_root+0xe5/0x160 [ 8480.590895] ? path_init+0x437/0x520 [ 8480.592731] path_openat+0xf7/0x500 [ 8480.596205] ? do_raw_spin_unlock+0x75/0x190 [ 8480.601944] do_filp_open+0x99/0x140 [ 8480.606253] ? getname_flags+0x6e/0x330 [ 8480.608876] ? __check_object_size+0xff/0x256 [ 8480.612152] ? do_raw_spin_unlock+0x75/0x190 [ 8480.613886] ? _raw_spin_unlock+0x12/0x30 [ 8480.616564] do_sys_openat2+0x2b4/0x410 [ 8480.622797] do_sys_open+0x73/0xa0 [ 8480.627904] __x64_sys_openat+0x24/0x30 [ 8480.632840] do_syscall_64+0xc1/0x440 [ 8480.636959] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8480.646893] RIP: 0033:0x7f53425a5332 [ 8480.650402] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8480.669552] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8480.673975] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8480.680130] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8480.682636] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8480.689205] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8480.696806] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8484.053303] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8484.067119] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8484.120312] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8484.125060] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8484.130816] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8484.135851] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8484.140308] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8484.153046] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8484.162670] RAX: 0000000000007c7d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8484.170506] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8484.186522] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8484.194090] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8484.197738] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8484.207750] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8484.220371] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8484.225051] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8484.226934] Call Trace: [ 8484.232418] [ 8484.233488] ? show_regs.cold.9+0x22/0x2f [ 8484.236541] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8484.239467] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8484.247224] ? watchdog+0x40/0x40 [ 8484.250333] ? hrtimer_interrupt+0x128/0x2c0 [ 8484.255931] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8484.266662] ? apic_timer_interrupt+0xf/0x20 [ 8484.268657] [ 8484.270446] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8484.278521] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8484.281212] ? newidle_balance.isra.93+0x207/0x580 [ 8484.287926] do_raw_spin_lock+0x177/0x190 [ 8484.289018] _raw_spin_lock+0x21/0x30 [ 8484.296616] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8484.302920] process_one_work+0x2c8/0x700 [ 8484.306464] worker_thread+0x296/0x6e0 [ 8484.308040] ? rescuer_thread+0x570/0x570 [ 8484.311714] kthread+0x1d1/0x200 [ 8484.312629] ? set_kthread_struct+0x70/0x70 [ 8484.322506] ret_from_fork+0x1f/0x30 [ 8489.440041] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5852s! [ 8489.444491] Showing busy workqueues and worker pools: [ 8489.447642] workqueue events: flags=0x0 [ 8489.456485] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8489.468908] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8489.482597] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8489.494718] pending: cpu_latency_work [ptlrpc] [ 8489.497546] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8489.501597] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8489.506564] workqueue events_long: flags=0x0 [ 8489.508744] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8489.514459] pending: lu_objects_flush [obdclass] [ 8489.517666] workqueue events_unbound: flags=0x2 [ 8489.520164] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8489.523644] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8489.527294] workqueue events_power_efficient: flags=0x80 [ 8489.529168] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8489.532590] pending: check_lifetime, do_cache_clean [ 8489.535052] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8489.538388] pending: neigh_periodic_work, neigh_periodic_work [ 8489.541339] workqueue mm_percpu_wq: flags=0x8 [ 8489.543404] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8489.547672] pending: vmstat_update, lru_add_drain_per_cpu [ 8489.550574] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8489.557723] pending: vmstat_update, lru_add_drain_per_cpu [ 8489.564071] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8489.575935] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8489.583742] workqueue xprtiod: flags=0xa [ 8489.593050] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8489.597918] pending: xprt_autoclose, xs_error_handle [ 8489.599712] workqueue ipv6_addrconf: flags=0x40008 [ 8489.613368] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8489.622229] pending: addrconf_verify_work [ 8489.628574] workqueue knbd0-recv: flags=0x1a [ 8489.632184] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8489.634305] in-flight: 651:recv_work [ 8489.642759] workqueue ptlrpc_pinger: flags=0x2 [ 8489.647324] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8489.650349] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8489.655919] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5852s workers=4 idle: 15979 31824 8 [ 8489.659990] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8480s workers=2 idle: 654 [ 8504.058026] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 8504.061643] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8504.078255] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8504.081899] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8504.085344] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8504.087244] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8504.093843] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8504.095689] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8504.097776] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8504.100438] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8504.103129] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8504.105708] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8504.108585] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8504.111422] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8504.113915] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8504.116630] Call Trace: [ 8504.117750] [ 8504.118434] ? show_regs.cold.9+0x22/0x2f [ 8504.119835] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8504.122102] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8504.123602] ? watchdog+0x40/0x40 [ 8504.124803] ? hrtimer_interrupt+0x128/0x2c0 [ 8504.127108] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8504.129010] ? apic_timer_interrupt+0xf/0x20 [ 8504.130535] [ 8504.131660] ? native_safe_halt+0x2e/0x50 [ 8504.133197] kvm_wait+0xcd/0xe0 [ 8504.134405] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8504.136695] do_raw_spin_lock+0x177/0x190 [ 8504.138363] _raw_spin_lock+0x21/0x30 [ 8504.139796] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8504.141889] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8504.143750] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8504.146530] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8504.149029] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8504.151045] ? woken_wake_function+0x30/0x30 [ 8504.152425] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8504.154726] kthread+0x1d1/0x200 [ 8504.156200] ? set_kthread_struct+0x70/0x70 [ 8504.157702] ret_from_fork+0x1f/0x30 [ 8508.055031] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 8508.057058] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 8508.057715] Modules linked in: lustre(O) osp(O) [ 8508.062967] Modules linked in: [ 8508.062969] ofd(O) lod(O) mdt(O) [ 8508.064787] lustre(O) [ 8508.066212] mdd(O) [ 8508.069346] osp(O) [ 8508.070700] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8508.070770] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8508.070778] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8508.070782] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8508.070796] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8508.070801] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8508.070806] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8508.070808] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8508.070810] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8508.070812] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8508.070815] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8508.070818] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8508.070822] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8508.070824] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8508.070833] Call Trace: [ 8508.070839] [ 8508.070845] ? show_regs.cold.9+0x22/0x2f [ 8508.070855] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8508.070861] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8508.070865] ? watchdog+0x40/0x40 [ 8508.070869] ? hrtimer_interrupt+0x128/0x2c0 [ 8508.070874] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8508.070880] ? apic_timer_interrupt+0xf/0x20 [ 8508.070884] [ 8508.070887] ? native_safe_halt+0x2e/0x50 [ 8508.070891] kvm_wait+0xcd/0xe0 [ 8508.070898] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8508.070906] do_raw_spin_lock+0x177/0x190 [ 8508.070910] _raw_spin_lock+0x21/0x30 [ 8508.070914] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8508.071412] ? do_raw_spin_unlock+0x75/0x190 [ 8508.071418] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8508.071687] ? do_wait_intr+0xf0/0xf0 [ 8508.071696] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8508.071907] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8508.072261] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8508.073264] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8508.073332] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8508.073339] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8508.073342] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8508.073352] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8508.073356] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8508.073360] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8508.073362] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8508.073364] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8508.073366] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8508.073369] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8508.073372] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8508.073374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8508.073377] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8508.073386] Call Trace: [ 8508.073390] [ 8508.073394] ? show_regs.cold.9+0x22/0x2f [ 8508.073405] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8508.073413] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8508.073417] ? watchdog+0x40/0x40 [ 8508.073421] ? hrtimer_interrupt+0x128/0x2c0 [ 8508.073425] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8508.073433] ? apic_timer_interrupt+0xf/0x20 [ 8508.073438] [ 8508.073441] ? native_safe_halt+0x2e/0x50 [ 8508.073446] kvm_wait+0xcd/0xe0 [ 8508.073454] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8508.073462] do_raw_spin_lock+0x177/0x190 [ 8508.073466] _raw_spin_lock+0x21/0x30 [ 8508.073471] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8508.073727] obd_mod_free+0x46/0x140 [ptlrpc] [ 8508.073915] mdc_commit_open+0x91/0xb0 [mdc] [ 8508.073963] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8508.074151] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8508.074339] after_reply+0xe2c/0x1780 [ptlrpc] [ 8508.074532] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8508.074736] ? __next_timer_interrupt+0x160/0x160 [ 8508.074745] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8508.074945] ? do_wait_intr+0xf0/0xf0 [ 8508.074956] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8508.075712] kthread+0x1d1/0x200 [ 8508.075720] ? set_kthread_struct+0x70/0x70 [ 8508.075724] ret_from_fork+0x1f/0x30 [ 8508.398008] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8508.401047] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8508.404871] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8508.406793] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8508.408745] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8508.413439] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8508.417731] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8508.420519] lmv_intent_open+0x338/0x1150 [lmv] [ 8508.424326] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8508.426168] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8508.429972] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8508.431998] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8508.434539] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8508.438877] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8508.441382] ? do_raw_spin_unlock+0x75/0x190 [ 8508.445315] ? _raw_spin_unlock+0x12/0x30 [ 8508.446829] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8508.448925] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8508.451174] ll_atomic_open+0x36a/0xd70 [lustre] [ 8508.455516] lookup_open+0x356/0xb40 [ 8508.456463] do_last+0x313/0xfc0 [ 8508.462018] ? nd_jump_root+0xe5/0x160 [ 8508.466940] ? path_init+0x437/0x520 [ 8508.468279] path_openat+0xf7/0x500 [ 8508.470022] ? do_raw_spin_unlock+0x75/0x190 [ 8508.471432] do_filp_open+0x99/0x140 [ 8508.473964] ? getname_flags+0x6e/0x330 [ 8508.475644] ? __check_object_size+0xff/0x256 [ 8508.477857] ? do_raw_spin_unlock+0x75/0x190 [ 8508.479790] ? _raw_spin_unlock+0x12/0x30 [ 8508.481737] do_sys_openat2+0x2b4/0x410 [ 8508.483776] do_sys_open+0x73/0xa0 [ 8508.487179] __x64_sys_openat+0x24/0x30 [ 8508.489488] do_syscall_64+0xc1/0x440 [ 8508.491425] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8508.493917] RIP: 0033:0x7f53425a5332 [ 8508.497373] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8508.511793] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8508.517262] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8508.521157] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8508.528049] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8508.536106] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8508.544967] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8516.052517] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 8516.058535] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8516.094382] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8516.100881] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8516.109032] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8516.116060] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8516.124914] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8516.146301] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8516.153382] RAX: 0000000000006a0e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8516.161012] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8516.169575] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8516.177736] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8516.189832] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8516.194236] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8516.198711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8516.201925] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8516.204904] Call Trace: [ 8516.206191] [ 8516.207066] ? show_regs.cold.9+0x22/0x2f [ 8516.210020] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8516.212901] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8516.215280] ? watchdog+0x40/0x40 [ 8516.222002] ? hrtimer_interrupt+0x128/0x2c0 [ 8516.224307] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8516.230908] ? apic_timer_interrupt+0xf/0x20 [ 8516.235466] [ 8516.238811] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8516.243099] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8516.246770] ? newidle_balance.isra.93+0x207/0x580 [ 8516.249701] do_raw_spin_lock+0x177/0x190 [ 8516.257325] _raw_spin_lock+0x21/0x30 [ 8516.260138] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8516.262142] process_one_work+0x2c8/0x700 [ 8516.265335] worker_thread+0x296/0x6e0 [ 8516.267346] ? rescuer_thread+0x570/0x570 [ 8516.269322] kthread+0x1d1/0x200 [ 8516.270801] ? set_kthread_struct+0x70/0x70 [ 8516.272735] ret_from_fork+0x1f/0x30 [ 8520.160131] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5883s! [ 8520.163880] Showing busy workqueues and worker pools: [ 8520.170487] workqueue events: flags=0x0 [ 8520.172044] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8520.183536] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8520.198602] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8520.206916] pending: cpu_latency_work [ptlrpc] [ 8520.212700] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8520.217860] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8520.236367] workqueue events_long: flags=0x0 [ 8520.243680] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8520.253166] pending: lu_objects_flush [obdclass] [ 8520.258307] workqueue events_unbound: flags=0x2 [ 8520.263905] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8520.265857] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8520.274418] workqueue events_power_efficient: flags=0x80 [ 8520.278088] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8520.292030] pending: check_lifetime, do_cache_clean [ 8520.293945] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8520.298876] pending: neigh_periodic_work, neigh_periodic_work [ 8520.303592] workqueue mm_percpu_wq: flags=0x8 [ 8520.305531] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8520.311057] pending: vmstat_update, lru_add_drain_per_cpu [ 8520.316183] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8520.321339] pending: vmstat_update, lru_add_drain_per_cpu [ 8520.326340] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8520.329797] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8520.332579] workqueue xprtiod: flags=0xa [ 8520.334628] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8520.337480] pending: xprt_autoclose, xs_error_handle [ 8520.339715] workqueue ipv6_addrconf: flags=0x40008 [ 8520.341806] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8520.344523] pending: addrconf_verify_work [ 8520.346306] workqueue knbd0-recv: flags=0x1a [ 8520.348066] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8520.350547] in-flight: 651:recv_work [ 8520.352096] workqueue ptlrpc_pinger: flags=0x2 [ 8520.354292] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8520.356838] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8520.359718] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5883s workers=4 idle: 15979 31824 8 [ 8520.362961] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8510s workers=2 idle: 654 [ 8532.067754] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 8532.073316] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8532.121579] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8532.129977] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8532.135014] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8532.138313] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8532.157205] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8532.165673] RAX: 0000000000007f84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8532.169321] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8532.174023] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8532.177941] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8532.182008] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8532.185668] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8532.190926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8532.193869] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8532.198470] Call Trace: [ 8532.199317] [ 8532.199959] ? show_regs.cold.9+0x22/0x2f [ 8532.202713] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8532.204926] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8532.207070] ? watchdog+0x40/0x40 [ 8532.209158] ? hrtimer_interrupt+0x128/0x2c0 [ 8532.213812] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8532.216869] ? apic_timer_interrupt+0xf/0x20 [ 8532.219890] [ 8532.220843] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8532.223422] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8532.225946] do_raw_spin_lock+0x177/0x190 [ 8532.227979] _raw_spin_lock+0x21/0x30 [ 8532.229674] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8532.233142] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8532.236507] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8532.239295] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8532.243453] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8532.246378] ? woken_wake_function+0x30/0x30 [ 8532.248248] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8532.250699] kthread+0x1d1/0x200 [ 8532.252914] ? set_kthread_struct+0x70/0x70 [ 8532.255633] ret_from_fork+0x1f/0x30 [ 8536.055131] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 8536.057025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 8536.058284] Modules linked in: lustre(O) [ 8536.060914] Modules linked in: [ 8536.060916] osp(O) ofd(O) [ 8536.062472] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8536.062547] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8536.062555] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8536.062558] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8536.062572] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8536.062576] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8536.062581] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8536.062583] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8536.062586] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8536.062588] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8536.062591] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8536.062594] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8536.062597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8536.062599] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8536.062608] Call Trace: [ 8536.062614] [ 8536.062620] ? show_regs.cold.9+0x22/0x2f [ 8536.062629] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8536.062636] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8536.062640] ? watchdog+0x40/0x40 [ 8536.062645] ? hrtimer_interrupt+0x128/0x2c0 [ 8536.062649] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8536.062657] ? apic_timer_interrupt+0xf/0x20 [ 8536.062662] [ 8536.062665] ? native_safe_halt+0x2e/0x50 [ 8536.062669] kvm_wait+0xcd/0xe0 [ 8536.062676] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8536.062684] do_raw_spin_lock+0x177/0x190 [ 8536.062688] _raw_spin_lock+0x21/0x30 [ 8536.062694] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8536.062967] obd_mod_free+0x46/0x140 [ptlrpc] [ 8536.063189] mdc_commit_open+0x91/0xb0 [mdc] [ 8536.063234] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8536.063437] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8536.063641] after_reply+0xe2c/0x1780 [ptlrpc] [ 8536.063841] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8536.064034] ? __next_timer_interrupt+0x160/0x160 [ 8536.064044] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8536.064260] ? do_wait_intr+0xf0/0xf0 [ 8536.064270] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8536.064471] kthread+0x1d1/0x200 [ 8536.064477] ? set_kthread_struct+0x70/0x70 [ 8536.064482] ret_from_fork+0x1f/0x30 [ 8536.342221] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8536.387459] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8536.393943] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8536.399662] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8536.403726] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8536.418096] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 8536.421754] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8536.435303] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8536.443499] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8536.465347] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8536.467618] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8536.470867] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8536.483299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8536.495096] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8536.503638] Call Trace: [ 8536.510070] [ 8536.510759] ? show_regs.cold.9+0x22/0x2f [ 8536.514628] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8536.520424] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8536.530227] ? watchdog+0x40/0x40 [ 8536.533548] ? hrtimer_interrupt+0x128/0x2c0 [ 8536.535771] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8536.539772] ? apic_timer_interrupt+0xf/0x20 [ 8536.543390] [ 8536.544111] ? native_safe_halt+0x2e/0x50 [ 8536.557819] kvm_wait+0xcd/0xe0 [ 8536.560640] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8536.564507] do_raw_spin_lock+0x177/0x190 [ 8536.565819] _raw_spin_lock+0x21/0x30 [ 8536.571008] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8536.580904] ? do_raw_spin_unlock+0x75/0x190 [ 8536.584824] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8536.586826] ? do_wait_intr+0xf0/0xf0 [ 8536.596951] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8536.598957] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8536.603975] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8536.612548] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8536.614427] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8536.619738] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8536.622630] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8536.625974] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8536.629732] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8536.631859] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8536.635579] lmv_intent_open+0x338/0x1150 [lmv] [ 8536.638154] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8536.641843] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8536.646627] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8536.653594] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8536.655732] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8536.659152] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8536.661408] ? do_raw_spin_unlock+0x75/0x190 [ 8536.663689] ? _raw_spin_unlock+0x12/0x30 [ 8536.665623] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8536.667639] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8536.669929] ll_atomic_open+0x36a/0xd70 [lustre] [ 8536.671480] lookup_open+0x356/0xb40 [ 8536.672414] do_last+0x313/0xfc0 [ 8536.673533] ? nd_jump_root+0xe5/0x160 [ 8536.674885] ? path_init+0x437/0x520 [ 8536.675965] path_openat+0xf7/0x500 [ 8536.678652] ? do_raw_spin_unlock+0x75/0x190 [ 8536.682997] do_filp_open+0x99/0x140 [ 8536.685693] ? getname_flags+0x6e/0x330 [ 8536.687523] ? __check_object_size+0xff/0x256 [ 8536.688778] ? do_raw_spin_unlock+0x75/0x190 [ 8536.694268] ? _raw_spin_unlock+0x12/0x30 [ 8536.699611] do_sys_openat2+0x2b4/0x410 [ 8536.701126] do_sys_open+0x73/0xa0 [ 8536.702051] __x64_sys_openat+0x24/0x30 [ 8536.702953] do_syscall_64+0xc1/0x440 [ 8536.706354] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8536.709161] RIP: 0033:0x7f53425a5332 [ 8536.710642] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8536.721475] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8536.723952] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8536.727404] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8536.730590] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8536.738422] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8536.746849] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8548.052061] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8548.059086] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8548.097933] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8548.105271] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8548.108985] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8548.112356] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8548.115492] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8548.122460] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8548.126499] RAX: 0000000000004920 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8548.130654] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8548.134173] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8548.139733] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8548.146669] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8548.154970] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8548.164114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8548.166812] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8548.168718] Call Trace: [ 8548.176216] [ 8548.182364] ? show_regs.cold.9+0x22/0x2f [ 8548.190785] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8548.195426] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8548.202048] ? watchdog+0x40/0x40 [ 8548.203604] ? hrtimer_interrupt+0x128/0x2c0 [ 8548.209509] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8548.212507] ? apic_timer_interrupt+0xf/0x20 [ 8548.221243] [ 8548.225478] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8548.233499] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8548.236212] ? newidle_balance.isra.93+0x207/0x580 [ 8548.244813] do_raw_spin_lock+0x177/0x190 [ 8548.248717] _raw_spin_lock+0x21/0x30 [ 8548.255393] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8548.264869] process_one_work+0x2c8/0x700 [ 8548.267334] worker_thread+0x296/0x6e0 [ 8548.275387] ? rescuer_thread+0x570/0x570 [ 8548.280749] kthread+0x1d1/0x200 [ 8548.286792] ? set_kthread_struct+0x70/0x70 [ 8548.288983] ret_from_fork+0x1f/0x30 [ 8550.373920] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5913s! [ 8550.381946] Showing busy workqueues and worker pools: [ 8550.385098] workqueue events: flags=0x0 [ 8550.391235] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8550.399375] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8550.408773] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8550.419673] pending: cpu_latency_work [ptlrpc] [ 8550.428651] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8550.435753] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8550.445805] workqueue events_long: flags=0x0 [ 8550.451582] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8550.453621] pending: lu_objects_flush [obdclass] [ 8550.459520] workqueue events_unbound: flags=0x2 [ 8550.464029] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8550.470327] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8550.483889] workqueue events_power_efficient: flags=0x80 [ 8550.490871] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8550.501896] pending: check_lifetime, do_cache_clean [ 8550.504567] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8550.506385] pending: neigh_periodic_work, neigh_periodic_work [ 8550.515445] workqueue mm_percpu_wq: flags=0x8 [ 8550.524416] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8550.535741] pending: vmstat_update, lru_add_drain_per_cpu [ 8550.540211] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8550.550203] pending: vmstat_update, lru_add_drain_per_cpu [ 8550.556027] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8550.565098] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8550.569108] workqueue xprtiod: flags=0xa [ 8550.573728] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8550.577362] pending: xprt_autoclose, xs_error_handle [ 8550.581393] workqueue ipv6_addrconf: flags=0x40008 [ 8550.592376] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8550.595662] pending: addrconf_verify_work [ 8550.600261] workqueue knbd0-recv: flags=0x1a [ 8550.604127] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8550.607687] in-flight: 651:recv_work [ 8550.609466] workqueue ptlrpc_pinger: flags=0x2 [ 8550.617552] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8550.621419] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8550.626382] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5913s workers=4 idle: 15979 31824 8 [ 8550.632858] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8541s workers=2 idle: 654 [ 8560.058346] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 8560.063487] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8560.084446] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8560.089119] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8560.092788] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8560.095472] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8560.104003] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8560.107242] RAX: 000000000000652a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8560.110458] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8560.113449] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8560.116937] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8560.120110] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8560.123467] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8560.127489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8560.130114] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8560.133876] Call Trace: [ 8560.135117] [ 8560.136173] ? show_regs.cold.9+0x22/0x2f [ 8560.138162] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8560.140412] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8560.142507] ? watchdog+0x40/0x40 [ 8560.143950] ? hrtimer_interrupt+0x128/0x2c0 [ 8560.145684] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8560.147915] ? apic_timer_interrupt+0xf/0x20 [ 8560.150136] [ 8560.151241] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8560.154294] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8560.157128] do_raw_spin_lock+0x177/0x190 [ 8560.159934] _raw_spin_lock+0x21/0x30 [ 8560.161932] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8560.165604] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8560.167967] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8560.170579] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8560.173955] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8560.176534] ? woken_wake_function+0x30/0x30 [ 8560.179270] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8560.182214] kthread+0x1d1/0x200 [ 8560.183988] ? set_kthread_struct+0x70/0x70 [ 8560.185853] ret_from_fork+0x1f/0x30 [ 8564.060019] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 8564.061536] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 8564.063243] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8564.063324] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8564.063332] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8564.063334] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8564.063347] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8564.063351] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8564.063356] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8564.063359] RDX: 0000000000003991 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8564.063361] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8564.063364] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8564.063366] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8564.063369] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8564.063371] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8564.063374] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8564.063382] Call Trace: [ 8564.063387] [ 8564.063392] ? show_regs.cold.9+0x22/0x2f [ 8564.063401] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8564.063407] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8564.063411] ? watchdog+0x40/0x40 [ 8564.063416] ? hrtimer_interrupt+0x128/0x2c0 [ 8564.063420] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8564.063428] ? apic_timer_interrupt+0xf/0x20 [ 8564.063432] [ 8564.063435] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8564.063439] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8564.063444] do_raw_spin_lock+0x177/0x190 [ 8564.063448] _raw_spin_lock+0x21/0x30 [ 8564.063453] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8564.063952] obd_mod_free+0x46/0x140 [ptlrpc] [ 8564.064151] mdc_commit_open+0x91/0xb0 [mdc] [ 8564.064186] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8564.064453] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8564.064918] after_reply+0xe2c/0x1780 [ptlrpc] [ 8564.065115] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8564.065310] ? __next_timer_interrupt+0x160/0x160 [ 8564.065321] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8564.066666] ? do_wait_intr+0xf0/0xf0 [ 8564.066676] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8564.075479] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8564.077013] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8564.077021] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8564.077024] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8564.077037] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8564.077041] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8564.077045] RAX: 0000000000007c86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8564.077048] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8564.077050] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8564.077052] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8564.077054] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8564.077058] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8564.077061] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8564.077063] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8564.077072] Call Trace: [ 8564.077078] [ 8564.077083] ? show_regs.cold.9+0x22/0x2f [ 8564.077092] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8564.077099] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8564.077104] ? watchdog+0x40/0x40 [ 8564.077108] ? hrtimer_interrupt+0x128/0x2c0 [ 8564.077112] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8564.077119] ? apic_timer_interrupt+0xf/0x20 [ 8564.077123] [ 8564.077126] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8564.077133] do_raw_spin_lock+0x177/0x190 [ 8564.077138] _raw_spin_lock+0x21/0x30 [ 8564.077143] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8564.077465] ? do_raw_spin_unlock+0x75/0x190 [ 8564.077472] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8564.077801] ? do_wait_intr+0xf0/0xf0 [ 8564.077812] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8564.078006] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8564.078217] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8564.081699] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8564.132895] kthread+0x1d1/0x200 [ 8564.143054] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8564.143109] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8564.143691] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8564.143727] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8564.143904] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8564.144189] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8564.144217] lmv_intent_open+0x338/0x1150 [lmv] [ 8564.144256] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8564.146030] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8564.146134] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8564.155178] ? set_kthread_struct+0x70/0x70 [ 8564.155201] ret_from_fork+0x1f/0x30 [ 8564.626741] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8564.629950] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8564.633329] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8564.637935] ? do_raw_spin_unlock+0x75/0x190 [ 8564.643803] ? _raw_spin_unlock+0x12/0x30 [ 8564.647666] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8564.651382] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8564.655530] ll_atomic_open+0x36a/0xd70 [lustre] [ 8564.657664] lookup_open+0x356/0xb40 [ 8564.659095] do_last+0x313/0xfc0 [ 8564.660267] ? nd_jump_root+0xe5/0x160 [ 8564.661971] ? path_init+0x437/0x520 [ 8564.663303] path_openat+0xf7/0x500 [ 8564.664994] ? do_raw_spin_unlock+0x75/0x190 [ 8564.666714] do_filp_open+0x99/0x140 [ 8564.668051] ? getname_flags+0x6e/0x330 [ 8564.669670] ? __check_object_size+0xff/0x256 [ 8564.671163] ? do_raw_spin_unlock+0x75/0x190 [ 8564.672746] ? _raw_spin_unlock+0x12/0x30 [ 8564.674077] do_sys_openat2+0x2b4/0x410 [ 8564.675180] do_sys_open+0x73/0xa0 [ 8564.676046] __x64_sys_openat+0x24/0x30 [ 8564.677260] do_syscall_64+0xc1/0x440 [ 8564.678622] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8564.680952] RIP: 0033:0x7f53425a5332 [ 8564.682290] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8564.688419] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8564.691316] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8564.693739] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8564.696429] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8564.699262] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8564.707753] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8576.052144] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 8576.057651] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8576.131052] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8576.164422] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8576.181930] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8576.196787] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8576.202699] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8576.222697] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8576.228242] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8576.246923] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8576.263061] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8576.273265] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8576.281245] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8576.291974] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8576.298319] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8576.305845] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8576.317787] Call Trace: [ 8576.319714] [ 8576.320772] ? show_regs.cold.9+0x22/0x2f [ 8576.326177] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8576.330698] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8576.333276] ? watchdog+0x40/0x40 [ 8576.335048] ? hrtimer_interrupt+0x128/0x2c0 [ 8576.337397] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8576.339544] ? apic_timer_interrupt+0xf/0x20 [ 8576.341517] [ 8576.342857] ? native_safe_halt+0x2e/0x50 [ 8576.352829] kvm_wait+0xcd/0xe0 [ 8576.354981] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8576.362758] ? newidle_balance.isra.93+0x207/0x580 [ 8576.364894] do_raw_spin_lock+0x177/0x190 [ 8576.372531] _raw_spin_lock+0x21/0x30 [ 8576.377282] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8576.380682] process_one_work+0x2c8/0x700 [ 8576.381637] worker_thread+0x296/0x6e0 [ 8576.382404] ? rescuer_thread+0x570/0x570 [ 8576.383314] kthread+0x1d1/0x200 [ 8576.384216] ? set_kthread_struct+0x70/0x70 [ 8576.385659] ret_from_fork+0x1f/0x30 [ 8581.096115] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5944s! [ 8581.103030] Showing busy workqueues and worker pools: [ 8581.106954] workqueue events: flags=0x0 [ 8581.115427] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8581.117765] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8581.126928] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8581.138064] pending: cpu_latency_work [ptlrpc] [ 8581.141244] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8581.155109] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8581.161679] workqueue events_long: flags=0x0 [ 8581.165170] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8581.172647] pending: lu_objects_flush [obdclass] [ 8581.174913] workqueue events_unbound: flags=0x2 [ 8581.184979] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8581.197972] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8581.205156] workqueue events_power_efficient: flags=0x80 [ 8581.206991] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8581.211529] pending: check_lifetime, do_cache_clean [ 8581.214481] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8581.216506] pending: neigh_periodic_work, neigh_periodic_work [ 8581.220904] workqueue mm_percpu_wq: flags=0x8 [ 8581.222707] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8581.226876] pending: vmstat_update, lru_add_drain_per_cpu [ 8581.232572] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8581.235607] pending: vmstat_update, lru_add_drain_per_cpu [ 8581.238013] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8581.245889] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8581.248098] workqueue xprtiod: flags=0xa [ 8581.257012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8581.262009] pending: xprt_autoclose, xs_error_handle [ 8581.273529] workqueue ipv6_addrconf: flags=0x40008 [ 8581.274992] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8581.276985] pending: addrconf_verify_work [ 8581.279798] workqueue knbd0-recv: flags=0x1a [ 8581.282947] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8581.288403] in-flight: 651:recv_work [ 8581.292231] workqueue ptlrpc_pinger: flags=0x2 [ 8581.294438] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8581.298766] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8581.302804] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5944s workers=4 idle: 15979 31824 8 [ 8581.311422] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8571s workers=2 idle: 654 [ 8588.058786] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 8588.064527] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8588.105149] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8588.112787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8588.117624] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8588.119907] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8588.135599] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8588.140277] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8588.144219] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8588.147456] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8588.151291] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8588.155477] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8588.159312] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8588.163247] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8588.166224] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8588.168681] Call Trace: [ 8588.169597] [ 8588.171334] ? show_regs.cold.9+0x22/0x2f [ 8588.177706] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8588.187307] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8588.192189] ? watchdog+0x40/0x40 [ 8588.194367] ? hrtimer_interrupt+0x128/0x2c0 [ 8588.197110] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8588.205446] ? apic_timer_interrupt+0xf/0x20 [ 8588.209423] [ 8588.211576] ? native_safe_halt+0x2e/0x50 [ 8588.217091] kvm_wait+0xcd/0xe0 [ 8588.218278] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8588.228196] do_raw_spin_lock+0x177/0x190 [ 8588.230658] _raw_spin_lock+0x21/0x30 [ 8588.239507] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8588.252917] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8588.260896] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8588.269591] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8588.272311] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8588.277761] ? woken_wake_function+0x30/0x30 [ 8588.278844] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8588.284393] kthread+0x1d1/0x200 [ 8588.291428] ? set_kthread_struct+0x70/0x70 [ 8588.297662] ret_from_fork+0x1f/0x30 [ 8592.056171] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 8592.057026] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 8592.063361] Modules linked in: [ 8592.065470] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) [ 8592.067011] lustre(O) [ 8592.067017] libcfs(O) rpcsec_gss_krb5 auth_rpcgss [ 8592.075678] osp(O) [ 8592.076728] nfsv4 [ 8592.079859] ofd(O) [ 8592.080760] dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8592.080798] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8592.080804] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8592.080807] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8592.080822] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8592.080826] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8592.080831] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8592.080834] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8592.080836] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8592.080839] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8592.080841] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8592.080845] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8592.080848] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8592.080850] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8592.080860] Call Trace: [ 8592.080867] [ 8592.080875] ? show_regs.cold.9+0x22/0x2f [ 8592.080888] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8592.080894] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8592.080899] ? watchdog+0x40/0x40 [ 8592.080903] ? hrtimer_interrupt+0x128/0x2c0 [ 8592.080907] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8592.080914] ? apic_timer_interrupt+0xf/0x20 [ 8592.080919] [ 8592.080921] ? native_safe_halt+0x2e/0x50 [ 8592.080925] kvm_wait+0xcd/0xe0 [ 8592.080933] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8592.080941] do_raw_spin_lock+0x177/0x190 [ 8592.080945] _raw_spin_lock+0x21/0x30 [ 8592.080952] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8592.081399] obd_mod_free+0x46/0x140 [ptlrpc] [ 8592.081645] mdc_commit_open+0x91/0xb0 [mdc] [ 8592.081690] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8592.081944] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8592.082152] after_reply+0xe2c/0x1780 [ptlrpc] [ 8592.082441] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8592.082637] ? __next_timer_interrupt+0x160/0x160 [ 8592.082648] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8592.082867] ? do_wait_intr+0xf0/0xf0 [ 8592.082878] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8592.083104] kthread+0x1d1/0x200 [ 8592.083110] ? set_kthread_struct+0x70/0x70 [ 8592.083114] ret_from_fork+0x1f/0x30 [ 8592.325435] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8592.379071] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8592.385176] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8592.393938] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8592.398324] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8592.423579] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8592.429142] RAX: 0000000000007a19 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8592.446619] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8592.451622] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8592.456549] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8592.465838] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8592.471920] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8592.488229] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8592.497942] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8592.506948] Call Trace: [ 8592.509351] [ 8592.511358] ? show_regs.cold.9+0x22/0x2f [ 8592.515339] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8592.517627] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8592.522360] ? watchdog+0x40/0x40 [ 8592.523828] ? hrtimer_interrupt+0x128/0x2c0 [ 8592.528330] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8592.532782] ? apic_timer_interrupt+0xf/0x20 [ 8592.536310] [ 8592.537538] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8592.540010] do_raw_spin_lock+0x177/0x190 [ 8592.543961] _raw_spin_lock+0x21/0x30 [ 8592.545284] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8592.546961] ? do_raw_spin_unlock+0x75/0x190 [ 8592.549916] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8592.555669] ? do_wait_intr+0xf0/0xf0 [ 8592.557137] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8592.559222] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8592.566047] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8592.568991] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8592.579282] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8592.584732] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8592.589464] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8592.592480] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8592.595543] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8592.601612] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8592.609811] lmv_intent_open+0x338/0x1150 [lmv] [ 8592.613957] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8592.618085] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8592.628208] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8592.631724] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8592.633977] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8592.635906] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8592.638995] ? do_raw_spin_unlock+0x75/0x190 [ 8592.641756] ? _raw_spin_unlock+0x12/0x30 [ 8592.645641] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8592.647862] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8592.651946] ll_atomic_open+0x36a/0xd70 [lustre] [ 8592.653658] lookup_open+0x356/0xb40 [ 8592.654920] do_last+0x313/0xfc0 [ 8592.656317] ? nd_jump_root+0xe5/0x160 [ 8592.657787] ? path_init+0x437/0x520 [ 8592.659001] path_openat+0xf7/0x500 [ 8592.660410] ? do_raw_spin_unlock+0x75/0x190 [ 8592.661914] do_filp_open+0x99/0x140 [ 8592.662934] ? getname_flags+0x6e/0x330 [ 8592.664231] ? __check_object_size+0xff/0x256 [ 8592.665698] ? do_raw_spin_unlock+0x75/0x190 [ 8592.667256] ? _raw_spin_unlock+0x12/0x30 [ 8592.668652] do_sys_openat2+0x2b4/0x410 [ 8592.669977] do_sys_open+0x73/0xa0 [ 8592.671276] __x64_sys_openat+0x24/0x30 [ 8592.672946] do_syscall_64+0xc1/0x440 [ 8592.675252] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8592.676806] RIP: 0033:0x7f53425a5332 [ 8592.677665] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8592.686037] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8592.688809] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8592.691809] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8592.694417] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8592.696917] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8592.699459] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8608.054766] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8608.060730] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8608.084438] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8608.089773] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8608.098087] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8608.102710] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8608.109058] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8608.122208] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8608.124636] RAX: 0000000000007f8a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8608.131840] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8608.135349] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8608.139316] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8608.145621] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8608.151279] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8608.159289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8608.163567] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8608.176061] Call Trace: [ 8608.178016] [ 8608.180809] ? show_regs.cold.9+0x22/0x2f [ 8608.183549] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8608.186406] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8608.188968] ? watchdog+0x40/0x40 [ 8608.190733] ? hrtimer_interrupt+0x128/0x2c0 [ 8608.192560] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8608.199754] ? apic_timer_interrupt+0xf/0x20 [ 8608.201145] [ 8608.205560] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8608.207409] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8608.211011] ? newidle_balance.isra.93+0x207/0x580 [ 8608.213518] do_raw_spin_lock+0x177/0x190 [ 8608.216015] _raw_spin_lock+0x21/0x30 [ 8608.217960] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8608.223862] process_one_work+0x2c8/0x700 [ 8608.226408] worker_thread+0x296/0x6e0 [ 8608.229465] ? rescuer_thread+0x570/0x570 [ 8608.231619] kthread+0x1d1/0x200 [ 8608.233369] ? set_kthread_struct+0x70/0x70 [ 8608.235705] ret_from_fork+0x1f/0x30 [ 8611.817488] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 5974s! [ 8611.821278] Showing busy workqueues and worker pools: [ 8611.823941] workqueue events: flags=0x0 [ 8611.826032] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8611.830171] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8611.839447] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8611.842758] pending: cpu_latency_work [ptlrpc] [ 8611.845119] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8611.850584] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8611.858370] workqueue events_long: flags=0x0 [ 8611.868021] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8611.878263] pending: lu_objects_flush [obdclass] [ 8611.882701] workqueue events_unbound: flags=0x2 [ 8611.884291] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8611.895669] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8611.897308] workqueue events_power_efficient: flags=0x80 [ 8611.904021] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8611.906308] pending: check_lifetime, do_cache_clean [ 8611.911015] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8611.914119] pending: neigh_periodic_work, neigh_periodic_work [ 8611.921677] workqueue mm_percpu_wq: flags=0x8 [ 8611.924991] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8611.928962] pending: vmstat_update, lru_add_drain_per_cpu [ 8611.932470] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8611.936059] pending: vmstat_update, lru_add_drain_per_cpu [ 8611.939507] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8611.945063] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8611.948289] workqueue xprtiod: flags=0xa [ 8611.949980] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8611.951601] pending: xprt_autoclose, xs_error_handle [ 8611.957542] workqueue ipv6_addrconf: flags=0x40008 [ 8611.960203] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8611.964149] pending: addrconf_verify_work [ 8611.968286] workqueue knbd0-recv: flags=0x1a [ 8611.970777] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8611.974263] in-flight: 651:recv_work [ 8611.976668] workqueue ptlrpc_pinger: flags=0x2 [ 8611.979371] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8611.982531] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8611.988524] pool 8: cpus=0-3 flags=0x4 nice=0 hung=5974s workers=4 idle: 15979 31824 8 [ 8611.997669] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8602s workers=2 idle: 654 [ 8616.059014] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 8616.063817] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8616.095444] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8616.101563] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8616.105621] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8616.107589] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8616.116921] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8616.120325] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8616.124199] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8616.126537] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8616.136515] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8616.141930] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8616.151612] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8616.159481] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8616.162977] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8616.166959] Call Trace: [ 8616.167815] [ 8616.168490] ? show_regs.cold.9+0x22/0x2f [ 8616.173968] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8616.176628] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8616.182102] ? watchdog+0x40/0x40 [ 8616.184479] ? hrtimer_interrupt+0x128/0x2c0 [ 8616.190375] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8616.192657] ? apic_timer_interrupt+0xf/0x20 [ 8616.195422] [ 8616.196470] ? native_safe_halt+0x2e/0x50 [ 8616.198630] kvm_wait+0xcd/0xe0 [ 8616.201049] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8616.207124] do_raw_spin_lock+0x177/0x190 [ 8616.210986] _raw_spin_lock+0x21/0x30 [ 8616.213607] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8616.219083] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8616.231886] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8616.239001] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8616.248269] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8616.250314] ? woken_wake_function+0x30/0x30 [ 8616.256024] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8616.264280] kthread+0x1d1/0x200 [ 8616.267104] ? set_kthread_struct+0x70/0x70 [ 8616.271974] ret_from_fork+0x1f/0x30 [ 8620.055011] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 8620.058603] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8620.062014] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 8620.082713] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8620.082729] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8620.082732] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8620.082747] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8620.082752] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8620.082758] RAX: 0000000000004d4f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8620.082760] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8620.082763] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8620.082765] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8620.082768] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8620.082772] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8620.082775] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8620.082778] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8620.082785] Call Trace: [ 8620.082792] [ 8620.082798] ? show_regs.cold.9+0x22/0x2f [ 8620.082808] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8620.082814] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8620.082819] ? watchdog+0x40/0x40 [ 8620.082823] ? hrtimer_interrupt+0x128/0x2c0 [ 8620.082828] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8620.082836] ? apic_timer_interrupt+0xf/0x20 [ 8620.082841] [ 8620.082844] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8620.082855] do_raw_spin_lock+0x177/0x190 [ 8620.088016] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8620.088322] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8620.088331] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8620.088334] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8620.088348] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8620.088352] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8620.088356] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8620.088358] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8620.088364] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8620.088367] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8620.088369] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8620.088372] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8620.088376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8620.088378] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8620.088385] Call Trace: [ 8620.088399] [ 8620.088405] ? show_regs.cold.9+0x22/0x2f [ 8620.088417] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8620.088424] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8620.088428] ? watchdog+0x40/0x40 [ 8620.088433] ? hrtimer_interrupt+0x128/0x2c0 [ 8620.088437] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8620.088444] ? apic_timer_interrupt+0xf/0x20 [ 8620.088448] [ 8620.088451] ? native_safe_halt+0x2e/0x50 [ 8620.088455] kvm_wait+0xcd/0xe0 [ 8620.088463] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8620.088471] do_raw_spin_lock+0x177/0x190 [ 8620.088476] _raw_spin_lock+0x21/0x30 [ 8620.088482] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8620.089348] obd_mod_free+0x46/0x140 [ptlrpc] [ 8620.089722] mdc_commit_open+0x91/0xb0 [mdc] [ 8620.089768] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8620.090050] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8620.090249] after_reply+0xe2c/0x1780 [ptlrpc] [ 8620.096376] _raw_spin_lock+0x21/0x30 [ 8620.096401] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8620.097088] ? do_raw_spin_unlock+0x75/0x190 [ 8620.097099] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8620.097309] ? do_wait_intr+0xf0/0xf0 [ 8620.097318] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8620.097547] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8620.102092] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8620.105018] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8620.128085] ? __next_timer_interrupt+0x160/0x160 [ 8620.128112] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8620.128495] ? do_wait_intr+0xf0/0xf0 [ 8620.128504] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8620.129758] kthread+0x1d1/0x200 [ 8620.129766] ? set_kthread_struct+0x70/0x70 [ 8620.129842] ret_from_fork+0x1f/0x30 [ 8620.563674] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8620.576709] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8620.579969] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8620.584939] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8620.589031] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8620.598901] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8620.605757] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8620.613711] lmv_intent_open+0x338/0x1150 [lmv] [ 8620.615908] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8620.621311] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8620.628616] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8620.634070] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8620.638201] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8620.644338] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8620.650542] ? do_raw_spin_unlock+0x75/0x190 [ 8620.653610] ? _raw_spin_unlock+0x12/0x30 [ 8620.660028] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8620.661918] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8620.666230] ll_atomic_open+0x36a/0xd70 [lustre] [ 8620.672730] lookup_open+0x356/0xb40 [ 8620.674879] do_last+0x313/0xfc0 [ 8620.680989] ? nd_jump_root+0xe5/0x160 [ 8620.684466] ? path_init+0x437/0x520 [ 8620.689401] path_openat+0xf7/0x500 [ 8620.697085] ? do_raw_spin_unlock+0x75/0x190 [ 8620.702249] do_filp_open+0x99/0x140 [ 8620.709860] ? getname_flags+0x6e/0x330 [ 8620.711272] ? __check_object_size+0xff/0x256 [ 8620.713634] ? do_raw_spin_unlock+0x75/0x190 [ 8620.717681] ? _raw_spin_unlock+0x12/0x30 [ 8620.722062] do_sys_openat2+0x2b4/0x410 [ 8620.724827] do_sys_open+0x73/0xa0 [ 8620.727349] __x64_sys_openat+0x24/0x30 [ 8620.730317] do_syscall_64+0xc1/0x440 [ 8620.732280] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8620.734678] RIP: 0033:0x7f53425a5332 [ 8620.737301] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8620.747002] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8620.751241] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8620.754397] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8620.757925] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8620.761066] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8620.763391] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8634.594032] rcu: INFO: rcu_sched self-detected stall on CPU [ 8634.596182] rcu: 2-....: (3555330 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 8634.599665] (t=6540168 jiffies g=274185 q=65506) [ 8634.601726] rcu: rcu_sched kthread starved for 5998534 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 8634.605256] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8634.608858] rcu: RCU grace-period kthread stack dump: [ 8634.610743] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8634.614399] Call Trace: [ 8634.615166] __schedule+0x351/0xcb0 [ 8634.616540] schedule+0xc0/0x180 [ 8634.617840] schedule_timeout+0xb4/0x190 [ 8634.619122] ? do_raw_spin_unlock+0x75/0x190 [ 8634.620579] ? __next_timer_interrupt+0x160/0x160 [ 8634.621989] ? prepare_to_swait_event+0xf3/0x140 [ 8634.623491] rcu_gp_kthread+0x8dd/0x15b0 [ 8634.624610] ? force_qs_rnp+0x2c0/0x2c0 [ 8634.625905] kthread+0x1d1/0x200 [ 8634.627069] ? set_kthread_struct+0x70/0x70 [ 8634.628315] ret_from_fork+0x1f/0x30 [ 8634.629263] rcu: Stack dump where RCU GP kthread last ran: [ 8634.630852] Sending NMI from CPU 2 to CPUs 1: [ 8634.633843] NMI backtrace for cpu 1 [ 8634.633851] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8634.633853] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8634.633854] RIP: 0010:interrupt_entry+0xb5/0xb9 [ 8634.633856] Code: 00 00 00 e8 8d 36 00 ff 48 8d 7c 24 08 65 ff 04 25 cc dc 01 00 75 12 65 48 89 3c 25 f8 5f 00 00 65 48 8b 24 25 d0 dc 01 00 57 77 f8 c3 48 83 04 24 80 e8 3d ff ff ff e8 f8 23 00 00 eb 35 66 [ 8634.633858] RSP: 0018:ffffbb5380d24ff8 EFLAGS: 00000056 [ 8634.633860] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000 [ 8634.633861] RDX: 0000000000000000 RSI: ffffffff91a0188a RDI: ffffbb5381503448 [ 8634.633863] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 [ 8634.633864] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 8634.633866] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 8634.633867] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8634.633869] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8634.633870] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8634.633871] Call Trace: [ 8634.633872] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8634.633874] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8634.633876] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8634.633878] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8634.633880] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8634.633881] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8634.633882] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8634.633884] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8634.633885] ? kvm_wait+0xcd/0xe0 [ 8634.633886] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8634.633888] ? do_raw_spin_lock+0x177/0x190 [ 8634.633889] ? _raw_spin_lock+0x21/0x30 [ 8634.633890] ? ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8634.633891] ? do_raw_spin_unlock+0x75/0x190 [ 8634.633892] ? ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8634.633893] ? do_wait_intr+0xf0/0xf0 [ 8634.633894] ? ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8634.633895] ? ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8634.633897] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8634.633898] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8634.633899] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8634.633900] ? mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8634.633901] ? mdc_intent_lock+0x353/0x7e0 [mdc] [ 8634.633902] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8634.633903] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8634.633904] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8634.633905] ? lmv_intent_open+0x338/0x1150 [lmv] [ 8634.633907] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8634.633908] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8634.633909] ? lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8634.633910] ? ll_intent_lock+0xc5/0xaa0 [lustre] [ 8634.633911] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8634.633912] ? ll_lookup_it+0xe0e/0x2270 [lustre] [ 8634.633914] ? do_raw_spin_unlock+0x75/0x190 [ 8634.633915] ? _raw_spin_unlock+0x12/0x30 [ 8634.633916] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8634.633917] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8634.633918] ? ll_atomic_open+0x36a/0xd70 [lustre] [ 8634.633920] ? lookup_open+0x356/0xb40 [ 8634.633920] ? do_last+0x313/0xfc0 [ 8634.633921] ? nd_jump_root+0xe5/0x160 [ 8634.633922] ? path_init+0x437/0x520 [ 8634.633924] ? path_openat+0xf7/0x500 [ 8634.633925] ? do_raw_spin_unlock+0x75/0x190 [ 8634.633926] ? do_filp_open+0x99/0x140 [ 8634.633927] ? getname_flags+0x6e/0x330 [ 8634.633928] ? __check_object_size+0xff/0x256 [ 8634.633929] ? do_raw_spin_unlock+0x75/0x190 [ 8634.633930] ? _raw_spin_unlock+0x12/0x30 [ 8634.633931] ? do_sys_openat2+0x2b4/0x410 [ 8634.633932] ? do_sys_open+0x73/0xa0 [ 8634.633933] ? __x64_sys_openat+0x24/0x30 [ 8634.633934] ? do_syscall_64+0xc1/0x440 [ 8634.633935] ? entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8634.634295] Sending NMI from CPU 2 to CPUs 1: [ 8634.792246] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 8634.794442] NMI backtrace for cpu 2 [ 8634.796847] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8634.801760] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8634.805813] Call Trace: [ 8634.806935] [ 8634.807907] ? dump_stack+0xbb/0x10e [ 8634.809584] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8634.811757] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8634.813815] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8634.815957] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8634.817834] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8634.819737] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8634.821186] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8634.823120] ? update_process_times+0x6d/0xc0 [ 8634.824602] ? tick_sched_handle.isra.18+0x27/0x90 [ 8634.825942] ? tick_sched_timer+0x53/0xb0 [ 8634.827435] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8634.829128] ? tick_sched_handle.isra.18+0x90/0x90 [ 8634.830997] ? hrtimer_interrupt+0x128/0x2c0 [ 8634.832698] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8634.834242] ? apic_timer_interrupt+0xf/0x20 [ 8634.836269] [ 8634.836812] ? native_safe_halt+0x2e/0x50 [ 8634.838095] ? kvm_wait+0xcd/0xe0 [ 8634.839285] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8634.841415] ? do_raw_spin_lock+0x177/0x190 [ 8634.842796] ? _raw_spin_lock+0x21/0x30 [ 8634.844122] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8634.845848] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8634.847479] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8634.849196] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8634.851453] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8634.853968] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8634.856201] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8634.857919] ? __next_timer_interrupt+0x160/0x160 [ 8634.859611] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8634.861405] ? do_wait_intr+0xf0/0xf0 [ 8634.862287] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8634.864493] ? kthread+0x1d1/0x200 [ 8634.865925] ? set_kthread_struct+0x70/0x70 [ 8634.867493] ? ret_from_fork+0x1f/0x30 [ 8636.052032] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8636.056603] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8636.094949] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8636.106318] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8636.111206] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8636.114630] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8636.117635] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8636.133157] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8636.140418] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8636.145145] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8636.151622] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8636.156493] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8636.163925] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8636.171896] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8636.186001] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8636.193923] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8636.205847] Call Trace: [ 8636.207676] [ 8636.209791] ? show_regs.cold.9+0x22/0x2f [ 8636.211161] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8636.213859] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8636.218635] ? watchdog+0x40/0x40 [ 8636.220235] ? hrtimer_interrupt+0x128/0x2c0 [ 8636.224807] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8636.233986] ? apic_timer_interrupt+0xf/0x20 [ 8636.237143] [ 8636.240068] ? native_safe_halt+0x2e/0x50 [ 8636.244782] kvm_wait+0xcd/0xe0 [ 8636.245969] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8636.249695] ? newidle_balance.isra.93+0x207/0x580 [ 8636.251618] do_raw_spin_lock+0x177/0x190 [ 8636.252945] _raw_spin_lock+0x21/0x30 [ 8636.254230] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8636.260024] process_one_work+0x2c8/0x700 [ 8636.265515] worker_thread+0x296/0x6e0 [ 8636.266862] ? rescuer_thread+0x570/0x570 [ 8636.268262] kthread+0x1d1/0x200 [ 8636.275950] ? set_kthread_struct+0x70/0x70 [ 8636.277380] ret_from_fork+0x1f/0x30 [ 8642.017172] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6004s! [ 8642.021839] Showing busy workqueues and worker pools: [ 8642.024036] workqueue events: flags=0x0 [ 8642.026234] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8642.029457] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8642.034768] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8642.038540] pending: cpu_latency_work [ptlrpc] [ 8642.040940] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8642.044424] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8642.048662] workqueue events_long: flags=0x0 [ 8642.050094] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8642.052923] pending: lu_objects_flush [obdclass] [ 8642.055018] workqueue events_unbound: flags=0x2 [ 8642.056636] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8642.058888] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8642.061809] workqueue events_power_efficient: flags=0x80 [ 8642.064109] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8642.066898] pending: check_lifetime, do_cache_clean [ 8642.069492] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8642.071832] pending: neigh_periodic_work, neigh_periodic_work [ 8642.074520] workqueue mm_percpu_wq: flags=0x8 [ 8642.076383] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8642.079615] pending: vmstat_update, lru_add_drain_per_cpu [ 8642.081534] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8642.084405] pending: vmstat_update, lru_add_drain_per_cpu [ 8642.087944] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8642.091172] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8642.094413] workqueue xprtiod: flags=0xa [ 8642.096387] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8642.099159] pending: xprt_autoclose, xs_error_handle [ 8642.101084] workqueue ipv6_addrconf: flags=0x40008 [ 8642.103583] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8642.105875] pending: addrconf_verify_work [ 8642.108102] workqueue knbd0-recv: flags=0x1a [ 8642.111214] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8642.114105] in-flight: 651:recv_work [ 8642.116072] workqueue ptlrpc_pinger: flags=0x2 [ 8642.117966] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8642.120530] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8642.123571] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6005s workers=4 idle: 15979 31824 8 [ 8642.127059] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8632s workers=2 idle: 654 [ 8644.058043] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 8644.061338] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8644.093291] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8644.100097] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8644.107967] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8644.109071] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8644.121507] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8644.137433] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8644.141747] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8644.153904] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8644.160532] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8644.163943] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8644.167842] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8644.175895] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8644.179937] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8644.187918] Call Trace: [ 8644.192797] [ 8644.195247] ? show_regs.cold.9+0x22/0x2f [ 8644.199296] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8644.203023] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8644.207743] ? watchdog+0x40/0x40 [ 8644.209910] ? hrtimer_interrupt+0x128/0x2c0 [ 8644.213102] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8644.215770] ? apic_timer_interrupt+0xf/0x20 [ 8644.217976] [ 8644.220238] ? native_safe_halt+0x2e/0x50 [ 8644.222843] kvm_wait+0xcd/0xe0 [ 8644.224643] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8644.227215] do_raw_spin_lock+0x177/0x190 [ 8644.228972] _raw_spin_lock+0x21/0x30 [ 8644.230915] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8644.234339] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8644.237478] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8644.240277] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8644.245337] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8644.247143] ? woken_wake_function+0x30/0x30 [ 8644.253767] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8644.260754] kthread+0x1d1/0x200 [ 8644.263957] ? set_kthread_struct+0x70/0x70 [ 8644.266772] ret_from_fork+0x1f/0x30 [ 8660.055694] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 8660.058224] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 8660.060534] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8660.060624] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8660.060634] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8660.060637] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8660.060653] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8660.060657] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8660.060661] RAX: 00000000000076e4 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8660.060664] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8660.060666] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8660.060668] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8660.060670] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8660.060673] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8660.060675] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8660.060677] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8660.060687] Call Trace: [ 8660.060693] [ 8660.060703] ? show_regs.cold.9+0x22/0x2f [ 8660.060713] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8660.060719] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8660.060724] ? watchdog+0x40/0x40 [ 8660.060728] ? hrtimer_interrupt+0x128/0x2c0 [ 8660.060731] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8660.060740] ? apic_timer_interrupt+0xf/0x20 [ 8660.060744] [ 8660.060747] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8660.060754] do_raw_spin_lock+0x177/0x190 [ 8660.060757] _raw_spin_lock+0x21/0x30 [ 8660.060762] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8660.061924] ? do_raw_spin_unlock+0x75/0x190 [ 8660.061932] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8660.062241] ? do_wait_intr+0xf0/0xf0 [ 8660.062251] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8660.062581] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8660.062803] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8660.063267] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8660.063411] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8660.063452] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8660.063481] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8660.063508] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8660.063593] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8660.063820] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8660.064022] lmv_intent_open+0x338/0x1150 [lmv] [ 8660.064053] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8660.064139] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8660.064474] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8660.064503] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8660.071743] Modules linked in: lustre(O) [ 8660.119615] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8660.119777] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8660.120847] ? do_raw_spin_unlock+0x75/0x190 [ 8660.120856] ? _raw_spin_unlock+0x12/0x30 [ 8660.120864] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8660.120906] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8660.128308] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8660.128385] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8660.128393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8660.128396] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8660.128411] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8660.128415] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8660.128420] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8660.128422] RDX: 0000000000007f81 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8660.128425] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8660.128427] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8660.128429] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8660.128431] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8660.128434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8660.128436] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8660.128444] Call Trace: [ 8660.128449] [ 8660.128454] ? show_regs.cold.9+0x22/0x2f [ 8660.128463] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8660.128470] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8660.128474] ? watchdog+0x40/0x40 [ 8660.128478] ? hrtimer_interrupt+0x128/0x2c0 [ 8660.128481] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8660.128488] ? apic_timer_interrupt+0xf/0x20 [ 8660.128498] [ 8660.128501] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8660.128505] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8660.128510] do_raw_spin_lock+0x177/0x190 [ 8660.128514] _raw_spin_lock+0x21/0x30 [ 8660.128519] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8660.128823] obd_mod_free+0x46/0x140 [ptlrpc] [ 8660.139497] ll_atomic_open+0x36a/0xd70 [lustre] [ 8660.139638] lookup_open+0x356/0xb40 [ 8660.139649] do_last+0x313/0xfc0 [ 8660.139655] ? nd_jump_root+0xe5/0x160 [ 8660.139660] ? path_init+0x437/0x520 [ 8660.139665] path_openat+0xf7/0x500 [ 8660.139670] ? do_raw_spin_unlock+0x75/0x190 [ 8660.139677] do_filp_open+0x99/0x140 [ 8660.139684] ? getname_flags+0x6e/0x330 [ 8660.139689] ? __check_object_size+0xff/0x256 [ 8660.139697] ? do_raw_spin_unlock+0x75/0x190 [ 8660.139701] ? _raw_spin_unlock+0x12/0x30 [ 8660.139708] do_sys_openat2+0x2b4/0x410 [ 8660.139713] do_sys_open+0x73/0xa0 [ 8660.139718] __x64_sys_openat+0x24/0x30 [ 8660.139721] do_syscall_64+0xc1/0x440 [ 8660.139729] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8660.139735] RIP: 0033:0x7f53425a5332 [ 8660.139781] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8660.139786] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8660.139791] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8660.139794] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8660.139796] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8660.139799] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8660.139801] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8660.634513] mdc_commit_open+0x91/0xb0 [mdc] [ 8660.636553] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8660.638763] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8660.641613] after_reply+0xe2c/0x1780 [ptlrpc] [ 8660.644224] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8660.647175] ? __next_timer_interrupt+0x160/0x160 [ 8660.649527] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8660.651617] ? do_wait_intr+0xf0/0xf0 [ 8660.653641] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8660.656485] kthread+0x1d1/0x200 [ 8660.657955] ? set_kthread_struct+0x70/0x70 [ 8660.660108] ret_from_fork+0x1f/0x30 [ 8668.056480] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8668.066013] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8668.128600] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8668.144530] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8668.154721] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8668.167989] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8668.171592] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8668.190467] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8668.193552] RAX: 00000000000015f8 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8668.196205] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8668.199709] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8668.202792] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8668.205187] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8668.208662] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8668.213618] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8668.215581] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8668.220509] Call Trace: [ 8668.221400] [ 8668.222317] ? show_regs.cold.9+0x22/0x2f [ 8668.225982] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8668.227859] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8668.231558] ? watchdog+0x40/0x40 [ 8668.232812] ? hrtimer_interrupt+0x128/0x2c0 [ 8668.234451] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8668.237588] ? apic_timer_interrupt+0xf/0x20 [ 8668.240291] [ 8668.241142] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8668.243125] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8668.246497] ? newidle_balance.isra.93+0x207/0x580 [ 8668.248664] do_raw_spin_lock+0x177/0x190 [ 8668.251437] _raw_spin_lock+0x21/0x30 [ 8668.252687] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8668.257148] process_one_work+0x2c8/0x700 [ 8668.258604] worker_thread+0x296/0x6e0 [ 8668.260124] ? rescuer_thread+0x570/0x570 [ 8668.262187] kthread+0x1d1/0x200 [ 8668.270661] ? set_kthread_struct+0x70/0x70 [ 8668.274062] ret_from_fork+0x1f/0x30 [ 8672.058678] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 8672.077633] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8672.122415] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8672.132480] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8672.136893] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8672.140963] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8672.157746] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8672.172777] RAX: 0000000000007f83 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8672.179028] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8672.183659] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8672.190194] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8672.199459] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8672.205063] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8672.217196] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8672.223253] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8672.225248] BUG: workqueue lockup - pool [ 8672.227446] Call Trace: [ 8672.227458] [ 8672.227464] ? show_regs.cold.9+0x22/0x2f [ 8672.227629] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8672.227639] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8672.227643] ? watchdog+0x40/0x40 [ 8672.227648] ? hrtimer_interrupt+0x128/0x2c0 [ 8672.227653] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8672.228541] ? apic_timer_interrupt+0xf/0x20 [ 8672.228550] [ 8672.228554] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8672.228561] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8672.228568] do_raw_spin_lock+0x177/0x190 [ 8672.228574] _raw_spin_lock+0x21/0x30 [ 8672.228579] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8672.229362] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8672.230450] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8672.234899] cpus=0-3 flags=0x4 nice=0 stuck for 6035s! [ 8672.234921] Showing busy workqueues and worker pools: [ 8672.234924] workqueue events: flags=0x0 [ 8672.234928] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8672.234936] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4] [ 8672.236296] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8672.237267] , push_to_pool [ 8672.237305] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8672.237323] pending: cpu_latency_work [ptlrpc] [ 8672.237585] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8672.237594] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8672.237894] workqueue events_long: flags=0x0 [ 8672.237897] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8672.237904] pending: lu_objects_flush [obdclass] [ 8672.238100] workqueue events_unbound: flags=0x2 [ 8672.238103] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8672.238110] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8672.238135] workqueue events_power_efficient: flags=0x80 [ 8672.238138] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8672.238145] pending: check_lifetime, do_cache_clean [ 8672.238158] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8672.238165] pending: neigh_periodic_work, neigh_periodic_work [ 8672.238178] workqueue mm_percpu_wq: flags=0x8 [ 8672.238183] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8672.238188] pending: vmstat_update, lru_add_drain_per_cpu [ 8672.238200] pwq 2: [ 8672.241285] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8672.246285] cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8672.246310] pending: vmstat_update [ 8672.248948] ? woken_wake_function+0x30/0x30 [ 8672.250493] , lru_add_drain_per_cpu [ 8672.250528] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8672.250541] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8672.250565] workqueue xprtiod: flags=0xa [ 8672.250567] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8672.250575] pending: xprt_autoclose, xs_error_handle [ 8672.250592] workqueue ipv6_addrconf: flags=0x40008 [ 8672.250595] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8672.250603] pending: addrconf_verify_work [ 8672.250615] workqueue knbd0-recv: flags=0x1a [ 8672.250618] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8672.250625] in-flight: 651:recv_work [ 8672.250634] workqueue ptlrpc_pinger: flags=0x2 [ 8672.250637] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8672.250642] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8672.250937] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6035s workers=4 idle: 15979 31824 8 [ 8672.250950] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8662s workers=2 idle: 654 [ 8672.492401] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8672.497754] kthread+0x1d1/0x200 [ 8672.499582] ? set_kthread_struct+0x70/0x70 [ 8672.502113] ret_from_fork+0x1f/0x30 [ 8688.055075] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 8688.057157] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 8688.064782] Modules linked in: [ 8688.073636] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8688.073730] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8688.073739] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8688.073744] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8688.073996] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8688.074000] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8688.074005] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8688.074008] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8688.074010] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8688.074012] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8688.074014] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8688.074018] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8688.074021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8688.074024] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8688.074032] Call Trace: [ 8688.074039] [ 8688.074047] ? show_regs.cold.9+0x22/0x2f [ 8688.074067] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8688.074076] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8688.074082] ? watchdog+0x40/0x40 [ 8688.074117] ? hrtimer_interrupt+0x128/0x2c0 [ 8688.074127] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8688.074136] ? apic_timer_interrupt+0xf/0x20 [ 8688.074141] [ 8688.074145] ? native_safe_halt+0x2e/0x50 [ 8688.074149] kvm_wait+0xcd/0xe0 [ 8688.074156] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8688.074164] do_raw_spin_lock+0x177/0x190 [ 8688.074168] _raw_spin_lock+0x21/0x30 [ 8688.074174] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8688.074909] obd_mod_free+0x46/0x140 [ptlrpc] [ 8688.075163] mdc_commit_open+0x91/0xb0 [mdc] [ 8688.075238] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8688.076121] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8688.076680] after_reply+0xe2c/0x1780 [ptlrpc] [ 8688.077770] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8688.145139] lustre(O) [ 8688.150345] ? __next_timer_interrupt+0x160/0x160 [ 8688.150368] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8688.150699] ? do_wait_intr+0xf0/0xf0 [ 8688.150710] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8688.150917] kthread+0x1d1/0x200 [ 8688.150923] ? set_kthread_struct+0x70/0x70 [ 8688.150927] ret_from_fork+0x1f/0x30 [ 8688.363936] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8688.391252] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8688.398791] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8688.403970] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8688.406432] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8688.419422] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8688.423745] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8688.429302] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8688.433860] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8688.438931] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8688.443785] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8688.449222] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8688.455258] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8688.459353] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8688.464845] Call Trace: [ 8688.466354] [ 8688.467884] ? show_regs.cold.9+0x22/0x2f [ 8688.470071] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8688.473344] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8688.476653] ? watchdog+0x40/0x40 [ 8688.478779] ? hrtimer_interrupt+0x128/0x2c0 [ 8688.482224] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8688.486126] ? apic_timer_interrupt+0xf/0x20 [ 8688.489339] [ 8688.491144] ? native_safe_halt+0x2e/0x50 [ 8688.493414] kvm_wait+0xcd/0xe0 [ 8688.495686] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8688.498798] do_raw_spin_lock+0x177/0x190 [ 8688.501283] _raw_spin_lock+0x21/0x30 [ 8688.503893] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8688.508299] ? do_raw_spin_unlock+0x75/0x190 [ 8688.510955] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8688.515364] ? do_wait_intr+0xf0/0xf0 [ 8688.518236] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8688.522917] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8688.527893] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8688.533477] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8688.538137] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8688.542080] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8688.545974] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8688.550068] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8688.553927] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8688.558171] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8688.563462] lmv_intent_open+0x338/0x1150 [lmv] [ 8688.567885] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8688.572350] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8688.577263] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8688.580941] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8688.585243] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8688.589755] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8688.594207] ? do_raw_spin_unlock+0x75/0x190 [ 8688.597868] ? _raw_spin_unlock+0x12/0x30 [ 8688.601960] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8688.605070] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8688.609759] ll_atomic_open+0x36a/0xd70 [lustre] [ 8688.612133] lookup_open+0x356/0xb40 [ 8688.615470] do_last+0x313/0xfc0 [ 8688.617956] ? nd_jump_root+0xe5/0x160 [ 8688.621469] ? path_init+0x437/0x520 [ 8688.630489] path_openat+0xf7/0x500 [ 8688.632024] ? do_raw_spin_unlock+0x75/0x190 [ 8688.634758] do_filp_open+0x99/0x140 [ 8688.636803] ? getname_flags+0x6e/0x330 [ 8688.644734] ? __check_object_size+0xff/0x256 [ 8688.646937] ? do_raw_spin_unlock+0x75/0x190 [ 8688.649222] ? _raw_spin_unlock+0x12/0x30 [ 8688.654974] do_sys_openat2+0x2b4/0x410 [ 8688.658711] do_sys_open+0x73/0xa0 [ 8688.660629] __x64_sys_openat+0x24/0x30 [ 8688.669184] do_syscall_64+0xc1/0x440 [ 8688.670523] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8688.677243] RIP: 0033:0x7f53425a5332 [ 8688.683492] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8688.691341] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8688.695695] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8688.699300] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8688.703685] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8688.715265] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8688.725840] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8700.052032] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8700.056441] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8700.060098] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 8700.081114] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8700.081129] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8700.081136] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8700.081437] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8700.081449] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8700.081453] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8700.081458] RAX: 00000000000019a2 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8700.081460] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8700.081462] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8700.081465] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8700.081467] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8700.081471] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8700.081473] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8700.081476] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8700.081484] Call Trace: [ 8700.081490] [ 8700.081495] ? show_regs.cold.9+0x22/0x2f [ 8700.081506] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8700.081513] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8700.081518] ? watchdog+0x40/0x40 [ 8700.081522] ? hrtimer_interrupt+0x128/0x2c0 [ 8700.081526] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8700.081536] ? apic_timer_interrupt+0xf/0x20 [ 8700.081541] [ 8700.081544] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8700.081549] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8700.081553] ? newidle_balance.isra.93+0x207/0x580 [ 8700.081558] do_raw_spin_lock+0x177/0x190 [ 8700.081564] _raw_spin_lock+0x21/0x30 [ 8700.081570] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8700.081774] process_one_work+0x2c8/0x700 [ 8700.081782] worker_thread+0x296/0x6e0 [ 8700.081787] ? rescuer_thread+0x570/0x570 [ 8700.081791] kthread+0x1d1/0x200 [ 8700.081798] ? set_kthread_struct+0x70/0x70 [ 8700.081801] ret_from_fork+0x1f/0x30 [ 8700.231647] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8700.272541] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8700.283535] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8700.289518] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8700.294536] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8700.310795] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8700.317216] RAX: 0000000000006484 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8700.321749] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8700.329528] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8700.333578] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8700.342764] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8700.349638] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8700.365033] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8700.370817] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8700.379251] Call Trace: [ 8700.383447] [ 8700.385813] ? show_regs.cold.9+0x22/0x2f [ 8700.392892] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8700.399386] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8700.404160] ? watchdog+0x40/0x40 [ 8700.413126] ? hrtimer_interrupt+0x128/0x2c0 [ 8700.419620] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8700.424723] ? apic_timer_interrupt+0xf/0x20 [ 8700.428866] [ 8700.430568] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8700.432917] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8700.436142] do_raw_spin_lock+0x177/0x190 [ 8700.439254] _raw_spin_lock+0x21/0x30 [ 8700.443701] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8700.448324] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8700.453652] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8700.460186] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8700.470725] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8700.474542] ? woken_wake_function+0x30/0x30 [ 8700.477153] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8700.487781] kthread+0x1d1/0x200 [ 8700.493662] ? set_kthread_struct+0x70/0x70 [ 8700.499007] ret_from_fork+0x1f/0x30 [ 8702.431209] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6065s! [ 8702.434693] Showing busy workqueues and worker pools: [ 8702.443650] workqueue events: flags=0x0 [ 8702.449543] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8702.453121] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8702.459907] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8702.482994] pending: cpu_latency_work [ptlrpc] [ 8702.503652] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8702.517814] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8702.538162] workqueue events_long: flags=0x0 [ 8702.545255] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8702.552804] pending: lu_objects_flush [obdclass] [ 8702.564553] workqueue events_unbound: flags=0x2 [ 8702.580729] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8702.591868] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8702.602683] workqueue events_power_efficient: flags=0x80 [ 8702.609223] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8702.620944] pending: check_lifetime, do_cache_clean [ 8702.624509] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8702.638539] pending: neigh_periodic_work, neigh_periodic_work [ 8702.643995] workqueue mm_percpu_wq: flags=0x8 [ 8702.646905] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8702.663748] pending: vmstat_update, lru_add_drain_per_cpu [ 8702.669480] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8702.683036] pending: vmstat_update, lru_add_drain_per_cpu [ 8702.691981] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8702.700706] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8702.709683] workqueue xprtiod: flags=0xa [ 8702.713594] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8702.719001] pending: xprt_autoclose, xs_error_handle [ 8702.720842] workqueue ipv6_addrconf: flags=0x40008 [ 8702.727594] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8702.734624] pending: addrconf_verify_work [ 8702.744286] workqueue knbd0-recv: flags=0x1a [ 8702.748676] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8702.756390] in-flight: 651:recv_work [ 8702.757904] workqueue ptlrpc_pinger: flags=0x2 [ 8702.766397] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8702.774909] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8702.782738] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6065s workers=4 idle: 15979 31824 8 [ 8702.794406] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8693s workers=2 idle: 654 [ 8716.057203] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 8716.062080] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8716.065129] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 8716.091938] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8716.091955] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8716.091958] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8716.091972] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8716.091977] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8716.091982] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8716.091985] RDX: 0000000000004ce5 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8716.091987] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8716.091989] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8716.091992] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8716.091995] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8716.091998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8716.092000] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8716.092009] Call Trace: [ 8716.092015] [ 8716.092020] ? show_regs.cold.9+0x22/0x2f [ 8716.092031] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8716.092037] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8716.092042] ? watchdog+0x40/0x40 [ 8716.092046] ? hrtimer_interrupt+0x128/0x2c0 [ 8716.092050] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8716.092059] ? apic_timer_interrupt+0xf/0x20 [ 8716.092063] [ 8716.092066] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8716.092180] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8716.092186] do_raw_spin_lock+0x177/0x190 [ 8716.092191] _raw_spin_lock+0x21/0x30 [ 8716.092199] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8716.092455] obd_mod_free+0x46/0x140 [ptlrpc] [ 8716.093017] mdc_commit_open+0x91/0xb0 [mdc] [ 8716.093055] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8716.093366] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8716.100420] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8716.100499] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8716.100507] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8716.100510] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8716.100525] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8716.100530] RSP: 0018:ffffbb5381503500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 8716.100535] RAX: 0000000000007a93 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8716.100537] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8716.100540] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8716.100542] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8716.100545] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8716.100548] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8716.100551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8716.100553] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8716.100563] Call Trace: [ 8716.100568] [ 8716.100575] ? show_regs.cold.9+0x22/0x2f [ 8716.100584] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8716.100591] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8716.100596] ? watchdog+0x40/0x40 [ 8716.100601] ? hrtimer_interrupt+0x128/0x2c0 [ 8716.100605] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8716.100613] ? apic_timer_interrupt+0xf/0x20 [ 8716.100618] [ 8716.100621] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8716.100627] do_raw_spin_lock+0x177/0x190 [ 8716.106622] after_reply+0xe2c/0x1780 [ptlrpc] [ 8716.107202] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8716.107404] ? __next_timer_interrupt+0x160/0x160 [ 8716.107414] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8716.107620] ? do_wait_intr+0xf0/0xf0 [ 8716.107846] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8716.108228] kthread+0x1d1/0x200 [ 8716.108235] ? set_kthread_struct+0x70/0x70 [ 8716.108239] ret_from_fork+0x1f/0x30 [ 8716.356954] _raw_spin_lock+0x21/0x30 [ 8716.358875] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8716.361695] ? do_raw_spin_unlock+0x75/0x190 [ 8716.363839] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8716.366186] ? do_wait_intr+0xf0/0xf0 [ 8716.368202] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8716.370579] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8716.373348] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8716.376530] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8716.379039] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8716.381662] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8716.383856] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8716.386224] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8716.388779] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8716.391757] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8716.394383] lmv_intent_open+0x338/0x1150 [lmv] [ 8716.396441] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8716.398796] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8716.402405] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8716.404478] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8716.406948] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8716.410038] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8716.412870] ? do_raw_spin_unlock+0x75/0x190 [ 8716.415527] ? _raw_spin_unlock+0x12/0x30 [ 8716.417688] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8716.421051] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8716.425791] ll_atomic_open+0x36a/0xd70 [lustre] [ 8716.429847] lookup_open+0x356/0xb40 [ 8716.433414] do_last+0x313/0xfc0 [ 8716.435321] ? nd_jump_root+0xe5/0x160 [ 8716.438673] ? path_init+0x437/0x520 [ 8716.440635] path_openat+0xf7/0x500 [ 8716.442468] ? do_raw_spin_unlock+0x75/0x190 [ 8716.445575] do_filp_open+0x99/0x140 [ 8716.447490] ? getname_flags+0x6e/0x330 [ 8716.449541] ? __check_object_size+0xff/0x256 [ 8716.453862] ? do_raw_spin_unlock+0x75/0x190 [ 8716.456444] ? _raw_spin_unlock+0x12/0x30 [ 8716.459020] do_sys_openat2+0x2b4/0x410 [ 8716.461284] do_sys_open+0x73/0xa0 [ 8716.465850] __x64_sys_openat+0x24/0x30 [ 8716.472388] do_syscall_64+0xc1/0x440 [ 8716.474637] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8716.479645] RIP: 0033:0x7f53425a5332 [ 8716.481704] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8716.503563] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8716.510924] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8716.518398] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8716.525587] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8716.530426] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8716.536639] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8728.052042] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8728.062023] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8728.063105] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 8728.117632] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8728.117646] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8728.117652] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8728.118643] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8728.118658] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8728.118662] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8728.118666] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8728.118668] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8728.118670] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8728.118672] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8728.118674] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8728.118677] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8728.118679] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8728.118681] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8728.129423] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8728.129697] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8728.129703] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8728.129706] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8728.129719] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8728.129723] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8728.129727] RAX: 0000000000006773 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8728.129729] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8728.129732] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8728.129734] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8728.129737] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8728.129740] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8728.129743] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8728.129813] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8728.129821] Call Trace: [ 8728.129827] [ 8728.129833] ? show_regs.cold.9+0x22/0x2f [ 8728.129843] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8728.129849] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8728.129854] ? watchdog+0x40/0x40 [ 8728.129858] ? hrtimer_interrupt+0x128/0x2c0 [ 8728.129862] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8728.129871] ? apic_timer_interrupt+0xf/0x20 [ 8728.129876] [ 8728.129879] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8728.129883] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8728.129889] do_raw_spin_lock+0x177/0x190 [ 8728.129894] _raw_spin_lock+0x21/0x30 [ 8728.129899] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8728.130607] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8728.130953] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8728.131898] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8728.142157] Call Trace: [ 8728.142171] [ 8728.142178] ? show_regs.cold.9+0x22/0x2f [ 8728.142192] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8728.142199] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8728.142204] ? watchdog+0x40/0x40 [ 8728.142208] ? hrtimer_interrupt+0x128/0x2c0 [ 8728.142212] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8728.142220] ? apic_timer_interrupt+0xf/0x20 [ 8728.142226] [ 8728.142228] ? native_safe_halt+0x2e/0x50 [ 8728.142232] kvm_wait+0xcd/0xe0 [ 8728.142240] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8728.142246] ? newidle_balance.isra.93+0x207/0x580 [ 8728.142254] do_raw_spin_lock+0x177/0x190 [ 8728.142259] _raw_spin_lock+0x21/0x30 [ 8728.142263] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8728.142638] process_one_work+0x2c8/0x700 [ 8728.142646] worker_thread+0x296/0x6e0 [ 8728.142651] ? rescuer_thread+0x570/0x570 [ 8728.142655] kthread+0x1d1/0x200 [ 8728.142660] ? set_kthread_struct+0x70/0x70 [ 8728.142664] ret_from_fork+0x1f/0x30 [ 8728.569828] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8728.577833] ? woken_wake_function+0x30/0x30 [ 8728.582893] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8728.591265] kthread+0x1d1/0x200 [ 8728.592625] ? set_kthread_struct+0x70/0x70 [ 8728.598803] ret_from_fork+0x1f/0x30 [ 8733.154270] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6096s! [ 8733.157449] Showing busy workqueues and worker pools: [ 8733.159630] workqueue events: flags=0x0 [ 8733.161190] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8733.163914] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8733.168399] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8733.171140] pending: cpu_latency_work [ptlrpc] [ 8733.173490] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8733.176276] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8733.180117] workqueue events_long: flags=0x0 [ 8733.181900] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8733.183906] pending: lu_objects_flush [obdclass] [ 8733.186068] workqueue events_unbound: flags=0x2 [ 8733.187593] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8733.189972] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8733.191975] workqueue events_power_efficient: flags=0x80 [ 8733.193920] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8733.196555] pending: check_lifetime, do_cache_clean [ 8733.198546] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8733.201104] pending: neigh_periodic_work, neigh_periodic_work [ 8733.203432] workqueue mm_percpu_wq: flags=0x8 [ 8733.205050] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8733.207673] pending: vmstat_update, lru_add_drain_per_cpu [ 8733.209808] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8733.212296] pending: vmstat_update, lru_add_drain_per_cpu [ 8733.214403] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8733.216836] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8733.219246] workqueue xprtiod: flags=0xa [ 8733.220831] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8733.223219] pending: xprt_autoclose, xs_error_handle [ 8733.225292] workqueue ipv6_addrconf: flags=0x40008 [ 8733.227158] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8733.229794] pending: addrconf_verify_work [ 8733.231461] workqueue knbd0-recv: flags=0x1a [ 8733.233150] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8733.235672] in-flight: 651:recv_work [ 8733.237245] workqueue ptlrpc_pinger: flags=0x2 [ 8733.239079] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8733.241462] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8733.244024] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6096s workers=4 idle: 15979 31824 8 [ 8733.247067] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8723s workers=2 idle: 654 [ 8744.058237] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 8744.062085] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 8744.070253] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8744.070335] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8744.076254] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8744.076355] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8744.076508] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8744.076512] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8744.076533] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8744.076539] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8744.076544] RAX: 000000000000079a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8744.076547] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8744.076549] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8744.076552] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8744.076555] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8744.076559] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8744.076562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8744.076564] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8744.076575] Call Trace: [ 8744.076583] [ 8744.076592] ? show_regs.cold.9+0x22/0x2f [ 8744.076606] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8744.076617] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8744.076622] ? watchdog+0x40/0x40 [ 8744.076627] ? hrtimer_interrupt+0x128/0x2c0 [ 8744.076631] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8744.076642] ? apic_timer_interrupt+0xf/0x20 [ 8744.076647] [ 8744.076650] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8744.076657] do_raw_spin_lock+0x177/0x190 [ 8744.076663] _raw_spin_lock+0x21/0x30 [ 8744.076668] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8744.077454] ? do_raw_spin_unlock+0x75/0x190 [ 8744.077462] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8744.077673] ? do_wait_intr+0xf0/0xf0 [ 8744.077687] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8744.078145] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8744.078519] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8744.078974] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8744.079113] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8744.079313] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8744.079349] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8744.079376] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8744.079477] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8744.079769] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8744.079801] lmv_intent_open+0x338/0x1150 [lmv] [ 8744.079852] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8744.079937] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8744.080033] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8744.080060] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8744.080152] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8744.080259] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8744.080350] ? do_raw_spin_unlock+0x75/0x190 [ 8744.080360] ? _raw_spin_unlock+0x12/0x30 [ 8744.080370] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8744.080402] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8744.080436] ll_atomic_open+0x36a/0xd70 [lustre] [ 8744.080528] lookup_open+0x356/0xb40 [ 8744.080545] do_last+0x313/0xfc0 [ 8744.080555] ? nd_jump_root+0xe5/0x160 [ 8744.144159] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8744.144176] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8744.144195] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8744.144200] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8744.144207] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8744.144210] RDX: 000000000000365b RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8744.144212] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8744.144215] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8744.144217] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8744.144221] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8744.144224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8744.144226] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8744.144799] Call Trace: [ 8744.144808] [ 8744.144814] ? show_regs.cold.9+0x22/0x2f [ 8744.144828] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8744.144834] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8744.144838] ? watchdog+0x40/0x40 [ 8744.144843] ? hrtimer_interrupt+0x128/0x2c0 [ 8744.144847] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8744.144857] ? apic_timer_interrupt+0xf/0x20 [ 8744.144862] [ 8744.144876] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8744.144882] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8744.144886] do_raw_spin_lock+0x177/0x190 [ 8744.144891] _raw_spin_lock+0x21/0x30 [ 8744.144900] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8744.146379] obd_mod_free+0x46/0x140 [ptlrpc] [ 8744.149877] mdc_commit_open+0x91/0xb0 [mdc] [ 8744.149953] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8744.150274] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8744.186845] ? path_init+0x437/0x520 [ 8744.186868] path_openat+0xf7/0x500 [ 8744.186877] ? do_raw_spin_unlock+0x75/0x190 [ 8744.231873] after_reply+0xe2c/0x1780 [ptlrpc] [ 8744.232149] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8744.241224] do_filp_open+0x99/0x140 [ 8744.241249] ? getname_flags+0x6e/0x330 [ 8744.241255] ? __check_object_size+0xff/0x256 [ 8744.241263] ? do_raw_spin_unlock+0x75/0x190 [ 8744.241270] ? _raw_spin_unlock+0x12/0x30 [ 8744.241277] do_sys_openat2+0x2b4/0x410 [ 8744.241282] do_sys_open+0x73/0xa0 [ 8744.241287] __x64_sys_openat+0x24/0x30 [ 8744.241291] do_syscall_64+0xc1/0x440 [ 8744.241297] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8744.241304] RIP: 0033:0x7f53425a5332 [ 8744.241314] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8744.241318] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8744.241325] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8744.241327] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8744.241329] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8744.241331] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8744.241334] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8744.784983] ? __next_timer_interrupt+0x160/0x160 [ 8744.789084] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8744.791457] ? do_wait_intr+0xf0/0xf0 [ 8744.792745] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8744.802815] kthread+0x1d1/0x200 [ 8744.804807] ? set_kthread_struct+0x70/0x70 [ 8744.809212] ret_from_fork+0x1f/0x30 [ 8756.058025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 8756.070526] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8756.114901] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8756.120100] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8756.123581] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8756.126541] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8756.134127] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8756.137849] RAX: 0000000000006702 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8756.140768] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8756.144304] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8756.147087] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8756.149711] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8756.151558] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8756.154165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8756.155885] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8756.158041] Call Trace: [ 8756.158792] [ 8756.159405] ? show_regs.cold.9+0x22/0x2f [ 8756.160868] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8756.162255] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8756.163472] ? watchdog+0x40/0x40 [ 8756.164378] ? hrtimer_interrupt+0x128/0x2c0 [ 8756.165490] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8756.166660] ? apic_timer_interrupt+0xf/0x20 [ 8756.167849] [ 8756.168395] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8756.169930] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8756.171297] do_raw_spin_lock+0x177/0x190 [ 8756.172292] _raw_spin_lock+0x21/0x30 [ 8756.173183] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8756.175415] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8756.177536] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8756.179925] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8756.182564] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8756.184812] ? woken_wake_function+0x30/0x30 [ 8756.186465] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8756.189161] kthread+0x1d1/0x200 [ 8756.190091] ? set_kthread_struct+0x70/0x70 [ 8756.191517] ret_from_fork+0x1f/0x30 [ 8760.052623] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8760.062868] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8760.133705] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8760.152071] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8760.168980] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8760.175627] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8760.179113] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8760.201519] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8760.210880] RAX: 0000000000006097 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8760.217027] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8760.224614] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8760.240998] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8760.249489] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8760.264406] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8760.278356] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8760.280319] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8760.297785] Call Trace: [ 8760.298703] [ 8760.305863] ? show_regs.cold.9+0x22/0x2f [ 8760.311078] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8760.316046] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8760.321889] ? watchdog+0x40/0x40 [ 8760.328493] ? hrtimer_interrupt+0x128/0x2c0 [ 8760.333254] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8760.338617] ? apic_timer_interrupt+0xf/0x20 [ 8760.342935] [ 8760.343725] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8760.345657] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8760.354382] ? newidle_balance.isra.93+0x207/0x580 [ 8760.357647] do_raw_spin_lock+0x177/0x190 [ 8760.362593] _raw_spin_lock+0x21/0x30 [ 8760.367525] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8760.371334] process_one_work+0x2c8/0x700 [ 8760.374183] worker_thread+0x296/0x6e0 [ 8760.376637] ? rescuer_thread+0x570/0x570 [ 8760.378934] kthread+0x1d1/0x200 [ 8760.380057] ? set_kthread_struct+0x70/0x70 [ 8760.382392] ret_from_fork+0x1f/0x30 [ 8763.360060] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6126s! [ 8763.364008] Showing busy workqueues and worker pools: [ 8763.370226] workqueue events: flags=0x0 [ 8763.373440] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8763.387825] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8763.396525] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8763.398409] pending: cpu_latency_work [ptlrpc] [ 8763.408165] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8763.422769] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8763.433215] workqueue events_long: flags=0x0 [ 8763.434782] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8763.440882] pending: lu_objects_flush [obdclass] [ 8763.444347] workqueue events_unbound: flags=0x2 [ 8763.460759] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8763.467577] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8763.472936] workqueue events_power_efficient: flags=0x80 [ 8763.478758] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8763.485517] pending: check_lifetime, do_cache_clean [ 8763.488740] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8763.495588] pending: neigh_periodic_work, neigh_periodic_work [ 8763.502681] workqueue mm_percpu_wq: flags=0x8 [ 8763.506818] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8763.510791] pending: vmstat_update, lru_add_drain_per_cpu [ 8763.512992] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8763.518342] pending: vmstat_update, lru_add_drain_per_cpu [ 8763.523862] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8763.531182] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8763.539496] workqueue xprtiod: flags=0xa [ 8763.542858] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8763.549045] pending: xprt_autoclose, xs_error_handle [ 8763.553316] workqueue ipv6_addrconf: flags=0x40008 [ 8763.555884] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8763.557559] pending: addrconf_verify_work [ 8763.559876] workqueue knbd0-recv: flags=0x1a [ 8763.561687] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8763.568285] in-flight: 651:recv_work [ 8763.571618] workqueue ptlrpc_pinger: flags=0x2 [ 8763.574410] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8763.576443] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8763.578369] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6126s workers=4 idle: 15979 31824 8 [ 8763.585809] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8754s workers=2 idle: 654 [ 8772.055037] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 8772.057011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 8772.061119] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul [ 8772.064505] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8772.064593] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8772.064602] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8772.064606] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8772.064680] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8772.064684] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8772.064689] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8772.064691] RDX: 000000000000686b RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8772.064693] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8772.064696] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8772.064698] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8772.064702] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8772.064705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8772.064708] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8772.064716] Call Trace: [ 8772.064723] [ 8772.064730] ? show_regs.cold.9+0x22/0x2f [ 8772.064741] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8772.064749] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8772.064753] ? watchdog+0x40/0x40 [ 8772.064758] ? hrtimer_interrupt+0x128/0x2c0 [ 8772.064762] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8772.064771] ? apic_timer_interrupt+0xf/0x20 [ 8772.064776] [ 8772.064779] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8772.064784] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8772.064789] do_raw_spin_lock+0x177/0x190 [ 8772.064794] _raw_spin_lock+0x21/0x30 [ 8772.064801] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8772.065289] obd_mod_free+0x46/0x140 [ptlrpc] [ 8772.065496] mdc_commit_open+0x91/0xb0 [mdc] [ 8772.065533] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8772.065739] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8772.066057] after_reply+0xe2c/0x1780 [ptlrpc] [ 8772.097211] crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8772.097239] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8772.097245] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8772.097249] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8772.097262] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8772.097267] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8772.097271] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8772.097274] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8772.097276] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8772.097279] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8772.097281] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8772.097284] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8772.097287] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8772.097289] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8772.097298] Call Trace: [ 8772.097304] [ 8772.097310] ? show_regs.cold.9+0x22/0x2f [ 8772.097321] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8772.097329] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8772.097334] ? watchdog+0x40/0x40 [ 8772.097338] ? hrtimer_interrupt+0x128/0x2c0 [ 8772.097342] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8772.097349] ? apic_timer_interrupt+0xf/0x20 [ 8772.097354] [ 8772.097357] ? native_safe_halt+0x2e/0x50 [ 8772.097360] kvm_wait+0xcd/0xe0 [ 8772.097367] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8772.097377] do_raw_spin_lock+0x177/0x190 [ 8772.097382] _raw_spin_lock+0x21/0x30 [ 8772.097387] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8772.131531] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8772.142673] ? do_raw_spin_unlock+0x75/0x190 [ 8772.142699] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8772.143061] ? do_wait_intr+0xf0/0xf0 [ 8772.143071] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8772.143573] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8772.150987] ? __next_timer_interrupt+0x160/0x160 [ 8772.156298] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8772.156620] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8772.156762] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8772.156806] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8772.156836] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8772.156864] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8772.171841] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8772.176322] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8772.176979] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8772.177022] lmv_intent_open+0x338/0x1150 [lmv] [ 8772.177056] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8772.177250] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8772.177345] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8772.177368] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8772.181055] ? do_wait_intr+0xf0/0xf0 [ 8772.181137] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8772.181515] kthread+0x1d1/0x200 [ 8772.181521] ? set_kthread_struct+0x70/0x70 [ 8772.181525] ret_from_fork+0x1f/0x30 [ 8772.527801] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8772.534546] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8772.543804] ? do_raw_spin_unlock+0x75/0x190 [ 8772.551280] ? _raw_spin_unlock+0x12/0x30 [ 8772.552688] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8772.567276] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8772.569854] ll_atomic_open+0x36a/0xd70 [lustre] [ 8772.575621] lookup_open+0x356/0xb40 [ 8772.578448] do_last+0x313/0xfc0 [ 8772.581564] ? nd_jump_root+0xe5/0x160 [ 8772.583336] ? path_init+0x437/0x520 [ 8772.585329] path_openat+0xf7/0x500 [ 8772.588373] ? do_raw_spin_unlock+0x75/0x190 [ 8772.591264] do_filp_open+0x99/0x140 [ 8772.593770] ? getname_flags+0x6e/0x330 [ 8772.595464] ? __check_object_size+0xff/0x256 [ 8772.597386] ? do_raw_spin_unlock+0x75/0x190 [ 8772.600759] ? _raw_spin_unlock+0x12/0x30 [ 8772.602638] do_sys_openat2+0x2b4/0x410 [ 8772.604486] do_sys_open+0x73/0xa0 [ 8772.605617] __x64_sys_openat+0x24/0x30 [ 8772.606841] do_syscall_64+0xc1/0x440 [ 8772.609723] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8772.614804] RIP: 0033:0x7f53425a5332 [ 8772.617465] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8772.643530] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8772.652411] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8772.666806] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8772.672389] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8772.680482] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8772.689401] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8784.059351] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 8784.071640] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8784.131056] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8784.136685] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8784.149022] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8784.151210] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8784.192294] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8784.200798] RAX: 0000000000007f0f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8784.205245] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8784.217600] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8784.220453] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8784.231428] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8784.239928] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8784.243240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8784.250869] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8784.257016] Call Trace: [ 8784.259245] [ 8784.261939] ? show_regs.cold.9+0x22/0x2f [ 8784.266370] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8784.271618] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8784.274379] ? watchdog+0x40/0x40 [ 8784.275557] ? hrtimer_interrupt+0x128/0x2c0 [ 8784.286229] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8784.289544] ? apic_timer_interrupt+0xf/0x20 [ 8784.296544] [ 8784.298878] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8784.300627] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8784.304024] do_raw_spin_lock+0x177/0x190 [ 8784.305838] _raw_spin_lock+0x21/0x30 [ 8784.307306] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8784.310262] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8784.312490] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8784.315330] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8784.317471] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8784.319636] ? woken_wake_function+0x30/0x30 [ 8784.321286] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8784.323986] kthread+0x1d1/0x200 [ 8784.325226] ? set_kthread_struct+0x70/0x70 [ 8784.326694] ret_from_fork+0x1f/0x30 [ 8788.055523] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8788.059315] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8788.080180] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8788.091936] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8788.095183] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8788.102753] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8788.111275] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8788.132383] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8788.136191] RAX: 0000000000007f83 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8788.140840] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8788.148216] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8788.156547] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8788.163981] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8788.172024] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8788.176286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8788.181024] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8788.187842] Call Trace: [ 8788.189099] [ 8788.190884] ? show_regs.cold.9+0x22/0x2f [ 8788.193583] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8788.197348] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8788.201045] ? watchdog+0x40/0x40 [ 8788.203292] ? hrtimer_interrupt+0x128/0x2c0 [ 8788.207087] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8788.210253] ? apic_timer_interrupt+0xf/0x20 [ 8788.211778] [ 8788.212563] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8788.214520] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8788.219088] ? newidle_balance.isra.93+0x207/0x580 [ 8788.221097] do_raw_spin_lock+0x177/0x190 [ 8788.222491] _raw_spin_lock+0x21/0x30 [ 8788.224746] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8788.229912] process_one_work+0x2c8/0x700 [ 8788.231605] worker_thread+0x296/0x6e0 [ 8788.232761] ? rescuer_thread+0x570/0x570 [ 8788.244033] kthread+0x1d1/0x200 [ 8788.247461] ? set_kthread_struct+0x70/0x70 [ 8788.257150] ret_from_fork+0x1f/0x30 [ 8794.080130] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6156s! [ 8794.084964] Showing busy workqueues and worker pools: [ 8794.088325] workqueue events: flags=0x0 [ 8794.090858] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8794.095304] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8794.103388] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8794.107910] pending: cpu_latency_work [ptlrpc] [ 8794.112216] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8794.117018] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8794.122477] workqueue events_long: flags=0x0 [ 8794.124682] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8794.130539] pending: lu_objects_flush [obdclass] [ 8794.133745] workqueue events_unbound: flags=0x2 [ 8794.139225] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8794.144895] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8794.151043] workqueue events_power_efficient: flags=0x80 [ 8794.157036] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8794.164955] pending: check_lifetime, do_cache_clean [ 8794.166846] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8794.173426] pending: neigh_periodic_work, neigh_periodic_work [ 8794.180807] workqueue mm_percpu_wq: flags=0x8 [ 8794.186421] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8794.192765] pending: vmstat_update, lru_add_drain_per_cpu [ 8794.196239] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8794.199380] pending: vmstat_update, lru_add_drain_per_cpu [ 8794.209376] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8794.213132] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8794.216452] workqueue xprtiod: flags=0xa [ 8794.218784] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8794.221897] pending: xprt_autoclose, xs_error_handle [ 8794.224385] workqueue ipv6_addrconf: flags=0x40008 [ 8794.226343] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8794.229434] pending: addrconf_verify_work [ 8794.232429] workqueue knbd0-recv: flags=0x1a [ 8794.234204] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8794.237816] in-flight: 651:recv_work [ 8794.240058] workqueue ptlrpc_pinger: flags=0x2 [ 8794.242889] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8794.246201] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8794.249280] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6157s workers=4 idle: 15979 31824 8 [ 8794.252010] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8784s workers=2 idle: 654 [ 8800.055394] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 8800.057061] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 8800.058043] Modules linked in: lustre(O) [ 8800.072686] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) [ 8800.074364] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8800.076670] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8800.076680] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8800.076684] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8800.076701] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8800.076706] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8800.076710] RAX: 00000000000028eb RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8800.076712] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8800.076715] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8800.076717] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8800.076720] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8800.076723] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8800.076727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8800.076729] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8800.076739] Call Trace: [ 8800.076745] [ 8800.076752] ? show_regs.cold.9+0x22/0x2f [ 8800.076763] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8800.076769] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8800.076774] ? watchdog+0x40/0x40 [ 8800.076777] ? hrtimer_interrupt+0x128/0x2c0 [ 8800.076781] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8800.076789] ? apic_timer_interrupt+0xf/0x20 [ 8800.076794] [ 8800.076797] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8800.076805] do_raw_spin_lock+0x177/0x190 [ 8800.076811] _raw_spin_lock+0x21/0x30 [ 8800.076816] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8800.077127] ? do_raw_spin_unlock+0x75/0x190 [ 8800.077133] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8800.077325] ? do_wait_intr+0xf0/0xf0 [ 8800.077334] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8800.078159] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8800.078388] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8800.088358] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8800.088489] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8800.088497] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8800.088499] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8800.088514] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8800.088518] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8800.088523] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8800.088525] RDX: 000000000000727d RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8800.088528] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8800.088530] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8800.088532] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8800.088536] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8800.088538] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8800.088541] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8800.088549] Call Trace: [ 8800.088555] [ 8800.088560] ? show_regs.cold.9+0x22/0x2f [ 8800.088569] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8800.088576] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8800.088581] ? watchdog+0x40/0x40 [ 8800.088584] ? hrtimer_interrupt+0x128/0x2c0 [ 8800.088588] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8800.088597] ? apic_timer_interrupt+0xf/0x20 [ 8800.088601] [ 8800.088604] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8800.088609] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8800.088613] do_raw_spin_lock+0x177/0x190 [ 8800.088618] _raw_spin_lock+0x21/0x30 [ 8800.088624] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8800.088883] obd_mod_free+0x46/0x140 [ptlrpc] [ 8800.089081] mdc_commit_open+0x91/0xb0 [mdc] [ 8800.120662] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8800.120796] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8800.120833] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8800.120863] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8800.120890] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8800.120973] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8800.121274] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8800.121308] lmv_intent_open+0x338/0x1150 [lmv] [ 8800.121339] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8800.121443] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8800.121530] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8800.121552] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8800.121636] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8800.121719] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8800.121802] ? do_raw_spin_unlock+0x75/0x190 [ 8800.121810] ? _raw_spin_unlock+0x12/0x30 [ 8800.121817] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8800.121847] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8800.121866] ll_atomic_open+0x36a/0xd70 [lustre] [ 8800.121949] lookup_open+0x356/0xb40 [ 8800.121959] do_last+0x313/0xfc0 [ 8800.121964] ? nd_jump_root+0xe5/0x160 [ 8800.121969] ? path_init+0x437/0x520 [ 8800.121973] path_openat+0xf7/0x500 [ 8800.121978] ? do_raw_spin_unlock+0x75/0x190 [ 8800.121982] do_filp_open+0x99/0x140 [ 8800.121989] ? getname_flags+0x6e/0x330 [ 8800.121994] ? __check_object_size+0xff/0x256 [ 8800.122000] ? do_raw_spin_unlock+0x75/0x190 [ 8800.122004] ? _raw_spin_unlock+0x12/0x30 [ 8800.122008] do_sys_openat2+0x2b4/0x410 [ 8800.122012] do_sys_open+0x73/0xa0 [ 8800.122016] __x64_sys_openat+0x24/0x30 [ 8800.122019] do_syscall_64+0xc1/0x440 [ 8800.122026] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8800.122032] RIP: 0033:0x7f53425a5332 [ 8800.122040] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8800.122044] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8800.122049] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8800.122051] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8800.122053] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8800.122055] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8800.122057] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8800.798659] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8800.801044] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8800.805481] after_reply+0xe2c/0x1780 [ptlrpc] [ 8800.809522] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8800.815190] ? __next_timer_interrupt+0x160/0x160 [ 8800.817261] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8800.823050] ? do_wait_intr+0xf0/0xf0 [ 8800.825214] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8800.830147] kthread+0x1d1/0x200 [ 8800.834474] ? set_kthread_struct+0x70/0x70 [ 8800.836051] ret_from_fork+0x1f/0x30 [ 8812.061120] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 8812.069501] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8812.119584] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8812.133163] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8812.138425] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8812.141665] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8812.155292] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8812.161032] RAX: 000000000000678a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8812.165709] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8812.173863] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8812.181668] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8812.185964] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8812.193930] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8812.204037] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8812.211870] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8812.219671] Call Trace: [ 8812.226610] [ 8812.227669] ? show_regs.cold.9+0x22/0x2f [ 8812.228665] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8812.230174] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8812.238162] ? watchdog+0x40/0x40 [ 8812.240950] ? hrtimer_interrupt+0x128/0x2c0 [ 8812.242964] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8812.245648] ? apic_timer_interrupt+0xf/0x20 [ 8812.249998] [ 8812.253442] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8812.261339] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8812.266846] do_raw_spin_lock+0x177/0x190 [ 8812.268668] _raw_spin_lock+0x21/0x30 [ 8812.271135] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8812.276789] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8812.279663] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8812.285819] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8812.291757] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8812.302623] ? woken_wake_function+0x30/0x30 [ 8812.309191] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8812.317136] kthread+0x1d1/0x200 [ 8812.322343] ? set_kthread_struct+0x70/0x70 [ 8812.328746] ret_from_fork+0x1f/0x30 [ 8814.597945] rcu: INFO: rcu_sched self-detected stall on CPU [ 8814.601225] rcu: 2-....: (3652786 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 8814.616390] (t=6720171 jiffies g=274185 q=65577) [ 8814.622089] rcu: rcu_sched kthread starved for 6178537 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 8814.630899] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8814.643038] rcu: RCU grace-period kthread stack dump: [ 8814.644851] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8814.654105] Call Trace: [ 8814.656666] __schedule+0x351/0xcb0 [ 8814.666429] schedule+0xc0/0x180 [ 8814.668192] schedule_timeout+0xb4/0x190 [ 8814.670888] ? do_raw_spin_unlock+0x75/0x190 [ 8814.672809] ? __next_timer_interrupt+0x160/0x160 [ 8814.675529] ? prepare_to_swait_event+0xf3/0x140 [ 8814.677116] rcu_gp_kthread+0x8dd/0x15b0 [ 8814.683353] ? force_qs_rnp+0x2c0/0x2c0 [ 8814.690105] kthread+0x1d1/0x200 [ 8814.693963] ? set_kthread_struct+0x70/0x70 [ 8814.697131] ret_from_fork+0x1f/0x30 [ 8814.700402] rcu: Stack dump where RCU GP kthread last ran: [ 8814.703588] Sending NMI from CPU 2 to CPUs 1: [ 8814.712541] NMI backtrace for cpu 1 [ 8814.712549] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8814.712551] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8814.712552] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8814.712554] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8814.712556] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [ 8814.712559] RAX: 000000000000038b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8814.712561] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8814.712562] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8814.712564] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8814.712565] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8814.712567] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8814.712569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8814.712570] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8814.712572] Call Trace: [ 8814.712572] [ 8814.712573] ? show_regs.cold.9+0x22/0x2f [ 8814.712575] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8814.712576] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8814.712577] ? nmi_handle+0xbd/0x210 [ 8814.712578] ? default_do_nmi+0x4b/0x1b0 [ 8814.712579] ? do_nmi+0x280/0x380 [ 8814.712580] ? end_repeat_nmi+0x16/0x1f [ 8814.712582] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8814.712583] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8814.712584] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8814.712585] [ 8814.712586] do_raw_spin_lock+0x177/0x190 [ 8814.712587] _raw_spin_lock+0x21/0x30 [ 8814.712589] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8814.712590] ? do_raw_spin_unlock+0x75/0x190 [ 8814.712591] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8814.712592] ? do_wait_intr+0xf0/0xf0 [ 8814.712593] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8814.712594] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8814.712595] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8814.712596] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8814.712598] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8814.712599] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8814.712600] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8814.712602] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8814.712603] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8814.712604] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8814.712606] lmv_intent_open+0x338/0x1150 [lmv] [ 8814.712607] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8814.712608] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8814.712609] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8814.712610] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8814.712693] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8814.712695] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8814.712695] ? do_raw_spin_unlock+0x75/0x190 [ 8814.712697] ? _raw_spin_unlock+0x12/0x30 [ 8814.712698] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8814.712699] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8814.712700] ll_atomic_open+0x36a/0xd70 [lustre] [ 8814.712701] lookup_open+0x356/0xb40 [ 8814.712703] do_last+0x313/0xfc0 [ 8814.712704] ? nd_jump_root+0xe5/0x160 [ 8814.712705] ? path_init+0x437/0x520 [ 8814.712706] path_openat+0xf7/0x500 [ 8814.712780] ? do_raw_spin_unlock+0x75/0x190 [ 8814.712783] do_filp_open+0x99/0x140 [ 8814.712784] ? getname_flags+0x6e/0x330 [ 8814.712789] ? __check_object_size+0xff/0x256 [ 8814.712791] ? do_raw_spin_unlock+0x75/0x190 [ 8814.712792] ? _raw_spin_unlock+0x12/0x30 [ 8814.712793] do_sys_openat2+0x2b4/0x410 [ 8814.712795] do_sys_open+0x73/0xa0 [ 8814.712796] __x64_sys_openat+0x24/0x30 [ 8814.712797] do_syscall_64+0xc1/0x440 [ 8814.712799] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8814.712800] RIP: 0033:0x7f53425a5332 [ 8814.712802] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8814.712804] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8814.712808] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8814.712810] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8814.712812] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8814.712814] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8814.712815] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8814.712980] Sending NMI from CPU 2 to CPUs 1: [ 8815.111482] NMI backtrace for cpu 1 [ 8815.111489] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8815.111491] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8815.111492] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8815.111494] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8815.111496] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [ 8815.111498] RAX: 0000000000003b92 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8815.111499] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8815.111501] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8815.111502] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8815.111503] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8815.111504] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8815.111505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8815.111506] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8815.111507] Call Trace: [ 8815.111508] [ 8815.111509] ? show_regs.cold.9+0x22/0x2f [ 8815.111510] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8815.111511] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8815.111512] ? nmi_handle+0xbd/0x210 [ 8815.111513] ? default_do_nmi+0x4b/0x1b0 [ 8815.111514] ? do_nmi+0x280/0x380 [ 8815.111515] ? end_repeat_nmi+0x16/0x1f [ 8815.111516] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8815.111517] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8815.111518] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8815.111519] [ 8815.111520] do_raw_spin_lock+0x177/0x190 [ 8815.111521] _raw_spin_lock+0x21/0x30 [ 8815.111521] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8815.111523] ? do_raw_spin_unlock+0x75/0x190 [ 8815.111523] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8815.111524] ? do_wait_intr+0xf0/0xf0 [ 8815.111525] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8815.111526] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8815.111527] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8815.111529] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8815.111530] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8815.111531] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8815.111532] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8815.111533] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8815.111534] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8815.111535] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8815.111535] lmv_intent_open+0x338/0x1150 [lmv] [ 8815.111536] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8815.111537] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8815.111538] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8815.111539] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8815.111540] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8815.111541] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8815.111542] ? do_raw_spin_unlock+0x75/0x190 [ 8815.111543] ? _raw_spin_unlock+0x12/0x30 [ 8815.111544] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8815.111546] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8815.111547] ll_atomic_open+0x36a/0xd70 [lustre] [ 8815.111548] lookup_open+0x356/0xb40 [ 8815.111549] do_last+0x313/0xfc0 [ 8815.111550] ? nd_jump_root+0xe5/0x160 [ 8815.111551] ? path_init+0x437/0x520 [ 8815.111551] path_openat+0xf7/0x500 [ 8815.111552] ? do_raw_spin_unlock+0x75/0x190 [ 8815.111553] do_filp_open+0x99/0x140 [ 8815.111554] ? getname_flags+0x6e/0x330 [ 8815.111555] ? __check_object_size+0xff/0x256 [ 8815.111556] ? do_raw_spin_unlock+0x75/0x190 [ 8815.111557] ? _raw_spin_unlock+0x12/0x30 [ 8815.111558] do_sys_openat2+0x2b4/0x410 [ 8815.111559] do_sys_open+0x73/0xa0 [ 8815.111560] __x64_sys_openat+0x24/0x30 [ 8815.111561] do_syscall_64+0xc1/0x440 [ 8815.111562] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8815.111563] RIP: 0033:0x7f53425a5332 [ 8815.111564] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8815.111566] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8815.111569] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8815.111570] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8815.111571] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8815.111572] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8815.111574] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8815.112395] NMI backtrace for cpu 2 [ 8815.531784] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8815.540436] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8815.546625] Call Trace: [ 8815.548724] [ 8815.552640] ? dump_stack+0xbb/0x10e [ 8815.555862] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8815.565905] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8815.567666] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8815.577415] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8815.585196] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8815.587127] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8815.595751] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8815.600629] ? update_process_times+0x6d/0xc0 [ 8815.615598] ? tick_sched_handle.isra.18+0x27/0x90 [ 8815.626538] ? tick_sched_timer+0x53/0xb0 [ 8815.630252] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8815.633602] ? tick_sched_handle.isra.18+0x90/0x90 [ 8815.637649] ? hrtimer_interrupt+0x128/0x2c0 [ 8815.648118] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8815.650675] ? apic_timer_interrupt+0xf/0x20 [ 8815.656752] [ 8815.666544] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8815.681087] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8815.690850] ? do_raw_spin_lock+0x177/0x190 [ 8815.700230] ? _raw_spin_lock+0x21/0x30 [ 8815.701964] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8815.711811] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8815.719745] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8815.724246] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8815.734354] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8815.750494] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8815.760528] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8815.768318] ? __next_timer_interrupt+0x160/0x160 [ 8815.774514] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8815.777393] ? do_wait_intr+0xf0/0xf0 [ 8815.780774] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8815.788838] ? kthread+0x1d1/0x200 [ 8815.792730] ? set_kthread_struct+0x70/0x70 [ 8815.797548] ? ret_from_fork+0x1f/0x30 [ 8820.052240] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 8820.058552] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8820.082330] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8820.087293] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8820.095761] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8820.098960] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8820.101011] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8820.109678] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8820.112825] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8820.116604] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8820.123090] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8820.127815] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8820.132322] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8820.135819] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8820.140508] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8820.145053] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8820.149661] Call Trace: [ 8820.151053] [ 8820.152294] ? show_regs.cold.9+0x22/0x2f [ 8820.157398] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8820.159949] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8820.164053] ? watchdog+0x40/0x40 [ 8820.169069] ? hrtimer_interrupt+0x128/0x2c0 [ 8820.171367] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8820.173700] ? apic_timer_interrupt+0xf/0x20 [ 8820.177557] [ 8820.180349] ? native_safe_halt+0x2e/0x50 [ 8820.187601] kvm_wait+0xcd/0xe0 [ 8820.192845] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8820.203387] ? newidle_balance.isra.93+0x207/0x580 [ 8820.209533] do_raw_spin_lock+0x177/0x190 [ 8820.213766] _raw_spin_lock+0x21/0x30 [ 8820.217561] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8820.222737] process_one_work+0x2c8/0x700 [ 8820.226578] worker_thread+0x296/0x6e0 [ 8820.228424] ? rescuer_thread+0x570/0x570 [ 8820.230577] kthread+0x1d1/0x200 [ 8820.231812] ? set_kthread_struct+0x70/0x70 [ 8820.233292] ret_from_fork+0x1f/0x30 [ 8824.293457] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6187s! [ 8824.301377] Showing busy workqueues and worker pools: [ 8824.308050] workqueue events: flags=0x0 [ 8824.309347] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8824.320943] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8824.334453] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8824.346527] pending: cpu_latency_work [ptlrpc] [ 8824.352239] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8824.364948] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8824.372924] workqueue events_long: flags=0x0 [ 8824.381794] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8824.393109] pending: lu_objects_flush [obdclass] [ 8824.399167] workqueue events_unbound: flags=0x2 [ 8824.402536] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8824.408993] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8824.411903] workqueue events_power_efficient: flags=0x80 [ 8824.416940] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8824.432735] pending: check_lifetime, do_cache_clean [ 8824.436555] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8824.450753] pending: neigh_periodic_work, neigh_periodic_work [ 8824.460396] workqueue mm_percpu_wq: flags=0x8 [ 8824.468127] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8824.474523] pending: vmstat_update, lru_add_drain_per_cpu [ 8824.478396] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8824.483534] pending: vmstat_update, lru_add_drain_per_cpu [ 8824.491404] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8824.503509] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8824.515659] workqueue xprtiod: flags=0xa [ 8824.519068] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8824.522443] pending: xprt_autoclose, xs_error_handle [ 8824.533555] workqueue ipv6_addrconf: flags=0x40008 [ 8824.537768] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8824.546512] pending: addrconf_verify_work [ 8824.555416] workqueue knbd0-recv: flags=0x1a [ 8824.566399] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8824.574151] in-flight: 651:recv_work [ 8824.577137] workqueue ptlrpc_pinger: flags=0x2 [ 8824.580030] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8824.586349] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8824.595828] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6187s workers=4 idle: 15979 31824 8 [ 8824.607655] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8815s workers=2 idle: 654 [ 8840.057793] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 8840.057888] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 8840.060181] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 8840.060195] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8840.060348] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8840.060353] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8840.060357] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8840.060386] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8840.060390] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8840.060395] RAX: 0000000000007385 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8840.060397] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8840.060400] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8840.060402] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8840.060404] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8840.060407] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8840.060411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8840.060413] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8840.060422] Call Trace: [ 8840.060438] [ 8840.060444] ? show_regs.cold.9+0x22/0x2f [ 8840.060458] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8840.060464] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8840.060468] ? watchdog+0x40/0x40 [ 8840.060472] ? hrtimer_interrupt+0x128/0x2c0 [ 8840.060477] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8840.060486] ? apic_timer_interrupt+0xf/0x20 [ 8840.060491] [ 8840.060494] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8840.060499] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8840.060505] do_raw_spin_lock+0x177/0x190 [ 8840.060511] _raw_spin_lock+0x21/0x30 [ 8840.060515] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8840.060814] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8840.061015] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8840.063526] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8840.063755] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8840.063965] ? woken_wake_function+0x30/0x30 [ 8840.063974] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8840.065297] kthread+0x1d1/0x200 [ 8840.065305] ? set_kthread_struct+0x70/0x70 [ 8840.065309] ret_from_fork+0x1f/0x30 [ 8840.068958] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8840.069035] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8840.069043] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8840.069046] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8840.069060] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8840.069065] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8840.069070] RAX: 0000000000006d92 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8840.069072] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8840.069075] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8840.069077] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8840.069079] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8840.069083] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8840.069085] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8840.069088] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8840.069097] Call Trace: [ 8840.069102] [ 8840.069108] ? show_regs.cold.9+0x22/0x2f [ 8840.069117] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8840.069124] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8840.069129] ? watchdog+0x40/0x40 [ 8840.069133] ? hrtimer_interrupt+0x128/0x2c0 [ 8840.069137] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8840.069147] ? apic_timer_interrupt+0xf/0x20 [ 8840.069152] [ 8840.069155] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8840.069162] do_raw_spin_lock+0x177/0x190 [ 8840.069167] _raw_spin_lock+0x21/0x30 [ 8840.069172] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8840.069453] ? do_raw_spin_unlock+0x75/0x190 [ 8840.069459] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8840.088856] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8840.088933] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8840.088940] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8840.088943] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8840.088971] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8840.088976] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8840.088981] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8840.088983] RDX: 0000000000002410 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8840.088986] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8840.088989] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8840.088992] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8840.088995] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8840.088998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8840.089000] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8840.089008] Call Trace: [ 8840.089015] [ 8840.089026] ? show_regs.cold.9+0x22/0x2f [ 8840.089036] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8840.089042] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8840.089047] ? watchdog+0x40/0x40 [ 8840.089056] ? hrtimer_interrupt+0x128/0x2c0 [ 8840.089062] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8840.089073] ? apic_timer_interrupt+0xf/0x20 [ 8840.089078] [ 8840.089080] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8840.089086] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8840.089091] do_raw_spin_lock+0x177/0x190 [ 8840.089095] _raw_spin_lock+0x21/0x30 [ 8840.089102] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8840.089615] obd_mod_free+0x46/0x140 [ptlrpc] [ 8840.089846] mdc_commit_open+0x91/0xb0 [mdc] [ 8840.089886] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8840.090202] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8840.090410] after_reply+0xe2c/0x1780 [ptlrpc] [ 8840.090689] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8840.091042] ? __next_timer_interrupt+0x160/0x160 [ 8840.091052] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8840.091405] ? do_wait_intr+0xf0/0xf0 [ 8840.091414] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8840.119398] ? do_wait_intr+0xf0/0xf0 [ 8840.128965] kthread+0x1d1/0x200 [ 8840.129807] ? set_kthread_struct+0x70/0x70 [ 8840.129812] ret_from_fork+0x1f/0x30 [ 8841.049032] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8841.057143] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8841.063217] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8841.070225] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8841.072838] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8841.079677] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8841.087550] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8841.095137] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8841.105283] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8841.111585] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8841.125076] lmv_intent_open+0x338/0x1150 [lmv] [ 8841.132074] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8841.134065] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8841.147434] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8841.152152] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8841.159390] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8841.167400] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8841.172332] ? do_raw_spin_unlock+0x75/0x190 [ 8841.174300] ? _raw_spin_unlock+0x12/0x30 [ 8841.181157] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8841.187304] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8841.189443] ll_atomic_open+0x36a/0xd70 [lustre] [ 8841.195554] lookup_open+0x356/0xb40 [ 8841.196816] do_last+0x313/0xfc0 [ 8841.197936] ? nd_jump_root+0xe5/0x160 [ 8841.199921] ? path_init+0x437/0x520 [ 8841.201967] path_openat+0xf7/0x500 [ 8841.207434] ? do_raw_spin_unlock+0x75/0x190 [ 8841.209555] do_filp_open+0x99/0x140 [ 8841.218335] ? getname_flags+0x6e/0x330 [ 8841.220617] ? __check_object_size+0xff/0x256 [ 8841.225820] ? do_raw_spin_unlock+0x75/0x190 [ 8841.230278] ? _raw_spin_unlock+0x12/0x30 [ 8841.240157] do_sys_openat2+0x2b4/0x410 [ 8841.243026] do_sys_open+0x73/0xa0 [ 8841.246765] __x64_sys_openat+0x24/0x30 [ 8841.251633] do_syscall_64+0xc1/0x440 [ 8841.256964] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8841.265008] RIP: 0033:0x7f53425a5332 [ 8841.270164] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8841.286193] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8841.294553] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8841.299636] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8841.306950] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8841.311141] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8841.324688] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8852.052088] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 8852.056799] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8852.095122] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8852.104159] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8852.111793] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8852.117095] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8852.120475] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8852.134077] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8852.136437] RAX: 0000000000007aaa RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8852.143912] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8852.153170] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8852.162116] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8852.169189] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8852.175803] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8852.183400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8852.188268] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8852.192162] Call Trace: [ 8852.198498] [ 8852.200466] ? show_regs.cold.9+0x22/0x2f [ 8852.205946] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8852.207838] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8852.213098] ? watchdog+0x40/0x40 [ 8852.214300] ? hrtimer_interrupt+0x128/0x2c0 [ 8852.215574] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8852.223422] ? apic_timer_interrupt+0xf/0x20 [ 8852.224603] [ 8852.225324] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8852.230571] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8852.232392] ? newidle_balance.isra.93+0x207/0x580 [ 8852.237672] do_raw_spin_lock+0x177/0x190 [ 8852.239376] _raw_spin_lock+0x21/0x30 [ 8852.244855] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8852.248324] process_one_work+0x2c8/0x700 [ 8852.252811] worker_thread+0x296/0x6e0 [ 8852.253800] ? rescuer_thread+0x570/0x570 [ 8852.258960] kthread+0x1d1/0x200 [ 8852.259794] ? set_kthread_struct+0x70/0x70 [ 8852.261280] ret_from_fork+0x1f/0x30 [ 8855.009431] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6217s! [ 8855.012674] Showing busy workqueues and worker pools: [ 8855.014692] workqueue events: flags=0x0 [ 8855.016176] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8855.018687] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8855.023250] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8855.025785] pending: cpu_latency_work [ptlrpc] [ 8855.027879] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8855.030659] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8855.034226] workqueue events_long: flags=0x0 [ 8855.035877] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8855.038213] pending: lu_objects_flush [obdclass] [ 8855.043383] workqueue events_unbound: flags=0x2 [ 8855.045439] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8855.050800] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8855.053363] workqueue events_power_efficient: flags=0x80 [ 8855.058031] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8855.062301] pending: check_lifetime, do_cache_clean [ 8855.068137] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8855.070880] pending: neigh_periodic_work, neigh_periodic_work [ 8855.077270] workqueue mm_percpu_wq: flags=0x8 [ 8855.079208] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8855.088718] pending: vmstat_update, lru_add_drain_per_cpu [ 8855.093407] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8855.096160] pending: vmstat_update, lru_add_drain_per_cpu [ 8855.102231] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8855.107235] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8855.112815] workqueue xprtiod: flags=0xa [ 8855.115715] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8855.117789] pending: xprt_autoclose, xs_error_handle [ 8855.123456] workqueue ipv6_addrconf: flags=0x40008 [ 8855.126872] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8855.129065] pending: addrconf_verify_work [ 8855.133580] workqueue knbd0-recv: flags=0x1a [ 8855.135943] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8855.138402] in-flight: 651:recv_work [ 8855.143171] workqueue ptlrpc_pinger: flags=0x2 [ 8855.146088] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8855.151289] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8855.157695] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6218s workers=4 idle: 15979 31824 8 [ 8855.164129] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8845s workers=2 idle: 654 [ 8868.056091] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 8868.057070] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 8868.058029] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 8868.058038] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8868.058115] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8868.058122] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8868.058125] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8868.058141] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8868.058148] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8868.058153] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8868.058156] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8868.058158] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8868.058160] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8868.058163] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8868.058173] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8868.058176] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8868.058178] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8868.058187] Call Trace: [ 8868.058198] [ 8868.058208] ? show_regs.cold.9+0x22/0x2f [ 8868.058223] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8868.058232] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8868.058246] ? watchdog+0x40/0x40 [ 8868.058253] ? hrtimer_interrupt+0x128/0x2c0 [ 8868.058257] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8868.058271] ? apic_timer_interrupt+0xf/0x20 [ 8868.058280] [ 8868.058283] ? native_safe_halt+0x2e/0x50 [ 8868.058288] kvm_wait+0xcd/0xe0 [ 8868.058304] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8868.058317] do_raw_spin_lock+0x177/0x190 [ 8868.058322] _raw_spin_lock+0x21/0x30 [ 8868.058326] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8868.058832] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8868.059068] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8868.059329] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8868.059557] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8868.059790] ? woken_wake_function+0x30/0x30 [ 8868.059800] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8868.060205] kthread+0x1d1/0x200 [ 8868.060212] ? set_kthread_struct+0x70/0x70 [ 8868.060216] ret_from_fork+0x1f/0x30 [ 8868.067500] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8868.067588] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8868.067595] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8868.067599] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8868.067613] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8868.067617] RSP: 0018:ffffbb5381503500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 8868.067622] RAX: 0000000000002f03 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8868.067625] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8868.067627] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8868.067630] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8868.067632] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8868.067636] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8868.067639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8868.067641] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8868.067655] Call Trace: [ 8868.067661] [ 8868.067667] ? show_regs.cold.9+0x22/0x2f [ 8868.067676] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8868.067682] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8868.067686] ? watchdog+0x40/0x40 [ 8868.067690] ? hrtimer_interrupt+0x128/0x2c0 [ 8868.067694] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8868.067703] ? apic_timer_interrupt+0xf/0x20 [ 8868.067707] [ 8868.067709] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8868.067715] do_raw_spin_lock+0x177/0x190 [ 8868.067720] _raw_spin_lock+0x21/0x30 [ 8868.067724] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8868.068033] ? do_raw_spin_unlock+0x75/0x190 [ 8868.068039] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8868.068266] ? do_wait_intr+0xf0/0xf0 [ 8868.068275] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8868.068475] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8868.068699] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8868.068916] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8868.069047] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8868.069090] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8868.069121] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8868.069147] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8868.069231] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8868.069468] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8868.069497] lmv_intent_open+0x338/0x1150 [lmv] [ 8868.069527] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8868.069616] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8868.069699] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8868.069722] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8868.075934] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8868.076025] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8868.076032] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8868.076035] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8868.076049] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8868.076053] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8868.076057] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8868.076060] RDX: 0000000000002714 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8868.076062] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8868.076065] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8868.076068] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8868.076071] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8868.076074] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8868.076076] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8868.076084] Call Trace: [ 8868.076089] [ 8868.076095] ? show_regs.cold.9+0x22/0x2f [ 8868.076111] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8868.076118] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8868.076123] ? watchdog+0x40/0x40 [ 8868.076127] ? hrtimer_interrupt+0x128/0x2c0 [ 8868.076131] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8868.076141] ? apic_timer_interrupt+0xf/0x20 [ 8868.076146] [ 8868.076149] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8868.076153] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8868.076158] do_raw_spin_lock+0x177/0x190 [ 8868.076165] _raw_spin_lock+0x21/0x30 [ 8868.076170] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8868.076549] obd_mod_free+0x46/0x140 [ptlrpc] [ 8868.076774] mdc_commit_open+0x91/0xb0 [mdc] [ 8868.076896] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8868.077399] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8868.077625] after_reply+0xe2c/0x1780 [ptlrpc] [ 8868.077854] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8868.078082] ? __next_timer_interrupt+0x160/0x160 [ 8868.078094] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8868.078309] ? do_wait_intr+0xf0/0xf0 [ 8868.078319] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8868.078527] kthread+0x1d1/0x200 [ 8868.078534] ? set_kthread_struct+0x70/0x70 [ 8868.078537] ret_from_fork+0x1f/0x30 [ 8868.689162] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8868.692256] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8868.695802] ? do_raw_spin_unlock+0x75/0x190 [ 8868.698044] ? _raw_spin_unlock+0x12/0x30 [ 8868.700431] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8868.703788] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8868.709855] ll_atomic_open+0x36a/0xd70 [lustre] [ 8868.714246] lookup_open+0x356/0xb40 [ 8868.719114] do_last+0x313/0xfc0 [ 8868.723032] ? nd_jump_root+0xe5/0x160 [ 8868.727680] ? path_init+0x437/0x520 [ 8868.731805] path_openat+0xf7/0x500 [ 8868.735900] ? do_raw_spin_unlock+0x75/0x190 [ 8868.740034] do_filp_open+0x99/0x140 [ 8868.744060] ? getname_flags+0x6e/0x330 [ 8868.747895] ? __check_object_size+0xff/0x256 [ 8868.755789] ? do_raw_spin_unlock+0x75/0x190 [ 8868.762070] ? _raw_spin_unlock+0x12/0x30 [ 8868.765492] do_sys_openat2+0x2b4/0x410 [ 8868.771800] do_sys_open+0x73/0xa0 [ 8868.774077] __x64_sys_openat+0x24/0x30 [ 8868.776215] do_syscall_64+0xc1/0x440 [ 8868.778305] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8868.782093] RIP: 0033:0x7f53425a5332 [ 8868.783869] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8868.802060] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8868.808735] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8868.816655] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8868.823715] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8868.829279] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8868.835561] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8880.052841] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 8880.055753] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8880.085896] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8880.091695] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8880.097167] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8880.101829] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8880.104142] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8880.115788] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8880.122442] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8880.124908] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8880.129303] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8880.133898] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8880.138083] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8880.143264] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8880.148694] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8880.159885] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8880.167908] Call Trace: [ 8880.171125] [ 8880.175689] ? show_regs.cold.9+0x22/0x2f [ 8880.179820] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8880.186901] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8880.188281] ? watchdog+0x40/0x40 [ 8880.188991] ? hrtimer_interrupt+0x128/0x2c0 [ 8880.192272] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8880.194744] ? apic_timer_interrupt+0xf/0x20 [ 8880.198927] [ 8880.199687] ? native_safe_halt+0x2e/0x50 [ 8880.201537] kvm_wait+0xcd/0xe0 [ 8880.202593] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8880.204285] ? newidle_balance.isra.93+0x207/0x580 [ 8880.208570] do_raw_spin_lock+0x177/0x190 [ 8880.209735] _raw_spin_lock+0x21/0x30 [ 8880.211401] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8880.213172] process_one_work+0x2c8/0x700 [ 8880.214606] worker_thread+0x296/0x6e0 [ 8880.216888] ? rescuer_thread+0x570/0x570 [ 8880.218778] kthread+0x1d1/0x200 [ 8880.219722] ? set_kthread_struct+0x70/0x70 [ 8880.222007] ret_from_fork+0x1f/0x30 [ 8885.219030] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6248s! [ 8885.223988] Showing busy workqueues and worker pools: [ 8885.226344] workqueue events: flags=0x0 [ 8885.228425] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8885.233024] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8885.238767] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8885.243901] pending: cpu_latency_work [ptlrpc] [ 8885.247255] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8885.251230] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8885.257137] workqueue events_long: flags=0x0 [ 8885.259103] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8885.266259] pending: lu_objects_flush [obdclass] [ 8885.271166] workqueue events_unbound: flags=0x2 [ 8885.277657] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8885.288753] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8885.299183] workqueue events_power_efficient: flags=0x80 [ 8885.307174] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8885.311874] pending: check_lifetime, do_cache_clean [ 8885.317050] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8885.330336] pending: neigh_periodic_work, neigh_periodic_work [ 8885.335916] workqueue mm_percpu_wq: flags=0x8 [ 8885.337527] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8885.361197] pending: vmstat_update, lru_add_drain_per_cpu [ 8885.370059] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8885.380778] pending: vmstat_update, lru_add_drain_per_cpu [ 8885.387284] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8885.392518] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8885.396629] workqueue xprtiod: flags=0xa [ 8885.401612] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8885.407583] pending: xprt_autoclose, xs_error_handle [ 8885.411582] workqueue ipv6_addrconf: flags=0x40008 [ 8885.414760] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8885.418894] pending: addrconf_verify_work [ 8885.422226] workqueue knbd0-recv: flags=0x1a [ 8885.425251] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8885.429229] in-flight: 651:recv_work [ 8885.433221] workqueue ptlrpc_pinger: flags=0x2 [ 8885.436134] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8885.442237] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8885.455702] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6248s workers=4 idle: 15979 31824 8 [ 8885.472013] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8875s workers=2 idle: 654 [ 8896.055933] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 8896.057318] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 8896.058013] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 8896.058029] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8896.058103] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8896.058109] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8896.058112] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8896.058125] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8896.058129] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8896.058134] RAX: 0000000000004035 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8896.058137] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8896.058139] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8896.058141] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8896.058144] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8896.058148] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8896.058151] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8896.058153] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8896.058161] Call Trace: [ 8896.058166] [ 8896.058171] ? show_regs.cold.9+0x22/0x2f [ 8896.058180] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8896.058186] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8896.058190] ? watchdog+0x40/0x40 [ 8896.058194] ? hrtimer_interrupt+0x128/0x2c0 [ 8896.058198] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8896.058206] ? apic_timer_interrupt+0xf/0x20 [ 8896.058211] [ 8896.058214] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8896.058218] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8896.058224] do_raw_spin_lock+0x177/0x190 [ 8896.058229] _raw_spin_lock+0x21/0x30 [ 8896.058234] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8896.058786] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8896.059011] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8896.059235] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8896.059446] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8896.059806] ? woken_wake_function+0x30/0x30 [ 8896.059816] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8896.060138] kthread+0x1d1/0x200 [ 8896.060144] ? set_kthread_struct+0x70/0x70 [ 8896.060148] ret_from_fork+0x1f/0x30 [ 8896.063455] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8896.063537] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8896.063546] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8896.063549] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8896.063563] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8896.063568] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8896.063573] RAX: 0000000000007aa0 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8896.063576] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8896.063590] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8896.063593] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8896.063595] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8896.063598] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8896.063601] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8896.063604] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8896.063612] Call Trace: [ 8896.063617] [ 8896.063623] ? show_regs.cold.9+0x22/0x2f [ 8896.063632] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8896.063639] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8896.063644] ? watchdog+0x40/0x40 [ 8896.063648] ? hrtimer_interrupt+0x128/0x2c0 [ 8896.063652] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8896.063660] ? apic_timer_interrupt+0xf/0x20 [ 8896.063665] [ 8896.063668] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8896.063674] do_raw_spin_lock+0x177/0x190 [ 8896.063679] _raw_spin_lock+0x21/0x30 [ 8896.063684] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8896.064031] ? do_raw_spin_unlock+0x75/0x190 [ 8896.064037] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8896.064237] ? do_wait_intr+0xf0/0xf0 [ 8896.064246] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8896.064439] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8896.064651] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8896.064877] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8896.065014] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8896.065059] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8896.065090] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8896.065119] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8896.065204] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8896.065418] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8896.065446] lmv_intent_open+0x338/0x1150 [lmv] [ 8896.065487] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8896.065572] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8896.065659] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8896.065683] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8896.065789] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8896.065878] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8896.065961] ? do_raw_spin_unlock+0x75/0x190 [ 8896.065969] ? _raw_spin_unlock+0x12/0x30 [ 8896.065977] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8896.066012] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8896.066032] ll_atomic_open+0x36a/0xd70 [lustre] [ 8896.066113] lookup_open+0x356/0xb40 [ 8896.066128] do_last+0x313/0xfc0 [ 8896.066132] ? nd_jump_root+0xe5/0x160 [ 8896.066137] ? path_init+0x437/0x520 [ 8896.066142] path_openat+0xf7/0x500 [ 8896.066147] ? do_raw_spin_unlock+0x75/0x190 [ 8896.066152] do_filp_open+0x99/0x140 [ 8896.066160] ? getname_flags+0x6e/0x330 [ 8896.066165] ? __check_object_size+0xff/0x256 [ 8896.066173] ? do_raw_spin_unlock+0x75/0x190 [ 8896.066177] ? _raw_spin_unlock+0x12/0x30 [ 8896.066182] do_sys_openat2+0x2b4/0x410 [ 8896.066187] do_sys_open+0x73/0xa0 [ 8896.066192] __x64_sys_openat+0x24/0x30 [ 8896.066196] do_syscall_64+0xc1/0x440 [ 8896.066204] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8896.066211] RIP: 0033:0x7f53425a5332 [ 8896.066219] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8896.066223] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8896.066228] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8896.066231] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8896.066233] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8896.066235] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8896.066237] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8896.754377] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8896.793947] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8896.798693] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8896.803192] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8896.806303] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8896.818771] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8896.826849] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8896.828992] RDX: 00000000000031bd RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8896.839374] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8896.846248] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8896.852579] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8896.856987] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8896.865630] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8896.868621] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8896.871043] Call Trace: [ 8896.872006] [ 8896.872863] ? show_regs.cold.9+0x22/0x2f [ 8896.874104] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8896.875971] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8896.878004] ? watchdog+0x40/0x40 [ 8896.879094] ? hrtimer_interrupt+0x128/0x2c0 [ 8896.880648] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8896.882461] ? apic_timer_interrupt+0xf/0x20 [ 8896.884164] [ 8896.885038] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8896.889920] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8896.897390] do_raw_spin_lock+0x177/0x190 [ 8896.901539] _raw_spin_lock+0x21/0x30 [ 8896.905586] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8896.912489] obd_mod_free+0x46/0x140 [ptlrpc] [ 8896.914954] mdc_commit_open+0x91/0xb0 [mdc] [ 8896.917512] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8896.920201] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8896.923331] after_reply+0xe2c/0x1780 [ptlrpc] [ 8896.925159] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8896.931150] ? __next_timer_interrupt+0x160/0x160 [ 8896.933487] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8896.936851] ? do_wait_intr+0xf0/0xf0 [ 8896.938560] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8896.943118] kthread+0x1d1/0x200 [ 8896.944561] ? set_kthread_struct+0x70/0x70 [ 8896.950137] ret_from_fork+0x1f/0x30 [ 8912.053277] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8912.059416] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8912.092053] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8912.097640] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8912.105527] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8912.108684] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8912.111451] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8912.123293] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8912.127036] RAX: 000000000000212a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8912.131145] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8912.135348] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8912.142127] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8912.149300] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8912.156925] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8912.170384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8912.181164] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8912.192297] Call Trace: [ 8912.193495] [ 8912.194791] ? show_regs.cold.9+0x22/0x2f [ 8912.201218] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8912.205330] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8912.216045] ? watchdog+0x40/0x40 [ 8912.219417] ? hrtimer_interrupt+0x128/0x2c0 [ 8912.220941] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8912.226567] ? apic_timer_interrupt+0xf/0x20 [ 8912.230713] [ 8912.231881] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8912.236763] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8912.241992] ? newidle_balance.isra.93+0x207/0x580 [ 8912.246327] do_raw_spin_lock+0x177/0x190 [ 8912.250392] _raw_spin_lock+0x21/0x30 [ 8912.253824] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8912.256849] process_one_work+0x2c8/0x700 [ 8912.263372] worker_thread+0x296/0x6e0 [ 8912.265206] ? rescuer_thread+0x570/0x570 [ 8912.274211] kthread+0x1d1/0x200 [ 8912.275956] ? set_kthread_struct+0x70/0x70 [ 8912.278253] ret_from_fork+0x1f/0x30 [ 8915.936122] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6278s! [ 8915.939224] Showing busy workqueues and worker pools: [ 8915.940929] workqueue events: flags=0x0 [ 8915.951255] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8915.956328] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8915.964564] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8915.968393] pending: cpu_latency_work [ptlrpc] [ 8915.971532] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8915.977160] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8915.982526] workqueue events_long: flags=0x0 [ 8915.984346] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8915.990207] pending: lu_objects_flush [obdclass] [ 8915.992948] workqueue events_unbound: flags=0x2 [ 8915.997120] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8916.000365] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8916.003185] workqueue events_power_efficient: flags=0x80 [ 8916.006153] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8916.009060] pending: check_lifetime, do_cache_clean [ 8916.012439] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8916.017070] pending: neigh_periodic_work, neigh_periodic_work [ 8916.019685] workqueue mm_percpu_wq: flags=0x8 [ 8916.021385] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8916.026211] pending: vmstat_update, lru_add_drain_per_cpu [ 8916.032504] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8916.038006] pending: vmstat_update, lru_add_drain_per_cpu [ 8916.040331] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8916.043346] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8916.049836] workqueue xprtiod: flags=0xa [ 8916.051727] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8916.054635] pending: xprt_autoclose, xs_error_handle [ 8916.057658] workqueue ipv6_addrconf: flags=0x40008 [ 8916.059961] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8916.063032] pending: addrconf_verify_work [ 8916.065032] workqueue knbd0-recv: flags=0x1a [ 8916.069046] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8916.072319] in-flight: 651:recv_work [ 8916.074141] workqueue ptlrpc_pinger: flags=0x2 [ 8916.075718] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8916.078341] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8916.081798] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6278s workers=4 idle: 15979 31824 8 [ 8916.087287] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8906s workers=2 idle: 654 [ 8924.055020] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 8924.057101] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 8924.058310] Modules linked in: [ 8924.061678] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr [ 8924.062575] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 8924.062584] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) [ 8924.062970] lustre(O) osp(O) ofd(O) lod(O) mdt(O) [ 8924.063001] ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8924.063048] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8924.063273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8924.063276] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8924.063290] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8924.063294] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8924.063299] RAX: 000000000000731e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8924.063302] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8924.063304] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8924.063306] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8924.063308] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8924.063312] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8924.063315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8924.063317] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8924.063325] Call Trace: [ 8924.063332] [ 8924.063338] ? show_regs.cold.9+0x22/0x2f [ 8924.064133] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8924.064145] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8924.064150] ? watchdog+0x40/0x40 [ 8924.064155] ? hrtimer_interrupt+0x128/0x2c0 [ 8924.064159] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8924.064168] ? apic_timer_interrupt+0xf/0x20 [ 8924.064173] [ 8924.064176] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8924.064182] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8924.064272] do_raw_spin_lock+0x177/0x190 [ 8924.064279] _raw_spin_lock+0x21/0x30 [ 8924.064285] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8924.064941] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8924.066687] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8924.066920] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8924.068823] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8924.071509] ? woken_wake_function+0x30/0x30 [ 8924.071525] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8924.073538] kthread+0x1d1/0x200 [ 8924.073548] ? set_kthread_struct+0x70/0x70 [ 8924.073553] ret_from_fork+0x1f/0x30 [ 8924.081997] intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8924.082048] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8924.082059] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8924.082063] RIP: 0010:native_safe_halt+0x2e/0x50 [ 8924.082082] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 8924.082088] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8924.082093] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8924.082096] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8924.082099] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8924.082102] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8924.082104] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8924.082109] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8924.082112] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8924.082115] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8924.082125] Call Trace: [ 8924.082134] [ 8924.082143] ? show_regs.cold.9+0x22/0x2f [ 8924.082162] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8924.082170] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8924.082175] ? watchdog+0x40/0x40 [ 8924.082180] ? hrtimer_interrupt+0x128/0x2c0 [ 8924.082185] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8924.082192] ? apic_timer_interrupt+0xf/0x20 [ 8924.082197] [ 8924.082200] ? native_safe_halt+0x2e/0x50 [ 8924.082204] kvm_wait+0xcd/0xe0 [ 8924.082213] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8924.082221] do_raw_spin_lock+0x177/0x190 [ 8924.082225] _raw_spin_lock+0x21/0x30 [ 8924.082231] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8924.083235] obd_mod_free+0x46/0x140 [ptlrpc] [ 8924.083465] mdc_commit_open+0x91/0xb0 [mdc] [ 8924.083515] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8924.083784] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8924.083980] after_reply+0xe2c/0x1780 [ptlrpc] [ 8924.084179] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8924.084393] ? __next_timer_interrupt+0x160/0x160 [ 8924.084407] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8924.084614] ? do_wait_intr+0xf0/0xf0 [ 8924.084623] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8924.085233] kthread+0x1d1/0x200 [ 8924.085240] ? set_kthread_struct+0x70/0x70 [ 8924.085244] ret_from_fork+0x1f/0x30 [ 8924.088576] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8924.088640] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8924.088647] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8924.088650] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8924.088732] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8924.088738] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8924.088743] RAX: 00000000000049a3 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8924.088745] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8924.088747] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8924.088749] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8924.088751] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8924.088754] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8924.088757] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8924.088759] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8924.088767] Call Trace: [ 8924.088772] [ 8924.088778] ? show_regs.cold.9+0x22/0x2f [ 8924.088787] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8924.088793] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8924.088797] ? watchdog+0x40/0x40 [ 8924.088801] ? hrtimer_interrupt+0x128/0x2c0 [ 8924.088805] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8924.088812] ? apic_timer_interrupt+0xf/0x20 [ 8924.088817] [ 8924.088820] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8924.088826] do_raw_spin_lock+0x177/0x190 [ 8924.088830] _raw_spin_lock+0x21/0x30 [ 8924.088835] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8924.089098] ? do_raw_spin_unlock+0x75/0x190 [ 8924.089104] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8924.451057] ? do_wait_intr+0xf0/0xf0 [ 8924.452544] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8924.454412] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8924.456171] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8924.458439] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8924.460525] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8924.462712] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8924.464466] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8924.466186] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8924.468580] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8924.471095] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8924.473140] lmv_intent_open+0x338/0x1150 [lmv] [ 8924.474771] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8924.476616] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8924.478641] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8924.481836] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8924.484031] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8924.486630] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8924.489044] ? do_raw_spin_unlock+0x75/0x190 [ 8924.490811] ? _raw_spin_unlock+0x12/0x30 [ 8924.492731] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8924.495189] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8924.497465] ll_atomic_open+0x36a/0xd70 [lustre] [ 8924.499487] lookup_open+0x356/0xb40 [ 8924.500627] do_last+0x313/0xfc0 [ 8924.501939] ? nd_jump_root+0xe5/0x160 [ 8924.503376] ? path_init+0x437/0x520 [ 8924.504794] path_openat+0xf7/0x500 [ 8924.506695] ? do_raw_spin_unlock+0x75/0x190 [ 8924.508355] do_filp_open+0x99/0x140 [ 8924.509690] ? getname_flags+0x6e/0x330 [ 8924.511195] ? __check_object_size+0xff/0x256 [ 8924.512722] ? do_raw_spin_unlock+0x75/0x190 [ 8924.514677] ? _raw_spin_unlock+0x12/0x30 [ 8924.516193] do_sys_openat2+0x2b4/0x410 [ 8924.517386] do_sys_open+0x73/0xa0 [ 8924.518401] __x64_sys_openat+0x24/0x30 [ 8924.519601] do_syscall_64+0xc1/0x440 [ 8924.520903] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8924.522886] RIP: 0033:0x7f53425a5332 [ 8924.524223] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8924.531845] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8924.535066] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8924.538118] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8924.541021] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8924.543858] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8924.547479] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8944.052009] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8944.057628] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8944.090573] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8944.098361] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8944.104687] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8944.108523] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8944.114344] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8944.127742] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8944.133544] RAX: 00000000000071bc RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8944.143365] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8944.151006] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8944.155014] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8944.160823] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8944.173982] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8944.178720] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8944.182470] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8944.187457] Call Trace: [ 8944.188372] [ 8944.190446] ? show_regs.cold.9+0x22/0x2f [ 8944.194558] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8944.201652] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8944.204655] ? watchdog+0x40/0x40 [ 8944.206524] ? hrtimer_interrupt+0x128/0x2c0 [ 8944.208114] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8944.215666] ? apic_timer_interrupt+0xf/0x20 [ 8944.217812] [ 8944.220000] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8944.229799] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8944.232771] ? newidle_balance.isra.93+0x207/0x580 [ 8944.241556] do_raw_spin_lock+0x177/0x190 [ 8944.243021] _raw_spin_lock+0x21/0x30 [ 8944.250771] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8944.256041] process_one_work+0x2c8/0x700 [ 8944.259378] worker_thread+0x296/0x6e0 [ 8944.269184] ? rescuer_thread+0x570/0x570 [ 8944.271985] kthread+0x1d1/0x200 [ 8944.273603] ? set_kthread_struct+0x70/0x70 [ 8944.275694] ret_from_fork+0x1f/0x30 [ 8946.144144] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6309s! [ 8946.159480] Showing busy workqueues and worker pools: [ 8946.166498] workqueue events: flags=0x0 [ 8946.171630] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8946.185585] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8946.197823] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8946.204176] pending: cpu_latency_work [ptlrpc] [ 8946.208592] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8946.217150] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8946.238341] workqueue events_long: flags=0x0 [ 8946.244621] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8946.254384] pending: lu_objects_flush [obdclass] [ 8946.264576] workqueue events_unbound: flags=0x2 [ 8946.269657] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8946.278110] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8946.286347] workqueue events_power_efficient: flags=0x80 [ 8946.293377] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8946.299043] pending: check_lifetime, do_cache_clean [ 8946.305702] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8946.317080] pending: neigh_periodic_work, neigh_periodic_work [ 8946.326344] workqueue mm_percpu_wq: flags=0x8 [ 8946.331621] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8946.340548] pending: vmstat_update, lru_add_drain_per_cpu [ 8946.345367] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8946.353298] pending: vmstat_update, lru_add_drain_per_cpu [ 8946.363808] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8946.372515] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8946.383558] workqueue xprtiod: flags=0xa [ 8946.385584] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8946.389726] pending: xprt_autoclose, xs_error_handle [ 8946.394565] workqueue ipv6_addrconf: flags=0x40008 [ 8946.402971] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8946.411145] pending: addrconf_verify_work [ 8946.417844] workqueue knbd0-recv: flags=0x1a [ 8946.429690] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8946.438774] in-flight: 651:recv_work [ 8946.441615] workqueue ptlrpc_pinger: flags=0x2 [ 8946.449652] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8946.460720] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8946.467628] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6309s workers=4 idle: 15979 31824 8 [ 8946.475425] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8936s workers=2 idle: 654 [ 8952.055021] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 8952.058273] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8952.064437] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 8952.065298] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 8952.068282] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8952.068368] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8952.068376] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8952.068379] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8952.068394] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8952.068398] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8952.068402] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8952.068404] RDX: 0000000000004254 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8952.068407] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8952.068409] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8952.068411] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8952.068414] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8952.068418] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8952.068420] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8952.068429] Call Trace: [ 8952.068435] [ 8952.068441] ? show_regs.cold.9+0x22/0x2f [ 8952.068449] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8952.068454] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8952.068458] ? watchdog+0x40/0x40 [ 8952.068462] ? hrtimer_interrupt+0x128/0x2c0 [ 8952.068465] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8952.068475] ? apic_timer_interrupt+0xf/0x20 [ 8952.068480] [ 8952.068483] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8952.068488] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8952.068492] do_raw_spin_lock+0x177/0x190 [ 8952.068496] _raw_spin_lock+0x21/0x30 [ 8952.068502] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8952.068789] obd_mod_free+0x46/0x140 [ptlrpc] [ 8952.068978] mdc_commit_open+0x91/0xb0 [mdc] [ 8952.069013] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8952.069218] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8952.069414] after_reply+0xe2c/0x1780 [ptlrpc] [ 8952.069614] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8952.069854] ? __next_timer_interrupt+0x160/0x160 [ 8952.069864] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8952.070070] ? do_wait_intr+0xf0/0xf0 [ 8952.070079] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8952.070283] kthread+0x1d1/0x200 [ 8952.070289] ? set_kthread_struct+0x70/0x70 [ 8952.070293] ret_from_fork+0x1f/0x30 [ 8952.077716] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8952.077731] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8952.077734] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8952.077748] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8952.077752] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8952.077758] RAX: 0000000000004583 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8952.077760] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8952.077777] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8952.077780] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8952.077782] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8952.077786] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8952.077789] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8952.077791] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8952.077801] Call Trace: [ 8952.077808] [ 8952.077816] ? show_regs.cold.9+0x22/0x2f [ 8952.077828] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8952.077835] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8952.077840] ? watchdog+0x40/0x40 [ 8952.077844] ? hrtimer_interrupt+0x128/0x2c0 [ 8952.077848] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8952.077858] ? apic_timer_interrupt+0xf/0x20 [ 8952.077863] [ 8952.077866] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8952.077874] do_raw_spin_lock+0x177/0x190 [ 8952.077879] _raw_spin_lock+0x21/0x30 [ 8952.077883] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8952.078206] ? do_raw_spin_unlock+0x75/0x190 [ 8952.078211] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8952.078403] ? do_wait_intr+0xf0/0xf0 [ 8952.078413] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8952.078841] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8952.079049] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8952.079262] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8952.079384] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8952.079422] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8952.079449] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8952.079648] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8952.079745] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8952.080053] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8952.080087] lmv_intent_open+0x338/0x1150 [lmv] [ 8952.080123] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8952.080207] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8952.080287] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8952.080310] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8952.080392] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8952.090824] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8952.090901] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8952.090911] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8952.090914] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8952.090930] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8952.090934] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8952.090939] RAX: 0000000000004929 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8952.090941] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8952.090943] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8952.090946] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8952.090948] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8952.090951] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8952.090954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8952.090957] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8952.090965] Call Trace: [ 8952.090971] [ 8952.090976] ? show_regs.cold.9+0x22/0x2f [ 8952.090985] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8952.090992] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8952.090997] ? watchdog+0x40/0x40 [ 8952.091001] ? hrtimer_interrupt+0x128/0x2c0 [ 8952.091005] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8952.091014] ? apic_timer_interrupt+0xf/0x20 [ 8952.091019] [ 8952.091023] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8952.091028] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8952.091034] do_raw_spin_lock+0x177/0x190 [ 8952.091039] _raw_spin_lock+0x21/0x30 [ 8952.091043] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8952.091300] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8952.095211] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8952.095376] ? do_raw_spin_unlock+0x75/0x190 [ 8952.095387] ? _raw_spin_unlock+0x12/0x30 [ 8952.095398] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8952.095437] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8952.095458] ll_atomic_open+0x36a/0xd70 [lustre] [ 8952.095947] lookup_open+0x356/0xb40 [ 8952.095958] do_last+0x313/0xfc0 [ 8952.095964] ? nd_jump_root+0xe5/0x160 [ 8952.095968] ? path_init+0x437/0x520 [ 8952.095973] path_openat+0xf7/0x500 [ 8952.095979] ? do_raw_spin_unlock+0x75/0x190 [ 8952.095985] do_filp_open+0x99/0x140 [ 8952.095992] ? getname_flags+0x6e/0x330 [ 8952.095997] ? __check_object_size+0xff/0x256 [ 8952.096004] ? do_raw_spin_unlock+0x75/0x190 [ 8952.096008] ? _raw_spin_unlock+0x12/0x30 [ 8952.096013] do_sys_openat2+0x2b4/0x410 [ 8952.096018] do_sys_open+0x73/0xa0 [ 8952.096023] __x64_sys_openat+0x24/0x30 [ 8952.096027] do_syscall_64+0xc1/0x440 [ 8952.096034] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8952.096041] RIP: 0033:0x7f53425a5332 [ 8952.096051] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8952.096055] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8952.096062] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8952.096065] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8952.096068] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8952.096070] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8952.096073] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8952.506608] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8952.509319] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8952.512307] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8952.515037] ? woken_wake_function+0x30/0x30 [ 8952.516818] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8952.519679] kthread+0x1d1/0x200 [ 8952.521217] ? set_kthread_struct+0x70/0x70 [ 8952.523141] ret_from_fork+0x1f/0x30 [ 8972.054683] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 8972.064423] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8972.107929] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8972.128721] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8972.134619] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 8972.140930] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8972.147337] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8972.169687] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8972.178740] RAX: 0000000000003510 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8972.185215] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 8972.189139] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 8972.196193] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 8972.201840] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 8972.205995] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 8972.212387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8972.223774] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 8972.226178] Call Trace: [ 8972.233630] [ 8972.239273] ? show_regs.cold.9+0x22/0x2f [ 8972.246891] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8972.253116] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8972.255308] ? watchdog+0x40/0x40 [ 8972.256575] ? hrtimer_interrupt+0x128/0x2c0 [ 8972.259713] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8972.260919] ? apic_timer_interrupt+0xf/0x20 [ 8972.262328] [ 8972.263569] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8972.268784] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8972.271660] ? newidle_balance.isra.93+0x207/0x580 [ 8972.275289] do_raw_spin_lock+0x177/0x190 [ 8972.278307] _raw_spin_lock+0x21/0x30 [ 8972.280257] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 8972.284943] process_one_work+0x2c8/0x700 [ 8972.285978] worker_thread+0x296/0x6e0 [ 8972.287012] ? rescuer_thread+0x570/0x570 [ 8972.288379] kthread+0x1d1/0x200 [ 8972.291853] ? set_kthread_struct+0x70/0x70 [ 8972.295429] ret_from_fork+0x1f/0x30 [ 8976.871980] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6339s! [ 8976.892243] Showing busy workqueues and worker pools: [ 8976.906510] workqueue events: flags=0x0 [ 8976.916318] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 8976.930722] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 8976.957071] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8976.971841] pending: cpu_latency_work [ptlrpc] [ 8976.981219] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 8976.984062] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 8976.990016] workqueue events_long: flags=0x0 [ 8976.991774] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 8976.995776] pending: lu_objects_flush [obdclass] [ 8976.999065] workqueue events_unbound: flags=0x2 [ 8977.001523] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 8977.005902] pending: flush_memcg_stats_dwork, idle_cull_fn [ 8977.009259] workqueue events_power_efficient: flags=0x80 [ 8977.011319] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8977.014047] pending: check_lifetime, do_cache_clean [ 8977.016834] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8977.020373] pending: neigh_periodic_work, neigh_periodic_work [ 8977.022558] workqueue mm_percpu_wq: flags=0x8 [ 8977.023771] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8977.026575] pending: vmstat_update, lru_add_drain_per_cpu [ 8977.028486] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 8977.032523] pending: vmstat_update, lru_add_drain_per_cpu [ 8977.037902] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 8977.040882] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 8977.043988] workqueue xprtiod: flags=0xa [ 8977.045883] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 8977.048518] pending: xprt_autoclose, xs_error_handle [ 8977.050300] workqueue ipv6_addrconf: flags=0x40008 [ 8977.052178] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 8977.059566] pending: addrconf_verify_work [ 8977.060715] workqueue knbd0-recv: flags=0x1a [ 8977.061848] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 8977.065828] in-flight: 651:recv_work [ 8977.067247] workqueue ptlrpc_pinger: flags=0x2 [ 8977.070058] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 8977.073895] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 8977.076505] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6339s workers=4 idle: 15979 31824 8 [ 8977.078755] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8967s workers=2 idle: 654 [ 8980.055283] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 8980.058439] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8980.063756] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 8980.063869] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 8980.063877] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8980.063953] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8980.063960] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8980.063963] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8980.063979] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 8980.063984] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8980.063989] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 8980.063991] RDX: 000000000000144e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 8980.063994] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 8980.063997] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 8980.064000] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 8980.064005] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 8980.064008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8980.064011] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 8980.064019] Call Trace: [ 8980.064026] [ 8980.064033] ? show_regs.cold.9+0x22/0x2f [ 8980.064044] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8980.064052] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8980.064056] ? watchdog+0x40/0x40 [ 8980.064060] ? hrtimer_interrupt+0x128/0x2c0 [ 8980.064064] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8980.064073] ? apic_timer_interrupt+0xf/0x20 [ 8980.064077] [ 8980.064081] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8980.064085] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8980.064090] do_raw_spin_lock+0x177/0x190 [ 8980.064095] _raw_spin_lock+0x21/0x30 [ 8980.064101] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8980.071868] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8980.071876] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8980.071879] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8980.071891] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8980.071895] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 8980.071900] RAX: 0000000000001c8a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8980.071903] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8980.071905] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8980.071907] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8980.071909] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8980.071912] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8980.071915] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8980.071917] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8980.071926] Call Trace: [ 8980.071930] [ 8980.071933] ? show_regs.cold.9+0x22/0x2f [ 8980.071942] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8980.071949] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8980.071953] ? watchdog+0x40/0x40 [ 8980.071957] ? hrtimer_interrupt+0x128/0x2c0 [ 8980.071961] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8980.071969] ? apic_timer_interrupt+0xf/0x20 [ 8980.071973] [ 8980.071976] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8980.071982] do_raw_spin_lock+0x177/0x190 [ 8980.071986] _raw_spin_lock+0x21/0x30 [ 8980.071884] obd_mod_free+0x46/0x140 [ptlrpc] [ 8980.071991] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8980.072322] mdc_commit_open+0x91/0xb0 [mdc] [ 8980.072506] ? do_raw_spin_unlock+0x75/0x190 [ 8980.072412] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8980.072810] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8980.072512] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8980.073380] ? do_wait_intr+0xf0/0xf0 [ 8980.073393] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8980.073204] after_reply+0xe2c/0x1780 [ptlrpc] [ 8980.073905] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8980.073923] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8980.074506] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8980.074711] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8980.074842] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8980.074884] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8980.074912] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8980.074938] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8980.075022] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8980.075240] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8980.075409] lmv_intent_open+0x338/0x1150 [lmv] [ 8980.075448] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8980.075538] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8980.075619] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8980.075642] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8980.075728] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8980.075814] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8980.075900] ? do_raw_spin_unlock+0x75/0x190 [ 8980.075908] ? _raw_spin_unlock+0x12/0x30 [ 8980.075915] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8980.075943] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8980.075962] ll_atomic_open+0x36a/0xd70 [lustre] [ 8980.076046] lookup_open+0x356/0xb40 [ 8980.076057] do_last+0x313/0xfc0 [ 8980.076061] ? nd_jump_root+0xe5/0x160 [ 8980.076066] ? path_init+0x437/0x520 [ 8980.076070] path_openat+0xf7/0x500 [ 8980.076074] ? do_raw_spin_unlock+0x75/0x190 [ 8980.076079] do_filp_open+0x99/0x140 [ 8980.076085] ? getname_flags+0x6e/0x330 [ 8980.076089] ? __check_object_size+0xff/0x256 [ 8980.076097] ? do_raw_spin_unlock+0x75/0x190 [ 8980.076101] ? _raw_spin_unlock+0x12/0x30 [ 8980.076105] do_sys_openat2+0x2b4/0x410 [ 8980.076109] do_sys_open+0x73/0xa0 [ 8980.076113] __x64_sys_openat+0x24/0x30 [ 8980.076116] do_syscall_64+0xc1/0x440 [ 8980.076122] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8980.076128] RIP: 0033:0x7f53425a5332 [ 8980.076136] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8980.076140] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8980.076145] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8980.076147] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8980.076149] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8980.076151] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8980.076154] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8980.076797] ? __next_timer_interrupt+0x160/0x160 [ 8980.076916] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8980.089332] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 8980.089442] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8980.089452] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8980.089455] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8980.089473] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8980.089479] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 8980.089483] RAX: 000000000000728e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8980.089487] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 8980.089490] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8980.089492] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 8980.089495] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 8980.089499] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 8980.089502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8980.089505] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 8980.089513] Call Trace: [ 8980.089520] [ 8980.089527] ? show_regs.cold.9+0x22/0x2f [ 8980.089537] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 8980.089546] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8980.089551] ? watchdog+0x40/0x40 [ 8980.089555] ? hrtimer_interrupt+0x128/0x2c0 [ 8980.089560] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8980.089570] ? apic_timer_interrupt+0xf/0x20 [ 8980.089575] [ 8980.089578] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8980.089582] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 8980.089589] do_raw_spin_lock+0x177/0x190 [ 8980.089594] _raw_spin_lock+0x21/0x30 [ 8980.089600] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 8980.090029] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 8980.090449] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 8980.090781] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 8980.091236] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 8980.099109] ? do_wait_intr+0xf0/0xf0 [ 8980.099146] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8980.099689] kthread+0x1d1/0x200 [ 8980.099696] ? set_kthread_struct+0x70/0x70 [ 8980.099701] ret_from_fork+0x1f/0x30 [ 8981.245463] ? woken_wake_function+0x30/0x30 [ 8981.247378] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 8981.249832] kthread+0x1d1/0x200 [ 8981.251376] ? set_kthread_struct+0x70/0x70 [ 8981.252691] ret_from_fork+0x1f/0x30 [ 8994.601449] rcu: INFO: rcu_sched self-detected stall on CPU [ 8994.604219] rcu: 2-....: (3754309 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 8994.609942] (t=6900175 jiffies g=274185 q=65626) [ 8994.613189] rcu: rcu_sched kthread starved for 6358541 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 8994.623920] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 8994.631269] rcu: RCU grace-period kthread stack dump: [ 8994.640568] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 8994.653321] Call Trace: [ 8994.654238] __schedule+0x351/0xcb0 [ 8994.658325] schedule+0xc0/0x180 [ 8994.660830] schedule_timeout+0xb4/0x190 [ 8994.663191] ? do_raw_spin_unlock+0x75/0x190 [ 8994.665314] ? __next_timer_interrupt+0x160/0x160 [ 8994.668711] ? prepare_to_swait_event+0xf3/0x140 [ 8994.672131] rcu_gp_kthread+0x8dd/0x15b0 [ 8994.673557] ? force_qs_rnp+0x2c0/0x2c0 [ 8994.676037] kthread+0x1d1/0x200 [ 8994.677138] ? set_kthread_struct+0x70/0x70 [ 8994.680951] ret_from_fork+0x1f/0x30 [ 8994.683527] rcu: Stack dump where RCU GP kthread last ran: [ 8994.688568] Sending NMI from CPU 2 to CPUs 1: [ 8994.695656] NMI backtrace for cpu 1 [ 8994.695670] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8994.695673] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8994.695674] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8994.695676] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8994.695678] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [ 8994.695681] RAX: 0000000000007e88 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8994.695683] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8994.695684] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8994.695686] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8994.695687] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8994.695688] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8994.695689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8994.695690] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8994.695691] Call Trace: [ 8994.695692] [ 8994.695693] ? show_regs.cold.9+0x22/0x2f [ 8994.695694] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8994.695695] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8994.695696] ? nmi_handle+0xbd/0x210 [ 8994.695697] ? default_do_nmi+0x4b/0x1b0 [ 8994.695698] ? do_nmi+0x280/0x380 [ 8994.695699] ? end_repeat_nmi+0x16/0x1f [ 8994.695700] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8994.695701] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8994.695703] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8994.695704] [ 8994.695705] do_raw_spin_lock+0x177/0x190 [ 8994.695706] _raw_spin_lock+0x21/0x30 [ 8994.695707] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8994.695708] ? do_raw_spin_unlock+0x75/0x190 [ 8994.695709] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8994.695710] ? do_wait_intr+0xf0/0xf0 [ 8994.695711] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8994.695712] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8994.695714] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8994.695715] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8994.695716] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8994.695718] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8994.695719] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8994.695720] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8994.695721] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8994.695722] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8994.695723] lmv_intent_open+0x338/0x1150 [lmv] [ 8994.695724] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8994.695725] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8994.695726] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8994.695727] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8994.695728] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8994.695729] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8994.695730] ? do_raw_spin_unlock+0x75/0x190 [ 8994.695731] ? _raw_spin_unlock+0x12/0x30 [ 8994.695732] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8994.695733] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8994.695734] ll_atomic_open+0x36a/0xd70 [lustre] [ 8994.695735] lookup_open+0x356/0xb40 [ 8994.695736] do_last+0x313/0xfc0 [ 8994.695737] ? nd_jump_root+0xe5/0x160 [ 8994.695738] ? path_init+0x437/0x520 [ 8994.695739] path_openat+0xf7/0x500 [ 8994.695740] ? do_raw_spin_unlock+0x75/0x190 [ 8994.695741] do_filp_open+0x99/0x140 [ 8994.695742] ? getname_flags+0x6e/0x330 [ 8994.695743] ? __check_object_size+0xff/0x256 [ 8994.695744] ? do_raw_spin_unlock+0x75/0x190 [ 8994.695745] ? _raw_spin_unlock+0x12/0x30 [ 8994.695746] do_sys_openat2+0x2b4/0x410 [ 8994.695747] do_sys_open+0x73/0xa0 [ 8994.695748] __x64_sys_openat+0x24/0x30 [ 8994.695749] do_syscall_64+0xc1/0x440 [ 8994.695750] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8994.695751] RIP: 0033:0x7f53425a5332 [ 8994.695752] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8994.695754] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8994.695757] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8994.695759] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8994.695760] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8994.695762] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8994.695763] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8994.697464] Sending NMI from CPU 2 to CPUs 1: [ 8995.056972] NMI backtrace for cpu 1 [ 8995.056979] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8995.056981] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8995.056982] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8995.056983] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 8995.056985] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [ 8995.056987] RAX: 0000000000007811 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 8995.056988] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 8995.056989] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 8995.056990] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 8995.056991] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 8995.056992] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 8995.056992] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8995.056993] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 8995.056994] Call Trace: [ 8995.056994] [ 8995.056995] ? show_regs.cold.9+0x22/0x2f [ 8995.056996] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 8995.056996] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 8995.056997] ? nmi_handle+0xbd/0x210 [ 8995.056997] ? default_do_nmi+0x4b/0x1b0 [ 8995.056998] ? do_nmi+0x280/0x380 [ 8995.056998] ? end_repeat_nmi+0x16/0x1f [ 8995.056999] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8995.057000] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8995.057000] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 8995.057001] [ 8995.057001] do_raw_spin_lock+0x177/0x190 [ 8995.057002] _raw_spin_lock+0x21/0x30 [ 8995.057002] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 8995.057003] ? do_raw_spin_unlock+0x75/0x190 [ 8995.057004] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 8995.057004] ? do_wait_intr+0xf0/0xf0 [ 8995.057005] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 8995.057005] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 8995.057006] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8995.057007] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8995.057007] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8995.057008] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 8995.057008] mdc_intent_lock+0x353/0x7e0 [mdc] [ 8995.057009] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8995.057010] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 8995.057010] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 8995.057026] lmv_intent_open+0x338/0x1150 [lmv] [ 8995.057028] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8995.057029] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8995.057031] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 8995.057032] ll_intent_lock+0xc5/0xaa0 [lustre] [ 8995.057033] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 8995.057034] ll_lookup_it+0xe0e/0x2270 [lustre] [ 8995.057036] ? do_raw_spin_unlock+0x75/0x190 [ 8995.057037] ? _raw_spin_unlock+0x12/0x30 [ 8995.057038] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 8995.057039] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 8995.057040] ll_atomic_open+0x36a/0xd70 [lustre] [ 8995.057041] lookup_open+0x356/0xb40 [ 8995.057041] do_last+0x313/0xfc0 [ 8995.057046] ? nd_jump_root+0xe5/0x160 [ 8995.057047] ? path_init+0x437/0x520 [ 8995.057048] path_openat+0xf7/0x500 [ 8995.057049] ? do_raw_spin_unlock+0x75/0x190 [ 8995.057050] do_filp_open+0x99/0x140 [ 8995.057051] ? getname_flags+0x6e/0x330 [ 8995.057052] ? __check_object_size+0xff/0x256 [ 8995.057053] ? do_raw_spin_unlock+0x75/0x190 [ 8995.057054] ? _raw_spin_unlock+0x12/0x30 [ 8995.057055] do_sys_openat2+0x2b4/0x410 [ 8995.057056] do_sys_open+0x73/0xa0 [ 8995.057056] __x64_sys_openat+0x24/0x30 [ 8995.057057] do_syscall_64+0xc1/0x440 [ 8995.057058] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 8995.057059] RIP: 0033:0x7f53425a5332 [ 8995.057060] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 8995.057062] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 8995.057065] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 8995.057066] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 8995.057068] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 8995.057069] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 8995.057070] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 8995.057884] NMI backtrace for cpu 2 [ 8995.345918] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 8995.358399] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 8995.363736] Call Trace: [ 8995.364427] [ 8995.365690] ? dump_stack+0xbb/0x10e [ 8995.367299] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 8995.373361] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 8995.374984] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 8995.379831] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 8995.382362] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 8995.387019] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 8995.391390] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 8995.395415] ? update_process_times+0x6d/0xc0 [ 8995.400436] ? tick_sched_handle.isra.18+0x27/0x90 [ 8995.403814] ? tick_sched_timer+0x53/0xb0 [ 8995.406673] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 8995.414691] ? tick_sched_handle.isra.18+0x90/0x90 [ 8995.419552] ? hrtimer_interrupt+0x128/0x2c0 [ 8995.421185] ? smp_apic_timer_interrupt+0xa6/0x270 [ 8995.422991] ? apic_timer_interrupt+0xf/0x20 [ 8995.424354] [ 8995.425193] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 8995.428004] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 8995.432092] ? do_raw_spin_lock+0x177/0x190 [ 8995.435255] ? _raw_spin_lock+0x21/0x30 [ 8995.436777] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 8995.439777] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 8995.445190] ? mdc_commit_open+0x91/0xb0 [mdc] [ 8995.450509] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 8995.455312] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 8995.460355] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 8995.463615] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 8995.473873] ? __next_timer_interrupt+0x160/0x160 [ 8995.481526] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 8995.489461] ? do_wait_intr+0xf0/0xf0 [ 8995.491942] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 8995.501315] ? kthread+0x1d1/0x200 [ 8995.502360] ? set_kthread_struct+0x70/0x70 [ 8995.508557] ? ret_from_fork+0x1f/0x30 [ 9004.055139] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 9004.061205] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9004.085121] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9004.090646] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9004.095505] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9004.100389] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9004.103403] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9004.112338] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9004.117390] RAX: 0000000000006fee RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9004.120752] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9004.124634] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9004.127371] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9004.130453] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9004.134062] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9004.138076] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9004.140553] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9004.143797] Call Trace: [ 9004.145143] [ 9004.146200] ? show_regs.cold.9+0x22/0x2f [ 9004.148166] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9004.151034] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9004.153225] ? watchdog+0x40/0x40 [ 9004.155044] ? hrtimer_interrupt+0x128/0x2c0 [ 9004.157180] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9004.159718] ? apic_timer_interrupt+0xf/0x20 [ 9004.161838] [ 9004.164026] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9004.166570] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9004.169764] ? newidle_balance.isra.93+0x207/0x580 [ 9004.172211] do_raw_spin_lock+0x177/0x190 [ 9004.174376] _raw_spin_lock+0x21/0x30 [ 9004.176133] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9004.178645] process_one_work+0x2c8/0x700 [ 9004.180678] worker_thread+0x296/0x6e0 [ 9004.182564] ? rescuer_thread+0x570/0x570 [ 9004.184563] kthread+0x1d1/0x200 [ 9004.186206] ? set_kthread_struct+0x70/0x70 [ 9004.188352] ret_from_fork+0x1f/0x30 [ 9007.588418] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6370s! [ 9007.599450] Showing busy workqueues and worker pools: [ 9007.602100] workqueue events: flags=0x0 [ 9007.607034] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9007.609389] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9007.613820] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9007.617103] pending: cpu_latency_work [ptlrpc] [ 9007.619831] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9007.622893] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9007.627051] workqueue events_long: flags=0x0 [ 9007.629501] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9007.632495] pending: lu_objects_flush [obdclass] [ 9007.635357] workqueue events_unbound: flags=0x2 [ 9007.637369] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9007.640811] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9007.643331] workqueue events_power_efficient: flags=0x80 [ 9007.646176] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9007.649440] pending: check_lifetime, do_cache_clean [ 9007.651727] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9007.654986] pending: neigh_periodic_work, neigh_periodic_work [ 9007.657906] workqueue mm_percpu_wq: flags=0x8 [ 9007.660299] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9007.663574] pending: vmstat_update, lru_add_drain_per_cpu [ 9007.666105] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9007.669393] pending: vmstat_update, lru_add_drain_per_cpu [ 9007.672166] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9007.674871] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9007.677921] workqueue xprtiod: flags=0xa [ 9007.679653] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9007.683243] pending: xprt_autoclose, xs_error_handle [ 9007.686099] workqueue ipv6_addrconf: flags=0x40008 [ 9007.688553] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9007.692459] pending: addrconf_verify_work [ 9007.695053] workqueue knbd0-recv: flags=0x1a [ 9007.697253] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9007.700226] in-flight: 651:recv_work [ 9007.704613] workqueue ptlrpc_pinger: flags=0x2 [ 9007.707278] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9007.710425] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9007.713361] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6370s workers=4 idle: 15979 31824 8 [ 9007.716684] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=8998s workers=2 idle: 654 [ 9008.059144] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 9008.064550] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9008.104697] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9008.111396] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9008.119568] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9008.126210] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9008.142738] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9008.153256] RAX: 00000000000032cd RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9008.157558] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9008.165574] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9008.172971] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9008.178483] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9008.183164] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9008.185490] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9008.189628] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9008.194734] Call Trace: [ 9008.195246] [ 9008.195665] ? show_regs.cold.9+0x22/0x2f [ 9008.203708] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9008.206737] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9008.210410] ? watchdog+0x40/0x40 [ 9008.213159] ? hrtimer_interrupt+0x128/0x2c0 [ 9008.215747] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9008.216766] ? apic_timer_interrupt+0xf/0x20 [ 9008.219081] [ 9008.219443] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9008.224214] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9008.228197] do_raw_spin_lock+0x177/0x190 [ 9008.233357] _raw_spin_lock+0x21/0x30 [ 9008.234531] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9008.238389] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9008.242238] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9008.245824] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9008.251083] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9008.254545] ? woken_wake_function+0x30/0x30 [ 9008.265508] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9008.275369] kthread+0x1d1/0x200 [ 9008.279584] ? set_kthread_struct+0x70/0x70 [ 9008.282139] ret_from_fork+0x1f/0x30 [ 9020.055013] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9020.057817] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9020.058588] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9020.058663] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9020.058671] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9020.058673] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9020.058688] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9020.058693] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9020.058698] RAX: 0000000000000b8a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9020.058892] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9020.058895] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9020.058897] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9020.058900] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9020.058903] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9020.058905] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9020.058912] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9020.058921] Call Trace: [ 9020.058928] [ 9020.058934] ? show_regs.cold.9+0x22/0x2f [ 9020.058951] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9020.058958] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9020.058964] ? watchdog+0x40/0x40 [ 9020.058968] ? hrtimer_interrupt+0x128/0x2c0 [ 9020.058973] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9020.058982] ? apic_timer_interrupt+0xf/0x20 [ 9020.058986] [ 9020.058990] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9020.058997] do_raw_spin_lock+0x177/0x190 [ 9020.059008] _raw_spin_lock+0x21/0x30 [ 9020.059014] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9020.059497] ? do_raw_spin_unlock+0x75/0x190 [ 9020.059504] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9020.059830] ? do_wait_intr+0xf0/0xf0 [ 9020.059839] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9020.060109] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9020.060411] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9020.060696] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9020.060814] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9020.060851] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9020.060882] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9020.060908] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9020.060992] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9020.061386] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9020.061422] lmv_intent_open+0x338/0x1150 [lmv] [ 9020.061456] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9020.061539] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9020.064966] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9020.065135] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9020.065143] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9020.065146] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9020.065161] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9020.065165] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9020.065170] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9020.065172] RDX: 0000000000001c56 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9020.065175] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9020.065177] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9020.065179] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9020.065182] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9020.065186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9020.065188] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9020.065261] Call Trace: [ 9020.065269] [ 9020.065274] ? show_regs.cold.9+0x22/0x2f [ 9020.065283] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9020.065297] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9020.065301] ? watchdog+0x40/0x40 [ 9020.065305] ? hrtimer_interrupt+0x128/0x2c0 [ 9020.065308] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9020.065317] ? apic_timer_interrupt+0xf/0x20 [ 9020.065326] [ 9020.065329] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9020.065333] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9020.065338] do_raw_spin_lock+0x177/0x190 [ 9020.065342] _raw_spin_lock+0x21/0x30 [ 9020.065348] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9020.065749] obd_mod_free+0x46/0x140 [ptlrpc] [ 9020.065973] mdc_commit_open+0x91/0xb0 [mdc] [ 9020.066019] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9020.066407] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9020.085532] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9020.085642] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9020.085758] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9020.085962] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9020.086108] ? do_raw_spin_unlock+0x75/0x190 [ 9020.086117] ? _raw_spin_unlock+0x12/0x30 [ 9020.086125] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9020.086154] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9020.086176] ll_atomic_open+0x36a/0xd70 [lustre] [ 9020.086324] lookup_open+0x356/0xb40 [ 9020.086335] do_last+0x313/0xfc0 [ 9020.086340] ? nd_jump_root+0xe5/0x160 [ 9020.086345] ? path_init+0x437/0x520 [ 9020.086350] path_openat+0xf7/0x500 [ 9020.086356] ? do_raw_spin_unlock+0x75/0x190 [ 9020.086366] do_filp_open+0x99/0x140 [ 9020.086374] ? getname_flags+0x6e/0x330 [ 9020.086379] ? __check_object_size+0xff/0x256 [ 9020.086386] ? do_raw_spin_unlock+0x75/0x190 [ 9020.086390] ? _raw_spin_unlock+0x12/0x30 [ 9020.086395] do_sys_openat2+0x2b4/0x410 [ 9020.086400] do_sys_open+0x73/0xa0 [ 9020.086404] __x64_sys_openat+0x24/0x30 [ 9020.086408] do_syscall_64+0xc1/0x440 [ 9020.086414] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9020.086422] RIP: 0033:0x7f53425a5332 [ 9020.086430] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9020.086435] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9020.086441] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9020.086443] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9020.086446] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9020.086448] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9020.086450] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9020.421358] after_reply+0xe2c/0x1780 [ptlrpc] [ 9020.424177] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9020.426506] ? __next_timer_interrupt+0x160/0x160 [ 9020.435579] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9020.438404] ? do_wait_intr+0xf0/0xf0 [ 9020.443952] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9020.450323] kthread+0x1d1/0x200 [ 9020.452137] ? set_kthread_struct+0x70/0x70 [ 9020.458328] ret_from_fork+0x1f/0x30 [ 9032.058877] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 9032.067351] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9032.111095] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9032.119998] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9032.128613] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9032.136859] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9032.140911] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9032.157017] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9032.167053] RAX: 0000000000005087 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9032.179005] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9032.187945] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9032.192079] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9032.195625] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9032.203235] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9032.211253] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9032.216722] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9032.222270] Call Trace: [ 9032.223268] [ 9032.224749] ? show_regs.cold.9+0x22/0x2f [ 9032.227095] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9032.230069] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9032.237667] ? watchdog+0x40/0x40 [ 9032.239911] ? hrtimer_interrupt+0x128/0x2c0 [ 9032.242005] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9032.244556] ? apic_timer_interrupt+0xf/0x20 [ 9032.246605] [ 9032.247514] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9032.249895] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9032.255629] ? newidle_balance.isra.93+0x207/0x580 [ 9032.259544] do_raw_spin_lock+0x177/0x190 [ 9032.261339] _raw_spin_lock+0x21/0x30 [ 9032.267942] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9032.273261] process_one_work+0x2c8/0x700 [ 9032.278072] worker_thread+0x296/0x6e0 [ 9032.280305] ? rescuer_thread+0x570/0x570 [ 9032.284341] kthread+0x1d1/0x200 [ 9032.286230] ? set_kthread_struct+0x70/0x70 [ 9032.291556] ret_from_fork+0x1f/0x30 [ 9036.058076] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 9036.061002] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9036.079129] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9036.083121] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9036.086532] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9036.088691] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9036.095972] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9036.098970] RAX: 0000000000005950 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9036.101595] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9036.104146] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9036.106825] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9036.109621] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9036.112078] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9036.115182] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9036.117586] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9036.119965] Call Trace: [ 9036.121011] [ 9036.121862] ? show_regs.cold.9+0x22/0x2f [ 9036.123220] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9036.124973] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9036.126720] ? watchdog+0x40/0x40 [ 9036.127929] ? hrtimer_interrupt+0x128/0x2c0 [ 9036.129689] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9036.131287] ? apic_timer_interrupt+0xf/0x20 [ 9036.133093] [ 9036.133812] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9036.135809] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9036.137894] do_raw_spin_lock+0x177/0x190 [ 9036.139282] _raw_spin_lock+0x21/0x30 [ 9036.140761] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9036.143580] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9036.145881] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9036.148806] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9036.151104] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9036.153873] ? woken_wake_function+0x30/0x30 [ 9036.155590] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9036.158340] kthread+0x1d1/0x200 [ 9036.159611] ? set_kthread_struct+0x70/0x70 [ 9036.161269] ret_from_fork+0x1f/0x30 [ 9037.792781] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6400s! [ 9037.798183] Showing busy workqueues and worker pools: [ 9037.800907] workqueue events: flags=0x0 [ 9037.803049] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9037.816687] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9037.825008] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9037.833930] pending: cpu_latency_work [ptlrpc] [ 9037.843169] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9037.860516] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9037.869092] workqueue events_long: flags=0x0 [ 9037.873954] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9037.880542] pending: lu_objects_flush [obdclass] [ 9037.889899] workqueue events_unbound: flags=0x2 [ 9037.892231] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9037.902249] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9037.906041] workqueue events_power_efficient: flags=0x80 [ 9037.912016] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9037.921575] pending: check_lifetime, do_cache_clean [ 9037.928157] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9037.938515] pending: neigh_periodic_work, neigh_periodic_work [ 9037.947765] workqueue mm_percpu_wq: flags=0x8 [ 9037.951130] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9037.962182] pending: vmstat_update, lru_add_drain_per_cpu [ 9037.969159] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9037.979642] pending: vmstat_update, lru_add_drain_per_cpu [ 9037.995885] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9038.013868] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9038.022275] workqueue xprtiod: flags=0xa [ 9038.027178] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9038.035433] pending: xprt_autoclose, xs_error_handle [ 9038.047058] workqueue ipv6_addrconf: flags=0x40008 [ 9038.062980] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9038.064935] pending: addrconf_verify_work [ 9038.074910] workqueue knbd0-recv: flags=0x1a [ 9038.089039] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9038.108822] in-flight: 651:recv_work [ 9038.119287] workqueue ptlrpc_pinger: flags=0x2 [ 9038.125872] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9038.134179] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9038.152223] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6401s workers=4 idle: 15979 31824 8 [ 9038.172359] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9028s workers=2 idle: 654 [ 9048.055214] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9048.057098] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9048.065561] Modules linked in: lustre(O) [ 9048.072180] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9048.072268] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9048.072274] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9048.072277] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9048.072290] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9048.072294] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9048.072298] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9048.072301] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9048.072303] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9048.072305] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9048.072307] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9048.072310] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9048.072313] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9048.072315] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9048.072323] Call Trace: [ 9048.072327] [ 9048.072332] ? show_regs.cold.9+0x22/0x2f [ 9048.072341] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9048.072347] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9048.072350] ? watchdog+0x40/0x40 [ 9048.072355] ? hrtimer_interrupt+0x128/0x2c0 [ 9048.072359] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9048.072365] ? apic_timer_interrupt+0xf/0x20 [ 9048.072369] [ 9048.072372] ? native_safe_halt+0x2e/0x50 [ 9048.072375] kvm_wait+0xcd/0xe0 [ 9048.072382] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9048.072389] do_raw_spin_lock+0x177/0x190 [ 9048.072393] _raw_spin_lock+0x21/0x30 [ 9048.072398] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9048.073839] obd_mod_free+0x46/0x140 [ptlrpc] [ 9048.074062] mdc_commit_open+0x91/0xb0 [mdc] [ 9048.074876] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9048.075097] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9048.078737] after_reply+0xe2c/0x1780 [ptlrpc] [ 9048.078944] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9048.080811] ? __next_timer_interrupt+0x160/0x160 [ 9048.080825] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9048.081041] ? do_wait_intr+0xf0/0xf0 [ 9048.081051] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9048.085698] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9048.085767] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9048.085774] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9048.085777] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9048.085789] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9048.085792] RSP: 0018:ffffbb5381503500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9048.085797] RAX: 000000000000391c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9048.085799] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9048.085801] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9048.085802] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9048.085804] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9048.085808] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9048.085810] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9048.085812] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9048.085821] Call Trace: [ 9048.085827] [ 9048.085833] ? show_regs.cold.9+0x22/0x2f [ 9048.085841] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9048.085849] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9048.085853] ? watchdog+0x40/0x40 [ 9048.085858] ? hrtimer_interrupt+0x128/0x2c0 [ 9048.085862] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9048.085870] ? apic_timer_interrupt+0xf/0x20 [ 9048.085873] [ 9048.085876] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9048.085883] do_raw_spin_lock+0x177/0x190 [ 9048.085887] _raw_spin_lock+0x21/0x30 [ 9048.085892] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9048.086168] ? do_raw_spin_unlock+0x75/0x190 [ 9048.086175] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9048.086386] ? do_wait_intr+0xf0/0xf0 [ 9048.086394] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9048.086588] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9048.086801] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9048.087007] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9048.091039] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9048.161777] kthread+0x1d1/0x200 [ 9048.161796] ? set_kthread_struct+0x70/0x70 [ 9048.161802] ret_from_fork+0x1f/0x30 [ 9048.933784] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9048.941609] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9048.949313] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9048.957003] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9048.972973] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9048.989306] lmv_intent_open+0x338/0x1150 [lmv] [ 9048.998915] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9049.017347] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9049.029934] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9049.043850] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9049.048774] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9049.058806] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9049.072044] ? do_raw_spin_unlock+0x75/0x190 [ 9049.079942] ? _raw_spin_unlock+0x12/0x30 [ 9049.087889] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9049.105281] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9049.107020] ll_atomic_open+0x36a/0xd70 [lustre] [ 9049.114030] lookup_open+0x356/0xb40 [ 9049.115733] do_last+0x313/0xfc0 [ 9049.118510] ? nd_jump_root+0xe5/0x160 [ 9049.121113] ? path_init+0x437/0x520 [ 9049.132184] path_openat+0xf7/0x500 [ 9049.133368] ? do_raw_spin_unlock+0x75/0x190 [ 9049.146453] do_filp_open+0x99/0x140 [ 9049.153582] ? getname_flags+0x6e/0x330 [ 9049.173816] ? __check_object_size+0xff/0x256 [ 9049.174915] ? do_raw_spin_unlock+0x75/0x190 [ 9049.176441] ? _raw_spin_unlock+0x12/0x30 [ 9049.196003] do_sys_openat2+0x2b4/0x410 [ 9049.202759] do_sys_open+0x73/0xa0 [ 9049.204578] __x64_sys_openat+0x24/0x30 [ 9049.212363] do_syscall_64+0xc1/0x440 [ 9049.214760] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9049.226240] RIP: 0033:0x7f53425a5332 [ 9049.227569] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9049.263180] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9049.279738] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9049.291209] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9049.306845] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9049.329495] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9049.357707] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9064.052045] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 9064.056150] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9064.065551] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 9064.131725] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9064.131745] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9064.131752] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9064.132050] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9064.132062] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9064.132067] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9064.132072] RAX: 0000000000005984 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9064.132074] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9064.132077] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9064.132079] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9064.132081] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9064.132085] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9064.132088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9064.132091] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9064.132100] Call Trace: [ 9064.132106] [ 9064.132111] ? show_regs.cold.9+0x22/0x2f [ 9064.132122] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9064.132128] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9064.132133] ? watchdog+0x40/0x40 [ 9064.132138] ? hrtimer_interrupt+0x128/0x2c0 [ 9064.132142] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9064.132150] ? apic_timer_interrupt+0xf/0x20 [ 9064.132155] [ 9064.132158] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9064.132164] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9064.132169] ? newidle_balance.isra.93+0x207/0x580 [ 9064.132175] do_raw_spin_lock+0x177/0x190 [ 9064.132180] _raw_spin_lock+0x21/0x30 [ 9064.132185] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9064.132391] process_one_work+0x2c8/0x700 [ 9064.132398] worker_thread+0x296/0x6e0 [ 9064.132402] ? rescuer_thread+0x570/0x570 [ 9064.132406] kthread+0x1d1/0x200 [ 9064.132410] ? set_kthread_struct+0x70/0x70 [ 9064.132413] ret_from_fork+0x1f/0x30 [ 9064.328024] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9064.355404] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9064.364177] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9064.370001] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9064.374663] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9064.389785] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9064.394163] RAX: 0000000000003711 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9064.397743] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9064.402449] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9064.405216] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9064.408296] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9064.411235] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9064.415367] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9064.417711] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9064.435560] Call Trace: [ 9064.438451] [ 9064.444386] ? show_regs.cold.9+0x22/0x2f [ 9064.446498] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9064.454177] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9064.461289] ? watchdog+0x40/0x40 [ 9064.464103] ? hrtimer_interrupt+0x128/0x2c0 [ 9064.468936] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9064.470945] ? apic_timer_interrupt+0xf/0x20 [ 9064.478453] [ 9064.479216] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9064.485848] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9064.488339] do_raw_spin_lock+0x177/0x190 [ 9064.493868] _raw_spin_lock+0x21/0x30 [ 9064.500927] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9064.511096] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9064.516985] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9064.521314] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9064.524695] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9064.542493] ? woken_wake_function+0x30/0x30 [ 9064.546988] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9064.552554] kthread+0x1d1/0x200 [ 9064.556882] ? set_kthread_struct+0x70/0x70 [ 9064.560025] ret_from_fork+0x1f/0x30 [ 9068.516686] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6431s! [ 9068.519656] Showing busy workqueues and worker pools: [ 9068.535408] workqueue events: flags=0x0 [ 9068.540985] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9068.552815] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9068.573550] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9068.578081] pending: cpu_latency_work [ptlrpc] [ 9068.583411] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9068.585885] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9068.598637] workqueue events_long: flags=0x0 [ 9068.605391] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9068.617378] pending: lu_objects_flush [obdclass] [ 9068.625718] workqueue events_unbound: flags=0x2 [ 9068.639178] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9068.641361] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9068.651352] workqueue events_power_efficient: flags=0x80 [ 9068.658717] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9068.667810] pending: check_lifetime, do_cache_clean [ 9068.684889] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9068.687363] pending: neigh_periodic_work, neigh_periodic_work [ 9068.706839] workqueue mm_percpu_wq: flags=0x8 [ 9068.713367] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9068.733368] pending: vmstat_update, lru_add_drain_per_cpu [ 9068.741573] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9068.756329] pending: vmstat_update, lru_add_drain_per_cpu [ 9068.762331] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9068.772855] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9068.789536] workqueue xprtiod: flags=0xa [ 9068.803708] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9068.821337] pending: xprt_autoclose, xs_error_handle [ 9068.829700] workqueue ipv6_addrconf: flags=0x40008 [ 9068.842669] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9068.856681] pending: addrconf_verify_work [ 9068.871934] workqueue knbd0-recv: flags=0x1a [ 9068.873508] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9068.890247] in-flight: 651:recv_work [ 9068.897481] workqueue ptlrpc_pinger: flags=0x2 [ 9068.904349] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9068.911360] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9068.927593] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6431s workers=4 idle: 15979 31824 8 [ 9068.942124] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9059s workers=2 idle: 654 [ 9076.062083] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9076.062143] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9076.068048] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9076.068135] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9076.068143] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9076.068145] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9076.068161] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9076.068165] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9076.068170] RAX: 0000000000002b81 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9076.068173] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9076.068175] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9076.068177] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9076.068180] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9076.068183] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9076.068186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9076.068188] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9076.068198] Call Trace: [ 9076.068205] [ 9076.068211] ? show_regs.cold.9+0x22/0x2f [ 9076.068223] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9076.068231] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9076.068235] ? watchdog+0x40/0x40 [ 9076.068239] ? hrtimer_interrupt+0x128/0x2c0 [ 9076.068244] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9076.068253] ? apic_timer_interrupt+0xf/0x20 [ 9076.068261] [ 9076.068265] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9076.068272] do_raw_spin_lock+0x177/0x190 [ 9076.068277] _raw_spin_lock+0x21/0x30 [ 9076.068282] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9076.068582] ? do_raw_spin_unlock+0x75/0x190 [ 9076.068588] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9076.070258] ? do_wait_intr+0xf0/0xf0 [ 9076.070268] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9076.070456] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9076.070678] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9076.070888] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9076.081094] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9076.081173] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9076.081180] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9076.081183] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9076.081198] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9076.081203] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9076.081207] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9076.115219] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9076.115285] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9076.115316] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9076.115344] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9076.115473] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9076.116198] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9076.116233] lmv_intent_open+0x338/0x1150 [lmv] [ 9076.116271] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9076.116365] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9076.116442] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9076.116464] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9076.116547] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9076.117181] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9076.117263] ? do_raw_spin_unlock+0x75/0x190 [ 9076.117273] ? _raw_spin_unlock+0x12/0x30 [ 9076.117281] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9076.117316] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9076.117335] ll_atomic_open+0x36a/0xd70 [lustre] [ 9076.117414] lookup_open+0x356/0xb40 [ 9076.117424] do_last+0x313/0xfc0 [ 9076.117429] ? nd_jump_root+0xe5/0x160 [ 9076.117434] ? path_init+0x437/0x520 [ 9076.117439] path_openat+0xf7/0x500 [ 9076.117444] ? do_raw_spin_unlock+0x75/0x190 [ 9076.117449] do_filp_open+0x99/0x140 [ 9076.117455] ? getname_flags+0x6e/0x330 [ 9076.117460] ? __check_object_size+0xff/0x256 [ 9076.117468] ? do_raw_spin_unlock+0x75/0x190 [ 9076.117471] ? _raw_spin_unlock+0x12/0x30 [ 9076.117475] do_sys_openat2+0x2b4/0x410 [ 9076.117480] do_sys_open+0x73/0xa0 [ 9076.117485] __x64_sys_openat+0x24/0x30 [ 9076.117489] do_syscall_64+0xc1/0x440 [ 9076.117497] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9076.117503] RIP: 0033:0x7f53425a5332 [ 9076.117511] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9076.117516] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9076.117521] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9076.117524] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9076.117526] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9076.117528] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9076.117531] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9076.885949] RDX: 0000000000005754 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9076.900385] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9076.905889] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9076.921372] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9076.937682] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9076.952935] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9076.964396] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9076.977380] Call Trace: [ 9076.979821] [ 9076.985419] ? show_regs.cold.9+0x22/0x2f [ 9076.992645] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9076.994316] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9077.003419] ? watchdog+0x40/0x40 [ 9077.005398] ? hrtimer_interrupt+0x128/0x2c0 [ 9077.007785] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9077.013412] ? apic_timer_interrupt+0xf/0x20 [ 9077.015516] [ 9077.016423] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9077.022248] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9077.025390] do_raw_spin_lock+0x177/0x190 [ 9077.027917] _raw_spin_lock+0x21/0x30 [ 9077.029839] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9077.032196] obd_mod_free+0x46/0x140 [ptlrpc] [ 9077.038886] mdc_commit_open+0x91/0xb0 [mdc] [ 9077.042454] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9077.045567] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9077.060339] after_reply+0xe2c/0x1780 [ptlrpc] [ 9077.072980] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9077.084462] ? __next_timer_interrupt+0x160/0x160 [ 9077.095739] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9077.097773] ? do_wait_intr+0xf0/0xf0 [ 9077.109293] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9077.125008] kthread+0x1d1/0x200 [ 9077.130554] ? set_kthread_struct+0x70/0x70 [ 9077.134215] ret_from_fork+0x1f/0x30 [ 9092.058019] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 9092.064197] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9092.090443] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9092.103456] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9092.108429] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9092.109916] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9092.132350] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9092.141585] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9092.147354] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9092.153227] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9092.161035] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9092.164584] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9092.167237] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9092.171478] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9092.173266] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9092.178065] Call Trace: [ 9092.178724] [ 9092.179283] ? show_regs.cold.9+0x22/0x2f [ 9092.182715] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9092.184847] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9092.186867] ? watchdog+0x40/0x40 [ 9092.189035] ? hrtimer_interrupt+0x128/0x2c0 [ 9092.190915] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9092.195874] ? apic_timer_interrupt+0xf/0x20 [ 9092.204125] [ 9092.208585] ? native_safe_halt+0x2e/0x50 [ 9092.210720] kvm_wait+0xcd/0xe0 [ 9092.211834] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9092.215675] do_raw_spin_lock+0x177/0x190 [ 9092.217643] _raw_spin_lock+0x21/0x30 [ 9092.218948] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9092.221845] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9092.226582] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9092.230225] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9092.232911] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9092.245775] ? woken_wake_function+0x30/0x30 [ 9092.253563] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9092.264168] kthread+0x1d1/0x200 [ 9092.268617] ? set_kthread_struct+0x70/0x70 [ 9092.272075] ret_from_fork+0x1f/0x30 [ 9096.055896] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 9096.058821] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9096.097052] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9096.110130] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9096.113058] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9096.115598] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9096.117827] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9096.129686] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9096.132196] RAX: 0000000000004b7a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9096.135352] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9096.139719] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9096.143594] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9096.147219] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9096.153224] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9096.156045] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9096.158813] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9096.163729] Call Trace: [ 9096.164925] [ 9096.165994] ? show_regs.cold.9+0x22/0x2f [ 9096.167608] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9096.169015] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9096.170780] ? watchdog+0x40/0x40 [ 9096.171810] ? hrtimer_interrupt+0x128/0x2c0 [ 9096.175240] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9096.177400] ? apic_timer_interrupt+0xf/0x20 [ 9096.178945] [ 9096.180869] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9096.184660] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9096.189050] ? newidle_balance.isra.93+0x207/0x580 [ 9096.190689] do_raw_spin_lock+0x177/0x190 [ 9096.195766] _raw_spin_lock+0x21/0x30 [ 9096.199562] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9096.202815] process_one_work+0x2c8/0x700 [ 9096.206842] worker_thread+0x296/0x6e0 [ 9096.209819] ? rescuer_thread+0x570/0x570 [ 9096.211941] kthread+0x1d1/0x200 [ 9096.213338] ? set_kthread_struct+0x70/0x70 [ 9096.215484] ret_from_fork+0x1f/0x30 [ 9099.231109] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6462s! [ 9099.237263] Showing busy workqueues and worker pools: [ 9099.241258] workqueue events: flags=0x0 [ 9099.245181] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9099.252365] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9099.260780] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9099.277706] pending: cpu_latency_work [ptlrpc] [ 9099.284370] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9099.293422] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9099.302011] workqueue events_long: flags=0x0 [ 9099.306842] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9099.313469] pending: lu_objects_flush [obdclass] [ 9099.318470] workqueue events_unbound: flags=0x2 [ 9099.323383] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9099.327347] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9099.333837] workqueue events_power_efficient: flags=0x80 [ 9099.339564] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9099.342055] pending: check_lifetime, do_cache_clean [ 9099.351902] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9099.362398] pending: neigh_periodic_work, neigh_periodic_work [ 9099.373596] workqueue mm_percpu_wq: flags=0x8 [ 9099.381387] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9099.391625] pending: vmstat_update, lru_add_drain_per_cpu [ 9099.402907] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9099.411680] pending: vmstat_update, lru_add_drain_per_cpu [ 9099.413269] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9099.426209] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9099.431007] workqueue xprtiod: flags=0xa [ 9099.432356] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9099.444581] pending: xprt_autoclose, xs_error_handle [ 9099.449494] workqueue ipv6_addrconf: flags=0x40008 [ 9099.455438] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9099.462055] pending: addrconf_verify_work [ 9099.463917] workqueue knbd0-recv: flags=0x1a [ 9099.467466] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9099.471386] in-flight: 651:recv_work [ 9099.474658] workqueue ptlrpc_pinger: flags=0x2 [ 9099.479357] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9099.483840] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9099.488742] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6462s workers=4 idle: 15979 31824 8 [ 9099.492355] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9089s workers=2 idle: 654 [ 9104.057033] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9104.057289] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9104.062843] Modules linked in: lustre(O) osp(O) ofd(O) [ 9104.067557] Modules linked in: [ 9104.067561] lod(O) mdt(O) mdd(O) [ 9104.069252] lustre(O) [ 9104.070481] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9104.070543] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9104.070551] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9104.070553] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9104.070566] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9104.070570] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9104.070575] RAX: 0000000000007f86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9104.070577] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9104.070580] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9104.070581] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9104.070584] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9104.070586] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9104.070589] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9104.070591] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9104.070600] Call Trace: [ 9104.070607] [ 9104.070613] ? show_regs.cold.9+0x22/0x2f [ 9104.070625] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9104.070632] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9104.070636] ? watchdog+0x40/0x40 [ 9104.070640] ? hrtimer_interrupt+0x128/0x2c0 [ 9104.070643] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9104.070654] ? apic_timer_interrupt+0xf/0x20 [ 9104.070659] [ 9104.070662] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9104.070669] do_raw_spin_lock+0x177/0x190 [ 9104.070674] _raw_spin_lock+0x21/0x30 [ 9104.070679] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9104.072018] osp(O) [ 9104.076216] ? do_raw_spin_unlock+0x75/0x190 [ 9104.094914] ofd(O) [ 9104.099670] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9104.103264] lod(O) [ 9104.105851] ? do_wait_intr+0xf0/0xf0 [ 9104.105878] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9104.112839] mdt(O) [ 9104.115890] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9104.119306] mdd(O) [ 9104.123887] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9104.129387] mgs(O) [ 9104.133942] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9104.135316] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9104.138744] lquota(O) lfsck(O) [ 9104.146915] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9104.149795] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9104.149863] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9104.149871] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9104.149874] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9104.149888] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9104.149893] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9104.149897] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9104.149899] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9104.149902] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9104.149904] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9104.149906] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9104.149909] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9104.149912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9104.149914] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9104.149922] Call Trace: [ 9104.149930] [ 9104.149938] ? show_regs.cold.9+0x22/0x2f [ 9104.149949] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9104.157691] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9104.159094] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9104.159118] ? watchdog+0x40/0x40 [ 9104.159125] ? hrtimer_interrupt+0x128/0x2c0 [ 9104.159130] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9104.159138] ? apic_timer_interrupt+0xf/0x20 [ 9104.159143] [ 9104.159148] ? native_safe_halt+0x2e/0x50 [ 9104.159152] kvm_wait+0xcd/0xe0 [ 9104.159161] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9104.159170] do_raw_spin_lock+0x177/0x190 [ 9104.159175] _raw_spin_lock+0x21/0x30 [ 9104.159183] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9104.159622] obd_mod_free+0x46/0x140 [ptlrpc] [ 9104.159892] mdc_commit_open+0x91/0xb0 [mdc] [ 9104.159938] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9104.160216] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9104.160809] after_reply+0xe2c/0x1780 [ptlrpc] [ 9104.161015] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9104.162207] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9104.164187] ? __next_timer_interrupt+0x160/0x160 [ 9104.168110] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9104.171858] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9104.173798] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9104.176457] ? do_wait_intr+0xf0/0xf0 [ 9104.179343] lmv_intent_open+0x338/0x1150 [lmv] [ 9104.182057] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9104.182466] kthread+0x1d1/0x200 [ 9104.182474] ? set_kthread_struct+0x70/0x70 [ 9104.182478] ret_from_fork+0x1f/0x30 [ 9104.560610] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9104.562622] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9104.564863] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9104.567717] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9104.572555] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9104.576938] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9104.581079] ? do_raw_spin_unlock+0x75/0x190 [ 9104.582956] ? _raw_spin_unlock+0x12/0x30 [ 9104.584938] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9104.592522] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9104.596501] ll_atomic_open+0x36a/0xd70 [lustre] [ 9104.602299] lookup_open+0x356/0xb40 [ 9104.604197] do_last+0x313/0xfc0 [ 9104.605192] ? nd_jump_root+0xe5/0x160 [ 9104.608833] ? path_init+0x437/0x520 [ 9104.610085] path_openat+0xf7/0x500 [ 9104.611648] ? do_raw_spin_unlock+0x75/0x190 [ 9104.617879] do_filp_open+0x99/0x140 [ 9104.619761] ? getname_flags+0x6e/0x330 [ 9104.622967] ? __check_object_size+0xff/0x256 [ 9104.627733] ? do_raw_spin_unlock+0x75/0x190 [ 9104.629620] ? _raw_spin_unlock+0x12/0x30 [ 9104.631509] do_sys_openat2+0x2b4/0x410 [ 9104.633520] do_sys_open+0x73/0xa0 [ 9104.635796] __x64_sys_openat+0x24/0x30 [ 9104.637669] do_syscall_64+0xc1/0x440 [ 9104.639372] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9104.641712] RIP: 0033:0x7f53425a5332 [ 9104.643280] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9104.664378] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9104.673975] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9104.685072] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9104.695153] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9104.703578] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9104.710912] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9120.058059] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 9120.062392] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9120.084186] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9120.089471] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9120.093455] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9120.095685] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9120.102578] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9120.104697] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9120.107496] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9120.110315] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9120.113542] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9120.117109] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9120.119996] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9120.123586] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9120.125673] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9120.128498] Call Trace: [ 9120.129481] [ 9120.130438] ? show_regs.cold.9+0x22/0x2f [ 9120.131956] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9120.133533] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9120.136110] ? watchdog+0x40/0x40 [ 9120.137372] ? hrtimer_interrupt+0x128/0x2c0 [ 9120.139125] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9120.141054] ? apic_timer_interrupt+0xf/0x20 [ 9120.142553] [ 9120.143257] ? native_safe_halt+0x2e/0x50 [ 9120.145862] kvm_wait+0xcd/0xe0 [ 9120.147413] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9120.149319] do_raw_spin_lock+0x177/0x190 [ 9120.151765] _raw_spin_lock+0x21/0x30 [ 9120.153266] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9120.157497] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9120.167813] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9120.173197] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9120.176350] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9120.178645] ? woken_wake_function+0x30/0x30 [ 9120.181034] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9120.184607] kthread+0x1d1/0x200 [ 9120.185715] ? set_kthread_struct+0x70/0x70 [ 9120.188179] ret_from_fork+0x1f/0x30 [ 9124.053910] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 9124.060973] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9124.096108] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9124.104153] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9124.109525] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9124.115852] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9124.121313] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9124.136772] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9124.140884] RAX: 000000000000640f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9124.147189] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9124.149880] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9124.153118] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9124.157882] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9124.163869] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9124.168977] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9124.172820] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9124.178273] Call Trace: [ 9124.179144] [ 9124.179826] ? show_regs.cold.9+0x22/0x2f [ 9124.184116] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9124.186206] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9124.190738] ? watchdog+0x40/0x40 [ 9124.193191] ? hrtimer_interrupt+0x128/0x2c0 [ 9124.197552] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9124.201993] ? apic_timer_interrupt+0xf/0x20 [ 9124.205832] [ 9124.207906] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9124.212198] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9124.218957] ? newidle_balance.isra.93+0x207/0x580 [ 9124.221110] do_raw_spin_lock+0x177/0x190 [ 9124.223381] _raw_spin_lock+0x21/0x30 [ 9124.225902] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9124.230243] process_one_work+0x2c8/0x700 [ 9124.233926] worker_thread+0x296/0x6e0 [ 9124.237530] ? rescuer_thread+0x570/0x570 [ 9124.240639] kthread+0x1d1/0x200 [ 9124.244160] ? set_kthread_struct+0x70/0x70 [ 9124.248487] ret_from_fork+0x1f/0x30 [ 9129.951275] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6492s! [ 9129.954487] Showing busy workqueues and worker pools: [ 9129.957321] workqueue events: flags=0x0 [ 9129.958957] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9129.961641] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9129.967442] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9129.970065] pending: cpu_latency_work [ptlrpc] [ 9129.972569] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9129.975561] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9129.979506] workqueue events_long: flags=0x0 [ 9129.980915] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9129.983623] pending: lu_objects_flush [obdclass] [ 9129.986571] workqueue events_unbound: flags=0x2 [ 9129.989076] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9129.991545] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9129.994283] workqueue events_power_efficient: flags=0x80 [ 9129.996433] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9129.999941] pending: check_lifetime, do_cache_clean [ 9130.002259] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9130.005320] pending: neigh_periodic_work, neigh_periodic_work [ 9130.008017] workqueue mm_percpu_wq: flags=0x8 [ 9130.010275] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9130.013303] pending: vmstat_update, lru_add_drain_per_cpu [ 9130.015618] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9130.021538] pending: vmstat_update, lru_add_drain_per_cpu [ 9130.025279] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9130.028141] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9130.030990] workqueue xprtiod: flags=0xa [ 9130.036526] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9130.039108] pending: xprt_autoclose, xs_error_handle [ 9130.045406] workqueue ipv6_addrconf: flags=0x40008 [ 9130.047402] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9130.049726] pending: addrconf_verify_work [ 9130.051452] workqueue knbd0-recv: flags=0x1a [ 9130.053115] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9130.055518] in-flight: 651:recv_work [ 9130.057212] workqueue ptlrpc_pinger: flags=0x2 [ 9130.062134] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9130.064675] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9130.066815] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6492s workers=4 idle: 15979 31824 8 [ 9130.069289] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9120s workers=2 idle: 654 [ 9132.055032] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9132.057014] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9132.058909] Modules linked in: [ 9132.061644] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac [ 9132.063572] lustre(O) [ 9132.063603] rapl [ 9132.078372] osp(O) [ 9132.079459] i2c_piix4 [ 9132.080261] ofd(O) [ 9132.081145] pcspkr squashfs ata_generic crct10dif_pclmul [ 9132.081929] lod(O) [ 9132.082850] crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9132.082906] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9132.082913] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9132.088373] mdt(O) [ 9132.089268] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9132.089333] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9132.089338] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9132.089344] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9132.089346] RDX: 000000000000359c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9132.089349] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9132.089351] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9132.089353] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9132.089356] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9132.089359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9132.089362] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9132.089370] Call Trace: [ 9132.089388] [ 9132.089394] ? show_regs.cold.9+0x22/0x2f [ 9132.089404] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9132.089409] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9132.089414] ? watchdog+0x40/0x40 [ 9132.089417] ? hrtimer_interrupt+0x128/0x2c0 [ 9132.089421] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9132.089430] ? apic_timer_interrupt+0xf/0x20 [ 9132.089434] [ 9132.089437] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9132.089442] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9132.089446] do_raw_spin_lock+0x177/0x190 [ 9132.089450] _raw_spin_lock+0x21/0x30 [ 9132.089457] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9132.103794] mdd(O) [ 9132.108503] obd_mod_free+0x46/0x140 [ptlrpc] [ 9132.114043] mgs(O) [ 9132.115209] mdc_commit_open+0x91/0xb0 [mdc] [ 9132.115273] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9132.115587] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9132.115817] after_reply+0xe2c/0x1780 [ptlrpc] [ 9132.119039] lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9132.119202] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9132.119211] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9132.119214] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9132.119233] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9132.119238] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9132.119243] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9132.119246] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9132.119248] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9132.119250] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9132.119253] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9132.119258] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9132.119261] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9132.119263] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9132.119272] Call Trace: [ 9132.119280] [ 9132.119287] ? show_regs.cold.9+0x22/0x2f [ 9132.119301] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9132.119308] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9132.119313] ? watchdog+0x40/0x40 [ 9132.119319] ? hrtimer_interrupt+0x128/0x2c0 [ 9132.119324] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9132.119333] ? apic_timer_interrupt+0xf/0x20 [ 9132.119338] [ 9132.119342] ? native_safe_halt+0x2e/0x50 [ 9132.119346] kvm_wait+0xcd/0xe0 [ 9132.119353] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9132.119362] do_raw_spin_lock+0x177/0x190 [ 9132.119368] _raw_spin_lock+0x21/0x30 [ 9132.119372] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9132.119732] ? do_raw_spin_unlock+0x75/0x190 [ 9132.119739] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9132.119949] ? do_wait_intr+0xf0/0xf0 [ 9132.119958] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9132.120289] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9132.120500] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9132.120913] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9132.121071] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9132.121122] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9132.121154] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9132.121181] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9132.121262] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9132.129872] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9132.130393] ? __next_timer_interrupt+0x160/0x160 [ 9132.130407] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9132.130684] ? do_wait_intr+0xf0/0xf0 [ 9132.130695] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9132.130904] kthread+0x1d1/0x200 [ 9132.130910] ? set_kthread_struct+0x70/0x70 [ 9132.130914] ret_from_fork+0x1f/0x30 [ 9132.360962] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9132.362975] lmv_intent_open+0x338/0x1150 [lmv] [ 9132.364799] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9132.366908] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9132.369181] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9132.371506] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9132.373708] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9132.376931] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9132.379015] ? do_raw_spin_unlock+0x75/0x190 [ 9132.381205] ? _raw_spin_unlock+0x12/0x30 [ 9132.382579] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9132.385404] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9132.388712] ll_atomic_open+0x36a/0xd70 [lustre] [ 9132.391945] lookup_open+0x356/0xb40 [ 9132.393071] do_last+0x313/0xfc0 [ 9132.394226] ? nd_jump_root+0xe5/0x160 [ 9132.396393] ? path_init+0x437/0x520 [ 9132.397672] path_openat+0xf7/0x500 [ 9132.399201] ? do_raw_spin_unlock+0x75/0x190 [ 9132.401425] do_filp_open+0x99/0x140 [ 9132.403401] ? getname_flags+0x6e/0x330 [ 9132.405418] ? __check_object_size+0xff/0x256 [ 9132.406916] ? do_raw_spin_unlock+0x75/0x190 [ 9132.409874] ? _raw_spin_unlock+0x12/0x30 [ 9132.412629] do_sys_openat2+0x2b4/0x410 [ 9132.415940] do_sys_open+0x73/0xa0 [ 9132.418481] __x64_sys_openat+0x24/0x30 [ 9132.419959] do_syscall_64+0xc1/0x440 [ 9132.421913] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9132.424731] RIP: 0033:0x7f53425a5332 [ 9132.427829] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9132.436376] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9132.440303] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9132.442829] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9132.445426] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9132.447904] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9132.450478] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9148.059615] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 9148.067330] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9148.103753] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9148.109733] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9148.113158] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9148.118668] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9148.128002] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9148.131520] RAX: 0000000000002583 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9148.134968] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9148.137618] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9148.140245] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9148.142741] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9148.146994] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9148.152457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9148.154083] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9148.156716] Call Trace: [ 9148.157657] [ 9148.158305] ? show_regs.cold.9+0x22/0x2f [ 9148.159766] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9148.161716] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9148.163425] ? watchdog+0x40/0x40 [ 9148.165749] ? hrtimer_interrupt+0x128/0x2c0 [ 9148.172194] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9148.173811] ? apic_timer_interrupt+0xf/0x20 [ 9148.175210] [ 9148.177621] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9148.182576] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9148.188224] do_raw_spin_lock+0x177/0x190 [ 9148.192856] _raw_spin_lock+0x21/0x30 [ 9148.194011] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9148.197576] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9148.203576] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9148.213239] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9148.217144] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9148.219787] ? woken_wake_function+0x30/0x30 [ 9148.221080] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9148.226486] kthread+0x1d1/0x200 [ 9148.227625] ? set_kthread_struct+0x70/0x70 [ 9148.228834] ret_from_fork+0x1f/0x30 [ 9156.052553] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 9156.056253] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9156.075844] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9156.080783] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9156.084760] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9156.087813] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9156.089726] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9156.097072] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9156.100106] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9156.103161] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9156.106057] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9156.108160] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9156.112348] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9156.115055] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9156.118715] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9156.121119] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9156.123984] Call Trace: [ 9156.124990] [ 9156.125961] ? show_regs.cold.9+0x22/0x2f [ 9156.127785] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9156.129470] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9156.130726] ? watchdog+0x40/0x40 [ 9156.132674] ? hrtimer_interrupt+0x128/0x2c0 [ 9156.135471] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9156.137569] ? apic_timer_interrupt+0xf/0x20 [ 9156.139110] [ 9156.140584] ? native_safe_halt+0x2e/0x50 [ 9156.141995] kvm_wait+0xcd/0xe0 [ 9156.143533] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9156.145685] ? newidle_balance.isra.93+0x207/0x580 [ 9156.147606] do_raw_spin_lock+0x177/0x190 [ 9156.149481] _raw_spin_lock+0x21/0x30 [ 9156.151454] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9156.154063] process_one_work+0x2c8/0x700 [ 9156.156066] worker_thread+0x296/0x6e0 [ 9156.158004] ? rescuer_thread+0x570/0x570 [ 9156.160053] kthread+0x1d1/0x200 [ 9156.161728] ? set_kthread_struct+0x70/0x70 [ 9156.163763] ret_from_fork+0x1f/0x30 [ 9160.055151] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9160.057027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9160.059610] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9160.059687] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9160.059695] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9160.059697] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9160.059711] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9160.059715] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9160.059720] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9160.059722] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9160.059724] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9160.059726] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9160.059728] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9160.059731] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9160.059734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9160.059736] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9160.059810] Call Trace: [ 9160.059818] [ 9160.059825] ? show_regs.cold.9+0x22/0x2f [ 9160.059837] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9160.059843] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9160.059847] ? watchdog+0x40/0x40 [ 9160.059851] ? hrtimer_interrupt+0x128/0x2c0 [ 9160.059856] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9160.059864] ? apic_timer_interrupt+0xf/0x20 [ 9160.059869] [ 9160.059872] ? native_safe_halt+0x2e/0x50 [ 9160.059876] kvm_wait+0xcd/0xe0 [ 9160.059884] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9160.061693] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9160.061749] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9160.061755] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9160.061757] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9160.061766] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9160.061769] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9160.061773] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9160.061775] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9160.061776] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9160.061778] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9160.061779] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9160.061782] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9160.061783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9160.061785] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9160.061790] Call Trace: [ 9160.061793] [ 9160.061798] ? show_regs.cold.9+0x22/0x2f [ 9160.061806] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9160.061810] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9160.061812] ? watchdog+0x40/0x40 [ 9160.061815] ? hrtimer_interrupt+0x128/0x2c0 [ 9160.061818] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9160.061823] ? apic_timer_interrupt+0xf/0x20 [ 9160.061826] [ 9160.061828] ? native_safe_halt+0x2e/0x50 [ 9160.061831] kvm_wait+0xcd/0xe0 [ 9160.061837] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9160.061841] do_raw_spin_lock+0x177/0x190 [ 9160.061844] _raw_spin_lock+0x21/0x30 [ 9160.061848] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9160.062062] obd_mod_free+0x46/0x140 [ptlrpc] [ 9160.062219] mdc_commit_open+0x91/0xb0 [mdc] [ 9160.062250] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9160.062413] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9160.062537] after_reply+0xe2c/0x1780 [ptlrpc] [ 9160.062661] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9160.062784] ? __next_timer_interrupt+0x160/0x160 [ 9160.062790] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9160.062937] ? do_wait_intr+0xf0/0xf0 [ 9160.062946] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9160.063075] kthread+0x1d1/0x200 [ 9160.063079] ? set_kthread_struct+0x70/0x70 [ 9160.063082] ret_from_fork+0x1f/0x30 [ 9160.161527] BUG: workqueue lockup - pool [ 9160.164439] do_raw_spin_lock+0x177/0x190 [ 9160.164461] _raw_spin_lock+0x21/0x30 [ 9160.164468] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9160.166893] cpus=0-3 [ 9160.168731] ? do_raw_spin_unlock+0x75/0x190 [ 9160.168749] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9160.170626] flags=0x4 nice=0 stuck for 6523s! [ 9160.170640] Showing busy workqueues and worker pools: [ 9160.170643] workqueue events: flags=0x0 [ 9160.170645] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9160.170657] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9160.171508] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9160.171520] pending: cpu_latency_work [ptlrpc] [ 9160.171745] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9160.171754] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9160.172058] workqueue events_long: flags=0x0 [ 9160.172062] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9160.172070] pending: lu_objects_flush [obdclass] [ 9160.172660] workqueue events_unbound: flags=0x2 [ 9160.172663] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9160.172672] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9160.172722] workqueue events_power_efficient: flags=0x80 [ 9160.172726] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9160.172734] pending: check_lifetime, do_cache_clean [ 9160.172750] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9160.172757] pending: neigh_periodic_work, neigh_periodic_work [ 9160.172774] workqueue mm_percpu_wq: flags=0x8 [ 9160.172777] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9160.172782] pending: vmstat_update, lru_add_drain_per_cpu [ 9160.172793] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9160.172799] pending: vmstat_update, lru_add_drain_per_cpu [ 9160.172807] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9160.172813] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9160.172833] workqueue xprtiod: flags=0xa [ 9160.172835] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9160.172841] pending: xprt_autoclose, xs_error_handle [ 9160.172857] workqueue ipv6_addrconf: flags=0x40008 [ 9160.172859] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9160.172866] pending: addrconf_verify_work [ 9160.172878] workqueue knbd0-recv: flags=0x1a [ 9160.172880] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9160.172886] in-flight: 651:recv_work [ 9160.172896] workqueue ptlrpc_pinger: flags=0x2 [ 9160.172899] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9160.172903] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9160.173178] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6523s workers=4 idle: 15979 31824 8 [ 9160.173193] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9150s workers=2 idle: 654 [ 9160.588750] ? do_wait_intr+0xf0/0xf0 [ 9160.590194] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9160.600931] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9160.602733] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9160.609237] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9160.610883] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9160.614273] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9160.615910] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9160.622603] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9160.630571] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9160.641878] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9160.648778] lmv_intent_open+0x338/0x1150 [lmv] [ 9160.652070] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9160.664447] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9160.678950] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9160.680281] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9160.681766] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9160.691358] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9160.692919] ? do_raw_spin_unlock+0x75/0x190 [ 9160.702468] ? _raw_spin_unlock+0x12/0x30 [ 9160.708754] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9160.710908] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9160.716513] ll_atomic_open+0x36a/0xd70 [lustre] [ 9160.721487] lookup_open+0x356/0xb40 [ 9160.725667] do_last+0x313/0xfc0 [ 9160.728668] ? nd_jump_root+0xe5/0x160 [ 9160.732607] ? path_init+0x437/0x520 [ 9160.734632] path_openat+0xf7/0x500 [ 9160.735995] ? do_raw_spin_unlock+0x75/0x190 [ 9160.738575] do_filp_open+0x99/0x140 [ 9160.742614] ? getname_flags+0x6e/0x330 [ 9160.746506] ? __check_object_size+0xff/0x256 [ 9160.748591] ? do_raw_spin_unlock+0x75/0x190 [ 9160.750480] ? _raw_spin_unlock+0x12/0x30 [ 9160.752443] do_sys_openat2+0x2b4/0x410 [ 9160.754523] do_sys_open+0x73/0xa0 [ 9160.755725] __x64_sys_openat+0x24/0x30 [ 9160.756979] do_syscall_64+0xc1/0x440 [ 9160.761322] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9160.765898] RIP: 0033:0x7f53425a5332 [ 9160.769757] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9160.786988] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9160.792629] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9160.797791] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9160.806403] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9160.811417] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9160.815601] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9174.604038] rcu: INFO: rcu_sched self-detected stall on CPU [ 9174.607249] rcu: 2-....: (3860439 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 9174.611615] (t=7080178 jiffies g=274185 q=65685) [ 9174.612931] rcu: rcu_sched kthread starved for 6538544 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 9174.636695] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9174.653746] rcu: RCU grace-period kthread stack dump: [ 9174.666516] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9174.683419] Call Trace: [ 9174.686817] __schedule+0x351/0xcb0 [ 9174.688756] schedule+0xc0/0x180 [ 9174.696317] schedule_timeout+0xb4/0x190 [ 9174.697720] ? do_raw_spin_unlock+0x75/0x190 [ 9174.716793] ? __next_timer_interrupt+0x160/0x160 [ 9174.727564] ? prepare_to_swait_event+0xf3/0x140 [ 9174.735164] rcu_gp_kthread+0x8dd/0x15b0 [ 9174.736582] ? force_qs_rnp+0x2c0/0x2c0 [ 9174.740573] kthread+0x1d1/0x200 [ 9174.743755] ? set_kthread_struct+0x70/0x70 [ 9174.748814] ret_from_fork+0x1f/0x30 [ 9174.752745] rcu: Stack dump where RCU GP kthread last ran: [ 9174.761299] Sending NMI from CPU 2 to CPUs 0: [ 9174.772492] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [ 9174.774469] Sending NMI from CPU 2 to CPUs 1: [ 9174.789058] NMI backtrace for cpu 1 [ 9174.789067] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9174.789069] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9174.789071] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9174.789073] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9174.789075] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [ 9174.789078] RAX: 0000000000007c8b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9174.789079] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9174.789081] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9174.789083] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9174.789084] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9174.789085] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9174.789087] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9174.789088] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9174.789089] Call Trace: [ 9174.789090] [ 9174.789091] ? show_regs.cold.9+0x22/0x2f [ 9174.789093] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9174.789094] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9174.789095] ? nmi_handle+0xbd/0x210 [ 9174.789096] ? default_do_nmi+0x4b/0x1b0 [ 9174.789097] ? do_nmi+0x280/0x380 [ 9174.789098] ? end_repeat_nmi+0x16/0x1f [ 9174.789099] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9174.789100] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9174.789102] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9174.789102] [ 9174.789103] do_raw_spin_lock+0x177/0x190 [ 9174.789104] _raw_spin_lock+0x21/0x30 [ 9174.789105] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9174.789107] ? do_raw_spin_unlock+0x75/0x190 [ 9174.789108] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9174.789109] ? do_wait_intr+0xf0/0xf0 [ 9174.789110] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9174.789111] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9174.789112] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9174.789113] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9174.789114] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9174.789116] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9174.789117] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9174.789118] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9174.789119] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9174.789120] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9174.789121] lmv_intent_open+0x338/0x1150 [lmv] [ 9174.789122] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9174.789124] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9174.789125] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9174.789126] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9174.789127] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9174.789128] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9174.789129] ? do_raw_spin_unlock+0x75/0x190 [ 9174.789130] ? _raw_spin_unlock+0x12/0x30 [ 9174.789131] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9174.789132] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9174.789133] ll_atomic_open+0x36a/0xd70 [lustre] [ 9174.789134] lookup_open+0x356/0xb40 [ 9174.789135] do_last+0x313/0xfc0 [ 9174.789136] ? nd_jump_root+0xe5/0x160 [ 9174.789137] ? path_init+0x437/0x520 [ 9174.789138] path_openat+0xf7/0x500 [ 9174.789139] ? do_raw_spin_unlock+0x75/0x190 [ 9174.789140] do_filp_open+0x99/0x140 [ 9174.789141] ? getname_flags+0x6e/0x330 [ 9174.789142] ? __check_object_size+0xff/0x256 [ 9174.789143] ? do_raw_spin_unlock+0x75/0x190 [ 9174.789144] ? _raw_spin_unlock+0x12/0x30 [ 9174.789145] do_sys_openat2+0x2b4/0x410 [ 9174.789146] do_sys_open+0x73/0xa0 [ 9174.789346] __x64_sys_openat+0x24/0x30 [ 9174.789347] do_syscall_64+0xc1/0x440 [ 9174.789348] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9174.789349] RIP: 0033:0x7f53425a5332 [ 9174.789350] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9174.789352] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9174.789355] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9174.789356] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9174.789357] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9174.789359] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9174.789360] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9174.789946] NMI backtrace for cpu 2 [ 9175.204971] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9175.223121] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9175.231275] Call Trace: [ 9175.235553] [ 9175.238517] ? dump_stack+0xbb/0x10e [ 9175.243154] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9175.248807] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9175.261878] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9175.268301] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9175.272383] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9175.275716] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9175.281202] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9175.288535] ? update_process_times+0x6d/0xc0 [ 9175.294275] ? tick_sched_handle.isra.18+0x27/0x90 [ 9175.304523] ? tick_sched_timer+0x53/0xb0 [ 9175.305950] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9175.310699] ? tick_sched_handle.isra.18+0x90/0x90 [ 9175.313395] ? hrtimer_interrupt+0x128/0x2c0 [ 9175.317410] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9175.319782] ? apic_timer_interrupt+0xf/0x20 [ 9175.321623] [ 9175.322355] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9175.323791] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9175.332940] ? do_raw_spin_lock+0x177/0x190 [ 9175.336557] ? _raw_spin_lock+0x21/0x30 [ 9175.342154] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9175.348794] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9175.355655] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9175.358809] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9175.362468] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9175.365533] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9175.371666] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9175.376825] ? __next_timer_interrupt+0x160/0x160 [ 9175.380252] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9175.381869] ? do_wait_intr+0xf0/0xf0 [ 9175.383991] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9175.391126] ? kthread+0x1d1/0x200 [ 9175.394765] ? set_kthread_struct+0x70/0x70 [ 9175.396579] ? ret_from_fork+0x1f/0x30 [ 9176.059045] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 9176.069963] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9176.112114] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9176.124150] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9176.130957] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9176.134502] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9176.147935] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9176.157531] RAX: 0000000000005e04 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9176.164438] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9176.171430] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9176.177395] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9176.184536] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9176.189614] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9176.195822] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9176.200372] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9176.204526] Call Trace: [ 9176.206637] [ 9176.208630] ? show_regs.cold.9+0x22/0x2f [ 9176.213445] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9176.218371] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9176.220383] ? watchdog+0x40/0x40 [ 9176.223102] ? hrtimer_interrupt+0x128/0x2c0 [ 9176.225659] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9176.231188] ? apic_timer_interrupt+0xf/0x20 [ 9176.233181] [ 9176.234826] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9176.243229] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9176.246075] do_raw_spin_lock+0x177/0x190 [ 9176.248098] _raw_spin_lock+0x21/0x30 [ 9176.249315] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9176.252407] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9176.255323] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9176.258095] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9176.260220] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9176.262953] ? woken_wake_function+0x30/0x30 [ 9176.264963] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9176.267779] kthread+0x1d1/0x200 [ 9176.269279] ? set_kthread_struct+0x70/0x70 [ 9176.271456] ret_from_fork+0x1f/0x30 [ 9190.370097] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6553s! [ 9190.375499] Showing busy workqueues and worker pools: [ 9190.382089] workqueue events: flags=0x0 [ 9190.383475] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9190.389122] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9190.398035] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9190.402060] pending: cpu_latency_work [ptlrpc] [ 9190.413448] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9190.415851] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9190.430363] workqueue events_long: flags=0x0 [ 9190.432371] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9190.439851] pending: lu_objects_flush [obdclass] [ 9190.442563] workqueue events_unbound: flags=0x2 [ 9190.445356] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9190.449866] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9190.454031] workqueue events_power_efficient: flags=0x80 [ 9190.457675] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9190.459398] pending: check_lifetime, do_cache_clean [ 9190.469016] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9190.478541] pending: neigh_periodic_work, neigh_periodic_work [ 9190.485158] workqueue mm_percpu_wq: flags=0x8 [ 9190.488278] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9190.499958] pending: vmstat_update, lru_add_drain_per_cpu [ 9190.511222] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9190.522429] pending: vmstat_update, lru_add_drain_per_cpu [ 9190.524482] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9190.533501] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9190.541317] workqueue xprtiod: flags=0xa [ 9190.543497] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9190.551234] pending: xprt_autoclose, xs_error_handle [ 9190.554339] workqueue ipv6_addrconf: flags=0x40008 [ 9190.558291] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9190.564579] pending: addrconf_verify_work [ 9190.567250] workqueue knbd0-recv: flags=0x1a [ 9190.585216] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9190.587562] in-flight: 651:recv_work [ 9190.593636] workqueue ptlrpc_pinger: flags=0x2 [ 9190.600016] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9190.608042] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9190.611613] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6553s workers=4 idle: 15979 31824 8 [ 9190.628444] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9181s workers=2 idle: 654 [ 9200.055041] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 9200.057037] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 9200.064369] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9200.064459] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9200.064468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9200.064471] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9200.064487] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9200.064491] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9200.064495] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9200.064498] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9200.064500] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9200.064502] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9200.064505] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9200.064508] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9200.064512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9200.064514] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9200.064523] Call Trace: [ 9200.064530] [ 9200.064538] ? show_regs.cold.9+0x22/0x2f [ 9200.064550] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9200.064557] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9200.064563] ? watchdog+0x40/0x40 [ 9200.064567] ? hrtimer_interrupt+0x128/0x2c0 [ 9200.064572] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9200.064582] ? apic_timer_interrupt+0xf/0x20 [ 9200.064586] [ 9200.064590] ? native_safe_halt+0x2e/0x50 [ 9200.064594] kvm_wait+0xcd/0xe0 [ 9200.064604] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9200.064615] do_raw_spin_lock+0x177/0x190 [ 9200.064621] _raw_spin_lock+0x21/0x30 [ 9200.064625] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9200.065282] ? do_raw_spin_unlock+0x75/0x190 [ 9200.065290] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9200.065497] ? do_wait_intr+0xf0/0xf0 [ 9200.065509] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9200.065709] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9200.111215] Modules linked in: [ 9200.119795] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9200.120130] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9200.120258] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9200.120297] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9200.120328] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9200.120357] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9200.120460] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9200.120680] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9200.120708] lmv_intent_open+0x338/0x1150 [lmv] [ 9200.120740] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9200.120828] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9200.120911] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9200.120934] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9200.133274] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9200.133355] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9200.133363] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9200.133366] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9200.133384] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9200.133389] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9200.133395] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9200.133398] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9200.133400] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9200.133403] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9200.133406] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9200.133410] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9200.133413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9200.133416] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9200.135183] Call Trace: [ 9200.135206] [ 9200.135216] ? show_regs.cold.9+0x22/0x2f [ 9200.135242] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9200.135253] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9200.135258] ? watchdog+0x40/0x40 [ 9200.135269] ? hrtimer_interrupt+0x128/0x2c0 [ 9200.135272] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9200.135282] ? apic_timer_interrupt+0xf/0x20 [ 9200.135287] [ 9200.135290] ? native_safe_halt+0x2e/0x50 [ 9200.135295] kvm_wait+0xcd/0xe0 [ 9200.135304] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9200.135314] do_raw_spin_lock+0x177/0x190 [ 9200.135318] _raw_spin_lock+0x21/0x30 [ 9200.135326] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9200.141557] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9200.141717] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9200.141806] ? do_raw_spin_unlock+0x75/0x190 [ 9200.141819] ? _raw_spin_unlock+0x12/0x30 [ 9200.141829] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9200.141865] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9200.141885] ll_atomic_open+0x36a/0xd70 [lustre] [ 9200.141968] lookup_open+0x356/0xb40 [ 9200.141979] do_last+0x313/0xfc0 [ 9200.141986] ? nd_jump_root+0xe5/0x160 [ 9200.141991] ? path_init+0x437/0x520 [ 9200.141996] path_openat+0xf7/0x500 [ 9200.142001] ? do_raw_spin_unlock+0x75/0x190 [ 9200.142006] do_filp_open+0x99/0x140 [ 9200.142014] ? getname_flags+0x6e/0x330 [ 9200.142018] ? __check_object_size+0xff/0x256 [ 9200.142026] ? do_raw_spin_unlock+0x75/0x190 [ 9200.142031] ? _raw_spin_unlock+0x12/0x30 [ 9200.142036] do_sys_openat2+0x2b4/0x410 [ 9200.142041] do_sys_open+0x73/0xa0 [ 9200.142046] __x64_sys_openat+0x24/0x30 [ 9200.142050] do_syscall_64+0xc1/0x440 [ 9200.142057] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9200.142063] RIP: 0033:0x7f53425a5332 [ 9200.142071] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9200.142075] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9200.142080] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9200.142083] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9200.142086] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9200.142088] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9200.142091] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9200.768287] obd_mod_free+0x46/0x140 [ptlrpc] [ 9200.775967] mdc_commit_open+0x91/0xb0 [mdc] [ 9200.778377] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9200.781782] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9200.785022] after_reply+0xe2c/0x1780 [ptlrpc] [ 9200.788001] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9200.791797] ? __next_timer_interrupt+0x160/0x160 [ 9200.797472] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9200.799280] ? do_wait_intr+0xf0/0xf0 [ 9200.807654] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9200.814821] kthread+0x1d1/0x200 [ 9200.815981] ? set_kthread_struct+0x70/0x70 [ 9200.823284] ret_from_fork+0x1f/0x30 [ 9204.064609] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 9204.072022] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9204.144343] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9204.167326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9204.182325] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9204.190102] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9204.218199] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9204.230651] RAX: 0000000000004789 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9204.243130] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9204.255782] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9204.267600] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9204.278007] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9204.292187] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9204.306314] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9204.313636] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9204.326118] Call Trace: [ 9204.331786] [ 9204.334562] ? show_regs.cold.9+0x22/0x2f [ 9204.340705] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9204.348585] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9204.352600] ? watchdog+0x40/0x40 [ 9204.357311] ? hrtimer_interrupt+0x128/0x2c0 [ 9204.365175] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9204.368502] ? apic_timer_interrupt+0xf/0x20 [ 9204.374468] [ 9204.377320] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9204.385889] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9204.393134] do_raw_spin_lock+0x177/0x190 [ 9204.398296] _raw_spin_lock+0x21/0x30 [ 9204.406675] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9204.413822] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9204.419038] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9204.428505] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9204.438022] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9204.444672] ? woken_wake_function+0x30/0x30 [ 9204.448561] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9204.457153] kthread+0x1d1/0x200 [ 9204.461742] ? set_kthread_struct+0x70/0x70 [ 9204.470246] ret_from_fork+0x1f/0x30 [ 9216.055956] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 9216.060926] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9216.106745] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9216.121997] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9216.128496] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9216.134746] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9216.139095] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9216.150079] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9216.155072] RAX: 0000000000003a9e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9216.169073] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9216.177697] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9216.186702] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9216.190158] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9216.206491] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9216.218677] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9216.233825] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9216.251874] Call Trace: [ 9216.252740] [ 9216.259774] ? show_regs.cold.9+0x22/0x2f [ 9216.264611] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9216.269679] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9216.273927] ? watchdog+0x40/0x40 [ 9216.279092] ? hrtimer_interrupt+0x128/0x2c0 [ 9216.288060] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9216.291062] ? apic_timer_interrupt+0xf/0x20 [ 9216.297493] [ 9216.299955] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9216.303122] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9216.311608] ? newidle_balance.isra.93+0x207/0x580 [ 9216.315091] do_raw_spin_lock+0x177/0x190 [ 9216.318511] _raw_spin_lock+0x21/0x30 [ 9216.319963] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9216.326668] process_one_work+0x2c8/0x700 [ 9216.329792] worker_thread+0x296/0x6e0 [ 9216.332768] ? rescuer_thread+0x570/0x570 [ 9216.334135] kthread+0x1d1/0x200 [ 9216.337909] ? set_kthread_struct+0x70/0x70 [ 9216.339215] ret_from_fork+0x1f/0x30 [ 9221.088147] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6583s! [ 9221.093903] Showing busy workqueues and worker pools: [ 9221.096931] workqueue events: flags=0x0 [ 9221.099670] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9221.103610] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9221.112433] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9221.115413] pending: cpu_latency_work [ptlrpc] [ 9221.118081] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9221.121551] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9221.126526] workqueue events_long: flags=0x0 [ 9221.128778] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9221.131984] pending: lu_objects_flush [obdclass] [ 9221.135426] workqueue events_unbound: flags=0x2 [ 9221.137447] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9221.140608] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9221.143601] workqueue events_power_efficient: flags=0x80 [ 9221.146484] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9221.149447] pending: check_lifetime, do_cache_clean [ 9221.151885] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9221.155763] pending: neigh_periodic_work, neigh_periodic_work [ 9221.158660] workqueue mm_percpu_wq: flags=0x8 [ 9221.160795] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9221.164002] pending: vmstat_update, lru_add_drain_per_cpu [ 9221.166834] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9221.169939] pending: vmstat_update, lru_add_drain_per_cpu [ 9221.172623] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9221.176016] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9221.179631] workqueue xprtiod: flags=0xa [ 9221.181635] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9221.184641] pending: xprt_autoclose, xs_error_handle [ 9221.187394] workqueue ipv6_addrconf: flags=0x40008 [ 9221.189527] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9221.192519] pending: addrconf_verify_work [ 9221.195036] workqueue knbd0-recv: flags=0x1a [ 9221.197440] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9221.200963] in-flight: 651:recv_work [ 9221.202947] workqueue ptlrpc_pinger: flags=0x2 [ 9221.205208] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9221.208214] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9221.211558] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6584s workers=4 idle: 15979 31824 8 [ 9221.215119] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9211s workers=2 idle: 654 [ 9228.056098] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 9228.059403] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 9228.060895] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9228.060999] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9228.061007] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9228.061011] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9228.061031] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9228.061036] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9228.061041] RAX: 0000000000004382 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9228.061044] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9228.061047] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9228.061049] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9228.061052] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9228.061056] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9228.061060] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9228.061062] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9228.061071] Call Trace: [ 9228.061077] [ 9228.061084] ? show_regs.cold.9+0x22/0x2f [ 9228.061097] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9228.061105] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9228.061109] ? watchdog+0x40/0x40 [ 9228.061115] ? hrtimer_interrupt+0x128/0x2c0 [ 9228.061119] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9228.061130] ? apic_timer_interrupt+0xf/0x20 [ 9228.061135] [ 9228.061139] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9228.061147] do_raw_spin_lock+0x177/0x190 [ 9228.061152] _raw_spin_lock+0x21/0x30 [ 9228.061157] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9228.062428] ? do_raw_spin_unlock+0x75/0x190 [ 9228.062447] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9228.062660] ? do_wait_intr+0xf0/0xf0 [ 9228.062670] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9228.067760] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9228.067840] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9228.067849] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9228.067852] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9228.067869] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9228.067874] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9228.067878] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9228.067881] RDX: 0000000000000ae5 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9228.067883] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9228.067885] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9228.067888] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9228.067891] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9228.067894] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9228.067896] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9228.067905] Call Trace: [ 9228.067913] [ 9228.067921] ? show_regs.cold.9+0x22/0x2f [ 9228.102031] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9228.102546] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9228.102784] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9228.103487] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9228.103545] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9228.112855] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9228.112880] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9228.112884] ? watchdog+0x40/0x40 [ 9228.112889] ? hrtimer_interrupt+0x128/0x2c0 [ 9228.112893] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9228.112904] ? apic_timer_interrupt+0xf/0x20 [ 9228.112909] [ 9228.112914] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9228.112921] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9228.112925] do_raw_spin_lock+0x177/0x190 [ 9228.113050] _raw_spin_lock+0x21/0x30 [ 9228.113058] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9228.115842] obd_mod_free+0x46/0x140 [ptlrpc] [ 9228.116886] mdc_commit_open+0x91/0xb0 [mdc] [ 9228.116996] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9228.117200] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9228.117468] after_reply+0xe2c/0x1780 [ptlrpc] [ 9228.117689] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9228.117886] ? __next_timer_interrupt+0x160/0x160 [ 9228.117900] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9228.118364] ? do_wait_intr+0xf0/0xf0 [ 9228.118375] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9228.118571] kthread+0x1d1/0x200 [ 9228.118578] ? set_kthread_struct+0x70/0x70 [ 9228.118581] ret_from_fork+0x1f/0x30 [ 9228.570200] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9228.573959] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9228.579205] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9228.581373] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9228.583902] lmv_intent_open+0x338/0x1150 [lmv] [ 9228.585688] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9228.588714] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9228.595823] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9228.597442] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9228.600427] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9228.603014] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9228.605545] ? do_raw_spin_unlock+0x75/0x190 [ 9228.611317] ? _raw_spin_unlock+0x12/0x30 [ 9228.612891] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9228.617905] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9228.620404] ll_atomic_open+0x36a/0xd70 [lustre] [ 9228.623222] lookup_open+0x356/0xb40 [ 9228.625297] do_last+0x313/0xfc0 [ 9228.628687] ? nd_jump_root+0xe5/0x160 [ 9228.632523] ? path_init+0x437/0x520 [ 9228.633493] path_openat+0xf7/0x500 [ 9228.635375] ? do_raw_spin_unlock+0x75/0x190 [ 9228.640997] do_filp_open+0x99/0x140 [ 9228.643027] ? getname_flags+0x6e/0x330 [ 9228.653393] ? __check_object_size+0xff/0x256 [ 9228.654801] ? do_raw_spin_unlock+0x75/0x190 [ 9228.661834] ? _raw_spin_unlock+0x12/0x30 [ 9228.663556] do_sys_openat2+0x2b4/0x410 [ 9228.672808] do_sys_open+0x73/0xa0 [ 9228.678722] __x64_sys_openat+0x24/0x30 [ 9228.683334] do_syscall_64+0xc1/0x440 [ 9228.686036] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9228.687623] RIP: 0033:0x7f53425a5332 [ 9228.695681] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9228.710606] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9228.713986] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9228.716490] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9228.719200] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9228.725761] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9228.728691] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9232.058263] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 9232.060989] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9232.081729] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9232.089385] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9232.095958] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9232.099819] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9232.113387] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9232.116013] RAX: 0000000000007386 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9232.119374] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9232.122507] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9232.125547] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9232.128975] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9232.132553] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9232.137404] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9232.139991] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9232.148519] Call Trace: [ 9232.150748] [ 9232.151585] ? show_regs.cold.9+0x22/0x2f [ 9232.153640] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9232.155785] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9232.157870] ? watchdog+0x40/0x40 [ 9232.159567] ? hrtimer_interrupt+0x128/0x2c0 [ 9232.162593] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9232.166338] ? apic_timer_interrupt+0xf/0x20 [ 9232.172570] [ 9232.174014] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9232.177646] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9232.181395] do_raw_spin_lock+0x177/0x190 [ 9232.183271] _raw_spin_lock+0x21/0x30 [ 9232.185074] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9232.187378] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9232.190016] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9232.192519] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9232.194700] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9232.197200] ? woken_wake_function+0x30/0x30 [ 9232.199338] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9232.201805] kthread+0x1d1/0x200 [ 9232.203601] ? set_kthread_struct+0x70/0x70 [ 9232.205499] ret_from_fork+0x1f/0x30 [ 9248.052267] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 9248.056543] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9248.080758] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9248.094999] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9248.102483] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9248.106255] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9248.108955] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9248.127830] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9248.135339] RAX: 00000000000023e4 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9248.139358] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9248.142942] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9248.146301] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9248.149333] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9248.152523] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9248.155362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9248.158073] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9248.161023] Call Trace: [ 9248.162180] [ 9248.163025] ? show_regs.cold.9+0x22/0x2f [ 9248.165021] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9248.167130] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9248.169223] ? watchdog+0x40/0x40 [ 9248.170760] ? hrtimer_interrupt+0x128/0x2c0 [ 9248.172728] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9248.174851] ? apic_timer_interrupt+0xf/0x20 [ 9248.176879] [ 9248.177833] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9248.180636] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9248.182918] ? newidle_balance.isra.93+0x207/0x580 [ 9248.185145] do_raw_spin_lock+0x177/0x190 [ 9248.186922] _raw_spin_lock+0x21/0x30 [ 9248.188752] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9248.191766] process_one_work+0x2c8/0x700 [ 9248.193264] worker_thread+0x296/0x6e0 [ 9248.194727] ? rescuer_thread+0x570/0x570 [ 9248.196362] kthread+0x1d1/0x200 [ 9248.198971] ? set_kthread_struct+0x70/0x70 [ 9248.200888] ret_from_fork+0x1f/0x30 [ 9251.295109] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6614s! [ 9251.298355] Showing busy workqueues and worker pools: [ 9251.299653] workqueue events: flags=0x0 [ 9251.301974] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9251.307226] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9251.312974] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9251.315453] pending: cpu_latency_work [ptlrpc] [ 9251.318418] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9251.321146] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9251.325466] workqueue events_long: flags=0x0 [ 9251.327471] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9251.330671] pending: lu_objects_flush [obdclass] [ 9251.333358] workqueue events_unbound: flags=0x2 [ 9251.335513] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9251.339975] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9251.342214] workqueue events_power_efficient: flags=0x80 [ 9251.346913] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9251.349717] pending: check_lifetime, do_cache_clean [ 9251.351419] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9251.355368] pending: neigh_periodic_work, neigh_periodic_work [ 9251.358210] workqueue mm_percpu_wq: flags=0x8 [ 9251.360253] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9251.363740] pending: vmstat_update, lru_add_drain_per_cpu [ 9251.366768] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9251.372289] pending: vmstat_update, lru_add_drain_per_cpu [ 9251.375076] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9251.379815] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9251.382806] workqueue xprtiod: flags=0xa [ 9251.384844] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9251.388703] pending: xprt_autoclose, xs_error_handle [ 9251.392793] workqueue ipv6_addrconf: flags=0x40008 [ 9251.399070] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9251.403102] pending: addrconf_verify_work [ 9251.405931] workqueue knbd0-recv: flags=0x1a [ 9251.407705] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9251.410444] in-flight: 651:recv_work [ 9251.413324] workqueue ptlrpc_pinger: flags=0x2 [ 9251.418602] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9251.422802] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9251.427075] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6614s workers=4 idle: 15979 31824 8 [ 9251.432374] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9241s workers=2 idle: 654 [ 9256.057096] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9256.057876] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9256.063552] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9256.063638] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9256.063648] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9256.063651] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9256.063669] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9256.063674] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9256.063679] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9256.063682] RDX: 00000000000067fd RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9256.063685] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9256.063687] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9256.063689] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9256.063694] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9256.063698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9256.063701] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9256.063731] Call Trace: [ 9256.063743] [ 9256.063754] ? show_regs.cold.9+0x22/0x2f [ 9256.063774] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9256.063788] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9256.063797] ? watchdog+0x40/0x40 [ 9256.063806] ? hrtimer_interrupt+0x128/0x2c0 [ 9256.063814] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9256.063835] ? apic_timer_interrupt+0xf/0x20 [ 9256.063848] [ 9256.063853] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9256.063866] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9256.063876] do_raw_spin_lock+0x177/0x190 [ 9256.063888] _raw_spin_lock+0x21/0x30 [ 9256.063903] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9256.072615] Modules linked in: [ 9256.106815] obd_mod_free+0x46/0x140 [ptlrpc] [ 9256.107122] mdc_commit_open+0x91/0xb0 [mdc] [ 9256.107166] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9256.107361] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9256.114326] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9256.114387] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9256.114393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9256.114395] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9256.114412] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9256.114417] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9256.114421] RAX: 000000000000743a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9256.114424] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9256.114427] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9256.114429] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9256.114432] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9256.114436] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9256.114439] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9256.114441] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9256.116247] Call Trace: [ 9256.116254] [ 9256.116260] ? show_regs.cold.9+0x22/0x2f [ 9256.116274] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9256.116281] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9256.116285] ? watchdog+0x40/0x40 [ 9256.116289] ? hrtimer_interrupt+0x128/0x2c0 [ 9256.116292] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9256.116298] ? apic_timer_interrupt+0xf/0x20 [ 9256.116302] [ 9256.116304] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9256.116312] do_raw_spin_lock+0x177/0x190 [ 9256.116315] _raw_spin_lock+0x21/0x30 [ 9256.116319] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9256.117978] ? do_raw_spin_unlock+0x75/0x190 [ 9256.120715] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9256.134258] after_reply+0xe2c/0x1780 [ptlrpc] [ 9256.134680] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9256.135025] ? __next_timer_interrupt+0x160/0x160 [ 9256.135038] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9256.135246] ? do_wait_intr+0xf0/0xf0 [ 9256.135255] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9256.141350] ? do_wait_intr+0xf0/0xf0 [ 9256.141639] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9256.143659] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9256.143913] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9256.144829] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9256.145158] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9256.166265] kthread+0x1d1/0x200 [ 9256.166284] ? set_kthread_struct+0x70/0x70 [ 9256.166287] ret_from_fork+0x1f/0x30 [ 9256.347623] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9256.349338] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9256.351009] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9256.353057] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9256.355656] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9256.357922] lmv_intent_open+0x338/0x1150 [lmv] [ 9256.359337] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9256.361342] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9256.362883] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9256.364179] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9256.365580] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9256.367360] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9256.368836] ? do_raw_spin_unlock+0x75/0x190 [ 9256.369955] ? _raw_spin_unlock+0x12/0x30 [ 9256.371204] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9256.372746] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9256.374143] ll_atomic_open+0x36a/0xd70 [lustre] [ 9256.375616] lookup_open+0x356/0xb40 [ 9256.376671] do_last+0x313/0xfc0 [ 9256.378007] ? nd_jump_root+0xe5/0x160 [ 9256.379843] ? path_init+0x437/0x520 [ 9256.381275] path_openat+0xf7/0x500 [ 9256.382722] ? do_raw_spin_unlock+0x75/0x190 [ 9256.384316] do_filp_open+0x99/0x140 [ 9256.385861] ? getname_flags+0x6e/0x330 [ 9256.387186] ? __check_object_size+0xff/0x256 [ 9256.388857] ? do_raw_spin_unlock+0x75/0x190 [ 9256.390617] ? _raw_spin_unlock+0x12/0x30 [ 9256.392114] do_sys_openat2+0x2b4/0x410 [ 9256.393725] do_sys_open+0x73/0xa0 [ 9256.395308] __x64_sys_openat+0x24/0x30 [ 9256.396930] do_syscall_64+0xc1/0x440 [ 9256.398283] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9256.400349] RIP: 0033:0x7f53425a5332 [ 9256.401828] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9256.408968] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9256.412114] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9256.414846] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9256.417127] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9256.419062] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9256.421576] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9260.060198] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 9260.067787] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9260.109642] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9260.117118] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9260.124010] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9260.131237] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9260.152287] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9260.160910] RAX: 000000000000735e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9260.171552] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9260.179746] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9260.187414] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9260.193723] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9260.198285] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9260.205922] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9260.213926] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9260.224296] Call Trace: [ 9260.225236] [ 9260.226033] ? show_regs.cold.9+0x22/0x2f [ 9260.228103] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9260.230917] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9260.233874] ? watchdog+0x40/0x40 [ 9260.237310] ? hrtimer_interrupt+0x128/0x2c0 [ 9260.244702] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9260.246809] ? apic_timer_interrupt+0xf/0x20 [ 9260.249228] [ 9260.250283] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9260.252922] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9260.255758] do_raw_spin_lock+0x177/0x190 [ 9260.257341] _raw_spin_lock+0x21/0x30 [ 9260.258441] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9260.261874] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9260.265115] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9260.268661] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9260.274345] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9260.277020] ? woken_wake_function+0x30/0x30 [ 9260.278898] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9260.281098] kthread+0x1d1/0x200 [ 9260.283967] ? set_kthread_struct+0x70/0x70 [ 9260.286150] ret_from_fork+0x1f/0x30 [ 9276.055602] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 9276.064999] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9276.123182] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9276.132656] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9276.143829] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9276.155020] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9276.162664] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9276.193281] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 9276.208243] RAX: 0000000000006c20 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9276.216969] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9276.226529] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9276.232677] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9276.237697] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9276.246174] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9276.252555] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9276.257467] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9276.264697] Call Trace: [ 9276.265406] [ 9276.266057] ? show_regs.cold.9+0x22/0x2f [ 9276.272473] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9276.278055] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9276.286626] ? watchdog+0x40/0x40 [ 9276.287829] ? hrtimer_interrupt+0x128/0x2c0 [ 9276.292241] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9276.295686] ? apic_timer_interrupt+0xf/0x20 [ 9276.298724] [ 9276.299660] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9276.301892] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9276.303737] ? newidle_balance.isra.93+0x207/0x580 [ 9276.305339] do_raw_spin_lock+0x177/0x190 [ 9276.310385] _raw_spin_lock+0x21/0x30 [ 9276.312018] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9276.314107] process_one_work+0x2c8/0x700 [ 9276.317176] worker_thread+0x296/0x6e0 [ 9276.318936] ? rescuer_thread+0x570/0x570 [ 9276.320810] kthread+0x1d1/0x200 [ 9276.323204] ? set_kthread_struct+0x70/0x70 [ 9276.324974] ret_from_fork+0x1f/0x30 [ 9281.506779] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6644s! [ 9281.515759] Showing busy workqueues and worker pools: [ 9281.518772] workqueue events: flags=0x0 [ 9281.524407] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9281.528894] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9281.540752] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9281.546140] pending: cpu_latency_work [ptlrpc] [ 9281.550368] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9281.553151] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9281.562025] workqueue events_long: flags=0x0 [ 9281.563821] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9281.572733] pending: lu_objects_flush [obdclass] [ 9281.577726] workqueue events_unbound: flags=0x2 [ 9281.583226] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9281.588345] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9281.590929] workqueue events_power_efficient: flags=0x80 [ 9281.597861] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9281.603924] pending: check_lifetime, do_cache_clean [ 9281.606214] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9281.611979] pending: neigh_periodic_work, neigh_periodic_work [ 9281.614989] workqueue mm_percpu_wq: flags=0x8 [ 9281.617320] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9281.620846] pending: vmstat_update, lru_add_drain_per_cpu [ 9281.623853] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9281.627422] pending: vmstat_update, lru_add_drain_per_cpu [ 9281.631221] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9281.642472] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9281.647257] workqueue xprtiod: flags=0xa [ 9281.649705] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9281.652964] pending: xprt_autoclose, xs_error_handle [ 9281.657686] workqueue ipv6_addrconf: flags=0x40008 [ 9281.662466] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9281.669940] pending: addrconf_verify_work [ 9281.679131] workqueue knbd0-recv: flags=0x1a [ 9281.684480] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9281.688893] in-flight: 651:recv_work [ 9281.690526] workqueue ptlrpc_pinger: flags=0x2 [ 9281.695827] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9281.700541] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9281.707720] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6644s workers=4 idle: 15979 31824 8 [ 9281.711882] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9272s workers=2 idle: 654 [ 9284.059071] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9284.059107] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9284.067583] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9284.067670] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9284.067680] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9284.067683] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9284.067699] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9284.067704] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9284.067709] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9284.067711] RDX: 00000000000048aa RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9284.067713] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9284.067716] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9284.067718] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9284.067721] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9284.067723] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9284.067726] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9284.067735] Call Trace: [ 9284.067742] [ 9284.067750] ? show_regs.cold.9+0x22/0x2f [ 9284.067760] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9284.067768] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9284.067772] ? watchdog+0x40/0x40 [ 9284.067776] ? hrtimer_interrupt+0x128/0x2c0 [ 9284.067780] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9284.067789] ? apic_timer_interrupt+0xf/0x20 [ 9284.067794] [ 9284.067797] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9284.067801] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9284.067806] do_raw_spin_lock+0x177/0x190 [ 9284.067810] _raw_spin_lock+0x21/0x30 [ 9284.067816] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9284.068337] obd_mod_free+0x46/0x140 [ptlrpc] [ 9284.068538] mdc_commit_open+0x91/0xb0 [mdc] [ 9284.068579] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9284.068774] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9284.068986] after_reply+0xe2c/0x1780 [ptlrpc] [ 9284.069417] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9284.069624] ? __next_timer_interrupt+0x160/0x160 [ 9284.069636] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9284.069847] ? do_wait_intr+0xf0/0xf0 [ 9284.069857] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9284.070384] kthread+0x1d1/0x200 [ 9284.070390] ? set_kthread_struct+0x70/0x70 [ 9284.070394] ret_from_fork+0x1f/0x30 [ 9284.429149] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9284.482487] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9284.490864] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9284.505893] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9284.511420] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9284.522903] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9284.527048] RAX: 0000000000007488 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9284.531016] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9284.534615] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9284.539849] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9284.544670] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9284.548028] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9284.551679] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9284.554102] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9284.565725] Call Trace: [ 9284.567576] [ 9284.570526] ? show_regs.cold.9+0x22/0x2f [ 9284.574058] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9284.578589] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9284.582040] ? watchdog+0x40/0x40 [ 9284.583688] ? hrtimer_interrupt+0x128/0x2c0 [ 9284.595614] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9284.601616] ? apic_timer_interrupt+0xf/0x20 [ 9284.605389] [ 9284.606139] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9284.607940] do_raw_spin_lock+0x177/0x190 [ 9284.626285] _raw_spin_lock+0x21/0x30 [ 9284.628756] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9284.630756] ? do_raw_spin_unlock+0x75/0x190 [ 9284.642110] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9284.646811] ? do_wait_intr+0xf0/0xf0 [ 9284.649658] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9284.653879] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9284.656551] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9284.663764] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9284.669758] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9284.671436] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9284.672634] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9284.673812] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9284.681841] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9284.688957] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9284.692912] lmv_intent_open+0x338/0x1150 [lmv] [ 9284.698576] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9284.703872] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9284.705827] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9284.710409] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9284.711970] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9284.720633] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9284.725304] ? do_raw_spin_unlock+0x75/0x190 [ 9284.729811] ? _raw_spin_unlock+0x12/0x30 [ 9284.731357] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9284.734299] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9284.737822] ll_atomic_open+0x36a/0xd70 [lustre] [ 9284.740293] lookup_open+0x356/0xb40 [ 9284.742083] do_last+0x313/0xfc0 [ 9284.746692] ? nd_jump_root+0xe5/0x160 [ 9284.748709] ? path_init+0x437/0x520 [ 9284.750319] path_openat+0xf7/0x500 [ 9284.753092] ? do_raw_spin_unlock+0x75/0x190 [ 9284.757726] do_filp_open+0x99/0x140 [ 9284.761705] ? getname_flags+0x6e/0x330 [ 9284.766096] ? __check_object_size+0xff/0x256 [ 9284.767892] ? do_raw_spin_unlock+0x75/0x190 [ 9284.771594] ? _raw_spin_unlock+0x12/0x30 [ 9284.773316] do_sys_openat2+0x2b4/0x410 [ 9284.775334] do_sys_open+0x73/0xa0 [ 9284.778824] __x64_sys_openat+0x24/0x30 [ 9284.783505] do_syscall_64+0xc1/0x440 [ 9284.786004] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9284.788873] RIP: 0033:0x7f53425a5332 [ 9284.790834] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9284.802827] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9284.810415] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9284.813469] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9284.817034] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9284.821269] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9284.826719] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9288.060359] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 9288.075938] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9288.158256] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9288.169418] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9288.177738] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9288.183097] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9288.198376] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9288.205470] RAX: 0000000000006f09 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9288.211704] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9288.222640] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9288.228277] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9288.236845] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9288.243407] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9288.251241] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9288.257705] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9288.264326] Call Trace: [ 9288.266641] [ 9288.268766] ? show_regs.cold.9+0x22/0x2f [ 9288.271660] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9288.277783] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9288.279403] ? watchdog+0x40/0x40 [ 9288.281727] ? hrtimer_interrupt+0x128/0x2c0 [ 9288.285562] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9288.289402] ? apic_timer_interrupt+0xf/0x20 [ 9288.293209] [ 9288.295193] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9288.299459] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9288.305027] do_raw_spin_lock+0x177/0x190 [ 9288.308604] _raw_spin_lock+0x21/0x30 [ 9288.313214] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9288.317087] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9288.321860] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9288.326339] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9288.328772] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9288.331319] ? woken_wake_function+0x30/0x30 [ 9288.333219] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9288.335822] kthread+0x1d1/0x200 [ 9288.337234] ? set_kthread_struct+0x70/0x70 [ 9288.338779] ret_from_fork+0x1f/0x30 [ 9308.052737] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 9308.083132] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9308.118996] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9308.130418] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9308.139065] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9308.144358] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9308.147980] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9308.166822] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9308.175857] RAX: 0000000000007c07 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9308.186717] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9308.193231] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9308.204585] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9308.217057] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9308.220240] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9308.227806] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9308.235899] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9308.244802] Call Trace: [ 9308.247817] [ 9308.251686] ? show_regs.cold.9+0x22/0x2f [ 9308.256370] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9308.260854] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9308.265896] ? watchdog+0x40/0x40 [ 9308.267878] ? hrtimer_interrupt+0x128/0x2c0 [ 9308.271750] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9308.273993] ? apic_timer_interrupt+0xf/0x20 [ 9308.276260] [ 9308.276914] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9308.283001] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9308.288963] ? newidle_balance.isra.93+0x207/0x580 [ 9308.291575] do_raw_spin_lock+0x177/0x190 [ 9308.298892] _raw_spin_lock+0x21/0x30 [ 9308.303799] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9308.309901] process_one_work+0x2c8/0x700 [ 9308.317863] worker_thread+0x296/0x6e0 [ 9308.323336] ? rescuer_thread+0x570/0x570 [ 9308.329881] kthread+0x1d1/0x200 [ 9308.334723] ? set_kthread_struct+0x70/0x70 [ 9308.344410] ret_from_fork+0x1f/0x30 [ 9312.055185] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9312.057120] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9312.057806] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9312.057878] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9312.057886] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9312.075889] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9312.075964] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9312.075972] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9312.075974] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9312.075989] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9312.075993] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9312.075997] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9312.075999] RDX: 000000000000340a RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9312.076002] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9312.076004] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9312.076006] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9312.076009] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9312.076012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9312.076014] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9312.076022] Call Trace: [ 9312.076026] [ 9312.076033] ? show_regs.cold.9+0x22/0x2f [ 9312.076043] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9312.076050] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9312.076054] ? watchdog+0x40/0x40 [ 9312.076059] ? hrtimer_interrupt+0x128/0x2c0 [ 9312.076224] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9312.076233] ? apic_timer_interrupt+0xf/0x20 [ 9312.076237] [ 9312.076240] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9312.076245] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9312.076249] do_raw_spin_lock+0x177/0x190 [ 9312.076255] _raw_spin_lock+0x21/0x30 [ 9312.076261] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9312.076630] obd_mod_free+0x46/0x140 [ptlrpc] [ 9312.076911] mdc_commit_open+0x91/0xb0 [mdc] [ 9312.076954] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9312.078003] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9312.126013] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9312.126037] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9312.126042] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9312.126048] RAX: 0000000000004708 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9312.126051] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9312.126053] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9312.126055] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9312.126058] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9312.126061] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9312.126063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9312.126065] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9312.126074] Call Trace: [ 9312.126081] [ 9312.126087] ? show_regs.cold.9+0x22/0x2f [ 9312.126097] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9312.126106] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9312.126110] ? watchdog+0x40/0x40 [ 9312.126114] ? hrtimer_interrupt+0x128/0x2c0 [ 9312.126118] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9312.126127] ? apic_timer_interrupt+0xf/0x20 [ 9312.126132] [ 9312.126135] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9312.126143] do_raw_spin_lock+0x177/0x190 [ 9312.126148] _raw_spin_lock+0x21/0x30 [ 9312.126155] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9312.126433] ? do_raw_spin_unlock+0x75/0x190 [ 9312.126439] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9312.127772] ? do_wait_intr+0xf0/0xf0 [ 9312.127782] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9312.128145] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9312.128360] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9312.128744] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9312.128859] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9312.129612] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9312.129642] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9312.129667] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9312.129763] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9312.131154] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9312.131202] lmv_intent_open+0x338/0x1150 [lmv] [ 9312.131235] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9312.131349] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9312.131434] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9312.131457] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9312.133635] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9312.133717] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9312.133810] ? do_raw_spin_unlock+0x75/0x190 [ 9312.133826] ? _raw_spin_unlock+0x12/0x30 [ 9312.133834] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9312.133868] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9312.133889] ll_atomic_open+0x36a/0xd70 [lustre] [ 9312.134177] lookup_open+0x356/0xb40 [ 9312.134187] do_last+0x313/0xfc0 [ 9312.134193] ? nd_jump_root+0xe5/0x160 [ 9312.134198] ? path_init+0x437/0x520 [ 9312.134202] path_openat+0xf7/0x500 [ 9312.134207] ? do_raw_spin_unlock+0x75/0x190 [ 9312.134211] do_filp_open+0x99/0x140 [ 9312.134218] ? getname_flags+0x6e/0x330 [ 9312.134222] ? __check_object_size+0xff/0x256 [ 9312.134230] ? do_raw_spin_unlock+0x75/0x190 [ 9312.134234] ? _raw_spin_unlock+0x12/0x30 [ 9312.134239] do_sys_openat2+0x2b4/0x410 [ 9312.134244] do_sys_open+0x73/0xa0 [ 9312.134248] __x64_sys_openat+0x24/0x30 [ 9312.134252] do_syscall_64+0xc1/0x440 [ 9312.134259] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9312.134265] RIP: 0033:0x7f53425a5332 [ 9312.134273] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9312.134278] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9312.134283] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9312.134286] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9312.134288] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9312.134291] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9312.134293] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9312.842704] after_reply+0xe2c/0x1780 [ptlrpc] [ 9312.844596] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9312.846715] ? __next_timer_interrupt+0x160/0x160 [ 9312.848542] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9312.850387] ? do_wait_intr+0xf0/0xf0 [ 9312.852286] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9312.855771] kthread+0x1d1/0x200 [ 9312.857692] ? set_kthread_struct+0x70/0x70 [ 9312.859961] ret_from_fork+0x1f/0x30 [ 9312.861977] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6675s! [ 9312.865344] Showing busy workqueues and worker pools: [ 9312.875660] workqueue events: flags=0x0 [ 9312.881294] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9312.887485] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9312.895287] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9312.901065] pending: cpu_latency_work [ptlrpc] [ 9312.904130] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9312.907280] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9312.918240] workqueue events_long: flags=0x0 [ 9312.920637] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9312.924771] pending: lu_objects_flush [obdclass] [ 9312.930594] workqueue events_unbound: flags=0x2 [ 9312.932122] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9312.938621] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9312.946146] workqueue events_power_efficient: flags=0x80 [ 9312.951653] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9312.955910] pending: check_lifetime, do_cache_clean [ 9312.957599] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9312.961093] pending: neigh_periodic_work, neigh_periodic_work [ 9312.963422] workqueue mm_percpu_wq: flags=0x8 [ 9312.967940] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9312.974027] pending: vmstat_update, lru_add_drain_per_cpu [ 9312.984163] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9312.999119] pending: vmstat_update, lru_add_drain_per_cpu [ 9313.005181] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9313.008471] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9313.012708] workqueue xprtiod: flags=0xa [ 9313.014901] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9313.017291] pending: xprt_autoclose, xs_error_handle [ 9313.019431] workqueue ipv6_addrconf: flags=0x40008 [ 9313.021297] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9313.023811] pending: addrconf_verify_work [ 9313.025362] workqueue knbd0-recv: flags=0x1a [ 9313.026991] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9313.029280] in-flight: 651:recv_work [ 9313.030826] workqueue ptlrpc_pinger: flags=0x2 [ 9313.032379] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9313.034603] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9313.036955] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6675s workers=4 idle: 15979 31824 8 [ 9313.039807] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9303s workers=2 idle: 654 [ 9316.061559] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 9316.069104] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9316.120100] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9316.132580] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9316.146596] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9316.156914] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9316.176177] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9316.184706] RAX: 0000000000004b8e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9316.193231] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9316.202263] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9316.205086] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9316.208946] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9316.214279] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9316.219949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9316.223102] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9316.233015] Call Trace: [ 9316.234179] [ 9316.235556] ? show_regs.cold.9+0x22/0x2f [ 9316.238152] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9316.240442] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9316.244052] ? watchdog+0x40/0x40 [ 9316.245333] ? hrtimer_interrupt+0x128/0x2c0 [ 9316.250718] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9316.255572] ? apic_timer_interrupt+0xf/0x20 [ 9316.266207] [ 9316.267291] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9316.275424] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9316.277229] do_raw_spin_lock+0x177/0x190 [ 9316.290711] _raw_spin_lock+0x21/0x30 [ 9316.293441] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9316.298360] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9316.306507] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9316.315485] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9316.317579] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9316.320785] ? woken_wake_function+0x30/0x30 [ 9316.323310] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9316.327566] kthread+0x1d1/0x200 [ 9316.330018] ? set_kthread_struct+0x70/0x70 [ 9316.334500] ret_from_fork+0x1f/0x30 [ 9340.052148] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 9340.055895] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9340.056844] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9340.056923] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9340.056930] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9340.056938] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9340.057353] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9340.057365] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9340.057370] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9340.057375] RAX: 0000000000005f76 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9340.057377] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9340.057379] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9340.057381] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9340.057383] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9340.057387] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9340.057390] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9340.057392] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9340.057402] Call Trace: [ 9340.057408] [ 9340.057413] ? show_regs.cold.9+0x22/0x2f [ 9340.057424] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9340.057431] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9340.057436] ? watchdog+0x40/0x40 [ 9340.057441] ? hrtimer_interrupt+0x128/0x2c0 [ 9340.057445] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9340.057454] ? apic_timer_interrupt+0xf/0x20 [ 9340.057458] [ 9340.057461] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9340.057466] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9340.057470] ? newidle_balance.isra.93+0x207/0x580 [ 9340.057475] do_raw_spin_lock+0x177/0x190 [ 9340.057480] _raw_spin_lock+0x21/0x30 [ 9340.057485] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9340.057697] process_one_work+0x2c8/0x700 [ 9340.057706] worker_thread+0x296/0x6e0 [ 9340.058075] ? rescuer_thread+0x570/0x570 [ 9340.059174] kthread+0x1d1/0x200 [ 9340.059179] ? set_kthread_struct+0x70/0x70 [ 9340.059184] ret_from_fork+0x1f/0x30 [ 9340.061155] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9340.074092] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9340.074166] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9340.074174] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9340.074176] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9340.074191] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9340.074195] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9340.074200] RAX: 0000000000002331 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9340.074203] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9340.074205] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9340.074207] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9340.074210] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9340.074213] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9340.074216] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9340.074218] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9340.074228] Call Trace: [ 9340.074234] [ 9340.074239] ? show_regs.cold.9+0x22/0x2f [ 9340.074249] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9340.074256] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9340.074260] ? watchdog+0x40/0x40 [ 9340.074265] ? hrtimer_interrupt+0x128/0x2c0 [ 9340.074269] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9340.074278] ? apic_timer_interrupt+0xf/0x20 [ 9340.074283] [ 9340.074285] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9340.074292] do_raw_spin_lock+0x177/0x190 [ 9340.074297] _raw_spin_lock+0x21/0x30 [ 9340.074302] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9340.121449] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9340.121531] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9340.121540] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9340.121542] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9340.121557] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9340.121562] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9340.121566] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9340.121569] RDX: 0000000000006ff5 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9340.121571] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9340.121574] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9340.121576] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9340.121579] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9340.121583] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9340.121585] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9340.121594] Call Trace: [ 9340.121599] [ 9340.121605] ? show_regs.cold.9+0x22/0x2f [ 9340.121614] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9340.121620] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9340.121624] ? watchdog+0x40/0x40 [ 9340.121628] ? hrtimer_interrupt+0x128/0x2c0 [ 9340.121633] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9340.121641] ? apic_timer_interrupt+0xf/0x20 [ 9340.121645] [ 9340.121648] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9340.121653] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9340.121657] do_raw_spin_lock+0x177/0x190 [ 9340.121662] _raw_spin_lock+0x21/0x30 [ 9340.121667] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9340.122591] obd_mod_free+0x46/0x140 [ptlrpc] [ 9340.122789] mdc_commit_open+0x91/0xb0 [mdc] [ 9340.122832] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9340.139716] ? do_raw_spin_unlock+0x75/0x190 [ 9340.139738] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9340.140052] ? do_wait_intr+0xf0/0xf0 [ 9340.140062] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9340.140268] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9340.140551] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9340.140773] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9340.141175] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9340.141213] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9340.141281] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9340.141314] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9340.141493] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9340.141960] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9340.142004] lmv_intent_open+0x338/0x1150 [lmv] [ 9340.142040] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9340.142135] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9340.142218] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9340.142241] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9340.142325] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9340.142402] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9340.142627] ? do_raw_spin_unlock+0x75/0x190 [ 9340.142635] ? _raw_spin_unlock+0x12/0x30 [ 9340.142642] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9340.142677] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9340.142698] ll_atomic_open+0x36a/0xd70 [lustre] [ 9340.142777] lookup_open+0x356/0xb40 [ 9340.142788] do_last+0x313/0xfc0 [ 9340.142793] ? nd_jump_root+0xe5/0x160 [ 9340.142797] ? path_init+0x437/0x520 [ 9340.142802] path_openat+0xf7/0x500 [ 9340.142806] ? do_raw_spin_unlock+0x75/0x190 [ 9340.142810] do_filp_open+0x99/0x140 [ 9340.142817] ? getname_flags+0x6e/0x330 [ 9340.142821] ? __check_object_size+0xff/0x256 [ 9340.142828] ? do_raw_spin_unlock+0x75/0x190 [ 9340.142831] ? _raw_spin_unlock+0x12/0x30 [ 9340.142835] do_sys_openat2+0x2b4/0x410 [ 9340.142839] do_sys_open+0x73/0xa0 [ 9340.142843] __x64_sys_openat+0x24/0x30 [ 9340.142846] do_syscall_64+0xc1/0x440 [ 9340.142853] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9340.142860] RIP: 0033:0x7f53425a5332 [ 9340.142869] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9340.142874] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9340.142878] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9340.142881] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9340.142883] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9340.142885] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9340.142886] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9340.865794] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9340.869322] after_reply+0xe2c/0x1780 [ptlrpc] [ 9340.874350] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9340.878946] ? __next_timer_interrupt+0x160/0x160 [ 9340.882313] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9340.885342] ? do_wait_intr+0xf0/0xf0 [ 9340.887421] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9340.890594] kthread+0x1d1/0x200 [ 9340.892067] ? set_kthread_struct+0x70/0x70 [ 9340.894519] ret_from_fork+0x1f/0x30 [ 9343.459951] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6706s! [ 9343.468402] Showing busy workqueues and worker pools: [ 9343.473596] workqueue events: flags=0x0 [ 9343.478313] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9343.491145] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9343.504871] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9343.519748] pending: cpu_latency_work [ptlrpc] [ 9343.527765] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9343.534315] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9343.548985] workqueue events_long: flags=0x0 [ 9343.556342] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9343.564672] pending: lu_objects_flush [obdclass] [ 9343.569835] workqueue events_unbound: flags=0x2 [ 9343.574634] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9343.580673] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9343.589705] workqueue events_power_efficient: flags=0x80 [ 9343.597802] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9343.603241] pending: check_lifetime, do_cache_clean [ 9343.612373] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9343.617593] pending: neigh_periodic_work, neigh_periodic_work [ 9343.620326] workqueue mm_percpu_wq: flags=0x8 [ 9343.622262] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9343.628588] pending: vmstat_update, lru_add_drain_per_cpu [ 9343.639702] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9343.647288] pending: vmstat_update, lru_add_drain_per_cpu [ 9343.652529] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9343.663118] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9343.668165] workqueue xprtiod: flags=0xa [ 9343.670105] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9343.677380] pending: xprt_autoclose, xs_error_handle [ 9343.680479] workqueue ipv6_addrconf: flags=0x40008 [ 9343.686616] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9343.694654] pending: addrconf_verify_work [ 9343.700578] workqueue knbd0-recv: flags=0x1a [ 9343.705692] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9343.715684] in-flight: 651:recv_work [ 9343.720423] workqueue ptlrpc_pinger: flags=0x2 [ 9343.723256] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9343.727636] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9343.731563] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6706s workers=4 idle: 15979 31824 8 [ 9343.735847] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9334s workers=2 idle: 654 [ 9344.058030] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 9344.069392] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9344.115450] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9344.124451] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9344.127658] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9344.132389] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9344.148823] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9344.155626] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9344.162399] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9344.169251] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9344.174509] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9344.180872] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9344.188706] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9344.195032] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9344.199435] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9344.205307] Call Trace: [ 9344.206810] [ 9344.207752] ? show_regs.cold.9+0x22/0x2f [ 9344.212011] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9344.213273] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9344.214352] ? watchdog+0x40/0x40 [ 9344.215779] ? hrtimer_interrupt+0x128/0x2c0 [ 9344.217618] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9344.221356] ? apic_timer_interrupt+0xf/0x20 [ 9344.224629] [ 9344.225815] ? native_safe_halt+0x2e/0x50 [ 9344.228739] kvm_wait+0xcd/0xe0 [ 9344.229837] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9344.231785] do_raw_spin_lock+0x177/0x190 [ 9344.233430] _raw_spin_lock+0x21/0x30 [ 9344.234963] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9344.237612] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9344.240101] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9344.243871] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9344.245513] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9344.249887] ? woken_wake_function+0x30/0x30 [ 9344.251453] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9344.253784] kthread+0x1d1/0x200 [ 9344.255084] ? set_kthread_struct+0x70/0x70 [ 9344.256696] ret_from_fork+0x1f/0x30 [ 9354.607023] rcu: INFO: rcu_sched self-detected stall on CPU [ 9354.609440] rcu: 2-....: (3956209 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 9354.613388] (t=7260181 jiffies g=274185 q=65702) [ 9354.615130] rcu: rcu_sched kthread starved for 6718547 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 9354.619475] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9354.623267] rcu: RCU grace-period kthread stack dump: [ 9354.628668] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9354.634653] Call Trace: [ 9354.635622] __schedule+0x351/0xcb0 [ 9354.637630] schedule+0xc0/0x180 [ 9354.638504] schedule_timeout+0xb4/0x190 [ 9354.639622] ? do_raw_spin_unlock+0x75/0x190 [ 9354.641181] ? __next_timer_interrupt+0x160/0x160 [ 9354.642934] ? prepare_to_swait_event+0xf3/0x140 [ 9354.644882] rcu_gp_kthread+0x8dd/0x15b0 [ 9354.646421] ? force_qs_rnp+0x2c0/0x2c0 [ 9354.647945] kthread+0x1d1/0x200 [ 9354.649142] ? set_kthread_struct+0x70/0x70 [ 9354.650643] ret_from_fork+0x1f/0x30 [ 9354.651998] rcu: Stack dump where RCU GP kthread last ran: [ 9354.653909] Sending NMI from CPU 2 to CPUs 3: [ 9354.656462] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [ 9354.659427] Sending NMI from CPU 2 to CPUs 1: [ 9354.661179] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 9354.662135] NMI backtrace for cpu 2 [ 9354.674489] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9354.683585] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9354.686918] Call Trace: [ 9354.688126] [ 9354.688838] ? dump_stack+0xbb/0x10e [ 9354.690871] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9354.695426] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9354.696685] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9354.698638] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9354.700556] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9354.702411] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9354.704373] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9354.706529] ? update_process_times+0x6d/0xc0 [ 9354.708464] ? tick_sched_handle.isra.18+0x27/0x90 [ 9354.710499] ? tick_sched_timer+0x53/0xb0 [ 9354.712143] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9354.714766] ? tick_sched_handle.isra.18+0x90/0x90 [ 9354.716805] ? hrtimer_interrupt+0x128/0x2c0 [ 9354.718466] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9354.720565] ? apic_timer_interrupt+0xf/0x20 [ 9354.722516] [ 9354.723436] ? native_safe_halt+0x2e/0x50 [ 9354.724978] ? kvm_wait+0xcd/0xe0 [ 9354.726404] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9354.728576] ? do_raw_spin_lock+0x177/0x190 [ 9354.730631] ? _raw_spin_lock+0x21/0x30 [ 9354.733665] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9354.742409] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9354.746384] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9354.751829] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9354.756641] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9354.761211] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9354.765461] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9354.767740] ? __next_timer_interrupt+0x160/0x160 [ 9354.769824] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9354.772028] ? do_wait_intr+0xf0/0xf0 [ 9354.773886] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9354.776173] ? kthread+0x1d1/0x200 [ 9354.777322] ? set_kthread_struct+0x70/0x70 [ 9354.778932] ? ret_from_fork+0x1f/0x30 [ 9368.055052] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 9368.068847] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9368.105991] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9368.114119] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9368.121030] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9368.129190] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9368.131462] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9368.148024] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9368.160872] RAX: 0000000000007c1f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9368.171256] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9368.176019] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9368.186752] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9368.194460] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9368.204768] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9368.211256] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9368.215951] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9368.220198] Call Trace: [ 9368.223340] [ 9368.226671] ? show_regs.cold.9+0x22/0x2f [ 9368.232666] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9368.236710] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9368.242466] ? watchdog+0x40/0x40 [ 9368.244409] ? hrtimer_interrupt+0x128/0x2c0 [ 9368.251262] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9368.254880] ? apic_timer_interrupt+0xf/0x20 [ 9368.258958] [ 9368.261836] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9368.263722] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9368.271413] ? newidle_balance.isra.93+0x207/0x580 [ 9368.274822] do_raw_spin_lock+0x177/0x190 [ 9368.276463] _raw_spin_lock+0x21/0x30 [ 9368.284210] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9368.293195] process_one_work+0x2c8/0x700 [ 9368.297811] worker_thread+0x296/0x6e0 [ 9368.299374] ? rescuer_thread+0x570/0x570 [ 9368.303209] kthread+0x1d1/0x200 [ 9368.304380] ? set_kthread_struct+0x70/0x70 [ 9368.316334] ret_from_fork+0x1f/0x30 [ 9374.186085] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6737s! [ 9374.193691] Showing busy workqueues and worker pools: [ 9374.196549] workqueue events: flags=0x0 [ 9374.202305] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9374.210802] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9374.217438] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9374.221266] pending: cpu_latency_work [ptlrpc] [ 9374.229416] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9374.237722] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9374.242398] workqueue events_long: flags=0x0 [ 9374.244871] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9374.248802] pending: lu_objects_flush [obdclass] [ 9374.254792] workqueue events_unbound: flags=0x2 [ 9374.265523] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9374.271946] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9374.276336] workqueue events_power_efficient: flags=0x80 [ 9374.283295] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9374.289506] pending: check_lifetime, do_cache_clean [ 9374.291283] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9374.296219] pending: neigh_periodic_work, neigh_periodic_work [ 9374.305949] workqueue mm_percpu_wq: flags=0x8 [ 9374.308467] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9374.320750] pending: vmstat_update, lru_add_drain_per_cpu [ 9374.330144] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9374.336307] pending: vmstat_update, lru_add_drain_per_cpu [ 9374.341265] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9374.346734] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9374.352981] workqueue xprtiod: flags=0xa [ 9374.355482] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9374.362338] pending: xprt_autoclose, xs_error_handle [ 9374.369804] workqueue ipv6_addrconf: flags=0x40008 [ 9374.375767] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9374.382358] pending: addrconf_verify_work [ 9374.386229] workqueue knbd0-recv: flags=0x1a [ 9374.391369] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9374.397273] in-flight: 651:recv_work [ 9374.399729] workqueue ptlrpc_pinger: flags=0x2 [ 9374.404803] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9374.410580] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9374.412852] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6737s workers=4 idle: 15979 31824 8 [ 9374.419533] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9364s workers=2 idle: 654 [ 9380.057034] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9380.058037] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 9380.058110] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9380.058119] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9380.058205] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9380.058211] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9380.058214] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9380.058230] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9380.058234] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9380.058239] RAX: 0000000000007f0d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9380.058242] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9380.058244] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9380.058246] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9380.058249] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9380.058253] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9380.058256] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9380.058259] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9380.058266] Call Trace: [ 9380.058280] [ 9380.058286] ? show_regs.cold.9+0x22/0x2f [ 9380.058296] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9380.058303] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9380.058307] ? watchdog+0x40/0x40 [ 9380.058312] ? hrtimer_interrupt+0x128/0x2c0 [ 9380.058316] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9380.058325] ? apic_timer_interrupt+0xf/0x20 [ 9380.058329] [ 9380.058332] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9380.058340] do_raw_spin_lock+0x177/0x190 [ 9380.058346] _raw_spin_lock+0x21/0x30 [ 9380.058472] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9380.058908] ? do_raw_spin_unlock+0x75/0x190 [ 9380.058914] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9380.059103] ? do_wait_intr+0xf0/0xf0 [ 9380.059114] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9380.059760] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9380.061073] Modules linked in: lustre(O) osp(O) [ 9380.061083] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9380.061972] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9380.063087] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9380.063134] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9380.063160] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9380.063184] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9380.063264] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9380.063860] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9380.063909] lmv_intent_open+0x338/0x1150 [lmv] [ 9380.063957] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9380.064060] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9380.064149] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9380.064173] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9380.064584] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9380.064655] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9380.064661] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9380.064664] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9380.064678] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9380.064682] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9380.064687] RAX: 000000000000677d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9380.064690] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9380.064693] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9380.064695] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9380.064698] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9380.064701] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9380.064705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9380.064707] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9380.064716] Call Trace: [ 9380.064721] [ 9380.064724] ? show_regs.cold.9+0x22/0x2f [ 9380.064735] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9380.064743] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9380.064747] ? watchdog+0x40/0x40 [ 9380.064750] ? hrtimer_interrupt+0x128/0x2c0 [ 9380.064754] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9380.064763] ? apic_timer_interrupt+0xf/0x20 [ 9380.064768] [ 9380.064771] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9380.064777] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9380.064782] do_raw_spin_lock+0x177/0x190 [ 9380.064787] _raw_spin_lock+0x21/0x30 [ 9380.064793] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9380.065190] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9380.065719] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9380.064269] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9380.066304] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9380.067042] ? do_raw_spin_unlock+0x75/0x190 [ 9380.067052] ? _raw_spin_unlock+0x12/0x30 [ 9380.067060] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9380.067096] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9380.067115] ll_atomic_open+0x36a/0xd70 [lustre] [ 9380.067211] lookup_open+0x356/0xb40 [ 9380.067222] do_last+0x313/0xfc0 [ 9380.067227] ? nd_jump_root+0xe5/0x160 [ 9380.067232] ? path_init+0x437/0x520 [ 9380.067237] path_openat+0xf7/0x500 [ 9380.067242] ? do_raw_spin_unlock+0x75/0x190 [ 9380.067247] do_filp_open+0x99/0x140 [ 9380.067254] ? getname_flags+0x6e/0x330 [ 9380.067258] ? __check_object_size+0xff/0x256 [ 9380.067266] ? do_raw_spin_unlock+0x75/0x190 [ 9380.067270] ? _raw_spin_unlock+0x12/0x30 [ 9380.067285] do_sys_openat2+0x2b4/0x410 [ 9380.067290] do_sys_open+0x73/0xa0 [ 9380.067295] __x64_sys_openat+0x24/0x30 [ 9380.067299] do_syscall_64+0xc1/0x440 [ 9380.067306] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9380.067313] RIP: 0033:0x7f53425a5332 [ 9380.067321] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9380.067325] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9380.067330] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9380.067333] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9380.067335] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9380.067338] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9380.067340] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9380.068761] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9380.068828] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9380.068833] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9380.068835] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9380.068855] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9380.069006] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9380.069011] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9380.069013] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9380.069015] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9380.069017] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9380.069020] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9380.069023] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9380.069026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9380.069029] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9380.069039] Call Trace: [ 9380.069045] [ 9380.069050] ? show_regs.cold.9+0x22/0x2f [ 9380.069062] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9380.069070] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9380.069074] ? watchdog+0x40/0x40 [ 9380.069078] ? hrtimer_interrupt+0x128/0x2c0 [ 9380.069083] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9380.069090] ? apic_timer_interrupt+0xf/0x20 [ 9380.109244] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9380.114797] [ 9380.114812] ? native_safe_halt+0x2e/0x50 [ 9380.114825] kvm_wait+0xcd/0xe0 [ 9380.114835] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9380.125658] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9380.145581] do_raw_spin_lock+0x177/0x190 [ 9380.145608] _raw_spin_lock+0x21/0x30 [ 9380.145619] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9380.151170] ? woken_wake_function+0x30/0x30 [ 9380.151191] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9380.151824] kthread+0x1d1/0x200 [ 9380.151831] ? set_kthread_struct+0x70/0x70 [ 9380.151836] ret_from_fork+0x1f/0x30 [ 9380.824868] obd_mod_free+0x46/0x140 [ptlrpc] [ 9380.827474] mdc_commit_open+0x91/0xb0 [mdc] [ 9380.829205] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9380.831927] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9380.834709] after_reply+0xe2c/0x1780 [ptlrpc] [ 9380.836845] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9380.839484] ? __next_timer_interrupt+0x160/0x160 [ 9380.841332] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9380.843387] ? do_wait_intr+0xf0/0xf0 [ 9380.844840] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9380.847211] kthread+0x1d1/0x200 [ 9380.848849] ? set_kthread_struct+0x70/0x70 [ 9380.850947] ret_from_fork+0x1f/0x30 [ 9400.052037] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 9400.054578] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9400.096200] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9400.107459] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9400.111092] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9400.113720] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9400.115740] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9400.127213] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9400.130608] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9400.133650] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9400.137184] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9400.140372] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9400.143532] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9400.146363] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9400.149759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9400.152354] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9400.155278] Call Trace: [ 9400.156084] [ 9400.156577] ? show_regs.cold.9+0x22/0x2f [ 9400.158457] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9400.160277] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9400.162193] ? watchdog+0x40/0x40 [ 9400.163616] ? hrtimer_interrupt+0x128/0x2c0 [ 9400.165569] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9400.167615] ? apic_timer_interrupt+0xf/0x20 [ 9400.169544] [ 9400.170470] ? native_safe_halt+0x2e/0x50 [ 9400.172485] kvm_wait+0xcd/0xe0 [ 9400.173794] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9400.176301] ? newidle_balance.isra.93+0x207/0x580 [ 9400.178588] do_raw_spin_lock+0x177/0x190 [ 9400.180493] _raw_spin_lock+0x21/0x30 [ 9400.182415] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9400.185050] process_one_work+0x2c8/0x700 [ 9400.187104] worker_thread+0x296/0x6e0 [ 9400.188661] ? rescuer_thread+0x570/0x570 [ 9400.190478] kthread+0x1d1/0x200 [ 9400.192005] ? set_kthread_struct+0x70/0x70 [ 9400.197658] ret_from_fork+0x1f/0x30 [ 9404.899937] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6767s! [ 9404.903735] Showing busy workqueues and worker pools: [ 9404.907255] workqueue events: flags=0x0 [ 9404.916052] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9404.932710] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9404.941099] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9404.945548] pending: cpu_latency_work [ptlrpc] [ 9404.947543] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9404.958873] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9404.968491] workqueue events_long: flags=0x0 [ 9404.973151] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9404.975770] pending: lu_objects_flush [obdclass] [ 9404.982595] workqueue events_unbound: flags=0x2 [ 9404.987136] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9404.990376] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9404.996595] workqueue events_power_efficient: flags=0x80 [ 9405.002527] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9405.006703] pending: check_lifetime, do_cache_clean [ 9405.013960] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9405.020411] pending: neigh_periodic_work, neigh_periodic_work [ 9405.024052] workqueue mm_percpu_wq: flags=0x8 [ 9405.035865] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9405.038767] pending: vmstat_update, lru_add_drain_per_cpu [ 9405.045289] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9405.056337] pending: vmstat_update, lru_add_drain_per_cpu [ 9405.064799] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9405.072626] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9405.081322] workqueue xprtiod: flags=0xa [ 9405.084823] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9405.088812] pending: xprt_autoclose, xs_error_handle [ 9405.091557] workqueue ipv6_addrconf: flags=0x40008 [ 9405.095949] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9405.099285] pending: addrconf_verify_work [ 9405.101378] workqueue knbd0-recv: flags=0x1a [ 9405.106311] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9405.108895] in-flight: 651:recv_work [ 9405.117617] workqueue ptlrpc_pinger: flags=0x2 [ 9405.119234] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9405.136382] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9405.145275] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6768s workers=4 idle: 15979 31824 8 [ 9405.157279] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9395s workers=2 idle: 654 [ 9408.057037] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9408.060408] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9408.061283] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9408.068081] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 9408.068099] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9408.068174] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9408.068180] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9408.068182] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9408.068200] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9408.068204] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9408.068208] RAX: 000000000000191c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9408.071192] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9408.071197] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9408.071199] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9408.071201] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9408.071206] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9408.071209] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9408.071212] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9408.071221] Call Trace: [ 9408.071228] [ 9408.071235] ? show_regs.cold.9+0x22/0x2f [ 9408.071250] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9408.071257] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9408.071262] ? watchdog+0x40/0x40 [ 9408.071266] ? hrtimer_interrupt+0x128/0x2c0 [ 9408.071270] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9408.071280] ? apic_timer_interrupt+0xf/0x20 [ 9408.071286] [ 9408.071289] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9408.071296] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9408.071305] do_raw_spin_lock+0x177/0x190 [ 9408.071311] _raw_spin_lock+0x21/0x30 [ 9408.071317] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9408.071623] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9408.071847] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9408.072054] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9408.072279] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9408.072482] ? woken_wake_function+0x30/0x30 [ 9408.072491] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9408.072691] kthread+0x1d1/0x200 [ 9408.072698] ? set_kthread_struct+0x70/0x70 [ 9408.072701] ret_from_fork+0x1f/0x30 [ 9408.081158] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9408.081245] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9408.081248] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9408.081266] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9408.081271] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9408.081277] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9408.081279] RDX: 00000000000047cd RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9408.081281] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9408.081284] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9408.081286] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9408.081290] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9408.081293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9408.081295] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9408.081303] Call Trace: [ 9408.081309] [ 9408.081316] ? show_regs.cold.9+0x22/0x2f [ 9408.081329] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9408.081335] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9408.081339] ? watchdog+0x40/0x40 [ 9408.081343] ? hrtimer_interrupt+0x128/0x2c0 [ 9408.081346] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9408.081358] ? apic_timer_interrupt+0xf/0x20 [ 9408.081362] [ 9408.081365] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9408.081369] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9408.081373] do_raw_spin_lock+0x177/0x190 [ 9408.081378] _raw_spin_lock+0x21/0x30 [ 9408.081388] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9408.081904] obd_mod_free+0x46/0x140 [ptlrpc] [ 9408.082099] mdc_commit_open+0x91/0xb0 [mdc] [ 9408.082141] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9408.082470] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9408.082834] after_reply+0xe2c/0x1780 [ptlrpc] [ 9408.083034] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9408.099425] Modules linked in: lustre(O) [ 9408.105740] ? __next_timer_interrupt+0x160/0x160 [ 9408.105768] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9408.183151] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) [ 9408.187806] ? do_wait_intr+0xf0/0xf0 [ 9408.187829] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9408.188320] kthread+0x1d1/0x200 [ 9408.188328] ? set_kthread_struct+0x70/0x70 [ 9408.188332] ret_from_fork+0x1f/0x30 [ 9408.523440] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9408.551391] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9408.557582] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9408.563737] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9408.568270] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9408.579670] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9408.584194] RAX: 0000000000007222 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9408.588312] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9408.592078] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9408.596666] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9408.600368] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9408.608126] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9408.621492] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9408.623583] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9408.632838] Call Trace: [ 9408.634510] [ 9408.635708] ? show_regs.cold.9+0x22/0x2f [ 9408.639727] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9408.642476] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9408.646905] ? watchdog+0x40/0x40 [ 9408.649827] ? hrtimer_interrupt+0x128/0x2c0 [ 9408.656502] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9408.660470] ? apic_timer_interrupt+0xf/0x20 [ 9408.662843] [ 9408.664145] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9408.665880] do_raw_spin_lock+0x177/0x190 [ 9408.668462] _raw_spin_lock+0x21/0x30 [ 9408.671722] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9408.676224] ? do_raw_spin_unlock+0x75/0x190 [ 9408.680728] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9408.686335] ? do_wait_intr+0xf0/0xf0 [ 9408.688490] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9408.694584] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9408.702713] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9408.709545] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9408.712797] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9408.719612] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9408.721429] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9408.726547] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9408.733510] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9408.743665] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9408.749466] lmv_intent_open+0x338/0x1150 [lmv] [ 9408.752379] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9408.758050] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9408.765179] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9408.770647] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9408.774492] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9408.777983] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9408.779958] ? do_raw_spin_unlock+0x75/0x190 [ 9408.783217] ? _raw_spin_unlock+0x12/0x30 [ 9408.786076] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9408.787381] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9408.794170] ll_atomic_open+0x36a/0xd70 [lustre] [ 9408.798437] lookup_open+0x356/0xb40 [ 9408.800292] do_last+0x313/0xfc0 [ 9408.801392] ? nd_jump_root+0xe5/0x160 [ 9408.802690] ? path_init+0x437/0x520 [ 9408.807496] path_openat+0xf7/0x500 [ 9408.809237] ? do_raw_spin_unlock+0x75/0x190 [ 9408.811837] do_filp_open+0x99/0x140 [ 9408.814970] ? getname_flags+0x6e/0x330 [ 9408.823540] ? __check_object_size+0xff/0x256 [ 9408.830452] ? do_raw_spin_unlock+0x75/0x190 [ 9408.833120] ? _raw_spin_unlock+0x12/0x30 [ 9408.835165] do_sys_openat2+0x2b4/0x410 [ 9408.837167] do_sys_open+0x73/0xa0 [ 9408.846467] __x64_sys_openat+0x24/0x30 [ 9408.850105] do_syscall_64+0xc1/0x440 [ 9408.857569] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9408.865792] RIP: 0033:0x7f53425a5332 [ 9408.872731] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9408.891943] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9408.901157] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9408.905682] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9408.914026] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9408.919492] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9408.927665] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9432.052890] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 9432.068045] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9432.141770] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9432.163916] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9432.171599] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9432.178045] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9432.187245] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9432.221911] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9432.242879] RAX: 000000000000617b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9432.257487] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9432.268350] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9432.292612] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9432.312318] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9432.327062] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9432.340262] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9432.349841] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9432.361904] Call Trace: [ 9432.366103] [ 9432.368340] ? show_regs.cold.9+0x22/0x2f [ 9432.372858] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9432.379023] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9432.383108] ? watchdog+0x40/0x40 [ 9432.384297] ? hrtimer_interrupt+0x128/0x2c0 [ 9432.389085] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9432.402254] ? apic_timer_interrupt+0xf/0x20 [ 9432.410976] [ 9432.411782] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9432.422860] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9432.436308] ? newidle_balance.isra.93+0x207/0x580 [ 9432.438519] do_raw_spin_lock+0x177/0x190 [ 9432.447674] _raw_spin_lock+0x21/0x30 [ 9432.449218] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9432.470259] process_one_work+0x2c8/0x700 [ 9432.471610] worker_thread+0x296/0x6e0 [ 9432.477389] ? rescuer_thread+0x570/0x570 [ 9432.481931] kthread+0x1d1/0x200 [ 9432.486752] ? set_kthread_struct+0x70/0x70 [ 9432.493951] ret_from_fork+0x1f/0x30 [ 9435.616050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6798s! [ 9435.644331] Showing busy workqueues and worker pools: [ 9435.659950] workqueue events: flags=0x0 [ 9435.661296] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9435.888971] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9435.918445] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9435.934214] pending: cpu_latency_work [ptlrpc] [ 9435.951917] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9435.956700] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9435.961470] workqueue events_long: flags=0x0 [ 9435.962888] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9435.965429] pending: lu_objects_flush [obdclass] [ 9435.967749] workqueue events_unbound: flags=0x2 [ 9435.969189] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9435.971429] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9435.973558] workqueue events_power_efficient: flags=0x80 [ 9435.975518] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9435.978146] pending: check_lifetime, do_cache_clean [ 9435.979937] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9435.983606] pending: neigh_periodic_work, neigh_periodic_work [ 9435.987386] workqueue mm_percpu_wq: flags=0x8 [ 9435.988954] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9435.991414] pending: vmstat_update, lru_add_drain_per_cpu [ 9435.993511] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9435.998753] pending: vmstat_update, lru_add_drain_per_cpu [ 9436.001946] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9436.007509] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9436.019236] workqueue xprtiod: flags=0xa [ 9436.021752] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9436.025403] pending: xprt_autoclose, xs_error_handle [ 9436.027557] workqueue ipv6_addrconf: flags=0x40008 [ 9436.029287] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9436.031747] pending: addrconf_verify_work [ 9436.033501] workqueue knbd0-recv: flags=0x1a [ 9436.035036] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9436.037551] in-flight: 651:recv_work [ 9436.038917] workqueue ptlrpc_pinger: flags=0x2 [ 9436.040608] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9436.042744] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9436.045101] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6798s workers=4 idle: 15979 31824 8 [ 9436.047872] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9426s workers=2 idle: 654 [ 9436.057468] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 9436.058139] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 9436.058306] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 9436.058313] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9436.058390] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9436.058395] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9436.058397] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9436.058412] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9436.058416] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9436.058421] RAX: 0000000000001c05 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9436.058423] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9436.058425] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9436.058428] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9436.058430] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9436.058434] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9436.058437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9436.058440] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9436.058448] Call Trace: [ 9436.058453] [ 9436.058459] ? show_regs.cold.9+0x22/0x2f [ 9436.058479] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9436.058485] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9436.058490] ? watchdog+0x40/0x40 [ 9436.058494] ? hrtimer_interrupt+0x128/0x2c0 [ 9436.058499] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9436.058765] ? apic_timer_interrupt+0xf/0x20 [ 9436.058770] [ 9436.058774] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9436.058780] do_raw_spin_lock+0x177/0x190 [ 9436.058785] _raw_spin_lock+0x21/0x30 [ 9436.058790] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9436.059064] ? do_raw_spin_unlock+0x75/0x190 [ 9436.059070] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9436.059839] ? do_wait_intr+0xf0/0xf0 [ 9436.059848] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9436.060107] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9436.063455] Modules linked in: lustre(O) [ 9436.065135] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9436.067189] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9436.067323] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9436.067373] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9436.067403] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9436.068714] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9436.068815] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9436.069096] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9436.069134] lmv_intent_open+0x338/0x1150 [lmv] [ 9436.069163] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9436.069251] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9436.069346] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9436.069379] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9436.070020] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9436.070460] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9436.070467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9436.070470] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9436.070483] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9436.070487] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9436.070491] RAX: 0000000000007d10 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9436.070494] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9436.070495] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9436.070498] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9436.070500] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9436.070503] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9436.070505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9436.070508] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9436.070515] Call Trace: [ 9436.070519] [ 9436.070523] ? show_regs.cold.9+0x22/0x2f [ 9436.070532] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9436.070538] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9436.070543] ? watchdog+0x40/0x40 [ 9436.070547] ? hrtimer_interrupt+0x128/0x2c0 [ 9436.070551] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9436.070561] ? apic_timer_interrupt+0xf/0x20 [ 9436.070566] [ 9436.070569] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9436.070574] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9436.070579] do_raw_spin_lock+0x177/0x190 [ 9436.070584] _raw_spin_lock+0x21/0x30 [ 9436.070603] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9436.070875] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9436.071547] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9436.071769] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9436.071973] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9436.072515] ? woken_wake_function+0x30/0x30 [ 9436.072525] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9436.072727] kthread+0x1d1/0x200 [ 9436.072733] ? set_kthread_struct+0x70/0x70 [ 9436.072736] ret_from_fork+0x1f/0x30 [ 9436.069483] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9436.074855] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9436.074942] ? do_raw_spin_unlock+0x75/0x190 [ 9436.074949] ? _raw_spin_unlock+0x12/0x30 [ 9436.074955] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9436.080345] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9436.080417] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9436.080425] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9436.080427] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9436.080442] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9436.080447] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9436.080452] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9436.080455] RDX: 0000000000004922 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9436.080457] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9436.080460] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9436.080462] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9436.080466] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9436.080469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9436.080471] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9436.080480] Call Trace: [ 9436.080486] [ 9436.080492] ? show_regs.cold.9+0x22/0x2f [ 9436.080503] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9436.080510] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9436.080515] ? watchdog+0x40/0x40 [ 9436.080520] ? hrtimer_interrupt+0x128/0x2c0 [ 9436.080524] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9436.080533] ? apic_timer_interrupt+0xf/0x20 [ 9436.080538] [ 9436.080542] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9436.080547] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9436.080552] do_raw_spin_lock+0x177/0x190 [ 9436.080557] _raw_spin_lock+0x21/0x30 [ 9436.080563] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9436.080826] obd_mod_free+0x46/0x140 [ptlrpc] [ 9436.081015] mdc_commit_open+0x91/0xb0 [mdc] [ 9436.081050] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9436.081242] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9436.081474] after_reply+0xe2c/0x1780 [ptlrpc] [ 9436.081672] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9436.081870] ? __next_timer_interrupt+0x160/0x160 [ 9436.081880] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9436.082081] ? do_wait_intr+0xf0/0xf0 [ 9436.082090] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9436.082306] kthread+0x1d1/0x200 [ 9436.082312] ? set_kthread_struct+0x70/0x70 [ 9436.082315] ret_from_fork+0x1f/0x30 [ 9436.756522] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9436.759075] ll_atomic_open+0x36a/0xd70 [lustre] [ 9436.763636] lookup_open+0x356/0xb40 [ 9436.764921] do_last+0x313/0xfc0 [ 9436.765988] ? nd_jump_root+0xe5/0x160 [ 9436.767518] ? path_init+0x437/0x520 [ 9436.769271] path_openat+0xf7/0x500 [ 9436.770542] ? do_raw_spin_unlock+0x75/0x190 [ 9436.771978] do_filp_open+0x99/0x140 [ 9436.773034] ? getname_flags+0x6e/0x330 [ 9436.773942] ? __check_object_size+0xff/0x256 [ 9436.776913] ? do_raw_spin_unlock+0x75/0x190 [ 9436.778093] ? _raw_spin_unlock+0x12/0x30 [ 9436.779037] do_sys_openat2+0x2b4/0x410 [ 9436.781997] do_sys_open+0x73/0xa0 [ 9436.782828] __x64_sys_openat+0x24/0x30 [ 9436.783801] do_syscall_64+0xc1/0x440 [ 9436.786875] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9436.788727] RIP: 0033:0x7f53425a5332 [ 9436.790110] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9436.797723] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9436.802516] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9436.806229] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9436.809944] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9436.816678] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9436.819715] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9460.053662] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 9460.058616] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9460.089773] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9460.095478] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9460.099262] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9460.103606] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9460.106786] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9460.114267] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9460.121302] RAX: 0000000000005472 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9460.129477] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9460.136269] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9460.143021] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9460.148714] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9460.154989] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9460.160821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9460.165999] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9460.170032] Call Trace: [ 9460.171987] [ 9460.172846] ? show_regs.cold.9+0x22/0x2f [ 9460.176838] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9460.179251] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9460.181164] ? watchdog+0x40/0x40 [ 9460.182623] ? hrtimer_interrupt+0x128/0x2c0 [ 9460.184452] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9460.190038] ? apic_timer_interrupt+0xf/0x20 [ 9460.192617] [ 9460.193475] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9460.198395] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9460.200799] ? newidle_balance.isra.93+0x207/0x580 [ 9460.203200] do_raw_spin_lock+0x177/0x190 [ 9460.204736] _raw_spin_lock+0x21/0x30 [ 9460.207168] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9460.209943] process_one_work+0x2c8/0x700 [ 9460.212017] worker_thread+0x296/0x6e0 [ 9460.215408] ? rescuer_thread+0x570/0x570 [ 9460.217228] kthread+0x1d1/0x200 [ 9460.218884] ? set_kthread_struct+0x70/0x70 [ 9460.221325] ret_from_fork+0x1f/0x30 [ 9464.055023] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 9464.057136] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 9464.058028] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 9464.058037] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9464.058109] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9464.058115] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9464.058117] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9464.058130] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9464.058135] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9464.058139] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9464.058142] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9464.058144] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9464.058147] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9464.058149] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9464.058152] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9464.058155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9464.058157] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9464.058166] Call Trace: [ 9464.058171] [ 9464.058177] ? show_regs.cold.9+0x22/0x2f [ 9464.058188] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9464.058194] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9464.058199] ? watchdog+0x40/0x40 [ 9464.058204] ? hrtimer_interrupt+0x128/0x2c0 [ 9464.058208] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9464.058214] ? apic_timer_interrupt+0xf/0x20 [ 9464.058219] [ 9464.058222] ? native_safe_halt+0x2e/0x50 [ 9464.058226] kvm_wait+0xcd/0xe0 [ 9464.058232] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9464.058240] do_raw_spin_lock+0x177/0x190 [ 9464.058244] _raw_spin_lock+0x21/0x30 [ 9464.058358] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9464.058412] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9464.058417] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9464.058420] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9464.058426] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9464.058430] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 [ 9464.058255] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9464.058434] ORIG_RAX: ffffffffffffff13 [ 9464.058436] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9464.058439] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9464.058441] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9464.058443] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9464.058445] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9464.058448] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9464.058450] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9464.058452] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9464.058458] Call Trace: [ 9464.058461] [ 9464.058464] ? show_regs.cold.9+0x22/0x2f [ 9464.058470] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9464.058474] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9464.058478] ? watchdog+0x40/0x40 [ 9464.058482] ? hrtimer_interrupt+0x128/0x2c0 [ 9464.058486] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9464.058492] ? apic_timer_interrupt+0xf/0x20 [ 9464.058496] [ 9464.058500] ? native_safe_halt+0x2e/0x50 [ 9464.058503] kvm_wait+0xcd/0xe0 [ 9464.058508] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9464.058514] do_raw_spin_lock+0x177/0x190 [ 9464.058518] _raw_spin_lock+0x21/0x30 [ 9464.058575] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9464.058921] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9464.058521] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9464.059625] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9464.060384] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9464.060612] ? woken_wake_function+0x30/0x30 [ 9464.060621] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9464.060854] kthread+0x1d1/0x200 [ 9464.060861] ? set_kthread_struct+0x70/0x70 [ 9464.060865] ret_from_fork+0x1f/0x30 [ 9464.060909] ? do_raw_spin_unlock+0x75/0x190 [ 9464.060917] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9464.065656] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9464.065740] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9464.065748] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9464.065751] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9464.065767] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9464.065772] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9464.065776] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9464.065778] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9464.065780] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9464.065783] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9464.065785] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9464.065789] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9464.065792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9464.065794] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9464.065812] Call Trace: [ 9464.065819] [ 9464.065826] ? show_regs.cold.9+0x22/0x2f [ 9464.065837] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9464.065844] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9464.065849] ? watchdog+0x40/0x40 [ 9464.065853] ? hrtimer_interrupt+0x128/0x2c0 [ 9464.065857] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9464.065865] ? apic_timer_interrupt+0xf/0x20 [ 9464.065869] [ 9464.065872] ? native_safe_halt+0x2e/0x50 [ 9464.065876] kvm_wait+0xcd/0xe0 [ 9464.065884] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9464.065891] do_raw_spin_lock+0x177/0x190 [ 9464.065896] _raw_spin_lock+0x21/0x30 [ 9464.065902] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9464.066285] obd_mod_free+0x46/0x140 [ptlrpc] [ 9464.066568] mdc_commit_open+0x91/0xb0 [mdc] [ 9464.066608] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9464.067439] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9464.067717] after_reply+0xe2c/0x1780 [ptlrpc] [ 9464.073642] ? do_wait_intr+0xf0/0xf0 [ 9464.090680] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9464.101172] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9464.101397] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9464.101555] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9464.101717] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9464.101839] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9464.101877] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9464.101897] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9464.101914] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9464.101969] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9464.107374] ? __next_timer_interrupt+0x160/0x160 [ 9464.108524] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9464.118382] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9464.123193] lmv_intent_open+0x338/0x1150 [lmv] [ 9464.123238] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9464.123363] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9464.123460] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9464.123483] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9464.123644] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9464.127559] ? do_wait_intr+0xf0/0xf0 [ 9464.134241] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9464.138400] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9464.138722] kthread+0x1d1/0x200 [ 9464.138730] ? set_kthread_struct+0x70/0x70 [ 9464.138734] ret_from_fork+0x1f/0x30 [ 9464.685369] ? do_raw_spin_unlock+0x75/0x190 [ 9464.692696] ? _raw_spin_unlock+0x12/0x30 [ 9464.696204] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9464.702363] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9464.709909] ll_atomic_open+0x36a/0xd70 [lustre] [ 9464.712477] lookup_open+0x356/0xb40 [ 9464.719506] do_last+0x313/0xfc0 [ 9464.724449] ? nd_jump_root+0xe5/0x160 [ 9464.727354] ? path_init+0x437/0x520 [ 9464.730082] path_openat+0xf7/0x500 [ 9464.732337] ? do_raw_spin_unlock+0x75/0x190 [ 9464.734529] do_filp_open+0x99/0x140 [ 9464.737135] ? getname_flags+0x6e/0x330 [ 9464.740260] ? __check_object_size+0xff/0x256 [ 9464.747424] ? do_raw_spin_unlock+0x75/0x190 [ 9464.755733] ? _raw_spin_unlock+0x12/0x30 [ 9464.764167] do_sys_openat2+0x2b4/0x410 [ 9464.765223] do_sys_open+0x73/0xa0 [ 9464.766191] __x64_sys_openat+0x24/0x30 [ 9464.767651] do_syscall_64+0xc1/0x440 [ 9464.769302] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9464.771374] RIP: 0033:0x7f53425a5332 [ 9464.772888] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9464.780205] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9464.783217] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9464.786097] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9464.789391] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9464.792542] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9464.796643] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9466.338254] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6829s! [ 9466.349099] Showing busy workqueues and worker pools: [ 9466.357235] workqueue events: flags=0x0 [ 9466.360739] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9466.371602] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9466.389526] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9466.394576] pending: cpu_latency_work [ptlrpc] [ 9466.398388] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9466.400950] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9466.405354] workqueue events_long: flags=0x0 [ 9466.409517] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9466.413338] pending: lu_objects_flush [obdclass] [ 9466.416121] workqueue events_unbound: flags=0x2 [ 9466.420130] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9466.426298] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9466.432521] workqueue events_power_efficient: flags=0x80 [ 9466.435321] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9466.439405] pending: check_lifetime, do_cache_clean [ 9466.443959] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9466.451943] pending: neigh_periodic_work, neigh_periodic_work [ 9466.460435] workqueue mm_percpu_wq: flags=0x8 [ 9466.465302] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9466.469139] pending: vmstat_update, lru_add_drain_per_cpu [ 9466.474289] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9466.477650] pending: vmstat_update, lru_add_drain_per_cpu [ 9466.485833] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9466.489959] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9466.494258] workqueue xprtiod: flags=0xa [ 9466.495970] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9466.506749] pending: xprt_autoclose, xs_error_handle [ 9466.514413] workqueue ipv6_addrconf: flags=0x40008 [ 9466.518486] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9466.529062] pending: addrconf_verify_work [ 9466.534675] workqueue knbd0-recv: flags=0x1a [ 9466.538824] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9466.546303] in-flight: 651:recv_work [ 9466.551979] workqueue ptlrpc_pinger: flags=0x2 [ 9466.553538] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9466.561849] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9466.565062] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6829s workers=4 idle: 15979 31824 8 [ 9466.577583] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9457s workers=2 idle: 654 [ 9492.052539] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 9492.055008] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 9492.057087] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 9492.057100] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9492.057668] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9492.057674] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9492.057677] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9492.057691] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9492.057696] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9492.057700] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9492.057702] RDX: 0000000000005c7e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9492.057704] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9492.057706] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9492.057708] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9492.057711] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9492.057714] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9492.057716] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9492.057724] Call Trace: [ 9492.057730] [ 9492.057735] ? show_regs.cold.9+0x22/0x2f [ 9492.057746] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9492.057753] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9492.057757] ? watchdog+0x40/0x40 [ 9492.057761] ? hrtimer_interrupt+0x128/0x2c0 [ 9492.057765] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9492.057774] ? apic_timer_interrupt+0xf/0x20 [ 9492.057779] [ 9492.057782] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9492.057787] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9492.057791] do_raw_spin_lock+0x177/0x190 [ 9492.057795] _raw_spin_lock+0x21/0x30 [ 9492.057801] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9492.060657] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9492.060739] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9492.060746] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9492.058062] obd_mod_free+0x46/0x140 [ptlrpc] [ 9492.061081] mdc_commit_open+0x91/0xb0 [mdc] [ 9492.061121] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9492.061401] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9492.061592] after_reply+0xe2c/0x1780 [ptlrpc] [ 9492.062072] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9492.062530] ? __next_timer_interrupt+0x160/0x160 [ 9492.062541] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9492.060753] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9492.062945] ? do_wait_intr+0xf0/0xf0 [ 9492.062955] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9492.063179] kthread+0x1d1/0x200 [ 9492.063185] ? set_kthread_struct+0x70/0x70 [ 9492.063189] ret_from_fork+0x1f/0x30 [ 9492.063828] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 9492.063836] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic [ 9492.064851] [ 9492.064855] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9492.064867] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9492.064872] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9492.064877] RAX: 0000000000006888 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9492.064879] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9492.064882] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9492.064884] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9492.064886] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9492.064889] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9492.064892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9492.064894] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9492.064902] Call Trace: [ 9492.064906] [ 9492.064910] ? show_regs.cold.9+0x22/0x2f [ 9492.064919] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9492.064925] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9492.065000] ? watchdog+0x40/0x40 [ 9492.065005] ? hrtimer_interrupt+0x128/0x2c0 [ 9492.065010] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9492.065018] ? apic_timer_interrupt+0xf/0x20 [ 9492.065034] [ 9492.065038] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9492.065042] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9492.065047] ? newidle_balance.isra.93+0x207/0x580 [ 9492.065053] do_raw_spin_lock+0x177/0x190 [ 9492.065058] _raw_spin_lock+0x21/0x30 [ 9492.065063] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9492.066597] crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9492.066622] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9492.066630] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9492.066632] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9492.066642] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9492.066646] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9492.066651] RAX: 000000000000618b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9492.066653] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9492.066656] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9492.066658] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9492.066661] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9492.066664] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9492.066667] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9492.066669] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9492.066677] Call Trace: [ 9492.066681] [ 9492.066686] ? show_regs.cold.9+0x22/0x2f [ 9492.066698] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9492.066704] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9492.066708] ? watchdog+0x40/0x40 [ 9492.066712] ? hrtimer_interrupt+0x128/0x2c0 [ 9492.066716] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9492.066723] ? apic_timer_interrupt+0xf/0x20 [ 9492.066728] [ 9492.066731] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9492.066736] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9492.066741] do_raw_spin_lock+0x177/0x190 [ 9492.066746] _raw_spin_lock+0x21/0x30 [ 9492.066751] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9492.067080] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9492.067272] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9492.067497] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9492.067708] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9492.067923] ? woken_wake_function+0x30/0x30 [ 9492.068586] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9492.068817] kthread+0x1d1/0x200 [ 9492.068824] ? set_kthread_struct+0x70/0x70 [ 9492.068832] ret_from_fork+0x1f/0x30 [ 9492.081664] Modules linked in: [ 9492.086852] process_one_work+0x2c8/0x700 [ 9492.086884] worker_thread+0x296/0x6e0 [ 9492.086889] ? rescuer_thread+0x570/0x570 [ 9492.086894] kthread+0x1d1/0x200 [ 9492.086898] ? set_kthread_struct+0x70/0x70 [ 9492.086902] ret_from_fork+0x1f/0x30 [ 9492.467317] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9492.498783] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9492.505442] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9492.509254] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9492.511154] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9492.525218] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9492.528601] RAX: 00000000000075b5 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9492.535359] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9492.539003] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9492.544253] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9492.550358] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9492.554513] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9492.561363] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9492.567939] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9492.571288] Call Trace: [ 9492.572268] [ 9492.573284] ? show_regs.cold.9+0x22/0x2f [ 9492.574960] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9492.577506] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9492.579195] ? watchdog+0x40/0x40 [ 9492.580754] ? hrtimer_interrupt+0x128/0x2c0 [ 9492.583894] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9492.587977] ? apic_timer_interrupt+0xf/0x20 [ 9492.589862] [ 9492.593566] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9492.595910] do_raw_spin_lock+0x177/0x190 [ 9492.597939] _raw_spin_lock+0x21/0x30 [ 9492.598923] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9492.603217] ? do_raw_spin_unlock+0x75/0x190 [ 9492.604812] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9492.608848] ? do_wait_intr+0xf0/0xf0 [ 9492.610284] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9492.612276] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9492.614155] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9492.616204] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9492.618552] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9492.624436] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9492.626189] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9492.628131] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9492.634437] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9492.642465] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9492.644186] lmv_intent_open+0x338/0x1150 [lmv] [ 9492.647106] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9492.649407] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9492.651435] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9492.653528] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9492.655403] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9492.657909] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9492.663995] ? do_raw_spin_unlock+0x75/0x190 [ 9492.669419] ? _raw_spin_unlock+0x12/0x30 [ 9492.671027] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9492.675789] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9492.679288] ll_atomic_open+0x36a/0xd70 [lustre] [ 9492.681615] lookup_open+0x356/0xb40 [ 9492.683418] do_last+0x313/0xfc0 [ 9492.684580] ? nd_jump_root+0xe5/0x160 [ 9492.686023] ? path_init+0x437/0x520 [ 9492.687409] path_openat+0xf7/0x500 [ 9492.688443] ? do_raw_spin_unlock+0x75/0x190 [ 9492.689675] do_filp_open+0x99/0x140 [ 9492.690786] ? getname_flags+0x6e/0x330 [ 9492.691866] ? __check_object_size+0xff/0x256 [ 9492.693532] ? do_raw_spin_unlock+0x75/0x190 [ 9492.695192] ? _raw_spin_unlock+0x12/0x30 [ 9492.699580] do_sys_openat2+0x2b4/0x410 [ 9492.701836] do_sys_open+0x73/0xa0 [ 9492.704363] __x64_sys_openat+0x24/0x30 [ 9492.705981] do_syscall_64+0xc1/0x440 [ 9492.709956] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9492.712244] RIP: 0033:0x7f53425a5332 [ 9492.713985] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9492.727562] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9492.731099] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9492.734128] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9492.739712] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9492.744879] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9492.748605] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9497.056182] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6859s! [ 9497.062154] Showing busy workqueues and worker pools: [ 9497.066703] workqueue events: flags=0x0 [ 9497.069973] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9497.075655] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9497.093813] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9497.097114] pending: cpu_latency_work [ptlrpc] [ 9497.106614] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9497.109081] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9497.119916] workqueue events_long: flags=0x0 [ 9497.122575] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9497.128012] pending: lu_objects_flush [obdclass] [ 9497.134114] workqueue events_unbound: flags=0x2 [ 9497.137762] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9497.139651] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9497.143636] workqueue events_power_efficient: flags=0x80 [ 9497.144931] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9497.150977] pending: check_lifetime, do_cache_clean [ 9497.157759] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9497.166748] pending: neigh_periodic_work, neigh_periodic_work [ 9497.173908] workqueue mm_percpu_wq: flags=0x8 [ 9497.175044] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9497.179299] pending: vmstat_update, lru_add_drain_per_cpu [ 9497.181425] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9497.183401] pending: vmstat_update, lru_add_drain_per_cpu [ 9497.185390] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9497.189831] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9497.195024] workqueue xprtiod: flags=0xa [ 9497.196304] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9497.203532] pending: xprt_autoclose, xs_error_handle [ 9497.206578] workqueue ipv6_addrconf: flags=0x40008 [ 9497.210895] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9497.218090] pending: addrconf_verify_work [ 9497.220931] workqueue knbd0-recv: flags=0x1a [ 9497.222428] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9497.225082] in-flight: 651:recv_work [ 9497.229034] workqueue ptlrpc_pinger: flags=0x2 [ 9497.234719] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9497.240375] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9497.247031] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6860s workers=4 idle: 15979 31824 8 [ 9497.251786] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9487s workers=2 idle: 654 [ 9520.057590] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 9520.058884] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 9520.059086] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 9520.059093] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9520.059166] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9520.059172] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9520.059174] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9520.059188] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9520.059192] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9520.059196] RAX: 000000000000148a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9520.059199] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9520.059201] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9520.059204] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9520.059206] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9520.059210] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9520.059213] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9520.059215] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9520.059225] Call Trace: [ 9520.059230] [ 9520.059235] ? show_regs.cold.9+0x22/0x2f [ 9520.059245] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9520.059250] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9520.059255] ? watchdog+0x40/0x40 [ 9520.059259] ? hrtimer_interrupt+0x128/0x2c0 [ 9520.059262] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9520.059270] ? apic_timer_interrupt+0xf/0x20 [ 9520.059275] [ 9520.059277] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9520.059282] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9520.059288] do_raw_spin_lock+0x177/0x190 [ 9520.059293] _raw_spin_lock+0x21/0x30 [ 9520.059298] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9520.060268] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9520.060477] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9520.062732] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9520.063396] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9520.063692] ? woken_wake_function+0x30/0x30 [ 9520.063702] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9520.064272] kthread+0x1d1/0x200 [ 9520.064278] ? set_kthread_struct+0x70/0x70 [ 9520.064282] ret_from_fork+0x1f/0x30 [ 9520.073136] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9520.073219] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9520.073227] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9520.073231] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9520.073245] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9520.073249] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9520.073254] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9520.073256] RDX: 0000000000007f86 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9520.073259] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9520.073261] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9520.073264] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9520.073268] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9520.073270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9520.073273] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9520.073282] Call Trace: [ 9520.073288] [ 9520.073294] ? show_regs.cold.9+0x22/0x2f [ 9520.073304] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9520.073311] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9520.073316] ? watchdog+0x40/0x40 [ 9520.073320] ? hrtimer_interrupt+0x128/0x2c0 [ 9520.073325] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9520.073333] ? apic_timer_interrupt+0xf/0x20 [ 9520.073338] [ 9520.073340] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9520.073345] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9520.073350] do_raw_spin_lock+0x177/0x190 [ 9520.073355] _raw_spin_lock+0x21/0x30 [ 9520.073360] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9520.073647] obd_mod_free+0x46/0x140 [ptlrpc] [ 9520.080663] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9520.080749] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9520.080757] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9520.080760] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9520.080779] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9520.080784] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9520.080789] RAX: 0000000000002b84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9520.080791] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9520.080793] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9520.080796] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9520.080799] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9520.080802] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9520.080805] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9520.080808] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9520.080817] Call Trace: [ 9520.086312] [ 9520.089105] mdc_commit_open+0x91/0xb0 [mdc] [ 9520.089223] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9520.089600] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9520.089809] after_reply+0xe2c/0x1780 [ptlrpc] [ 9520.090016] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9520.090204] ? __next_timer_interrupt+0x160/0x160 [ 9520.090212] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9520.090369] ? do_wait_intr+0xf0/0xf0 [ 9520.090376] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9520.090573] kthread+0x1d1/0x200 [ 9520.090580] ? set_kthread_struct+0x70/0x70 [ 9520.090584] ret_from_fork+0x1f/0x30 [ 9520.722754] ? show_regs.cold.9+0x22/0x2f [ 9520.724547] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9520.726618] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9520.728597] ? watchdog+0x40/0x40 [ 9520.730733] ? hrtimer_interrupt+0x128/0x2c0 [ 9520.735189] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9520.739553] ? apic_timer_interrupt+0xf/0x20 [ 9520.743763] [ 9520.745669] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9520.750759] do_raw_spin_lock+0x177/0x190 [ 9520.753004] _raw_spin_lock+0x21/0x30 [ 9520.755703] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9520.757628] ? do_raw_spin_unlock+0x75/0x190 [ 9520.759907] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9520.763089] ? do_wait_intr+0xf0/0xf0 [ 9520.765238] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9520.767129] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9520.769647] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9520.771710] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9520.778289] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9520.788910] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9520.796773] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9520.808747] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9520.821640] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9520.833710] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9520.847482] lmv_intent_open+0x338/0x1150 [lmv] [ 9520.853497] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9520.855289] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9520.862870] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9520.875427] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9520.880711] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9520.890479] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9520.897268] ? do_raw_spin_unlock+0x75/0x190 [ 9520.898590] ? _raw_spin_unlock+0x12/0x30 [ 9520.906591] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9520.908375] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9520.914270] ll_atomic_open+0x36a/0xd70 [lustre] [ 9520.920526] lookup_open+0x356/0xb40 [ 9520.929736] do_last+0x313/0xfc0 [ 9520.933929] ? nd_jump_root+0xe5/0x160 [ 9520.938440] ? path_init+0x437/0x520 [ 9520.939731] path_openat+0xf7/0x500 [ 9520.944602] ? do_raw_spin_unlock+0x75/0x190 [ 9520.952097] do_filp_open+0x99/0x140 [ 9520.953274] ? getname_flags+0x6e/0x330 [ 9520.954757] ? __check_object_size+0xff/0x256 [ 9520.960561] ? do_raw_spin_unlock+0x75/0x190 [ 9520.967726] ? _raw_spin_unlock+0x12/0x30 [ 9520.974242] do_sys_openat2+0x2b4/0x410 [ 9520.975545] do_sys_open+0x73/0xa0 [ 9520.981142] __x64_sys_openat+0x24/0x30 [ 9520.982456] do_syscall_64+0xc1/0x440 [ 9520.993362] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9521.000532] RIP: 0033:0x7f53425a5332 [ 9521.002689] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9521.013236] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9521.022952] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9521.029409] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9521.036409] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9521.046412] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9521.051296] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9524.052110] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 9524.057504] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9524.078262] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9524.083532] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9524.086577] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9524.088895] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9524.090843] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9524.099409] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9524.102707] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9524.105881] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9524.109396] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9524.113144] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9524.116401] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9524.120022] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9524.124148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9524.127458] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9524.131168] Call Trace: [ 9524.132859] [ 9524.135986] ? show_regs.cold.9+0x22/0x2f [ 9524.137296] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9524.143995] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9524.145565] ? watchdog+0x40/0x40 [ 9524.148929] ? hrtimer_interrupt+0x128/0x2c0 [ 9524.151535] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9524.158653] ? apic_timer_interrupt+0xf/0x20 [ 9524.160273] [ 9524.161257] ? native_safe_halt+0x2e/0x50 [ 9524.163052] kvm_wait+0xcd/0xe0 [ 9524.164289] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9524.168646] ? newidle_balance.isra.93+0x207/0x580 [ 9524.170168] do_raw_spin_lock+0x177/0x190 [ 9524.176091] _raw_spin_lock+0x21/0x30 [ 9524.177503] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9524.179899] process_one_work+0x2c8/0x700 [ 9524.181437] worker_thread+0x296/0x6e0 [ 9524.183077] ? rescuer_thread+0x570/0x570 [ 9524.184653] kthread+0x1d1/0x200 [ 9524.187102] ? set_kthread_struct+0x70/0x70 [ 9524.188411] ret_from_fork+0x1f/0x30 [ 9527.265573] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6890s! [ 9527.270099] Showing busy workqueues and worker pools: [ 9527.274684] workqueue events: flags=0x0 [ 9527.276069] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9527.282416] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9527.295397] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9527.302238] pending: cpu_latency_work [ptlrpc] [ 9527.306987] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9527.314158] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9527.321348] workqueue events_long: flags=0x0 [ 9527.325957] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9527.332262] pending: lu_objects_flush [obdclass] [ 9527.338007] workqueue events_unbound: flags=0x2 [ 9527.341304] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9527.347121] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9527.353245] workqueue events_power_efficient: flags=0x80 [ 9527.355858] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9527.362942] pending: check_lifetime, do_cache_clean [ 9527.369504] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9527.376395] pending: neigh_periodic_work, neigh_periodic_work [ 9527.380280] workqueue mm_percpu_wq: flags=0x8 [ 9527.387016] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9527.390103] pending: vmstat_update, lru_add_drain_per_cpu [ 9527.395814] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9527.402263] pending: vmstat_update, lru_add_drain_per_cpu [ 9527.408478] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9527.415080] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9527.423432] workqueue xprtiod: flags=0xa [ 9527.424787] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9527.430540] pending: xprt_autoclose, xs_error_handle [ 9527.437782] workqueue ipv6_addrconf: flags=0x40008 [ 9527.440051] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9527.447114] pending: addrconf_verify_work [ 9527.453097] workqueue knbd0-recv: flags=0x1a [ 9527.454250] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9527.460287] in-flight: 651:recv_work [ 9527.461339] workqueue ptlrpc_pinger: flags=0x2 [ 9527.464203] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9527.469357] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9527.476287] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6890s workers=4 idle: 15979 31824 8 [ 9527.484136] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9517s workers=2 idle: 654 [ 9534.610008] rcu: INFO: rcu_sched self-detected stall on CPU [ 9534.616905] rcu: 2-....: (4059549 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 9534.622181] (t=7440184 jiffies g=274185 q=65748) [ 9534.624759] rcu: rcu_sched kthread starved for 6898550 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [ 9534.629450] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9534.634023] rcu: RCU grace-period kthread stack dump: [ 9534.636747] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9534.641410] Call Trace: [ 9534.642496] __schedule+0x351/0xcb0 [ 9534.644325] schedule+0xc0/0x180 [ 9534.645984] schedule_timeout+0xb4/0x190 [ 9534.648109] ? do_raw_spin_unlock+0x75/0x190 [ 9534.650635] ? __next_timer_interrupt+0x160/0x160 [ 9534.652965] ? prepare_to_swait_event+0xf3/0x140 [ 9534.655170] rcu_gp_kthread+0x8dd/0x15b0 [ 9534.657060] ? force_qs_rnp+0x2c0/0x2c0 [ 9534.659203] kthread+0x1d1/0x200 [ 9534.660714] ? set_kthread_struct+0x70/0x70 [ 9534.662696] ret_from_fork+0x1f/0x30 [ 9534.664069] rcu: Stack dump where RCU GP kthread last ran: [ 9534.665853] Sending NMI from CPU 2 to CPUs 1: [ 9534.667807] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [ 9534.668779] Sending NMI from CPU 2 to CPUs 1: [ 9534.675447] NMI backtrace for cpu 1 [ 9534.675455] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9534.675457] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9534.675458] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9534.675460] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9534.675462] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [ 9534.675465] RAX: 000000000000038e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9534.675466] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9534.675468] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9534.675469] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9534.675470] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9534.675472] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9534.675473] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9534.675475] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9534.675476] Call Trace: [ 9534.675477] [ 9534.675478] ? show_regs.cold.9+0x22/0x2f [ 9534.675479] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9534.675480] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9534.675481] ? nmi_handle+0xbd/0x210 [ 9534.675482] ? default_do_nmi+0x4b/0x1b0 [ 9534.675483] ? do_nmi+0x280/0x380 [ 9534.675484] ? end_repeat_nmi+0x16/0x1f [ 9534.675485] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9534.675486] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9534.675488] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9534.675489] [ 9534.675490] do_raw_spin_lock+0x177/0x190 [ 9534.675491] _raw_spin_lock+0x21/0x30 [ 9534.675492] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9534.675493] ? do_raw_spin_unlock+0x75/0x190 [ 9534.675494] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9534.675495] ? do_wait_intr+0xf0/0xf0 [ 9534.675496] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9534.675497] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9534.675498] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9534.675499] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9534.675501] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9534.675502] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9534.675504] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9534.675505] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9534.675506] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9534.675507] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9534.675508] lmv_intent_open+0x338/0x1150 [lmv] [ 9534.675509] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9534.675510] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9534.675511] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9534.675512] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9534.675513] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9534.675514] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9534.675515] ? do_raw_spin_unlock+0x75/0x190 [ 9534.675516] ? _raw_spin_unlock+0x12/0x30 [ 9534.675517] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9534.675518] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9534.675519] ll_atomic_open+0x36a/0xd70 [lustre] [ 9534.675521] lookup_open+0x356/0xb40 [ 9534.675522] do_last+0x313/0xfc0 [ 9534.675523] ? nd_jump_root+0xe5/0x160 [ 9534.675524] ? path_init+0x437/0x520 [ 9534.675525] path_openat+0xf7/0x500 [ 9534.675526] ? do_raw_spin_unlock+0x75/0x190 [ 9534.675527] do_filp_open+0x99/0x140 [ 9534.675528] ? getname_flags+0x6e/0x330 [ 9534.675529] ? __check_object_size+0xff/0x256 [ 9534.675530] ? do_raw_spin_unlock+0x75/0x190 [ 9534.675531] ? _raw_spin_unlock+0x12/0x30 [ 9534.675532] do_sys_openat2+0x2b4/0x410 [ 9534.675533] do_sys_open+0x73/0xa0 [ 9534.675535] __x64_sys_openat+0x24/0x30 [ 9534.675536] do_syscall_64+0xc1/0x440 [ 9534.675537] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9534.675538] RIP: 0033:0x7f53425a5332 [ 9534.675540] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9534.675542] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9534.675545] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9534.675546] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9534.675547] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9534.675549] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9534.675550] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9534.675806] NMI backtrace for cpu 2 [ 9534.880814] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9534.886274] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9534.890050] Call Trace: [ 9534.891197] [ 9534.892106] ? dump_stack+0xbb/0x10e [ 9534.893718] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9534.895918] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9534.897939] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9534.900206] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9534.902584] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9534.904667] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9534.906432] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9534.908723] ? update_process_times+0x6d/0xc0 [ 9534.910774] ? tick_sched_handle.isra.18+0x27/0x90 [ 9534.913016] ? tick_sched_timer+0x53/0xb0 [ 9534.914782] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9534.916748] ? tick_sched_handle.isra.18+0x90/0x90 [ 9534.918917] ? hrtimer_interrupt+0x128/0x2c0 [ 9534.920889] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9534.923387] ? apic_timer_interrupt+0xf/0x20 [ 9534.925361] [ 9534.926329] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9534.928784] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9534.931054] ? do_raw_spin_lock+0x177/0x190 [ 9534.932672] ? _raw_spin_lock+0x21/0x30 [ 9534.934363] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9534.936888] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9534.939421] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9534.941552] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9534.944938] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9534.947780] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9534.950292] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9534.952899] ? __next_timer_interrupt+0x160/0x160 [ 9534.955270] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9534.958177] ? do_wait_intr+0xf0/0xf0 [ 9534.962292] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9534.967073] ? kthread+0x1d1/0x200 [ 9534.968837] ? set_kthread_struct+0x70/0x70 [ 9534.972308] ? ret_from_fork+0x1f/0x30 [ 9548.058084] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 9548.061933] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9548.081198] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9548.086073] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9548.090460] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9548.093487] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9548.101666] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9548.104852] RAX: 0000000000002e62 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9548.107750] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9548.110671] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9548.113667] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9548.116520] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9548.119617] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9548.123039] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9548.125193] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9548.129241] Call Trace: [ 9548.130439] [ 9548.131619] ? show_regs.cold.9+0x22/0x2f [ 9548.133322] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9548.135308] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9548.136748] ? watchdog+0x40/0x40 [ 9548.137681] ? hrtimer_interrupt+0x128/0x2c0 [ 9548.138948] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9548.141252] ? apic_timer_interrupt+0xf/0x20 [ 9548.143048] [ 9548.143875] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9548.146304] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9548.148668] do_raw_spin_lock+0x177/0x190 [ 9548.150434] _raw_spin_lock+0x21/0x30 [ 9548.152009] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9548.155174] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9548.157699] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9548.159689] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9548.162273] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9548.165273] ? woken_wake_function+0x30/0x30 [ 9548.166960] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9548.169828] kthread+0x1d1/0x200 [ 9548.171299] ? set_kthread_struct+0x70/0x70 [ 9548.173310] ret_from_fork+0x1f/0x30 [ 9552.052101] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 9552.057872] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9552.096469] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9552.103780] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9552.108985] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9552.113841] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9552.117241] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9552.128724] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9552.132787] RAX: 0000000000001286 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9552.137715] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9552.141473] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9552.144916] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9552.149348] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9552.153937] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9552.160244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9552.164940] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9552.169541] Call Trace: [ 9552.172715] [ 9552.173681] ? show_regs.cold.9+0x22/0x2f [ 9552.175843] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9552.178246] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9552.180757] ? watchdog+0x40/0x40 [ 9552.182560] ? hrtimer_interrupt+0x128/0x2c0 [ 9552.184724] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9552.187763] ? apic_timer_interrupt+0xf/0x20 [ 9552.189911] [ 9552.191318] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9552.194208] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9552.196446] ? newidle_balance.isra.93+0x207/0x580 [ 9552.198517] do_raw_spin_lock+0x177/0x190 [ 9552.200560] _raw_spin_lock+0x21/0x30 [ 9552.207980] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9552.217360] process_one_work+0x2c8/0x700 [ 9552.220650] worker_thread+0x296/0x6e0 [ 9552.222681] ? rescuer_thread+0x570/0x570 [ 9552.229013] kthread+0x1d1/0x200 [ 9552.230708] ? set_kthread_struct+0x70/0x70 [ 9552.234754] ret_from_fork+0x1f/0x30 [ 9557.984044] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6920s! [ 9557.986902] Showing busy workqueues and worker pools: [ 9558.005601] workqueue events: flags=0x0 [ 9558.006996] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9558.020903] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9558.034213] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9558.042594] pending: cpu_latency_work [ptlrpc] [ 9558.047863] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9558.052353] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9558.062323] workqueue events_long: flags=0x0 [ 9558.065261] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9558.070099] pending: lu_objects_flush [obdclass] [ 9558.080456] workqueue events_unbound: flags=0x2 [ 9558.083304] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9558.085485] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9558.091179] workqueue events_power_efficient: flags=0x80 [ 9558.093742] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9558.100446] pending: check_lifetime, do_cache_clean [ 9558.103132] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9558.109528] pending: neigh_periodic_work, neigh_periodic_work [ 9558.115531] workqueue mm_percpu_wq: flags=0x8 [ 9558.122024] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9558.130742] pending: vmstat_update, lru_add_drain_per_cpu [ 9558.151599] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9558.161446] pending: vmstat_update, lru_add_drain_per_cpu [ 9558.165270] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9558.169681] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9558.174483] workqueue xprtiod: flags=0xa [ 9558.177118] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9558.181249] pending: xprt_autoclose, xs_error_handle [ 9558.183775] workqueue ipv6_addrconf: flags=0x40008 [ 9558.185402] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9558.191576] pending: addrconf_verify_work [ 9558.194369] workqueue knbd0-recv: flags=0x1a [ 9558.196021] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9558.199329] in-flight: 651:recv_work [ 9558.200754] workqueue ptlrpc_pinger: flags=0x2 [ 9558.207189] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9558.214425] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9558.219914] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6921s workers=4 idle: 15979 31824 8 [ 9558.224526] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9548s workers=2 idle: 654 [ 9560.057618] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9560.060813] Modules linked in: lustre(O) osp(O) [ 9560.061215] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9560.061223] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log [ 9560.069849] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9560.070070] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9560.070077] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9560.070080] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9560.070093] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9560.070098] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9560.070102] RAX: 0000000000007519 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9560.070104] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9560.070107] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9560.070110] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9560.070112] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9560.070115] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9560.070118] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9560.070133] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9560.070140] Call Trace: [ 9560.070146] [ 9560.070152] ? show_regs.cold.9+0x22/0x2f [ 9560.070163] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9560.070170] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9560.070174] ? watchdog+0x40/0x40 [ 9560.070179] ? hrtimer_interrupt+0x128/0x2c0 [ 9560.070184] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9560.070191] ? apic_timer_interrupt+0xf/0x20 [ 9560.070196] [ 9560.070198] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9560.070205] do_raw_spin_lock+0x177/0x190 [ 9560.070210] _raw_spin_lock+0x21/0x30 [ 9560.070215] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9560.070510] ? do_raw_spin_unlock+0x75/0x190 [ 9560.070515] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9560.070717] ? do_wait_intr+0xf0/0xf0 [ 9560.070726] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9560.073054] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9560.073498] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9560.083556] dm_mod sha512_ssse3 sha512_generic [ 9560.083665] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9560.083671] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9560.083678] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9560.083695] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9560.083700] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9560.083708] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9560.083711] RDX: 000000000000558c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9560.083714] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9560.083717] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9560.083719] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9560.083727] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9560.083730] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9560.083733] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9560.083743] Call Trace: [ 9560.083753] [ 9560.083759] ? show_regs.cold.9+0x22/0x2f [ 9560.083773] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9560.083784] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9560.083788] ? watchdog+0x40/0x40 [ 9560.083796] ? hrtimer_interrupt+0x128/0x2c0 [ 9560.083801] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9560.083813] ? apic_timer_interrupt+0xf/0x20 [ 9560.083821] [ 9560.083825] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9560.083830] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9560.083839] do_raw_spin_lock+0x177/0x190 [ 9560.083844] _raw_spin_lock+0x21/0x30 [ 9560.083919] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9560.084501] obd_mod_free+0x46/0x140 [ptlrpc] [ 9560.124589] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9560.124733] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9560.124778] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9560.124810] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9560.124841] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9560.158086] mdc_commit_open+0x91/0xb0 [mdc] [ 9560.158198] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9560.158629] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9560.158846] after_reply+0xe2c/0x1780 [ptlrpc] [ 9560.159738] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9560.159947] ? __next_timer_interrupt+0x160/0x160 [ 9560.168104] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9560.174888] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9560.176200] ? do_wait_intr+0xf0/0xf0 [ 9560.181650] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9560.190677] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9560.190996] kthread+0x1d1/0x200 [ 9560.191007] ? set_kthread_struct+0x70/0x70 [ 9560.191011] ret_from_fork+0x1f/0x30 [ 9560.626464] lmv_intent_open+0x338/0x1150 [lmv] [ 9560.633661] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9560.647963] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9560.649780] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9560.665262] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9560.666857] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9560.683298] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9560.684958] ? do_raw_spin_unlock+0x75/0x190 [ 9560.692183] ? _raw_spin_unlock+0x12/0x30 [ 9560.694010] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9560.699813] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9560.703991] ll_atomic_open+0x36a/0xd70 [lustre] [ 9560.706571] lookup_open+0x356/0xb40 [ 9560.712461] do_last+0x313/0xfc0 [ 9560.716308] ? nd_jump_root+0xe5/0x160 [ 9560.717936] ? path_init+0x437/0x520 [ 9560.720475] path_openat+0xf7/0x500 [ 9560.722812] ? do_raw_spin_unlock+0x75/0x190 [ 9560.724877] do_filp_open+0x99/0x140 [ 9560.731622] ? getname_flags+0x6e/0x330 [ 9560.737087] ? __check_object_size+0xff/0x256 [ 9560.746063] ? do_raw_spin_unlock+0x75/0x190 [ 9560.751643] ? _raw_spin_unlock+0x12/0x30 [ 9560.759948] do_sys_openat2+0x2b4/0x410 [ 9560.762274] do_sys_open+0x73/0xa0 [ 9560.768660] __x64_sys_openat+0x24/0x30 [ 9560.773098] do_syscall_64+0xc1/0x440 [ 9560.779043] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9560.784409] RIP: 0033:0x7f53425a5332 [ 9560.788459] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9560.812729] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9560.823407] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9560.830060] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9560.837918] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9560.842989] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9560.854532] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9576.060039] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 9576.068370] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9576.108607] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9576.117042] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9576.124736] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9576.134101] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9576.150483] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9576.158624] RAX: 0000000000007e0f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9576.162363] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9576.164155] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9576.170074] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9576.177382] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9576.187060] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9576.193073] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9576.202077] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9576.210264] Call Trace: [ 9576.211508] [ 9576.212680] ? show_regs.cold.9+0x22/0x2f [ 9576.215893] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9576.224091] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9576.244318] ? watchdog+0x40/0x40 [ 9576.254135] ? hrtimer_interrupt+0x128/0x2c0 [ 9576.262347] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9576.276370] ? apic_timer_interrupt+0xf/0x20 [ 9576.277987] [ 9576.278773] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9576.297407] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9576.302152] do_raw_spin_lock+0x177/0x190 [ 9576.308439] _raw_spin_lock+0x21/0x30 [ 9576.313620] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9576.326498] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9576.341015] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9576.350383] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9576.356133] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9576.360240] ? woken_wake_function+0x30/0x30 [ 9576.363510] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9576.365770] kthread+0x1d1/0x200 [ 9576.366747] ? set_kthread_struct+0x70/0x70 [ 9576.368024] ret_from_fork+0x1f/0x30 [ 9584.055055] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 9584.067607] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9584.124208] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9584.133795] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9584.142020] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9584.148249] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9584.152668] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9584.164634] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 9584.171985] RAX: 0000000000007d42 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9584.179332] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9584.183442] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9584.188955] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9584.193076] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9584.196436] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9584.205083] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9584.217803] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9584.221998] Call Trace: [ 9584.222901] [ 9584.223944] ? show_regs.cold.9+0x22/0x2f [ 9584.227008] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9584.234938] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9584.241007] ? watchdog+0x40/0x40 [ 9584.242267] ? hrtimer_interrupt+0x128/0x2c0 [ 9584.250100] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9584.254635] ? apic_timer_interrupt+0xf/0x20 [ 9584.258669] [ 9584.259779] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9584.265606] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9584.270584] ? newidle_balance.isra.93+0x207/0x580 [ 9584.275760] do_raw_spin_lock+0x177/0x190 [ 9584.278034] _raw_spin_lock+0x21/0x30 [ 9584.281209] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9584.285938] process_one_work+0x2c8/0x700 [ 9584.287646] worker_thread+0x296/0x6e0 [ 9584.295415] ? rescuer_thread+0x570/0x570 [ 9584.300022] kthread+0x1d1/0x200 [ 9584.303300] ? set_kthread_struct+0x70/0x70 [ 9584.312037] ret_from_fork+0x1f/0x30 [ 9588.055796] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9588.057004] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9588.060995] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9588.061086] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9588.061093] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9588.061096] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9588.061109] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9588.061113] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9588.061118] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9588.061120] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9588.061123] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9588.061125] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9588.061128] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9588.061131] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9588.061134] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9588.061137] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9588.061145] Call Trace: [ 9588.061151] [ 9588.061156] ? show_regs.cold.9+0x22/0x2f [ 9588.061167] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9588.061173] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9588.061177] ? watchdog+0x40/0x40 [ 9588.061182] ? hrtimer_interrupt+0x128/0x2c0 [ 9588.061185] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9588.061192] ? apic_timer_interrupt+0xf/0x20 [ 9588.061197] [ 9588.061200] ? native_safe_halt+0x2e/0x50 [ 9588.061204] kvm_wait+0xcd/0xe0 [ 9588.061211] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9588.061220] do_raw_spin_lock+0x177/0x190 [ 9588.061225] _raw_spin_lock+0x21/0x30 [ 9588.061229] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9588.061513] ? do_raw_spin_unlock+0x75/0x190 [ 9588.061520] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9588.061759] ? do_wait_intr+0xf0/0xf0 [ 9588.061768] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9588.061999] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9588.062235] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9588.062441] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9588.062558] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9588.062604] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9588.076068] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9588.076147] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9588.076156] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9588.076158] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9588.076172] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9588.076177] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9588.076181] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9588.076184] RDX: 0000000000003c6f RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9588.076186] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9588.076188] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9588.076191] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9588.076194] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9588.076197] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9588.076199] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9588.076208] Call Trace: [ 9588.076215] [ 9588.076223] ? show_regs.cold.9+0x22/0x2f [ 9588.076233] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9588.076240] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9588.076245] ? watchdog+0x40/0x40 [ 9588.076249] ? hrtimer_interrupt+0x128/0x2c0 [ 9588.076254] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9588.076262] ? apic_timer_interrupt+0xf/0x20 [ 9588.076268] [ 9588.081447] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9588.112238] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9588.115909] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9588.119017] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9588.121150] do_raw_spin_lock+0x177/0x190 [ 9588.128242] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9588.131772] _raw_spin_lock+0x21/0x30 [ 9588.131792] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9588.132051] obd_mod_free+0x46/0x140 [ptlrpc] [ 9588.132244] mdc_commit_open+0x91/0xb0 [mdc] [ 9588.132283] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9588.132489] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9588.132696] after_reply+0xe2c/0x1780 [ptlrpc] [ 9588.132897] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9588.133089] ? __next_timer_interrupt+0x160/0x160 [ 9588.133099] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9588.133298] ? do_wait_intr+0xf0/0xf0 [ 9588.133308] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9588.133506] kthread+0x1d1/0x200 [ 9588.133512] ? set_kthread_struct+0x70/0x70 [ 9588.133516] ret_from_fork+0x1f/0x30 [ 9588.507269] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9588.511686] lmv_intent_open+0x338/0x1150 [lmv] [ 9588.517697] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9588.523969] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9588.528126] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9588.531724] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9588.533861] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9588.538287] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9588.544123] ? do_raw_spin_unlock+0x75/0x190 [ 9588.549671] ? _raw_spin_unlock+0x12/0x30 [ 9588.551116] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9588.558187] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9588.560009] ll_atomic_open+0x36a/0xd70 [lustre] [ 9588.564645] lookup_open+0x356/0xb40 [ 9588.566818] do_last+0x313/0xfc0 [ 9588.568914] ? nd_jump_root+0xe5/0x160 [ 9588.569993] ? path_init+0x437/0x520 [ 9588.571733] path_openat+0xf7/0x500 [ 9588.573342] ? do_raw_spin_unlock+0x75/0x190 [ 9588.575745] do_filp_open+0x99/0x140 [ 9588.585729] ? getname_flags+0x6e/0x330 [ 9588.587121] ? __check_object_size+0xff/0x256 [ 9588.588537] ? do_raw_spin_unlock+0x75/0x190 [ 9588.590673] ? _raw_spin_unlock+0x12/0x30 [ 9588.591786] do_sys_openat2+0x2b4/0x410 [ 9588.594315] do_sys_open+0x73/0xa0 [ 9588.595634] __x64_sys_openat+0x24/0x30 [ 9588.596913] do_syscall_64+0xc1/0x440 [ 9588.600518] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9588.603118] RIP: 0033:0x7f53425a5332 [ 9588.605367] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9588.621755] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9588.625918] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9588.632551] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9588.644040] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9588.648120] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9588.655646] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9588.705443] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6951s! [ 9588.713528] Showing busy workqueues and worker pools: [ 9588.723115] workqueue events: flags=0x0 [ 9588.726075] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9588.732043] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9588.739410] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9588.743241] pending: cpu_latency_work [ptlrpc] [ 9588.747121] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9588.751892] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9588.756525] workqueue events_long: flags=0x0 [ 9588.762559] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9588.765480] pending: lu_objects_flush [obdclass] [ 9588.767197] workqueue events_unbound: flags=0x2 [ 9588.768536] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9588.779846] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9588.783465] workqueue events_power_efficient: flags=0x80 [ 9588.785022] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9588.792790] pending: check_lifetime, do_cache_clean [ 9588.795656] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9588.797806] pending: neigh_periodic_work, neigh_periodic_work [ 9588.802084] workqueue mm_percpu_wq: flags=0x8 [ 9588.806827] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9588.810317] pending: vmstat_update, lru_add_drain_per_cpu [ 9588.817098] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9588.821067] pending: vmstat_update, lru_add_drain_per_cpu [ 9588.827855] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9588.834006] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9588.843597] workqueue xprtiod: flags=0xa [ 9588.845454] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9588.849744] pending: xprt_autoclose, xs_error_handle [ 9588.854436] workqueue ipv6_addrconf: flags=0x40008 [ 9588.856700] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9588.859903] pending: addrconf_verify_work [ 9588.866402] workqueue knbd0-recv: flags=0x1a [ 9588.868001] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9588.872823] in-flight: 651:recv_work [ 9588.874957] workqueue ptlrpc_pinger: flags=0x2 [ 9588.876794] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9588.879845] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9588.883437] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6951s workers=4 idle: 15979 31824 8 [ 9588.897109] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9579s workers=2 idle: 654 [ 9604.058666] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 9604.060798] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9604.108617] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9604.113665] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9604.117854] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9604.120595] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9604.129118] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9604.132803] RAX: 0000000000005b17 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9604.136625] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9604.140431] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9604.144223] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9604.147635] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9604.151075] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9604.155195] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9604.158454] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9604.162162] Call Trace: [ 9604.163321] [ 9604.164290] ? show_regs.cold.9+0x22/0x2f [ 9604.166296] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9604.168084] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9604.169695] ? watchdog+0x40/0x40 [ 9604.170807] ? hrtimer_interrupt+0x128/0x2c0 [ 9604.172423] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9604.174891] ? apic_timer_interrupt+0xf/0x20 [ 9604.177048] [ 9604.178252] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9604.180594] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9604.182289] do_raw_spin_lock+0x177/0x190 [ 9604.184879] _raw_spin_lock+0x21/0x30 [ 9604.186682] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9604.190951] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9604.193145] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9604.197049] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9604.200332] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9604.204346] ? woken_wake_function+0x30/0x30 [ 9604.205364] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9604.207625] kthread+0x1d1/0x200 [ 9604.208667] ? set_kthread_struct+0x70/0x70 [ 9604.210061] ret_from_fork+0x1f/0x30 [ 9616.052015] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 9616.054432] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9616.055606] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9616.060240] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9616.060253] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9616.060329] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9616.060334] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9616.060337] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9616.060351] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9616.060355] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9616.060360] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9616.060362] RDX: 0000000000002d96 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9616.060365] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9616.060367] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9616.060370] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9616.060373] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9616.060376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9616.060378] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9616.060388] Call Trace: [ 9616.060393] [ 9616.060399] ? show_regs.cold.9+0x22/0x2f [ 9616.060408] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9616.060414] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9616.060419] ? watchdog+0x40/0x40 [ 9616.060423] ? hrtimer_interrupt+0x128/0x2c0 [ 9616.060427] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9616.060435] ? apic_timer_interrupt+0xf/0x20 [ 9616.060439] [ 9616.060442] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9616.060447] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9616.060452] do_raw_spin_lock+0x177/0x190 [ 9616.060456] _raw_spin_lock+0x21/0x30 [ 9616.060462] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9616.060725] obd_mod_free+0x46/0x140 [ptlrpc] [ 9616.060930] mdc_commit_open+0x91/0xb0 [mdc] [ 9616.060968] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9616.061198] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9616.061413] after_reply+0xe2c/0x1780 [ptlrpc] [ 9616.061614] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9616.061817] ? __next_timer_interrupt+0x160/0x160 [ 9616.061827] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9616.062046] ? do_wait_intr+0xf0/0xf0 [ 9616.062055] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9616.062284] kthread+0x1d1/0x200 [ 9616.062290] ? set_kthread_struct+0x70/0x70 [ 9616.062294] ret_from_fork+0x1f/0x30 [ 9616.073196] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9616.073210] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9616.073218] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9616.073487] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9616.073497] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9616.073501] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9616.073505] RAX: 0000000000006f44 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9616.073508] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9616.073510] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9616.073512] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9616.073520] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9616.073523] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9616.073526] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9616.073528] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9616.073534] Call Trace: [ 9616.073539] [ 9616.073544] ? show_regs.cold.9+0x22/0x2f [ 9616.073553] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9616.073558] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9616.073563] ? watchdog+0x40/0x40 [ 9616.073567] ? hrtimer_interrupt+0x128/0x2c0 [ 9616.073571] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9616.073578] ? apic_timer_interrupt+0xf/0x20 [ 9616.073583] [ 9616.073586] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9616.073591] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9616.073595] ? newidle_balance.isra.93+0x207/0x580 [ 9616.073599] do_raw_spin_lock+0x177/0x190 [ 9616.073604] _raw_spin_lock+0x21/0x30 [ 9616.073609] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9616.073809] process_one_work+0x2c8/0x700 [ 9616.073816] worker_thread+0x296/0x6e0 [ 9616.073821] ? rescuer_thread+0x570/0x570 [ 9616.073824] kthread+0x1d1/0x200 [ 9616.073828] ? set_kthread_struct+0x70/0x70 [ 9616.073832] ret_from_fork+0x1f/0x30 [ 9616.479106] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9616.517797] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9616.526105] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9616.530039] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9616.536761] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9616.556812] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9616.561836] RAX: 0000000000003551 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9616.568855] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9616.572787] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9616.578466] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9616.585894] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9616.589792] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9616.594614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9616.599787] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9616.603849] Call Trace: [ 9616.607973] [ 9616.608982] ? show_regs.cold.9+0x22/0x2f [ 9616.612779] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9616.616544] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9616.623631] ? watchdog+0x40/0x40 [ 9616.625491] ? hrtimer_interrupt+0x128/0x2c0 [ 9616.627711] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9616.630370] ? apic_timer_interrupt+0xf/0x20 [ 9616.631546] [ 9616.635798] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9616.638901] do_raw_spin_lock+0x177/0x190 [ 9616.645291] _raw_spin_lock+0x21/0x30 [ 9616.647663] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9616.652623] ? do_raw_spin_unlock+0x75/0x190 [ 9616.654596] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9616.657269] ? do_wait_intr+0xf0/0xf0 [ 9616.658585] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9616.675541] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9616.686053] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9616.692458] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9616.694305] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9616.701140] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9616.703821] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9616.708861] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9616.713216] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9616.718481] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9616.721196] lmv_intent_open+0x338/0x1150 [lmv] [ 9616.723856] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9616.729773] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9616.737175] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9616.742254] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9616.744646] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9616.747614] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9616.750397] ? do_raw_spin_unlock+0x75/0x190 [ 9616.752074] ? _raw_spin_unlock+0x12/0x30 [ 9616.758273] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9616.760875] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9616.763093] ll_atomic_open+0x36a/0xd70 [lustre] [ 9616.765046] lookup_open+0x356/0xb40 [ 9616.769983] do_last+0x313/0xfc0 [ 9616.771914] ? nd_jump_root+0xe5/0x160 [ 9616.773766] ? path_init+0x437/0x520 [ 9616.776040] path_openat+0xf7/0x500 [ 9616.779751] ? do_raw_spin_unlock+0x75/0x190 [ 9616.781788] do_filp_open+0x99/0x140 [ 9616.786015] ? getname_flags+0x6e/0x330 [ 9616.788044] ? __check_object_size+0xff/0x256 [ 9616.795221] ? do_raw_spin_unlock+0x75/0x190 [ 9616.796728] ? _raw_spin_unlock+0x12/0x30 [ 9616.799883] do_sys_openat2+0x2b4/0x410 [ 9616.807882] do_sys_open+0x73/0xa0 [ 9616.809527] __x64_sys_openat+0x24/0x30 [ 9616.816822] do_syscall_64+0xc1/0x440 [ 9616.818834] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9616.825791] RIP: 0033:0x7f53425a5332 [ 9616.828124] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9616.850856] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9616.855459] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9616.859123] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9616.864835] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9616.868382] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9616.874829] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9618.918030] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 6981s! [ 9618.931768] Showing busy workqueues and worker pools: [ 9618.943555] workqueue events: flags=0x0 [ 9618.947688] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9618.950140] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9618.962616] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9618.976608] pending: cpu_latency_work [ptlrpc] [ 9618.988902] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9618.998809] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9619.004816] workqueue events_long: flags=0x0 [ 9619.016211] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9619.023223] pending: lu_objects_flush [obdclass] [ 9619.042767] workqueue events_unbound: flags=0x2 [ 9619.044449] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9619.053760] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9619.063107] workqueue events_power_efficient: flags=0x80 [ 9619.066322] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9619.080804] pending: check_lifetime, do_cache_clean [ 9619.084346] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9619.089998] pending: neigh_periodic_work, neigh_periodic_work [ 9619.095729] workqueue mm_percpu_wq: flags=0x8 [ 9619.098627] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9619.104657] pending: vmstat_update, lru_add_drain_per_cpu [ 9619.111254] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9619.116855] pending: vmstat_update, lru_add_drain_per_cpu [ 9619.123977] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9619.129693] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9619.135908] workqueue xprtiod: flags=0xa [ 9619.137417] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9619.145958] pending: xprt_autoclose, xs_error_handle [ 9619.148836] workqueue ipv6_addrconf: flags=0x40008 [ 9619.163853] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9619.168440] pending: addrconf_verify_work [ 9619.174592] workqueue knbd0-recv: flags=0x1a [ 9619.177917] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9619.189858] in-flight: 651:recv_work [ 9619.195195] workqueue ptlrpc_pinger: flags=0x2 [ 9619.206056] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9619.207708] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9619.212760] pool 8: cpus=0-3 flags=0x4 nice=0 hung=6982s workers=4 idle: 15979 31824 8 [ 9619.226247] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9609s workers=2 idle: 654 [ 9632.058544] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 9632.067702] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9632.107409] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9632.114533] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9632.119932] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9632.122834] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9632.133482] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9632.139507] RAX: 0000000000007f06 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9632.142750] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9632.148323] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9632.154297] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9632.157501] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9632.161370] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9632.166101] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9632.169589] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9632.173741] Call Trace: [ 9632.174782] [ 9632.177328] ? show_regs.cold.9+0x22/0x2f [ 9632.179677] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9632.181831] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9632.189523] ? watchdog+0x40/0x40 [ 9632.192018] ? hrtimer_interrupt+0x128/0x2c0 [ 9632.193547] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9632.199434] ? apic_timer_interrupt+0xf/0x20 [ 9632.202481] [ 9632.203529] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9632.209566] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9632.212017] do_raw_spin_lock+0x177/0x190 [ 9632.213450] _raw_spin_lock+0x21/0x30 [ 9632.214707] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9632.218248] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9632.222281] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9632.225556] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9632.228377] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9632.232487] ? woken_wake_function+0x30/0x30 [ 9632.235823] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9632.238735] kthread+0x1d1/0x200 [ 9632.240357] ? set_kthread_struct+0x70/0x70 [ 9632.242385] ret_from_fork+0x1f/0x30 [ 9644.052038] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 9644.054578] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9644.057011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9644.058382] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9644.058391] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9644.058460] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9644.058466] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9644.058468] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9644.058483] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9644.058487] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9644.058492] RAX: 000000000000380b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9644.058494] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9644.058496] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9644.058498] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9644.058501] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9644.058504] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9644.058506] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9644.058509] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9644.060775] Call Trace: [ 9644.060786] [ 9644.060793] ? show_regs.cold.9+0x22/0x2f [ 9644.060807] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9644.060814] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9644.060819] ? watchdog+0x40/0x40 [ 9644.060824] ? hrtimer_interrupt+0x128/0x2c0 [ 9644.060828] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9644.060836] ? apic_timer_interrupt+0xf/0x20 [ 9644.060841] [ 9644.060844] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9644.060854] do_raw_spin_lock+0x177/0x190 [ 9644.060859] _raw_spin_lock+0x21/0x30 [ 9644.060865] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9644.061127] ? do_raw_spin_unlock+0x75/0x190 [ 9644.061134] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9644.061330] ? do_wait_intr+0xf0/0xf0 [ 9644.061339] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9644.061566] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9644.061796] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9644.062013] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9644.062128] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9644.062165] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9644.062195] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9644.062223] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9644.062308] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9644.062535] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9644.062570] lmv_intent_open+0x338/0x1150 [lmv] [ 9644.062598] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9644.062701] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9644.062796] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9644.062821] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9644.062920] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9644.063014] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9644.063097] ? do_raw_spin_unlock+0x75/0x190 [ 9644.063104] ? _raw_spin_unlock+0x12/0x30 [ 9644.063110] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9644.063134] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9644.063155] ll_atomic_open+0x36a/0xd70 [lustre] [ 9644.063240] lookup_open+0x356/0xb40 [ 9644.063250] do_last+0x313/0xfc0 [ 9644.063256] ? nd_jump_root+0xe5/0x160 [ 9644.063260] ? path_init+0x437/0x520 [ 9644.063265] path_openat+0xf7/0x500 [ 9644.063270] ? do_raw_spin_unlock+0x75/0x190 [ 9644.063275] do_filp_open+0x99/0x140 [ 9644.063282] ? getname_flags+0x6e/0x330 [ 9644.063286] ? __check_object_size+0xff/0x256 [ 9644.063294] ? do_raw_spin_unlock+0x75/0x190 [ 9644.063300] ? _raw_spin_unlock+0x12/0x30 [ 9644.063306] do_sys_openat2+0x2b4/0x410 [ 9644.063311] do_sys_open+0x73/0xa0 [ 9644.063315] __x64_sys_openat+0x24/0x30 [ 9644.063319] do_syscall_64+0xc1/0x440 [ 9644.063326] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9644.063332] RIP: 0033:0x7f53425a5332 [ 9644.063340] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9644.063344] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9644.063349] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9644.063351] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9644.063354] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9644.063356] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9644.063359] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9644.076876] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9644.076889] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9644.076896] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9644.082115] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9644.082198] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9644.082206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9644.082209] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9644.082223] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9644.082228] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9644.082232] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9644.082235] RDX: 0000000000006737 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9644.082238] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9644.082241] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9644.082244] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9644.082247] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9644.082250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9644.082252] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9644.082261] Call Trace: [ 9644.082327] [ 9644.082333] ? show_regs.cold.9+0x22/0x2f [ 9644.082345] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9644.082352] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9644.082356] ? watchdog+0x40/0x40 [ 9644.082360] ? hrtimer_interrupt+0x128/0x2c0 [ 9644.082365] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9644.082373] ? apic_timer_interrupt+0xf/0x20 [ 9644.082378] [ 9644.082381] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9644.082387] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9644.082392] do_raw_spin_lock+0x177/0x190 [ 9644.082397] _raw_spin_lock+0x21/0x30 [ 9644.082403] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9644.082842] obd_mod_free+0x46/0x140 [ptlrpc] [ 9644.087685] [ 9644.087694] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9644.087711] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9644.087715] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9644.087720] RAX: 0000000000000e9d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9644.087722] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9644.087724] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9644.087726] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9644.087729] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9644.087732] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9644.087735] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9644.087738] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9644.087747] Call Trace: [ 9644.087751] [ 9644.087757] ? show_regs.cold.9+0x22/0x2f [ 9644.087765] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9644.087771] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9644.087775] ? watchdog+0x40/0x40 [ 9644.087779] ? hrtimer_interrupt+0x128/0x2c0 [ 9644.087783] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9644.087791] ? apic_timer_interrupt+0xf/0x20 [ 9644.087796] [ 9644.087799] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9644.087804] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9644.087808] ? newidle_balance.isra.93+0x207/0x580 [ 9644.087815] do_raw_spin_lock+0x177/0x190 [ 9644.087820] _raw_spin_lock+0x21/0x30 [ 9644.087837] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9644.088242] process_one_work+0x2c8/0x700 [ 9644.088249] worker_thread+0x296/0x6e0 [ 9644.088254] ? rescuer_thread+0x570/0x570 [ 9644.088258] kthread+0x1d1/0x200 [ 9644.088262] ? set_kthread_struct+0x70/0x70 [ 9644.088266] ret_from_fork+0x1f/0x30 [ 9644.480215] mdc_commit_open+0x91/0xb0 [mdc] [ 9644.482417] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9644.487625] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9644.491300] after_reply+0xe2c/0x1780 [ptlrpc] [ 9644.494465] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9644.496999] ? __next_timer_interrupt+0x160/0x160 [ 9644.499202] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9644.501040] ? do_wait_intr+0xf0/0xf0 [ 9644.502484] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9644.504871] kthread+0x1d1/0x200 [ 9644.506245] ? set_kthread_struct+0x70/0x70 [ 9644.508046] ret_from_fork+0x1f/0x30 [ 9649.631068] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7012s! [ 9649.641210] Showing busy workqueues and worker pools: [ 9649.645510] workqueue events: flags=0x0 [ 9649.647916] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9649.657018] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9649.666149] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9649.670548] pending: cpu_latency_work [ptlrpc] [ 9649.679732] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9649.684012] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9649.695713] workqueue events_long: flags=0x0 [ 9649.697893] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9649.703301] pending: lu_objects_flush [obdclass] [ 9649.709788] workqueue events_unbound: flags=0x2 [ 9649.711243] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9649.718080] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9649.726174] workqueue events_power_efficient: flags=0x80 [ 9649.730111] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9649.742134] pending: check_lifetime, do_cache_clean [ 9649.749122] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9649.756046] pending: neigh_periodic_work, neigh_periodic_work [ 9649.760130] workqueue mm_percpu_wq: flags=0x8 [ 9649.770121] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9649.772205] pending: vmstat_update, lru_add_drain_per_cpu [ 9649.782883] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9649.792979] pending: vmstat_update, lru_add_drain_per_cpu [ 9649.803409] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9649.815109] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9649.825016] workqueue xprtiod: flags=0xa [ 9649.827410] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9649.838095] pending: xprt_autoclose, xs_error_handle [ 9649.840854] workqueue ipv6_addrconf: flags=0x40008 [ 9649.848130] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9649.852583] pending: addrconf_verify_work [ 9649.860108] workqueue knbd0-recv: flags=0x1a [ 9649.863330] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9649.871161] in-flight: 651:recv_work [ 9649.877459] workqueue ptlrpc_pinger: flags=0x2 [ 9649.880610] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9649.888244] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9649.902416] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7012s workers=4 idle: 15979 31824 8 [ 9649.904981] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9640s workers=2 idle: 654 [ 9660.061302] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 9660.068580] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9660.108087] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9660.121612] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9660.128578] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9660.131525] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9660.144257] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 9660.148919] RAX: 0000000000003d22 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9660.153430] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9660.157839] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9660.161539] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9660.171162] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9660.178951] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9660.183786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9660.185892] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9660.188102] Call Trace: [ 9660.189002] [ 9660.189523] ? show_regs.cold.9+0x22/0x2f [ 9660.191177] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9660.192955] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9660.194474] ? watchdog+0x40/0x40 [ 9660.198112] ? hrtimer_interrupt+0x128/0x2c0 [ 9660.203204] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9660.207697] ? apic_timer_interrupt+0xf/0x20 [ 9660.209304] [ 9660.211659] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9660.216070] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9660.218923] do_raw_spin_lock+0x177/0x190 [ 9660.220961] _raw_spin_lock+0x21/0x30 [ 9660.222452] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9660.224947] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9660.230623] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9660.234480] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9660.239268] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9660.242629] ? woken_wake_function+0x30/0x30 [ 9660.244279] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9660.248721] kthread+0x1d1/0x200 [ 9660.255846] ? set_kthread_struct+0x70/0x70 [ 9660.258704] ret_from_fork+0x1f/0x30 [ 9672.055114] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9672.057031] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9672.060475] Modules linked in: [ 9672.064109] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9672.064196] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9672.064204] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9672.064207] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9672.064221] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9672.064226] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9672.064230] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9672.064233] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9672.064235] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9672.064238] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9672.064241] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9672.064244] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9672.064247] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9672.064249] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9672.064259] Call Trace: [ 9672.064264] [ 9672.064271] ? show_regs.cold.9+0x22/0x2f [ 9672.064281] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9672.064287] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9672.064292] ? watchdog+0x40/0x40 [ 9672.064296] ? hrtimer_interrupt+0x128/0x2c0 [ 9672.064300] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9672.064306] ? apic_timer_interrupt+0xf/0x20 [ 9672.064311] [ 9672.064315] ? native_safe_halt+0x2e/0x50 [ 9672.064319] kvm_wait+0xcd/0xe0 [ 9672.064326] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9672.064332] do_raw_spin_lock+0x177/0x190 [ 9672.064337] _raw_spin_lock+0x21/0x30 [ 9672.064343] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9672.064604] obd_mod_free+0x46/0x140 [ptlrpc] [ 9672.065087] mdc_commit_open+0x91/0xb0 [mdc] [ 9672.065133] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9672.068510] lustre(O) [ 9672.089177] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9672.089524] after_reply+0xe2c/0x1780 [ptlrpc] [ 9672.089730] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9672.090010] ? __next_timer_interrupt+0x160/0x160 [ 9672.090020] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9672.090249] ? do_wait_intr+0xf0/0xf0 [ 9672.090261] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9672.090467] kthread+0x1d1/0x200 [ 9672.090473] ? set_kthread_struct+0x70/0x70 [ 9672.090477] ret_from_fork+0x1f/0x30 [ 9672.366512] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9672.416945] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9672.431607] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9672.441761] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9672.447125] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9672.462036] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9672.465126] RAX: 0000000000003191 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9672.477246] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9672.479330] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9672.488620] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9672.495462] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9672.503018] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9672.514026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9672.520907] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9672.530513] Call Trace: [ 9672.535647] [ 9672.537529] ? show_regs.cold.9+0x22/0x2f [ 9672.540456] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9672.544967] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9672.550987] ? watchdog+0x40/0x40 [ 9672.552053] ? hrtimer_interrupt+0x128/0x2c0 [ 9672.557025] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9672.566640] ? apic_timer_interrupt+0xf/0x20 [ 9672.569985] [ 9672.572848] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9672.576093] do_raw_spin_lock+0x177/0x190 [ 9672.583428] _raw_spin_lock+0x21/0x30 [ 9672.584737] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9672.593829] ? do_raw_spin_unlock+0x75/0x190 [ 9672.597444] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9672.607268] ? do_wait_intr+0xf0/0xf0 [ 9672.608403] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9672.615921] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9672.626663] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9672.633867] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9672.641018] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9672.654071] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9672.658831] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9672.665625] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9672.667586] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9672.670873] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9672.674352] lmv_intent_open+0x338/0x1150 [lmv] [ 9672.675807] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9672.679034] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9672.683319] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9672.685112] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9672.686974] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9672.691081] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9672.695293] ? do_raw_spin_unlock+0x75/0x190 [ 9672.696293] ? _raw_spin_unlock+0x12/0x30 [ 9672.697766] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9672.701863] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9672.703146] ll_atomic_open+0x36a/0xd70 [lustre] [ 9672.707018] lookup_open+0x356/0xb40 [ 9672.710004] do_last+0x313/0xfc0 [ 9672.710766] ? nd_jump_root+0xe5/0x160 [ 9672.713007] ? path_init+0x437/0x520 [ 9672.715867] path_openat+0xf7/0x500 [ 9672.717511] ? do_raw_spin_unlock+0x75/0x190 [ 9672.719071] do_filp_open+0x99/0x140 [ 9672.720462] ? getname_flags+0x6e/0x330 [ 9672.721950] ? __check_object_size+0xff/0x256 [ 9672.723730] ? do_raw_spin_unlock+0x75/0x190 [ 9672.725443] ? _raw_spin_unlock+0x12/0x30 [ 9672.727314] do_sys_openat2+0x2b4/0x410 [ 9672.730846] do_sys_open+0x73/0xa0 [ 9672.733135] __x64_sys_openat+0x24/0x30 [ 9672.736993] do_syscall_64+0xc1/0x440 [ 9672.740092] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9672.744119] RIP: 0033:0x7f53425a5332 [ 9672.747856] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9672.767295] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9672.772299] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9672.780265] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9672.786640] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9672.790492] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9672.793977] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9676.053149] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 9676.058454] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9676.090170] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9676.096762] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9676.102443] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9676.105262] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9676.107789] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9676.120615] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9676.124712] RAX: 0000000000007e97 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9676.128373] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9676.132627] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9676.141762] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9676.147768] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9676.151329] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9676.154966] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9676.156562] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9676.158883] Call Trace: [ 9676.159983] [ 9676.161250] ? show_regs.cold.9+0x22/0x2f [ 9676.163383] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9676.165888] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9676.168843] ? watchdog+0x40/0x40 [ 9676.170552] ? hrtimer_interrupt+0x128/0x2c0 [ 9676.173826] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9676.175662] ? apic_timer_interrupt+0xf/0x20 [ 9676.178520] [ 9676.179593] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9676.182914] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9676.186040] ? newidle_balance.isra.93+0x207/0x580 [ 9676.190441] do_raw_spin_lock+0x177/0x190 [ 9676.192363] _raw_spin_lock+0x21/0x30 [ 9676.194772] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9676.198579] process_one_work+0x2c8/0x700 [ 9676.200931] worker_thread+0x296/0x6e0 [ 9676.202929] ? rescuer_thread+0x570/0x570 [ 9676.204936] kthread+0x1d1/0x200 [ 9676.209126] ? set_kthread_struct+0x70/0x70 [ 9676.210769] ret_from_fork+0x1f/0x30 [ 9680.355146] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7043s! [ 9680.368323] Showing busy workqueues and worker pools: [ 9680.369935] workqueue events: flags=0x0 [ 9680.374482] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9680.381494] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9680.387420] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9680.394972] pending: cpu_latency_work [ptlrpc] [ 9680.399182] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9680.405802] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9680.413897] workqueue events_long: flags=0x0 [ 9680.417433] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9680.422400] pending: lu_objects_flush [obdclass] [ 9680.428899] workqueue events_unbound: flags=0x2 [ 9680.431789] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9680.435008] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9680.438672] workqueue events_power_efficient: flags=0x80 [ 9680.445149] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9680.454434] pending: check_lifetime, do_cache_clean [ 9680.456263] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9680.468355] pending: neigh_periodic_work, neigh_periodic_work [ 9680.475408] workqueue mm_percpu_wq: flags=0x8 [ 9680.478743] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9680.485228] pending: vmstat_update, lru_add_drain_per_cpu [ 9680.488441] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9680.491862] pending: vmstat_update, lru_add_drain_per_cpu [ 9680.494673] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9680.499182] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9680.502832] workqueue xprtiod: flags=0xa [ 9680.504853] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9680.512163] pending: xprt_autoclose, xs_error_handle [ 9680.514081] workqueue ipv6_addrconf: flags=0x40008 [ 9680.520513] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9680.527858] pending: addrconf_verify_work [ 9680.532276] workqueue knbd0-recv: flags=0x1a [ 9680.534904] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9680.540922] in-flight: 651:recv_work [ 9680.543486] workqueue ptlrpc_pinger: flags=0x2 [ 9680.546952] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9680.552758] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9680.557749] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7043s workers=4 idle: 15979 31824 8 [ 9680.564295] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9671s workers=2 idle: 654 [ 9688.058043] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 9688.067083] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9688.112132] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9688.122081] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9688.126545] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9688.130178] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9688.142534] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9688.146968] RAX: 0000000000006a83 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9688.150909] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9688.156520] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9688.161092] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9688.166270] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9688.171156] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9688.180178] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9688.183290] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9688.187928] Call Trace: [ 9688.189040] [ 9688.192153] ? show_regs.cold.9+0x22/0x2f [ 9688.194149] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9688.196625] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9688.198785] ? watchdog+0x40/0x40 [ 9688.201676] ? hrtimer_interrupt+0x128/0x2c0 [ 9688.203581] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9688.209940] ? apic_timer_interrupt+0xf/0x20 [ 9688.211732] [ 9688.213070] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9688.219804] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9688.222095] do_raw_spin_lock+0x177/0x190 [ 9688.227247] _raw_spin_lock+0x21/0x30 [ 9688.229577] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9688.233740] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9688.236092] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9688.239675] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9688.243336] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9688.246674] ? woken_wake_function+0x30/0x30 [ 9688.249659] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9688.253358] kthread+0x1d1/0x200 [ 9688.255587] ? set_kthread_struct+0x70/0x70 [ 9688.257824] ret_from_fork+0x1f/0x30 [ 9700.057015] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9700.058813] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9700.061848] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9700.061993] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9700.062010] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9700.062013] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9700.062031] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9700.062036] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9700.062041] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9700.062043] RDX: 0000000000003863 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9700.062101] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9700.062104] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9700.062106] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9700.062109] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9700.062112] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9700.062115] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9700.062123] Call Trace: [ 9700.062144] [ 9700.062151] ? show_regs.cold.9+0x22/0x2f [ 9700.062160] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9700.062168] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9700.062177] ? watchdog+0x40/0x40 [ 9700.062182] ? hrtimer_interrupt+0x128/0x2c0 [ 9700.062187] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9700.062201] ? apic_timer_interrupt+0xf/0x20 [ 9700.062206] [ 9700.062209] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9700.062214] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9700.062219] do_raw_spin_lock+0x177/0x190 [ 9700.062224] _raw_spin_lock+0x21/0x30 [ 9700.062230] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9700.062915] obd_mod_free+0x46/0x140 [ptlrpc] [ 9700.063164] mdc_commit_open+0x91/0xb0 [mdc] [ 9700.063214] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9700.071078] Modules linked in: [ 9700.103944] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9700.121352] lustre(O) [ 9700.124632] after_reply+0xe2c/0x1780 [ptlrpc] [ 9700.125007] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9700.125201] ? __next_timer_interrupt+0x160/0x160 [ 9700.125281] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9700.133517] osp(O) ofd(O) [ 9700.143170] ? do_wait_intr+0xf0/0xf0 [ 9700.143196] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9700.143575] kthread+0x1d1/0x200 [ 9700.143582] ? set_kthread_struct+0x70/0x70 [ 9700.143586] ret_from_fork+0x1f/0x30 [ 9700.342832] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9700.400154] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9700.419322] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9700.431744] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9700.439843] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9700.451358] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9700.460544] RAX: 0000000000007c14 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9700.465227] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9700.469726] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9700.478141] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9700.482012] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9700.488135] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9700.494246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9700.503156] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9700.508223] Call Trace: [ 9700.509022] [ 9700.509671] ? show_regs.cold.9+0x22/0x2f [ 9700.515194] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9700.519239] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9700.522283] ? watchdog+0x40/0x40 [ 9700.525145] ? hrtimer_interrupt+0x128/0x2c0 [ 9700.532019] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9700.535785] ? apic_timer_interrupt+0xf/0x20 [ 9700.537341] [ 9700.538277] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9700.540963] do_raw_spin_lock+0x177/0x190 [ 9700.542489] _raw_spin_lock+0x21/0x30 [ 9700.545108] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9700.547038] ? do_raw_spin_unlock+0x75/0x190 [ 9700.551958] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9700.558487] ? do_wait_intr+0xf0/0xf0 [ 9700.560236] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9700.562068] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9700.567371] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9700.576973] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9700.580092] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9700.585786] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9700.587389] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9700.592380] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9700.595576] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9700.599848] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9700.607296] lmv_intent_open+0x338/0x1150 [lmv] [ 9700.614188] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9700.625134] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9700.631152] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9700.636068] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9700.640149] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9700.643051] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9700.651416] ? do_raw_spin_unlock+0x75/0x190 [ 9700.655091] ? _raw_spin_unlock+0x12/0x30 [ 9700.657298] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9700.659855] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9700.661129] ll_atomic_open+0x36a/0xd70 [lustre] [ 9700.668262] lookup_open+0x356/0xb40 [ 9700.673036] do_last+0x313/0xfc0 [ 9700.677011] ? nd_jump_root+0xe5/0x160 [ 9700.680395] ? path_init+0x437/0x520 [ 9700.683265] path_openat+0xf7/0x500 [ 9700.687133] ? do_raw_spin_unlock+0x75/0x190 [ 9700.690281] do_filp_open+0x99/0x140 [ 9700.694164] ? getname_flags+0x6e/0x330 [ 9700.698802] ? __check_object_size+0xff/0x256 [ 9700.700398] ? do_raw_spin_unlock+0x75/0x190 [ 9700.706140] ? _raw_spin_unlock+0x12/0x30 [ 9700.710967] do_sys_openat2+0x2b4/0x410 [ 9700.713529] do_sys_open+0x73/0xa0 [ 9700.719315] __x64_sys_openat+0x24/0x30 [ 9700.724387] do_syscall_64+0xc1/0x440 [ 9700.727163] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9700.732139] RIP: 0033:0x7f53425a5332 [ 9700.734773] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9700.754908] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9700.765135] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9700.769278] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9700.777399] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9700.784965] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9700.792644] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9708.052812] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 9708.061131] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9708.111425] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9708.127467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9708.136164] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9708.143466] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9708.152915] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9708.172984] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9708.179463] RAX: 0000000000006bb2 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9708.189824] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9708.200141] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9708.214903] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9708.225242] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9708.240136] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9708.254605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9708.269639] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9708.272156] Call Trace: [ 9708.282170] [ 9708.285265] ? show_regs.cold.9+0x22/0x2f [ 9708.288185] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9708.296446] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9708.303249] ? watchdog+0x40/0x40 [ 9708.306383] ? hrtimer_interrupt+0x128/0x2c0 [ 9708.309111] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9708.313788] ? apic_timer_interrupt+0xf/0x20 [ 9708.316473] [ 9708.321163] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9708.329386] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9708.333471] ? newidle_balance.isra.93+0x207/0x580 [ 9708.336487] do_raw_spin_lock+0x177/0x190 [ 9708.339128] _raw_spin_lock+0x21/0x30 [ 9708.343166] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9708.347204] process_one_work+0x2c8/0x700 [ 9708.349368] worker_thread+0x296/0x6e0 [ 9708.356419] ? rescuer_thread+0x570/0x570 [ 9708.360312] kthread+0x1d1/0x200 [ 9708.363611] ? set_kthread_struct+0x70/0x70 [ 9708.365014] ret_from_fork+0x1f/0x30 [ 9711.075037] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7073s! [ 9711.084958] Showing busy workqueues and worker pools: [ 9711.099722] workqueue events: flags=0x0 [ 9711.111659] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9711.126994] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9711.150828] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9711.163488] pending: cpu_latency_work [ptlrpc] [ 9711.175569] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9711.177901] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9711.204891] workqueue events_long: flags=0x0 [ 9711.206466] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9711.219918] pending: lu_objects_flush [obdclass] [ 9711.235778] workqueue events_unbound: flags=0x2 [ 9711.245562] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9711.253570] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9711.260184] workqueue events_power_efficient: flags=0x80 [ 9711.278619] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9711.286970] pending: check_lifetime, do_cache_clean [ 9711.301018] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9711.314706] pending: neigh_periodic_work, neigh_periodic_work [ 9711.321517] workqueue mm_percpu_wq: flags=0x8 [ 9711.327085] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9711.340040] pending: vmstat_update, lru_add_drain_per_cpu [ 9711.351366] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9711.366035] pending: vmstat_update, lru_add_drain_per_cpu [ 9711.372151] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9711.380358] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9711.399652] workqueue xprtiod: flags=0xa [ 9711.409014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9711.428549] pending: xprt_autoclose, xs_error_handle [ 9711.441662] workqueue ipv6_addrconf: flags=0x40008 [ 9711.443332] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9711.455996] pending: addrconf_verify_work [ 9711.464212] workqueue knbd0-recv: flags=0x1a [ 9711.468013] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9711.470095] in-flight: 651:recv_work [ 9711.471863] workqueue ptlrpc_pinger: flags=0x2 [ 9711.476064] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9711.478839] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9711.483011] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7074s workers=4 idle: 15979 31824 8 [ 9711.490120] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9701s workers=2 idle: 654 [ 9714.613017] rcu: INFO: rcu_sched self-detected stall on CPU [ 9714.617921] rcu: 2-....: (4165320 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 9714.626647] (t=7620187 jiffies g=274185 q=65796) [ 9714.629824] rcu: rcu_sched kthread starved for 7078553 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [ 9714.638191] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9714.644363] rcu: RCU grace-period kthread stack dump: [ 9714.647273] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9714.652132] Call Trace: [ 9714.654137] __schedule+0x351/0xcb0 [ 9714.657952] schedule+0xc0/0x180 [ 9714.659213] schedule_timeout+0xb4/0x190 [ 9714.668450] ? do_raw_spin_unlock+0x75/0x190 [ 9714.672305] ? __next_timer_interrupt+0x160/0x160 [ 9714.678040] ? prepare_to_swait_event+0xf3/0x140 [ 9714.682577] rcu_gp_kthread+0x8dd/0x15b0 [ 9714.684560] ? force_qs_rnp+0x2c0/0x2c0 [ 9714.686782] kthread+0x1d1/0x200 [ 9714.688044] ? set_kthread_struct+0x70/0x70 [ 9714.692708] ret_from_fork+0x1f/0x30 [ 9714.694453] rcu: Stack dump where RCU GP kthread last ran: [ 9714.698651] Sending NMI from CPU 2 to CPUs 0: [ 9714.701103] NMI backtrace for cpu 0 [ 9714.701110] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9714.701112] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9714.701114] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9714.701117] RIP: 0010:__pv_queued_spin_lock_slowpath+0x302/0x6d0 [ 9714.701118] Code: 05 48 98 49 81 c5 40 41 03 00 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 <84> d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 [ 9714.701120] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 [ 9714.701122] RAX: 0000000000003c00 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9714.701124] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9714.701126] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9714.701314] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9714.701317] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9714.701318] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9714.701319] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9714.701321] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9714.701322] Call Trace: [ 9714.701323] [ 9714.701324] ? show_regs.cold.9+0x22/0x2f [ 9714.701325] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9714.701326] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9714.701328] ? nmi_handle+0xbd/0x210 [ 9714.701329] ? default_do_nmi+0x4b/0x1b0 [ 9714.701330] ? do_nmi+0x280/0x380 [ 9714.701331] ? end_repeat_nmi+0x16/0x1f [ 9714.701332] ? __pv_queued_spin_lock_slowpath+0x302/0x6d0 [ 9714.701333] ? __pv_queued_spin_lock_slowpath+0x302/0x6d0 [ 9714.701334] ? __pv_queued_spin_lock_slowpath+0x302/0x6d0 [ 9714.701335] [ 9714.701337] ? newidle_balance.isra.93+0x207/0x580 [ 9714.701337] do_raw_spin_lock+0x177/0x190 [ 9714.701339] _raw_spin_lock+0x21/0x30 [ 9714.701340] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9714.701341] process_one_work+0x2c8/0x700 [ 9714.701342] worker_thread+0x296/0x6e0 [ 9714.701343] ? rescuer_thread+0x570/0x570 [ 9714.701344] kthread+0x1d1/0x200 [ 9714.701345] ? set_kthread_struct+0x70/0x70 [ 9714.701346] ret_from_fork+0x1f/0x30 [ 9714.701687] Sending NMI from CPU 2 to CPUs 1: [ 9714.822181] NMI backtrace for cpu 1 [ 9714.822187] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9714.822190] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9714.822191] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9714.822193] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9714.822196] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [ 9714.822198] RAX: 0000000000005b82 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9714.822200] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9714.822201] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9714.822202] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9714.822204] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9714.822205] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9714.822207] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9714.822209] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9714.822210] Call Trace: [ 9714.822211] [ 9714.822212] ? show_regs.cold.9+0x22/0x2f [ 9714.822213] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9714.822215] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9714.822216] ? nmi_handle+0xbd/0x210 [ 9714.822217] ? default_do_nmi+0x4b/0x1b0 [ 9714.822218] ? do_nmi+0x280/0x380 [ 9714.822219] ? end_repeat_nmi+0x16/0x1f [ 9714.822220] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9714.822222] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9714.822223] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9714.822224] [ 9714.822225] do_raw_spin_lock+0x177/0x190 [ 9714.822226] _raw_spin_lock+0x21/0x30 [ 9714.822227] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9714.822228] ? do_raw_spin_unlock+0x75/0x190 [ 9714.822229] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9714.822230] ? do_wait_intr+0xf0/0xf0 [ 9714.822231] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9714.822232] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9714.822233] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9714.822235] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9714.822236] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9714.822237] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9714.822238] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9714.822239] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9714.822241] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9714.822242] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9714.822244] lmv_intent_open+0x338/0x1150 [lmv] [ 9714.822245] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9714.822246] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9714.822247] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9714.822248] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9714.822250] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9714.822251] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9714.822252] ? do_raw_spin_unlock+0x75/0x190 [ 9714.822253] ? _raw_spin_unlock+0x12/0x30 [ 9714.822254] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9714.822255] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9714.822256] ll_atomic_open+0x36a/0xd70 [lustre] [ 9714.822257] lookup_open+0x356/0xb40 [ 9714.822258] do_last+0x313/0xfc0 [ 9714.822259] ? nd_jump_root+0xe5/0x160 [ 9714.822259] ? path_init+0x437/0x520 [ 9714.822260] path_openat+0xf7/0x500 [ 9714.822262] ? do_raw_spin_unlock+0x75/0x190 [ 9714.822263] do_filp_open+0x99/0x140 [ 9714.822264] ? getname_flags+0x6e/0x330 [ 9714.822265] ? __check_object_size+0xff/0x256 [ 9714.822266] ? do_raw_spin_unlock+0x75/0x190 [ 9714.822267] ? _raw_spin_unlock+0x12/0x30 [ 9714.822268] do_sys_openat2+0x2b4/0x410 [ 9714.822269] do_sys_open+0x73/0xa0 [ 9714.822271] __x64_sys_openat+0x24/0x30 [ 9714.822272] do_syscall_64+0xc1/0x440 [ 9714.822273] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9714.822274] RIP: 0033:0x7f53425a5332 [ 9714.822275] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9714.822278] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9714.822281] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9714.822283] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9714.822284] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9714.822286] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9714.822288] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9714.826891] NMI backtrace for cpu 2 [ 9715.006237] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9715.010087] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9715.013034] Call Trace: [ 9715.013921] [ 9715.014627] ? dump_stack+0xbb/0x10e [ 9715.015798] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9715.017269] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9715.018785] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9715.020040] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9715.021352] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9715.022937] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9715.024403] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9715.026188] ? update_process_times+0x6d/0xc0 [ 9715.027781] ? tick_sched_handle.isra.18+0x27/0x90 [ 9715.029474] ? tick_sched_timer+0x53/0xb0 [ 9715.030799] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9715.032321] ? tick_sched_handle.isra.18+0x90/0x90 [ 9715.034080] ? hrtimer_interrupt+0x128/0x2c0 [ 9715.035592] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9715.037260] ? apic_timer_interrupt+0xf/0x20 [ 9715.040069] [ 9715.041073] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9715.042997] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9715.044584] ? do_raw_spin_lock+0x177/0x190 [ 9715.045753] ? _raw_spin_lock+0x21/0x30 [ 9715.046777] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9715.049601] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9715.051262] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9715.052587] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9715.054886] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9715.059563] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9715.061546] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9715.063358] ? __next_timer_interrupt+0x160/0x160 [ 9715.065029] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9715.067137] ? do_wait_intr+0xf0/0xf0 [ 9715.068983] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9715.070985] ? kthread+0x1d1/0x200 [ 9715.072286] ? set_kthread_struct+0x70/0x70 [ 9715.073679] ? ret_from_fork+0x1f/0x30 [ 9716.060250] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 9716.063226] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9716.103064] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9716.108235] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9716.113062] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9716.116093] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9716.124059] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9716.133634] RAX: 0000000000007e8a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9716.139601] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9716.144091] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9716.148873] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9716.154945] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9716.161061] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9716.167823] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9716.173400] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9716.176858] Call Trace: [ 9716.177994] [ 9716.178960] ? show_regs.cold.9+0x22/0x2f [ 9716.180558] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9716.182559] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9716.184446] ? watchdog+0x40/0x40 [ 9716.185670] ? hrtimer_interrupt+0x128/0x2c0 [ 9716.187731] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9716.189296] ? apic_timer_interrupt+0xf/0x20 [ 9716.190976] [ 9716.191838] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9716.193876] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9716.196031] do_raw_spin_lock+0x177/0x190 [ 9716.197757] _raw_spin_lock+0x21/0x30 [ 9716.199545] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9716.202318] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9716.204335] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9716.206061] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9716.208131] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9716.210437] ? woken_wake_function+0x30/0x30 [ 9716.211701] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9716.213968] kthread+0x1d1/0x200 [ 9716.215269] ? set_kthread_struct+0x70/0x70 [ 9716.216780] ret_from_fork+0x1f/0x30 [ 9740.053030] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 9740.055545] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 9740.059579] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 [ 9740.062672] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 9740.062687] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9740.062752] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9740.062757] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9740.062760] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9740.062773] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9740.062777] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9740.062782] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9740.062784] RDX: 00000000000077b9 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9740.062786] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9740.062788] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9740.062791] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9740.062794] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9740.062796] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9740.062799] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9740.062806] Call Trace: [ 9740.062812] [ 9740.062818] ? show_regs.cold.9+0x22/0x2f [ 9740.062828] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9740.062835] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9740.062839] ? watchdog+0x40/0x40 [ 9740.062843] ? hrtimer_interrupt+0x128/0x2c0 [ 9740.062847] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9740.063265] ? apic_timer_interrupt+0xf/0x20 [ 9740.063270] [ 9740.063273] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9740.063277] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9740.063281] do_raw_spin_lock+0x177/0x190 [ 9740.063286] _raw_spin_lock+0x21/0x30 [ 9740.063293] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9740.063576] obd_mod_free+0x46/0x140 [ptlrpc] [ 9740.064308] mdc_commit_open+0x91/0xb0 [mdc] [ 9740.064346] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9740.064542] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9740.065211] after_reply+0xe2c/0x1780 [ptlrpc] [ 9740.065400] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9740.065473] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9740.065480] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9740.065482] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9740.065493] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9740.065497] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9740.065501] RAX: 000000000000610a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9740.065503] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9740.065505] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9740.065507] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9740.065509] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9740.065512] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9740.065515] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9740.065517] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9740.065525] Call Trace: [ 9740.065529] [ 9740.065532] ? show_regs.cold.9+0x22/0x2f [ 9740.065434] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9740.065544] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9740.065552] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9740.065557] ? watchdog+0x40/0x40 [ 9740.065561] ? hrtimer_interrupt+0x128/0x2c0 [ 9740.065565] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9740.065573] ? apic_timer_interrupt+0xf/0x20 [ 9740.065578] [ 9740.065582] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9740.065587] do_raw_spin_lock+0x177/0x190 [ 9740.065592] _raw_spin_lock+0x21/0x30 [ 9740.065841] ? __next_timer_interrupt+0x160/0x160 [ 9740.065850] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9740.065610] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9740.066370] ? do_raw_spin_unlock+0x75/0x190 [ 9740.066320] ? do_wait_intr+0xf0/0xf0 [ 9740.066418] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9740.066377] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9740.067318] ? do_wait_intr+0xf0/0xf0 [ 9740.067327] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9740.067523] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9740.067749] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9740.068838] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9740.068945] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9740.069530] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9740.069560] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9740.069587] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9740.069684] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9740.069938] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9740.072035] kthread+0x1d1/0x200 [ 9740.072046] ? set_kthread_struct+0x70/0x70 [ 9740.072050] ret_from_fork+0x1f/0x30 [ 9740.072303] lmv_intent_open+0x338/0x1150 [lmv] [ 9740.072338] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9740.072446] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9740.072531] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9740.072556] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9740.075773] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9740.094173] auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9740.094215] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9740.094222] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9740.094229] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9740.094500] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9740.094510] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9740.094515] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9740.094519] RAX: 0000000000003291 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9740.094522] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9740.094523] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9740.094526] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9740.094528] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9740.094531] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9740.094534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9740.094536] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9740.094547] Call Trace: [ 9740.094553] [ 9740.094558] ? show_regs.cold.9+0x22/0x2f [ 9740.094568] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9740.094574] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9740.094578] ? watchdog+0x40/0x40 [ 9740.094582] ? hrtimer_interrupt+0x128/0x2c0 [ 9740.094586] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9740.094594] ? apic_timer_interrupt+0xf/0x20 [ 9740.094599] [ 9740.094602] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9740.094607] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9740.094611] ? newidle_balance.isra.93+0x207/0x580 [ 9740.094616] do_raw_spin_lock+0x177/0x190 [ 9740.094621] _raw_spin_lock+0x21/0x30 [ 9740.094626] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9740.095125] process_one_work+0x2c8/0x700 [ 9740.095132] worker_thread+0x296/0x6e0 [ 9740.095136] ? rescuer_thread+0x570/0x570 [ 9740.095139] kthread+0x1d1/0x200 [ 9740.095143] ? set_kthread_struct+0x70/0x70 [ 9740.095146] ret_from_fork+0x1f/0x30 [ 9740.754664] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9740.757348] ? do_raw_spin_unlock+0x75/0x190 [ 9740.759370] ? _raw_spin_unlock+0x12/0x30 [ 9740.761279] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9740.763583] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9740.765912] ll_atomic_open+0x36a/0xd70 [lustre] [ 9740.769843] lookup_open+0x356/0xb40 [ 9740.772972] do_last+0x313/0xfc0 [ 9740.777046] ? nd_jump_root+0xe5/0x160 [ 9740.779617] ? path_init+0x437/0x520 [ 9740.782613] path_openat+0xf7/0x500 [ 9740.789082] ? do_raw_spin_unlock+0x75/0x190 [ 9740.793658] do_filp_open+0x99/0x140 [ 9740.795603] ? getname_flags+0x6e/0x330 [ 9740.802653] ? __check_object_size+0xff/0x256 [ 9740.806511] ? do_raw_spin_unlock+0x75/0x190 [ 9740.810554] ? _raw_spin_unlock+0x12/0x30 [ 9740.816468] do_sys_openat2+0x2b4/0x410 [ 9740.817619] do_sys_open+0x73/0xa0 [ 9740.819851] __x64_sys_openat+0x24/0x30 [ 9740.823350] do_syscall_64+0xc1/0x440 [ 9740.824863] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9740.827034] RIP: 0033:0x7f53425a5332 [ 9740.829272] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9740.847396] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9740.855278] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9740.865695] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9740.870370] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9740.881113] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9740.891361] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9741.794311] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7104s! [ 9741.797785] Showing busy workqueues and worker pools: [ 9741.799499] workqueue events: flags=0x0 [ 9741.809915] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9741.818018] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9741.843742] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9741.850519] pending: cpu_latency_work [ptlrpc] [ 9741.859859] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9741.864289] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9741.873621] workqueue events_long: flags=0x0 [ 9741.883171] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9741.889026] pending: lu_objects_flush [obdclass] [ 9741.897653] workqueue events_unbound: flags=0x2 [ 9741.901440] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9741.920403] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9741.930668] workqueue events_power_efficient: flags=0x80 [ 9741.932907] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9741.944395] pending: check_lifetime, do_cache_clean [ 9741.947792] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9741.957804] pending: neigh_periodic_work, neigh_periodic_work [ 9741.963690] workqueue mm_percpu_wq: flags=0x8 [ 9741.969032] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9741.973781] pending: vmstat_update, lru_add_drain_per_cpu [ 9741.986651] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9741.997919] pending: vmstat_update, lru_add_drain_per_cpu [ 9742.004298] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9742.008403] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9742.013250] workqueue xprtiod: flags=0xa [ 9742.015793] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9742.020627] pending: xprt_autoclose, xs_error_handle [ 9742.022425] workqueue ipv6_addrconf: flags=0x40008 [ 9742.034448] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9742.042626] pending: addrconf_verify_work [ 9742.048698] workqueue knbd0-recv: flags=0x1a [ 9742.053594] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9742.065617] in-flight: 651:recv_work [ 9742.076270] workqueue ptlrpc_pinger: flags=0x2 [ 9742.089674] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9742.091687] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9742.110361] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7105s workers=4 idle: 15979 31824 8 [ 9742.125593] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9732s workers=2 idle: 654 [ 9744.059241] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 9744.065807] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9744.104749] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9744.115980] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9744.122695] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9744.126569] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9744.138084] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9744.148857] RAX: 0000000000002f91 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9744.156845] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9744.163711] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9744.165948] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9744.177129] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9744.181859] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9744.191884] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9744.198577] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9744.205767] Call Trace: [ 9744.209858] [ 9744.212523] ? show_regs.cold.9+0x22/0x2f [ 9744.217188] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9744.222351] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9744.226828] ? watchdog+0x40/0x40 [ 9744.227961] ? hrtimer_interrupt+0x128/0x2c0 [ 9744.234550] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9744.240085] ? apic_timer_interrupt+0xf/0x20 [ 9744.245231] [ 9744.250126] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9744.254524] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9744.259306] do_raw_spin_lock+0x177/0x190 [ 9744.260767] _raw_spin_lock+0x21/0x30 [ 9744.266652] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9744.274210] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9744.279589] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9744.291117] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9744.300927] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9744.308264] ? woken_wake_function+0x30/0x30 [ 9744.309765] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9744.320586] kthread+0x1d1/0x200 [ 9744.325354] ? set_kthread_struct+0x70/0x70 [ 9744.328744] ret_from_fork+0x1f/0x30 [ 9768.053842] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 9768.055087] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 9768.059184] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 9768.059199] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9768.059265] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9768.059271] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9768.059273] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9768.059287] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9768.059292] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9768.059296] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9768.059298] RDX: 00000000000013ae RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9768.059300] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9768.059302] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9768.059304] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9768.059307] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9768.059310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9768.059312] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9768.059320] Call Trace: [ 9768.059326] [ 9768.059331] ? show_regs.cold.9+0x22/0x2f [ 9768.059341] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9768.059347] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9768.059351] ? watchdog+0x40/0x40 [ 9768.059356] ? hrtimer_interrupt+0x128/0x2c0 [ 9768.059360] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9768.059368] ? apic_timer_interrupt+0xf/0x20 [ 9768.059373] [ 9768.059376] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9768.059380] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9768.059385] do_raw_spin_lock+0x177/0x190 [ 9768.059389] _raw_spin_lock+0x21/0x30 [ 9768.059395] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9768.059696] obd_mod_free+0x46/0x140 [ptlrpc] [ 9768.059899] mdc_commit_open+0x91/0xb0 [mdc] [ 9768.059941] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9768.060144] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9768.060337] after_reply+0xe2c/0x1780 [ptlrpc] [ 9768.060608] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9768.060809] ? __next_timer_interrupt+0x160/0x160 [ 9768.060818] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9768.061032] ? do_wait_intr+0xf0/0xf0 [ 9768.061041] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9768.061250] kthread+0x1d1/0x200 [ 9768.061256] ? set_kthread_struct+0x70/0x70 [ 9768.061260] ret_from_fork+0x1f/0x30 [ 9768.065993] Modules linked in: lustre(O) [ 9768.069544] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9768.069634] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9768.069642] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9768.069645] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9768.069725] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9768.069730] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9768.069735] RAX: 0000000000005587 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9768.069737] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9768.069740] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9768.069742] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9768.069744] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9768.069747] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9768.069750] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9768.069752] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9768.069763] Call Trace: [ 9768.069769] [ 9768.069774] ? show_regs.cold.9+0x22/0x2f [ 9768.069786] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9768.069798] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9768.069802] ? watchdog+0x40/0x40 [ 9768.069806] ? hrtimer_interrupt+0x128/0x2c0 [ 9768.069810] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9768.069818] ? apic_timer_interrupt+0xf/0x20 [ 9768.069823] [ 9768.069826] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9768.069834] do_raw_spin_lock+0x177/0x190 [ 9768.069839] _raw_spin_lock+0x21/0x30 [ 9768.069844] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9768.070131] ? do_raw_spin_unlock+0x75/0x190 [ 9768.070137] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9768.070414] ? do_wait_intr+0xf0/0xf0 [ 9768.070423] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9768.070639] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9768.071023] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9768.071414] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9768.071533] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9768.071577] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9768.071609] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9768.071636] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9768.071801] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9768.072026] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9768.072055] lmv_intent_open+0x338/0x1150 [lmv] [ 9768.072087] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9768.072172] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9768.072334] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9768.072355] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9768.072440] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9768.072538] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9768.072632] ? do_raw_spin_unlock+0x75/0x190 [ 9768.072638] ? _raw_spin_unlock+0x12/0x30 [ 9768.072645] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9768.072770] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9768.072791] ll_atomic_open+0x36a/0xd70 [lustre] [ 9768.072929] lookup_open+0x356/0xb40 [ 9768.072938] do_last+0x313/0xfc0 [ 9768.072943] ? nd_jump_root+0xe5/0x160 [ 9768.072948] ? path_init+0x437/0x520 [ 9768.072952] path_openat+0xf7/0x500 [ 9768.072956] ? do_raw_spin_unlock+0x75/0x190 [ 9768.072961] do_filp_open+0x99/0x140 [ 9768.072967] ? getname_flags+0x6e/0x330 [ 9768.072971] ? __check_object_size+0xff/0x256 [ 9768.072979] ? do_raw_spin_unlock+0x75/0x190 [ 9768.072983] ? _raw_spin_unlock+0x12/0x30 [ 9768.072987] do_sys_openat2+0x2b4/0x410 [ 9768.072991] do_sys_open+0x73/0xa0 [ 9768.072995] __x64_sys_openat+0x24/0x30 [ 9768.072998] do_syscall_64+0xc1/0x440 [ 9768.073005] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9768.073011] RIP: 0033:0x7f53425a5332 [ 9768.073018] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9768.073022] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9768.073026] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9768.073029] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9768.073031] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9768.073033] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9768.073035] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9768.665132] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9768.708372] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9768.726358] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9768.734177] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9768.737753] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9768.741435] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9768.757831] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9768.767026] RAX: 000000000000680c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9768.772354] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9768.778779] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9768.783405] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9768.786794] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9768.790121] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9768.801150] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9768.806337] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9768.809454] Call Trace: [ 9768.810664] [ 9768.815828] ? show_regs.cold.9+0x22/0x2f [ 9768.817852] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9768.822904] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9768.826679] ? watchdog+0x40/0x40 [ 9768.828764] ? hrtimer_interrupt+0x128/0x2c0 [ 9768.831030] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9768.838780] ? apic_timer_interrupt+0xf/0x20 [ 9768.843956] [ 9768.849162] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9768.851725] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9768.865157] ? newidle_balance.isra.93+0x207/0x580 [ 9768.872973] do_raw_spin_lock+0x177/0x190 [ 9768.878229] _raw_spin_lock+0x21/0x30 [ 9768.882736] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9768.885863] process_one_work+0x2c8/0x700 [ 9768.888995] worker_thread+0x296/0x6e0 [ 9768.891340] ? rescuer_thread+0x570/0x570 [ 9768.893076] kthread+0x1d1/0x200 [ 9768.894193] ? set_kthread_struct+0x70/0x70 [ 9768.899193] ret_from_fork+0x1f/0x30 [ 9772.058189] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 9772.063356] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9772.117781] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9772.127703] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9772.145901] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9772.149361] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9772.170245] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9772.177009] RAX: 0000000000002684 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9772.192366] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9772.202231] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9772.220948] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9772.223216] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9772.242223] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9772.254008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9772.255738] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9772.275632] Call Trace: [ 9772.276516] [ 9772.277168] ? show_regs.cold.9+0x22/0x2f [ 9772.282076] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9772.289202] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9772.293617] ? watchdog+0x40/0x40 [ 9772.297156] ? hrtimer_interrupt+0x128/0x2c0 [ 9772.300994] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9772.312033] ? apic_timer_interrupt+0xf/0x20 [ 9772.316513] [ 9772.318386] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9772.322114] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9772.329727] do_raw_spin_lock+0x177/0x190 [ 9772.333152] _raw_spin_lock+0x21/0x30 [ 9772.336571] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9772.338780] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9772.349915] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9772.361154] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9772.362998] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9772.376457] ? woken_wake_function+0x30/0x30 [ 9772.377787] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9772.391065] kthread+0x1d1/0x200 [ 9772.394351] ? set_kthread_struct+0x70/0x70 [ 9772.398800] ret_from_fork+0x1f/0x30 [ 9772.513041] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7135s! [ 9772.515653] Showing busy workqueues and worker pools: [ 9772.522870] workqueue events: flags=0x0 [ 9772.523965] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9772.527825] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9772.533916] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9772.536778] pending: cpu_latency_work [ptlrpc] [ 9772.539287] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9772.541586] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9772.545018] workqueue events_long: flags=0x0 [ 9772.546042] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9772.547934] pending: lu_objects_flush [obdclass] [ 9772.549486] workqueue events_unbound: flags=0x2 [ 9772.550637] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9772.552254] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9772.553749] workqueue events_power_efficient: flags=0x80 [ 9772.555063] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9772.556816] pending: check_lifetime, do_cache_clean [ 9772.558165] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9772.559887] pending: neigh_periodic_work, neigh_periodic_work [ 9772.561516] workqueue mm_percpu_wq: flags=0x8 [ 9772.562573] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9772.564338] pending: vmstat_update, lru_add_drain_per_cpu [ 9772.565834] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9772.567528] pending: vmstat_update, lru_add_drain_per_cpu [ 9772.568906] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9772.570889] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9772.572635] workqueue xprtiod: flags=0xa [ 9772.573783] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9772.575565] pending: xprt_autoclose, xs_error_handle [ 9772.577024] workqueue ipv6_addrconf: flags=0x40008 [ 9772.578557] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9772.580368] pending: addrconf_verify_work [ 9772.581596] workqueue knbd0-recv: flags=0x1a [ 9772.582733] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9772.584567] in-flight: 651:recv_work [ 9772.585631] workqueue ptlrpc_pinger: flags=0x2 [ 9772.587218] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9772.589632] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9772.592016] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7135s workers=4 idle: 15979 31824 8 [ 9772.594827] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9763s workers=2 idle: 654 [ 9796.055080] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [ 9796.057163] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [ 9796.061197] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9796.061370] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9796.061378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9796.061382] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9796.061396] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9796.061400] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9796.061405] RAX: 000000000000558f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9796.061407] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9796.061410] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9796.061412] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9796.061415] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9796.061418] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9796.061421] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9796.061424] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9796.061432] Call Trace: [ 9796.061438] [ 9796.061445] ? show_regs.cold.9+0x22/0x2f [ 9796.061455] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9796.061460] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9796.061464] ? watchdog+0x40/0x40 [ 9796.061469] ? hrtimer_interrupt+0x128/0x2c0 [ 9796.061473] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9796.061481] ? apic_timer_interrupt+0xf/0x20 [ 9796.061486] [ 9796.061489] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9796.061496] do_raw_spin_lock+0x177/0x190 [ 9796.061502] _raw_spin_lock+0x21/0x30 [ 9796.061507] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9796.062319] ? do_raw_spin_unlock+0x75/0x190 [ 9796.062327] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9796.062527] ? do_wait_intr+0xf0/0xf0 [ 9796.062537] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9796.067701] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9796.067781] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9796.067789] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9796.067792] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9796.067806] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9796.067810] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9796.067815] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9796.067909] RDX: 0000000000001bd2 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9796.067912] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9796.067914] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9796.067916] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9796.067920] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9796.067922] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9796.067925] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9796.067933] Call Trace: [ 9796.067938] [ 9796.067943] ? show_regs.cold.9+0x22/0x2f [ 9796.067952] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9796.067958] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9796.067962] ? watchdog+0x40/0x40 [ 9796.067967] ? hrtimer_interrupt+0x128/0x2c0 [ 9796.067971] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9796.067980] ? apic_timer_interrupt+0xf/0x20 [ 9796.067984] [ 9796.068043] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9796.068049] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9796.068053] do_raw_spin_lock+0x177/0x190 [ 9796.068058] _raw_spin_lock+0x21/0x30 [ 9796.068064] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9796.068362] obd_mod_free+0x46/0x140 [ptlrpc] [ 9796.068555] mdc_commit_open+0x91/0xb0 [mdc] [ 9796.068593] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9796.099931] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9796.103738] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9796.104043] after_reply+0xe2c/0x1780 [ptlrpc] [ 9796.111221] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9796.113258] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9796.122498] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9796.122629] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9796.122671] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9796.122703] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9796.122732] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9796.122817] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9796.123360] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9796.123395] lmv_intent_open+0x338/0x1150 [lmv] [ 9796.123431] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9796.123532] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9796.123624] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9796.123646] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9796.123729] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9796.123811] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9796.123897] ? do_raw_spin_unlock+0x75/0x190 [ 9796.123905] ? _raw_spin_unlock+0x12/0x30 [ 9796.123911] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9796.123938] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9796.123957] ll_atomic_open+0x36a/0xd70 [lustre] [ 9796.126728] ? __next_timer_interrupt+0x160/0x160 [ 9796.126753] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9796.127065] ? do_wait_intr+0xf0/0xf0 [ 9796.127075] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9796.127366] kthread+0x1d1/0x200 [ 9796.127373] ? set_kthread_struct+0x70/0x70 [ 9796.127377] ret_from_fork+0x1f/0x30 [ 9796.370867] lookup_open+0x356/0xb40 [ 9796.374976] do_last+0x313/0xfc0 [ 9796.376369] ? nd_jump_root+0xe5/0x160 [ 9796.381890] ? path_init+0x437/0x520 [ 9796.387299] path_openat+0xf7/0x500 [ 9796.388651] ? do_raw_spin_unlock+0x75/0x190 [ 9796.392442] do_filp_open+0x99/0x140 [ 9796.394481] ? getname_flags+0x6e/0x330 [ 9796.397818] ? __check_object_size+0xff/0x256 [ 9796.399805] ? do_raw_spin_unlock+0x75/0x190 [ 9796.405888] ? _raw_spin_unlock+0x12/0x30 [ 9796.413938] do_sys_openat2+0x2b4/0x410 [ 9796.416348] do_sys_open+0x73/0xa0 [ 9796.420272] __x64_sys_openat+0x24/0x30 [ 9796.423656] do_syscall_64+0xc1/0x440 [ 9796.429107] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9796.430893] RIP: 0033:0x7f53425a5332 [ 9796.432956] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9796.449125] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9796.459085] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9796.466742] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9796.468883] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9796.475402] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9796.487180] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9800.054230] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 9800.056484] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9800.062017] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 9800.080009] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9800.080027] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9800.080033] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9800.080439] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 9800.080452] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [ 9800.080456] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9800.080461] RAX: 000000000000475a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9800.080463] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9800.080465] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9800.080468] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9800.080470] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9800.080473] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9800.080476] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9800.080479] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9800.080488] Call Trace: [ 9800.080493] [ 9800.080498] ? show_regs.cold.9+0x22/0x2f [ 9800.080509] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9800.080515] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9800.080520] ? watchdog+0x40/0x40 [ 9800.080524] ? hrtimer_interrupt+0x128/0x2c0 [ 9800.080528] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9800.080537] ? apic_timer_interrupt+0xf/0x20 [ 9800.080541] [ 9800.080544] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [ 9800.080549] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9800.080553] ? newidle_balance.isra.93+0x207/0x580 [ 9800.080558] do_raw_spin_lock+0x177/0x190 [ 9800.080563] _raw_spin_lock+0x21/0x30 [ 9800.080568] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9800.080777] process_one_work+0x2c8/0x700 [ 9800.080785] worker_thread+0x296/0x6e0 [ 9800.080790] ? rescuer_thread+0x570/0x570 [ 9800.080794] kthread+0x1d1/0x200 [ 9800.080798] ? set_kthread_struct+0x70/0x70 [ 9800.080803] ret_from_fork+0x1f/0x30 [ 9800.222046] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9800.240201] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9800.244870] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9800.254663] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9800.264278] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9800.279683] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [ 9800.284539] RAX: 0000000000004389 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9800.287998] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9800.291200] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9800.294156] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9800.296467] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9800.299530] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9800.304098] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9800.306598] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9800.313413] Call Trace: [ 9800.315287] [ 9800.317780] ? show_regs.cold.9+0x22/0x2f [ 9800.322120] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9800.327991] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9800.329890] ? watchdog+0x40/0x40 [ 9800.332248] ? hrtimer_interrupt+0x128/0x2c0 [ 9800.335554] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9800.337828] ? apic_timer_interrupt+0xf/0x20 [ 9800.340203] [ 9800.341532] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9800.347095] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9800.349774] do_raw_spin_lock+0x177/0x190 [ 9800.351602] _raw_spin_lock+0x21/0x30 [ 9800.353286] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9800.356570] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9800.359535] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9800.362247] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9800.364827] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9800.369574] ? woken_wake_function+0x30/0x30 [ 9800.372824] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9800.375599] kthread+0x1d1/0x200 [ 9800.378298] ? set_kthread_struct+0x70/0x70 [ 9800.382501] ret_from_fork+0x1f/0x30 [ 9802.721085] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7165s! [ 9802.724822] Showing busy workqueues and worker pools: [ 9802.727088] workqueue events: flags=0x0 [ 9802.729293] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9802.732343] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9802.739617] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9802.744739] pending: cpu_latency_work [ptlrpc] [ 9802.747163] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9802.751616] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9802.756899] workqueue events_long: flags=0x0 [ 9802.758713] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9802.761310] pending: lu_objects_flush [obdclass] [ 9802.762780] workqueue events_unbound: flags=0x2 [ 9802.764552] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9802.767067] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9802.769456] workqueue events_power_efficient: flags=0x80 [ 9802.773530] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9802.778161] pending: check_lifetime, do_cache_clean [ 9802.782295] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9802.785206] pending: neigh_periodic_work, neigh_periodic_work [ 9802.787780] workqueue mm_percpu_wq: flags=0x8 [ 9802.792037] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9802.797694] pending: vmstat_update, lru_add_drain_per_cpu [ 9802.799743] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9802.804820] pending: vmstat_update, lru_add_drain_per_cpu [ 9802.808121] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9802.813754] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9802.818444] workqueue xprtiod: flags=0xa [ 9802.822655] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9802.825972] pending: xprt_autoclose, xs_error_handle [ 9802.831372] workqueue ipv6_addrconf: flags=0x40008 [ 9802.835208] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9802.838368] pending: addrconf_verify_work [ 9802.842025] workqueue knbd0-recv: flags=0x1a [ 9802.843739] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9802.849476] in-flight: 651:recv_work [ 9802.851566] workqueue ptlrpc_pinger: flags=0x2 [ 9802.853489] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9802.855758] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9802.858814] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7165s workers=4 idle: 15979 31824 8 [ 9802.862061] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9793s workers=2 idle: 654 [ 9824.059664] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9824.063227] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9824.065100] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9824.113340] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9824.113359] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9824.113364] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9824.113384] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9824.113389] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9824.113395] RAX: 0000000000007e86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9824.113398] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9824.113400] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9824.113402] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9824.113405] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9824.113408] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9824.113411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9824.113414] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9824.113423] Call Trace: [ 9824.113430] [ 9824.113436] ? show_regs.cold.9+0x22/0x2f [ 9824.113447] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9824.113454] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9824.113460] ? watchdog+0x40/0x40 [ 9824.113465] ? hrtimer_interrupt+0x128/0x2c0 [ 9824.113469] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9824.113480] ? apic_timer_interrupt+0xf/0x20 [ 9824.113485] [ 9824.113488] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9824.113508] do_raw_spin_lock+0x177/0x190 [ 9824.113514] _raw_spin_lock+0x21/0x30 [ 9824.113519] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9824.113839] ? do_raw_spin_unlock+0x75/0x190 [ 9824.113845] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9824.114047] ? do_wait_intr+0xf0/0xf0 [ 9824.114056] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9824.114252] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9824.114466] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9824.124165] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9824.124253] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9824.124261] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9824.124264] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9824.124278] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9824.124282] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9824.124287] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9824.124289] RDX: 000000000000682c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9824.124292] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9824.124295] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9824.124297] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9824.124301] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9824.124304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9824.124306] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9824.124315] Call Trace: [ 9824.124321] [ 9824.124327] ? show_regs.cold.9+0x22/0x2f [ 9824.124337] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9824.124343] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9824.124348] ? watchdog+0x40/0x40 [ 9824.124351] ? hrtimer_interrupt+0x128/0x2c0 [ 9824.124355] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9824.124363] ? apic_timer_interrupt+0xf/0x20 [ 9824.124367] [ 9824.124370] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9824.124374] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9824.124379] do_raw_spin_lock+0x177/0x190 [ 9824.125659] _raw_spin_lock+0x21/0x30 [ 9824.125668] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9824.125947] obd_mod_free+0x46/0x140 [ptlrpc] [ 9824.126159] mdc_commit_open+0x91/0xb0 [mdc] [ 9824.126200] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9824.147489] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9824.148892] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9824.148936] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9824.148970] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9824.149001] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9824.149083] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9824.149383] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9824.149413] lmv_intent_open+0x338/0x1150 [lmv] [ 9824.149450] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9824.149548] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9824.149638] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9824.149664] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9824.149749] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9824.149835] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9824.149926] ? do_raw_spin_unlock+0x75/0x190 [ 9824.149935] ? _raw_spin_unlock+0x12/0x30 [ 9824.149942] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9824.149971] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9824.149992] ll_atomic_open+0x36a/0xd70 [lustre] [ 9824.150078] lookup_open+0x356/0xb40 [ 9824.150089] do_last+0x313/0xfc0 [ 9824.150094] ? nd_jump_root+0xe5/0x160 [ 9824.150100] ? path_init+0x437/0x520 [ 9824.150105] path_openat+0xf7/0x500 [ 9824.150110] ? do_raw_spin_unlock+0x75/0x190 [ 9824.150115] do_filp_open+0x99/0x140 [ 9824.150121] ? getname_flags+0x6e/0x330 [ 9824.150126] ? __check_object_size+0xff/0x256 [ 9824.150134] ? do_raw_spin_unlock+0x75/0x190 [ 9824.150138] ? _raw_spin_unlock+0x12/0x30 [ 9824.150143] do_sys_openat2+0x2b4/0x410 [ 9824.150147] do_sys_open+0x73/0xa0 [ 9824.150152] __x64_sys_openat+0x24/0x30 [ 9824.150155] do_syscall_64+0xc1/0x440 [ 9824.150163] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9824.150169] RIP: 0033:0x7f53425a5332 [ 9824.150218] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9824.150223] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9824.150229] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9824.150231] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9824.150233] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9824.150235] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9824.150237] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9824.858465] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9824.862335] after_reply+0xe2c/0x1780 [ptlrpc] [ 9824.867312] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9824.871005] ? __next_timer_interrupt+0x160/0x160 [ 9824.874360] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9824.879381] ? do_wait_intr+0xf0/0xf0 [ 9824.880553] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9824.882260] kthread+0x1d1/0x200 [ 9824.885714] ? set_kthread_struct+0x70/0x70 [ 9824.891245] ret_from_fork+0x1f/0x30 [ 9828.056950] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [ 9828.058022] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 9828.076537] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9828.076619] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9828.076627] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9828.076635] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9828.076913] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9828.076925] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9828.076929] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9828.076933] RAX: 0000000000007364 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9828.076936] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9828.076938] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9828.076940] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9828.076942] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9828.076945] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9828.076948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9828.076950] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9828.076959] Call Trace: [ 9828.076965] [ 9828.076971] ? show_regs.cold.9+0x22/0x2f [ 9828.076980] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9828.076986] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9828.076990] ? watchdog+0x40/0x40 [ 9828.076995] ? hrtimer_interrupt+0x128/0x2c0 [ 9828.076998] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9828.077007] ? apic_timer_interrupt+0xf/0x20 [ 9828.077012] [ 9828.077014] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9828.077019] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9828.077022] ? newidle_balance.isra.93+0x207/0x580 [ 9828.077027] do_raw_spin_lock+0x177/0x190 [ 9828.077031] _raw_spin_lock+0x21/0x30 [ 9828.077035] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9828.077251] process_one_work+0x2c8/0x700 [ 9828.077259] worker_thread+0x296/0x6e0 [ 9828.077263] ? rescuer_thread+0x570/0x570 [ 9828.077267] kthread+0x1d1/0x200 [ 9828.077272] ? set_kthread_struct+0x70/0x70 [ 9828.077276] ret_from_fork+0x1f/0x30 [ 9828.238274] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9828.260261] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9828.266300] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9828.270825] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9828.274582] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9828.286258] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9828.289745] RAX: 0000000000002025 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9828.293369] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9828.296979] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9828.302905] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9828.307531] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9828.311133] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9828.315818] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9828.318582] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9828.322701] Call Trace: [ 9828.323617] [ 9828.324408] ? show_regs.cold.9+0x22/0x2f [ 9828.327216] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9828.330806] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9828.333297] ? watchdog+0x40/0x40 [ 9828.335497] ? hrtimer_interrupt+0x128/0x2c0 [ 9828.338002] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9828.340667] ? apic_timer_interrupt+0xf/0x20 [ 9828.343163] [ 9828.344680] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9828.348849] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9828.352327] do_raw_spin_lock+0x177/0x190 [ 9828.354403] _raw_spin_lock+0x21/0x30 [ 9828.356178] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9828.360106] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9828.363253] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9828.366132] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9828.368541] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9828.371502] ? woken_wake_function+0x30/0x30 [ 9828.373790] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9828.378756] kthread+0x1d1/0x200 [ 9828.380547] ? set_kthread_struct+0x70/0x70 [ 9828.383358] ret_from_fork+0x1f/0x30 [ 9832.930238] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7195s! [ 9832.935059] Showing busy workqueues and worker pools: [ 9832.937279] workqueue events: flags=0x0 [ 9832.944264] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9832.951018] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9832.964643] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9832.973666] pending: cpu_latency_work [ptlrpc] [ 9832.976158] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9832.982086] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9832.990968] workqueue events_long: flags=0x0 [ 9832.993171] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9832.995975] pending: lu_objects_flush [obdclass] [ 9832.999729] workqueue events_unbound: flags=0x2 [ 9833.002405] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9833.005097] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9833.008405] workqueue events_power_efficient: flags=0x80 [ 9833.010860] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9833.018835] pending: check_lifetime, do_cache_clean [ 9833.022027] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9833.024887] pending: neigh_periodic_work, neigh_periodic_work [ 9833.027540] workqueue mm_percpu_wq: flags=0x8 [ 9833.029520] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9833.032182] pending: vmstat_update, lru_add_drain_per_cpu [ 9833.035900] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9833.039658] pending: vmstat_update, lru_add_drain_per_cpu [ 9833.051023] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9833.053218] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9833.059767] workqueue xprtiod: flags=0xa [ 9833.062682] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9833.066251] pending: xprt_autoclose, xs_error_handle [ 9833.075238] workqueue ipv6_addrconf: flags=0x40008 [ 9833.083105] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9833.085040] pending: addrconf_verify_work [ 9833.088607] workqueue knbd0-recv: flags=0x1a [ 9833.092034] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9833.096769] in-flight: 651:recv_work [ 9833.099024] workqueue ptlrpc_pinger: flags=0x2 [ 9833.101228] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9833.105618] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9833.107816] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7196s workers=4 idle: 15979 31824 8 [ 9833.115665] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9823s workers=2 idle: 654 [ 9852.058605] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9852.059202] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9852.062950] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9852.063057] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9852.063066] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9852.063068] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9852.063084] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9852.063088] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9852.063092] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9852.063094] RDX: 00000000000049ed RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9852.063096] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9852.063098] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9852.063100] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9852.063104] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9852.063107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9852.063109] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9852.063119] Call Trace: [ 9852.063126] [ 9852.063132] ? show_regs.cold.9+0x22/0x2f [ 9852.063141] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9852.063148] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9852.063152] ? watchdog+0x40/0x40 [ 9852.063155] ? hrtimer_interrupt+0x128/0x2c0 [ 9852.063159] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9852.063167] ? apic_timer_interrupt+0xf/0x20 [ 9852.063172] [ 9852.063175] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9852.063179] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9852.063183] do_raw_spin_lock+0x177/0x190 [ 9852.063188] _raw_spin_lock+0x21/0x30 [ 9852.063201] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9852.063472] obd_mod_free+0x46/0x140 [ptlrpc] [ 9852.063743] mdc_commit_open+0x91/0xb0 [mdc] [ 9852.063783] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9852.063975] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9852.064227] after_reply+0xe2c/0x1780 [ptlrpc] [ 9852.064419] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9852.064666] ? __next_timer_interrupt+0x160/0x160 [ 9852.064674] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9852.064870] ? do_wait_intr+0xf0/0xf0 [ 9852.064878] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9852.065103] kthread+0x1d1/0x200 [ 9852.065109] ? set_kthread_struct+0x70/0x70 [ 9852.065113] ret_from_fork+0x1f/0x30 [ 9852.403542] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9852.458404] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9852.465552] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9852.475681] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9852.480545] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9852.491270] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9852.494555] RAX: 0000000000006972 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9852.497720] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9852.500970] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9852.504175] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9852.506723] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9852.509807] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9852.513803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9852.516414] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9852.519484] Call Trace: [ 9852.520498] [ 9852.521395] ? show_regs.cold.9+0x22/0x2f [ 9852.523433] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9852.525535] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9852.527550] ? watchdog+0x40/0x40 [ 9852.529113] ? hrtimer_interrupt+0x128/0x2c0 [ 9852.530802] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9852.533280] ? apic_timer_interrupt+0xf/0x20 [ 9852.534898] [ 9852.535816] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9852.540655] do_raw_spin_lock+0x177/0x190 [ 9852.543359] _raw_spin_lock+0x21/0x30 [ 9852.544957] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9852.547514] ? do_raw_spin_unlock+0x75/0x190 [ 9852.549549] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9852.552340] ? do_wait_intr+0xf0/0xf0 [ 9852.553761] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9852.556830] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9852.559536] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9852.562586] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9852.564979] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9852.567759] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9852.571409] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9852.574089] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9852.577845] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9852.584350] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9852.591526] lmv_intent_open+0x338/0x1150 [lmv] [ 9852.594766] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9852.598826] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9852.603530] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9852.607742] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9852.610274] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9852.613799] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9852.617304] ? do_raw_spin_unlock+0x75/0x190 [ 9852.622365] ? _raw_spin_unlock+0x12/0x30 [ 9852.625577] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9852.628767] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9852.632278] ll_atomic_open+0x36a/0xd70 [lustre] [ 9852.635184] lookup_open+0x356/0xb40 [ 9852.637339] do_last+0x313/0xfc0 [ 9852.639243] ? nd_jump_root+0xe5/0x160 [ 9852.640604] ? path_init+0x437/0x520 [ 9852.641878] path_openat+0xf7/0x500 [ 9852.644053] ? do_raw_spin_unlock+0x75/0x190 [ 9852.646917] do_filp_open+0x99/0x140 [ 9852.648657] ? getname_flags+0x6e/0x330 [ 9852.650979] ? __check_object_size+0xff/0x256 [ 9852.653621] ? do_raw_spin_unlock+0x75/0x190 [ 9852.655790] ? _raw_spin_unlock+0x12/0x30 [ 9852.658171] do_sys_openat2+0x2b4/0x410 [ 9852.660300] do_sys_open+0x73/0xa0 [ 9852.662500] __x64_sys_openat+0x24/0x30 [ 9852.664945] do_syscall_64+0xc1/0x440 [ 9852.667121] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9852.670253] RIP: 0033:0x7f53425a5332 [ 9852.672400] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9852.700974] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9852.705991] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9852.712396] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9852.721535] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9852.727351] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9852.737956] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9856.058960] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 9856.063557] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9856.105717] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9856.114513] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9856.118358] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9856.125972] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9856.135652] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [ 9856.141664] RAX: 0000000000007c27 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9856.145603] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9856.149417] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9856.155053] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9856.159198] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9856.161242] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9856.164680] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9856.168357] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9856.170892] Call Trace: [ 9856.172279] [ 9856.173419] ? show_regs.cold.9+0x22/0x2f [ 9856.175818] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9856.178178] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9856.180797] ? watchdog+0x40/0x40 [ 9856.182431] ? hrtimer_interrupt+0x128/0x2c0 [ 9856.184803] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9856.187275] ? apic_timer_interrupt+0xf/0x20 [ 9856.189422] [ 9856.190414] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9856.193447] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9856.196389] do_raw_spin_lock+0x177/0x190 [ 9856.200412] _raw_spin_lock+0x21/0x30 [ 9856.202547] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9856.206166] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9856.208555] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9856.211819] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9856.215090] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9856.217598] ? woken_wake_function+0x30/0x30 [ 9856.219837] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9856.222863] kthread+0x1d1/0x200 [ 9856.224228] ? set_kthread_struct+0x70/0x70 [ 9856.225879] ret_from_fork+0x1f/0x30 [ 9860.059085] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 9860.064539] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9860.090199] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9860.095885] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9860.100328] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9860.103837] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9860.107495] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9860.115703] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9860.118793] RAX: 0000000000001e86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9860.121538] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9860.124726] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9860.127819] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9860.130507] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9860.133480] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9860.137126] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9860.139453] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9860.142822] Call Trace: [ 9860.143863] [ 9860.144508] ? show_regs.cold.9+0x22/0x2f [ 9860.146433] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9860.148821] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9860.151147] ? watchdog+0x40/0x40 [ 9860.152447] ? hrtimer_interrupt+0x128/0x2c0 [ 9860.154374] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9860.156715] ? apic_timer_interrupt+0xf/0x20 [ 9860.158438] [ 9860.159275] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9860.162069] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9860.164411] ? newidle_balance.isra.93+0x207/0x580 [ 9860.166391] do_raw_spin_lock+0x177/0x190 [ 9860.168253] _raw_spin_lock+0x21/0x30 [ 9860.169838] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9860.172309] process_one_work+0x2c8/0x700 [ 9860.174184] worker_thread+0x296/0x6e0 [ 9860.176268] ? rescuer_thread+0x570/0x570 [ 9860.178491] kthread+0x1d1/0x200 [ 9860.180335] ? set_kthread_struct+0x70/0x70 [ 9860.182348] ret_from_fork+0x1f/0x30 [ 9863.141616] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7226s! [ 9863.145377] Showing busy workqueues and worker pools: [ 9863.149098] workqueue events: flags=0x0 [ 9863.150812] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9863.153837] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9863.159619] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9863.163499] pending: cpu_latency_work [ptlrpc] [ 9863.167130] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9863.170637] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9863.175754] workqueue events_long: flags=0x0 [ 9863.177386] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9863.180614] pending: lu_objects_flush [obdclass] [ 9863.183045] workqueue events_unbound: flags=0x2 [ 9863.185071] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9863.187725] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9863.190372] workqueue events_power_efficient: flags=0x80 [ 9863.192761] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9863.195769] pending: check_lifetime, do_cache_clean [ 9863.198245] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9863.201385] pending: neigh_periodic_work, neigh_periodic_work [ 9863.206646] workqueue mm_percpu_wq: flags=0x8 [ 9863.208693] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9863.212259] pending: vmstat_update, lru_add_drain_per_cpu [ 9863.215650] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9863.219364] pending: vmstat_update, lru_add_drain_per_cpu [ 9863.222211] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9863.225360] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9863.227763] workqueue xprtiod: flags=0xa [ 9863.229727] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9863.232851] pending: xprt_autoclose, xs_error_handle [ 9863.235374] workqueue ipv6_addrconf: flags=0x40008 [ 9863.237678] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9863.240245] pending: addrconf_verify_work [ 9863.241749] workqueue knbd0-recv: flags=0x1a [ 9863.243467] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9863.246176] in-flight: 651:recv_work [ 9863.247613] workqueue ptlrpc_pinger: flags=0x2 [ 9863.249449] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9863.251765] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9863.254521] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7226s workers=4 idle: 15979 31824 8 [ 9863.257799] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9853s workers=2 idle: 654 [ 9880.055017] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9880.057916] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9880.062839] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9880.086647] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9880.086665] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9880.086669] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9880.086683] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9880.086687] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9880.086692] RAX: 0000000000006983 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9880.086695] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9880.086697] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9880.086700] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9880.086702] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9880.086706] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9880.086709] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9880.086712] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9880.086720] Call Trace: [ 9880.086726] [ 9880.086800] ? show_regs.cold.9+0x22/0x2f [ 9880.086812] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9880.086820] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9880.086825] ? watchdog+0x40/0x40 [ 9880.086830] ? hrtimer_interrupt+0x128/0x2c0 [ 9880.086835] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9880.086844] ? apic_timer_interrupt+0xf/0x20 [ 9880.086848] [ 9880.086852] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9880.090135] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) [ 9880.096181] do_raw_spin_lock+0x177/0x190 [ 9880.100008] lquota(O) lfsck(O) obdecho(O) mgc(O) [ 9880.102762] _raw_spin_lock+0x21/0x30 [ 9880.102785] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9880.110186] mdc(O) lov(O) osc(O) lmv(O) ec(O) [ 9880.116226] ? do_raw_spin_unlock+0x75/0x190 [ 9880.119261] fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9880.119319] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9880.119327] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9880.119330] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9880.119344] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9880.119349] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9880.119353] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9880.119356] RDX: 0000000000003053 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9880.119358] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9880.119361] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9880.119363] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9880.119367] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9880.119370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9880.119372] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9880.119380] Call Trace: [ 9880.119386] [ 9880.119391] ? show_regs.cold.9+0x22/0x2f [ 9880.119400] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9880.119407] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9880.119411] ? watchdog+0x40/0x40 [ 9880.119415] ? hrtimer_interrupt+0x128/0x2c0 [ 9880.119418] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9880.119426] ? apic_timer_interrupt+0xf/0x20 [ 9880.119431] [ 9880.119434] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9880.119439] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9880.119788] do_raw_spin_lock+0x177/0x190 [ 9880.122076] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9880.122396] ? do_wait_intr+0xf0/0xf0 [ 9880.122407] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9880.122825] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9880.123094] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9880.123299] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9880.123421] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9880.123726] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9880.123804] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9880.123840] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9880.123936] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9880.124181] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9880.124212] lmv_intent_open+0x338/0x1150 [lmv] [ 9880.124247] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9880.127125] _raw_spin_lock+0x21/0x30 [ 9880.127146] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9880.127416] obd_mod_free+0x46/0x140 [ptlrpc] [ 9880.127891] mdc_commit_open+0x91/0xb0 [mdc] [ 9880.127941] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9880.128156] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9880.128375] after_reply+0xe2c/0x1780 [ptlrpc] [ 9880.128772] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9880.128980] ? __next_timer_interrupt+0x160/0x160 [ 9880.128992] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9880.129215] ? do_wait_intr+0xf0/0xf0 [ 9880.129225] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9880.129575] kthread+0x1d1/0x200 [ 9880.129581] ? set_kthread_struct+0x70/0x70 [ 9880.129586] ret_from_fork+0x1f/0x30 [ 9880.336889] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9880.339490] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9880.341573] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9880.344164] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9880.346381] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9880.348727] ? do_raw_spin_unlock+0x75/0x190 [ 9880.350550] ? _raw_spin_unlock+0x12/0x30 [ 9880.352308] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9880.354198] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9880.356292] ll_atomic_open+0x36a/0xd70 [lustre] [ 9880.358235] lookup_open+0x356/0xb40 [ 9880.360397] do_last+0x313/0xfc0 [ 9880.361481] ? nd_jump_root+0xe5/0x160 [ 9880.362535] ? path_init+0x437/0x520 [ 9880.364157] path_openat+0xf7/0x500 [ 9880.365731] ? do_raw_spin_unlock+0x75/0x190 [ 9880.368003] do_filp_open+0x99/0x140 [ 9880.369316] ? getname_flags+0x6e/0x330 [ 9880.370715] ? __check_object_size+0xff/0x256 [ 9880.372300] ? do_raw_spin_unlock+0x75/0x190 [ 9880.373725] ? _raw_spin_unlock+0x12/0x30 [ 9880.375152] do_sys_openat2+0x2b4/0x410 [ 9880.376990] do_sys_open+0x73/0xa0 [ 9880.378400] __x64_sys_openat+0x24/0x30 [ 9880.380370] do_syscall_64+0xc1/0x440 [ 9880.381987] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9880.384078] RIP: 0033:0x7f53425a5332 [ 9880.385964] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9880.394080] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9880.397444] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9880.400226] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9880.403128] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9880.406296] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9880.408899] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9884.058091] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [ 9884.072971] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9884.102296] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9884.120054] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9884.125900] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9884.128550] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9884.149607] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9884.161320] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9884.168242] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9884.177249] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9884.189963] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9884.199184] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9884.204884] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9884.209366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9884.212756] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9884.220286] Call Trace: [ 9884.224000] [ 9884.230974] ? show_regs.cold.9+0x22/0x2f [ 9884.232390] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9884.240987] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9884.249882] ? watchdog+0x40/0x40 [ 9884.256357] ? hrtimer_interrupt+0x128/0x2c0 [ 9884.259321] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9884.263302] ? apic_timer_interrupt+0xf/0x20 [ 9884.265822] [ 9884.266904] ? native_safe_halt+0x2e/0x50 [ 9884.269225] kvm_wait+0xcd/0xe0 [ 9884.271355] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9884.274439] do_raw_spin_lock+0x177/0x190 [ 9884.278574] _raw_spin_lock+0x21/0x30 [ 9884.280056] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9884.283964] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9884.287241] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9884.290433] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9884.293258] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9884.296095] ? woken_wake_function+0x30/0x30 [ 9884.298722] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9884.303802] kthread+0x1d1/0x200 [ 9884.305345] ? set_kthread_struct+0x70/0x70 [ 9884.308056] ret_from_fork+0x1f/0x30 [ 9888.052017] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 9888.055192] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9888.074051] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9888.078699] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9888.081986] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9888.084855] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9888.087197] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9888.095133] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9888.098373] RAX: 00000000000069ce RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9888.101580] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9888.104530] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9888.107273] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9888.110201] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9888.113240] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9888.116937] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9888.119266] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9888.122567] Call Trace: [ 9888.123768] [ 9888.124559] ? show_regs.cold.9+0x22/0x2f [ 9888.126133] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9888.128125] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9888.130000] ? watchdog+0x40/0x40 [ 9888.131260] ? hrtimer_interrupt+0x128/0x2c0 [ 9888.132852] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9888.134799] ? apic_timer_interrupt+0xf/0x20 [ 9888.136657] [ 9888.137626] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9888.140063] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9888.142982] ? newidle_balance.isra.93+0x207/0x580 [ 9888.144916] do_raw_spin_lock+0x177/0x190 [ 9888.146506] _raw_spin_lock+0x21/0x30 [ 9888.147891] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9888.150290] process_one_work+0x2c8/0x700 [ 9888.151975] worker_thread+0x296/0x6e0 [ 9888.153687] ? rescuer_thread+0x570/0x570 [ 9888.155180] kthread+0x1d1/0x200 [ 9888.156777] ? set_kthread_struct+0x70/0x70 [ 9888.158527] ret_from_fork+0x1f/0x30 [ 9893.346136] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7256s! [ 9893.349902] Showing busy workqueues and worker pools: [ 9893.354516] workqueue events: flags=0x0 [ 9893.356717] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9893.361241] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9893.368482] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9893.374471] pending: cpu_latency_work [ptlrpc] [ 9893.377988] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9893.382716] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9893.393478] workqueue events_long: flags=0x0 [ 9893.396909] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9893.399246] pending: lu_objects_flush [obdclass] [ 9893.407039] workqueue events_unbound: flags=0x2 [ 9893.410180] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9893.414865] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9893.417507] workqueue events_power_efficient: flags=0x80 [ 9893.420547] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9893.424065] pending: check_lifetime, do_cache_clean [ 9893.428165] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9893.435423] pending: neigh_periodic_work, neigh_periodic_work [ 9893.438690] workqueue mm_percpu_wq: flags=0x8 [ 9893.440699] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9893.448658] pending: vmstat_update, lru_add_drain_per_cpu [ 9893.455244] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9893.461679] pending: vmstat_update, lru_add_drain_per_cpu [ 9893.469126] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9893.476824] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9893.480256] workqueue xprtiod: flags=0xa [ 9893.488703] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9893.496761] pending: xprt_autoclose, xs_error_handle [ 9893.501207] workqueue ipv6_addrconf: flags=0x40008 [ 9893.509940] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9893.523987] pending: addrconf_verify_work [ 9893.531911] workqueue knbd0-recv: flags=0x1a [ 9893.536048] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9893.540707] in-flight: 651:recv_work [ 9893.544098] workqueue ptlrpc_pinger: flags=0x2 [ 9893.545459] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9893.552007] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9893.557751] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7256s workers=4 idle: 15979 31824 8 [ 9893.575472] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9884s workers=2 idle: 654 [ 9894.616762] rcu: INFO: rcu_sched self-detected stall on CPU [ 9894.619913] rcu: 2-....: (4262012 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [ 9894.625081] (t=7800190 jiffies g=274185 q=65830) [ 9894.627459] rcu: rcu_sched kthread starved for 7258556 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [ 9894.632237] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 9894.636047] rcu: RCU grace-period kthread stack dump: [ 9894.638443] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [ 9894.642941] Call Trace: [ 9894.644132] __schedule+0x351/0xcb0 [ 9894.645935] schedule+0xc0/0x180 [ 9894.647102] schedule_timeout+0xb4/0x190 [ 9894.649710] ? do_raw_spin_unlock+0x75/0x190 [ 9894.651398] ? __next_timer_interrupt+0x160/0x160 [ 9894.653954] ? prepare_to_swait_event+0xf3/0x140 [ 9894.658179] rcu_gp_kthread+0x8dd/0x15b0 [ 9894.664094] ? force_qs_rnp+0x2c0/0x2c0 [ 9894.666784] kthread+0x1d1/0x200 [ 9894.670304] ? set_kthread_struct+0x70/0x70 [ 9894.671984] ret_from_fork+0x1f/0x30 [ 9894.673994] rcu: Stack dump where RCU GP kthread last ran: [ 9894.676618] Sending NMI from CPU 2 to CPUs 3: [ 9894.683732] NMI backtrace for cpu 3 [ 9894.683740] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9894.683742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9894.683743] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9894.683745] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9894.683748] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 [ 9894.683751] RAX: 0000000000001c0d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9894.683752] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9894.683754] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9894.683756] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9894.683757] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9894.683759] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9894.683760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9894.683762] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9894.683763] Call Trace: [ 9894.683764] [ 9894.683765] ? show_regs.cold.9+0x22/0x2f [ 9894.683766] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9894.683767] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9894.683768] ? nmi_handle+0xbd/0x210 [ 9894.683769] ? default_do_nmi+0x4b/0x1b0 [ 9894.683770] ? do_nmi+0x280/0x380 [ 9894.683771] ? end_repeat_nmi+0x16/0x1f [ 9894.683772] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9894.683773] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9894.683774] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9894.683775] [ 9894.683776] do_raw_spin_lock+0x177/0x190 [ 9894.683777] _raw_spin_lock+0x21/0x30 [ 9894.683778] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9894.683780] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9894.683781] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9894.683782] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9894.683783] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9894.683784] ? woken_wake_function+0x30/0x30 [ 9894.683785] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9894.683786] kthread+0x1d1/0x200 [ 9894.683787] ? set_kthread_struct+0x70/0x70 [ 9894.683788] ret_from_fork+0x1f/0x30 [ 9894.684088] Sending NMI from CPU 2 to CPUs 1: [ 9894.823081] NMI backtrace for cpu 1 [ 9894.823089] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9894.823091] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9894.823093] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9894.823095] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9894.823097] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [ 9894.823099] RAX: 0000000000006c96 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9894.823100] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9894.823102] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9894.823103] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9894.823104] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9894.823106] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9894.823107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9894.823109] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9894.823110] Call Trace: [ 9894.823111] [ 9894.823112] ? show_regs.cold.9+0x22/0x2f [ 9894.823113] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [ 9894.823114] ? nmi_cpu_backtrace_handler+0x15/0x40 [ 9894.823115] ? nmi_handle+0xbd/0x210 [ 9894.823116] ? default_do_nmi+0x4b/0x1b0 [ 9894.823117] ? do_nmi+0x280/0x380 [ 9894.823118] ? end_repeat_nmi+0x16/0x1f [ 9894.823119] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9894.823121] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9894.823122] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9894.823123] [ 9894.823124] do_raw_spin_lock+0x177/0x190 [ 9894.823125] _raw_spin_lock+0x21/0x30 [ 9894.823127] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9894.823128] ? do_raw_spin_unlock+0x75/0x190 [ 9894.823129] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9894.823130] ? do_wait_intr+0xf0/0xf0 [ 9894.823131] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9894.823132] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9894.823134] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9894.823135] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9894.823136] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9894.823138] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9894.823139] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9894.823140] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9894.823141] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9894.823142] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9894.823143] lmv_intent_open+0x338/0x1150 [lmv] [ 9894.823144] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9894.823149] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9894.823150] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9894.823151] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9894.823152] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9894.823153] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9894.823154] ? do_raw_spin_unlock+0x75/0x190 [ 9894.823155] ? _raw_spin_unlock+0x12/0x30 [ 9894.823156] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9894.823157] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9894.823158] ll_atomic_open+0x36a/0xd70 [lustre] [ 9894.823159] lookup_open+0x356/0xb40 [ 9894.823160] do_last+0x313/0xfc0 [ 9894.823161] ? nd_jump_root+0xe5/0x160 [ 9894.823163] ? path_init+0x437/0x520 [ 9894.823164] path_openat+0xf7/0x500 [ 9894.823165] ? do_raw_spin_unlock+0x75/0x190 [ 9894.823166] do_filp_open+0x99/0x140 [ 9894.823167] ? getname_flags+0x6e/0x330 [ 9894.823168] ? __check_object_size+0xff/0x256 [ 9894.823169] ? do_raw_spin_unlock+0x75/0x190 [ 9894.823170] ? _raw_spin_unlock+0x12/0x30 [ 9894.823171] do_sys_openat2+0x2b4/0x410 [ 9894.823172] do_sys_open+0x73/0xa0 [ 9894.823173] __x64_sys_openat+0x24/0x30 [ 9894.823174] do_syscall_64+0xc1/0x440 [ 9894.823175] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9894.823176] RIP: 0033:0x7f53425a5332 [ 9894.823177] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9894.823180] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9894.823182] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9894.823183] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9894.823184] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9894.823186] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9894.823187] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9894.823928] NMI backtrace for cpu 2 [ 9895.003924] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9895.008823] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9895.012451] Call Trace: [ 9895.013551] [ 9895.014376] ? dump_stack+0xbb/0x10e [ 9895.015941] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [ 9895.017777] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [ 9895.019615] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [ 9895.021748] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [ 9895.024089] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [ 9895.026092] ? rcu_dump_cpu_stacks+0x11b/0x171 [ 9895.028021] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [ 9895.030347] ? update_process_times+0x6d/0xc0 [ 9895.031912] ? tick_sched_handle.isra.18+0x27/0x90 [ 9895.033212] ? tick_sched_timer+0x53/0xb0 [ 9895.037205] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9895.038746] ? tick_sched_handle.isra.18+0x90/0x90 [ 9895.041244] ? hrtimer_interrupt+0x128/0x2c0 [ 9895.045228] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9895.046599] ? apic_timer_interrupt+0xf/0x20 [ 9895.047684] [ 9895.048684] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9895.051144] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9895.056880] ? do_raw_spin_lock+0x177/0x190 [ 9895.060869] ? _raw_spin_lock+0x21/0x30 [ 9895.063847] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9895.068116] ? obd_mod_free+0x46/0x140 [ptlrpc] [ 9895.074157] ? mdc_commit_open+0x91/0xb0 [mdc] [ 9895.079556] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9895.083741] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9895.095802] ? after_reply+0xe2c/0x1780 [ptlrpc] [ 9895.097112] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9895.100466] ? __next_timer_interrupt+0x160/0x160 [ 9895.105791] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9895.108519] ? do_wait_intr+0xf0/0xf0 [ 9895.109707] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9895.112112] ? kthread+0x1d1/0x200 [ 9895.115617] ? set_kthread_struct+0x70/0x70 [ 9895.116719] ? ret_from_fork+0x1f/0x30 [ 9920.052072] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 9920.055617] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9920.058029] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 9920.058085] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9920.058093] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9920.058164] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9920.058170] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9920.058173] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9920.058192] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9920.058200] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9920.058205] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9920.058207] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9920.058209] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9920.058212] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9920.058214] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9920.058217] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9920.058220] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9920.058222] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9920.058230] Call Trace: [ 9920.058236] [ 9920.058241] ? show_regs.cold.9+0x22/0x2f [ 9920.058260] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9920.058268] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9920.058272] ? watchdog+0x40/0x40 [ 9920.058277] ? hrtimer_interrupt+0x128/0x2c0 [ 9920.058289] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9920.058297] ? apic_timer_interrupt+0xf/0x20 [ 9920.058305] [ 9920.058308] ? native_safe_halt+0x2e/0x50 [ 9920.058325] kvm_wait+0xcd/0xe0 [ 9920.058339] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9920.058349] do_raw_spin_lock+0x177/0x190 [ 9920.058358] _raw_spin_lock+0x21/0x30 [ 9920.058362] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9920.058763] ? do_raw_spin_unlock+0x75/0x190 [ 9920.058770] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9920.058976] ? do_wait_intr+0xf0/0xf0 [ 9920.058985] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9920.059304] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9920.060547] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9920.060770] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9920.060894] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9920.060933] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9920.060965] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9920.060995] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9920.061195] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9920.061207] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul [ 9920.061077] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9920.061856] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9920.061889] lmv_intent_open+0x338/0x1150 [lmv] [ 9920.061921] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9920.062014] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9920.061273] crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9920.062142] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9920.062148] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9920.062150] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9920.062170] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9920.062178] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9920.062183] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9920.062185] RDX: 000000000000293a RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9920.062190] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9920.062193] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9920.062198] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9920.062201] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9920.062207] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9920.062210] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9920.062218] Call Trace: [ 9920.062226] [ 9920.062229] ? show_regs.cold.9+0x22/0x2f [ 9920.062241] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9920.062248] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9920.062256] ? watchdog+0x40/0x40 [ 9920.062261] ? hrtimer_interrupt+0x128/0x2c0 [ 9920.062265] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9920.062279] ? apic_timer_interrupt+0xf/0x20 [ 9920.062283] [ 9920.062286] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9920.062295] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9920.062300] do_raw_spin_lock+0x177/0x190 [ 9920.062309] _raw_spin_lock+0x21/0x30 [ 9920.062842] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9920.063083] obd_mod_free+0x46/0x140 [ptlrpc] [ 9920.062098] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9920.063658] mdc_commit_open+0x91/0xb0 [mdc] [ 9920.063692] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9920.063898] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9920.064087] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9920.065737] after_reply+0xe2c/0x1780 [ptlrpc] [ 9920.065974] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9920.065126] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9920.066706] ? __next_timer_interrupt+0x160/0x160 [ 9920.066717] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9920.066942] ? do_wait_intr+0xf0/0xf0 [ 9920.066951] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9920.067166] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9920.067282] ? do_raw_spin_unlock+0x75/0x190 [ 9920.067296] ? _raw_spin_unlock+0x12/0x30 [ 9920.067306] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9920.068156] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9920.068195] ll_atomic_open+0x36a/0xd70 [lustre] [ 9920.068572] kthread+0x1d1/0x200 [ 9920.068581] ? set_kthread_struct+0x70/0x70 [ 9920.068585] ret_from_fork+0x1f/0x30 [ 9920.069172] lookup_open+0x356/0xb40 [ 9920.069190] do_last+0x313/0xfc0 [ 9920.069195] ? nd_jump_root+0xe5/0x160 [ 9920.069203] ? path_init+0x437/0x520 [ 9920.069208] path_openat+0xf7/0x500 [ 9920.069215] ? do_raw_spin_unlock+0x75/0x190 [ 9920.069222] do_filp_open+0x99/0x140 [ 9920.069228] ? getname_flags+0x6e/0x330 [ 9920.069236] ? __check_object_size+0xff/0x256 [ 9920.069247] ? do_raw_spin_unlock+0x75/0x190 [ 9920.069254] ? _raw_spin_unlock+0x12/0x30 [ 9920.069260] do_sys_openat2+0x2b4/0x410 [ 9920.069265] do_sys_open+0x73/0xa0 [ 9920.069269] __x64_sys_openat+0x24/0x30 [ 9920.069273] do_syscall_64+0xc1/0x440 [ 9920.069288] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9920.069297] RIP: 0033:0x7f53425a5332 [ 9920.069307] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9920.069324] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9920.069329] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9920.069332] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9920.069334] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9920.069337] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9920.069339] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9920.077320] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9920.077333] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9920.077340] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9920.089737] Modules linked in: [ 9920.093835] [ 9920.093845] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9920.093860] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9920.093864] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9920.093868] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9920.093870] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9920.093872] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9920.093874] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9920.093876] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9920.093879] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9920.093881] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9920.093883] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9920.093890] Call Trace: [ 9920.093895] [ 9920.093900] ? show_regs.cold.9+0x22/0x2f [ 9920.093908] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9920.093913] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9920.093917] ? watchdog+0x40/0x40 [ 9920.093920] ? hrtimer_interrupt+0x128/0x2c0 [ 9920.093923] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9920.093929] ? apic_timer_interrupt+0xf/0x20 [ 9920.093933] [ 9920.093935] ? native_safe_halt+0x2e/0x50 [ 9920.093938] kvm_wait+0xcd/0xe0 [ 9920.093945] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9920.093951] ? newidle_balance.isra.93+0x207/0x580 [ 9920.093957] do_raw_spin_lock+0x177/0x190 [ 9920.093960] _raw_spin_lock+0x21/0x30 [ 9920.093964] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9920.114983] lustre(O) [ 9920.122973] process_one_work+0x2c8/0x700 [ 9920.127232] osp(O) ofd(O) [ 9920.128957] worker_thread+0x296/0x6e0 [ 9920.128972] ? rescuer_thread+0x570/0x570 [ 9920.128975] kthread+0x1d1/0x200 [ 9920.128979] ? set_kthread_struct+0x70/0x70 [ 9920.128982] ret_from_fork+0x1f/0x30 [ 9920.556353] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9920.571301] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9920.575907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9920.578925] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9920.580586] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9920.587180] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9920.590058] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9920.593174] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9920.595369] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9920.597989] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9920.600454] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9920.602663] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9920.606034] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9920.608388] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9920.610694] Call Trace: [ 9920.611619] [ 9920.612674] ? show_regs.cold.9+0x22/0x2f [ 9920.614023] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9920.616004] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9920.617676] ? watchdog+0x40/0x40 [ 9920.619069] ? hrtimer_interrupt+0x128/0x2c0 [ 9920.621578] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9920.623563] ? apic_timer_interrupt+0xf/0x20 [ 9920.625429] [ 9920.626316] ? native_safe_halt+0x2e/0x50 [ 9920.627687] kvm_wait+0xcd/0xe0 [ 9920.629229] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9920.631118] do_raw_spin_lock+0x177/0x190 [ 9920.632556] _raw_spin_lock+0x21/0x30 [ 9920.633755] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9920.637423] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9920.639870] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9920.642303] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9920.645370] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9920.647977] ? woken_wake_function+0x30/0x30 [ 9920.649588] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9920.652197] kthread+0x1d1/0x200 [ 9920.653651] ? set_kthread_struct+0x70/0x70 [ 9920.655073] ret_from_fork+0x1f/0x30 [ 9924.065100] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7286s! [ 9924.071395] Showing busy workqueues and worker pools: [ 9924.074269] workqueue events: flags=0x0 [ 9924.075873] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9924.079202] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9924.084924] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9924.088141] pending: cpu_latency_work [ptlrpc] [ 9924.090783] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9924.094068] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9924.098363] workqueue events_long: flags=0x0 [ 9924.100181] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9924.103226] pending: lu_objects_flush [obdclass] [ 9924.105674] workqueue events_unbound: flags=0x2 [ 9924.107662] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9924.110216] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9924.112737] workqueue events_power_efficient: flags=0x80 [ 9924.114958] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9924.117933] pending: check_lifetime, do_cache_clean [ 9924.119926] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9924.122337] pending: neigh_periodic_work, neigh_periodic_work [ 9924.124911] workqueue mm_percpu_wq: flags=0x8 [ 9924.126475] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9924.128963] pending: vmstat_update, lru_add_drain_per_cpu [ 9924.131111] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9924.135561] pending: vmstat_update, lru_add_drain_per_cpu [ 9924.139668] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9924.143922] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9924.146887] workqueue xprtiod: flags=0xa [ 9924.148660] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9924.152012] pending: xprt_autoclose, xs_error_handle [ 9924.155196] workqueue ipv6_addrconf: flags=0x40008 [ 9924.157882] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9924.160877] pending: addrconf_verify_work [ 9924.162670] workqueue knbd0-recv: flags=0x1a [ 9924.164502] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9924.166956] in-flight: 651:recv_work [ 9924.172416] workqueue ptlrpc_pinger: flags=0x2 [ 9924.175434] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9924.187622] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9924.197829] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7287s workers=4 idle: 15979 31824 8 [ 9924.201662] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9914s workers=2 idle: 654 [ 9948.056942] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9948.059072] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9948.060714] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 9948.060766] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9948.060773] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9948.060838] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9948.060843] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9948.060846] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9948.060860] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9948.060865] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9948.060869] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9948.060871] RDX: 0000000000007e87 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9948.060873] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9948.060875] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9948.060877] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9948.060880] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9948.060882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9948.060884] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9948.060892] Call Trace: [ 9948.060898] [ 9948.060903] ? show_regs.cold.9+0x22/0x2f [ 9948.060914] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9948.060920] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9948.060925] ? watchdog+0x40/0x40 [ 9948.060929] ? hrtimer_interrupt+0x128/0x2c0 [ 9948.060932] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9948.060941] ? apic_timer_interrupt+0xf/0x20 [ 9948.060946] [ 9948.060948] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9948.060953] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9948.060958] do_raw_spin_lock+0x177/0x190 [ 9948.060962] _raw_spin_lock+0x21/0x30 [ 9948.060967] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9948.061238] obd_mod_free+0x46/0x140 [ptlrpc] [ 9948.061442] mdc_commit_open+0x91/0xb0 [mdc] [ 9948.061484] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9948.061692] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9948.061899] after_reply+0xe2c/0x1780 [ptlrpc] [ 9948.062098] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9948.062297] ? __next_timer_interrupt+0x160/0x160 [ 9948.062306] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9948.062542] ? do_wait_intr+0xf0/0xf0 [ 9948.062552] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9948.062752] kthread+0x1d1/0x200 [ 9948.062757] ? set_kthread_struct+0x70/0x70 [ 9948.062761] ret_from_fork+0x1f/0x30 [ 9948.096952] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9948.096964] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9948.096966] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9948.096977] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9948.096980] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9948.096984] RAX: 0000000000007f85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9948.096986] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9948.096987] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9948.096989] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9948.096990] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9948.096993] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9948.096995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9948.096996] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9948.097002] Call Trace: [ 9948.097006] [ 9948.097011] ? show_regs.cold.9+0x22/0x2f [ 9948.097018] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9948.097024] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9948.097028] ? watchdog+0x40/0x40 [ 9948.097033] ? hrtimer_interrupt+0x128/0x2c0 [ 9948.097037] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9948.097045] ? apic_timer_interrupt+0xf/0x20 [ 9948.097050] [ 9948.097053] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9948.097061] do_raw_spin_lock+0x177/0x190 [ 9948.097066] _raw_spin_lock+0x21/0x30 [ 9948.097071] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9948.097360] ? do_raw_spin_unlock+0x75/0x190 [ 9948.097366] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9948.097563] ? do_wait_intr+0xf0/0xf0 [ 9948.097572] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9948.097764] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9948.097968] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9948.098152] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9948.112042] Modules linked in: lustre(O) osp(O) [ 9948.119555] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9948.119610] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9948.119640] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9948.119670] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9948.119786] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9948.120437] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9948.120469] lmv_intent_open+0x338/0x1150 [lmv] [ 9948.120503] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9948.120594] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9948.120681] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9948.120705] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9948.120792] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9948.120950] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9948.121034] ? do_raw_spin_unlock+0x75/0x190 [ 9948.121043] ? _raw_spin_unlock+0x12/0x30 [ 9948.121050] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9948.121079] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9948.121100] ll_atomic_open+0x36a/0xd70 [lustre] [ 9948.121486] lookup_open+0x356/0xb40 [ 9948.121496] do_last+0x313/0xfc0 [ 9948.121501] ? nd_jump_root+0xe5/0x160 [ 9948.121506] ? path_init+0x437/0x520 [ 9948.121511] path_openat+0xf7/0x500 [ 9948.121516] ? do_raw_spin_unlock+0x75/0x190 [ 9948.121522] do_filp_open+0x99/0x140 [ 9948.121529] ? getname_flags+0x6e/0x330 [ 9948.121533] ? __check_object_size+0xff/0x256 [ 9948.121541] ? do_raw_spin_unlock+0x75/0x190 [ 9948.121545] ? _raw_spin_unlock+0x12/0x30 [ 9948.121550] do_sys_openat2+0x2b4/0x410 [ 9948.121554] do_sys_open+0x73/0xa0 [ 9948.121559] __x64_sys_openat+0x24/0x30 [ 9948.121562] do_syscall_64+0xc1/0x440 [ 9948.121569] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9948.121575] RIP: 0033:0x7f53425a5332 [ 9948.121583] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9948.121588] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9948.121594] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9948.121596] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9948.121599] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9948.121601] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9948.121603] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9948.677706] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9948.702503] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9948.707912] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9948.712416] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9948.715172] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9948.729114] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9948.734616] RAX: 0000000000007f84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9948.737605] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9948.741854] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9948.745277] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9948.752707] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9948.757253] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9948.760949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9948.764776] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9948.768280] Call Trace: [ 9948.769422] [ 9948.770220] ? show_regs.cold.9+0x22/0x2f [ 9948.772227] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9948.776626] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9948.778766] ? watchdog+0x40/0x40 [ 9948.781287] ? hrtimer_interrupt+0x128/0x2c0 [ 9948.782749] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9948.785476] ? apic_timer_interrupt+0xf/0x20 [ 9948.796672] [ 9948.801270] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9948.804232] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9948.811234] do_raw_spin_lock+0x177/0x190 [ 9948.813910] _raw_spin_lock+0x21/0x30 [ 9948.815851] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9948.818912] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9948.823297] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9948.828813] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9948.834497] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9948.837673] ? woken_wake_function+0x30/0x30 [ 9948.842677] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9948.849742] kthread+0x1d1/0x200 [ 9948.851544] ? set_kthread_struct+0x70/0x70 [ 9948.854847] ret_from_fork+0x1f/0x30 [ 9952.055475] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 9952.067454] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9952.117437] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9952.133472] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9952.141118] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9952.147701] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9952.150105] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9952.165642] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [ 9952.172513] RAX: 0000000000007b17 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9952.176780] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9952.182751] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9952.188850] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9952.192691] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9952.198136] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9952.204726] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9952.208810] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9952.215085] Call Trace: [ 9952.216739] [ 9952.219171] ? show_regs.cold.9+0x22/0x2f [ 9952.221173] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9952.225645] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9952.228782] ? watchdog+0x40/0x40 [ 9952.232383] ? hrtimer_interrupt+0x128/0x2c0 [ 9952.234759] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9952.238218] ? apic_timer_interrupt+0xf/0x20 [ 9952.243128] [ 9952.244765] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9952.251189] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9952.253084] ? newidle_balance.isra.93+0x207/0x580 [ 9952.262836] do_raw_spin_lock+0x177/0x190 [ 9952.264202] _raw_spin_lock+0x21/0x30 [ 9952.267950] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9952.273490] process_one_work+0x2c8/0x700 [ 9952.274646] worker_thread+0x296/0x6e0 [ 9952.280703] ? rescuer_thread+0x570/0x570 [ 9952.282084] kthread+0x1d1/0x200 [ 9952.285070] ? set_kthread_struct+0x70/0x70 [ 9952.289544] ret_from_fork+0x1f/0x30 [ 9954.281062] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7317s! [ 9954.285182] Showing busy workqueues and worker pools: [ 9954.287210] workqueue events: flags=0x0 [ 9954.288707] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9954.290908] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9954.295577] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9954.298080] pending: cpu_latency_work [ptlrpc] [ 9954.300214] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9954.303037] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9954.306778] workqueue events_long: flags=0x0 [ 9954.308726] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9954.311118] pending: lu_objects_flush [obdclass] [ 9954.312802] workqueue events_unbound: flags=0x2 [ 9954.313897] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9954.315782] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9954.317174] workqueue events_power_efficient: flags=0x80 [ 9954.318861] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9954.321582] pending: check_lifetime, do_cache_clean [ 9954.323662] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9954.325446] pending: neigh_periodic_work, neigh_periodic_work [ 9954.327568] workqueue mm_percpu_wq: flags=0x8 [ 9954.328886] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9954.331550] pending: vmstat_update, lru_add_drain_per_cpu [ 9954.333908] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9954.335994] pending: vmstat_update, lru_add_drain_per_cpu [ 9954.337482] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9954.339267] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9954.340726] workqueue xprtiod: flags=0xa [ 9954.342173] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9954.344855] pending: xprt_autoclose, xs_error_handle [ 9954.347098] workqueue ipv6_addrconf: flags=0x40008 [ 9954.348841] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9954.351117] pending: addrconf_verify_work [ 9954.352491] workqueue knbd0-recv: flags=0x1a [ 9954.353857] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9954.356296] in-flight: 651:recv_work [ 9954.357854] workqueue ptlrpc_pinger: flags=0x2 [ 9954.359434] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9954.361935] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9954.364638] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7317s workers=4 idle: 15979 31824 8 [ 9954.367729] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9944s workers=2 idle: 654 [ 9976.055727] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [ 9976.057013] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [ 9976.059993] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [ 9976.060007] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9976.060088] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9976.060094] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9976.060096] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9976.060110] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9976.060114] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9976.060118] RAX: 0000000000007391 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9976.060121] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [ 9976.060123] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9976.060126] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [ 9976.060128] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [ 9976.060132] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [ 9976.060135] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9976.060137] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [ 9976.060146] Call Trace: [ 9976.060151] [ 9976.060157] ? show_regs.cold.9+0x22/0x2f [ 9976.060168] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9976.060174] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9976.060178] ? watchdog+0x40/0x40 [ 9976.060182] ? hrtimer_interrupt+0x128/0x2c0 [ 9976.060186] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9976.060193] ? apic_timer_interrupt+0xf/0x20 [ 9976.060198] [ 9976.060201] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9976.060206] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9976.060212] do_raw_spin_lock+0x177/0x190 [ 9976.060216] _raw_spin_lock+0x21/0x30 [ 9976.060221] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [ 9976.060470] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [ 9976.060662] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [ 9976.060894] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [ 9976.061123] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [ 9976.061357] ? woken_wake_function+0x30/0x30 [ 9976.061365] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [ 9976.061581] kthread+0x1d1/0x200 [ 9976.061587] ? set_kthread_struct+0x70/0x70 [ 9976.061590] ret_from_fork+0x1f/0x30 [ 9976.062709] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9976.062791] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9976.062799] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9976.068714] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9976.068793] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9976.068802] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9976.068805] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9976.068819] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [ 9976.068824] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9976.068829] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [ 9976.068831] RDX: 0000000000005c7e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [ 9976.068834] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [ 9976.073997] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [ 9976.074003] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [ 9976.074006] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [ 9976.074009] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9976.074012] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [ 9976.074020] Call Trace: [ 9976.074027] [ 9976.074032] ? show_regs.cold.9+0x22/0x2f [ 9976.074046] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9976.074054] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9976.074058] ? watchdog+0x40/0x40 [ 9976.074062] ? hrtimer_interrupt+0x128/0x2c0 [ 9976.074067] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9976.074075] ? apic_timer_interrupt+0xf/0x20 [ 9976.074080] [ 9976.074084] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [ 9976.079714] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9976.079737] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [ 9976.079741] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9976.079747] RAX: 000000000000778a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9976.079749] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [ 9976.079752] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [ 9976.079754] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [ 9976.079757] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [ 9976.079760] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [ 9976.079764] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9976.079765] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [ 9976.079774] Call Trace: [ 9976.079779] [ 9976.079785] ? show_regs.cold.9+0x22/0x2f [ 9976.079793] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9976.079800] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9976.079804] ? watchdog+0x40/0x40 [ 9976.079809] ? hrtimer_interrupt+0x128/0x2c0 [ 9976.079813] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9976.079821] ? apic_timer_interrupt+0xf/0x20 [ 9976.079826] [ 9976.079828] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [ 9976.079845] do_raw_spin_lock+0x177/0x190 [ 9976.079851] _raw_spin_lock+0x21/0x30 [ 9976.079856] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [ 9976.080393] ? do_raw_spin_unlock+0x75/0x190 [ 9976.080400] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [ 9976.081359] ? do_wait_intr+0xf0/0xf0 [ 9976.081369] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [ 9976.131421] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [ 9976.131441] do_raw_spin_lock+0x177/0x190 [ 9976.131447] _raw_spin_lock+0x21/0x30 [ 9976.131455] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [ 9976.131741] obd_mod_free+0x46/0x140 [ptlrpc] [ 9976.131950] mdc_commit_open+0x91/0xb0 [mdc] [ 9976.131987] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [ 9976.132181] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [ 9976.132379] after_reply+0xe2c/0x1780 [ptlrpc] [ 9976.141355] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [ 9976.141803] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9976.142147] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9976.142259] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9976.142308] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [ 9976.142339] mdc_intent_lock+0x353/0x7e0 [mdc] [ 9976.142367] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9976.142539] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [ 9976.142772] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [ 9976.142802] lmv_intent_open+0x338/0x1150 [lmv] [ 9976.142895] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9976.142989] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9976.143078] lmv_intent_lock+0x1dd/0x4c0 [lmv] [ 9976.143102] ll_intent_lock+0xc5/0xaa0 [lustre] [ 9976.143191] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [ 9976.143281] ll_lookup_it+0xe0e/0x2270 [lustre] [ 9976.143367] ? do_raw_spin_unlock+0x75/0x190 [ 9976.143375] ? _raw_spin_unlock+0x12/0x30 [ 9976.143382] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [ 9976.143408] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [ 9976.143425] ll_atomic_open+0x36a/0xd70 [lustre] [ 9976.143821] lookup_open+0x356/0xb40 [ 9976.143830] do_last+0x313/0xfc0 [ 9976.143900] ? nd_jump_root+0xe5/0x160 [ 9976.143905] ? path_init+0x437/0x520 [ 9976.143910] path_openat+0xf7/0x500 [ 9976.143915] ? do_raw_spin_unlock+0x75/0x190 [ 9976.143920] do_filp_open+0x99/0x140 [ 9976.143927] ? getname_flags+0x6e/0x330 [ 9976.143932] ? __check_object_size+0xff/0x256 [ 9976.143939] ? do_raw_spin_unlock+0x75/0x190 [ 9976.143943] ? _raw_spin_unlock+0x12/0x30 [ 9976.143948] do_sys_openat2+0x2b4/0x410 [ 9976.143953] do_sys_open+0x73/0xa0 [ 9976.143958] __x64_sys_openat+0x24/0x30 [ 9976.143962] do_syscall_64+0xc1/0x440 [ 9976.143968] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 9976.143974] RIP: 0033:0x7f53425a5332 [ 9976.143982] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [ 9976.143986] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [ 9976.143992] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [ 9976.143994] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [ 9976.143997] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [ 9976.143999] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [ 9976.144002] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [ 9976.507157] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [ 9976.510047] ? __next_timer_interrupt+0x160/0x160 [ 9976.512109] ptlrpcd+0xa82/0xe40 [ptlrpc] [ 9976.514434] ? do_wait_intr+0xf0/0xf0 [ 9976.516416] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [ 9976.520175] kthread+0x1d1/0x200 [ 9976.522204] ? set_kthread_struct+0x70/0x70 [ 9976.524838] ret_from_fork+0x1f/0x30 [ 9980.052690] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [ 9980.055380] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 9980.073608] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [ 9980.078757] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 9980.084136] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [ 9980.086945] RIP: 0010:native_safe_halt+0x2e/0x50 [ 9980.089161] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [ 9980.100620] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [ 9980.106216] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [ 9980.111522] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [ 9980.116198] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [ 9980.125008] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [ 9980.134516] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [ 9980.137472] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [ 9980.142450] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9980.144483] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [ 9980.149461] Call Trace: [ 9980.150410] [ 9980.151319] ? show_regs.cold.9+0x22/0x2f [ 9980.154159] ? watchdog_timer_fn.cold.6+0x62/0x10a [ 9980.155684] ? __hrtimer_run_queues+0x1c2/0x4d0 [ 9980.159365] ? watchdog+0x40/0x40 [ 9980.161395] ? hrtimer_interrupt+0x128/0x2c0 [ 9980.165276] ? smp_apic_timer_interrupt+0xa6/0x270 [ 9980.171415] ? apic_timer_interrupt+0xf/0x20 [ 9980.173395] [ 9980.174363] ? native_safe_halt+0x2e/0x50 [ 9980.176202] kvm_wait+0xcd/0xe0 [ 9980.177471] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [ 9980.180065] ? newidle_balance.isra.93+0x207/0x580 [ 9980.182461] do_raw_spin_lock+0x177/0x190 [ 9980.185755] _raw_spin_lock+0x21/0x30 [ 9980.187727] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [ 9980.190222] process_one_work+0x2c8/0x700 [ 9980.195260] worker_thread+0x296/0x6e0 [ 9980.198966] ? rescuer_thread+0x570/0x570 [ 9980.201945] kthread+0x1d1/0x200 [ 9980.204338] ? set_kthread_struct+0x70/0x70 [ 9980.211102] ret_from_fork+0x1f/0x30 [ 9984.481921] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7347s! [ 9984.489301] Showing busy workqueues and worker pools: [ 9984.494900] workqueue events: flags=0x0 [ 9984.497316] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [ 9984.500815] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [ 9984.509242] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9984.516285] pending: cpu_latency_work [ptlrpc] [ 9984.519058] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [ 9984.526258] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [ 9984.534969] workqueue events_long: flags=0x0 [ 9984.541973] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [ 9984.550796] pending: lu_objects_flush [obdclass] [ 9984.555299] workqueue events_unbound: flags=0x2 [ 9984.561953] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [ 9984.565296] pending: flush_memcg_stats_dwork, idle_cull_fn [ 9984.569011] workqueue events_power_efficient: flags=0x80 [ 9984.574487] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9984.580001] pending: check_lifetime, do_cache_clean [ 9984.587867] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9984.593238] pending: neigh_periodic_work, neigh_periodic_work [ 9984.596541] workqueue mm_percpu_wq: flags=0x8 [ 9984.600170] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9984.605131] pending: vmstat_update, lru_add_drain_per_cpu [ 9984.609638] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [ 9984.613141] pending: vmstat_update, lru_add_drain_per_cpu [ 9984.616982] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [ 9984.619388] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [ 9984.623835] workqueue xprtiod: flags=0xa [ 9984.627889] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [ 9984.630970] pending: xprt_autoclose, xs_error_handle [ 9984.633238] workqueue ipv6_addrconf: flags=0x40008 [ 9984.635537] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [ 9984.640464] pending: addrconf_verify_work [ 9984.642355] workqueue knbd0-recv: flags=0x1a [ 9984.645621] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [ 9984.649885] in-flight: 651:recv_work [ 9984.652217] workqueue ptlrpc_pinger: flags=0x2 [ 9984.654109] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [ 9984.656215] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [ 9984.660223] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7347s workers=4 idle: 15979 31824 8 [ 9984.664628] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=9975s workers=2 idle: 654 [10004.055553] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [10004.059372] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [10004.059421] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [10004.059428] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10004.059501] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10004.059508] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10004.059511] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10004.059525] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10004.059530] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10004.059534] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10004.059536] RDX: 0000000000004cdf RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10004.059538] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10004.059540] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10004.059543] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10004.059546] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10004.059549] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10004.059551] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10004.059560] Call Trace: [10004.059565] [10004.059571] ? show_regs.cold.9+0x22/0x2f [10004.059582] ? watchdog_timer_fn.cold.6+0x62/0x10a [10004.059588] ? __hrtimer_run_queues+0x1c2/0x4d0 [10004.059593] ? watchdog+0x40/0x40 [10004.059597] ? hrtimer_interrupt+0x128/0x2c0 [10004.059601] ? smp_apic_timer_interrupt+0xa6/0x270 [10004.059610] ? apic_timer_interrupt+0xf/0x20 [10004.059615] [10004.059618] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10004.059622] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10004.059627] do_raw_spin_lock+0x177/0x190 [10004.059632] _raw_spin_lock+0x21/0x30 [10004.059638] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10004.059927] obd_mod_free+0x46/0x140 [ptlrpc] [10004.060130] mdc_commit_open+0x91/0xb0 [mdc] [10004.060168] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10004.061901] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror [10004.061954] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10004.062203] dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10004.062213] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10004.062218] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10004.062221] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10004.062231] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10004.062235] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10004.062239] RAX: 0000000000000d7f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10004.062241] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10004.062243] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10004.062245] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10004.062247] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10004.062250] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10004.062253] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10004.062256] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10004.062264] Call Trace: [10004.062267] [10004.062270] ? show_regs.cold.9+0x22/0x2f [10004.062279] ? watchdog_timer_fn.cold.6+0x62/0x10a [10004.062284] ? __hrtimer_run_queues+0x1c2/0x4d0 [10004.062288] ? watchdog+0x40/0x40 [10004.062292] ? hrtimer_interrupt+0x128/0x2c0 [10004.062295] ? smp_apic_timer_interrupt+0xa6/0x270 [10004.062303] ? apic_timer_interrupt+0xf/0x20 [10004.062308] [10004.062311] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10004.062318] do_raw_spin_lock+0x177/0x190 [10004.062322] _raw_spin_lock+0x21/0x30 [10004.062327] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10004.063706] ? do_raw_spin_unlock+0x75/0x190 [10004.063714] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10004.064224] ? do_wait_intr+0xf0/0xf0 [10004.064234] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10004.065134] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10004.065382] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10004.065600] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10004.065702] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10004.065742] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10004.065773] mdc_intent_lock+0x353/0x7e0 [mdc] [10004.065801] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10004.065884] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10004.067117] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10004.067143] lmv_intent_open+0x338/0x1150 [lmv] [10004.067163] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10004.067921] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10004.068033] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10004.068052] ll_intent_lock+0xc5/0xaa0 [lustre] [10004.068114] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10004.068173] ll_lookup_it+0xe0e/0x2270 [lustre] [10004.068245] ? do_raw_spin_unlock+0x75/0x190 [10004.068253] ? _raw_spin_unlock+0x12/0x30 [10004.068262] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10004.068293] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10004.068315] ll_atomic_open+0x36a/0xd70 [lustre] [10004.068409] lookup_open+0x356/0xb40 [10004.068419] do_last+0x313/0xfc0 [10004.068424] ? nd_jump_root+0xe5/0x160 [10004.068796] ? path_init+0x437/0x520 [10004.068802] path_openat+0xf7/0x500 [10004.068806] ? do_raw_spin_unlock+0x75/0x190 [10004.068810] do_filp_open+0x99/0x140 [10004.068814] ? getname_flags+0x6e/0x330 [10004.068817] ? __check_object_size+0xff/0x256 [10004.068823] ? do_raw_spin_unlock+0x75/0x190 [10004.068826] ? _raw_spin_unlock+0x12/0x30 [10004.068829] do_sys_openat2+0x2b4/0x410 [10004.068832] do_sys_open+0x73/0xa0 [10004.068835] __x64_sys_openat+0x24/0x30 [10004.068837] do_syscall_64+0xc1/0x440 [10004.068842] entry_SYSCALL_64_after_hwframe+0x49/0xae [10004.068847] RIP: 0033:0x7f53425a5332 [10004.068854] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10004.068857] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10004.068861] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10004.068863] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10004.068864] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10004.068869] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10004.068871] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10004.071267] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10004.071878] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10004.071886] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10004.071889] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10004.071901] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10004.071906] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10004.071910] RAX: 0000000000007f73 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10004.071912] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10004.071914] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10004.071917] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10004.071919] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10004.071923] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10004.071926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10004.071928] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10004.071937] Call Trace: [10004.071942] [10004.071947] ? show_regs.cold.9+0x22/0x2f [10004.071958] ? watchdog_timer_fn.cold.6+0x62/0x10a [10004.071964] ? __hrtimer_run_queues+0x1c2/0x4d0 [10004.071968] ? watchdog+0x40/0x40 [10004.071973] ? hrtimer_interrupt+0x128/0x2c0 [10004.071977] ? smp_apic_timer_interrupt+0xa6/0x270 [10004.071985] ? apic_timer_interrupt+0xf/0x20 [10004.071990] [10004.071993] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10004.071998] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10004.072005] do_raw_spin_lock+0x177/0x190 [10004.072010] _raw_spin_lock+0x21/0x30 [10004.072016] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10004.072315] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10004.080116] after_reply+0xe2c/0x1780 [ptlrpc] [10004.080936] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10004.081152] ? __next_timer_interrupt+0x160/0x160 [10004.081163] ptlrpcd+0xa82/0xe40 [ptlrpc] [10004.081914] ? do_wait_intr+0xf0/0xf0 [10004.081925] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10004.082151] kthread+0x1d1/0x200 [10004.082157] ? set_kthread_struct+0x70/0x70 [10004.082161] ret_from_fork+0x1f/0x30 [10004.716295] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10004.727945] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10004.736133] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10004.742260] ? woken_wake_function+0x30/0x30 [10004.748624] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10004.753201] kthread+0x1d1/0x200 [10004.754711] ? set_kthread_struct+0x70/0x70 [10004.756009] ret_from_fork+0x1f/0x30 [10012.052006] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [10012.054272] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10012.084733] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10012.091717] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10012.096599] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10012.100425] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10012.104114] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10012.120129] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10012.126630] RAX: 00000000000072be RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10012.132281] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10012.135622] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10012.140787] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10012.144894] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10012.148091] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10012.153475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10012.157895] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10012.162137] Call Trace: [10012.163846] [10012.165000] ? show_regs.cold.9+0x22/0x2f [10012.166101] ? watchdog_timer_fn.cold.6+0x62/0x10a [10012.170372] ? __hrtimer_run_queues+0x1c2/0x4d0 [10012.171651] ? watchdog+0x40/0x40 [10012.175586] ? hrtimer_interrupt+0x128/0x2c0 [10012.177325] ? smp_apic_timer_interrupt+0xa6/0x270 [10012.181656] ? apic_timer_interrupt+0xf/0x20 [10012.183623] [10012.184353] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10012.188631] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10012.190612] ? newidle_balance.isra.93+0x207/0x580 [10012.194647] do_raw_spin_lock+0x177/0x190 [10012.196055] _raw_spin_lock+0x21/0x30 [10012.199656] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10012.202454] process_one_work+0x2c8/0x700 [10012.205626] worker_thread+0x296/0x6e0 [10012.207745] ? rescuer_thread+0x570/0x570 [10012.209353] kthread+0x1d1/0x200 [10012.213067] ? set_kthread_struct+0x70/0x70 [10012.214384] ret_from_fork+0x1f/0x30 [10014.689226] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7377s! [10014.693192] Showing busy workqueues and worker pools: [10014.700116] workqueue events: flags=0x0 [10014.704624] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10014.708223] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10014.729458] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10014.741285] pending: cpu_latency_work [ptlrpc] [10014.749430] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10014.757175] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10014.770105] workqueue events_long: flags=0x0 [10014.779526] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10014.792912] pending: lu_objects_flush [obdclass] [10014.798283] workqueue events_unbound: flags=0x2 [10014.800814] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10014.807136] pending: flush_memcg_stats_dwork, idle_cull_fn [10014.810822] workqueue events_power_efficient: flags=0x80 [10014.816608] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10014.820123] pending: check_lifetime, do_cache_clean [10014.824933] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10014.831811] pending: neigh_periodic_work, neigh_periodic_work [10014.837765] workqueue mm_percpu_wq: flags=0x8 [10014.841537] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10014.848447] pending: vmstat_update, lru_add_drain_per_cpu [10014.857486] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10014.862333] pending: vmstat_update, lru_add_drain_per_cpu [10014.868033] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10014.874230] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10014.879081] workqueue xprtiod: flags=0xa [10014.885533] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10014.895418] pending: xprt_autoclose, xs_error_handle [10014.900659] workqueue ipv6_addrconf: flags=0x40008 [10014.905138] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10014.911792] pending: addrconf_verify_work [10014.917686] workqueue knbd0-recv: flags=0x1a [10014.924718] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10014.928021] in-flight: 651:recv_work [10014.932740] workqueue ptlrpc_pinger: flags=0x2 [10014.937290] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10014.946529] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10014.955880] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7377s workers=4 idle: 15979 31824 8 [10014.964237] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10005s workers=2 idle: 654 [10032.055086] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [10032.057007] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [10032.058007] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [10032.058014] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10032.058095] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10032.058101] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10032.058104] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10032.058117] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10032.058121] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10032.058126] RAX: 0000000000004d37 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10032.058128] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10032.058137] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10032.058139] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10032.058142] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10032.058145] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10032.058148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10032.058151] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10032.058158] Call Trace: [10032.058163] [10032.058168] ? show_regs.cold.9+0x22/0x2f [10032.058177] ? watchdog_timer_fn.cold.6+0x62/0x10a [10032.058183] ? __hrtimer_run_queues+0x1c2/0x4d0 [10032.058187] ? watchdog+0x40/0x40 [10032.058287] ? hrtimer_interrupt+0x128/0x2c0 [10032.058292] ? smp_apic_timer_interrupt+0xa6/0x270 [10032.058300] ? apic_timer_interrupt+0xf/0x20 [10032.058305] [10032.058308] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10032.058313] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10032.058319] do_raw_spin_lock+0x177/0x190 [10032.058324] _raw_spin_lock+0x21/0x30 [10032.058379] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10032.058446] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10032.058451] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10032.058453] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10032.058461] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10032.058475] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10032.058479] RAX: 0000000000003cc1 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10032.058482] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10032.058484] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10032.058486] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10032.058489] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10032.058495] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10032.058498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10032.058501] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10032.058506] Call Trace: [10032.058508] [10032.058329] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10032.058511] ? show_regs.cold.9+0x22/0x2f [10032.058520] ? watchdog_timer_fn.cold.6+0x62/0x10a [10032.058525] ? __hrtimer_run_queues+0x1c2/0x4d0 [10032.058528] ? watchdog+0x40/0x40 [10032.058532] ? hrtimer_interrupt+0x128/0x2c0 [10032.058537] ? smp_apic_timer_interrupt+0xa6/0x270 [10032.058542] ? apic_timer_interrupt+0xf/0x20 [10032.058546] [10032.058549] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10032.058554] do_raw_spin_lock+0x177/0x190 [10032.058559] _raw_spin_lock+0x21/0x30 [10032.058563] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10032.059186] ? do_raw_spin_unlock+0x75/0x190 [10032.059192] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10032.058627] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10032.059881] ? do_wait_intr+0xf0/0xf0 [10032.059890] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10032.059642] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10032.060400] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10032.061058] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10032.061297] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10032.061428] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10032.061540] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10032.061631] mdc_intent_lock+0x353/0x7e0 [mdc] [10032.061660] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10032.061745] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10032.060613] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10032.063035] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10032.063471] ? woken_wake_function+0x30/0x30 [10032.063481] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10032.063801] kthread+0x1d1/0x200 [10032.063808] ? set_kthread_struct+0x70/0x70 [10032.063813] ret_from_fork+0x1f/0x30 [10032.064846] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) [10032.068186] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10032.087598] ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) [10032.091529] lmv_intent_open+0x338/0x1150 [lmv] [10032.091623] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10032.091782] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10032.091844] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10032.091878] ll_intent_lock+0xc5/0xaa0 [lustre] [10032.091959] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10032.092028] ll_lookup_it+0xe0e/0x2270 [lustre] [10032.092092] ? do_raw_spin_unlock+0x75/0x190 [10032.092098] ? _raw_spin_unlock+0x12/0x30 [10032.092103] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10032.092122] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10032.092136] ll_atomic_open+0x36a/0xd70 [lustre] [10032.092199] lookup_open+0x356/0xb40 [10032.092207] do_last+0x313/0xfc0 [10032.092213] ? nd_jump_root+0xe5/0x160 [10032.092218] ? path_init+0x437/0x520 [10032.092223] path_openat+0xf7/0x500 [10032.092228] ? do_raw_spin_unlock+0x75/0x190 [10032.092233] do_filp_open+0x99/0x140 [10032.092238] ? getname_flags+0x6e/0x330 [10032.092241] ? __check_object_size+0xff/0x256 [10032.092246] ? do_raw_spin_unlock+0x75/0x190 [10032.092249] ? _raw_spin_unlock+0x12/0x30 [10032.092252] do_sys_openat2+0x2b4/0x410 [10032.092255] do_sys_open+0x73/0xa0 [10032.092257] __x64_sys_openat+0x24/0x30 [10032.092260] do_syscall_64+0xc1/0x440 [10032.092264] entry_SYSCALL_64_after_hwframe+0x49/0xae [10032.092269] RIP: 0033:0x7f53425a5332 [10032.092279] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10032.092282] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10032.092286] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10032.092287] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10032.092289] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10032.092290] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10032.092292] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10032.609172] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10032.630429] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10032.648613] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10032.664177] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10032.668339] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10032.676633] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10032.682651] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10032.687067] RDX: 0000000000007da9 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10032.689831] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10032.694988] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10032.697386] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10032.707694] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10032.710995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10032.714221] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10032.718260] Call Trace: [10032.720231] [10032.721428] ? show_regs.cold.9+0x22/0x2f [10032.723664] ? watchdog_timer_fn.cold.6+0x62/0x10a [10032.726298] ? __hrtimer_run_queues+0x1c2/0x4d0 [10032.728201] ? watchdog+0x40/0x40 [10032.729571] ? hrtimer_interrupt+0x128/0x2c0 [10032.731377] ? smp_apic_timer_interrupt+0xa6/0x270 [10032.734687] ? apic_timer_interrupt+0xf/0x20 [10032.736607] [10032.738031] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10032.740584] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10032.751776] do_raw_spin_lock+0x177/0x190 [10032.753240] _raw_spin_lock+0x21/0x30 [10032.757146] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10032.758904] obd_mod_free+0x46/0x140 [ptlrpc] [10032.761657] mdc_commit_open+0x91/0xb0 [mdc] [10032.765499] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10032.771655] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10032.782457] after_reply+0xe2c/0x1780 [ptlrpc] [10032.784280] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10032.786541] ? __next_timer_interrupt+0x160/0x160 [10032.788692] ptlrpcd+0xa82/0xe40 [ptlrpc] [10032.791228] ? do_wait_intr+0xf0/0xf0 [10032.792913] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10032.795266] kthread+0x1d1/0x200 [10032.796526] ? set_kthread_struct+0x70/0x70 [10032.798306] ret_from_fork+0x1f/0x30 [10040.054898] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [10040.061043] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10040.106323] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10040.117022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10040.125198] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10040.135152] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10040.137150] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10040.157131] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10040.163771] RAX: 0000000000005d83 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10040.166560] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10040.172889] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10040.179166] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10040.191504] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10040.196207] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10040.209080] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10040.216115] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10040.218177] Call Trace: [10040.222081] [10040.223533] ? show_regs.cold.9+0x22/0x2f [10040.226201] ? watchdog_timer_fn.cold.6+0x62/0x10a [10040.230985] ? __hrtimer_run_queues+0x1c2/0x4d0 [10040.238154] ? watchdog+0x40/0x40 [10040.241170] ? hrtimer_interrupt+0x128/0x2c0 [10040.246141] ? smp_apic_timer_interrupt+0xa6/0x270 [10040.249108] ? apic_timer_interrupt+0xf/0x20 [10040.251064] [10040.252308] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10040.259588] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10040.267142] ? newidle_balance.isra.93+0x207/0x580 [10040.269665] do_raw_spin_lock+0x177/0x190 [10040.274863] _raw_spin_lock+0x21/0x30 [10040.281326] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10040.285163] process_one_work+0x2c8/0x700 [10040.287005] worker_thread+0x296/0x6e0 [10040.290844] ? rescuer_thread+0x570/0x570 [10040.294926] kthread+0x1d1/0x200 [10040.296241] ? set_kthread_struct+0x70/0x70 [10040.298281] ret_from_fork+0x1f/0x30 [10045.407094] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7408s! [10045.412323] Showing busy workqueues and worker pools: [10045.415498] workqueue events: flags=0x0 [10045.417788] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10045.421551] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10045.428576] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10045.435686] pending: cpu_latency_work [ptlrpc] [10045.443915] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10045.452323] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10045.461600] workqueue events_long: flags=0x0 [10045.467340] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10045.478508] pending: lu_objects_flush [obdclass] [10045.482908] workqueue events_unbound: flags=0x2 [10045.488848] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10045.493278] pending: flush_memcg_stats_dwork, idle_cull_fn [10045.496328] workqueue events_power_efficient: flags=0x80 [10045.498114] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10045.502448] pending: check_lifetime, do_cache_clean [10045.505455] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10045.513117] pending: neigh_periodic_work, neigh_periodic_work [10045.517170] workqueue mm_percpu_wq: flags=0x8 [10045.521404] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10045.525195] pending: vmstat_update, lru_add_drain_per_cpu [10045.527452] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10045.533509] pending: vmstat_update, lru_add_drain_per_cpu [10045.536957] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10045.540063] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10045.542878] workqueue xprtiod: flags=0xa [10045.552144] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10045.559377] pending: xprt_autoclose, xs_error_handle [10045.561680] workqueue ipv6_addrconf: flags=0x40008 [10045.562861] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10045.568123] pending: addrconf_verify_work [10045.569614] workqueue knbd0-recv: flags=0x1a [10045.573306] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10045.579554] in-flight: 651:recv_work [10045.582441] workqueue ptlrpc_pinger: flags=0x2 [10045.585123] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10045.590992] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10045.601313] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7408s workers=4 idle: 15979 31824 8 [10045.610077] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10036s workers=2 idle: 654 [10060.055908] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [10060.059430] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10060.062032] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [10060.063763] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [10060.063854] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10060.063927] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10060.063933] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10060.063937] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10060.063951] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10060.063956] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10060.063961] RAX: 0000000000005b03 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10060.063964] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10060.063967] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10060.063969] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10060.063972] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10060.063976] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10060.063980] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10060.063982] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10060.063992] Call Trace: [10060.063999] [10060.064006] ? show_regs.cold.9+0x22/0x2f [10060.064017] ? watchdog_timer_fn.cold.6+0x62/0x10a [10060.064024] ? __hrtimer_run_queues+0x1c2/0x4d0 [10060.064030] ? watchdog+0x40/0x40 [10060.064034] ? hrtimer_interrupt+0x128/0x2c0 [10060.064039] ? smp_apic_timer_interrupt+0xa6/0x270 [10060.064049] ? apic_timer_interrupt+0xf/0x20 [10060.064054] [10060.064057] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10060.064132] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10060.064139] do_raw_spin_lock+0x177/0x190 [10060.064144] _raw_spin_lock+0x21/0x30 [10060.064150] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10060.066413] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10060.068934] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10060.069449] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10060.070954] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10060.075560] ? woken_wake_function+0x30/0x30 [10060.075574] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10060.075871] kthread+0x1d1/0x200 [10060.075877] ? set_kthread_struct+0x70/0x70 [10060.075881] ret_from_fork+0x1f/0x30 [10060.096659] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10060.096838] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10060.096841] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10060.096856] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10060.096860] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10060.096866] RAX: 0000000000000017 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10060.096869] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10060.096871] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10060.096874] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10060.096876] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10060.096879] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10060.096882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10060.096885] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10060.096893] Call Trace: [10060.096899] [10060.096905] ? show_regs.cold.9+0x22/0x2f [10060.096914] ? watchdog_timer_fn.cold.6+0x62/0x10a [10060.096921] ? __hrtimer_run_queues+0x1c2/0x4d0 [10060.096925] ? watchdog+0x40/0x40 [10060.096929] ? hrtimer_interrupt+0x128/0x2c0 [10060.096933] ? smp_apic_timer_interrupt+0xa6/0x270 [10060.096942] ? apic_timer_interrupt+0xf/0x20 [10060.096948] [10060.096951] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10060.096960] do_raw_spin_lock+0x177/0x190 [10060.096965] _raw_spin_lock+0x21/0x30 [10060.096970] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10060.103054] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10060.103138] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10060.103145] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10060.103148] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10060.103162] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10060.103166] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10060.103170] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10060.103173] RDX: 00000000000052b0 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10060.103176] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10060.103178] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10060.103181] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10060.103184] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10060.103186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10060.103189] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10060.103197] Call Trace: [10060.103202] [10060.103207] ? show_regs.cold.9+0x22/0x2f [10060.103215] ? watchdog_timer_fn.cold.6+0x62/0x10a [10060.103221] ? __hrtimer_run_queues+0x1c2/0x4d0 [10060.103225] ? watchdog+0x40/0x40 [10060.103229] ? hrtimer_interrupt+0x128/0x2c0 [10060.103233] ? smp_apic_timer_interrupt+0xa6/0x270 [10060.103241] ? apic_timer_interrupt+0xf/0x20 [10060.103245] [10060.103248] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10060.103252] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10060.103257] do_raw_spin_lock+0x177/0x190 [10060.103261] _raw_spin_lock+0x21/0x30 [10060.103266] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10060.104815] obd_mod_free+0x46/0x140 [ptlrpc] [10060.105021] mdc_commit_open+0x91/0xb0 [mdc] [10060.105062] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10060.105262] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10060.106650] after_reply+0xe2c/0x1780 [ptlrpc] [10060.106882] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10060.114885] ? do_raw_spin_unlock+0x75/0x190 [10060.114907] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10060.115189] ? do_wait_intr+0xf0/0xf0 [10060.115198] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10060.117303] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10060.117620] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10060.144001] ? __next_timer_interrupt+0x160/0x160 [10060.152714] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10060.152891] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10060.152944] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10060.152980] mdc_intent_lock+0x353/0x7e0 [mdc] [10060.153015] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10060.153114] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10060.153546] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10060.153576] lmv_intent_open+0x338/0x1150 [lmv] [10060.153603] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10060.153878] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10060.153961] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10060.153988] ll_intent_lock+0xc5/0xaa0 [lustre] [10060.154071] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10060.154154] ll_lookup_it+0xe0e/0x2270 [lustre] [10060.154243] ? do_raw_spin_unlock+0x75/0x190 [10060.154255] ? _raw_spin_unlock+0x12/0x30 [10060.154265] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10060.154296] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10060.154314] ll_atomic_open+0x36a/0xd70 [lustre] [10060.154408] lookup_open+0x356/0xb40 [10060.154419] do_last+0x313/0xfc0 [10060.154424] ? nd_jump_root+0xe5/0x160 [10060.154429] ? path_init+0x437/0x520 [10060.154433] path_openat+0xf7/0x500 [10060.154438] ? do_raw_spin_unlock+0x75/0x190 [10060.154443] do_filp_open+0x99/0x140 [10060.154449] ? getname_flags+0x6e/0x330 [10060.154454] ? __check_object_size+0xff/0x256 [10060.154462] ? do_raw_spin_unlock+0x75/0x190 [10060.154466] ? _raw_spin_unlock+0x12/0x30 [10060.154471] do_sys_openat2+0x2b4/0x410 [10060.154475] do_sys_open+0x73/0xa0 [10060.154479] __x64_sys_openat+0x24/0x30 [10060.154483] do_syscall_64+0xc1/0x440 [10060.154490] entry_SYSCALL_64_after_hwframe+0x49/0xae [10060.154497] RIP: 0033:0x7f53425a5332 [10060.154505] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10060.154510] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10060.154516] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10060.154518] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10060.154520] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10060.154523] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10060.154525] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10060.878736] ptlrpcd+0xa82/0xe40 [ptlrpc] [10060.881690] ? do_wait_intr+0xf0/0xf0 [10060.883689] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10060.890779] kthread+0x1d1/0x200 [10060.895772] ? set_kthread_struct+0x70/0x70 [10060.898496] ret_from_fork+0x1f/0x30 [10072.054043] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [10072.062436] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10072.093812] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10072.098404] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10072.101876] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10072.106026] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10072.108968] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10072.117129] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10072.119492] RAX: 0000000000007577 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10072.126748] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10072.131088] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10072.133904] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10072.151481] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10072.155928] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10072.160981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10072.165397] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10072.168783] Call Trace: [10072.172307] [10072.173371] ? show_regs.cold.9+0x22/0x2f [10072.175087] ? watchdog_timer_fn.cold.6+0x62/0x10a [10072.177496] ? __hrtimer_run_queues+0x1c2/0x4d0 [10072.180607] ? watchdog+0x40/0x40 [10072.181865] ? hrtimer_interrupt+0x128/0x2c0 [10072.184260] ? smp_apic_timer_interrupt+0xa6/0x270 [10072.186600] ? apic_timer_interrupt+0xf/0x20 [10072.189735] [10072.190221] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10072.192799] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10072.197976] ? newidle_balance.isra.93+0x207/0x580 [10072.200985] do_raw_spin_lock+0x177/0x190 [10072.203516] _raw_spin_lock+0x21/0x30 [10072.205278] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10072.208070] process_one_work+0x2c8/0x700 [10072.212434] worker_thread+0x296/0x6e0 [10072.215044] ? rescuer_thread+0x570/0x570 [10072.217552] kthread+0x1d1/0x200 [10072.219098] ? set_kthread_struct+0x70/0x70 [10072.222044] ret_from_fork+0x1f/0x30 [10074.619027] rcu: INFO: rcu_sched self-detected stall on CPU [10074.620961] rcu: 2-....: (4369799 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [10074.624324] (t=7980193 jiffies g=274185 q=65831) [10074.625556] rcu: rcu_sched kthread starved for 7438559 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [10074.628651] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10074.631093] rcu: RCU grace-period kthread stack dump: [10074.632552] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10074.635493] Call Trace: [10074.636413] __schedule+0x351/0xcb0 [10074.637447] schedule+0xc0/0x180 [10074.638540] schedule_timeout+0xb4/0x190 [10074.639784] ? do_raw_spin_unlock+0x75/0x190 [10074.641017] ? __next_timer_interrupt+0x160/0x160 [10074.642492] ? prepare_to_swait_event+0xf3/0x140 [10074.643778] rcu_gp_kthread+0x8dd/0x15b0 [10074.645472] ? force_qs_rnp+0x2c0/0x2c0 [10074.646629] kthread+0x1d1/0x200 [10074.647563] ? set_kthread_struct+0x70/0x70 [10074.648862] ret_from_fork+0x1f/0x30 [10074.649865] rcu: Stack dump where RCU GP kthread last ran: [10074.651450] Sending NMI from CPU 2 to CPUs 3: [10074.652663] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [10074.653630] Sending NMI from CPU 2 to CPUs 1: [10074.657902] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [10074.660981] NMI backtrace for cpu 2 [10074.661987] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10074.667246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10074.671865] Call Trace: [10074.673059] [10074.673574] ? dump_stack+0xbb/0x10e [10074.675627] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10074.677406] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10074.680167] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10074.682489] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10074.684870] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10074.686857] ? rcu_dump_cpu_stacks+0x11b/0x171 [10074.688393] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10074.690624] ? update_process_times+0x6d/0xc0 [10074.692382] ? tick_sched_handle.isra.18+0x27/0x90 [10074.694223] ? tick_sched_timer+0x53/0xb0 [10074.695839] ? __hrtimer_run_queues+0x1c2/0x4d0 [10074.697252] ? tick_sched_handle.isra.18+0x90/0x90 [10074.698678] ? hrtimer_interrupt+0x128/0x2c0 [10074.700262] ? smp_apic_timer_interrupt+0xa6/0x270 [10074.701648] ? apic_timer_interrupt+0xf/0x20 [10074.702919] [10074.703604] ? native_safe_halt+0x2e/0x50 [10074.704812] ? kvm_wait+0xcd/0xe0 [10074.705853] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10074.707860] ? do_raw_spin_lock+0x177/0x190 [10074.709225] ? _raw_spin_lock+0x21/0x30 [10074.710470] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10074.712463] ? obd_mod_free+0x46/0x140 [ptlrpc] [10074.714246] ? mdc_commit_open+0x91/0xb0 [mdc] [10074.715608] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10074.717517] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10074.719515] ? after_reply+0xe2c/0x1780 [ptlrpc] [10074.721235] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10074.723172] ? __next_timer_interrupt+0x160/0x160 [10074.724544] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10074.726213] ? do_wait_intr+0xf0/0xf0 [10074.727399] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10074.729237] ? kthread+0x1d1/0x200 [10074.730207] ? set_kthread_struct+0x70/0x70 [10074.731502] ? ret_from_fork+0x1f/0x30 [10076.127061] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7439s! [10076.129396] Showing busy workqueues and worker pools: [10076.131617] workqueue events: flags=0x0 [10076.133177] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10076.146184] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10076.149873] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10076.151939] pending: cpu_latency_work [ptlrpc] [10076.156703] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10076.158641] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10076.164345] workqueue events_long: flags=0x0 [10076.166584] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10076.168644] pending: lu_objects_flush [obdclass] [10076.172579] workqueue events_unbound: flags=0x2 [10076.174351] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10076.179716] pending: flush_memcg_stats_dwork, idle_cull_fn [10076.181487] workqueue events_power_efficient: flags=0x80 [10076.183630] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10076.186438] pending: check_lifetime, do_cache_clean [10076.188572] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10076.190644] pending: neigh_periodic_work, neigh_periodic_work [10076.192632] workqueue mm_percpu_wq: flags=0x8 [10076.196965] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10076.198507] pending: vmstat_update, lru_add_drain_per_cpu [10076.202465] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10076.205952] pending: vmstat_update, lru_add_drain_per_cpu [10076.208529] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10076.213544] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10076.216496] workqueue xprtiod: flags=0xa [10076.217665] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10076.219518] pending: xprt_autoclose, xs_error_handle [10076.221377] workqueue ipv6_addrconf: flags=0x40008 [10076.228154] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10076.238648] pending: addrconf_verify_work [10076.240617] workqueue knbd0-recv: flags=0x1a [10076.243584] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10076.248339] in-flight: 651:recv_work [10076.250964] workqueue ptlrpc_pinger: flags=0x2 [10076.253592] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10076.256778] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10076.262895] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7439s workers=4 idle: 15979 31824 8 [10076.265408] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10066s workers=2 idle: 654 [10100.055739] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [10100.058699] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10100.061074] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [10100.062419] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [10100.062428] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10100.062514] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10100.062521] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10100.063095] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10100.063110] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10100.063114] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10100.063119] RAX: 0000000000003b0c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10100.063121] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10100.063123] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10100.063125] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10100.063128] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10100.063131] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10100.063134] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10100.063136] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10100.063146] Call Trace: [10100.063151] [10100.063157] ? show_regs.cold.9+0x22/0x2f [10100.063166] ? watchdog_timer_fn.cold.6+0x62/0x10a [10100.063173] ? __hrtimer_run_queues+0x1c2/0x4d0 [10100.063184] ? watchdog+0x40/0x40 [10100.063189] ? hrtimer_interrupt+0x128/0x2c0 [10100.063193] ? smp_apic_timer_interrupt+0xa6/0x270 [10100.063199] ? apic_timer_interrupt+0xf/0x20 [10100.063204] [10100.063207] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10100.063213] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10100.063219] do_raw_spin_lock+0x177/0x190 [10100.063224] _raw_spin_lock+0x21/0x30 [10100.063229] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10100.063522] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10100.063761] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10100.064090] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10100.064316] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10100.064544] ? woken_wake_function+0x30/0x30 [10100.064552] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10100.065408] kthread+0x1d1/0x200 [10100.065696] ? set_kthread_struct+0x70/0x70 [10100.065700] ret_from_fork+0x1f/0x30 [10100.097487] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10100.097501] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10100.097504] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10100.097519] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10100.097523] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10100.097528] RAX: 0000000000005919 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10100.097531] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10100.097533] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10100.097535] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10100.097538] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10100.097541] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10100.097544] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10100.097546] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10100.097555] Call Trace: [10100.097561] [10100.097567] ? show_regs.cold.9+0x22/0x2f [10100.097577] ? watchdog_timer_fn.cold.6+0x62/0x10a [10100.097583] ? __hrtimer_run_queues+0x1c2/0x4d0 [10100.097587] ? watchdog+0x40/0x40 [10100.097591] ? hrtimer_interrupt+0x128/0x2c0 [10100.097595] ? smp_apic_timer_interrupt+0xa6/0x270 [10100.097603] ? apic_timer_interrupt+0xf/0x20 [10100.097607] [10100.097609] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10100.097617] do_raw_spin_lock+0x177/0x190 [10100.097621] _raw_spin_lock+0x21/0x30 [10100.097625] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10100.097903] ? do_raw_spin_unlock+0x75/0x190 [10100.097908] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10100.098109] ? do_wait_intr+0xf0/0xf0 [10100.098119] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10100.098323] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10100.110716] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10100.110796] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10100.110803] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10100.110807] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10100.110822] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10100.110826] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10100.110831] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10100.110833] RDX: 0000000000007f08 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10100.110836] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10100.110838] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10100.110840] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10100.110844] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10100.110846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10100.110849] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10100.110858] Call Trace: [10100.110865] [10100.110872] ? show_regs.cold.9+0x22/0x2f [10100.110948] ? watchdog_timer_fn.cold.6+0x62/0x10a [10100.110956] ? __hrtimer_run_queues+0x1c2/0x4d0 [10100.110960] ? watchdog+0x40/0x40 [10100.111023] ? hrtimer_interrupt+0x128/0x2c0 [10100.111027] ? smp_apic_timer_interrupt+0xa6/0x270 [10100.111036] ? apic_timer_interrupt+0xf/0x20 [10100.111041] [10100.111044] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10100.111049] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10100.111053] do_raw_spin_lock+0x177/0x190 [10100.111058] _raw_spin_lock+0x21/0x30 [10100.111063] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10100.112059] obd_mod_free+0x46/0x140 [ptlrpc] [10100.112277] mdc_commit_open+0x91/0xb0 [mdc] [10100.112318] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10100.122401] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10100.125138] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10100.125840] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10100.125891] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10100.125977] mdc_intent_lock+0x353/0x7e0 [mdc] [10100.126012] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10100.126113] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10100.126418] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10100.126447] lmv_intent_open+0x338/0x1150 [lmv] [10100.126482] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10100.126567] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10100.126648] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10100.126793] ll_intent_lock+0xc5/0xaa0 [lustre] [10100.126880] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10100.126964] ll_lookup_it+0xe0e/0x2270 [lustre] [10100.127049] ? do_raw_spin_unlock+0x75/0x190 [10100.127057] ? _raw_spin_unlock+0x12/0x30 [10100.127065] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10100.127094] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10100.127116] ll_atomic_open+0x36a/0xd70 [lustre] [10100.127418] lookup_open+0x356/0xb40 [10100.127433] do_last+0x313/0xfc0 [10100.127438] ? nd_jump_root+0xe5/0x160 [10100.127448] ? path_init+0x437/0x520 [10100.127454] path_openat+0xf7/0x500 [10100.127464] ? do_raw_spin_unlock+0x75/0x190 [10100.127469] do_filp_open+0x99/0x140 [10100.127480] ? getname_flags+0x6e/0x330 [10100.127484] ? __check_object_size+0xff/0x256 [10100.127492] ? do_raw_spin_unlock+0x75/0x190 [10100.127496] ? _raw_spin_unlock+0x12/0x30 [10100.127505] do_sys_openat2+0x2b4/0x410 [10100.127509] do_sys_open+0x73/0xa0 [10100.127517] __x64_sys_openat+0x24/0x30 [10100.127521] do_syscall_64+0xc1/0x440 [10100.127527] entry_SYSCALL_64_after_hwframe+0x49/0xae [10100.127895] RIP: 0033:0x7f53425a5332 [10100.127906] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10100.127910] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10100.127915] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10100.127917] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10100.127919] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10100.127922] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10100.127925] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10100.728434] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10100.731297] after_reply+0xe2c/0x1780 [ptlrpc] [10100.734596] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10100.741053] ? __next_timer_interrupt+0x160/0x160 [10100.743368] ptlrpcd+0xa82/0xe40 [ptlrpc] [10100.745134] ? do_wait_intr+0xf0/0xf0 [10100.747606] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10100.750898] kthread+0x1d1/0x200 [10100.753771] ? set_kthread_struct+0x70/0x70 [10100.757560] ret_from_fork+0x1f/0x30 [10104.055868] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [10104.059479] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10104.093939] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10104.101918] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10104.108492] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10104.114093] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10104.116756] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10104.130619] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10104.135237] RAX: 0000000000002518 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10104.140686] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10104.145365] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10104.151116] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10104.155530] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10104.162814] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10104.165458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10104.169950] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10104.173993] Call Trace: [10104.176007] [10104.177841] ? show_regs.cold.9+0x22/0x2f [10104.179263] ? watchdog_timer_fn.cold.6+0x62/0x10a [10104.180503] ? __hrtimer_run_queues+0x1c2/0x4d0 [10104.181965] ? watchdog+0x40/0x40 [10104.183104] ? hrtimer_interrupt+0x128/0x2c0 [10104.184350] ? smp_apic_timer_interrupt+0xa6/0x270 [10104.187513] ? apic_timer_interrupt+0xf/0x20 [10104.190267] [10104.191772] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10104.193449] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10104.197207] ? newidle_balance.isra.93+0x207/0x580 [10104.200867] do_raw_spin_lock+0x177/0x190 [10104.205172] _raw_spin_lock+0x21/0x30 [10104.206297] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10104.209290] process_one_work+0x2c8/0x700 [10104.212104] worker_thread+0x296/0x6e0 [10104.213516] ? rescuer_thread+0x570/0x570 [10104.216476] kthread+0x1d1/0x200 [10104.218946] ? set_kthread_struct+0x70/0x70 [10104.221350] ret_from_fork+0x1f/0x30 [10106.344623] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7469s! [10106.348244] Showing busy workqueues and worker pools: [10106.352858] workqueue events: flags=0x0 [10106.354175] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10106.360328] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10106.373888] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10106.381484] pending: cpu_latency_work [ptlrpc] [10106.390039] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10106.397482] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10106.408373] workqueue events_long: flags=0x0 [10106.413045] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10106.420062] pending: lu_objects_flush [obdclass] [10106.426105] workqueue events_unbound: flags=0x2 [10106.430438] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10106.436495] pending: flush_memcg_stats_dwork, idle_cull_fn [10106.443491] workqueue events_power_efficient: flags=0x80 [10106.448962] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10106.459136] pending: check_lifetime, do_cache_clean [10106.470469] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10106.478041] pending: neigh_periodic_work, neigh_periodic_work [10106.487917] workqueue mm_percpu_wq: flags=0x8 [10106.490159] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10106.503952] pending: vmstat_update, lru_add_drain_per_cpu [10106.508468] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10106.517872] pending: vmstat_update, lru_add_drain_per_cpu [10106.526908] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10106.537608] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10106.546789] workqueue xprtiod: flags=0xa [10106.553710] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10106.561986] pending: xprt_autoclose, xs_error_handle [10106.568033] workqueue ipv6_addrconf: flags=0x40008 [10106.572870] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10106.583880] pending: addrconf_verify_work [10106.588648] workqueue knbd0-recv: flags=0x1a [10106.593740] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10106.601742] in-flight: 651:recv_work [10106.607163] workqueue ptlrpc_pinger: flags=0x2 [10106.614082] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10106.623969] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10106.629824] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7469s workers=4 idle: 15979 31824 8 [10106.635413] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10097s workers=2 idle: 654 [10128.057316] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [10128.057926] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [10128.059301] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [10128.059312] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10128.059391] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10128.059397] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10128.059400] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10128.059414] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10128.059419] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [10128.059424] RAX: 0000000000003116 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10128.059427] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10128.059431] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10128.059433] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10128.059436] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10128.059440] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10128.059443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10128.059446] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10128.059456] Call Trace: [10128.059461] [10128.059467] ? show_regs.cold.9+0x22/0x2f [10128.059476] ? watchdog_timer_fn.cold.6+0x62/0x10a [10128.059483] ? __hrtimer_run_queues+0x1c2/0x4d0 [10128.059488] ? watchdog+0x40/0x40 [10128.059493] ? hrtimer_interrupt+0x128/0x2c0 [10128.059497] ? smp_apic_timer_interrupt+0xa6/0x270 [10128.059505] ? apic_timer_interrupt+0xf/0x20 [10128.059510] [10128.059513] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10128.059518] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10128.059525] do_raw_spin_lock+0x177/0x190 [10128.059529] _raw_spin_lock+0x21/0x30 [10128.059534] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10128.059872] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10128.060069] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10128.060591] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10128.060896] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10128.061326] ? woken_wake_function+0x30/0x30 [10128.061335] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10128.061536] kthread+0x1d1/0x200 [10128.061543] ? set_kthread_struct+0x70/0x70 [10128.061546] ret_from_fork+0x1f/0x30 [10128.066441] Modules linked in: [10128.070758] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10128.070841] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10128.070848] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10128.070851] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10128.070866] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10128.070870] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10128.070874] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10128.070876] RDX: 000000000000008c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10128.070878] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10128.070880] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10128.070882] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10128.070885] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10128.070888] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10128.070890] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10128.070900] Call Trace: [10128.070906] [10128.071242] ? show_regs.cold.9+0x22/0x2f [10128.071253] ? watchdog_timer_fn.cold.6+0x62/0x10a [10128.071259] ? __hrtimer_run_queues+0x1c2/0x4d0 [10128.071263] ? watchdog+0x40/0x40 [10128.071268] ? hrtimer_interrupt+0x128/0x2c0 [10128.071272] ? smp_apic_timer_interrupt+0xa6/0x270 [10128.071279] ? apic_timer_interrupt+0xf/0x20 [10128.071285] [10128.071287] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10128.077656] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver [10128.113839] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10128.113860] do_raw_spin_lock+0x177/0x190 [10128.113866] _raw_spin_lock+0x21/0x30 [10128.113875] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10128.114137] obd_mod_free+0x46/0x140 [ptlrpc] [10128.114355] mdc_commit_open+0x91/0xb0 [mdc] [10128.114404] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10128.114606] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10128.114809] after_reply+0xe2c/0x1780 [ptlrpc] [10128.115011] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10128.115654] ? __next_timer_interrupt+0x160/0x160 [10128.115664] ptlrpcd+0xa82/0xe40 [ptlrpc] [10128.116079] ? do_wait_intr+0xf0/0xf0 [10128.116089] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10128.116350] kthread+0x1d1/0x200 [10128.116357] ? set_kthread_struct+0x70/0x70 [10128.116361] ret_from_fork+0x1f/0x30 [10128.463830] intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10128.481759] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10128.490787] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10128.495751] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10128.497581] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10128.521758] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10128.537136] RAX: 0000000000007a87 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10128.545598] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10128.556855] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10128.570286] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10128.583075] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10128.594888] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10128.607293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10128.613011] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10128.625231] Call Trace: [10128.626736] [10128.628534] ? show_regs.cold.9+0x22/0x2f [10128.641506] ? watchdog_timer_fn.cold.6+0x62/0x10a [10128.651438] ? __hrtimer_run_queues+0x1c2/0x4d0 [10128.665643] ? watchdog+0x40/0x40 [10128.683686] ? hrtimer_interrupt+0x128/0x2c0 [10128.685173] ? smp_apic_timer_interrupt+0xa6/0x270 [10128.693313] ? apic_timer_interrupt+0xf/0x20 [10128.694743] [10128.695497] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10128.699649] do_raw_spin_lock+0x177/0x190 [10128.700891] _raw_spin_lock+0x21/0x30 [10128.703773] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10128.706499] ? do_raw_spin_unlock+0x75/0x190 [10128.707926] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10128.711386] ? do_wait_intr+0xf0/0xf0 [10128.712672] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10128.717843] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10128.720282] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10128.722834] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10128.725692] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10128.728056] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10128.730228] mdc_intent_lock+0x353/0x7e0 [mdc] [10128.732333] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10128.734697] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10128.736785] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10128.739177] lmv_intent_open+0x338/0x1150 [lmv] [10128.740867] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10128.743062] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10128.745273] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10128.746983] ll_intent_lock+0xc5/0xaa0 [lustre] [10128.748495] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10128.751270] ll_lookup_it+0xe0e/0x2270 [lustre] [10128.753004] ? do_raw_spin_unlock+0x75/0x190 [10128.754300] ? _raw_spin_unlock+0x12/0x30 [10128.755972] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10128.758224] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10128.760226] ll_atomic_open+0x36a/0xd70 [lustre] [10128.762345] lookup_open+0x356/0xb40 [10128.763544] do_last+0x313/0xfc0 [10128.765091] ? nd_jump_root+0xe5/0x160 [10128.766381] ? path_init+0x437/0x520 [10128.768285] path_openat+0xf7/0x500 [10128.769685] ? do_raw_spin_unlock+0x75/0x190 [10128.772266] do_filp_open+0x99/0x140 [10128.773618] ? getname_flags+0x6e/0x330 [10128.775470] ? __check_object_size+0xff/0x256 [10128.777333] ? do_raw_spin_unlock+0x75/0x190 [10128.779458] ? _raw_spin_unlock+0x12/0x30 [10128.781485] do_sys_openat2+0x2b4/0x410 [10128.783049] do_sys_open+0x73/0xa0 [10128.784359] __x64_sys_openat+0x24/0x30 [10128.786126] do_syscall_64+0xc1/0x440 [10128.787580] entry_SYSCALL_64_after_hwframe+0x49/0xae [10128.789645] RIP: 0033:0x7f53425a5332 [10128.790683] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10128.796986] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10128.799698] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10128.802821] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10128.805229] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10128.807709] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10128.810935] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10132.052021] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [10132.056578] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10132.077903] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10132.085672] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10132.096843] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10132.099421] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10132.101603] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10132.112128] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10132.116096] RAX: 000000000000671b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10132.121747] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10132.125729] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10132.128790] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10132.131951] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10132.135068] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10132.139092] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10132.141631] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10132.145658] Call Trace: [10132.146803] [10132.147881] ? show_regs.cold.9+0x22/0x2f [10132.149572] ? watchdog_timer_fn.cold.6+0x62/0x10a [10132.151528] ? __hrtimer_run_queues+0x1c2/0x4d0 [10132.153159] ? watchdog+0x40/0x40 [10132.154693] ? hrtimer_interrupt+0x128/0x2c0 [10132.156771] ? smp_apic_timer_interrupt+0xa6/0x270 [10132.158973] ? apic_timer_interrupt+0xf/0x20 [10132.161070] [10132.161962] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10132.164466] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10132.166851] ? newidle_balance.isra.93+0x207/0x580 [10132.169358] do_raw_spin_lock+0x177/0x190 [10132.171465] _raw_spin_lock+0x21/0x30 [10132.173093] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10132.175497] process_one_work+0x2c8/0x700 [10132.176848] worker_thread+0x296/0x6e0 [10132.178893] ? rescuer_thread+0x570/0x570 [10132.180925] kthread+0x1d1/0x200 [10132.182722] ? set_kthread_struct+0x70/0x70 [10132.184615] ret_from_fork+0x1f/0x30 [10137.056050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7499s! [10137.059230] Showing busy workqueues and worker pools: [10137.061080] workqueue events: flags=0x0 [10137.062480] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10137.065162] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10137.073118] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10137.075780] pending: cpu_latency_work [ptlrpc] [10137.078035] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10137.080844] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10137.084912] workqueue events_long: flags=0x0 [10137.086757] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10137.090360] pending: lu_objects_flush [obdclass] [10137.094455] workqueue events_unbound: flags=0x2 [10137.096428] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10137.099084] pending: flush_memcg_stats_dwork, idle_cull_fn [10137.103374] workqueue events_power_efficient: flags=0x80 [10137.105401] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10137.108358] pending: check_lifetime, do_cache_clean [10137.110543] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10137.113390] pending: neigh_periodic_work, neigh_periodic_work [10137.116400] workqueue mm_percpu_wq: flags=0x8 [10137.118304] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10137.122234] pending: vmstat_update, lru_add_drain_per_cpu [10137.126384] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10137.132805] pending: vmstat_update, lru_add_drain_per_cpu [10137.136012] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10137.139515] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10137.143754] workqueue xprtiod: flags=0xa [10137.145065] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10137.147664] pending: xprt_autoclose, xs_error_handle [10137.149898] workqueue ipv6_addrconf: flags=0x40008 [10137.152602] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10137.155589] pending: addrconf_verify_work [10137.157681] workqueue knbd0-recv: flags=0x1a [10137.159711] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10137.162432] in-flight: 651:recv_work [10137.164088] workqueue ptlrpc_pinger: flags=0x2 [10137.165791] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10137.168056] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10137.169983] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7500s workers=4 idle: 15979 31824 8 [10137.172421] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10127s workers=2 idle: 654 [10156.055015] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [10156.058023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [10156.058524] Modules linked in: [10156.061011] Modules linked in: [10156.061016] lustre(O) [10156.062077] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10156.062169] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10156.063396] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10156.063463] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10156.063469] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10156.063471] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10156.063512] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10156.063518] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10156.063522] RAX: 0000000000002723 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10156.063525] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10156.063527] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10156.063530] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10156.063532] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10156.063535] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10156.063539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10156.063541] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10156.063549] Call Trace: [10156.063566] [10156.063572] ? show_regs.cold.9+0x22/0x2f [10156.063581] ? watchdog_timer_fn.cold.6+0x62/0x10a [10156.063589] ? __hrtimer_run_queues+0x1c2/0x4d0 [10156.063593] ? watchdog+0x40/0x40 [10156.063598] ? hrtimer_interrupt+0x128/0x2c0 [10156.063602] ? smp_apic_timer_interrupt+0xa6/0x270 [10156.063610] ? apic_timer_interrupt+0xf/0x20 [10156.063615] [10156.063618] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10156.063625] do_raw_spin_lock+0x177/0x190 [10156.063630] _raw_spin_lock+0x21/0x30 [10156.063315] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [10156.063696] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10156.063753] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10156.063766] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10156.063769] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10156.063776] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10156.063780] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10156.063785] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10156.063787] RDX: 000000000000423f RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10156.063790] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10156.063793] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10156.063795] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10156.063798] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10156.063801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10156.063804] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10156.063810] Call Trace: [10156.063813] [10156.063816] ? show_regs.cold.9+0x22/0x2f [10156.063823] ? watchdog_timer_fn.cold.6+0x62/0x10a [10156.063828] ? __hrtimer_run_queues+0x1c2/0x4d0 [10156.063832] ? watchdog+0x40/0x40 [10156.063836] ? hrtimer_interrupt+0x128/0x2c0 [10156.063841] ? smp_apic_timer_interrupt+0xa6/0x270 [10156.063847] ? apic_timer_interrupt+0xf/0x20 [10156.063852] [10156.063855] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10156.063860] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10156.063865] do_raw_spin_lock+0x177/0x190 [10156.063870] _raw_spin_lock+0x21/0x30 [10156.063635] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10156.064422] ? do_raw_spin_unlock+0x75/0x190 [10156.063874] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10156.064430] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10156.065136] ? do_wait_intr+0xf0/0xf0 [10156.065147] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10156.065352] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10156.065648] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10156.065654] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10156.065663] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10156.065667] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10156.065673] RAX: 00000000000074a3 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10156.065675] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10156.065677] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10156.065680] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10156.065682] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10156.065696] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10156.065699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10156.065701] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10156.065708] Call Trace: [10156.065712] [10156.065715] ? show_regs.cold.9+0x22/0x2f [10156.065724] ? watchdog_timer_fn.cold.6+0x62/0x10a [10156.065730] ? __hrtimer_run_queues+0x1c2/0x4d0 [10156.065734] ? watchdog+0x40/0x40 [10156.065744] ? hrtimer_interrupt+0x128/0x2c0 [10156.065749] ? smp_apic_timer_interrupt+0xa6/0x270 [10156.065756] ? apic_timer_interrupt+0xf/0x20 [10156.065761] [10156.065764] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10156.065770] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10156.065775] do_raw_spin_lock+0x177/0x190 [10156.065780] _raw_spin_lock+0x21/0x30 [10156.065785] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10156.065582] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10156.066529] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10156.069232] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10156.069816] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10156.070146] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10156.070481] ? woken_wake_function+0x30/0x30 [10156.070492] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10156.070740] kthread+0x1d1/0x200 [10156.070746] ? set_kthread_struct+0x70/0x70 [10156.070750] ret_from_fork+0x1f/0x30 [10156.101195] obd_mod_free+0x46/0x140 [ptlrpc] [10156.105838] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10156.105975] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10156.106015] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10156.106049] mdc_intent_lock+0x353/0x7e0 [mdc] [10156.106078] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10156.106340] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10156.106753] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10156.106787] lmv_intent_open+0x338/0x1150 [lmv] [10156.106820] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10156.106925] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10156.143598] mdc_commit_open+0x91/0xb0 [mdc] [10156.148118] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10156.148461] ll_intent_lock+0xc5/0xaa0 [lustre] [10156.148587] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10156.148857] ll_lookup_it+0xe0e/0x2270 [lustre] [10156.148962] ? do_raw_spin_unlock+0x75/0x190 [10156.148976] ? _raw_spin_unlock+0x12/0x30 [10156.148984] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10156.153898] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10156.156428] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10156.156569] ll_atomic_open+0x36a/0xd70 [lustre] [10156.156712] lookup_open+0x356/0xb40 [10156.156723] do_last+0x313/0xfc0 [10156.156729] ? nd_jump_root+0xe5/0x160 [10156.156734] ? path_init+0x437/0x520 [10156.156740] path_openat+0xf7/0x500 [10156.156745] ? do_raw_spin_unlock+0x75/0x190 [10156.156753] do_filp_open+0x99/0x140 [10156.156760] ? getname_flags+0x6e/0x330 [10156.156765] ? __check_object_size+0xff/0x256 [10156.156772] ? do_raw_spin_unlock+0x75/0x190 [10156.156777] ? _raw_spin_unlock+0x12/0x30 [10156.156784] do_sys_openat2+0x2b4/0x410 [10156.156789] do_sys_open+0x73/0xa0 [10156.156793] __x64_sys_openat+0x24/0x30 [10156.156797] do_syscall_64+0xc1/0x440 [10156.156804] entry_SYSCALL_64_after_hwframe+0x49/0xae [10156.156810] RIP: 0033:0x7f53425a5332 [10156.156898] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10156.156902] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10156.156907] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10156.156909] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10156.156912] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10156.156914] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10156.156916] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10156.975979] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10156.983634] after_reply+0xe2c/0x1780 [ptlrpc] [10156.987302] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10156.990059] ? __next_timer_interrupt+0x160/0x160 [10156.996012] ptlrpcd+0xa82/0xe40 [ptlrpc] [10157.000475] ? do_wait_intr+0xf0/0xf0 [10157.002258] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10157.006797] kthread+0x1d1/0x200 [10157.008913] ? set_kthread_struct+0x70/0x70 [10157.010734] ret_from_fork+0x1f/0x30 [10164.053925] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [10164.058362] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10164.082331] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10164.091796] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10164.095314] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10164.099279] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10164.104791] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10164.117485] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10164.122017] RAX: 0000000000007f85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10164.130526] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10164.137036] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10164.140802] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10164.147045] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10164.153301] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10164.160294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10164.166979] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10164.174070] Call Trace: [10164.175974] [10164.177999] ? show_regs.cold.9+0x22/0x2f [10164.180517] ? watchdog_timer_fn.cold.6+0x62/0x10a [10164.184950] ? __hrtimer_run_queues+0x1c2/0x4d0 [10164.187209] ? watchdog+0x40/0x40 [10164.191399] ? hrtimer_interrupt+0x128/0x2c0 [10164.195917] ? smp_apic_timer_interrupt+0xa6/0x270 [10164.199387] ? apic_timer_interrupt+0xf/0x20 [10164.204452] [10164.207745] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10164.210348] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10164.216321] ? newidle_balance.isra.93+0x207/0x580 [10164.221790] do_raw_spin_lock+0x177/0x190 [10164.225159] _raw_spin_lock+0x21/0x30 [10164.229001] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10164.231606] process_one_work+0x2c8/0x700 [10164.234886] worker_thread+0x296/0x6e0 [10164.236345] ? rescuer_thread+0x570/0x570 [10164.241498] kthread+0x1d1/0x200 [10164.243292] ? set_kthread_struct+0x70/0x70 [10164.244597] ret_from_fork+0x1f/0x30 [10167.265187] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7530s! [10167.272179] Showing busy workqueues and worker pools: [10167.276626] workqueue events: flags=0x0 [10167.280682] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10167.286810] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10167.296888] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10167.306766] pending: cpu_latency_work [ptlrpc] [10167.315745] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10167.324643] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10167.334336] workqueue events_long: flags=0x0 [10167.337653] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10167.340463] pending: lu_objects_flush [obdclass] [10167.345716] workqueue events_unbound: flags=0x2 [10167.350181] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10167.356067] pending: flush_memcg_stats_dwork, idle_cull_fn [10167.359793] workqueue events_power_efficient: flags=0x80 [10167.364766] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10167.371215] pending: check_lifetime, do_cache_clean [10167.375775] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10167.382721] pending: neigh_periodic_work, neigh_periodic_work [10167.386773] workqueue mm_percpu_wq: flags=0x8 [10167.390155] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10167.396424] pending: vmstat_update, lru_add_drain_per_cpu [10167.401399] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10167.404728] pending: vmstat_update, lru_add_drain_per_cpu [10167.410195] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10167.417655] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10167.419656] workqueue xprtiod: flags=0xa [10167.425601] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10167.431060] pending: xprt_autoclose, xs_error_handle [10167.436018] workqueue ipv6_addrconf: flags=0x40008 [10167.439920] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10167.446657] pending: addrconf_verify_work [10167.450049] workqueue knbd0-recv: flags=0x1a [10167.455353] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10167.460739] in-flight: 651:recv_work [10167.464343] workqueue ptlrpc_pinger: flags=0x2 [10167.466469] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10167.469405] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10167.476594] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7530s workers=4 idle: 15979 31824 8 [10167.485407] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10157s workers=2 idle: 654 [10184.056168] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [10184.057028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [10184.058013] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [10184.058020] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10184.058224] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10184.058230] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10184.058232] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10184.058246] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10184.058250] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10184.058254] RAX: 0000000000004e75 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10184.058257] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10184.058259] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10184.058262] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10184.058264] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10184.058268] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10184.058271] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10184.058273] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10184.058280] Call Trace: [10184.058285] [10184.058290] ? show_regs.cold.9+0x22/0x2f [10184.058299] ? watchdog_timer_fn.cold.6+0x62/0x10a [10184.058305] ? __hrtimer_run_queues+0x1c2/0x4d0 [10184.058309] ? watchdog+0x40/0x40 [10184.058313] ? hrtimer_interrupt+0x128/0x2c0 [10184.058318] ? smp_apic_timer_interrupt+0xa6/0x270 [10184.058325] ? apic_timer_interrupt+0xf/0x20 [10184.058329] [10184.058332] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10184.058337] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10184.058342] do_raw_spin_lock+0x177/0x190 [10184.058347] _raw_spin_lock+0x21/0x30 [10184.058351] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10184.058615] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10184.058676] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10184.058680] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10184.058682] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10184.058689] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10184.058692] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10184.058696] RAX: 0000000000007f07 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10184.058698] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10184.058700] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10184.058702] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10184.058704] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10184.058706] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10184.058709] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10184.058711] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10184.058717] Call Trace: [10184.058719] [10184.058722] ? show_regs.cold.9+0x22/0x2f [10184.058727] ? watchdog_timer_fn.cold.6+0x62/0x10a [10184.058732] ? __hrtimer_run_queues+0x1c2/0x4d0 [10184.058735] ? watchdog+0x40/0x40 [10184.058739] ? hrtimer_interrupt+0x128/0x2c0 [10184.058742] ? smp_apic_timer_interrupt+0xa6/0x270 [10184.058747] ? apic_timer_interrupt+0xf/0x20 [10184.058751] [10184.058754] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10184.058759] do_raw_spin_lock+0x177/0x190 [10184.058763] _raw_spin_lock+0x21/0x30 [10184.058669] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10184.058766] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10184.059277] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10184.059518] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10184.059737] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10184.059987] ? woken_wake_function+0x30/0x30 [10184.059996] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10184.062226] ? do_raw_spin_unlock+0x75/0x190 [10184.062235] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10184.062458] ? do_wait_intr+0xf0/0xf0 [10184.062467] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10184.063141] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10184.063229] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10184.063235] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10184.063239] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10184.063249] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10184.063254] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10184.063259] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10184.063261] RDX: 00000000000038fc RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10184.063264] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10184.063266] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10184.063269] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10184.063272] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10184.063276] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10184.063278] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10184.063286] Call Trace: [10184.063290] [10184.063293] ? show_regs.cold.9+0x22/0x2f [10184.063303] ? watchdog_timer_fn.cold.6+0x62/0x10a [10184.063310] ? __hrtimer_run_queues+0x1c2/0x4d0 [10184.063315] ? watchdog+0x40/0x40 [10184.063320] ? hrtimer_interrupt+0x128/0x2c0 [10184.063324] ? smp_apic_timer_interrupt+0xa6/0x270 [10184.063333] ? apic_timer_interrupt+0xf/0x20 [10184.063337] [10184.063341] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10184.063346] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10184.063351] do_raw_spin_lock+0x177/0x190 [10184.063355] _raw_spin_lock+0x21/0x30 [10184.063361] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10184.063567] obd_mod_free+0x46/0x140 [ptlrpc] [10184.063766] mdc_commit_open+0x91/0xb0 [mdc] [10184.063805] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10184.064003] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10184.064367] after_reply+0xe2c/0x1780 [ptlrpc] [10184.064566] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10184.064765] ? __next_timer_interrupt+0x160/0x160 [10184.062954] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10184.064776] ptlrpcd+0xa82/0xe40 [ptlrpc] [10184.066021] ? do_wait_intr+0xf0/0xf0 [10184.066033] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10184.067423] kthread+0x1d1/0x200 [10184.067432] ? set_kthread_struct+0x70/0x70 [10184.067437] ret_from_fork+0x1f/0x30 [10184.069899] kthread+0x1d1/0x200 [10184.069920] ? set_kthread_struct+0x70/0x70 [10184.069925] ret_from_fork+0x1f/0x30 [10184.572476] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10184.582398] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10184.586302] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10184.591522] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10184.596635] mdc_intent_lock+0x353/0x7e0 [mdc] [10184.598852] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10184.601080] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10184.603012] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10184.605829] lmv_intent_open+0x338/0x1150 [lmv] [10184.607750] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10184.609900] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10184.612707] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10184.614762] ll_intent_lock+0xc5/0xaa0 [lustre] [10184.617399] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10184.619796] ll_lookup_it+0xe0e/0x2270 [lustre] [10184.621937] ? do_raw_spin_unlock+0x75/0x190 [10184.623807] ? _raw_spin_unlock+0x12/0x30 [10184.625533] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10184.627789] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10184.630074] ll_atomic_open+0x36a/0xd70 [lustre] [10184.631662] lookup_open+0x356/0xb40 [10184.632844] do_last+0x313/0xfc0 [10184.633785] ? nd_jump_root+0xe5/0x160 [10184.634859] ? path_init+0x437/0x520 [10184.635866] path_openat+0xf7/0x500 [10184.636808] ? do_raw_spin_unlock+0x75/0x190 [10184.638069] do_filp_open+0x99/0x140 [10184.639507] ? getname_flags+0x6e/0x330 [10184.640590] ? __check_object_size+0xff/0x256 [10184.641805] ? do_raw_spin_unlock+0x75/0x190 [10184.642957] ? _raw_spin_unlock+0x12/0x30 [10184.644070] do_sys_openat2+0x2b4/0x410 [10184.645156] do_sys_open+0x73/0xa0 [10184.646055] __x64_sys_openat+0x24/0x30 [10184.647401] do_syscall_64+0xc1/0x440 [10184.648419] entry_SYSCALL_64_after_hwframe+0x49/0xae [10184.649976] RIP: 0033:0x7f53425a5332 [10184.650990] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10184.657094] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10184.659665] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10184.661703] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10184.663771] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10184.666542] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10184.669246] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10192.053739] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [10192.069183] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10192.120350] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10192.131326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10192.141426] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10192.151806] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10192.159020] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10192.188601] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10192.198187] RAX: 0000000000006c0d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10192.208020] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10192.215830] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10192.219982] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10192.238469] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10192.242148] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10192.246473] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10192.249296] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10192.256184] Call Trace: [10192.258469] [10192.260557] ? show_regs.cold.9+0x22/0x2f [10192.265495] ? watchdog_timer_fn.cold.6+0x62/0x10a [10192.270740] ? __hrtimer_run_queues+0x1c2/0x4d0 [10192.276539] ? watchdog+0x40/0x40 [10192.282798] ? hrtimer_interrupt+0x128/0x2c0 [10192.288760] ? smp_apic_timer_interrupt+0xa6/0x270 [10192.292329] ? apic_timer_interrupt+0xf/0x20 [10192.297959] [10192.299776] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10192.304415] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10192.312329] ? newidle_balance.isra.93+0x207/0x580 [10192.314493] do_raw_spin_lock+0x177/0x190 [10192.319501] _raw_spin_lock+0x21/0x30 [10192.326993] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10192.328934] process_one_work+0x2c8/0x700 [10192.336263] worker_thread+0x296/0x6e0 [10192.338728] ? rescuer_thread+0x570/0x570 [10192.342572] kthread+0x1d1/0x200 [10192.346941] ? set_kthread_struct+0x70/0x70 [10192.348813] ret_from_fork+0x1f/0x30 [10197.985195] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7560s! [10197.991647] Showing busy workqueues and worker pools: [10197.994447] workqueue events: flags=0x0 [10197.996459] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10197.999844] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10198.010311] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10198.015723] pending: cpu_latency_work [ptlrpc] [10198.018048] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10198.026633] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10198.030548] workqueue events_long: flags=0x0 [10198.031646] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10198.036943] pending: lu_objects_flush [obdclass] [10198.041056] workqueue events_unbound: flags=0x2 [10198.042633] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10198.044948] pending: flush_memcg_stats_dwork, idle_cull_fn [10198.047135] workqueue events_power_efficient: flags=0x80 [10198.048849] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10198.051445] pending: check_lifetime, do_cache_clean [10198.053367] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10198.055470] pending: neigh_periodic_work, neigh_periodic_work [10198.057341] workqueue mm_percpu_wq: flags=0x8 [10198.058268] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10198.061211] pending: vmstat_update, lru_add_drain_per_cpu [10198.065877] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10198.069508] pending: vmstat_update, lru_add_drain_per_cpu [10198.071200] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10198.073754] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10198.076087] workqueue xprtiod: flags=0xa [10198.077447] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10198.079676] pending: xprt_autoclose, xs_error_handle [10198.085279] workqueue ipv6_addrconf: flags=0x40008 [10198.092472] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10198.095742] pending: addrconf_verify_work [10198.103343] workqueue knbd0-recv: flags=0x1a [10198.104876] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10198.119364] in-flight: 651:recv_work [10198.124359] workqueue ptlrpc_pinger: flags=0x2 [10198.131259] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10198.143418] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10198.147621] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7561s workers=4 idle: 15979 31824 8 [10198.151547] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10188s workers=2 idle: 654 [10212.055678] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [10212.058019] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [10212.060572] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [10212.063826] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [10212.063833] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10212.063995] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10212.064001] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10212.064003] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10212.064028] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10212.064032] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10212.064037] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10212.064039] RDX: 0000000000005fad RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10212.064041] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10212.064043] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10212.064045] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10212.064048] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10212.064050] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10212.064052] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10212.064061] Call Trace: [10212.064076] [10212.064081] ? show_regs.cold.9+0x22/0x2f [10212.064091] ? watchdog_timer_fn.cold.6+0x62/0x10a [10212.064096] ? __hrtimer_run_queues+0x1c2/0x4d0 [10212.064100] ? watchdog+0x40/0x40 [10212.064105] ? hrtimer_interrupt+0x128/0x2c0 [10212.064109] ? smp_apic_timer_interrupt+0xa6/0x270 [10212.064117] ? apic_timer_interrupt+0xf/0x20 [10212.064122] [10212.064125] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10212.064129] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10212.064133] do_raw_spin_lock+0x177/0x190 [10212.064137] _raw_spin_lock+0x21/0x30 [10212.064143] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10212.064644] obd_mod_free+0x46/0x140 [ptlrpc] [10212.064907] mdc_commit_open+0x91/0xb0 [mdc] [10212.064948] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10212.065577] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10212.066082] after_reply+0xe2c/0x1780 [ptlrpc] [10212.068395] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10212.069837] ? __next_timer_interrupt+0x160/0x160 [10212.069849] ptlrpcd+0xa82/0xe40 [ptlrpc] [10212.070921] ? do_wait_intr+0xf0/0xf0 [10212.071087] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10212.071468] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10212.071475] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10212.071479] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10212.071490] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10212.071495] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10212.071501] RAX: 0000000000000014 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10212.071504] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10212.071506] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10212.071509] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10212.071513] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10212.071517] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10212.070934] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10212.071522] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10212.071525] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10212.071643] kthread+0x1d1/0x200 [10212.071651] ? set_kthread_struct+0x70/0x70 [10212.071655] ret_from_fork+0x1f/0x30 [10212.071975] Call Trace: [10212.071982] [10212.071985] ? show_regs.cold.9+0x22/0x2f [10212.071996] ? watchdog_timer_fn.cold.6+0x62/0x10a [10212.072002] ? __hrtimer_run_queues+0x1c2/0x4d0 [10212.072007] ? watchdog+0x40/0x40 [10212.072012] ? hrtimer_interrupt+0x128/0x2c0 [10212.072016] ? smp_apic_timer_interrupt+0xa6/0x270 [10212.072022] ? apic_timer_interrupt+0xf/0x20 [10212.072028] [10212.072031] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10212.072037] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10212.072042] do_raw_spin_lock+0x177/0x190 [10212.072048] _raw_spin_lock+0x21/0x30 [10212.072053] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10212.077166] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10212.077233] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10212.077245] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10212.077248] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10212.077262] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10212.077267] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10212.077271] RAX: 000000000000550d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10212.077274] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10212.077276] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10212.077279] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10212.077286] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10212.077289] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10212.077293] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10212.077295] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10212.077303] Call Trace: [10212.077308] [10212.077314] ? show_regs.cold.9+0x22/0x2f [10212.077324] ? watchdog_timer_fn.cold.6+0x62/0x10a [10212.077330] ? __hrtimer_run_queues+0x1c2/0x4d0 [10212.077334] ? watchdog+0x40/0x40 [10212.077339] ? hrtimer_interrupt+0x128/0x2c0 [10212.077342] ? smp_apic_timer_interrupt+0xa6/0x270 [10212.077350] ? apic_timer_interrupt+0xf/0x20 [10212.077355] [10212.077359] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10212.077366] do_raw_spin_lock+0x177/0x190 [10212.077371] _raw_spin_lock+0x21/0x30 [10212.077376] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10212.077798] ? do_raw_spin_unlock+0x75/0x190 [10212.077805] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10212.078176] ? do_wait_intr+0xf0/0xf0 [10212.078186] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10212.078396] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10212.079148] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10212.079441] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10212.079631] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10212.079697] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10212.079745] mdc_intent_lock+0x353/0x7e0 [mdc] [10212.079779] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10212.079985] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10212.080212] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10212.080244] lmv_intent_open+0x338/0x1150 [lmv] [10212.080277] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10212.080367] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10212.080456] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10212.080481] ll_intent_lock+0xc5/0xaa0 [lustre] [10212.082197] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10212.082300] ll_lookup_it+0xe0e/0x2270 [lustre] [10212.096257] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10212.096553] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10212.096777] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10212.130465] ? do_raw_spin_unlock+0x75/0x190 [10212.130486] ? _raw_spin_unlock+0x12/0x30 [10212.130494] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10212.130521] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10212.130541] ll_atomic_open+0x36a/0xd70 [lustre] [10212.130663] lookup_open+0x356/0xb40 [10212.130673] do_last+0x313/0xfc0 [10212.130679] ? nd_jump_root+0xe5/0x160 [10212.130684] ? path_init+0x437/0x520 [10212.130689] path_openat+0xf7/0x500 [10212.130694] ? do_raw_spin_unlock+0x75/0x190 [10212.130699] do_filp_open+0x99/0x140 [10212.130717] ? getname_flags+0x6e/0x330 [10212.130721] ? __check_object_size+0xff/0x256 [10212.130728] ? do_raw_spin_unlock+0x75/0x190 [10212.130733] ? _raw_spin_unlock+0x12/0x30 [10212.130737] do_sys_openat2+0x2b4/0x410 [10212.130742] do_sys_open+0x73/0xa0 [10212.130747] __x64_sys_openat+0x24/0x30 [10212.130751] do_syscall_64+0xc1/0x440 [10212.130757] entry_SYSCALL_64_after_hwframe+0x49/0xae [10212.130763] RIP: 0033:0x7f53425a5332 [10212.130813] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10212.130819] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10212.130824] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10212.130827] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10212.130829] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10212.130832] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10212.130834] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10212.765305] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10212.769915] ? woken_wake_function+0x30/0x30 [10212.772093] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10212.776473] kthread+0x1d1/0x200 [10212.778371] ? set_kthread_struct+0x70/0x70 [10212.782460] ret_from_fork+0x1f/0x30 [10224.055582] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [10224.066304] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10224.139349] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10224.155326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10224.176399] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10224.188220] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10224.203141] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10224.225437] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10224.235380] RAX: 0000000000006004 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10224.247299] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10224.260801] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10224.268381] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10224.274483] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10224.287160] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10224.298855] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10224.305844] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10224.321823] Call Trace: [10224.327212] [10224.331719] ? show_regs.cold.9+0x22/0x2f [10224.333396] ? watchdog_timer_fn.cold.6+0x62/0x10a [10224.336207] ? __hrtimer_run_queues+0x1c2/0x4d0 [10224.339385] ? watchdog+0x40/0x40 [10224.345428] ? hrtimer_interrupt+0x128/0x2c0 [10224.353571] ? smp_apic_timer_interrupt+0xa6/0x270 [10224.355967] ? apic_timer_interrupt+0xf/0x20 [10224.357064] [10224.357821] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10224.362311] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10224.366427] ? newidle_balance.isra.93+0x207/0x580 [10224.374786] do_raw_spin_lock+0x177/0x190 [10224.375739] _raw_spin_lock+0x21/0x30 [10224.376628] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10224.380826] process_one_work+0x2c8/0x700 [10224.383336] worker_thread+0x296/0x6e0 [10224.387163] ? rescuer_thread+0x570/0x570 [10224.389412] kthread+0x1d1/0x200 [10224.390668] ? set_kthread_struct+0x70/0x70 [10224.392799] ret_from_fork+0x1f/0x30 [10228.192807] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7591s! [10228.204766] Showing busy workqueues and worker pools: [10228.206354] workqueue events: flags=0x0 [10228.209386] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10228.218053] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10228.227072] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10228.230003] pending: cpu_latency_work [ptlrpc] [10228.235281] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10228.239213] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10228.244811] workqueue events_long: flags=0x0 [10228.247314] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10228.249361] pending: lu_objects_flush [obdclass] [10228.251152] workqueue events_unbound: flags=0x2 [10228.253824] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10228.256228] pending: flush_memcg_stats_dwork, idle_cull_fn [10228.259380] workqueue events_power_efficient: flags=0x80 [10228.262480] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10228.265308] pending: check_lifetime, do_cache_clean [10228.267580] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10228.271122] pending: neigh_periodic_work, neigh_periodic_work [10228.273701] workqueue mm_percpu_wq: flags=0x8 [10228.277249] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10228.284601] pending: vmstat_update, lru_add_drain_per_cpu [10228.288306] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10228.298101] pending: vmstat_update, lru_add_drain_per_cpu [10228.306884] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10228.312689] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10228.317138] workqueue xprtiod: flags=0xa [10228.319211] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10228.322367] pending: xprt_autoclose, xs_error_handle [10228.325576] workqueue ipv6_addrconf: flags=0x40008 [10228.328080] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10228.330890] pending: addrconf_verify_work [10228.334527] workqueue knbd0-recv: flags=0x1a [10228.336698] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10228.339755] in-flight: 651:recv_work [10228.342745] workqueue ptlrpc_pinger: flags=0x2 [10228.345370] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10228.348580] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10228.353164] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7591s workers=4 idle: 15979 31824 8 [10228.356518] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10218s workers=2 idle: 654 [10240.056464] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [10240.059671] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [10240.063563] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [10240.063577] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10240.064081] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10240.064087] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10240.064091] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10240.064105] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10240.064109] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10240.064114] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10240.064116] RDX: 0000000000003d9d RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10240.064119] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10240.064121] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10240.064123] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10240.064126] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10240.064129] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10240.064131] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10240.064141] Call Trace: [10240.064147] [10240.064152] ? show_regs.cold.9+0x22/0x2f [10240.064162] ? watchdog_timer_fn.cold.6+0x62/0x10a [10240.064169] ? __hrtimer_run_queues+0x1c2/0x4d0 [10240.064174] ? watchdog+0x40/0x40 [10240.064178] ? hrtimer_interrupt+0x128/0x2c0 [10240.064182] ? smp_apic_timer_interrupt+0xa6/0x270 [10240.064190] ? apic_timer_interrupt+0xf/0x20 [10240.064195] [10240.064198] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10240.064203] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10240.064208] do_raw_spin_lock+0x177/0x190 [10240.064223] _raw_spin_lock+0x21/0x30 [10240.064232] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10240.065409] obd_mod_free+0x46/0x140 [ptlrpc] [10240.065655] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10240.065737] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10240.065742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10240.065745] RIP: 0010:native_safe_halt+0x2e/0x50 [10240.065754] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10240.065758] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10240.065763] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10240.065766] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10240.065768] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10240.065770] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10240.065772] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10240.065775] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10240.065778] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10240.065780] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10240.065789] Call Trace: [10240.065793] [10240.065796] ? show_regs.cold.9+0x22/0x2f [10240.065806] ? watchdog_timer_fn.cold.6+0x62/0x10a [10240.065812] ? __hrtimer_run_queues+0x1c2/0x4d0 [10240.065816] ? watchdog+0x40/0x40 [10240.065820] ? hrtimer_interrupt+0x128/0x2c0 [10240.065825] ? smp_apic_timer_interrupt+0xa6/0x270 [10240.065831] ? apic_timer_interrupt+0xf/0x20 [10240.065835] [10240.065838] ? native_safe_halt+0x2e/0x50 [10240.065842] kvm_wait+0xcd/0xe0 [10240.065850] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10240.065857] do_raw_spin_lock+0x177/0x190 [10240.065862] _raw_spin_lock+0x21/0x30 [10240.065866] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10240.066068] ? do_raw_spin_unlock+0x75/0x190 [10240.067079] mdc_commit_open+0x91/0xb0 [mdc] [10240.067118] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10240.066074] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10240.067435] ? do_wait_intr+0xf0/0xf0 [10240.067444] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10240.068024] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10240.067919] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10240.069277] after_reply+0xe2c/0x1780 [ptlrpc] [10240.069603] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10240.069799] ? __next_timer_interrupt+0x160/0x160 [10240.069809] ptlrpcd+0xa82/0xe40 [ptlrpc] [10240.071492] ? do_wait_intr+0xf0/0xf0 [10240.071505] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10240.071715] kthread+0x1d1/0x200 [10240.071721] ? set_kthread_struct+0x70/0x70 [10240.071725] ret_from_fork+0x1f/0x30 [10240.077101] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10240.077181] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10240.077189] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10240.077192] RIP: 0010:native_safe_halt+0x2e/0x50 [10240.077205] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10240.077208] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10240.084458] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10240.086334] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10240.086448] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10240.086488] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10240.086787] mdc_intent_lock+0x353/0x7e0 [mdc] [10240.086820] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10240.086908] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10240.087773] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10240.134435] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10240.134446] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10240.134449] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10240.134451] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10240.134454] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10240.134457] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10240.134460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10240.134461] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10240.134469] Call Trace: [10240.134477] [10240.134483] ? show_regs.cold.9+0x22/0x2f [10240.134495] ? watchdog_timer_fn.cold.6+0x62/0x10a [10240.134501] ? __hrtimer_run_queues+0x1c2/0x4d0 [10240.134506] ? watchdog+0x40/0x40 [10240.134510] ? hrtimer_interrupt+0x128/0x2c0 [10240.134515] ? smp_apic_timer_interrupt+0xa6/0x270 [10240.134524] ? apic_timer_interrupt+0xf/0x20 [10240.134529] [10240.134532] ? native_safe_halt+0x2e/0x50 [10240.134537] kvm_wait+0xcd/0xe0 [10240.134545] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10240.134555] do_raw_spin_lock+0x177/0x190 [10240.134560] _raw_spin_lock+0x21/0x30 [10240.134564] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10240.134836] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10240.135027] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10240.135247] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10240.135452] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10240.135663] ? woken_wake_function+0x30/0x30 [10240.135672] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10240.135877] kthread+0x1d1/0x200 [10240.135882] ? set_kthread_struct+0x70/0x70 [10240.135886] ret_from_fork+0x1f/0x30 [10240.675428] lmv_intent_open+0x338/0x1150 [lmv] [10240.677765] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10240.680350] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10240.683006] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10240.685106] ll_intent_lock+0xc5/0xaa0 [lustre] [10240.687349] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10240.690080] ll_lookup_it+0xe0e/0x2270 [lustre] [10240.692306] ? do_raw_spin_unlock+0x75/0x190 [10240.693895] ? _raw_spin_unlock+0x12/0x30 [10240.696083] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10240.701022] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10240.703923] ll_atomic_open+0x36a/0xd70 [lustre] [10240.706949] lookup_open+0x356/0xb40 [10240.708858] do_last+0x313/0xfc0 [10240.711023] ? nd_jump_root+0xe5/0x160 [10240.712953] ? path_init+0x437/0x520 [10240.714398] path_openat+0xf7/0x500 [10240.719405] ? do_raw_spin_unlock+0x75/0x190 [10240.721837] do_filp_open+0x99/0x140 [10240.723269] ? getname_flags+0x6e/0x330 [10240.726521] ? __check_object_size+0xff/0x256 [10240.731037] ? do_raw_spin_unlock+0x75/0x190 [10240.736018] ? _raw_spin_unlock+0x12/0x30 [10240.739876] do_sys_openat2+0x2b4/0x410 [10240.741252] do_sys_open+0x73/0xa0 [10240.744100] __x64_sys_openat+0x24/0x30 [10240.745865] do_syscall_64+0xc1/0x440 [10240.747426] entry_SYSCALL_64_after_hwframe+0x49/0xae [10240.749792] RIP: 0033:0x7f53425a5332 [10240.752181] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10240.762798] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10240.766123] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10240.770440] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10240.776306] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10240.780471] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10240.785315] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10254.622989] rcu: INFO: rcu_sched self-detected stall on CPU [10254.629327] rcu: 2-....: (4472673 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [10254.641579] (t=8160196 jiffies g=274185 q=65874) [10254.644788] rcu: rcu_sched kthread starved for 7618562 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [10254.657297] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10254.671946] rcu: RCU grace-period kthread stack dump: [10254.677772] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10254.696399] Call Trace: [10254.699938] __schedule+0x351/0xcb0 [10254.708155] schedule+0xc0/0x180 [10254.712827] schedule_timeout+0xb4/0x190 [10254.718691] ? do_raw_spin_unlock+0x75/0x190 [10254.722670] ? __next_timer_interrupt+0x160/0x160 [10254.728575] ? prepare_to_swait_event+0xf3/0x140 [10254.730042] rcu_gp_kthread+0x8dd/0x15b0 [10254.731436] ? force_qs_rnp+0x2c0/0x2c0 [10254.745600] kthread+0x1d1/0x200 [10254.752777] ? set_kthread_struct+0x70/0x70 [10254.759409] ret_from_fork+0x1f/0x30 [10254.763901] rcu: Stack dump where RCU GP kthread last ran: [10254.772019] Sending NMI from CPU 2 to CPUs 1: [10254.784512] NMI backtrace for cpu 1 [10254.784524] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10254.784526] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10254.784528] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10254.784530] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10254.784532] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [10254.784535] RAX: 0000000000002d81 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10254.784537] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10254.784539] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10254.784540] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10254.784542] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10254.784546] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10254.784548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10254.784549] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10254.784551] Call Trace: [10254.784552] [10254.784553] ? show_regs.cold.9+0x22/0x2f [10254.784554] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10254.784555] ? nmi_cpu_backtrace_handler+0x15/0x40 [10254.784557] ? nmi_handle+0xbd/0x210 [10254.784558] ? default_do_nmi+0x4b/0x1b0 [10254.784559] ? do_nmi+0x280/0x380 [10254.784560] ? end_repeat_nmi+0x16/0x1f [10254.784562] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10254.784563] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10254.784564] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10254.784565] [10254.784566] do_raw_spin_lock+0x177/0x190 [10254.784567] _raw_spin_lock+0x21/0x30 [10254.784568] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10254.784569] ? do_raw_spin_unlock+0x75/0x190 [10254.784570] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10254.784571] ? do_wait_intr+0xf0/0xf0 [10254.784572] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10254.784573] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10254.784574] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10254.784575] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10254.784576] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10254.784580] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10254.784581] mdc_intent_lock+0x353/0x7e0 [mdc] [10254.784582] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10254.784583] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10254.784584] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10254.784585] lmv_intent_open+0x338/0x1150 [lmv] [10254.784587] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10254.784588] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10254.784589] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10254.784590] ll_intent_lock+0xc5/0xaa0 [lustre] [10254.784591] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10254.784592] ll_lookup_it+0xe0e/0x2270 [lustre] [10254.784593] ? do_raw_spin_unlock+0x75/0x190 [10254.784595] ? _raw_spin_unlock+0x12/0x30 [10254.784596] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10254.784597] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10254.784598] ll_atomic_open+0x36a/0xd70 [lustre] [10254.784599] lookup_open+0x356/0xb40 [10254.784600] do_last+0x313/0xfc0 [10254.784602] ? nd_jump_root+0xe5/0x160 [10254.784603] ? path_init+0x437/0x520 [10254.784604] path_openat+0xf7/0x500 [10254.784605] ? do_raw_spin_unlock+0x75/0x190 [10254.784607] do_filp_open+0x99/0x140 [10254.784608] ? getname_flags+0x6e/0x330 [10254.784609] ? __check_object_size+0xff/0x256 [10254.784610] ? do_raw_spin_unlock+0x75/0x190 [10254.784611] ? _raw_spin_unlock+0x12/0x30 [10254.784612] do_sys_openat2+0x2b4/0x410 [10254.784613] do_sys_open+0x73/0xa0 [10254.784614] __x64_sys_openat+0x24/0x30 [10254.784615] do_syscall_64+0xc1/0x440 [10254.784616] entry_SYSCALL_64_after_hwframe+0x49/0xae [10254.784618] RIP: 0033:0x7f53425a5332 [10254.784619] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10254.784621] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10254.784624] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10254.784628] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10254.784629] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10254.784630] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10254.784632] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10254.784927] Sending NMI from CPU 2 to CPUs 1: [10255.127547] NMI backtrace for cpu 1 [10255.127556] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10255.127558] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10255.127560] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10255.127561] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10255.127564] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [10255.127567] RAX: 0000000000001f84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10255.127569] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10255.127570] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10255.127571] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10255.127573] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10255.127574] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10255.127575] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10255.127577] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10255.127578] Call Trace: [10255.127579] [10255.127580] ? show_regs.cold.9+0x22/0x2f [10255.127581] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10255.127582] ? nmi_cpu_backtrace_handler+0x15/0x40 [10255.127584] ? nmi_handle+0xbd/0x210 [10255.127585] ? default_do_nmi+0x4b/0x1b0 [10255.127586] ? do_nmi+0x280/0x380 [10255.127587] ? end_repeat_nmi+0x16/0x1f [10255.127588] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10255.127589] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10255.127591] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10255.127592] [10255.127593] do_raw_spin_lock+0x177/0x190 [10255.127594] _raw_spin_lock+0x21/0x30 [10255.127595] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10255.127597] ? do_raw_spin_unlock+0x75/0x190 [10255.127597] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10255.127599] ? do_wait_intr+0xf0/0xf0 [10255.127605] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10255.127606] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10255.127607] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10255.127608] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10255.127609] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10255.127610] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10255.127611] mdc_intent_lock+0x353/0x7e0 [mdc] [10255.127612] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10255.127613] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10255.127614] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10255.127615] lmv_intent_open+0x338/0x1150 [lmv] [10255.127616] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10255.127618] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10255.127619] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10255.127620] ll_intent_lock+0xc5/0xaa0 [lustre] [10255.127621] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10255.127622] ll_lookup_it+0xe0e/0x2270 [lustre] [10255.127623] ? do_raw_spin_unlock+0x75/0x190 [10255.127624] ? _raw_spin_unlock+0x12/0x30 [10255.127625] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10255.127627] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10255.127628] ll_atomic_open+0x36a/0xd70 [lustre] [10255.127629] lookup_open+0x356/0xb40 [10255.127630] do_last+0x313/0xfc0 [10255.127631] ? nd_jump_root+0xe5/0x160 [10255.127632] ? path_init+0x437/0x520 [10255.127633] path_openat+0xf7/0x500 [10255.127635] ? do_raw_spin_unlock+0x75/0x190 [10255.127635] do_filp_open+0x99/0x140 [10255.127636] ? getname_flags+0x6e/0x330 [10255.127637] ? __check_object_size+0xff/0x256 [10255.127639] ? do_raw_spin_unlock+0x75/0x190 [10255.127640] ? _raw_spin_unlock+0x12/0x30 [10255.127641] do_sys_openat2+0x2b4/0x410 [10255.127642] do_sys_open+0x73/0xa0 [10255.127643] __x64_sys_openat+0x24/0x30 [10255.127644] do_syscall_64+0xc1/0x440 [10255.127645] entry_SYSCALL_64_after_hwframe+0x49/0xae [10255.127646] RIP: 0033:0x7f53425a5332 [10255.127647] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10255.127650] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10255.127652] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10255.127654] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10255.127656] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10255.127657] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10255.127658] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10255.128417] NMI backtrace for cpu 2 [10255.534444] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10255.547310] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10255.557777] Call Trace: [10255.558715] [10255.562534] ? dump_stack+0xbb/0x10e [10255.565345] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10255.569297] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10255.576216] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10255.579493] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10255.581884] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10255.584193] ? rcu_dump_cpu_stacks+0x11b/0x171 [10255.586494] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10255.588936] ? update_process_times+0x6d/0xc0 [10255.590382] ? tick_sched_handle.isra.18+0x27/0x90 [10255.592331] ? tick_sched_timer+0x53/0xb0 [10255.593769] ? __hrtimer_run_queues+0x1c2/0x4d0 [10255.595725] ? tick_sched_handle.isra.18+0x90/0x90 [10255.597322] ? hrtimer_interrupt+0x128/0x2c0 [10255.598751] ? smp_apic_timer_interrupt+0xa6/0x270 [10255.601122] ? apic_timer_interrupt+0xf/0x20 [10255.602553] [10255.603480] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10255.606363] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10255.612314] ? do_raw_spin_lock+0x177/0x190 [10255.615240] ? _raw_spin_lock+0x21/0x30 [10255.617284] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10255.620269] ? obd_mod_free+0x46/0x140 [ptlrpc] [10255.622393] ? mdc_commit_open+0x91/0xb0 [mdc] [10255.626872] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10255.633490] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10255.639312] ? after_reply+0xe2c/0x1780 [ptlrpc] [10255.642374] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10255.645861] ? __next_timer_interrupt+0x160/0x160 [10255.648320] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10255.651362] ? do_wait_intr+0xf0/0xf0 [10255.654446] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10255.658672] ? kthread+0x1d1/0x200 [10255.661822] ? set_kthread_struct+0x70/0x70 [10255.670680] ? ret_from_fork+0x1f/0x30 [10256.052045] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [10256.055160] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10256.070909] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10256.074853] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10256.078125] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10256.081287] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10256.083712] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10256.090565] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10256.093722] RAX: 000000000000221b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10256.096553] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10256.099742] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10256.102684] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10256.105164] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10256.107915] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10256.111326] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10256.113457] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10256.116436] Call Trace: [10256.117547] [10256.118383] ? show_regs.cold.9+0x22/0x2f [10256.120121] ? watchdog_timer_fn.cold.6+0x62/0x10a [10256.121850] ? __hrtimer_run_queues+0x1c2/0x4d0 [10256.123664] ? watchdog+0x40/0x40 [10256.124846] ? hrtimer_interrupt+0x128/0x2c0 [10256.126599] ? smp_apic_timer_interrupt+0xa6/0x270 [10256.128354] ? apic_timer_interrupt+0xf/0x20 [10256.130215] [10256.131075] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10256.133268] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10256.135597] ? newidle_balance.isra.93+0x207/0x580 [10256.137441] do_raw_spin_lock+0x177/0x190 [10256.138929] _raw_spin_lock+0x21/0x30 [10256.140242] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10256.142903] process_one_work+0x2c8/0x700 [10256.144763] worker_thread+0x296/0x6e0 [10256.146121] ? rescuer_thread+0x570/0x570 [10256.147694] kthread+0x1d1/0x200 [10256.148582] ? set_kthread_struct+0x70/0x70 [10256.150123] ret_from_fork+0x1f/0x30 [10258.399074] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7621s! [10258.402512] Showing busy workqueues and worker pools: [10258.409547] workqueue events: flags=0x0 [10258.414131] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10258.419412] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10258.429594] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10258.433610] pending: cpu_latency_work [ptlrpc] [10258.436530] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10258.439964] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10258.447495] workqueue events_long: flags=0x0 [10258.449575] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10258.458554] pending: lu_objects_flush [obdclass] [10258.463284] workqueue events_unbound: flags=0x2 [10258.469323] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10258.476853] pending: flush_memcg_stats_dwork, idle_cull_fn [10258.479735] workqueue events_power_efficient: flags=0x80 [10258.485920] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10258.494678] pending: check_lifetime, do_cache_clean [10258.496875] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10258.500560] pending: neigh_periodic_work, neigh_periodic_work [10258.502518] workqueue mm_percpu_wq: flags=0x8 [10258.503712] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10258.509181] pending: vmstat_update, lru_add_drain_per_cpu [10258.512289] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10258.517195] pending: vmstat_update, lru_add_drain_per_cpu [10258.519670] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10258.522623] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10258.525379] workqueue xprtiod: flags=0xa [10258.527158] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10258.530003] pending: xprt_autoclose, xs_error_handle [10258.533954] workqueue ipv6_addrconf: flags=0x40008 [10258.536159] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10258.540373] pending: addrconf_verify_work [10258.542794] workqueue knbd0-recv: flags=0x1a [10258.544789] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10258.548720] in-flight: 651:recv_work [10258.555336] workqueue ptlrpc_pinger: flags=0x2 [10258.557455] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10258.565777] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10258.573667] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7621s workers=4 idle: 15979 31824 8 [10258.576699] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10249s workers=2 idle: 654 [10268.058019] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [10268.061092] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10268.085411] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10268.093399] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10268.098785] RIP: 0010:native_safe_halt+0x2e/0x50 [10268.102351] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10268.115397] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10268.119215] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10268.123373] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10268.126970] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10268.131145] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10268.134540] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10268.138204] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10268.142452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10268.146134] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10268.151797] Call Trace: [10268.154394] [10268.157550] ? show_regs.cold.9+0x22/0x2f [10268.162457] ? watchdog_timer_fn.cold.6+0x62/0x10a [10268.167576] ? __hrtimer_run_queues+0x1c2/0x4d0 [10268.173926] ? watchdog+0x40/0x40 [10268.177384] ? hrtimer_interrupt+0x128/0x2c0 [10268.182093] ? smp_apic_timer_interrupt+0xa6/0x270 [10268.191229] ? apic_timer_interrupt+0xf/0x20 [10268.196210] [10268.197519] ? native_safe_halt+0x2e/0x50 [10268.201194] kvm_wait+0xcd/0xe0 [10268.203567] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10268.207579] do_raw_spin_lock+0x177/0x190 [10268.212151] _raw_spin_lock+0x21/0x30 [10268.213930] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10268.218497] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10268.222078] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10268.227237] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10268.230413] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10268.232497] ? woken_wake_function+0x30/0x30 [10268.236688] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10268.246631] kthread+0x1d1/0x200 [10268.248981] ? set_kthread_struct+0x70/0x70 [10268.258395] ret_from_fork+0x1f/0x30 [10280.057115] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [10280.061113] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10280.061498] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [10280.087133] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10280.087148] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10280.087151] RIP: 0010:native_safe_halt+0x2e/0x50 [10280.087164] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10280.087168] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10280.087173] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10280.087176] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10280.087178] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10280.087181] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10280.087183] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10280.087186] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10280.087190] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10280.087192] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10280.087200] Call Trace: [10280.087206] [10280.087212] ? show_regs.cold.9+0x22/0x2f [10280.087223] ? watchdog_timer_fn.cold.6+0x62/0x10a [10280.087231] ? __hrtimer_run_queues+0x1c2/0x4d0 [10280.087235] ? watchdog+0x40/0x40 [10280.087239] ? hrtimer_interrupt+0x128/0x2c0 [10280.087244] ? smp_apic_timer_interrupt+0xa6/0x270 [10280.087250] ? apic_timer_interrupt+0xf/0x20 [10280.087255] [10280.087258] ? native_safe_halt+0x2e/0x50 [10280.087262] kvm_wait+0xcd/0xe0 [10280.087269] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10280.087277] do_raw_spin_lock+0x177/0x190 [10280.087282] _raw_spin_lock+0x21/0x30 [10280.087290] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10280.087612] obd_mod_free+0x46/0x140 [ptlrpc] [10280.094939] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10280.095013] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10280.095021] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10280.095023] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10280.095037] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10280.095041] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10280.095046] RAX: 0000000000004299 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10280.095048] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10280.095050] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10280.095175] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10280.095177] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10280.095181] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10280.095183] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10280.095185] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10280.095193] Call Trace: [10280.095199] [10280.095205] ? show_regs.cold.9+0x22/0x2f [10280.095215] ? watchdog_timer_fn.cold.6+0x62/0x10a [10280.095221] ? __hrtimer_run_queues+0x1c2/0x4d0 [10280.095225] ? watchdog+0x40/0x40 [10280.095229] ? hrtimer_interrupt+0x128/0x2c0 [10280.095233] ? smp_apic_timer_interrupt+0xa6/0x270 [10280.095240] ? apic_timer_interrupt+0xf/0x20 [10280.095245] [10280.095248] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10280.095261] do_raw_spin_lock+0x177/0x190 [10280.095266] _raw_spin_lock+0x21/0x30 [10280.095272] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10280.095688] ? do_raw_spin_unlock+0x75/0x190 [10280.095694] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10280.096027] ? do_wait_intr+0xf0/0xf0 [10280.096037] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10280.096314] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10280.096976] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10280.097288] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10280.097400] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10280.097437] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10280.097466] mdc_intent_lock+0x353/0x7e0 [mdc] [10280.097492] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10280.097834] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10280.098381] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10280.098415] lmv_intent_open+0x338/0x1150 [lmv] [10280.098447] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10280.098705] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10280.098949] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10280.098972] ll_intent_lock+0xc5/0xaa0 [lustre] [10280.099132] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10280.099229] ll_lookup_it+0xe0e/0x2270 [lustre] [10280.099312] ? do_raw_spin_unlock+0x75/0x190 [10280.099319] ? _raw_spin_unlock+0x12/0x30 [10280.099326] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10280.099359] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10280.099380] ll_atomic_open+0x36a/0xd70 [lustre] [10280.110210] mdc_commit_open+0x91/0xb0 [mdc] [10280.110261] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10280.110583] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10280.110987] after_reply+0xe2c/0x1780 [ptlrpc] [10280.111181] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10280.130148] lookup_open+0x356/0xb40 [10280.130169] do_last+0x313/0xfc0 [10280.130175] ? nd_jump_root+0xe5/0x160 [10280.130180] ? path_init+0x437/0x520 [10280.130186] path_openat+0xf7/0x500 [10280.130191] ? do_raw_spin_unlock+0x75/0x190 [10280.130198] do_filp_open+0x99/0x140 [10280.130204] ? getname_flags+0x6e/0x330 [10280.130208] ? __check_object_size+0xff/0x256 [10280.130216] ? do_raw_spin_unlock+0x75/0x190 [10280.130220] ? _raw_spin_unlock+0x12/0x30 [10280.130227] do_sys_openat2+0x2b4/0x410 [10280.130231] do_sys_open+0x73/0xa0 [10280.130235] __x64_sys_openat+0x24/0x30 [10280.130239] do_syscall_64+0xc1/0x440 [10280.130246] entry_SYSCALL_64_after_hwframe+0x49/0xae [10280.130252] RIP: 0033:0x7f53425a5332 [10280.130261] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10280.130266] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10280.130272] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10280.130274] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10280.130277] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10280.130279] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10280.130282] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10280.759441] ? __next_timer_interrupt+0x160/0x160 [10280.763775] ptlrpcd+0xa82/0xe40 [ptlrpc] [10280.766749] ? do_wait_intr+0xf0/0xf0 [10280.769265] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10280.771468] kthread+0x1d1/0x200 [10280.773983] ? set_kthread_struct+0x70/0x70 [10280.778632] ret_from_fork+0x1f/0x30 [10284.053682] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [10284.069895] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10284.099849] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10284.110916] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10284.119591] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10284.125640] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10284.133957] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10284.150280] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10284.157384] RAX: 0000000000007e8f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10284.160639] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10284.162969] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10284.165849] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10284.169345] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10284.171964] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10284.179179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10284.182408] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10284.189793] Call Trace: [10284.192124] [10284.193133] ? show_regs.cold.9+0x22/0x2f [10284.194938] ? watchdog_timer_fn.cold.6+0x62/0x10a [10284.198063] ? __hrtimer_run_queues+0x1c2/0x4d0 [10284.201923] ? watchdog+0x40/0x40 [10284.205681] ? hrtimer_interrupt+0x128/0x2c0 [10284.208985] ? smp_apic_timer_interrupt+0xa6/0x270 [10284.214079] ? apic_timer_interrupt+0xf/0x20 [10284.216740] [10284.221255] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10284.222576] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10284.225907] ? newidle_balance.isra.93+0x207/0x580 [10284.229946] do_raw_spin_lock+0x177/0x190 [10284.233049] _raw_spin_lock+0x21/0x30 [10284.236040] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10284.240929] process_one_work+0x2c8/0x700 [10284.243840] worker_thread+0x296/0x6e0 [10284.247605] ? rescuer_thread+0x570/0x570 [10284.249090] kthread+0x1d1/0x200 [10284.252677] ? set_kthread_struct+0x70/0x70 [10284.255808] ret_from_fork+0x1f/0x30 [10288.610700] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7651s! [10288.614901] Showing busy workqueues and worker pools: [10288.618934] workqueue events: flags=0x0 [10288.620872] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10288.624825] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10288.632142] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10288.637138] pending: cpu_latency_work [ptlrpc] [10288.639766] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10288.642709] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10288.647831] workqueue events_long: flags=0x0 [10288.649918] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10288.653646] pending: lu_objects_flush [obdclass] [10288.656672] workqueue events_unbound: flags=0x2 [10288.658754] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10288.661887] pending: flush_memcg_stats_dwork, idle_cull_fn [10288.664912] workqueue events_power_efficient: flags=0x80 [10288.667817] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10288.671802] pending: check_lifetime, do_cache_clean [10288.674856] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10288.678591] pending: neigh_periodic_work, neigh_periodic_work [10288.682041] workqueue mm_percpu_wq: flags=0x8 [10288.684660] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10288.688480] pending: vmstat_update, lru_add_drain_per_cpu [10288.691484] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10288.695273] pending: vmstat_update, lru_add_drain_per_cpu [10288.698747] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10288.702546] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10288.706248] workqueue xprtiod: flags=0xa [10288.708439] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10288.711987] pending: xprt_autoclose, xs_error_handle [10288.714976] workqueue ipv6_addrconf: flags=0x40008 [10288.717838] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10288.721308] pending: addrconf_verify_work [10288.724208] workqueue knbd0-recv: flags=0x1a [10288.726438] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10288.732843] in-flight: 651:recv_work [10288.734622] workqueue ptlrpc_pinger: flags=0x2 [10288.737249] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10288.740948] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10288.744491] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7651s workers=4 idle: 15979 31824 8 [10288.750203] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10279s workers=2 idle: 654 [10296.063963] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [10296.078416] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10296.096832] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10296.101660] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10296.105618] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10296.109398] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10296.126510] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10296.133605] RAX: 0000000000002b1b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10296.138335] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10296.141801] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10296.144560] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10296.149628] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10296.154395] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10296.157560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10296.161706] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10296.168540] Call Trace: [10296.169316] [10296.169829] ? show_regs.cold.9+0x22/0x2f [10296.173004] ? watchdog_timer_fn.cold.6+0x62/0x10a [10296.174649] ? __hrtimer_run_queues+0x1c2/0x4d0 [10296.176547] ? watchdog+0x40/0x40 [10296.178205] ? hrtimer_interrupt+0x128/0x2c0 [10296.180650] ? smp_apic_timer_interrupt+0xa6/0x270 [10296.187675] ? apic_timer_interrupt+0xf/0x20 [10296.189360] [10296.190026] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10296.192204] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10296.194485] do_raw_spin_lock+0x177/0x190 [10296.196184] _raw_spin_lock+0x21/0x30 [10296.199528] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10296.201828] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10296.203801] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10296.207591] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10296.213023] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10296.215777] ? woken_wake_function+0x30/0x30 [10296.217522] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10296.220562] kthread+0x1d1/0x200 [10296.221941] ? set_kthread_struct+0x70/0x70 [10296.223843] ret_from_fork+0x1f/0x30 [10308.057436] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [10308.057934] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [10308.070702] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10308.070776] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10308.070784] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10308.070787] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10308.070801] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10308.070806] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10308.070810] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10308.070813] RDX: 0000000000007980 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10308.070815] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10308.070817] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10308.070819] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10308.070822] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10308.070825] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10308.070827] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10308.070836] Call Trace: [10308.070842] [10308.070847] ? show_regs.cold.9+0x22/0x2f [10308.070856] ? watchdog_timer_fn.cold.6+0x62/0x10a [10308.070862] ? __hrtimer_run_queues+0x1c2/0x4d0 [10308.070880] ? watchdog+0x40/0x40 [10308.070885] ? hrtimer_interrupt+0x128/0x2c0 [10308.070889] ? smp_apic_timer_interrupt+0xa6/0x270 [10308.070897] ? apic_timer_interrupt+0xf/0x20 [10308.070902] [10308.070905] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10308.070909] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10308.070914] do_raw_spin_lock+0x177/0x190 [10308.070919] _raw_spin_lock+0x21/0x30 [10308.070925] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10308.087852] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10308.087944] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10308.087951] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10308.087954] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10308.087969] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10308.087974] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10308.087978] RAX: 0000000000001456 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10308.087981] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10308.087983] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10308.087985] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10308.087988] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10308.087991] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10308.087995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10308.087997] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10308.088006] Call Trace: [10308.088013] [10308.088018] ? show_regs.cold.9+0x22/0x2f [10308.088039] ? watchdog_timer_fn.cold.6+0x62/0x10a [10308.088046] ? __hrtimer_run_queues+0x1c2/0x4d0 [10308.088050] ? watchdog+0x40/0x40 [10308.088054] ? hrtimer_interrupt+0x128/0x2c0 [10308.088059] ? smp_apic_timer_interrupt+0xa6/0x270 [10308.088067] ? apic_timer_interrupt+0xf/0x20 [10308.088072] [10308.088074] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10308.088081] do_raw_spin_lock+0x177/0x190 [10308.088086] _raw_spin_lock+0x21/0x30 [10308.088091] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10308.088847] ? do_raw_spin_unlock+0x75/0x190 [10308.089082] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10308.089917] ? do_wait_intr+0xf0/0xf0 [10308.089928] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10308.090148] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10308.090607] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10308.090840] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10308.091064] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10308.091105] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10308.091134] mdc_intent_lock+0x353/0x7e0 [mdc] [10308.091162] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10308.091245] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10308.136863] obd_mod_free+0x46/0x140 [ptlrpc] [10308.145981] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10308.146044] lmv_intent_open+0x338/0x1150 [lmv] [10308.146078] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10308.151324] mdc_commit_open+0x91/0xb0 [mdc] [10308.151375] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10308.151654] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10308.152098] after_reply+0xe2c/0x1780 [ptlrpc] [10308.152297] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10308.152491] ? __next_timer_interrupt+0x160/0x160 [10308.152502] ptlrpcd+0xa82/0xe40 [ptlrpc] [10308.152703] ? do_wait_intr+0xf0/0xf0 [10308.152713] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10308.153292] kthread+0x1d1/0x200 [10308.153299] ? set_kthread_struct+0x70/0x70 [10308.153303] ret_from_fork+0x1f/0x30 [10308.593450] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10308.595194] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10308.596396] ll_intent_lock+0xc5/0xaa0 [lustre] [10308.598703] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10308.602819] ll_lookup_it+0xe0e/0x2270 [lustre] [10308.604986] ? do_raw_spin_unlock+0x75/0x190 [10308.606619] ? _raw_spin_unlock+0x12/0x30 [10308.607990] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10308.609583] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10308.612739] ll_atomic_open+0x36a/0xd70 [lustre] [10308.614505] lookup_open+0x356/0xb40 [10308.616945] do_last+0x313/0xfc0 [10308.618436] ? nd_jump_root+0xe5/0x160 [10308.621540] ? path_init+0x437/0x520 [10308.624688] path_openat+0xf7/0x500 [10308.626124] ? do_raw_spin_unlock+0x75/0x190 [10308.628671] do_filp_open+0x99/0x140 [10308.630348] ? getname_flags+0x6e/0x330 [10308.634382] ? __check_object_size+0xff/0x256 [10308.636994] ? do_raw_spin_unlock+0x75/0x190 [10308.638354] ? _raw_spin_unlock+0x12/0x30 [10308.639547] do_sys_openat2+0x2b4/0x410 [10308.641229] do_sys_open+0x73/0xa0 [10308.644229] __x64_sys_openat+0x24/0x30 [10308.648772] do_syscall_64+0xc1/0x440 [10308.654030] entry_SYSCALL_64_after_hwframe+0x49/0xae [10308.657942] RIP: 0033:0x7f53425a5332 [10308.660366] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10308.669653] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10308.672205] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10308.677201] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10308.679754] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10308.684180] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10308.686967] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10316.052271] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [10316.055088] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10316.095134] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10316.111577] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10316.118391] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10316.120711] RIP: 0010:native_safe_halt+0x2e/0x50 [10316.124144] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10316.136254] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10316.138030] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10316.139877] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10316.142030] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10316.145596] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10316.149072] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10316.154959] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10316.160086] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10316.161868] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10316.164235] Call Trace: [10316.165011] [10316.165593] ? show_regs.cold.9+0x22/0x2f [10316.166638] ? watchdog_timer_fn.cold.6+0x62/0x10a [10316.168743] ? __hrtimer_run_queues+0x1c2/0x4d0 [10316.170663] ? watchdog+0x40/0x40 [10316.171675] ? hrtimer_interrupt+0x128/0x2c0 [10316.173228] ? smp_apic_timer_interrupt+0xa6/0x270 [10316.174808] ? apic_timer_interrupt+0xf/0x20 [10316.176164] [10316.176872] ? native_safe_halt+0x2e/0x50 [10316.179068] kvm_wait+0xcd/0xe0 [10316.181452] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10316.183671] ? newidle_balance.isra.93+0x207/0x580 [10316.187693] do_raw_spin_lock+0x177/0x190 [10316.189754] _raw_spin_lock+0x21/0x30 [10316.191762] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10316.198676] process_one_work+0x2c8/0x700 [10316.203407] worker_thread+0x296/0x6e0 [10316.207753] ? rescuer_thread+0x570/0x570 [10316.214417] kthread+0x1d1/0x200 [10316.217791] ? set_kthread_struct+0x70/0x70 [10316.220590] ret_from_fork+0x1f/0x30 [10318.816557] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7681s! [10318.820013] Showing busy workqueues and worker pools: [10318.821424] workqueue events: flags=0x0 [10318.827249] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10318.830568] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10318.838820] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10318.846627] pending: cpu_latency_work [ptlrpc] [10318.851785] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10318.865550] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10318.881124] workqueue events_long: flags=0x0 [10318.889887] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10318.894561] pending: lu_objects_flush [obdclass] [10318.901389] workqueue events_unbound: flags=0x2 [10318.910115] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10318.916474] pending: flush_memcg_stats_dwork, idle_cull_fn [10318.921659] workqueue events_power_efficient: flags=0x80 [10318.939793] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10318.953340] pending: check_lifetime, do_cache_clean [10318.960856] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10318.970063] pending: neigh_periodic_work, neigh_periodic_work [10318.975380] workqueue mm_percpu_wq: flags=0x8 [10318.976943] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10318.986618] pending: vmstat_update, lru_add_drain_per_cpu [10318.994542] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10319.001683] pending: vmstat_update, lru_add_drain_per_cpu [10319.006254] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10319.010494] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10319.018656] workqueue xprtiod: flags=0xa [10319.022570] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10319.028068] pending: xprt_autoclose, xs_error_handle [10319.033522] workqueue ipv6_addrconf: flags=0x40008 [10319.035960] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10319.039222] pending: addrconf_verify_work [10319.040789] workqueue knbd0-recv: flags=0x1a [10319.044086] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10319.051729] in-flight: 651:recv_work [10319.053608] workqueue ptlrpc_pinger: flags=0x2 [10319.057369] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10319.063583] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10319.071264] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7681s workers=4 idle: 15979 31824 8 [10319.079216] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10309s workers=2 idle: 654 [10324.063677] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [10324.070674] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10324.150627] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10324.157586] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10324.161550] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10324.179198] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10324.200248] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10324.209342] RAX: 0000000000007d87 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10324.212714] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10324.215681] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10324.218527] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10324.221916] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10324.229900] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10324.238617] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10324.241468] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10324.249274] Call Trace: [10324.250055] [10324.250709] ? show_regs.cold.9+0x22/0x2f [10324.253396] ? watchdog_timer_fn.cold.6+0x62/0x10a [10324.257310] ? __hrtimer_run_queues+0x1c2/0x4d0 [10324.264821] ? watchdog+0x40/0x40 [10324.267543] ? hrtimer_interrupt+0x128/0x2c0 [10324.269521] ? smp_apic_timer_interrupt+0xa6/0x270 [10324.272258] ? apic_timer_interrupt+0xf/0x20 [10324.273951] [10324.274760] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10324.277193] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10324.282596] do_raw_spin_lock+0x177/0x190 [10324.286289] _raw_spin_lock+0x21/0x30 [10324.294687] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10324.302626] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10324.310234] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10324.317285] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10324.321916] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10324.329207] ? woken_wake_function+0x30/0x30 [10324.333979] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10324.338101] kthread+0x1d1/0x200 [10324.342725] ? set_kthread_struct+0x70/0x70 [10324.345183] ret_from_fork+0x1f/0x30 [10336.056474] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [10336.057018] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [10336.063218] Modules linked in: lustre(O) [10336.065873] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10336.065947] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10336.065954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10336.065958] RIP: 0010:native_safe_halt+0x2e/0x50 [10336.065971] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10336.065976] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10336.065980] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10336.065982] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10336.065984] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10336.065987] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10336.065989] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10336.065992] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10336.065995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10336.065997] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10336.066005] Call Trace: [10336.066011] [10336.066016] ? show_regs.cold.9+0x22/0x2f [10336.066025] ? watchdog_timer_fn.cold.6+0x62/0x10a [10336.066032] ? __hrtimer_run_queues+0x1c2/0x4d0 [10336.066036] ? watchdog+0x40/0x40 [10336.066041] ? hrtimer_interrupt+0x128/0x2c0 [10336.066044] ? smp_apic_timer_interrupt+0xa6/0x270 [10336.066053] ? apic_timer_interrupt+0xf/0x20 [10336.066056] [10336.066059] ? native_safe_halt+0x2e/0x50 [10336.066063] kvm_wait+0xcd/0xe0 [10336.066070] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10336.066077] do_raw_spin_lock+0x177/0x190 [10336.066082] _raw_spin_lock+0x21/0x30 [10336.066087] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10336.066354] obd_mod_free+0x46/0x140 [ptlrpc] [10336.066560] mdc_commit_open+0x91/0xb0 [mdc] [10336.066602] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10336.066805] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10336.067007] after_reply+0xe2c/0x1780 [ptlrpc] [10336.071594] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10336.071666] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10336.071673] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10336.071676] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10336.071689] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10336.071694] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10336.071698] RAX: 0000000000007915 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10336.071701] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10336.071703] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10336.071705] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10336.071707] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10336.071710] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10336.071713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10336.071716] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10336.071724] Call Trace: [10336.071729] [10336.071734] ? show_regs.cold.9+0x22/0x2f [10336.071744] ? watchdog_timer_fn.cold.6+0x62/0x10a [10336.071749] ? __hrtimer_run_queues+0x1c2/0x4d0 [10336.071754] ? watchdog+0x40/0x40 [10336.071758] ? hrtimer_interrupt+0x128/0x2c0 [10336.071762] ? smp_apic_timer_interrupt+0xa6/0x270 [10336.071770] ? apic_timer_interrupt+0xf/0x20 [10336.071775] [10336.071778] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10336.071784] do_raw_spin_lock+0x177/0x190 [10336.071789] _raw_spin_lock+0x21/0x30 [10336.071794] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10336.072048] ? do_raw_spin_unlock+0x75/0x190 [10336.072053] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10336.072246] ? do_wait_intr+0xf0/0xf0 [10336.072253] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10336.072443] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10336.072666] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10336.072888] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10336.072994] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10336.073032] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10336.073060] mdc_intent_lock+0x353/0x7e0 [mdc] [10336.073088] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10336.073174] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10336.073388] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10336.084000] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10336.088581] lmv_intent_open+0x338/0x1150 [lmv] [10336.088628] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10336.088754] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10336.088836] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10336.088856] ll_intent_lock+0xc5/0xaa0 [lustre] [10336.088936] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10336.089025] ll_lookup_it+0xe0e/0x2270 [lustre] [10336.089109] ? do_raw_spin_unlock+0x75/0x190 [10336.089118] ? _raw_spin_unlock+0x12/0x30 [10336.089124] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10336.089149] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10336.089172] ll_atomic_open+0x36a/0xd70 [lustre] [10336.092745] ? __next_timer_interrupt+0x160/0x160 [10336.092764] ptlrpcd+0xa82/0xe40 [ptlrpc] [10336.093119] ? do_wait_intr+0xf0/0xf0 [10336.093127] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10336.095939] lookup_open+0x356/0xb40 [10336.101093] kthread+0x1d1/0x200 [10336.101112] ? set_kthread_struct+0x70/0x70 [10336.101116] ret_from_fork+0x1f/0x30 [10336.607899] do_last+0x313/0xfc0 [10336.608869] ? nd_jump_root+0xe5/0x160 [10336.609865] ? path_init+0x437/0x520 [10336.611231] path_openat+0xf7/0x500 [10336.612839] ? do_raw_spin_unlock+0x75/0x190 [10336.614355] do_filp_open+0x99/0x140 [10336.616597] ? getname_flags+0x6e/0x330 [10336.618941] ? __check_object_size+0xff/0x256 [10336.621041] ? do_raw_spin_unlock+0x75/0x190 [10336.623950] ? _raw_spin_unlock+0x12/0x30 [10336.625797] do_sys_openat2+0x2b4/0x410 [10336.627877] do_sys_open+0x73/0xa0 [10336.629134] __x64_sys_openat+0x24/0x30 [10336.631108] do_syscall_64+0xc1/0x440 [10336.636804] entry_SYSCALL_64_after_hwframe+0x49/0xae [10336.641514] RIP: 0033:0x7f53425a5332 [10336.644095] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10336.657997] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10336.662106] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10336.670266] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10336.675981] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10336.685832] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10336.688275] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10344.052293] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [10344.057495] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10344.085002] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10344.091097] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10344.095743] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10344.099053] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10344.101853] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10344.114387] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10344.118996] RAX: 0000000000003e03 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10344.123539] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10344.126723] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10344.130731] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10344.135680] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10344.142991] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10344.150658] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10344.153146] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10344.157863] Call Trace: [10344.159963] [10344.161030] ? show_regs.cold.9+0x22/0x2f [10344.163275] ? watchdog_timer_fn.cold.6+0x62/0x10a [10344.165374] ? __hrtimer_run_queues+0x1c2/0x4d0 [10344.167563] ? watchdog+0x40/0x40 [10344.169107] ? hrtimer_interrupt+0x128/0x2c0 [10344.171524] ? smp_apic_timer_interrupt+0xa6/0x270 [10344.173864] ? apic_timer_interrupt+0xf/0x20 [10344.177023] [10344.178166] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10344.187536] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10344.192373] ? newidle_balance.isra.93+0x207/0x580 [10344.195127] do_raw_spin_lock+0x177/0x190 [10344.197736] _raw_spin_lock+0x21/0x30 [10344.199028] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10344.201821] process_one_work+0x2c8/0x700 [10344.203355] worker_thread+0x296/0x6e0 [10344.205074] ? rescuer_thread+0x570/0x570 [10344.206706] kthread+0x1d1/0x200 [10344.208117] ? set_kthread_struct+0x70/0x70 [10344.210009] ret_from_fork+0x1f/0x30 [10349.539321] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7712s! [10349.546399] Showing busy workqueues and worker pools: [10349.551050] workqueue events: flags=0x0 [10349.555117] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10349.573942] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10349.595726] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10349.611410] pending: cpu_latency_work [ptlrpc] [10349.618113] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10349.628999] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10349.646414] workqueue events_long: flags=0x0 [10349.656892] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10349.672174] pending: lu_objects_flush [obdclass] [10349.680602] workqueue events_unbound: flags=0x2 [10349.688196] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10349.693172] pending: flush_memcg_stats_dwork, idle_cull_fn [10349.702589] workqueue events_power_efficient: flags=0x80 [10349.709884] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10349.719129] pending: check_lifetime, do_cache_clean [10349.726800] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10349.737793] pending: neigh_periodic_work, neigh_periodic_work [10349.748059] workqueue mm_percpu_wq: flags=0x8 [10349.750098] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10349.763952] pending: vmstat_update, lru_add_drain_per_cpu [10349.771127] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10349.786757] pending: vmstat_update, lru_add_drain_per_cpu [10349.799019] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10349.812066] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10349.816650] workqueue xprtiod: flags=0xa [10349.823235] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10349.833480] pending: xprt_autoclose, xs_error_handle [10349.835650] workqueue ipv6_addrconf: flags=0x40008 [10349.845183] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10349.854555] pending: addrconf_verify_work [10349.864065] workqueue knbd0-recv: flags=0x1a [10349.879989] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10349.881999] in-flight: 651:recv_work [10349.901551] workqueue ptlrpc_pinger: flags=0x2 [10349.909594] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10349.935811] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10349.938135] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7712s workers=4 idle: 15979 31824 8 [10349.959528] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10340s workers=2 idle: 654 [10352.064475] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [10352.071608] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10352.096318] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10352.102814] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10352.106568] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10352.110430] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10352.118474] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10352.122242] RAX: 0000000000007e94 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10352.125977] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10352.130161] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10352.133062] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10352.138139] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10352.141221] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10352.145244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10352.148452] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10352.153849] Call Trace: [10352.155026] [10352.157751] ? show_regs.cold.9+0x22/0x2f [10352.161460] ? watchdog_timer_fn.cold.6+0x62/0x10a [10352.164971] ? __hrtimer_run_queues+0x1c2/0x4d0 [10352.167215] ? watchdog+0x40/0x40 [10352.168864] ? hrtimer_interrupt+0x128/0x2c0 [10352.171838] ? smp_apic_timer_interrupt+0xa6/0x270 [10352.173661] ? apic_timer_interrupt+0xf/0x20 [10352.180108] [10352.180970] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10352.186844] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10352.189091] do_raw_spin_lock+0x177/0x190 [10352.191274] _raw_spin_lock+0x21/0x30 [10352.192554] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10352.195992] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10352.200222] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10352.203691] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10352.206165] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10352.211453] ? woken_wake_function+0x30/0x30 [10352.213075] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10352.217972] kthread+0x1d1/0x200 [10352.219699] ? set_kthread_struct+0x70/0x70 [10352.224642] ret_from_fork+0x1f/0x30 [10364.055135] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [10364.062698] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10364.062969] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [10364.087362] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10364.087378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10364.087381] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10364.087397] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10364.087401] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10364.087408] RAX: 0000000000007e2e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10364.087410] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10364.087413] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10364.087415] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10364.087417] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10364.087421] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10364.087424] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10364.087426] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10364.087435] Call Trace: [10364.087441] [10364.087446] ? show_regs.cold.9+0x22/0x2f [10364.087457] ? watchdog_timer_fn.cold.6+0x62/0x10a [10364.087463] ? __hrtimer_run_queues+0x1c2/0x4d0 [10364.087467] ? watchdog+0x40/0x40 [10364.087471] ? hrtimer_interrupt+0x128/0x2c0 [10364.087475] ? smp_apic_timer_interrupt+0xa6/0x270 [10364.087485] ? apic_timer_interrupt+0xf/0x20 [10364.087489] [10364.087493] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10364.087515] do_raw_spin_lock+0x177/0x190 [10364.087522] _raw_spin_lock+0x21/0x30 [10364.087527] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10364.087803] ? do_raw_spin_unlock+0x75/0x190 [10364.087809] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10364.088001] ? do_wait_intr+0xf0/0xf0 [10364.088011] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10364.088510] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10364.088793] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10364.089008] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10364.089335] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10364.089407] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10364.092817] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10364.093238] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10364.093246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10364.093249] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10364.093263] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10364.093268] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10364.093272] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10364.093275] RDX: 0000000000003d5a RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10364.093277] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10364.093279] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10364.093282] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10364.093285] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10364.093287] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10364.093290] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10364.093300] Call Trace: [10364.093305] [10364.093311] ? show_regs.cold.9+0x22/0x2f [10364.093321] ? watchdog_timer_fn.cold.6+0x62/0x10a [10364.093327] ? __hrtimer_run_queues+0x1c2/0x4d0 [10364.093331] ? watchdog+0x40/0x40 [10364.093335] ? hrtimer_interrupt+0x128/0x2c0 [10364.093339] ? smp_apic_timer_interrupt+0xa6/0x270 [10364.093347] ? apic_timer_interrupt+0xf/0x20 [10364.093352] [10364.093354] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10364.093359] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10364.093364] do_raw_spin_lock+0x177/0x190 [10364.093369] _raw_spin_lock+0x21/0x30 [10364.093375] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10364.093731] obd_mod_free+0x46/0x140 [ptlrpc] [10364.093937] mdc_commit_open+0x91/0xb0 [mdc] [10364.093975] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10364.094409] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10364.099892] mdc_intent_lock+0x353/0x7e0 [mdc] [10364.099950] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10364.100095] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10364.100642] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10364.100675] lmv_intent_open+0x338/0x1150 [lmv] [10364.100703] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10364.100799] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10364.100888] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10364.100911] ll_intent_lock+0xc5/0xaa0 [lustre] [10364.101560] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10364.113788] after_reply+0xe2c/0x1780 [ptlrpc] [10364.114220] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10364.115194] ? __next_timer_interrupt+0x160/0x160 [10364.115208] ptlrpcd+0xa82/0xe40 [ptlrpc] [10364.115691] ? do_wait_intr+0xf0/0xf0 [10364.115703] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10364.119895] ll_lookup_it+0xe0e/0x2270 [lustre] [10364.120024] ? do_raw_spin_unlock+0x75/0x190 [10364.120033] ? _raw_spin_unlock+0x12/0x30 [10364.120040] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10364.120081] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10364.120104] ll_atomic_open+0x36a/0xd70 [lustre] [10364.120403] lookup_open+0x356/0xb40 [10364.120412] do_last+0x313/0xfc0 [10364.120417] ? nd_jump_root+0xe5/0x160 [10364.120422] ? path_init+0x437/0x520 [10364.120427] path_openat+0xf7/0x500 [10364.120433] ? do_raw_spin_unlock+0x75/0x190 [10364.120438] do_filp_open+0x99/0x140 [10364.120445] ? getname_flags+0x6e/0x330 [10364.120450] ? __check_object_size+0xff/0x256 [10364.120456] ? do_raw_spin_unlock+0x75/0x190 [10364.120461] ? _raw_spin_unlock+0x12/0x30 [10364.120465] do_sys_openat2+0x2b4/0x410 [10364.120469] do_sys_open+0x73/0xa0 [10364.120474] __x64_sys_openat+0x24/0x30 [10364.120478] do_syscall_64+0xc1/0x440 [10364.120484] entry_SYSCALL_64_after_hwframe+0x49/0xae [10364.120491] RIP: 0033:0x7f53425a5332 [10364.120499] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10364.120503] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10364.120508] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10364.120511] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10364.120513] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10364.120515] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10364.120518] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10364.703128] kthread+0x1d1/0x200 [10364.704792] ? set_kthread_struct+0x70/0x70 [10364.707634] ret_from_fork+0x1f/0x30 [10376.053343] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [10376.059552] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10376.109162] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10376.124154] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10376.134443] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10376.140141] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10376.142591] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10376.154420] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10376.162288] RAX: 0000000000006b0b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10376.170781] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10376.173750] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10376.176696] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10376.189663] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10376.191828] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10376.201329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10376.210329] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10376.213107] Call Trace: [10376.214133] [10376.214845] ? show_regs.cold.9+0x22/0x2f [10376.216177] ? watchdog_timer_fn.cold.6+0x62/0x10a [10376.218157] ? __hrtimer_run_queues+0x1c2/0x4d0 [10376.219787] ? watchdog+0x40/0x40 [10376.221218] ? hrtimer_interrupt+0x128/0x2c0 [10376.224719] ? smp_apic_timer_interrupt+0xa6/0x270 [10376.227063] ? apic_timer_interrupt+0xf/0x20 [10376.228721] [10376.229600] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10376.231681] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10376.233840] ? newidle_balance.isra.93+0x207/0x580 [10376.239125] do_raw_spin_lock+0x177/0x190 [10376.240541] _raw_spin_lock+0x21/0x30 [10376.241731] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10376.244773] process_one_work+0x2c8/0x700 [10376.246544] worker_thread+0x296/0x6e0 [10376.247782] ? rescuer_thread+0x570/0x570 [10376.249372] kthread+0x1d1/0x200 [10376.250573] ? set_kthread_struct+0x70/0x70 [10376.256548] ret_from_fork+0x1f/0x30 [10380.066412] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [10380.074696] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10380.132413] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10380.149062] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10380.159471] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10380.166528] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10380.183160] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10380.193387] RAX: 0000000000007d8e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10380.200916] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10380.204959] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10380.207784] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10380.209679] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10380.213399] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10380.216062] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10380.218954] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10380.222732] Call Trace: [10380.224469] [10380.229528] ? show_regs.cold.9+0x22/0x2f [10380.232324] ? watchdog_timer_fn.cold.6+0x62/0x10a [10380.236438] ? __hrtimer_run_queues+0x1c2/0x4d0 [10380.240957] ? watchdog+0x40/0x40 [10380.242568] ? hrtimer_interrupt+0x128/0x2c0 [10380.243895] ? smp_apic_timer_interrupt+0xa6/0x270 [10380.245071] ? apic_timer_interrupt+0xf/0x20 [10380.246167] [10380.247662] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10380.249294] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10380.251545] do_raw_spin_lock+0x177/0x190 [10380.253141] _raw_spin_lock+0x21/0x30 [10380.255161] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10380.261107] BUG: workqueue lockup - pool [10380.263500] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10380.264067] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10380.264293] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10380.264511] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10380.264727] ? woken_wake_function+0x30/0x30 [10380.264737] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10380.265149] kthread+0x1d1/0x200 [10380.265156] ? set_kthread_struct+0x70/0x70 [10380.265160] ret_from_fork+0x1f/0x30 [10380.311259] cpus=0-3 flags=0x4 nice=0 stuck for 7743s! [10380.314799] Showing busy workqueues and worker pools: [10380.316752] workqueue events: flags=0x0 [10380.318494] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10380.323278] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10380.332278] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10380.338209] pending: cpu_latency_work [ptlrpc] [10380.350141] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10380.352251] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10380.361460] workqueue events_long: flags=0x0 [10380.364682] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10380.371990] pending: lu_objects_flush [obdclass] [10380.374940] workqueue events_unbound: flags=0x2 [10380.376923] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10380.380139] pending: flush_memcg_stats_dwork, idle_cull_fn [10380.382988] workqueue events_power_efficient: flags=0x80 [10380.385389] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10380.394603] pending: check_lifetime, do_cache_clean [10380.396626] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10380.404020] pending: neigh_periodic_work, neigh_periodic_work [10380.411032] workqueue mm_percpu_wq: flags=0x8 [10380.420634] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10380.423868] pending: vmstat_update, lru_add_drain_per_cpu [10380.434430] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10380.441739] pending: vmstat_update, lru_add_drain_per_cpu [10380.446686] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10380.453443] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10380.458202] workqueue xprtiod: flags=0xa [10380.463524] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10380.468390] pending: xprt_autoclose, xs_error_handle [10380.486189] workqueue ipv6_addrconf: flags=0x40008 [10380.488954] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10380.493617] pending: addrconf_verify_work [10380.495859] workqueue knbd0-recv: flags=0x1a [10380.497790] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10380.505476] in-flight: 651:recv_work [10380.506903] workqueue ptlrpc_pinger: flags=0x2 [10380.509404] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10380.518939] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10380.527542] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7743s workers=4 idle: 15979 31824 8 [10380.536457] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10370s workers=2 idle: 654 [10392.055047] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [10392.059055] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [10392.066297] Modules linked in: [10392.070143] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10392.070368] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10392.070376] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10392.070379] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10392.070392] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10392.070396] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10392.070401] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10392.070403] RDX: 0000000000002d4f RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10392.070405] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10392.070407] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10392.070409] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10392.070413] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10392.070415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10392.070418] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10392.070427] Call Trace: [10392.070433] [10392.070438] ? show_regs.cold.9+0x22/0x2f [10392.070447] ? watchdog_timer_fn.cold.6+0x62/0x10a [10392.070454] ? __hrtimer_run_queues+0x1c2/0x4d0 [10392.070459] ? watchdog+0x40/0x40 [10392.070463] ? hrtimer_interrupt+0x128/0x2c0 [10392.070466] ? smp_apic_timer_interrupt+0xa6/0x270 [10392.070473] ? apic_timer_interrupt+0xf/0x20 [10392.070478] [10392.070481] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10392.070486] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10392.070491] do_raw_spin_lock+0x177/0x190 [10392.070495] _raw_spin_lock+0x21/0x30 [10392.070820] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10392.071065] obd_mod_free+0x46/0x140 [ptlrpc] [10392.077277] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10392.077456] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10392.077463] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10392.077466] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [10392.077479] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [10392.077483] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10392.077488] RAX: 00000000000050bb RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10392.077490] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10392.077493] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10392.077495] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10392.077497] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10392.077516] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10392.077519] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10392.077522] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10392.077535] Call Trace: [10392.077541] [10392.077547] ? show_regs.cold.9+0x22/0x2f [10392.077567] ? watchdog_timer_fn.cold.6+0x62/0x10a [10392.077574] ? __hrtimer_run_queues+0x1c2/0x4d0 [10392.077578] ? watchdog+0x40/0x40 [10392.077588] ? hrtimer_interrupt+0x128/0x2c0 [10392.077592] ? smp_apic_timer_interrupt+0xa6/0x270 [10392.077600] ? apic_timer_interrupt+0xf/0x20 [10392.077609] [10392.077613] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [10392.077619] do_raw_spin_lock+0x177/0x190 [10392.077627] _raw_spin_lock+0x21/0x30 [10392.077632] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10392.078648] ? do_raw_spin_unlock+0x75/0x190 [10392.078658] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10392.107974] mdc_commit_open+0x91/0xb0 [mdc] [10392.108018] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10392.108464] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10392.110827] after_reply+0xe2c/0x1780 [ptlrpc] [10392.111041] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10392.121546] ? do_wait_intr+0xf0/0xf0 [10392.121771] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10392.123348] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10392.124477] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10392.124974] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10392.125354] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10392.125391] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10392.125423] mdc_intent_lock+0x353/0x7e0 [mdc] [10392.125452] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10392.126219] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10392.126945] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10392.126981] lmv_intent_open+0x338/0x1150 [lmv] [10392.127013] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10392.127347] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10392.127432] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10392.127454] ll_intent_lock+0xc5/0xaa0 [lustre] [10392.128693] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10392.128791] ll_lookup_it+0xe0e/0x2270 [lustre] [10392.128883] ? do_raw_spin_unlock+0x75/0x190 [10392.128891] ? _raw_spin_unlock+0x12/0x30 [10392.128899] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10392.128928] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10392.128949] ll_atomic_open+0x36a/0xd70 [lustre] [10392.132881] ? __next_timer_interrupt+0x160/0x160 [10392.132901] ptlrpcd+0xa82/0xe40 [ptlrpc] [10392.133905] ? do_wait_intr+0xf0/0xf0 [10392.133915] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10392.134875] kthread+0x1d1/0x200 [10392.134883] ? set_kthread_struct+0x70/0x70 [10392.134887] ret_from_fork+0x1f/0x30 [10392.460265] lookup_open+0x356/0xb40 [10392.461734] do_last+0x313/0xfc0 [10392.462860] ? nd_jump_root+0xe5/0x160 [10392.464124] ? path_init+0x437/0x520 [10392.465006] path_openat+0xf7/0x500 [10392.465845] ? do_raw_spin_unlock+0x75/0x190 [10392.467861] do_filp_open+0x99/0x140 [10392.469790] ? getname_flags+0x6e/0x330 [10392.472949] ? __check_object_size+0xff/0x256 [10392.475356] ? do_raw_spin_unlock+0x75/0x190 [10392.477884] ? _raw_spin_unlock+0x12/0x30 [10392.481333] do_sys_openat2+0x2b4/0x410 [10392.482829] do_sys_open+0x73/0xa0 [10392.484244] __x64_sys_openat+0x24/0x30 [10392.486832] do_syscall_64+0xc1/0x440 [10392.489062] entry_SYSCALL_64_after_hwframe+0x49/0xae [10392.490457] RIP: 0033:0x7f53425a5332 [10392.495958] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10392.506475] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10392.509787] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10392.513008] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10392.516793] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10392.525047] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10392.528877] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10408.052357] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [10408.055436] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10408.058063] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [10408.092534] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10408.092547] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10408.092553] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10408.101912] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10408.102265] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10408.102286] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10408.102288] RIP: 0010:native_safe_halt+0x2e/0x50 [10408.102302] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10408.102307] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [10408.102312] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10408.102315] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10408.102317] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10408.102319] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10408.102321] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10408.102325] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10408.102328] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10408.102330] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10408.102719] Call Trace: [10408.102847] [10408.102853] ? show_regs.cold.9+0x22/0x2f [10408.102867] ? watchdog_timer_fn.cold.6+0x62/0x10a [10408.102873] ? __hrtimer_run_queues+0x1c2/0x4d0 [10408.102876] ? watchdog+0x40/0x40 [10408.102881] ? hrtimer_interrupt+0x128/0x2c0 [10408.102885] ? smp_apic_timer_interrupt+0xa6/0x270 [10408.102892] ? apic_timer_interrupt+0xf/0x20 [10408.102897] [10408.102900] ? native_safe_halt+0x2e/0x50 [10408.102904] kvm_wait+0xcd/0xe0 [10408.102912] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10408.102921] do_raw_spin_lock+0x177/0x190 [10408.102926] _raw_spin_lock+0x21/0x30 [10408.102930] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10408.103248] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10408.103636] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10408.103922] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10408.104206] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10408.104659] ? woken_wake_function+0x30/0x30 [10408.104667] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10408.105067] kthread+0x1d1/0x200 [10408.105074] ? set_kthread_struct+0x70/0x70 [10408.105077] ret_from_fork+0x1f/0x30 [10408.309165] [10408.309967] RIP: 0010:native_safe_halt+0x2e/0x50 [10408.314503] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10408.324242] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10408.330249] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10408.337765] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [10408.341560] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10408.351181] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10408.360242] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10408.366350] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10408.371710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10408.381033] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10408.388247] Call Trace: [10408.389132] [10408.390163] ? show_regs.cold.9+0x22/0x2f [10408.393883] ? watchdog_timer_fn.cold.6+0x62/0x10a [10408.397978] ? __hrtimer_run_queues+0x1c2/0x4d0 [10408.400527] ? watchdog+0x40/0x40 [10408.402235] ? hrtimer_interrupt+0x128/0x2c0 [10408.405205] ? smp_apic_timer_interrupt+0xa6/0x270 [10408.408049] ? apic_timer_interrupt+0xf/0x20 [10408.411336] [10408.413089] ? native_safe_halt+0x2e/0x50 [10408.416988] kvm_wait+0xcd/0xe0 [10408.419376] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10408.423570] ? newidle_balance.isra.93+0x207/0x580 [10408.426932] do_raw_spin_lock+0x177/0x190 [10408.429130] _raw_spin_lock+0x21/0x30 [10408.431177] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10408.436891] process_one_work+0x2c8/0x700 [10408.439406] worker_thread+0x296/0x6e0 [10408.444190] ? rescuer_thread+0x570/0x570 [10408.447085] kthread+0x1d1/0x200 [10408.450199] ? set_kthread_struct+0x70/0x70 [10408.452528] ret_from_fork+0x1f/0x30 [10410.979341] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7773s! [10410.989754] Showing busy workqueues and worker pools: [10410.992988] workqueue events: flags=0x0 [10411.001870] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10411.015141] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10411.042594] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10411.054046] pending: cpu_latency_work [ptlrpc] [10411.055728] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10411.076804] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10411.093444] workqueue events_long: flags=0x0 [10411.099661] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10411.107260] pending: lu_objects_flush [obdclass] [10411.115414] workqueue events_unbound: flags=0x2 [10411.118559] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10411.129771] pending: flush_memcg_stats_dwork, idle_cull_fn [10411.146097] workqueue events_power_efficient: flags=0x80 [10411.156555] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10411.162106] pending: check_lifetime, do_cache_clean [10411.164464] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10411.167841] pending: neigh_periodic_work, neigh_periodic_work [10411.171320] workqueue mm_percpu_wq: flags=0x8 [10411.173483] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10411.180708] pending: vmstat_update, lru_add_drain_per_cpu [10411.183713] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10411.192379] pending: vmstat_update, lru_add_drain_per_cpu [10411.200687] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10411.211387] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10411.225844] workqueue xprtiod: flags=0xa [10411.226934] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10411.231907] pending: xprt_autoclose, xs_error_handle [10411.233320] workqueue ipv6_addrconf: flags=0x40008 [10411.241750] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10411.252278] pending: addrconf_verify_work [10411.253437] workqueue knbd0-recv: flags=0x1a [10411.254512] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10411.265669] in-flight: 651:recv_work [10411.271353] workqueue ptlrpc_pinger: flags=0x2 [10411.273101] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10411.276185] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10411.283326] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7774s workers=4 idle: 15979 31824 8 [10411.287037] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10401s workers=2 idle: 654 [10420.055645] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [10420.057910] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [10420.062337] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10420.062445] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10420.062462] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10420.062465] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10420.062480] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10420.062485] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10420.062490] RAX: 000000000000013a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10420.062493] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10420.062495] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10420.062498] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10420.062501] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10420.062504] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10420.062507] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10420.062510] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10420.062519] Call Trace: [10420.062525] [10420.062531] ? show_regs.cold.9+0x22/0x2f [10420.062541] ? watchdog_timer_fn.cold.6+0x62/0x10a [10420.062547] ? __hrtimer_run_queues+0x1c2/0x4d0 [10420.062551] ? watchdog+0x40/0x40 [10420.062556] ? hrtimer_interrupt+0x128/0x2c0 [10420.062560] ? smp_apic_timer_interrupt+0xa6/0x270 [10420.062570] ? apic_timer_interrupt+0xf/0x20 [10420.062575] [10420.062577] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10420.062584] do_raw_spin_lock+0x177/0x190 [10420.062589] _raw_spin_lock+0x21/0x30 [10420.062594] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10420.062896] ? do_raw_spin_unlock+0x75/0x190 [10420.062902] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10420.063984] ? do_wait_intr+0xf0/0xf0 [10420.063999] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10420.064395] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10420.064669] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10420.064889] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10420.065001] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10420.065546] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10420.065576] mdc_intent_lock+0x353/0x7e0 [mdc] [10420.065605] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10420.065694] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10420.065924] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10420.065954] lmv_intent_open+0x338/0x1150 [lmv] [10420.065985] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10420.066143] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10420.066380] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10420.066404] ll_intent_lock+0xc5/0xaa0 [lustre] [10420.066553] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10420.066753] ll_lookup_it+0xe0e/0x2270 [lustre] [10420.066838] ? do_raw_spin_unlock+0x75/0x190 [10420.066846] ? _raw_spin_unlock+0x12/0x30 [10420.066852] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10420.066880] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10420.066900] ll_atomic_open+0x36a/0xd70 [lustre] [10420.066989] lookup_open+0x356/0xb40 [10420.066999] do_last+0x313/0xfc0 [10420.067004] ? nd_jump_root+0xe5/0x160 [10420.067009] ? path_init+0x437/0x520 [10420.067013] path_openat+0xf7/0x500 [10420.067018] ? do_raw_spin_unlock+0x75/0x190 [10420.067022] do_filp_open+0x99/0x140 [10420.067029] ? getname_flags+0x6e/0x330 [10420.067035] ? __check_object_size+0xff/0x256 [10420.067043] ? do_raw_spin_unlock+0x75/0x190 [10420.067047] ? _raw_spin_unlock+0x12/0x30 [10420.067052] do_sys_openat2+0x2b4/0x410 [10420.067057] do_sys_open+0x73/0xa0 [10420.072078] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10420.072179] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10420.072190] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10420.072192] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10420.072211] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10420.072218] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10420.072223] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10420.072225] RDX: 0000000000003a4a RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10420.072230] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10420.072232] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10420.072235] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10420.072238] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10420.072244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10420.072247] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10420.072262] Call Trace: [10420.072267] [10420.072285] ? show_regs.cold.9+0x22/0x2f [10420.072294] ? watchdog_timer_fn.cold.6+0x62/0x10a [10420.072300] ? __hrtimer_run_queues+0x1c2/0x4d0 [10420.072305] ? watchdog+0x40/0x40 [10420.072309] ? hrtimer_interrupt+0x128/0x2c0 [10420.072313] ? smp_apic_timer_interrupt+0xa6/0x270 [10420.072321] ? apic_timer_interrupt+0xf/0x20 [10420.072325] [10420.072328] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10420.072333] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10420.072337] do_raw_spin_lock+0x177/0x190 [10420.072347] _raw_spin_lock+0x21/0x30 [10420.072356] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10420.072758] obd_mod_free+0x46/0x140 [ptlrpc] [10420.073126] mdc_commit_open+0x91/0xb0 [mdc] [10420.073176] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10420.073474] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10420.073976] after_reply+0xe2c/0x1780 [ptlrpc] [10420.074513] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10420.100058] __x64_sys_openat+0x24/0x30 [10420.100086] do_syscall_64+0xc1/0x440 [10420.100094] entry_SYSCALL_64_after_hwframe+0x49/0xae [10420.100103] RIP: 0033:0x7f53425a5332 [10420.100115] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10420.100120] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10420.100129] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10420.100131] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10420.100133] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10420.100135] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10420.100137] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10420.415614] ? __next_timer_interrupt+0x160/0x160 [10420.417197] ptlrpcd+0xa82/0xe40 [ptlrpc] [10420.419075] ? do_wait_intr+0xf0/0xf0 [10420.420571] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10420.423312] kthread+0x1d1/0x200 [10420.424504] ? set_kthread_struct+0x70/0x70 [10420.426192] ret_from_fork+0x1f/0x30 [10434.626082] rcu: INFO: rcu_sched self-detected stall on CPU [10434.632797] rcu: 2-....: (4570367 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [10434.649838] (t=8340200 jiffies g=274185 q=65924) [10434.651195] rcu: rcu_sched kthread starved for 7798566 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [10434.662626] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10434.675436] rcu: RCU grace-period kthread stack dump: [10434.685352] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10434.689238] Call Trace: [10434.690117] __schedule+0x351/0xcb0 [10434.691415] schedule+0xc0/0x180 [10434.694041] schedule_timeout+0xb4/0x190 [10434.697589] ? do_raw_spin_unlock+0x75/0x190 [10434.699766] ? __next_timer_interrupt+0x160/0x160 [10434.710236] ? prepare_to_swait_event+0xf3/0x140 [10434.716117] rcu_gp_kthread+0x8dd/0x15b0 [10434.722112] ? force_qs_rnp+0x2c0/0x2c0 [10434.724794] kthread+0x1d1/0x200 [10434.729132] ? set_kthread_struct+0x70/0x70 [10434.734418] ret_from_fork+0x1f/0x30 [10434.738118] rcu: Stack dump where RCU GP kthread last ran: [10434.742539] Sending NMI from CPU 2 to CPUs 1: [10434.748199] NMI backtrace for cpu 1 [10434.748207] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10434.748210] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10434.748212] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10434.748213] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10434.748216] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [10434.748219] RAX: 0000000000007c20 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10434.748220] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10434.748222] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10434.748223] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10434.748224] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10434.748226] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10434.748228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10434.748229] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10434.748230] Call Trace: [10434.748231] [10434.748232] ? show_regs.cold.9+0x22/0x2f [10434.748233] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10434.748234] ? nmi_cpu_backtrace_handler+0x15/0x40 [10434.748235] ? nmi_handle+0xbd/0x210 [10434.748237] ? default_do_nmi+0x4b/0x1b0 [10434.748238] ? do_nmi+0x280/0x380 [10434.748239] ? end_repeat_nmi+0x16/0x1f [10434.748240] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10434.748241] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10434.748243] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10434.748244] [10434.748245] do_raw_spin_lock+0x177/0x190 [10434.748246] _raw_spin_lock+0x21/0x30 [10434.748247] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10434.748248] ? do_raw_spin_unlock+0x75/0x190 [10434.748249] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10434.748250] ? do_wait_intr+0xf0/0xf0 [10434.748251] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10434.748252] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10434.748253] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10434.748255] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10434.748256] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10434.748257] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10434.748259] mdc_intent_lock+0x353/0x7e0 [mdc] [10434.748260] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10434.748261] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10434.748262] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10434.748263] lmv_intent_open+0x338/0x1150 [lmv] [10434.748264] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10434.748265] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10434.748266] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10434.748268] ll_intent_lock+0xc5/0xaa0 [lustre] [10434.748269] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10434.748270] ll_lookup_it+0xe0e/0x2270 [lustre] [10434.748271] ? do_raw_spin_unlock+0x75/0x190 [10434.748272] ? _raw_spin_unlock+0x12/0x30 [10434.748273] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10434.748275] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10434.748276] ll_atomic_open+0x36a/0xd70 [lustre] [10434.748277] lookup_open+0x356/0xb40 [10434.748278] do_last+0x313/0xfc0 [10434.748279] ? nd_jump_root+0xe5/0x160 [10434.748280] ? path_init+0x437/0x520 [10434.748281] path_openat+0xf7/0x500 [10434.748282] ? do_raw_spin_unlock+0x75/0x190 [10434.748283] do_filp_open+0x99/0x140 [10434.748284] ? getname_flags+0x6e/0x330 [10434.748285] ? __check_object_size+0xff/0x256 [10434.748286] ? do_raw_spin_unlock+0x75/0x190 [10434.748288] ? _raw_spin_unlock+0x12/0x30 [10434.748289] do_sys_openat2+0x2b4/0x410 [10434.748290] do_sys_open+0x73/0xa0 [10434.748291] __x64_sys_openat+0x24/0x30 [10434.748293] do_syscall_64+0xc1/0x440 [10434.748294] entry_SYSCALL_64_after_hwframe+0x49/0xae [10434.748295] RIP: 0033:0x7f53425a5332 [10434.748297] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10434.748299] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10434.748301] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10434.748303] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10434.748304] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10434.748306] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10434.748307] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10434.751527] Sending NMI from CPU 2 to CPUs 1: [10435.042708] NMI backtrace for cpu 1 [10435.042715] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10435.042717] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10435.042719] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10435.042720] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10435.042723] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [10435.042725] RAX: 0000000000007f86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10435.042727] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10435.042728] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10435.042729] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10435.042730] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10435.042732] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10435.042733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10435.042734] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10435.042735] Call Trace: [10435.042736] [10435.042737] ? show_regs.cold.9+0x22/0x2f [10435.042738] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10435.042739] ? nmi_cpu_backtrace_handler+0x15/0x40 [10435.042740] ? nmi_handle+0xbd/0x210 [10435.042741] ? default_do_nmi+0x4b/0x1b0 [10435.042742] ? do_nmi+0x280/0x380 [10435.042743] ? end_repeat_nmi+0x16/0x1f [10435.042744] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10435.042745] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10435.042746] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10435.042747] [10435.042748] do_raw_spin_lock+0x177/0x190 [10435.042749] _raw_spin_lock+0x21/0x30 [10435.042750] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10435.042752] ? do_raw_spin_unlock+0x75/0x190 [10435.042753] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10435.042754] ? do_wait_intr+0xf0/0xf0 [10435.042755] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10435.042757] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10435.042758] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10435.042759] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10435.042760] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10435.042761] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10435.042762] mdc_intent_lock+0x353/0x7e0 [mdc] [10435.042764] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10435.042766] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10435.042767] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10435.042768] lmv_intent_open+0x338/0x1150 [lmv] [10435.042770] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10435.042771] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10435.042772] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10435.042773] ll_intent_lock+0xc5/0xaa0 [lustre] [10435.042775] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10435.042776] ll_lookup_it+0xe0e/0x2270 [lustre] [10435.042777] ? do_raw_spin_unlock+0x75/0x190 [10435.042778] ? _raw_spin_unlock+0x12/0x30 [10435.042779] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10435.042780] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10435.042781] ll_atomic_open+0x36a/0xd70 [lustre] [10435.042782] lookup_open+0x356/0xb40 [10435.042783] do_last+0x313/0xfc0 [10435.042784] ? nd_jump_root+0xe5/0x160 [10435.042785] ? path_init+0x437/0x520 [10435.042787] path_openat+0xf7/0x500 [10435.042788] ? do_raw_spin_unlock+0x75/0x190 [10435.042789] do_filp_open+0x99/0x140 [10435.042790] ? getname_flags+0x6e/0x330 [10435.042791] ? __check_object_size+0xff/0x256 [10435.042793] ? do_raw_spin_unlock+0x75/0x190 [10435.042794] ? _raw_spin_unlock+0x12/0x30 [10435.042795] do_sys_openat2+0x2b4/0x410 [10435.042796] do_sys_open+0x73/0xa0 [10435.042797] __x64_sys_openat+0x24/0x30 [10435.042798] do_syscall_64+0xc1/0x440 [10435.042810] entry_SYSCALL_64_after_hwframe+0x49/0xae [10435.042811] RIP: 0033:0x7f53425a5332 [10435.042812] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10435.042815] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10435.042817] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10435.042819] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10435.042820] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10435.042822] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10435.042823] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10435.045518] NMI backtrace for cpu 2 [10435.491677] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10435.500903] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10435.513307] Call Trace: [10435.514116] [10435.518662] ? dump_stack+0xbb/0x10e [10435.524279] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10435.529455] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10435.531061] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10435.539480] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10435.546579] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10435.554661] ? rcu_dump_cpu_stacks+0x11b/0x171 [10435.560435] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10435.564515] ? update_process_times+0x6d/0xc0 [10435.575062] ? tick_sched_handle.isra.18+0x27/0x90 [10435.583349] ? tick_sched_timer+0x53/0xb0 [10435.586773] ? __hrtimer_run_queues+0x1c2/0x4d0 [10435.589185] ? tick_sched_handle.isra.18+0x90/0x90 [10435.592430] ? hrtimer_interrupt+0x128/0x2c0 [10435.596582] ? smp_apic_timer_interrupt+0xa6/0x270 [10435.598809] ? apic_timer_interrupt+0xf/0x20 [10435.600067] [10435.600657] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10435.607362] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10435.613165] ? do_raw_spin_lock+0x177/0x190 [10435.616555] ? _raw_spin_lock+0x21/0x30 [10435.620157] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10435.629601] ? obd_mod_free+0x46/0x140 [ptlrpc] [10435.636547] ? mdc_commit_open+0x91/0xb0 [mdc] [10435.648164] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10435.652727] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10435.660436] ? after_reply+0xe2c/0x1780 [ptlrpc] [10435.668670] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10435.671118] ? __next_timer_interrupt+0x160/0x160 [10435.673020] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10435.674762] ? do_wait_intr+0xf0/0xf0 [10435.676007] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10435.678011] ? kthread+0x1d1/0x200 [10435.679778] ? set_kthread_struct+0x70/0x70 [10435.681418] ? ret_from_fork+0x1f/0x30 [10436.053736] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [10436.059199] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [10436.061143] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10436.061218] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10436.061225] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10436.061232] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10436.061677] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10436.061688] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10436.061691] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10436.061696] RAX: 000000000000080b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10436.061698] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10436.061700] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10436.061702] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10436.061704] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10436.061707] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10436.061709] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10436.061711] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10436.061720] Call Trace: [10436.061725] [10436.061731] ? show_regs.cold.9+0x22/0x2f [10436.061740] ? watchdog_timer_fn.cold.6+0x62/0x10a [10436.061745] ? __hrtimer_run_queues+0x1c2/0x4d0 [10436.061749] ? watchdog+0x40/0x40 [10436.061753] ? hrtimer_interrupt+0x128/0x2c0 [10436.061756] ? smp_apic_timer_interrupt+0xa6/0x270 [10436.061765] ? apic_timer_interrupt+0xf/0x20 [10436.061770] [10436.061773] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10436.061778] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10436.061794] ? newidle_balance.isra.93+0x207/0x580 [10436.061800] do_raw_spin_lock+0x177/0x190 [10436.061805] _raw_spin_lock+0x21/0x30 [10436.061810] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10436.062170] process_one_work+0x2c8/0x700 [10436.062178] worker_thread+0x296/0x6e0 [10436.062182] ? rescuer_thread+0x570/0x570 [10436.062186] kthread+0x1d1/0x200 [10436.062192] ? set_kthread_struct+0x70/0x70 [10436.062195] ret_from_fork+0x1f/0x30 [10436.283758] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10436.306730] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10436.311302] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10436.315263] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10436.318274] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10436.326070] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10436.329150] RAX: 000000000000160d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10436.331954] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10436.334310] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10436.337018] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10436.339417] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10436.341985] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10436.345140] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10436.347287] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10436.349459] Call Trace: [10436.350158] [10436.350554] ? show_regs.cold.9+0x22/0x2f [10436.354075] ? watchdog_timer_fn.cold.6+0x62/0x10a [10436.356347] ? __hrtimer_run_queues+0x1c2/0x4d0 [10436.357469] ? watchdog+0x40/0x40 [10436.358241] ? hrtimer_interrupt+0x128/0x2c0 [10436.359715] ? smp_apic_timer_interrupt+0xa6/0x270 [10436.361482] ? apic_timer_interrupt+0xf/0x20 [10436.362989] [10436.363486] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10436.365260] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10436.367409] do_raw_spin_lock+0x177/0x190 [10436.369421] _raw_spin_lock+0x21/0x30 [10436.371058] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10436.373741] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10436.376551] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10436.378723] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10436.381060] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10436.383502] ? woken_wake_function+0x30/0x30 [10436.385184] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10436.387528] kthread+0x1d1/0x200 [10436.393847] ? set_kthread_struct+0x70/0x70 [10436.395558] ret_from_fork+0x1f/0x30 [10441.697033] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7804s! [10441.700318] Showing busy workqueues and worker pools: [10441.702147] workqueue events: flags=0x0 [10441.710393] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10441.715152] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10441.721408] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10441.724622] pending: cpu_latency_work [ptlrpc] [10441.729427] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10441.731829] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10441.742141] workqueue events_long: flags=0x0 [10441.747087] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10441.750488] pending: lu_objects_flush [obdclass] [10441.758361] workqueue events_unbound: flags=0x2 [10441.761362] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10441.764592] pending: flush_memcg_stats_dwork, idle_cull_fn [10441.774063] workqueue events_power_efficient: flags=0x80 [10441.776392] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10441.782440] pending: check_lifetime, do_cache_clean [10441.794120] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10441.802877] pending: neigh_periodic_work, neigh_periodic_work [10441.804563] workqueue mm_percpu_wq: flags=0x8 [10441.814750] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10441.819381] pending: vmstat_update, lru_add_drain_per_cpu [10441.831787] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10441.836464] pending: vmstat_update, lru_add_drain_per_cpu [10441.842110] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10441.851584] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10441.856814] workqueue xprtiod: flags=0xa [10441.858128] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10441.863936] pending: xprt_autoclose, xs_error_handle [10441.872861] workqueue ipv6_addrconf: flags=0x40008 [10441.874475] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10441.877122] pending: addrconf_verify_work [10441.878347] workqueue knbd0-recv: flags=0x1a [10441.879468] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10441.883166] in-flight: 651:recv_work [10441.886513] workqueue ptlrpc_pinger: flags=0x2 [10441.888666] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10441.892016] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10441.895298] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7804s workers=4 idle: 15979 31824 8 [10441.900210] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10432s workers=2 idle: 654 [10460.055344] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [10460.057241] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [10460.060072] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10460.060152] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10460.060159] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10460.060162] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10460.060177] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10460.060183] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10460.060188] RAX: 0000000000004899 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10460.060191] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10460.060193] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10460.060195] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10460.060198] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10460.060201] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10460.060203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10460.060205] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10460.060214] Call Trace: [10460.060219] [10460.060225] ? show_regs.cold.9+0x22/0x2f [10460.060234] ? watchdog_timer_fn.cold.6+0x62/0x10a [10460.060241] ? __hrtimer_run_queues+0x1c2/0x4d0 [10460.060245] ? watchdog+0x40/0x40 [10460.060250] ? hrtimer_interrupt+0x128/0x2c0 [10460.060253] ? smp_apic_timer_interrupt+0xa6/0x270 [10460.060261] ? apic_timer_interrupt+0xf/0x20 [10460.060265] [10460.060268] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10460.060275] do_raw_spin_lock+0x177/0x190 [10460.060280] _raw_spin_lock+0x21/0x30 [10460.060285] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10460.060782] ? do_raw_spin_unlock+0x75/0x190 [10460.060788] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10460.070864] Modules linked in: [10460.120000] ? do_wait_intr+0xf0/0xf0 [10460.120023] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10460.125792] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10460.125878] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10460.125886] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10460.125889] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10460.125903] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10460.125907] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10460.125912] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10460.125915] RDX: 0000000000007c7f RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10460.125917] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10460.125919] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10460.125921] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10460.125925] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10460.125928] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10460.125931] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10460.125939] Call Trace: [10460.125950] [10460.125956] ? show_regs.cold.9+0x22/0x2f [10460.125966] ? watchdog_timer_fn.cold.6+0x62/0x10a [10460.125975] ? __hrtimer_run_queues+0x1c2/0x4d0 [10460.133743] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10460.136689] ? watchdog+0x40/0x40 [10460.136768] ? hrtimer_interrupt+0x128/0x2c0 [10460.136775] ? smp_apic_timer_interrupt+0xa6/0x270 [10460.136782] ? apic_timer_interrupt+0xf/0x20 [10460.136788] [10460.136793] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10460.136862] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10460.136867] do_raw_spin_lock+0x177/0x190 [10460.136872] _raw_spin_lock+0x21/0x30 [10460.136880] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10460.137329] obd_mod_free+0x46/0x140 [ptlrpc] [10460.137526] mdc_commit_open+0x91/0xb0 [mdc] [10460.137576] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10460.137786] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10460.138221] after_reply+0xe2c/0x1780 [ptlrpc] [10460.138399] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10460.138592] ? __next_timer_interrupt+0x160/0x160 [10460.138602] ptlrpcd+0xa82/0xe40 [ptlrpc] [10460.138917] ? do_wait_intr+0xf0/0xf0 [10460.138925] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10460.139158] kthread+0x1d1/0x200 [10460.139163] ? set_kthread_struct+0x70/0x70 [10460.139167] ret_from_fork+0x1f/0x30 [10460.358460] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10460.360779] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10460.363398] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10460.365411] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10460.367537] mdc_intent_lock+0x353/0x7e0 [mdc] [10460.369422] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10460.372170] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10460.374937] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10460.377481] lmv_intent_open+0x338/0x1150 [lmv] [10460.379225] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10460.381745] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10460.384647] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10460.386758] ll_intent_lock+0xc5/0xaa0 [lustre] [10460.388919] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10460.391027] ll_lookup_it+0xe0e/0x2270 [lustre] [10460.393104] ? do_raw_spin_unlock+0x75/0x190 [10460.395142] ? _raw_spin_unlock+0x12/0x30 [10460.396740] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10460.399027] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10460.401151] ll_atomic_open+0x36a/0xd70 [lustre] [10460.403805] lookup_open+0x356/0xb40 [10460.405164] do_last+0x313/0xfc0 [10460.406347] ? nd_jump_root+0xe5/0x160 [10460.409482] ? path_init+0x437/0x520 [10460.411225] path_openat+0xf7/0x500 [10460.412606] ? do_raw_spin_unlock+0x75/0x190 [10460.414109] do_filp_open+0x99/0x140 [10460.415334] ? getname_flags+0x6e/0x330 [10460.416784] ? __check_object_size+0xff/0x256 [10460.418321] ? do_raw_spin_unlock+0x75/0x190 [10460.419964] ? _raw_spin_unlock+0x12/0x30 [10460.421331] do_sys_openat2+0x2b4/0x410 [10460.422829] do_sys_open+0x73/0xa0 [10460.424128] __x64_sys_openat+0x24/0x30 [10460.425501] do_syscall_64+0xc1/0x440 [10460.427183] entry_SYSCALL_64_after_hwframe+0x49/0xae [10460.429466] RIP: 0033:0x7f53425a5332 [10460.431062] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10460.437516] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10460.439286] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10460.443857] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10460.446894] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10460.450504] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10460.453191] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10464.058029] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [10464.061653] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10464.092440] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10464.103623] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10464.110545] RIP: 0010:native_safe_halt+0x2e/0x50 [10464.113794] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10464.133048] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10464.139787] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10464.145099] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10464.149903] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10464.154702] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10464.162792] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10464.168923] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10464.172831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10464.175230] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10464.179488] Call Trace: [10464.180865] [10464.182729] ? show_regs.cold.9+0x22/0x2f [10464.185355] ? watchdog_timer_fn.cold.6+0x62/0x10a [10464.188749] ? __hrtimer_run_queues+0x1c2/0x4d0 [10464.193050] ? watchdog+0x40/0x40 [10464.195023] ? hrtimer_interrupt+0x128/0x2c0 [10464.197137] ? smp_apic_timer_interrupt+0xa6/0x270 [10464.204640] ? apic_timer_interrupt+0xf/0x20 [10464.210129] [10464.211748] ? native_safe_halt+0x2e/0x50 [10464.213979] kvm_wait+0xcd/0xe0 [10464.215727] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10464.218162] do_raw_spin_lock+0x177/0x190 [10464.221086] _raw_spin_lock+0x21/0x30 [10464.222928] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10464.226227] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10464.229074] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10464.232614] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10464.236607] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10464.242184] ? woken_wake_function+0x30/0x30 [10464.244495] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10464.247924] kthread+0x1d1/0x200 [10464.250013] ? set_kthread_struct+0x70/0x70 [10464.252072] ret_from_fork+0x1f/0x30 [10468.054242] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [10468.065487] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10468.086654] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10468.090656] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10468.097078] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10468.103157] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10468.107083] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10468.116582] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10468.123455] RAX: 0000000000007dde RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10468.129488] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10468.135279] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10468.139676] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10468.142904] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10468.146894] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10468.153539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10468.158674] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10468.161203] Call Trace: [10468.162062] [10468.163880] ? show_regs.cold.9+0x22/0x2f [10468.167372] ? watchdog_timer_fn.cold.6+0x62/0x10a [10468.168883] ? __hrtimer_run_queues+0x1c2/0x4d0 [10468.170715] ? watchdog+0x40/0x40 [10468.171944] ? hrtimer_interrupt+0x128/0x2c0 [10468.173861] ? smp_apic_timer_interrupt+0xa6/0x270 [10468.178548] ? apic_timer_interrupt+0xf/0x20 [10468.181839] [10468.182545] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10468.184024] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10468.190418] ? newidle_balance.isra.93+0x207/0x580 [10468.198013] do_raw_spin_lock+0x177/0x190 [10468.200986] _raw_spin_lock+0x21/0x30 [10468.202007] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10468.204106] process_one_work+0x2c8/0x700 [10468.206005] worker_thread+0x296/0x6e0 [10468.208023] ? rescuer_thread+0x570/0x570 [10468.209786] kthread+0x1d1/0x200 [10468.211031] ? set_kthread_struct+0x70/0x70 [10468.212635] ret_from_fork+0x1f/0x30 [10471.904782] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7834s! [10471.908566] Showing busy workqueues and worker pools: [10471.911620] workqueue events: flags=0x0 [10471.913053] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10471.917273] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10471.925412] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10471.927597] pending: cpu_latency_work [ptlrpc] [10471.929432] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10471.931825] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10471.935400] workqueue events_long: flags=0x0 [10471.936868] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10471.939388] pending: lu_objects_flush [obdclass] [10471.941560] workqueue events_unbound: flags=0x2 [10471.943139] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10471.945560] pending: flush_memcg_stats_dwork, idle_cull_fn [10471.947627] workqueue events_power_efficient: flags=0x80 [10471.949457] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10471.951961] pending: check_lifetime, do_cache_clean [10471.953674] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10471.956055] pending: neigh_periodic_work, neigh_periodic_work [10471.958191] workqueue mm_percpu_wq: flags=0x8 [10471.959668] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10471.962544] pending: vmstat_update, lru_add_drain_per_cpu [10471.964881] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10471.967231] pending: vmstat_update, lru_add_drain_per_cpu [10471.969110] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10471.971339] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10471.973909] workqueue xprtiod: flags=0xa [10471.976653] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10471.981991] pending: xprt_autoclose, xs_error_handle [10471.986250] workqueue ipv6_addrconf: flags=0x40008 [10471.989625] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10471.994376] pending: addrconf_verify_work [10471.997352] workqueue knbd0-recv: flags=0x1a [10472.000933] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10472.004895] in-flight: 651:recv_work [10472.008106] workqueue ptlrpc_pinger: flags=0x2 [10472.011920] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10472.015563] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10472.020558] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7834s workers=4 idle: 15979 31824 8 [10472.026607] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10462s workers=2 idle: 654 [10488.055648] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [10488.057064] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [10488.061484] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel [10488.065489] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver [10488.079747] ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10488.079776] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10488.079789] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10488.079792] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10488.079806] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10488.079816] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10488.079821] RAX: 0000000000004a86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10488.079824] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10488.079826] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10488.079828] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10488.079831] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10488.079834] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10488.079837] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10488.079840] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10488.079850] Call Trace: [10488.079856] [10488.079861] ? show_regs.cold.9+0x22/0x2f [10488.079886] ? watchdog_timer_fn.cold.6+0x62/0x10a [10488.079894] ? __hrtimer_run_queues+0x1c2/0x4d0 [10488.079898] ? watchdog+0x40/0x40 [10488.079902] ? hrtimer_interrupt+0x128/0x2c0 [10488.079906] ? smp_apic_timer_interrupt+0xa6/0x270 [10488.079915] ? apic_timer_interrupt+0xf/0x20 [10488.079920] [10488.079922] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10488.079934] do_raw_spin_lock+0x177/0x190 [10488.079940] _raw_spin_lock+0x21/0x30 [10488.079945] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10488.080343] ? do_raw_spin_unlock+0x75/0x190 [10488.080350] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10488.080581] ? do_wait_intr+0xf0/0xf0 [10488.080590] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10488.080810] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10488.081176] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10488.081402] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10488.081520] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10488.081557] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10488.081587] mdc_intent_lock+0x353/0x7e0 [mdc] [10488.081659] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10488.081744] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10488.099036] intel_rapl_msr [10488.103172] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10488.116073] intel_rapl_common [10488.119160] lmv_intent_open+0x338/0x1150 [lmv] [10488.119206] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10488.119332] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10488.119425] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10488.119453] ll_intent_lock+0xc5/0xaa0 [lustre] [10488.119539] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10488.128038] sb_edac [10488.135712] ll_lookup_it+0xe0e/0x2270 [lustre] [10488.139182] rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log [10488.142240] ? do_raw_spin_unlock+0x75/0x190 [10488.142261] ? _raw_spin_unlock+0x12/0x30 [10488.142268] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10488.142294] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10488.142316] ll_atomic_open+0x36a/0xd70 [lustre] [10488.142512] lookup_open+0x356/0xb40 [10488.142522] do_last+0x313/0xfc0 [10488.142527] ? nd_jump_root+0xe5/0x160 [10488.142531] ? path_init+0x437/0x520 [10488.142536] path_openat+0xf7/0x500 [10488.142542] ? do_raw_spin_unlock+0x75/0x190 [10488.142547] do_filp_open+0x99/0x140 [10488.142553] ? getname_flags+0x6e/0x330 [10488.142558] ? __check_object_size+0xff/0x256 [10488.142565] ? do_raw_spin_unlock+0x75/0x190 [10488.142570] ? _raw_spin_unlock+0x12/0x30 [10488.142575] do_sys_openat2+0x2b4/0x410 [10488.142580] do_sys_open+0x73/0xa0 [10488.142585] __x64_sys_openat+0x24/0x30 [10488.142589] do_syscall_64+0xc1/0x440 [10488.142595] entry_SYSCALL_64_after_hwframe+0x49/0xae [10488.142602] RIP: 0033:0x7f53425a5332 [10488.142611] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10488.142615] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10488.142621] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10488.142623] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10488.142626] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10488.142628] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10488.142631] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10488.391669] dm_mod sha512_ssse3 sha512_generic [10488.393972] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10488.401335] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10488.411688] RIP: 0010:native_safe_halt+0x2e/0x50 [10488.415046] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10488.426794] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10488.431627] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10488.438299] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10488.442972] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10488.448602] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10488.453622] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10488.460328] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10488.467357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10488.472582] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10488.477941] Call Trace: [10488.480668] [10488.482667] ? show_regs.cold.9+0x22/0x2f [10488.484855] ? watchdog_timer_fn.cold.6+0x62/0x10a [10488.488265] ? __hrtimer_run_queues+0x1c2/0x4d0 [10488.492307] ? watchdog+0x40/0x40 [10488.497593] ? hrtimer_interrupt+0x128/0x2c0 [10488.504570] ? smp_apic_timer_interrupt+0xa6/0x270 [10488.507692] ? apic_timer_interrupt+0xf/0x20 [10488.511987] [10488.513365] ? native_safe_halt+0x2e/0x50 [10488.520116] kvm_wait+0xcd/0xe0 [10488.522302] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10488.526953] do_raw_spin_lock+0x177/0x190 [10488.529572] _raw_spin_lock+0x21/0x30 [10488.532338] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10488.538008] obd_mod_free+0x46/0x140 [ptlrpc] [10488.542322] mdc_commit_open+0x91/0xb0 [mdc] [10488.544948] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10488.548804] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10488.551369] after_reply+0xe2c/0x1780 [ptlrpc] [10488.556514] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10488.560850] ? __next_timer_interrupt+0x160/0x160 [10488.563245] ptlrpcd+0xa82/0xe40 [ptlrpc] [10488.566145] ? do_wait_intr+0xf0/0xf0 [10488.567761] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10488.570413] kthread+0x1d1/0x200 [10488.575220] ? set_kthread_struct+0x70/0x70 [10488.577246] ret_from_fork+0x1f/0x30 [10492.058658] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [10492.064605] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10492.088738] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10492.096530] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10492.101729] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10492.104848] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10492.115677] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10492.119661] RAX: 00000000000073a3 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10492.123248] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10492.127629] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10492.134791] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10492.138138] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10492.143234] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10492.148301] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10492.152090] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10492.156271] Call Trace: [10492.158792] [10492.159753] ? show_regs.cold.9+0x22/0x2f [10492.162563] ? watchdog_timer_fn.cold.6+0x62/0x10a [10492.164932] ? __hrtimer_run_queues+0x1c2/0x4d0 [10492.166691] ? watchdog+0x40/0x40 [10492.168025] ? hrtimer_interrupt+0x128/0x2c0 [10492.169741] ? smp_apic_timer_interrupt+0xa6/0x270 [10492.172332] ? apic_timer_interrupt+0xf/0x20 [10492.173538] [10492.174179] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10492.176297] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10492.178608] do_raw_spin_lock+0x177/0x190 [10492.180195] _raw_spin_lock+0x21/0x30 [10492.181714] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10492.185239] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10492.187699] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10492.190115] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10492.192672] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10492.202819] ? woken_wake_function+0x30/0x30 [10492.207561] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10492.209672] kthread+0x1d1/0x200 [10492.210756] ? set_kthread_struct+0x70/0x70 [10492.212172] ret_from_fork+0x1f/0x30 [10496.052025] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [10496.054284] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10496.085454] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10496.090924] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10496.094252] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10496.096765] RIP: 0010:native_safe_halt+0x2e/0x50 [10496.098502] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10496.106316] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10496.113185] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10496.122417] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [10496.132116] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10496.137045] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10496.144311] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10496.149163] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10496.157977] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10496.163217] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10496.166026] Call Trace: [10496.166609] [10496.167265] ? show_regs.cold.9+0x22/0x2f [10496.169328] ? watchdog_timer_fn.cold.6+0x62/0x10a [10496.172710] ? __hrtimer_run_queues+0x1c2/0x4d0 [10496.178623] ? watchdog+0x40/0x40 [10496.181355] ? hrtimer_interrupt+0x128/0x2c0 [10496.186069] ? smp_apic_timer_interrupt+0xa6/0x270 [10496.193225] ? apic_timer_interrupt+0xf/0x20 [10496.196625] [10496.197461] ? native_safe_halt+0x2e/0x50 [10496.200532] kvm_wait+0xcd/0xe0 [10496.205978] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10496.211098] ? newidle_balance.isra.93+0x207/0x580 [10496.216782] do_raw_spin_lock+0x177/0x190 [10496.219066] _raw_spin_lock+0x21/0x30 [10496.224595] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10496.230851] process_one_work+0x2c8/0x700 [10496.232515] worker_thread+0x296/0x6e0 [10496.234029] ? rescuer_thread+0x570/0x570 [10496.239680] kthread+0x1d1/0x200 [10496.243963] ? set_kthread_struct+0x70/0x70 [10496.246681] ret_from_fork+0x1f/0x30 [10502.115450] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7865s! [10502.119506] Showing busy workqueues and worker pools: [10502.121752] workqueue events: flags=0x0 [10502.123262] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10502.125936] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10502.132610] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10502.139425] pending: cpu_latency_work [ptlrpc] [10502.145145] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10502.150803] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10502.169223] workqueue events_long: flags=0x0 [10502.173360] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10502.178061] pending: lu_objects_flush [obdclass] [10502.184601] workqueue events_unbound: flags=0x2 [10502.188380] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10502.192740] pending: flush_memcg_stats_dwork, idle_cull_fn [10502.200785] workqueue events_power_efficient: flags=0x80 [10502.210571] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10502.217389] pending: check_lifetime, do_cache_clean [10502.223191] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10502.233102] pending: neigh_periodic_work, neigh_periodic_work [10502.239328] workqueue mm_percpu_wq: flags=0x8 [10502.242929] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10502.247339] pending: vmstat_update, lru_add_drain_per_cpu [10502.251700] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10502.253619] pending: vmstat_update, lru_add_drain_per_cpu [10502.264604] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10502.277140] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10502.283604] workqueue xprtiod: flags=0xa [10502.290518] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10502.297822] pending: xprt_autoclose, xs_error_handle [10502.307404] workqueue ipv6_addrconf: flags=0x40008 [10502.311403] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10502.324617] pending: addrconf_verify_work [10502.327226] workqueue knbd0-recv: flags=0x1a [10502.331514] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10502.338508] in-flight: 651:recv_work [10502.344480] workqueue ptlrpc_pinger: flags=0x2 [10502.352506] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10502.360813] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10502.372298] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7865s workers=4 idle: 15979 31824 8 [10502.375140] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10492s workers=2 idle: 654 [10516.055024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [10516.057028] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [10516.057697] Modules linked in: [10516.061950] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10516.062034] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10516.062041] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10516.062043] RIP: 0010:native_safe_halt+0x2e/0x50 [10516.062055] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10516.062060] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10516.062064] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10516.062067] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10516.062069] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10516.062072] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10516.062075] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10516.062078] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10516.062081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10516.062083] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10516.062091] Call Trace: [10516.062096] [10516.062106] ? show_regs.cold.9+0x22/0x2f [10516.062115] ? watchdog_timer_fn.cold.6+0x62/0x10a [10516.062122] ? __hrtimer_run_queues+0x1c2/0x4d0 [10516.062126] ? watchdog+0x40/0x40 [10516.062131] ? hrtimer_interrupt+0x128/0x2c0 [10516.062135] ? smp_apic_timer_interrupt+0xa6/0x270 [10516.062141] ? apic_timer_interrupt+0xf/0x20 [10516.062145] [10516.062148] ? native_safe_halt+0x2e/0x50 [10516.062152] kvm_wait+0xcd/0xe0 [10516.062159] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10516.062165] do_raw_spin_lock+0x177/0x190 [10516.062170] _raw_spin_lock+0x21/0x30 [10516.062175] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10516.062529] obd_mod_free+0x46/0x140 [ptlrpc] [10516.062723] mdc_commit_open+0x91/0xb0 [mdc] [10516.062760] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10516.062951] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10516.063143] after_reply+0xe2c/0x1780 [ptlrpc] [10516.063349] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10516.063691] ? __next_timer_interrupt+0x160/0x160 [10516.063700] ptlrpcd+0xa82/0xe40 [ptlrpc] [10516.063964] ? do_wait_intr+0xf0/0xf0 [10516.063974] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10516.065282] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10516.065352] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10516.065358] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10516.065360] RIP: 0010:native_safe_halt+0x2e/0x50 [10516.065370] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10516.065374] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10516.065394] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10516.065396] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10516.065399] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10516.065401] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10516.065404] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10516.065407] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10516.065410] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10516.065413] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10516.065420] Call Trace: [10516.065424] [10516.065427] ? show_regs.cold.9+0x22/0x2f [10516.065438] ? watchdog_timer_fn.cold.6+0x62/0x10a [10516.065443] ? __hrtimer_run_queues+0x1c2/0x4d0 [10516.065447] ? watchdog+0x40/0x40 [10516.065451] ? hrtimer_interrupt+0x128/0x2c0 [10516.065454] ? smp_apic_timer_interrupt+0xa6/0x270 [10516.065460] ? apic_timer_interrupt+0xf/0x20 [10516.065465] [10516.065467] ? native_safe_halt+0x2e/0x50 [10516.065471] kvm_wait+0xcd/0xe0 [10516.065478] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10516.065485] do_raw_spin_lock+0x177/0x190 [10516.065490] _raw_spin_lock+0x21/0x30 [10516.065494] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10516.065724] ? do_raw_spin_unlock+0x75/0x190 [10516.065731] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10516.065958] ? do_wait_intr+0xf0/0xf0 [10516.065966] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10516.096342] kthread+0x1d1/0x200 [10516.096360] ? set_kthread_struct+0x70/0x70 [10516.096365] ret_from_fork+0x1f/0x30 [10516.298722] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10516.301289] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10516.304158] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10516.306755] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10516.309993] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10516.311709] mdc_intent_lock+0x353/0x7e0 [mdc] [10516.313277] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10516.315724] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10516.318363] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10516.320824] lmv_intent_open+0x338/0x1150 [lmv] [10516.322983] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10516.325474] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10516.327704] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10516.329781] ll_intent_lock+0xc5/0xaa0 [lustre] [10516.332370] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10516.334774] ll_lookup_it+0xe0e/0x2270 [lustre] [10516.336870] ? do_raw_spin_unlock+0x75/0x190 [10516.338727] ? _raw_spin_unlock+0x12/0x30 [10516.340560] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10516.342321] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10516.345035] ll_atomic_open+0x36a/0xd70 [lustre] [10516.347309] lookup_open+0x356/0xb40 [10516.348997] do_last+0x313/0xfc0 [10516.350481] ? nd_jump_root+0xe5/0x160 [10516.352605] ? path_init+0x437/0x520 [10516.354365] path_openat+0xf7/0x500 [10516.356027] ? do_raw_spin_unlock+0x75/0x190 [10516.358483] do_filp_open+0x99/0x140 [10516.360001] ? getname_flags+0x6e/0x330 [10516.361693] ? __check_object_size+0xff/0x256 [10516.363731] ? do_raw_spin_unlock+0x75/0x190 [10516.366096] ? _raw_spin_unlock+0x12/0x30 [10516.367891] do_sys_openat2+0x2b4/0x410 [10516.369590] do_sys_open+0x73/0xa0 [10516.371292] __x64_sys_openat+0x24/0x30 [10516.373509] do_syscall_64+0xc1/0x440 [10516.375261] entry_SYSCALL_64_after_hwframe+0x49/0xae [10516.377659] RIP: 0033:0x7f53425a5332 [10516.379773] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10516.388689] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10516.392255] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10516.395857] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10516.399922] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10516.403522] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10516.407055] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10520.060145] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [10520.065255] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10520.119169] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10520.130857] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10520.146695] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10520.153767] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10520.191129] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10520.197984] RAX: 0000000000001883 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10520.204587] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10520.214029] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10520.219648] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10520.232075] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10520.243931] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10520.252729] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10520.255854] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10520.266050] Call Trace: [10520.266973] [10520.267475] ? show_regs.cold.9+0x22/0x2f [10520.277922] ? watchdog_timer_fn.cold.6+0x62/0x10a [10520.281389] ? __hrtimer_run_queues+0x1c2/0x4d0 [10520.287629] ? watchdog+0x40/0x40 [10520.290333] ? hrtimer_interrupt+0x128/0x2c0 [10520.299655] ? smp_apic_timer_interrupt+0xa6/0x270 [10520.304180] ? apic_timer_interrupt+0xf/0x20 [10520.308440] [10520.309856] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10520.323457] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10520.332379] do_raw_spin_lock+0x177/0x190 [10520.335066] _raw_spin_lock+0x21/0x30 [10520.337719] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10520.348290] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10520.352044] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10520.358456] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10520.362825] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10520.371017] ? woken_wake_function+0x30/0x30 [10520.373427] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10520.377011] kthread+0x1d1/0x200 [10520.378601] ? set_kthread_struct+0x70/0x70 [10520.380525] ret_from_fork+0x1f/0x30 [10528.052187] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [10528.059450] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10528.113509] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10528.121918] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10528.137371] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10528.144396] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10528.150379] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10528.170573] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [10528.174644] RAX: 0000000000006f87 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10528.177867] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10528.182439] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10528.186857] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10528.190465] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10528.193472] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10528.198075] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10528.200589] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10528.204950] Call Trace: [10528.206105] [10528.206997] ? show_regs.cold.9+0x22/0x2f [10528.213490] ? watchdog_timer_fn.cold.6+0x62/0x10a [10528.217989] ? __hrtimer_run_queues+0x1c2/0x4d0 [10528.223855] ? watchdog+0x40/0x40 [10528.229513] ? hrtimer_interrupt+0x128/0x2c0 [10528.236092] ? smp_apic_timer_interrupt+0xa6/0x270 [10528.238942] ? apic_timer_interrupt+0xf/0x20 [10528.241533] [10528.242468] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10528.245118] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10528.248573] ? newidle_balance.isra.93+0x207/0x580 [10528.252529] do_raw_spin_lock+0x177/0x190 [10528.256975] _raw_spin_lock+0x21/0x30 [10528.263534] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10528.270472] process_one_work+0x2c8/0x700 [10528.274139] worker_thread+0x296/0x6e0 [10528.277189] ? rescuer_thread+0x570/0x570 [10528.278997] kthread+0x1d1/0x200 [10528.285952] ? set_kthread_struct+0x70/0x70 [10528.291083] ret_from_fork+0x1f/0x30 [10532.844277] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7895s! [10532.858102] Showing busy workqueues and worker pools: [10532.895324] workqueue events: flags=0x0 [10532.896692] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10532.919718] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10532.945655] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10532.951498] pending: cpu_latency_work [ptlrpc] [10532.957675] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10532.961764] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10532.972469] workqueue events_long: flags=0x0 [10532.976135] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10532.980829] pending: lu_objects_flush [obdclass] [10532.990680] workqueue events_unbound: flags=0x2 [10533.002920] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10533.016732] pending: flush_memcg_stats_dwork, idle_cull_fn [10533.021667] workqueue events_power_efficient: flags=0x80 [10533.031618] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10533.039117] pending: check_lifetime, do_cache_clean [10533.043635] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10533.057844] pending: neigh_periodic_work, neigh_periodic_work [10533.064867] workqueue mm_percpu_wq: flags=0x8 [10533.068826] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10533.078658] pending: vmstat_update, lru_add_drain_per_cpu [10533.085904] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10533.099815] pending: vmstat_update, lru_add_drain_per_cpu [10533.109197] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10533.125764] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10533.139281] workqueue xprtiod: flags=0xa [10533.141744] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10533.150236] pending: xprt_autoclose, xs_error_handle [10533.154712] workqueue ipv6_addrconf: flags=0x40008 [10533.157899] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10533.163407] pending: addrconf_verify_work [10533.165623] workqueue knbd0-recv: flags=0x1a [10533.178306] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10533.196269] in-flight: 651:recv_work [10533.198678] workqueue ptlrpc_pinger: flags=0x2 [10533.210651] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10533.218887] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10533.226096] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7896s workers=4 idle: 15979 31824 8 [10533.233687] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10523s workers=2 idle: 654 [10544.055121] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [10544.057316] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [10544.057850] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10544.057929] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10544.057934] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10544.057937] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10544.057949] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10544.057953] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10544.057958] RAX: 000000000000038b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10544.057960] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10544.057962] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10544.057964] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10544.057967] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10544.057971] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10544.057974] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10544.057976] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10544.057985] Call Trace: [10544.057990] [10544.057996] ? show_regs.cold.9+0x22/0x2f [10544.058005] ? watchdog_timer_fn.cold.6+0x62/0x10a [10544.058013] ? __hrtimer_run_queues+0x1c2/0x4d0 [10544.058018] ? watchdog+0x40/0x40 [10544.058024] ? hrtimer_interrupt+0x128/0x2c0 [10544.058029] ? smp_apic_timer_interrupt+0xa6/0x270 [10544.058036] ? apic_timer_interrupt+0xf/0x20 [10544.058041] [10544.058044] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10544.058051] do_raw_spin_lock+0x177/0x190 [10544.058056] _raw_spin_lock+0x21/0x30 [10544.058061] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10544.060695] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10544.060774] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10544.060780] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10544.060783] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10544.060795] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10544.060800] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10544.060804] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10544.060807] RDX: 00000000000068a4 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10544.060810] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10544.060812] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10544.060814] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10544.060818] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10544.060821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10544.060824] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10544.060832] Call Trace: [10544.060837] [10544.060843] ? show_regs.cold.9+0x22/0x2f [10544.060852] ? watchdog_timer_fn.cold.6+0x62/0x10a [10544.060859] ? __hrtimer_run_queues+0x1c2/0x4d0 [10544.060862] ? watchdog+0x40/0x40 [10544.060867] ? hrtimer_interrupt+0x128/0x2c0 [10544.060871] ? smp_apic_timer_interrupt+0xa6/0x270 [10544.060879] ? apic_timer_interrupt+0xf/0x20 [10544.060884] [10544.060887] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10544.060902] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10544.060907] do_raw_spin_lock+0x177/0x190 [10544.060912] _raw_spin_lock+0x21/0x30 [10544.060917] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10544.061175] obd_mod_free+0x46/0x140 [ptlrpc] [10544.061369] mdc_commit_open+0x91/0xb0 [mdc] [10544.061406] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10544.078268] ? do_raw_spin_unlock+0x75/0x190 [10544.078287] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10544.078560] ? do_wait_intr+0xf0/0xf0 [10544.078568] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10544.078758] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10544.079077] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10544.079292] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10544.079402] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10544.079441] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10544.079792] mdc_intent_lock+0x353/0x7e0 [mdc] [10544.086438] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10544.086741] after_reply+0xe2c/0x1780 [ptlrpc] [10544.086948] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10544.087138] ? __next_timer_interrupt+0x160/0x160 [10544.087148] ptlrpcd+0xa82/0xe40 [ptlrpc] [10544.087348] ? do_wait_intr+0xf0/0xf0 [10544.087358] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10544.087552] kthread+0x1d1/0x200 [10544.087558] ? set_kthread_struct+0x70/0x70 [10544.087562] ret_from_fork+0x1f/0x30 [10544.361197] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10544.362834] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10544.365103] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10544.367336] lmv_intent_open+0x338/0x1150 [lmv] [10544.368812] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10544.370503] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10544.372296] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10544.373772] ll_intent_lock+0xc5/0xaa0 [lustre] [10544.375281] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10544.376943] ll_lookup_it+0xe0e/0x2270 [lustre] [10544.378440] ? do_raw_spin_unlock+0x75/0x190 [10544.379764] ? _raw_spin_unlock+0x12/0x30 [10544.381015] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10544.382662] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10544.384320] ll_atomic_open+0x36a/0xd70 [lustre] [10544.385818] lookup_open+0x356/0xb40 [10544.386953] do_last+0x313/0xfc0 [10544.387977] ? nd_jump_root+0xe5/0x160 [10544.389591] ? path_init+0x437/0x520 [10544.390961] path_openat+0xf7/0x500 [10544.392376] ? do_raw_spin_unlock+0x75/0x190 [10544.393949] do_filp_open+0x99/0x140 [10544.395276] ? getname_flags+0x6e/0x330 [10544.396562] ? __check_object_size+0xff/0x256 [10544.398384] ? do_raw_spin_unlock+0x75/0x190 [10544.399946] ? _raw_spin_unlock+0x12/0x30 [10544.401278] do_sys_openat2+0x2b4/0x410 [10544.402431] do_sys_open+0x73/0xa0 [10544.403501] __x64_sys_openat+0x24/0x30 [10544.404821] do_syscall_64+0xc1/0x440 [10544.406092] entry_SYSCALL_64_after_hwframe+0x49/0xae [10544.407714] RIP: 0033:0x7f53425a5332 [10544.408869] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10544.414705] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10544.416836] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10544.418947] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10544.421176] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10544.423321] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10544.425372] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10548.060697] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [10548.065119] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10548.102450] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10548.113448] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10548.120053] RIP: 0010:native_safe_halt+0x2e/0x50 [10548.123462] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10548.142914] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [10548.153187] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10548.158079] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10548.166140] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10548.173344] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10548.181059] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10548.192263] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10548.202539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10548.206240] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10548.209753] Call Trace: [10548.212095] [10548.212868] ? show_regs.cold.9+0x22/0x2f [10548.215661] ? watchdog_timer_fn.cold.6+0x62/0x10a [10548.217644] ? __hrtimer_run_queues+0x1c2/0x4d0 [10548.222244] ? watchdog+0x40/0x40 [10548.223575] ? hrtimer_interrupt+0x128/0x2c0 [10548.226169] ? smp_apic_timer_interrupt+0xa6/0x270 [10548.228374] ? apic_timer_interrupt+0xf/0x20 [10548.230492] [10548.231181] ? native_safe_halt+0x2e/0x50 [10548.232345] kvm_wait+0xcd/0xe0 [10548.233231] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10548.235119] do_raw_spin_lock+0x177/0x190 [10548.236771] _raw_spin_lock+0x21/0x30 [10548.238594] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10548.242563] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10548.245792] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10548.248371] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10548.250247] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10548.253249] ? woken_wake_function+0x30/0x30 [10548.255269] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10548.258122] kthread+0x1d1/0x200 [10548.259654] ? set_kthread_struct+0x70/0x70 [10548.261555] ret_from_fork+0x1f/0x30 [10560.054832] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [10560.058567] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10560.107959] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10560.123107] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10560.135063] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10560.142426] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10560.147523] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10560.158827] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [10560.164500] RAX: 0000000000007e0f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10560.168133] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10560.170813] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10560.172592] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10560.181005] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10560.184455] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10560.188423] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10560.191056] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10560.194673] Call Trace: [10560.195755] [10560.196523] ? show_regs.cold.9+0x22/0x2f [10560.198468] ? watchdog_timer_fn.cold.6+0x62/0x10a [10560.201002] ? __hrtimer_run_queues+0x1c2/0x4d0 [10560.203806] ? watchdog+0x40/0x40 [10560.205311] ? hrtimer_interrupt+0x128/0x2c0 [10560.207523] ? smp_apic_timer_interrupt+0xa6/0x270 [10560.215775] ? apic_timer_interrupt+0xf/0x20 [10560.217288] [10560.218116] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10560.222233] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10560.225009] ? newidle_balance.isra.93+0x207/0x580 [10560.227438] do_raw_spin_lock+0x177/0x190 [10560.229446] _raw_spin_lock+0x21/0x30 [10560.231275] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10560.234143] process_one_work+0x2c8/0x700 [10560.236145] worker_thread+0x296/0x6e0 [10560.238075] ? rescuer_thread+0x570/0x570 [10560.240315] kthread+0x1d1/0x200 [10560.242116] ? set_kthread_struct+0x70/0x70 [10560.243954] ret_from_fork+0x1f/0x30 [10563.552452] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7926s! [10563.558272] Showing busy workqueues and worker pools: [10563.561766] workqueue events: flags=0x0 [10563.563264] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10563.567322] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10563.577305] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10563.585962] pending: cpu_latency_work [ptlrpc] [10563.589385] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10563.592943] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10563.598193] workqueue events_long: flags=0x0 [10563.599734] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10563.602983] pending: lu_objects_flush [obdclass] [10563.606046] workqueue events_unbound: flags=0x2 [10563.612136] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10563.615949] pending: flush_memcg_stats_dwork, idle_cull_fn [10563.621184] workqueue events_power_efficient: flags=0x80 [10563.623365] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10563.632200] pending: check_lifetime, do_cache_clean [10563.635868] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10563.639185] pending: neigh_periodic_work, neigh_periodic_work [10563.642467] workqueue mm_percpu_wq: flags=0x8 [10563.644878] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10563.651404] pending: vmstat_update, lru_add_drain_per_cpu [10563.655136] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10563.660136] pending: vmstat_update, lru_add_drain_per_cpu [10563.663801] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10563.667267] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10563.671986] workqueue xprtiod: flags=0xa [10563.673130] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10563.679021] pending: xprt_autoclose, xs_error_handle [10563.684062] workqueue ipv6_addrconf: flags=0x40008 [10563.687023] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10563.698241] pending: addrconf_verify_work [10563.701313] workqueue knbd0-recv: flags=0x1a [10563.704960] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10563.709626] in-flight: 651:recv_work [10563.711312] workqueue ptlrpc_pinger: flags=0x2 [10563.719354] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10563.721407] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10563.728217] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7926s workers=4 idle: 15979 31824 8 [10563.738644] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10554s workers=2 idle: 654 [10572.055391] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [10572.058466] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [10572.068461] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10572.068544] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10572.068553] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10572.068556] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10572.068571] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10572.068576] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10572.068580] RAX: 0000000000007b29 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10572.068582] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10572.068584] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10572.068586] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10572.068589] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10572.068592] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10572.068595] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10572.068598] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10572.068606] Call Trace: [10572.068612] [10572.068618] ? show_regs.cold.9+0x22/0x2f [10572.068627] ? watchdog_timer_fn.cold.6+0x62/0x10a [10572.068634] ? __hrtimer_run_queues+0x1c2/0x4d0 [10572.068639] ? watchdog+0x40/0x40 [10572.068644] ? hrtimer_interrupt+0x128/0x2c0 [10572.068649] ? smp_apic_timer_interrupt+0xa6/0x270 [10572.068657] ? apic_timer_interrupt+0xf/0x20 [10572.068662] [10572.068665] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10572.068673] do_raw_spin_lock+0x177/0x190 [10572.068678] _raw_spin_lock+0x21/0x30 [10572.068683] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10572.068963] ? do_raw_spin_unlock+0x75/0x190 [10572.068968] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10572.069161] ? do_wait_intr+0xf0/0xf0 [10572.069170] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10572.073789] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10572.073868] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10572.073875] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10572.073878] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10572.073891] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10572.073895] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10572.073900] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10572.073902] RDX: 0000000000003fb5 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10572.073905] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10572.073907] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10572.073910] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10572.073913] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10572.073916] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10572.073919] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10572.073928] Call Trace: [10572.073934] [10572.073939] ? show_regs.cold.9+0x22/0x2f [10572.073949] ? watchdog_timer_fn.cold.6+0x62/0x10a [10572.073956] ? __hrtimer_run_queues+0x1c2/0x4d0 [10572.073960] ? watchdog+0x40/0x40 [10572.073965] ? hrtimer_interrupt+0x128/0x2c0 [10572.073969] ? smp_apic_timer_interrupt+0xa6/0x270 [10572.073977] ? apic_timer_interrupt+0xf/0x20 [10572.073982] [10572.073985] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10572.073989] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10572.073994] do_raw_spin_lock+0x177/0x190 [10572.073999] _raw_spin_lock+0x21/0x30 [10572.074051] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10572.074324] obd_mod_free+0x46/0x140 [ptlrpc] [10572.074709] mdc_commit_open+0x91/0xb0 [mdc] [10572.074748] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10572.074954] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10572.121325] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10572.121856] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10572.122100] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10572.122244] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10572.122411] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10572.122444] mdc_intent_lock+0x353/0x7e0 [mdc] [10572.122473] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10572.122934] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10572.123168] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10572.123198] lmv_intent_open+0x338/0x1150 [lmv] [10572.123227] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10572.123805] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10572.123929] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10572.123964] ll_intent_lock+0xc5/0xaa0 [lustre] [10572.124053] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10572.124138] ll_lookup_it+0xe0e/0x2270 [lustre] [10572.124335] ? do_raw_spin_unlock+0x75/0x190 [10572.124343] ? _raw_spin_unlock+0x12/0x30 [10572.124350] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10572.124374] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10572.124411] ll_atomic_open+0x36a/0xd70 [lustre] [10572.124506] lookup_open+0x356/0xb40 [10572.124516] do_last+0x313/0xfc0 [10572.124521] ? nd_jump_root+0xe5/0x160 [10572.124525] ? path_init+0x437/0x520 [10572.124530] path_openat+0xf7/0x500 [10572.124535] ? do_raw_spin_unlock+0x75/0x190 [10572.124540] do_filp_open+0x99/0x140 [10572.124547] ? getname_flags+0x6e/0x330 [10572.124551] ? __check_object_size+0xff/0x256 [10572.124558] ? do_raw_spin_unlock+0x75/0x190 [10572.124562] ? _raw_spin_unlock+0x12/0x30 [10572.124566] do_sys_openat2+0x2b4/0x410 [10572.124571] do_sys_open+0x73/0xa0 [10572.124576] __x64_sys_openat+0x24/0x30 [10572.124579] do_syscall_64+0xc1/0x440 [10572.124586] entry_SYSCALL_64_after_hwframe+0x49/0xae [10572.124592] RIP: 0033:0x7f53425a5332 [10572.124600] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10572.124604] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10572.124609] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10572.124611] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10572.124613] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10572.124615] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10572.124618] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10572.737959] after_reply+0xe2c/0x1780 [ptlrpc] [10572.747604] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10572.751835] ? __next_timer_interrupt+0x160/0x160 [10572.763117] ptlrpcd+0xa82/0xe40 [ptlrpc] [10572.770340] ? do_wait_intr+0xf0/0xf0 [10572.774758] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10572.784508] kthread+0x1d1/0x200 [10572.785877] ? set_kthread_struct+0x70/0x70 [10572.793150] ret_from_fork+0x1f/0x30 [10576.058047] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [10576.061354] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10576.079992] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10576.090657] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10576.101515] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10576.110931] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10576.132318] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10576.142927] RAX: 000000000000420a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10576.152758] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10576.165809] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10576.171368] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10576.180892] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10576.189851] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10576.196271] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10576.202062] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10576.213695] Call Trace: [10576.216214] [10576.219084] ? show_regs.cold.9+0x22/0x2f [10576.221590] ? watchdog_timer_fn.cold.6+0x62/0x10a [10576.227276] ? __hrtimer_run_queues+0x1c2/0x4d0 [10576.229810] ? watchdog+0x40/0x40 [10576.233873] ? hrtimer_interrupt+0x128/0x2c0 [10576.239712] ? smp_apic_timer_interrupt+0xa6/0x270 [10576.247544] ? apic_timer_interrupt+0xf/0x20 [10576.253091] [10576.256646] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10576.266812] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10576.275767] do_raw_spin_lock+0x177/0x190 [10576.277074] _raw_spin_lock+0x21/0x30 [10576.281287] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10576.294217] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10576.305107] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10576.309212] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10576.316164] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10576.323618] ? woken_wake_function+0x30/0x30 [10576.331287] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10576.336837] kthread+0x1d1/0x200 [10576.341615] ? set_kthread_struct+0x70/0x70 [10576.344613] ret_from_fork+0x1f/0x30 [10588.052044] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [10588.054180] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10588.080561] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10588.085280] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10588.088014] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10588.090789] RIP: 0010:native_safe_halt+0x2e/0x50 [10588.093027] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10588.100495] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10588.102454] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10588.104391] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [10588.106440] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10588.109103] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10588.111826] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10588.114504] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10588.117948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10588.120231] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10588.122911] Call Trace: [10588.123867] [10588.124659] ? show_regs.cold.9+0x22/0x2f [10588.126093] ? watchdog_timer_fn.cold.6+0x62/0x10a [10588.128439] ? __hrtimer_run_queues+0x1c2/0x4d0 [10588.130505] ? watchdog+0x40/0x40 [10588.131627] ? hrtimer_interrupt+0x128/0x2c0 [10588.133075] ? smp_apic_timer_interrupt+0xa6/0x270 [10588.135322] ? apic_timer_interrupt+0xf/0x20 [10588.137031] [10588.137924] ? native_safe_halt+0x2e/0x50 [10588.139635] kvm_wait+0xcd/0xe0 [10588.140959] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10588.143343] ? newidle_balance.isra.93+0x207/0x580 [10588.145204] do_raw_spin_lock+0x177/0x190 [10588.146859] _raw_spin_lock+0x21/0x30 [10588.148373] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10588.150790] process_one_work+0x2c8/0x700 [10588.152420] worker_thread+0x296/0x6e0 [10588.153953] ? rescuer_thread+0x570/0x570 [10588.155859] kthread+0x1d1/0x200 [10588.157150] ? set_kthread_struct+0x70/0x70 [10588.159109] ret_from_fork+0x1f/0x30 [10593.760763] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7956s! [10593.770831] Showing busy workqueues and worker pools: [10593.774144] workqueue events: flags=0x0 [10593.777840] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10593.782282] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10593.794745] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10593.798005] pending: cpu_latency_work [ptlrpc] [10593.807808] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10593.817951] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10593.830011] workqueue events_long: flags=0x0 [10593.835249] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10593.840547] pending: lu_objects_flush [obdclass] [10593.846112] workqueue events_unbound: flags=0x2 [10593.848938] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10593.854087] pending: flush_memcg_stats_dwork, idle_cull_fn [10593.860497] workqueue events_power_efficient: flags=0x80 [10593.866305] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10593.883651] pending: check_lifetime, do_cache_clean [10593.888043] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10593.893557] pending: neigh_periodic_work, neigh_periodic_work [10593.902918] workqueue mm_percpu_wq: flags=0x8 [10593.910124] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10593.921299] pending: vmstat_update, lru_add_drain_per_cpu [10593.929654] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10593.934760] pending: vmstat_update, lru_add_drain_per_cpu [10593.939303] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10593.948795] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10593.959362] workqueue xprtiod: flags=0xa [10593.960651] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10593.974077] pending: xprt_autoclose, xs_error_handle [10593.977839] workqueue ipv6_addrconf: flags=0x40008 [10593.982096] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10593.988328] pending: addrconf_verify_work [10593.991331] workqueue knbd0-recv: flags=0x1a [10593.996101] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10594.010707] in-flight: 651:recv_work [10594.014071] workqueue ptlrpc_pinger: flags=0x2 [10594.022455] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10594.028222] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10594.037878] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7956s workers=4 idle: 15979 31824 8 [10594.041677] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10584s workers=2 idle: 654 [10600.057147] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [10600.059018] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [10600.059769] Modules linked in: [10600.064315] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10600.064435] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10600.064444] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10600.064447] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10600.064461] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10600.064465] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10600.064470] RAX: 0000000000005b98 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10600.064472] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10600.064475] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10600.064477] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10600.064479] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10600.064483] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10600.064486] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10600.064489] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10600.064499] Call Trace: [10600.064504] [10600.064509] ? show_regs.cold.9+0x22/0x2f [10600.064520] ? watchdog_timer_fn.cold.6+0x62/0x10a [10600.064525] ? __hrtimer_run_queues+0x1c2/0x4d0 [10600.064530] ? watchdog+0x40/0x40 [10600.064534] ? hrtimer_interrupt+0x128/0x2c0 [10600.064538] ? smp_apic_timer_interrupt+0xa6/0x270 [10600.064546] ? apic_timer_interrupt+0xf/0x20 [10600.064550] [10600.064553] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10600.064560] do_raw_spin_lock+0x177/0x190 [10600.064564] _raw_spin_lock+0x21/0x30 [10600.064569] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10600.066673] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10600.066740] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10600.066747] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10600.066750] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10600.066762] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10600.066767] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10600.066771] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10600.066773] RDX: 0000000000001d86 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10600.066775] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10600.066777] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10600.066779] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10600.066782] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10600.066785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10600.066788] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10600.066796] Call Trace: [10600.066801] [10600.066805] ? show_regs.cold.9+0x22/0x2f [10600.066813] ? watchdog_timer_fn.cold.6+0x62/0x10a [10600.066818] ? __hrtimer_run_queues+0x1c2/0x4d0 [10600.066822] ? watchdog+0x40/0x40 [10600.066826] ? hrtimer_interrupt+0x128/0x2c0 [10600.066829] ? smp_apic_timer_interrupt+0xa6/0x270 [10600.066838] ? apic_timer_interrupt+0xf/0x20 [10600.066842] [10600.066844] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10600.066848] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10600.066853] do_raw_spin_lock+0x177/0x190 [10600.066856] _raw_spin_lock+0x21/0x30 [10600.066862] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10600.067160] obd_mod_free+0x46/0x140 [ptlrpc] [10600.067373] mdc_commit_open+0x91/0xb0 [mdc] [10600.067409] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10600.067633] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10600.067874] after_reply+0xe2c/0x1780 [ptlrpc] [10600.068108] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10600.068295] ? __next_timer_interrupt+0x160/0x160 [10600.068304] ptlrpcd+0xa82/0xe40 [ptlrpc] [10600.068509] ? do_wait_intr+0xf0/0xf0 [10600.068518] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10600.068777] kthread+0x1d1/0x200 [10600.068788] ? set_kthread_struct+0x70/0x70 [10600.068792] ret_from_fork+0x1f/0x30 [10600.487516] ? do_raw_spin_unlock+0x75/0x190 [10600.489541] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10600.493767] ? do_wait_intr+0xf0/0xf0 [10600.500788] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10600.509293] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10600.511247] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10600.514737] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10600.532410] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10600.534239] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10600.542949] mdc_intent_lock+0x353/0x7e0 [mdc] [10600.544109] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10600.552341] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10600.555632] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10600.559652] lmv_intent_open+0x338/0x1150 [lmv] [10600.563625] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10600.566842] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10600.569460] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10600.571730] ll_intent_lock+0xc5/0xaa0 [lustre] [10600.574238] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10600.579545] ll_lookup_it+0xe0e/0x2270 [lustre] [10600.581742] ? do_raw_spin_unlock+0x75/0x190 [10600.584988] ? _raw_spin_unlock+0x12/0x30 [10600.587132] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10600.588930] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10600.591696] ll_atomic_open+0x36a/0xd70 [lustre] [10600.593761] lookup_open+0x356/0xb40 [10600.595715] do_last+0x313/0xfc0 [10600.597937] ? nd_jump_root+0xe5/0x160 [10600.602793] ? path_init+0x437/0x520 [10600.609981] path_openat+0xf7/0x500 [10600.612374] ? do_raw_spin_unlock+0x75/0x190 [10600.616591] do_filp_open+0x99/0x140 [10600.619046] ? getname_flags+0x6e/0x330 [10600.620750] ? __check_object_size+0xff/0x256 [10600.623160] ? do_raw_spin_unlock+0x75/0x190 [10600.625713] ? _raw_spin_unlock+0x12/0x30 [10600.627795] do_sys_openat2+0x2b4/0x410 [10600.630673] do_sys_open+0x73/0xa0 [10600.632411] __x64_sys_openat+0x24/0x30 [10600.634207] do_syscall_64+0xc1/0x440 [10600.641080] entry_SYSCALL_64_after_hwframe+0x49/0xae [10600.642996] RIP: 0033:0x7f53425a5332 [10600.645033] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10600.662265] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10600.668676] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10600.671009] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10600.679277] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10600.682444] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10600.686315] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10604.058028] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [10604.060666] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10604.110466] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10604.125081] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10604.131079] RIP: 0010:native_safe_halt+0x2e/0x50 [10604.132813] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10604.156064] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10604.160610] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10604.167167] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10604.172425] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10604.177766] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10604.182448] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10604.186698] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10604.191239] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10604.197077] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10604.205731] Call Trace: [10604.207750] [10604.210661] ? show_regs.cold.9+0x22/0x2f [10604.215656] ? watchdog_timer_fn.cold.6+0x62/0x10a [10604.219806] ? __hrtimer_run_queues+0x1c2/0x4d0 [10604.223242] ? watchdog+0x40/0x40 [10604.225620] ? hrtimer_interrupt+0x128/0x2c0 [10604.230751] ? smp_apic_timer_interrupt+0xa6/0x270 [10604.237757] ? apic_timer_interrupt+0xf/0x20 [10604.246018] [10604.248671] ? native_safe_halt+0x2e/0x50 [10604.250135] kvm_wait+0xcd/0xe0 [10604.252102] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10604.255062] do_raw_spin_lock+0x177/0x190 [10604.258275] _raw_spin_lock+0x21/0x30 [10604.259688] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10604.265925] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10604.268415] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10604.274549] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10604.282436] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10604.286691] ? woken_wake_function+0x30/0x30 [10604.288740] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10604.291326] kthread+0x1d1/0x200 [10604.292382] ? set_kthread_struct+0x70/0x70 [10604.301171] ret_from_fork+0x1f/0x30 [10614.629016] rcu: INFO: rcu_sched self-detected stall on CPU [10614.631967] rcu: 2-....: (4679786 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [10614.637190] (t=8520203 jiffies g=274185 q=65984) [10614.639866] rcu: rcu_sched kthread starved for 7978569 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [10614.645493] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10614.650273] rcu: RCU grace-period kthread stack dump: [10614.652467] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10614.657862] Call Trace: [10614.659134] __schedule+0x351/0xcb0 [10614.661191] schedule+0xc0/0x180 [10614.662939] schedule_timeout+0xb4/0x190 [10614.665303] ? do_raw_spin_unlock+0x75/0x190 [10614.667545] ? __next_timer_interrupt+0x160/0x160 [10614.670201] ? prepare_to_swait_event+0xf3/0x140 [10614.672995] rcu_gp_kthread+0x8dd/0x15b0 [10614.674997] ? force_qs_rnp+0x2c0/0x2c0 [10614.677176] kthread+0x1d1/0x200 [10614.678984] ? set_kthread_struct+0x70/0x70 [10614.681247] ret_from_fork+0x1f/0x30 [10614.683092] rcu: Stack dump where RCU GP kthread last ran: [10614.685990] Sending NMI from CPU 2 to CPUs 3: [10614.691327] NMI backtrace for cpu 3 [10614.691335] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10614.691338] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10614.691340] RIP: 0010:_raw_spin_lock_irqsave+0xe/0x70 [10614.691341] Code: 83 05 55 8e f9 01 01 e8 60 d3 1c ff 48 83 05 50 8e f9 01 01 c3 0f 1f 80 00 00 00 00 0f 1f 44 00 00 53 48 83 05 22 8c f9 01 01 <48> 83 05 7a 8b f9 01 01 9c 58 0f 1f 44 00 00 48 83 05 73 8b f9 01 [10614.691344] RSP: 0018:ffffbb5380d8cf78 EFLAGS: 00000006 [10614.691347] RAX: 0000000000000000 RBX: ffff98a70219df40 RCX: 000000000000080b [10614.691348] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff98a7021a4e40 [10614.691350] RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000 [10614.691352] R10: 0000000000000000 R11: 0000000000000000 R12: 7fffffffffffffff [10614.691353] R13: 0000000000000000 R14: ffff98a7021a4e40 R15: ffff98a7021a4fb8 [10614.691355] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10614.691357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10614.691358] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10614.691359] Call Trace: [10614.691360] [10614.691361] ? show_regs.cold.9+0x22/0x2f [10614.691363] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10614.691364] ? nmi_cpu_backtrace_handler+0x15/0x40 [10614.691365] ? nmi_handle+0xbd/0x210 [10614.691366] ? default_do_nmi+0x4b/0x1b0 [10614.691368] ? do_nmi+0x280/0x380 [10614.691369] ? end_repeat_nmi+0x16/0x1f [10614.691370] ? _raw_spin_lock_irqsave+0xe/0x70 [10614.691371] ? _raw_spin_lock_irqsave+0xe/0x70 [10614.691372] ? _raw_spin_lock_irqsave+0xe/0x70 [10614.691373] [10614.691374] [10614.691375] hrtimer_interrupt+0x64/0x2c0 [10614.691376] ? sched_clock+0xd/0x20 [10614.691378] ? sched_clock_cpu+0x24/0x150 [10614.691379] smp_apic_timer_interrupt+0xa6/0x270 [10614.691380] apic_timer_interrupt+0xf/0x20 [10614.691381] [10614.691382] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10614.691384] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10614.691387] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10614.691390] RAX: 0000000000007b92 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10614.691391] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10614.691393] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10614.691394] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10614.691396] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10614.691397] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10614.691398] do_raw_spin_lock+0x177/0x190 [10614.691399] _raw_spin_lock+0x21/0x30 [10614.691400] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10614.691401] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10614.691402] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10614.691403] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10614.691404] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10614.691405] ? woken_wake_function+0x30/0x30 [10614.691406] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10614.691408] kthread+0x1d1/0x200 [10614.691409] ? set_kthread_struct+0x70/0x70 [10614.691410] ret_from_fork+0x1f/0x30 [10614.691506] Sending NMI from CPU 2 to CPUs 1: [10614.897231] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [10614.898186] NMI backtrace for cpu 2 [10614.903164] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10614.918080] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10614.938629] Call Trace: [10614.944559] [10614.945255] ? dump_stack+0xbb/0x10e [10614.953102] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10614.955926] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10614.963956] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10614.979471] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10614.983347] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10614.988545] ? rcu_dump_cpu_stacks+0x11b/0x171 [10614.991316] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10614.998845] ? update_process_times+0x6d/0xc0 [10615.004548] ? tick_sched_handle.isra.18+0x27/0x90 [10615.007596] ? tick_sched_timer+0x53/0xb0 [10615.010257] ? __hrtimer_run_queues+0x1c2/0x4d0 [10615.013049] ? tick_sched_handle.isra.18+0x90/0x90 [10615.017772] ? hrtimer_interrupt+0x128/0x2c0 [10615.028309] ? smp_apic_timer_interrupt+0xa6/0x270 [10615.031493] ? apic_timer_interrupt+0xf/0x20 [10615.037322] [10615.039332] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10615.046606] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10615.057061] ? do_raw_spin_lock+0x177/0x190 [10615.065159] ? _raw_spin_lock+0x21/0x30 [10615.069566] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10615.087624] ? obd_mod_free+0x46/0x140 [ptlrpc] [10615.094289] ? mdc_commit_open+0x91/0xb0 [mdc] [10615.095360] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10615.115551] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10615.130014] ? after_reply+0xe2c/0x1780 [ptlrpc] [10615.131388] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10615.152410] ? __next_timer_interrupt+0x160/0x160 [10615.157805] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10615.161013] ? do_wait_intr+0xf0/0xf0 [10615.161999] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10615.165037] ? kthread+0x1d1/0x200 [10615.168318] ? set_kthread_struct+0x70/0x70 [10615.174331] ? ret_from_fork+0x1f/0x30 [10620.056364] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [10620.061168] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10620.106323] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10620.113219] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10620.124846] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10620.133459] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10620.146443] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10620.175525] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10620.181630] RAX: 0000000000006b83 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10620.191439] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10620.208955] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10620.219286] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10620.224038] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10620.227840] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10620.232233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10620.235651] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10620.243167] Call Trace: [10620.247123] [10620.250499] ? show_regs.cold.9+0x22/0x2f [10620.261900] ? watchdog_timer_fn.cold.6+0x62/0x10a [10620.275245] ? __hrtimer_run_queues+0x1c2/0x4d0 [10620.283131] ? watchdog+0x40/0x40 [10620.284314] ? hrtimer_interrupt+0x128/0x2c0 [10620.289428] ? smp_apic_timer_interrupt+0xa6/0x270 [10620.295529] ? apic_timer_interrupt+0xf/0x20 [10620.301379] [10620.306082] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10620.318121] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10620.324149] ? newidle_balance.isra.93+0x207/0x580 [10620.329078] do_raw_spin_lock+0x177/0x190 [10620.337038] _raw_spin_lock+0x21/0x30 [10620.341326] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10620.350439] process_one_work+0x2c8/0x700 [10620.356342] worker_thread+0x296/0x6e0 [10620.358075] ? rescuer_thread+0x570/0x570 [10620.359678] kthread+0x1d1/0x200 [10620.361070] ? set_kthread_struct+0x70/0x70 [10620.363192] ret_from_fork+0x1f/0x30 [10624.488061] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 7987s! [10624.501323] Showing busy workqueues and worker pools: [10624.509444] workqueue events: flags=0x0 [10624.512364] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10624.516046] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10624.522273] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10624.525675] pending: cpu_latency_work [ptlrpc] [10624.529755] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10624.537926] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10624.546826] workqueue events_long: flags=0x0 [10624.548127] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10624.550597] pending: lu_objects_flush [obdclass] [10624.552616] workqueue events_unbound: flags=0x2 [10624.554338] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10624.557382] pending: flush_memcg_stats_dwork, idle_cull_fn [10624.559532] workqueue events_power_efficient: flags=0x80 [10624.565112] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10624.572760] pending: check_lifetime, do_cache_clean [10624.580312] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10624.584524] pending: neigh_periodic_work, neigh_periodic_work [10624.595249] workqueue mm_percpu_wq: flags=0x8 [10624.600271] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10624.604252] pending: vmstat_update, lru_add_drain_per_cpu [10624.616861] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10624.624347] pending: vmstat_update, lru_add_drain_per_cpu [10624.626466] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10624.634863] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10624.638874] workqueue xprtiod: flags=0xa [10624.641219] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10624.646926] pending: xprt_autoclose, xs_error_handle [10624.652017] workqueue ipv6_addrconf: flags=0x40008 [10624.653598] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10624.663065] pending: addrconf_verify_work [10624.665325] workqueue knbd0-recv: flags=0x1a [10624.672268] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10624.678144] in-flight: 651:recv_work [10624.683105] workqueue ptlrpc_pinger: flags=0x2 [10624.684621] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10624.693247] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10624.700240] pool 8: cpus=0-3 flags=0x4 nice=0 hung=7987s workers=4 idle: 15979 31824 8 [10624.705747] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10615s workers=2 idle: 654 [10640.055027] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [10640.058584] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10640.058773] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [10640.058810] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [10640.058815] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10640.058868] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10640.058873] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10640.058875] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10640.058889] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10640.058894] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10640.058898] RAX: 000000000000358b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10640.058901] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10640.058903] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10640.058905] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10640.058907] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10640.058910] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10640.058914] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10640.058916] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10640.058925] Call Trace: [10640.058929] [10640.058934] ? show_regs.cold.9+0x22/0x2f [10640.058943] ? watchdog_timer_fn.cold.6+0x62/0x10a [10640.058949] ? __hrtimer_run_queues+0x1c2/0x4d0 [10640.058953] ? watchdog+0x40/0x40 [10640.058957] ? hrtimer_interrupt+0x128/0x2c0 [10640.058961] ? smp_apic_timer_interrupt+0xa6/0x270 [10640.058968] ? apic_timer_interrupt+0xf/0x20 [10640.058972] [10640.058975] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10640.058979] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10640.058985] do_raw_spin_lock+0x177/0x190 [10640.058990] _raw_spin_lock+0x21/0x30 [10640.058995] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10640.059254] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10640.059450] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10640.065804] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10640.066037] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10640.066258] ? woken_wake_function+0x30/0x30 [10640.066269] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10640.066480] kthread+0x1d1/0x200 [10640.066486] ? set_kthread_struct+0x70/0x70 [10640.066490] ret_from_fork+0x1f/0x30 [10640.080482] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10640.086725] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10640.086798] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10640.086806] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10640.086808] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10640.086821] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10640.086825] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10640.086829] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10640.086832] RDX: 0000000000005693 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10640.086834] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10640.086836] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10640.086838] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10640.086841] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10640.086844] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10640.086847] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10640.086855] Call Trace: [10640.086861] [10640.086867] ? show_regs.cold.9+0x22/0x2f [10640.086876] ? watchdog_timer_fn.cold.6+0x62/0x10a [10640.086882] ? __hrtimer_run_queues+0x1c2/0x4d0 [10640.086886] ? watchdog+0x40/0x40 [10640.086890] ? hrtimer_interrupt+0x128/0x2c0 [10640.086894] ? smp_apic_timer_interrupt+0xa6/0x270 [10640.086901] ? apic_timer_interrupt+0xf/0x20 [10640.086906] [10640.086909] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10640.086913] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10640.086918] do_raw_spin_lock+0x177/0x190 [10640.087052] _raw_spin_lock+0x21/0x30 [10640.087061] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10640.094591] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10640.094601] RIP: 0010:native_safe_halt+0x2e/0x50 [10640.094615] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10640.094620] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10640.094626] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10640.094628] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10640.094630] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10640.094632] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10640.094635] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10640.094638] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10640.094642] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10640.094644] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10640.094653] Call Trace: [10640.094658] [10640.094664] ? show_regs.cold.9+0x22/0x2f [10640.094673] ? watchdog_timer_fn.cold.6+0x62/0x10a [10640.094679] ? __hrtimer_run_queues+0x1c2/0x4d0 [10640.094684] ? watchdog+0x40/0x40 [10640.094688] ? hrtimer_interrupt+0x128/0x2c0 [10640.094692] ? smp_apic_timer_interrupt+0xa6/0x270 [10640.094698] ? apic_timer_interrupt+0xf/0x20 [10640.094703] [10640.094706] ? native_safe_halt+0x2e/0x50 [10640.094710] kvm_wait+0xcd/0xe0 [10640.094718] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10640.094728] do_raw_spin_lock+0x177/0x190 [10640.094733] _raw_spin_lock+0x21/0x30 [10640.094737] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10640.095140] ? do_raw_spin_unlock+0x75/0x190 [10640.095146] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10640.136704] obd_mod_free+0x46/0x140 [ptlrpc] [10640.137274] mdc_commit_open+0x91/0xb0 [mdc] [10640.137310] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10640.137595] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10640.137790] after_reply+0xe2c/0x1780 [ptlrpc] [10640.147576] ? do_wait_intr+0xf0/0xf0 [10640.147599] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10640.147856] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10640.148290] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10640.148563] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10640.148671] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10640.148708] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10640.148738] mdc_intent_lock+0x353/0x7e0 [mdc] [10640.148768] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10640.149067] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10640.153905] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10640.154358] ? __next_timer_interrupt+0x160/0x160 [10640.154369] ptlrpcd+0xa82/0xe40 [ptlrpc] [10640.154679] ? do_wait_intr+0xf0/0xf0 [10640.154689] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10640.154890] kthread+0x1d1/0x200 [10640.154896] ? set_kthread_struct+0x70/0x70 [10640.154900] ret_from_fork+0x1f/0x30 [10640.968989] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10640.975663] lmv_intent_open+0x338/0x1150 [lmv] [10640.978993] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10640.981998] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10640.985819] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10640.988771] ll_intent_lock+0xc5/0xaa0 [lustre] [10640.996798] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10641.006073] ll_lookup_it+0xe0e/0x2270 [lustre] [10641.009604] ? do_raw_spin_unlock+0x75/0x190 [10641.012786] ? _raw_spin_unlock+0x12/0x30 [10641.018443] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10641.019744] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10641.021108] ll_atomic_open+0x36a/0xd70 [lustre] [10641.025747] lookup_open+0x356/0xb40 [10641.028269] do_last+0x313/0xfc0 [10641.031911] ? nd_jump_root+0xe5/0x160 [10641.034803] ? path_init+0x437/0x520 [10641.049300] path_openat+0xf7/0x500 [10641.051162] ? do_raw_spin_unlock+0x75/0x190 [10641.054605] do_filp_open+0x99/0x140 [10641.057283] ? getname_flags+0x6e/0x330 [10641.059221] ? __check_object_size+0xff/0x256 [10641.061740] ? do_raw_spin_unlock+0x75/0x190 [10641.063940] ? _raw_spin_unlock+0x12/0x30 [10641.071169] do_sys_openat2+0x2b4/0x410 [10641.074683] do_sys_open+0x73/0xa0 [10641.076938] __x64_sys_openat+0x24/0x30 [10641.078976] do_syscall_64+0xc1/0x440 [10641.083424] entry_SYSCALL_64_after_hwframe+0x49/0xae [10641.085321] RIP: 0033:0x7f53425a5332 [10641.087874] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10641.100664] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10641.104069] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10641.111167] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10641.115178] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10641.120241] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10641.124818] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10652.053080] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [10652.061821] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10652.125763] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10652.131466] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10652.140524] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10652.148379] RIP: 0010:native_safe_halt+0x2e/0x50 [10652.149882] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10652.165886] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10652.169937] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10652.177771] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [10652.192617] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10652.198408] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10652.201150] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10652.213694] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10652.222517] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10652.226576] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10652.240211] Call Trace: [10652.241147] [10652.241902] ? show_regs.cold.9+0x22/0x2f [10652.244316] ? watchdog_timer_fn.cold.6+0x62/0x10a [10652.247280] ? __hrtimer_run_queues+0x1c2/0x4d0 [10652.253942] ? watchdog+0x40/0x40 [10652.257459] ? hrtimer_interrupt+0x128/0x2c0 [10652.263930] ? smp_apic_timer_interrupt+0xa6/0x270 [10652.274889] ? apic_timer_interrupt+0xf/0x20 [10652.282837] [10652.288172] ? native_safe_halt+0x2e/0x50 [10652.291324] kvm_wait+0xcd/0xe0 [10652.292320] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10652.293712] ? newidle_balance.isra.93+0x207/0x580 [10652.307420] do_raw_spin_lock+0x177/0x190 [10652.309401] _raw_spin_lock+0x21/0x30 [10652.313263] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10652.315296] process_one_work+0x2c8/0x700 [10652.346256] worker_thread+0x296/0x6e0 [10652.347603] ? rescuer_thread+0x570/0x570 [10652.350088] kthread+0x1d1/0x200 [10652.351539] ? set_kthread_struct+0x70/0x70 [10652.352817] ret_from_fork+0x1f/0x30 [10655.200059] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8018s! [10655.203867] Showing busy workqueues and worker pools: [10655.206386] workqueue events: flags=0x0 [10655.208310] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10655.210999] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10655.220668] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10655.226583] pending: cpu_latency_work [ptlrpc] [10655.230632] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10655.236476] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10655.245532] workqueue events_long: flags=0x0 [10655.247788] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10655.251088] pending: lu_objects_flush [obdclass] [10655.254311] workqueue events_unbound: flags=0x2 [10655.258344] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10655.261578] pending: flush_memcg_stats_dwork, idle_cull_fn [10655.265928] workqueue events_power_efficient: flags=0x80 [10655.273357] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10655.285311] pending: check_lifetime, do_cache_clean [10655.292593] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10655.298466] pending: neigh_periodic_work, neigh_periodic_work [10655.306910] workqueue mm_percpu_wq: flags=0x8 [10655.311608] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10655.319641] pending: vmstat_update, lru_add_drain_per_cpu [10655.322488] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10655.330689] pending: vmstat_update, lru_add_drain_per_cpu [10655.335738] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10655.341963] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10655.347822] workqueue xprtiod: flags=0xa [10655.350575] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10655.356525] pending: xprt_autoclose, xs_error_handle [10655.359630] workqueue ipv6_addrconf: flags=0x40008 [10655.361562] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10655.368652] pending: addrconf_verify_work [10655.371624] workqueue knbd0-recv: flags=0x1a [10655.374052] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10655.378681] in-flight: 651:recv_work [10655.381483] workqueue ptlrpc_pinger: flags=0x2 [10655.386435] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10655.390594] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10655.399614] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8018s workers=4 idle: 15979 31824 8 [10655.407878] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10645s workers=2 idle: 654 [10668.055018] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [10668.059492] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [10668.060253] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [10668.060260] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver [10668.061566] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10668.061635] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10668.061643] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10668.061645] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10668.061659] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10668.061663] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10668.061668] RAX: 0000000000005296 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10668.061670] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10668.061673] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10668.061675] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10668.061677] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10668.061680] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10668.061684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10668.061686] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10668.061695] Call Trace: [10668.061700] [10668.061705] ? show_regs.cold.9+0x22/0x2f [10668.061896] ? watchdog_timer_fn.cold.6+0x62/0x10a [10668.061904] ? __hrtimer_run_queues+0x1c2/0x4d0 [10668.061908] ? watchdog+0x40/0x40 [10668.061912] ? hrtimer_interrupt+0x128/0x2c0 [10668.060302] intel_rapl_msr [10668.061915] ? smp_apic_timer_interrupt+0xa6/0x270 [10668.061924] intel_rapl_common sb_edac [10668.061926] ? apic_timer_interrupt+0xf/0x20 [10668.061931] rapl i2c_piix4 [10668.061937] [10668.061938] pcspkr squashfs ata_generic [10668.061944] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10668.061947] crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw [10668.061955] do_raw_spin_lock+0x177/0x190 [10668.061961] libata dm_mirror dm_region_hash [10668.061965] _raw_spin_lock+0x21/0x30 [10668.061969] dm_log dm_mod sha512_ssse3 sha512_generic [10668.061984] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10668.061988] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10668.061990] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10668.061997] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10668.062000] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10668.062004] RAX: 0000000000007d8b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10668.062006] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10668.062009] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10668.062011] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10668.062014] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10668.062025] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10668.062028] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10668.062030] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10668.062035] Call Trace: [10668.062037] [10668.062040] ? show_regs.cold.9+0x22/0x2f [10668.062129] ? watchdog_timer_fn.cold.6+0x62/0x10a [10668.062134] ? __hrtimer_run_queues+0x1c2/0x4d0 [10668.062138] ? watchdog+0x40/0x40 [10668.062142] ? hrtimer_interrupt+0x128/0x2c0 [10668.062146] ? smp_apic_timer_interrupt+0xa6/0x270 [10668.062152] ? apic_timer_interrupt+0xf/0x20 [10668.062156] [10668.062159] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10668.062163] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10668.062167] do_raw_spin_lock+0x177/0x190 [10668.062172] _raw_spin_lock+0x21/0x30 [10668.062176] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10668.061975] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10668.062889] ? do_raw_spin_unlock+0x75/0x190 [10668.062896] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10668.063461] ? do_wait_intr+0xf0/0xf0 [10668.063471] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10668.063226] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10668.063832] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10668.064168] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10668.064472] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10668.065024] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10668.065438] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10668.065549] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10668.065588] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10668.065620] mdc_intent_lock+0x353/0x7e0 [mdc] [10668.065649] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10668.065905] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10668.066173] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10668.066372] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10668.066403] lmv_intent_open+0x338/0x1150 [lmv] [10668.066432] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10668.066530] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10668.066737] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10668.066835] ll_intent_lock+0xc5/0xaa0 [lustre] [10668.066915] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10668.066994] ll_lookup_it+0xe0e/0x2270 [lustre] [10668.067071] ? do_raw_spin_unlock+0x75/0x190 [10668.067079] ? _raw_spin_unlock+0x12/0x30 [10668.067085] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10668.067258] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10668.067399] ll_atomic_open+0x36a/0xd70 [lustre] [10668.067488] lookup_open+0x356/0xb40 [10668.067498] do_last+0x313/0xfc0 [10668.067503] ? nd_jump_root+0xe5/0x160 [10668.067508] ? path_init+0x437/0x520 [10668.067512] path_openat+0xf7/0x500 [10668.067517] ? do_raw_spin_unlock+0x75/0x190 [10668.067521] do_filp_open+0x99/0x140 [10668.067526] ? getname_flags+0x6e/0x330 [10668.067531] ? __check_object_size+0xff/0x256 [10668.067538] ? do_raw_spin_unlock+0x75/0x190 [10668.067542] ? _raw_spin_unlock+0x12/0x30 [10668.067546] do_sys_openat2+0x2b4/0x410 [10668.067551] do_sys_open+0x73/0xa0 [10668.067556] __x64_sys_openat+0x24/0x30 [10668.067560] do_syscall_64+0xc1/0x440 [10668.067566] entry_SYSCALL_64_after_hwframe+0x49/0xae [10668.067572] RIP: 0033:0x7f53425a5332 [10668.067580] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10668.067583] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10668.067588] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10668.067591] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10668.067594] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10668.067596] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10668.067598] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10668.067705] ? woken_wake_function+0x30/0x30 [10668.067714] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10668.069137] kthread+0x1d1/0x200 [10668.069146] ? set_kthread_struct+0x70/0x70 [10668.069150] ret_from_fork+0x1f/0x30 [10668.977599] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10669.027958] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10669.037029] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10669.064092] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10669.074568] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10669.103054] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10669.120232] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10669.132123] RDX: 0000000000005921 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10669.149176] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10669.162221] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10669.168126] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10669.173454] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10669.182532] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10669.186988] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10669.196182] Call Trace: [10669.201090] [10669.218155] ? show_regs.cold.9+0x22/0x2f [10669.219666] ? watchdog_timer_fn.cold.6+0x62/0x10a [10669.236690] ? __hrtimer_run_queues+0x1c2/0x4d0 [10669.256014] ? watchdog+0x40/0x40 [10669.266825] ? hrtimer_interrupt+0x128/0x2c0 [10669.278060] ? smp_apic_timer_interrupt+0xa6/0x270 [10669.279466] ? apic_timer_interrupt+0xf/0x20 [10669.304293] [10669.308001] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10669.311101] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10669.316953] do_raw_spin_lock+0x177/0x190 [10669.320989] _raw_spin_lock+0x21/0x30 [10669.326072] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10669.329977] obd_mod_free+0x46/0x140 [ptlrpc] [10669.335531] mdc_commit_open+0x91/0xb0 [mdc] [10669.343097] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10669.354910] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10669.364461] after_reply+0xe2c/0x1780 [ptlrpc] [10669.376263] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10669.381931] ? __next_timer_interrupt+0x160/0x160 [10669.392622] ptlrpcd+0xa82/0xe40 [ptlrpc] [10669.397379] ? do_wait_intr+0xf0/0xf0 [10669.402711] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10669.405096] kthread+0x1d1/0x200 [10669.411164] ? set_kthread_struct+0x70/0x70 [10669.413923] ret_from_fork+0x1f/0x30 [10680.052021] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [10680.064154] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10680.088233] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10680.097344] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10680.114410] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10680.118967] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10680.122359] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10680.136071] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10680.141332] RAX: 0000000000004446 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10680.149687] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10680.157932] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10680.161698] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10680.172076] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10680.178981] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10680.182378] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10680.184733] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10680.190406] Call Trace: [10680.193399] [10680.195307] ? show_regs.cold.9+0x22/0x2f [10680.201052] ? watchdog_timer_fn.cold.6+0x62/0x10a [10680.208108] ? __hrtimer_run_queues+0x1c2/0x4d0 [10680.212485] ? watchdog+0x40/0x40 [10680.214730] ? hrtimer_interrupt+0x128/0x2c0 [10680.218122] ? smp_apic_timer_interrupt+0xa6/0x270 [10680.220573] ? apic_timer_interrupt+0xf/0x20 [10680.222629] [10680.223639] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10680.229133] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10680.232159] ? newidle_balance.isra.93+0x207/0x580 [10680.234720] do_raw_spin_lock+0x177/0x190 [10680.236940] _raw_spin_lock+0x21/0x30 [10680.238592] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10680.242890] process_one_work+0x2c8/0x700 [10680.245004] worker_thread+0x296/0x6e0 [10680.246903] ? rescuer_thread+0x570/0x570 [10680.250729] kthread+0x1d1/0x200 [10680.253648] ? set_kthread_struct+0x70/0x70 [10680.256580] ret_from_fork+0x1f/0x30 [10685.924242] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8048s! [10685.935043] Showing busy workqueues and worker pools: [10685.939797] workqueue events: flags=0x0 [10685.942885] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10685.950185] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10685.957147] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10685.962761] pending: cpu_latency_work [ptlrpc] [10685.967379] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10685.978932] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10685.988274] workqueue events_long: flags=0x0 [10685.994412] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10685.997558] pending: lu_objects_flush [obdclass] [10686.000093] workqueue events_unbound: flags=0x2 [10686.001875] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10686.008196] pending: flush_memcg_stats_dwork, idle_cull_fn [10686.013762] workqueue events_power_efficient: flags=0x80 [10686.018970] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10686.022896] pending: check_lifetime, do_cache_clean [10686.025866] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10686.028989] pending: neigh_periodic_work, neigh_periodic_work [10686.031910] workqueue mm_percpu_wq: flags=0x8 [10686.033896] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10686.037863] pending: vmstat_update, lru_add_drain_per_cpu [10686.040211] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10686.043780] pending: vmstat_update, lru_add_drain_per_cpu [10686.046531] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10686.050579] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10686.053623] workqueue xprtiod: flags=0xa [10686.055477] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10686.058016] pending: xprt_autoclose, xs_error_handle [10686.061593] workqueue ipv6_addrconf: flags=0x40008 [10686.063816] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10686.067883] pending: addrconf_verify_work [10686.070271] workqueue knbd0-recv: flags=0x1a [10686.072259] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10686.074961] in-flight: 651:recv_work [10686.076855] workqueue ptlrpc_pinger: flags=0x2 [10686.078791] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10686.081304] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10686.084293] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8048s workers=4 idle: 15979 31824 8 [10686.088797] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10676s workers=2 idle: 654 [10696.058024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [10696.059630] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [10696.060632] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [10696.060807] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10696.060901] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10696.060907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10696.060909] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10696.060924] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10696.060933] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10696.060937] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10696.060939] RDX: 0000000000007e8d RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10696.060941] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10696.060943] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10696.060945] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10696.060948] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10696.060951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10696.060954] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10696.060961] Call Trace: [10696.060967] [10696.060979] ? show_regs.cold.9+0x22/0x2f [10696.060989] ? watchdog_timer_fn.cold.6+0x62/0x10a [10696.060996] ? __hrtimer_run_queues+0x1c2/0x4d0 [10696.061001] ? watchdog+0x40/0x40 [10696.061005] ? hrtimer_interrupt+0x128/0x2c0 [10696.061009] ? smp_apic_timer_interrupt+0xa6/0x270 [10696.061018] ? apic_timer_interrupt+0xf/0x20 [10696.061022] [10696.061025] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10696.061030] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10696.061036] do_raw_spin_lock+0x177/0x190 [10696.061040] _raw_spin_lock+0x21/0x30 [10696.061047] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10696.061378] obd_mod_free+0x46/0x140 [ptlrpc] [10696.062483] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10696.062544] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10696.062549] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10696.062551] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10696.062560] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10696.062563] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10696.062316] mdc_commit_open+0x91/0xb0 [mdc] [10696.062569] RAX: 0000000000004685 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10696.062573] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10696.062575] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10696.062577] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10696.062579] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10696.062581] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10696.062584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10696.062589] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10696.062598] Call Trace: [10696.062601] [10696.062605] ? show_regs.cold.9+0x22/0x2f [10696.062612] ? watchdog_timer_fn.cold.6+0x62/0x10a [10696.062618] ? __hrtimer_run_queues+0x1c2/0x4d0 [10696.062622] ? watchdog+0x40/0x40 [10696.062626] ? hrtimer_interrupt+0x128/0x2c0 [10696.062629] ? smp_apic_timer_interrupt+0xa6/0x270 [10696.062636] ? apic_timer_interrupt+0xf/0x20 [10696.062640] [10696.062642] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10696.062647] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10696.062651] do_raw_spin_lock+0x177/0x190 [10696.062656] _raw_spin_lock+0x21/0x30 [10696.062660] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10696.062966] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10696.063176] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10696.063404] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10696.063654] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10696.064069] ? woken_wake_function+0x30/0x30 [10696.062592] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10696.064078] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10696.069056] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10696.072981] after_reply+0xe2c/0x1780 [ptlrpc] [10696.077463] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10696.079330] ? __next_timer_interrupt+0x160/0x160 [10696.079355] ptlrpcd+0xa82/0xe40 [ptlrpc] [10696.079588] ? do_wait_intr+0xf0/0xf0 [10696.079598] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10696.079825] kthread+0x1d1/0x200 [10696.079837] ? set_kthread_struct+0x70/0x70 [10696.079841] ret_from_fork+0x1f/0x30 [10696.080073] Modules linked in: lustre(O) [10696.086321] kthread+0x1d1/0x200 [10696.086354] ? set_kthread_struct+0x70/0x70 [10696.086358] ret_from_fork+0x1f/0x30 [10696.824060] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10696.885389] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10696.900888] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10696.918168] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10696.919587] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10696.951124] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10696.959342] RAX: 0000000000005a50 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10696.971344] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10696.975517] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10696.980699] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10696.984972] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10696.999431] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10697.009792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10697.013800] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10697.015326] Call Trace: [10697.015928] [10697.024956] ? show_regs.cold.9+0x22/0x2f [10697.026874] ? watchdog_timer_fn.cold.6+0x62/0x10a [10697.034389] ? __hrtimer_run_queues+0x1c2/0x4d0 [10697.039403] ? watchdog+0x40/0x40 [10697.042331] ? hrtimer_interrupt+0x128/0x2c0 [10697.050611] ? smp_apic_timer_interrupt+0xa6/0x270 [10697.055298] ? apic_timer_interrupt+0xf/0x20 [10697.056478] [10697.060576] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10697.067797] do_raw_spin_lock+0x177/0x190 [10697.070934] _raw_spin_lock+0x21/0x30 [10697.075950] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10697.079070] ? do_raw_spin_unlock+0x75/0x190 [10697.081621] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10697.082956] ? do_wait_intr+0xf0/0xf0 [10697.083845] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10697.090037] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10697.092947] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10697.096589] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10697.100622] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10697.105391] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10697.107409] mdc_intent_lock+0x353/0x7e0 [mdc] [10697.111108] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10697.115664] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10697.120027] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10697.121735] lmv_intent_open+0x338/0x1150 [lmv] [10697.124828] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10697.127754] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10697.132272] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10697.135699] ll_intent_lock+0xc5/0xaa0 [lustre] [10697.137300] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10697.142412] ll_lookup_it+0xe0e/0x2270 [lustre] [10697.145106] ? do_raw_spin_unlock+0x75/0x190 [10697.146592] ? _raw_spin_unlock+0x12/0x30 [10697.149496] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10697.157189] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10697.161359] ll_atomic_open+0x36a/0xd70 [lustre] [10697.168268] lookup_open+0x356/0xb40 [10697.169202] do_last+0x313/0xfc0 [10697.169964] ? nd_jump_root+0xe5/0x160 [10697.173095] ? path_init+0x437/0x520 [10697.175534] path_openat+0xf7/0x500 [10697.178039] ? do_raw_spin_unlock+0x75/0x190 [10697.179525] do_filp_open+0x99/0x140 [10697.180235] ? getname_flags+0x6e/0x330 [10697.181195] ? __check_object_size+0xff/0x256 [10697.184647] ? do_raw_spin_unlock+0x75/0x190 [10697.197201] ? _raw_spin_unlock+0x12/0x30 [10697.200272] do_sys_openat2+0x2b4/0x410 [10697.207938] do_sys_open+0x73/0xa0 [10697.215853] __x64_sys_openat+0x24/0x30 [10697.219685] do_syscall_64+0xc1/0x440 [10697.222286] entry_SYSCALL_64_after_hwframe+0x49/0xae [10697.228920] RIP: 0033:0x7f53425a5332 [10697.229983] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10697.246783] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10697.251122] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10697.254655] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10697.262838] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10697.266048] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10697.267588] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10712.053331] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [10712.059635] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10712.099185] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10712.109423] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10712.113593] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10712.116882] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10712.120031] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10712.128381] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10712.135028] RAX: 0000000000006016 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10712.145025] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10712.153395] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10712.161619] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10712.166714] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10712.177176] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10712.185498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10712.191416] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10712.196591] Call Trace: [10712.198356] [10712.199110] ? show_regs.cold.9+0x22/0x2f [10712.202611] ? watchdog_timer_fn.cold.6+0x62/0x10a [10712.206026] ? __hrtimer_run_queues+0x1c2/0x4d0 [10712.214412] ? watchdog+0x40/0x40 [10712.218547] ? hrtimer_interrupt+0x128/0x2c0 [10712.226561] ? smp_apic_timer_interrupt+0xa6/0x270 [10712.230930] ? apic_timer_interrupt+0xf/0x20 [10712.232700] [10712.233967] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10712.236137] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10712.242022] ? newidle_balance.isra.93+0x207/0x580 [10712.244878] do_raw_spin_lock+0x177/0x190 [10712.246437] _raw_spin_lock+0x21/0x30 [10712.248053] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10712.250766] process_one_work+0x2c8/0x700 [10712.255141] worker_thread+0x296/0x6e0 [10712.258107] ? rescuer_thread+0x570/0x570 [10712.261584] kthread+0x1d1/0x200 [10712.266263] ? set_kthread_struct+0x70/0x70 [10712.279345] ret_from_fork+0x1f/0x30 [10716.132360] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8079s! [10716.153526] Showing busy workqueues and worker pools: [10716.167140] workqueue events: flags=0x0 [10716.177340] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10716.186611] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10716.197803] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10716.212746] pending: cpu_latency_work [ptlrpc] [10716.222789] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10716.233737] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10716.247194] workqueue events_long: flags=0x0 [10716.248452] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10716.255018] pending: lu_objects_flush [obdclass] [10716.257742] workqueue events_unbound: flags=0x2 [10716.267470] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10716.274655] pending: flush_memcg_stats_dwork, idle_cull_fn [10716.276693] workqueue events_power_efficient: flags=0x80 [10716.284540] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10716.292166] pending: check_lifetime, do_cache_clean [10716.297304] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10716.307573] pending: neigh_periodic_work, neigh_periodic_work [10716.314679] workqueue mm_percpu_wq: flags=0x8 [10716.316159] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10716.328425] pending: vmstat_update, lru_add_drain_per_cpu [10716.338172] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10716.344830] pending: vmstat_update, lru_add_drain_per_cpu [10716.348800] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10716.361018] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10716.362799] workqueue xprtiod: flags=0xa [10716.369706] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10716.371467] pending: xprt_autoclose, xs_error_handle [10716.374951] workqueue ipv6_addrconf: flags=0x40008 [10716.376914] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10716.383007] pending: addrconf_verify_work [10716.384633] workqueue knbd0-recv: flags=0x1a [10716.386017] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10716.405378] in-flight: 651:recv_work [10716.406552] workqueue ptlrpc_pinger: flags=0x2 [10716.417033] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10716.436600] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10716.438810] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8079s workers=4 idle: 15979 31824 8 [10716.449330] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10706s workers=2 idle: 654 [10724.055085] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [10724.058952] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [10724.060595] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10724.060677] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10724.060685] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10724.060688] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10724.060703] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10724.060707] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10724.060712] RAX: 00000000000069dd RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10724.060715] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10724.060717] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10724.060720] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10724.060723] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10724.060726] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10724.060730] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10724.062350] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [10724.062359] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10724.062425] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10724.062430] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10724.062433] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10724.062444] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10724.062448] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10724.062453] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10724.062455] RDX: 000000000000661e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10724.062458] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10724.062460] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10724.062462] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10724.062465] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10724.062469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10724.062472] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10724.062482] Call Trace: [10724.062487] [10724.062493] ? show_regs.cold.9+0x22/0x2f [10724.062503] ? watchdog_timer_fn.cold.6+0x62/0x10a [10724.062510] ? __hrtimer_run_queues+0x1c2/0x4d0 [10724.062514] ? watchdog+0x40/0x40 [10724.062519] ? hrtimer_interrupt+0x128/0x2c0 [10724.062523] ? smp_apic_timer_interrupt+0xa6/0x270 [10724.062531] ? apic_timer_interrupt+0xf/0x20 [10724.062535] [10724.062538] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10724.062542] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10724.062546] do_raw_spin_lock+0x177/0x190 [10724.062550] _raw_spin_lock+0x21/0x30 [10724.062555] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10724.065113] obd_mod_free+0x46/0x140 [ptlrpc] [10724.065321] mdc_commit_open+0x91/0xb0 [mdc] [10724.065355] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10724.065554] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10724.065774] after_reply+0xe2c/0x1780 [ptlrpc] [10724.065966] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10724.066158] ? __next_timer_interrupt+0x160/0x160 [10724.066167] ptlrpcd+0xa82/0xe40 [ptlrpc] [10724.066357] ? do_wait_intr+0xf0/0xf0 [10724.066367] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10724.066562] kthread+0x1d1/0x200 [10724.066568] ? set_kthread_struct+0x70/0x70 [10724.066571] ret_from_fork+0x1f/0x30 [10724.075315] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10724.075392] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10724.075399] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10724.075402] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10724.075417] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10724.075423] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10724.075427] RAX: 0000000000007e50 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10724.075430] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10724.075432] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10724.075435] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10724.075437] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10724.075441] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10724.075444] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10724.075446] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10724.075455] Call Trace: [10724.075461] [10724.075467] ? show_regs.cold.9+0x22/0x2f [10724.075476] ? watchdog_timer_fn.cold.6+0x62/0x10a [10724.075482] ? __hrtimer_run_queues+0x1c2/0x4d0 [10724.075486] ? watchdog+0x40/0x40 [10724.075490] ? hrtimer_interrupt+0x128/0x2c0 [10724.075494] ? smp_apic_timer_interrupt+0xa6/0x270 [10724.075502] ? apic_timer_interrupt+0xf/0x20 [10724.075506] [10724.075509] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10724.075514] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10724.075519] do_raw_spin_lock+0x177/0x190 [10724.075524] _raw_spin_lock+0x21/0x30 [10724.075528] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10724.109538] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10724.109555] Call Trace: [10724.109561] [10724.109567] ? show_regs.cold.9+0x22/0x2f [10724.109580] ? watchdog_timer_fn.cold.6+0x62/0x10a [10724.109587] ? __hrtimer_run_queues+0x1c2/0x4d0 [10724.109592] ? watchdog+0x40/0x40 [10724.109596] ? hrtimer_interrupt+0x128/0x2c0 [10724.109600] ? smp_apic_timer_interrupt+0xa6/0x270 [10724.109609] ? apic_timer_interrupt+0xf/0x20 [10724.109613] [10724.109617] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10724.109628] do_raw_spin_lock+0x177/0x190 [10724.109633] _raw_spin_lock+0x21/0x30 [10724.109639] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10724.109917] ? do_raw_spin_unlock+0x75/0x190 [10724.109925] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10724.110121] ? do_wait_intr+0xf0/0xf0 [10724.110131] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10724.110330] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10724.110546] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10724.110771] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10724.110890] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10724.110931] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10724.110961] mdc_intent_lock+0x353/0x7e0 [mdc] [10724.110990] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10724.111074] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10724.111287] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10724.111318] lmv_intent_open+0x338/0x1150 [lmv] [10724.111348] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10724.111430] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10724.111518] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10724.111542] ll_intent_lock+0xc5/0xaa0 [lustre] [10724.111625] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10724.111704] ll_lookup_it+0xe0e/0x2270 [lustre] [10724.112392] ? do_raw_spin_unlock+0x75/0x190 [10724.112398] ? _raw_spin_unlock+0x12/0x30 [10724.112404] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10724.112434] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10724.112453] ll_atomic_open+0x36a/0xd70 [lustre] [10724.112541] lookup_open+0x356/0xb40 [10724.112551] do_last+0x313/0xfc0 [10724.112556] ? nd_jump_root+0xe5/0x160 [10724.112562] ? path_init+0x437/0x520 [10724.112566] path_openat+0xf7/0x500 [10724.112571] ? do_raw_spin_unlock+0x75/0x190 [10724.112577] do_filp_open+0x99/0x140 [10724.112583] ? getname_flags+0x6e/0x330 [10724.112588] ? __check_object_size+0xff/0x256 [10724.112596] ? do_raw_spin_unlock+0x75/0x190 [10724.112600] ? _raw_spin_unlock+0x12/0x30 [10724.112605] do_sys_openat2+0x2b4/0x410 [10724.112609] do_sys_open+0x73/0xa0 [10724.112614] __x64_sys_openat+0x24/0x30 [10724.112618] do_syscall_64+0xc1/0x440 [10724.112625] entry_SYSCALL_64_after_hwframe+0x49/0xae [10724.112631] RIP: 0033:0x7f53425a5332 [10724.112640] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10724.112644] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10724.112650] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10724.112653] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10724.112655] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10724.112658] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10724.112660] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10724.757195] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10724.767407] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10724.777980] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10724.786033] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10724.791477] ? woken_wake_function+0x30/0x30 [10724.798152] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10724.810312] kthread+0x1d1/0x200 [10724.811461] ? set_kthread_struct+0x70/0x70 [10724.816332] ret_from_fork+0x1f/0x30 [10744.052519] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [10744.060895] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10744.108716] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10744.118704] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10744.122924] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10744.129659] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10744.135205] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10744.151725] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10744.155935] RAX: 0000000000007a90 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10744.160624] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10744.165945] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10744.168914] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10744.172072] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10744.175644] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10744.179139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10744.182962] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10744.187345] Call Trace: [10744.189117] [10744.190273] ? show_regs.cold.9+0x22/0x2f [10744.193900] ? watchdog_timer_fn.cold.6+0x62/0x10a [10744.197283] ? __hrtimer_run_queues+0x1c2/0x4d0 [10744.200172] ? watchdog+0x40/0x40 [10744.201933] ? hrtimer_interrupt+0x128/0x2c0 [10744.204485] ? smp_apic_timer_interrupt+0xa6/0x270 [10744.207285] ? apic_timer_interrupt+0xf/0x20 [10744.209429] [10744.210710] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10744.213688] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10744.216745] ? newidle_balance.isra.93+0x207/0x580 [10744.219630] do_raw_spin_lock+0x177/0x190 [10744.221941] _raw_spin_lock+0x21/0x30 [10744.224248] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10744.227590] process_one_work+0x2c8/0x700 [10744.229941] worker_thread+0x296/0x6e0 [10744.232243] ? rescuer_thread+0x570/0x570 [10744.234450] kthread+0x1d1/0x200 [10744.235699] ? set_kthread_struct+0x70/0x70 [10744.237713] ret_from_fork+0x1f/0x30 [10746.849127] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8109s! [10746.859709] Showing busy workqueues and worker pools: [10746.865574] workqueue events: flags=0x0 [10746.878116] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10746.890348] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10746.916168] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10746.929414] pending: cpu_latency_work [ptlrpc] [10746.939398] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10746.953165] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10746.962346] workqueue events_long: flags=0x0 [10746.968284] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10746.979993] pending: lu_objects_flush [obdclass] [10746.986325] workqueue events_unbound: flags=0x2 [10746.991146] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10746.993162] pending: flush_memcg_stats_dwork, idle_cull_fn [10747.002996] workqueue events_power_efficient: flags=0x80 [10747.006236] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10747.011321] pending: check_lifetime, do_cache_clean [10747.024166] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10747.027305] pending: neigh_periodic_work, neigh_periodic_work [10747.031685] workqueue mm_percpu_wq: flags=0x8 [10747.037939] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10747.047219] pending: vmstat_update, lru_add_drain_per_cpu [10747.051914] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10747.055228] pending: vmstat_update, lru_add_drain_per_cpu [10747.063254] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10747.068457] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10747.074275] workqueue xprtiod: flags=0xa [10747.077971] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10747.088137] pending: xprt_autoclose, xs_error_handle [10747.097179] workqueue ipv6_addrconf: flags=0x40008 [10747.102637] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10747.113851] pending: addrconf_verify_work [10747.115215] workqueue knbd0-recv: flags=0x1a [10747.121959] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10747.128242] in-flight: 651:recv_work [10747.134324] workqueue ptlrpc_pinger: flags=0x2 [10747.138262] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10747.144914] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10747.149966] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8110s workers=4 idle: 15979 31824 8 [10747.157291] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10737s workers=2 idle: 654 [10752.055030] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [10752.058699] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10752.060446] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [10752.060838] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [10752.060844] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10752.060912] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10752.060917] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10752.060919] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10752.060934] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10752.060938] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10752.060943] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10752.060946] RDX: 0000000000007f84 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10752.060948] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10752.060951] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10752.060953] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10752.060956] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10752.060960] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10752.060962] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10752.060970] Call Trace: [10752.060976] [10752.060981] ? show_regs.cold.9+0x22/0x2f [10752.060991] ? watchdog_timer_fn.cold.6+0x62/0x10a [10752.060997] ? __hrtimer_run_queues+0x1c2/0x4d0 [10752.061001] ? watchdog+0x40/0x40 [10752.061005] ? hrtimer_interrupt+0x128/0x2c0 [10752.061009] ? smp_apic_timer_interrupt+0xa6/0x270 [10752.061018] ? apic_timer_interrupt+0xf/0x20 [10752.061023] [10752.061026] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10752.061031] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10752.061035] do_raw_spin_lock+0x177/0x190 [10752.061040] _raw_spin_lock+0x21/0x30 [10752.061046] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10752.061846] obd_mod_free+0x46/0x140 [ptlrpc] [10752.062036] mdc_commit_open+0x91/0xb0 [mdc] [10752.062074] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10752.063222] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10752.063474] after_reply+0xe2c/0x1780 [ptlrpc] [10752.063681] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10752.063896] ? __next_timer_interrupt+0x160/0x160 [10752.063906] ptlrpcd+0xa82/0xe40 [ptlrpc] [10752.064139] ? do_wait_intr+0xf0/0xf0 [10752.064150] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10752.064379] kthread+0x1d1/0x200 [10752.064386] ? set_kthread_struct+0x70/0x70 [10752.064390] ret_from_fork+0x1f/0x30 [10752.102538] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10752.102556] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10752.102559] RIP: 0010:native_safe_halt+0x2e/0x50 [10752.105574] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10752.105661] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10752.105670] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10752.105673] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10752.105688] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10752.105694] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10752.105699] RAX: 0000000000004cb7 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10752.105701] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10752.105704] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10752.105707] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10752.105710] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10752.105714] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10752.105717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10752.105720] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10752.105729] Call Trace: [10752.105737] [10752.105745] ? show_regs.cold.9+0x22/0x2f [10752.105758] ? watchdog_timer_fn.cold.6+0x62/0x10a [10752.105765] ? __hrtimer_run_queues+0x1c2/0x4d0 [10752.105770] ? watchdog+0x40/0x40 [10752.105775] ? hrtimer_interrupt+0x128/0x2c0 [10752.105779] ? smp_apic_timer_interrupt+0xa6/0x270 [10752.105788] ? apic_timer_interrupt+0xf/0x20 [10752.105793] [10752.105797] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10752.105802] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10752.105808] do_raw_spin_lock+0x177/0x190 [10752.105813] _raw_spin_lock+0x21/0x30 [10752.105818] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10752.107095] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10752.107394] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10752.107628] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10752.107863] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10752.121593] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10752.158831] ? woken_wake_function+0x30/0x30 [10752.158859] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10752.159182] kthread+0x1d1/0x200 [10752.159189] ? set_kthread_struct+0x70/0x70 [10752.159193] ret_from_fork+0x1f/0x30 [10752.723729] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10752.727770] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10752.731089] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10752.733594] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10752.738658] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10752.742052] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10752.747991] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10752.754413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10752.757508] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10752.760588] Call Trace: [10752.761567] [10752.762404] ? show_regs.cold.9+0x22/0x2f [10752.764118] ? watchdog_timer_fn.cold.6+0x62/0x10a [10752.765855] ? __hrtimer_run_queues+0x1c2/0x4d0 [10752.767693] ? watchdog+0x40/0x40 [10752.768739] ? hrtimer_interrupt+0x128/0x2c0 [10752.770446] ? smp_apic_timer_interrupt+0xa6/0x270 [10752.772959] ? apic_timer_interrupt+0xf/0x20 [10752.774809] [10752.775718] ? native_safe_halt+0x2e/0x50 [10752.777390] kvm_wait+0xcd/0xe0 [10752.778883] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10752.780918] do_raw_spin_lock+0x177/0x190 [10752.782626] _raw_spin_lock+0x21/0x30 [10752.788482] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10752.792950] ? do_raw_spin_unlock+0x75/0x190 [10752.797803] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10752.803784] ? do_wait_intr+0xf0/0xf0 [10752.807629] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10752.815447] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10752.822965] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10752.836428] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10752.838090] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10752.841709] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10752.847177] mdc_intent_lock+0x353/0x7e0 [mdc] [10752.849399] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10752.855465] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10752.859607] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10752.862921] lmv_intent_open+0x338/0x1150 [lmv] [10752.869859] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10752.879072] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10752.884377] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10752.890302] ll_intent_lock+0xc5/0xaa0 [lustre] [10752.892789] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10752.894788] ll_lookup_it+0xe0e/0x2270 [lustre] [10752.904598] ? do_raw_spin_unlock+0x75/0x190 [10752.906486] ? _raw_spin_unlock+0x12/0x30 [10752.912556] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10752.917527] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10752.919362] ll_atomic_open+0x36a/0xd70 [lustre] [10752.928846] lookup_open+0x356/0xb40 [10752.930188] do_last+0x313/0xfc0 [10752.931180] ? nd_jump_root+0xe5/0x160 [10752.936628] ? path_init+0x437/0x520 [10752.941122] path_openat+0xf7/0x500 [10752.943976] ? do_raw_spin_unlock+0x75/0x190 [10752.946222] do_filp_open+0x99/0x140 [10752.948129] ? getname_flags+0x6e/0x330 [10752.952075] ? __check_object_size+0xff/0x256 [10752.954520] ? do_raw_spin_unlock+0x75/0x190 [10752.962773] ? _raw_spin_unlock+0x12/0x30 [10752.964211] do_sys_openat2+0x2b4/0x410 [10752.966937] do_sys_open+0x73/0xa0 [10752.968068] __x64_sys_openat+0x24/0x30 [10752.969399] do_syscall_64+0xc1/0x440 [10752.973475] entry_SYSCALL_64_after_hwframe+0x49/0xae [10752.977396] RIP: 0033:0x7f53425a5332 [10752.978985] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10753.005525] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10753.019335] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10753.030288] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10753.036390] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10753.043802] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10753.049665] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10772.053295] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [10772.058837] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10772.119342] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10772.131039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10772.146216] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10772.170215] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10772.182020] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10772.214547] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10772.236822] RAX: 0000000000007962 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10772.241886] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10772.258414] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10772.269956] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10772.286480] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10772.298288] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10772.310977] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10772.322293] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10772.331689] Call Trace: [10772.338134] [10772.341762] ? show_regs.cold.9+0x22/0x2f [10772.344501] ? watchdog_timer_fn.cold.6+0x62/0x10a [10772.347733] ? __hrtimer_run_queues+0x1c2/0x4d0 [10772.351498] ? watchdog+0x40/0x40 [10772.352685] ? hrtimer_interrupt+0x128/0x2c0 [10772.354194] ? smp_apic_timer_interrupt+0xa6/0x270 [10772.356970] ? apic_timer_interrupt+0xf/0x20 [10772.358245] [10772.358809] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10772.365590] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10772.368923] ? newidle_balance.isra.93+0x207/0x580 [10772.372060] do_raw_spin_lock+0x177/0x190 [10772.374344] _raw_spin_lock+0x21/0x30 [10772.376271] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10772.385553] process_one_work+0x2c8/0x700 [10772.389974] worker_thread+0x296/0x6e0 [10772.391570] ? rescuer_thread+0x570/0x570 [10772.395045] kthread+0x1d1/0x200 [10772.396754] ? set_kthread_struct+0x70/0x70 [10772.399520] ret_from_fork+0x1f/0x30 [10777.568123] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8140s! [10777.570984] Showing busy workqueues and worker pools: [10777.574145] workqueue events: flags=0x0 [10777.577956] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10777.582077] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10777.590307] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10777.596274] pending: cpu_latency_work [ptlrpc] [10777.600203] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10777.609007] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10777.621019] workqueue events_long: flags=0x0 [10777.624274] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10777.629782] pending: lu_objects_flush [obdclass] [10777.633263] workqueue events_unbound: flags=0x2 [10777.637224] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10777.642401] pending: flush_memcg_stats_dwork, idle_cull_fn [10777.646869] workqueue events_power_efficient: flags=0x80 [10777.650218] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10777.654197] pending: check_lifetime, do_cache_clean [10777.657389] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10777.661535] pending: neigh_periodic_work, neigh_periodic_work [10777.664886] workqueue mm_percpu_wq: flags=0x8 [10777.667275] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10777.671231] pending: vmstat_update, lru_add_drain_per_cpu [10777.674676] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10777.678357] pending: vmstat_update, lru_add_drain_per_cpu [10777.681389] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10777.688048] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10777.691708] workqueue xprtiod: flags=0xa [10777.693396] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10777.695999] pending: xprt_autoclose, xs_error_handle [10777.698395] workqueue ipv6_addrconf: flags=0x40008 [10777.701123] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10777.710858] pending: addrconf_verify_work [10777.712764] workqueue knbd0-recv: flags=0x1a [10777.714387] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10777.717091] in-flight: 651:recv_work [10777.719268] workqueue ptlrpc_pinger: flags=0x2 [10777.722514] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10777.728044] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10777.735745] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8140s workers=4 idle: 15979 31824 8 [10777.745107] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10768s workers=2 idle: 654 [10780.055026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [10780.058212] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [10780.058225] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [10780.058234] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10780.058312] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10780.058318] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10780.058321] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10780.058335] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10780.058339] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10780.058344] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10780.058346] RDX: 0000000000000d02 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10780.058349] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10780.058351] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10780.058354] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10780.058357] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10780.058359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10780.059448] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10780.059460] Call Trace: [10780.059467] [10780.059473] ? show_regs.cold.9+0x22/0x2f [10780.059486] ? watchdog_timer_fn.cold.6+0x62/0x10a [10780.059493] ? __hrtimer_run_queues+0x1c2/0x4d0 [10780.059498] ? watchdog+0x40/0x40 [10780.059957] ? hrtimer_interrupt+0x128/0x2c0 [10780.059963] ? smp_apic_timer_interrupt+0xa6/0x270 [10780.060011] ? apic_timer_interrupt+0xf/0x20 [10780.060017] [10780.060020] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10780.060026] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10780.060030] do_raw_spin_lock+0x177/0x190 [10780.060036] _raw_spin_lock+0x21/0x30 [10780.060043] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10780.061270] obd_mod_free+0x46/0x140 [ptlrpc] [10780.062052] mdc_commit_open+0x91/0xb0 [mdc] [10780.062197] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10780.063217] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10780.063978] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10780.064054] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10780.064060] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10780.064063] RIP: 0010:native_safe_halt+0x2e/0x50 [10780.064075] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10780.064204] after_reply+0xe2c/0x1780 [ptlrpc] [10780.064449] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 [10780.064529] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10780.064765] ? __next_timer_interrupt+0x160/0x160 [10780.064776] ptlrpcd+0xa82/0xe40 [ptlrpc] [10780.065213] ? do_wait_intr+0xf0/0xf0 [10780.065224] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10780.065442] kthread+0x1d1/0x200 [10780.065449] ? set_kthread_struct+0x70/0x70 [10780.065453] ret_from_fork+0x1f/0x30 [10780.070638] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common [10780.074896] ORIG_RAX: ffffffffffffff13 [10780.074906] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10780.074909] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10780.074911] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10780.074913] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10780.074916] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10780.074919] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10780.074922] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10780.074925] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10780.074933] Call Trace: [10780.074939] [10780.074944] ? show_regs.cold.9+0x22/0x2f [10780.074955] ? watchdog_timer_fn.cold.6+0x62/0x10a [10780.074961] ? __hrtimer_run_queues+0x1c2/0x4d0 [10780.074965] ? watchdog+0x40/0x40 [10780.074970] ? hrtimer_interrupt+0x128/0x2c0 [10780.074974] ? smp_apic_timer_interrupt+0xa6/0x270 [10780.074981] ? apic_timer_interrupt+0xf/0x20 [10780.074986] [10780.074989] ? native_safe_halt+0x2e/0x50 [10780.074993] kvm_wait+0xcd/0xe0 [10780.074998] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10780.075007] do_raw_spin_lock+0x177/0x190 [10780.075011] _raw_spin_lock+0x21/0x30 [10780.075015] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10780.075353] ? do_raw_spin_unlock+0x75/0x190 [10780.075358] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10780.075960] ? do_wait_intr+0xf0/0xf0 [10780.075968] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10780.076175] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10780.114919] sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10780.114963] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10780.114973] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10780.114976] RIP: 0010:native_safe_halt+0x2e/0x50 [10780.114991] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10780.120336] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10780.120960] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10780.121079] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10780.121119] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10780.121149] mdc_intent_lock+0x353/0x7e0 [mdc] [10780.121182] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10780.121338] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10780.129019] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10780.129035] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10780.129041] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10780.137350] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10780.148529] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10780.148541] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10780.148544] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10780.148547] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10780.148551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10780.148554] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10780.148628] Call Trace: [10780.148636] [10780.148642] ? show_regs.cold.9+0x22/0x2f [10780.148658] ? watchdog_timer_fn.cold.6+0x62/0x10a [10780.148665] ? __hrtimer_run_queues+0x1c2/0x4d0 [10780.148670] ? watchdog+0x40/0x40 [10780.148675] ? hrtimer_interrupt+0x128/0x2c0 [10780.148679] ? smp_apic_timer_interrupt+0xa6/0x270 [10780.148687] ? apic_timer_interrupt+0xf/0x20 [10780.148693] [10780.148696] ? native_safe_halt+0x2e/0x50 [10780.148701] kvm_wait+0xcd/0xe0 [10780.148709] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10780.148719] do_raw_spin_lock+0x177/0x190 [10780.148724] _raw_spin_lock+0x21/0x30 [10780.148729] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10780.149485] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10780.150017] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10780.150831] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10780.151474] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10780.152029] ? woken_wake_function+0x30/0x30 [10780.152041] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10780.152742] kthread+0x1d1/0x200 [10780.153234] ? set_kthread_struct+0x70/0x70 [10780.153244] ret_from_fork+0x1f/0x30 [10780.625349] lmv_intent_open+0x338/0x1150 [lmv] [10780.630432] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10780.642328] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10780.650623] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10780.651954] ll_intent_lock+0xc5/0xaa0 [lustre] [10780.653452] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10780.657262] ll_lookup_it+0xe0e/0x2270 [lustre] [10780.661480] ? do_raw_spin_unlock+0x75/0x190 [10780.664426] ? _raw_spin_unlock+0x12/0x30 [10780.667628] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10780.673268] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10780.676485] ll_atomic_open+0x36a/0xd70 [lustre] [10780.678324] lookup_open+0x356/0xb40 [10780.679520] do_last+0x313/0xfc0 [10780.680683] ? nd_jump_root+0xe5/0x160 [10780.682040] ? path_init+0x437/0x520 [10780.683038] path_openat+0xf7/0x500 [10780.683860] ? do_raw_spin_unlock+0x75/0x190 [10780.685121] do_filp_open+0x99/0x140 [10780.686230] ? getname_flags+0x6e/0x330 [10780.687288] ? __check_object_size+0xff/0x256 [10780.688437] ? do_raw_spin_unlock+0x75/0x190 [10780.689654] ? _raw_spin_unlock+0x12/0x30 [10780.691365] do_sys_openat2+0x2b4/0x410 [10780.692443] do_sys_open+0x73/0xa0 [10780.693627] __x64_sys_openat+0x24/0x30 [10780.697387] do_syscall_64+0xc1/0x440 [10780.699286] entry_SYSCALL_64_after_hwframe+0x49/0xae [10780.700501] RIP: 0033:0x7f53425a5332 [10780.701988] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10780.708049] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10780.710001] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10780.713444] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10780.716571] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10780.724013] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10780.727619] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10794.634605] rcu: INFO: rcu_sched self-detected stall on CPU [10794.645029] rcu: 2-....: (4769358 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [10794.654291] (t=8700208 jiffies g=274185 q=66092) [10794.656235] rcu: rcu_sched kthread starved for 8158574 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [10794.663039] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10794.667114] rcu: RCU grace-period kthread stack dump: [10794.670804] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10794.681576] Call Trace: [10794.683871] __schedule+0x351/0xcb0 [10794.686207] schedule+0xc0/0x180 [10794.689859] schedule_timeout+0xb4/0x190 [10794.692917] ? do_raw_spin_unlock+0x75/0x190 [10794.695620] ? __next_timer_interrupt+0x160/0x160 [10794.698258] ? prepare_to_swait_event+0xf3/0x140 [10794.708140] rcu_gp_kthread+0x8dd/0x15b0 [10794.711443] ? force_qs_rnp+0x2c0/0x2c0 [10794.712687] kthread+0x1d1/0x200 [10794.715540] ? set_kthread_struct+0x70/0x70 [10794.717900] ret_from_fork+0x1f/0x30 [10794.721886] rcu: Stack dump where RCU GP kthread last ran: [10794.729765] Sending NMI from CPU 2 to CPUs 0: [10794.733981] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [10794.734938] Sending NMI from CPU 2 to CPUs 1: [10794.744751] NMI backtrace for cpu 1 [10794.744758] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10794.744761] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10794.744762] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10794.744764] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10794.744766] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [10794.744769] RAX: 0000000000004921 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10794.744771] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10794.744772] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10794.744774] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10794.744775] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10794.744777] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10794.744778] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10794.744779] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10794.744781] Call Trace: [10794.744782] [10794.744783] ? show_regs.cold.9+0x22/0x2f [10794.744784] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10794.744785] ? nmi_cpu_backtrace_handler+0x15/0x40 [10794.744786] ? nmi_handle+0xbd/0x210 [10794.744787] ? default_do_nmi+0x4b/0x1b0 [10794.744789] ? do_nmi+0x280/0x380 [10794.744790] ? end_repeat_nmi+0x16/0x1f [10794.744791] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10794.744792] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10794.744794] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10794.744795] [10794.744796] do_raw_spin_lock+0x177/0x190 [10794.744797] _raw_spin_lock+0x21/0x30 [10794.744799] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10794.744800] ? do_raw_spin_unlock+0x75/0x190 [10794.744801] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10794.744803] ? do_wait_intr+0xf0/0xf0 [10794.744804] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10794.744805] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10794.744806] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10794.744808] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10794.744809] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10794.744811] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10794.744812] mdc_intent_lock+0x353/0x7e0 [mdc] [10794.744813] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10794.744814] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10794.744815] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10794.744816] lmv_intent_open+0x338/0x1150 [lmv] [10794.744817] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10794.744818] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10794.744819] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10794.744820] ll_intent_lock+0xc5/0xaa0 [lustre] [10794.744821] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10794.744822] ll_lookup_it+0xe0e/0x2270 [lustre] [10794.744823] ? do_raw_spin_unlock+0x75/0x190 [10794.744824] ? _raw_spin_unlock+0x12/0x30 [10794.744825] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10794.744826] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10794.744828] ll_atomic_open+0x36a/0xd70 [lustre] [10794.744828] lookup_open+0x356/0xb40 [10794.744829] do_last+0x313/0xfc0 [10794.744830] ? nd_jump_root+0xe5/0x160 [10794.744831] ? path_init+0x437/0x520 [10794.744832] path_openat+0xf7/0x500 [10794.744833] ? do_raw_spin_unlock+0x75/0x190 [10794.744834] do_filp_open+0x99/0x140 [10794.744835] ? getname_flags+0x6e/0x330 [10794.744836] ? __check_object_size+0xff/0x256 [10794.744837] ? do_raw_spin_unlock+0x75/0x190 [10794.744839] ? _raw_spin_unlock+0x12/0x30 [10794.744840] do_sys_openat2+0x2b4/0x410 [10794.744841] do_sys_open+0x73/0xa0 [10794.744842] __x64_sys_openat+0x24/0x30 [10794.744844] do_syscall_64+0xc1/0x440 [10794.744845] entry_SYSCALL_64_after_hwframe+0x49/0xae [10794.744846] RIP: 0033:0x7f53425a5332 [10794.744848] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10794.744850] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10794.744853] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10794.744855] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10794.744856] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10794.744858] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10794.744859] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10794.745889] NMI backtrace for cpu 2 [10794.950940] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10794.956072] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10794.961082] Call Trace: [10794.962072] [10794.963052] ? dump_stack+0xbb/0x10e [10794.964903] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10794.966962] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10794.969287] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10794.971953] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10794.974304] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10794.977252] ? rcu_dump_cpu_stacks+0x11b/0x171 [10794.980352] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10794.982950] ? update_process_times+0x6d/0xc0 [10794.985002] ? tick_sched_handle.isra.18+0x27/0x90 [10794.988703] ? tick_sched_timer+0x53/0xb0 [10794.990846] ? __hrtimer_run_queues+0x1c2/0x4d0 [10794.993112] ? tick_sched_handle.isra.18+0x90/0x90 [10794.994966] ? hrtimer_interrupt+0x128/0x2c0 [10794.996969] ? smp_apic_timer_interrupt+0xa6/0x270 [10794.999119] ? apic_timer_interrupt+0xf/0x20 [10795.001168] [10795.002848] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10795.005436] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10795.008561] ? do_raw_spin_lock+0x177/0x190 [10795.010099] ? _raw_spin_lock+0x21/0x30 [10795.011962] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10795.014812] ? obd_mod_free+0x46/0x140 [ptlrpc] [10795.017146] ? mdc_commit_open+0x91/0xb0 [mdc] [10795.019120] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10795.021841] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10795.025095] ? after_reply+0xe2c/0x1780 [ptlrpc] [10795.027336] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10795.030380] ? __next_timer_interrupt+0x160/0x160 [10795.031888] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10795.033964] ? do_wait_intr+0xf0/0xf0 [10795.035131] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10795.039674] ? kthread+0x1d1/0x200 [10795.041425] ? set_kthread_struct+0x70/0x70 [10795.043330] ? ret_from_fork+0x1f/0x30 [10807.780805] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8170s! [10807.786230] Showing busy workqueues and worker pools: [10807.788621] workqueue events: flags=0x0 [10807.797807] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10807.806429] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10807.816180] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10807.820756] pending: cpu_latency_work [ptlrpc] [10807.823985] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10807.828639] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10807.840178] workqueue events_long: flags=0x0 [10807.844614] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10807.852091] pending: lu_objects_flush [obdclass] [10807.857161] workqueue events_unbound: flags=0x2 [10807.860070] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10807.867223] pending: flush_memcg_stats_dwork, idle_cull_fn [10807.870984] workqueue events_power_efficient: flags=0x80 [10807.874341] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10807.877196] pending: check_lifetime, do_cache_clean [10807.879573] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10807.883174] pending: neigh_periodic_work, neigh_periodic_work [10807.885964] workqueue mm_percpu_wq: flags=0x8 [10807.887853] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10807.891425] pending: vmstat_update, lru_add_drain_per_cpu [10807.894142] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10807.897633] pending: vmstat_update, lru_add_drain_per_cpu [10807.900549] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10807.905159] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10807.908870] workqueue xprtiod: flags=0xa [10807.911986] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10807.917605] pending: xprt_autoclose, xs_error_handle [10807.925441] workqueue ipv6_addrconf: flags=0x40008 [10807.927200] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10807.934561] pending: addrconf_verify_work [10807.939519] workqueue knbd0-recv: flags=0x1a [10807.943150] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10807.948076] in-flight: 651:recv_work [10807.950557] workqueue ptlrpc_pinger: flags=0x2 [10807.952343] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10807.955760] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10807.959582] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8170s workers=4 idle: 15979 31824 8 [10807.964191] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10798s workers=2 idle: 654 [10808.058032] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [10808.066148] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10808.105438] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10808.114962] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10808.120748] RIP: 0010:native_safe_halt+0x2e/0x50 [10808.127583] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10808.148713] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10808.156722] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10808.163996] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10808.171845] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10808.176444] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10808.182826] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10808.186646] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10808.197326] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10808.200750] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10808.206770] Call Trace: [10808.209840] [10808.211144] ? show_regs.cold.9+0x22/0x2f [10808.216720] ? watchdog_timer_fn.cold.6+0x62/0x10a [10808.222250] ? __hrtimer_run_queues+0x1c2/0x4d0 [10808.224992] ? watchdog+0x40/0x40 [10808.226181] ? hrtimer_interrupt+0x128/0x2c0 [10808.228440] ? smp_apic_timer_interrupt+0xa6/0x270 [10808.231237] ? apic_timer_interrupt+0xf/0x20 [10808.234968] [10808.237775] ? native_safe_halt+0x2e/0x50 [10808.240108] kvm_wait+0xcd/0xe0 [10808.241075] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10808.242882] do_raw_spin_lock+0x177/0x190 [10808.244816] _raw_spin_lock+0x21/0x30 [10808.246710] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10808.250642] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10808.253691] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10808.258545] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10808.264338] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10808.269592] ? woken_wake_function+0x30/0x30 [10808.273994] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10808.283306] kthread+0x1d1/0x200 [10808.286950] ? set_kthread_struct+0x70/0x70 [10808.293275] ret_from_fork+0x1f/0x30 [10820.055306] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [10820.061226] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10820.062517] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [10820.086893] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10820.086914] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10820.086916] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10820.086927] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10820.086930] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10820.086935] RAX: 00000000000045c1 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10820.086936] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10820.086938] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10820.086939] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10820.086941] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10820.086943] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10820.086945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10820.086946] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10820.086952] Call Trace: [10820.086957] [10820.086961] ? show_regs.cold.9+0x22/0x2f [10820.086967] ? watchdog_timer_fn.cold.6+0x62/0x10a [10820.086972] ? __hrtimer_run_queues+0x1c2/0x4d0 [10820.086974] ? watchdog+0x40/0x40 [10820.086977] ? hrtimer_interrupt+0x128/0x2c0 [10820.086979] ? smp_apic_timer_interrupt+0xa6/0x270 [10820.086985] ? apic_timer_interrupt+0xf/0x20 [10820.086988] [10820.086990] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10820.086997] do_raw_spin_lock+0x177/0x190 [10820.087000] _raw_spin_lock+0x21/0x30 [10820.087003] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10820.087289] ? do_raw_spin_unlock+0x75/0x190 [10820.087294] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10820.087479] ? do_wait_intr+0xf0/0xf0 [10820.087488] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10820.087687] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10820.087896] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10820.100003] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10820.103347] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10820.103357] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10820.103360] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10820.103373] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10820.103377] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10820.103382] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10820.103384] RDX: 0000000000000637 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10820.103387] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10820.103388] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10820.103391] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10820.103394] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10820.103396] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10820.103398] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10820.103406] Call Trace: [10820.103412] [10820.103418] ? show_regs.cold.9+0x22/0x2f [10820.103427] ? watchdog_timer_fn.cold.6+0x62/0x10a [10820.103433] ? __hrtimer_run_queues+0x1c2/0x4d0 [10820.103437] ? watchdog+0x40/0x40 [10820.103441] ? hrtimer_interrupt+0x128/0x2c0 [10820.103445] ? smp_apic_timer_interrupt+0xa6/0x270 [10820.103454] ? apic_timer_interrupt+0xf/0x20 [10820.103458] [10820.103461] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10820.103466] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10820.103470] do_raw_spin_lock+0x177/0x190 [10820.111208] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10820.111331] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10820.111372] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10820.111519] mdc_intent_lock+0x353/0x7e0 [mdc] [10820.111547] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10820.111630] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10820.111898] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10820.111927] lmv_intent_open+0x338/0x1150 [lmv] [10820.111955] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10820.112036] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10820.112258] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10820.112281] ll_intent_lock+0xc5/0xaa0 [lustre] [10820.112362] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10820.112524] ll_lookup_it+0xe0e/0x2270 [lustre] [10820.112609] ? do_raw_spin_unlock+0x75/0x190 [10820.112618] ? _raw_spin_unlock+0x12/0x30 [10820.112626] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10820.112652] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10820.112673] ll_atomic_open+0x36a/0xd70 [lustre] [10820.112756] lookup_open+0x356/0xb40 [10820.112767] do_last+0x313/0xfc0 [10820.112772] ? nd_jump_root+0xe5/0x160 [10820.112777] ? path_init+0x437/0x520 [10820.112781] path_openat+0xf7/0x500 [10820.112786] ? do_raw_spin_unlock+0x75/0x190 [10820.112791] do_filp_open+0x99/0x140 [10820.112796] ? getname_flags+0x6e/0x330 [10820.112802] ? __check_object_size+0xff/0x256 [10820.112808] ? do_raw_spin_unlock+0x75/0x190 [10820.112812] ? _raw_spin_unlock+0x12/0x30 [10820.112817] do_sys_openat2+0x2b4/0x410 [10820.112822] do_sys_open+0x73/0xa0 [10820.112826] __x64_sys_openat+0x24/0x30 [10820.112830] do_syscall_64+0xc1/0x440 [10820.112836] entry_SYSCALL_64_after_hwframe+0x49/0xae [10820.112843] RIP: 0033:0x7f53425a5332 [10820.112893] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10820.112899] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10820.112975] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10820.112977] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10820.112979] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10820.112981] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10820.112983] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10820.679002] _raw_spin_lock+0x21/0x30 [10820.680645] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10820.683744] obd_mod_free+0x46/0x140 [ptlrpc] [10820.689175] mdc_commit_open+0x91/0xb0 [mdc] [10820.692270] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10820.697261] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10820.701208] after_reply+0xe2c/0x1780 [ptlrpc] [10820.702564] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10820.704761] ? __next_timer_interrupt+0x160/0x160 [10820.706704] ptlrpcd+0xa82/0xe40 [ptlrpc] [10820.708593] ? do_wait_intr+0xf0/0xf0 [10820.709748] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10820.711823] kthread+0x1d1/0x200 [10820.712864] ? set_kthread_struct+0x70/0x70 [10820.722138] ret_from_fork+0x1f/0x30 [10832.052024] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [10832.054698] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10832.118429] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10832.135953] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10832.145369] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10832.151868] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10832.158811] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10832.181345] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10832.189817] RAX: 0000000000004266 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10832.197178] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10832.201903] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10832.210301] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10832.222349] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10832.231951] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10832.242636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10832.251552] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10832.262347] Call Trace: [10832.267387] [10832.274221] ? show_regs.cold.9+0x22/0x2f [10832.282383] ? watchdog_timer_fn.cold.6+0x62/0x10a [10832.288668] ? __hrtimer_run_queues+0x1c2/0x4d0 [10832.294113] ? watchdog+0x40/0x40 [10832.295325] ? hrtimer_interrupt+0x128/0x2c0 [10832.306777] ? smp_apic_timer_interrupt+0xa6/0x270 [10832.323147] ? apic_timer_interrupt+0xf/0x20 [10832.329100] [10832.331598] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10832.339790] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10832.350263] ? newidle_balance.isra.93+0x207/0x580 [10832.358923] do_raw_spin_lock+0x177/0x190 [10832.365773] _raw_spin_lock+0x21/0x30 [10832.370907] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10832.374375] process_one_work+0x2c8/0x700 [10832.376934] worker_thread+0x296/0x6e0 [10832.379270] ? rescuer_thread+0x570/0x570 [10832.385806] kthread+0x1d1/0x200 [10832.390194] ? set_kthread_struct+0x70/0x70 [10832.395333] ret_from_fork+0x1f/0x30 [10836.061129] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [10836.066117] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10836.100611] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10836.110936] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10836.115898] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10836.122749] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10836.146414] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10836.149934] RAX: 0000000000005705 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10836.153924] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10836.159416] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10836.168340] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10836.171687] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10836.175734] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10836.184932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10836.189523] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10836.193168] Call Trace: [10836.194397] [10836.195233] ? show_regs.cold.9+0x22/0x2f [10836.197520] ? watchdog_timer_fn.cold.6+0x62/0x10a [10836.200613] ? __hrtimer_run_queues+0x1c2/0x4d0 [10836.203429] ? watchdog+0x40/0x40 [10836.204667] ? hrtimer_interrupt+0x128/0x2c0 [10836.212022] ? smp_apic_timer_interrupt+0xa6/0x270 [10836.215093] ? apic_timer_interrupt+0xf/0x20 [10836.218682] [10836.220036] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10836.224257] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10836.227132] do_raw_spin_lock+0x177/0x190 [10836.231686] _raw_spin_lock+0x21/0x30 [10836.234389] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10836.239581] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10836.242823] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10836.245447] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10836.251135] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10836.253853] ? woken_wake_function+0x30/0x30 [10836.259577] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10836.268095] kthread+0x1d1/0x200 [10836.269138] ? set_kthread_struct+0x70/0x70 [10836.270336] ret_from_fork+0x1f/0x30 [10837.986038] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8200s! [10837.988526] Showing busy workqueues and worker pools: [10837.990169] workqueue events: flags=0x0 [10837.991197] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10837.993282] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10837.998714] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10838.000695] pending: cpu_latency_work [ptlrpc] [10838.002855] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10838.005259] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10838.008196] workqueue events_long: flags=0x0 [10838.009543] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10838.011280] pending: lu_objects_flush [obdclass] [10838.012860] workqueue events_unbound: flags=0x2 [10838.014107] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10838.015706] pending: flush_memcg_stats_dwork, idle_cull_fn [10838.017365] workqueue events_power_efficient: flags=0x80 [10838.019135] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10838.021106] pending: check_lifetime, do_cache_clean [10838.022647] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10838.024534] pending: neigh_periodic_work, neigh_periodic_work [10838.026252] workqueue mm_percpu_wq: flags=0x8 [10838.027493] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10838.030803] pending: vmstat_update, lru_add_drain_per_cpu [10838.032964] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10838.035678] pending: vmstat_update, lru_add_drain_per_cpu [10838.037331] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10838.039050] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10838.040139] workqueue xprtiod: flags=0xa [10838.041061] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10838.042872] pending: xprt_autoclose, xs_error_handle [10838.044518] workqueue ipv6_addrconf: flags=0x40008 [10838.046055] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10838.048300] pending: addrconf_verify_work [10838.050222] workqueue knbd0-recv: flags=0x1a [10838.052087] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10838.054964] in-flight: 651:recv_work [10838.056582] workqueue ptlrpc_pinger: flags=0x2 [10838.058580] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10838.068682] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10838.073282] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8200s workers=4 idle: 15979 31824 8 [10838.077236] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10828s workers=2 idle: 654 [10848.055025] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [10848.057407] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [10848.057782] Modules linked in: [10848.062940] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10848.064459] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel [10848.091755] [10848.091757] ata_piix serio_raw libata dm_mirror [10848.111902] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10848.112802] dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10848.112818] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10848.112824] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10848.112826] RIP: 0010:native_safe_halt+0x2e/0x50 [10848.112843] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10848.112848] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10848.112856] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10848.112858] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10848.112860] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10848.112861] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10848.112866] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10848.112869] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10848.112872] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10848.112874] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10848.112880] Call Trace: [10848.112889] [10848.112896] ? show_regs.cold.9+0x22/0x2f [10848.112905] ? watchdog_timer_fn.cold.6+0x62/0x10a [10848.112912] ? __hrtimer_run_queues+0x1c2/0x4d0 [10848.112916] ? watchdog+0x40/0x40 [10848.112921] ? hrtimer_interrupt+0x128/0x2c0 [10848.112924] ? smp_apic_timer_interrupt+0xa6/0x270 [10848.112931] ? apic_timer_interrupt+0xf/0x20 [10848.112935] [10848.112938] ? native_safe_halt+0x2e/0x50 [10848.112941] kvm_wait+0xcd/0xe0 [10848.112948] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10848.112965] do_raw_spin_lock+0x177/0x190 [10848.112972] _raw_spin_lock+0x21/0x30 [10848.112977] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10848.113478] ? do_raw_spin_unlock+0x75/0x190 [10848.113485] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10848.118472] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10848.118484] RIP: 0010:native_safe_halt+0x2e/0x50 [10848.118499] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10848.118503] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10848.118509] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10848.118512] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10848.118514] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10848.118516] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10848.118519] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10848.118522] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10848.118531] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10848.118534] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10848.118544] Call Trace: [10848.118550] [10848.118556] ? show_regs.cold.9+0x22/0x2f [10848.118580] ? watchdog_timer_fn.cold.6+0x62/0x10a [10848.118591] ? __hrtimer_run_queues+0x1c2/0x4d0 [10848.118595] ? watchdog+0x40/0x40 [10848.124996] ? do_wait_intr+0xf0/0xf0 [10848.125019] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10848.127591] ? hrtimer_interrupt+0x128/0x2c0 [10848.127605] ? smp_apic_timer_interrupt+0xa6/0x270 [10848.127613] ? apic_timer_interrupt+0xf/0x20 [10848.127618] [10848.127622] ? native_safe_halt+0x2e/0x50 [10848.127627] kvm_wait+0xcd/0xe0 [10848.127635] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10848.127643] do_raw_spin_lock+0x177/0x190 [10848.127647] _raw_spin_lock+0x21/0x30 [10848.127653] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10848.127954] obd_mod_free+0x46/0x140 [ptlrpc] [10848.128217] mdc_commit_open+0x91/0xb0 [mdc] [10848.128255] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10848.128457] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10848.128714] after_reply+0xe2c/0x1780 [ptlrpc] [10848.128900] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10848.129085] ? __next_timer_interrupt+0x160/0x160 [10848.136788] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10848.137110] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10848.137323] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10848.137441] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10848.137482] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10848.137513] mdc_intent_lock+0x353/0x7e0 [mdc] [10848.137544] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10848.137646] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10848.137859] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10848.137887] lmv_intent_open+0x338/0x1150 [lmv] [10848.137919] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10848.138002] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10848.138083] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10848.138107] ll_intent_lock+0xc5/0xaa0 [lustre] [10848.138192] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10848.138277] ll_lookup_it+0xe0e/0x2270 [lustre] [10848.138367] ? do_raw_spin_unlock+0x75/0x190 [10848.138375] ? _raw_spin_unlock+0x12/0x30 [10848.138383] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10848.138415] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10848.138435] ll_atomic_open+0x36a/0xd70 [lustre] [10848.138518] lookup_open+0x356/0xb40 [10848.138527] do_last+0x313/0xfc0 [10848.138533] ? nd_jump_root+0xe5/0x160 [10848.138538] ? path_init+0x437/0x520 [10848.138543] path_openat+0xf7/0x500 [10848.138548] ? do_raw_spin_unlock+0x75/0x190 [10848.138553] do_filp_open+0x99/0x140 [10848.138711] ? getname_flags+0x6e/0x330 [10848.138718] ? __check_object_size+0xff/0x256 [10848.138726] ? do_raw_spin_unlock+0x75/0x190 [10848.138730] ? _raw_spin_unlock+0x12/0x30 [10848.138735] do_sys_openat2+0x2b4/0x410 [10848.138740] do_sys_open+0x73/0xa0 [10848.138745] __x64_sys_openat+0x24/0x30 [10848.138748] do_syscall_64+0xc1/0x440 [10848.138757] entry_SYSCALL_64_after_hwframe+0x49/0xae [10848.138763] RIP: 0033:0x7f53425a5332 [10848.138771] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10848.138776] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10848.138782] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10848.138785] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10848.138787] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10848.138789] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10848.138792] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10848.508787] ptlrpcd+0xa82/0xe40 [ptlrpc] [10848.515243] ? do_wait_intr+0xf0/0xf0 [10848.524037] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10848.531019] kthread+0x1d1/0x200 [10848.533869] ? set_kthread_struct+0x70/0x70 [10848.535346] ret_from_fork+0x1f/0x30 [10864.052007] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [10864.058452] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [10864.061399] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [10864.066404] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10864.066551] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10864.066559] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10864.066562] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10864.066575] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10864.066579] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10864.066584] RAX: 000000000000240f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10864.066586] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10864.066588] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10864.066590] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10864.066593] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10864.066596] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10864.066599] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10864.066602] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10864.066610] Call Trace: [10864.066615] [10864.066620] ? show_regs.cold.9+0x22/0x2f [10864.066630] ? watchdog_timer_fn.cold.6+0x62/0x10a [10864.066636] ? __hrtimer_run_queues+0x1c2/0x4d0 [10864.066640] ? watchdog+0x40/0x40 [10864.066644] ? hrtimer_interrupt+0x128/0x2c0 [10864.066647] ? smp_apic_timer_interrupt+0xa6/0x270 [10864.066656] ? apic_timer_interrupt+0xf/0x20 [10864.066660] [10864.066663] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10864.066668] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10864.066674] do_raw_spin_lock+0x177/0x190 [10864.066678] _raw_spin_lock+0x21/0x30 [10864.066684] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10864.066972] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10864.067230] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10864.067577] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10864.067897] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10864.068228] ? woken_wake_function+0x30/0x30 [10864.068238] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10864.068485] kthread+0x1d1/0x200 [10864.068491] ? set_kthread_struct+0x70/0x70 [10864.068495] ret_from_fork+0x1f/0x30 [10864.229934] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10864.242651] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10864.247566] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10864.257766] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10864.260198] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10864.266181] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10864.278524] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10864.287649] RAX: 0000000000004ab0 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10864.293336] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10864.296850] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10864.299618] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10864.303143] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10864.307186] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10864.314889] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10864.316943] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10864.324596] Call Trace: [10864.325728] [10864.326743] ? show_regs.cold.9+0x22/0x2f [10864.329130] ? watchdog_timer_fn.cold.6+0x62/0x10a [10864.334597] ? __hrtimer_run_queues+0x1c2/0x4d0 [10864.337303] ? watchdog+0x40/0x40 [10864.340060] ? hrtimer_interrupt+0x128/0x2c0 [10864.342228] ? smp_apic_timer_interrupt+0xa6/0x270 [10864.347120] ? apic_timer_interrupt+0xf/0x20 [10864.349171] [10864.349718] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10864.350924] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10864.355002] ? newidle_balance.isra.93+0x207/0x580 [10864.360890] do_raw_spin_lock+0x177/0x190 [10864.362747] _raw_spin_lock+0x21/0x30 [10864.365273] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10864.367547] process_one_work+0x2c8/0x700 [10864.369204] worker_thread+0x296/0x6e0 [10864.371556] ? rescuer_thread+0x570/0x570 [10864.376643] kthread+0x1d1/0x200 [10864.380336] ? set_kthread_struct+0x70/0x70 [10864.385011] ret_from_fork+0x1f/0x30 [10868.193360] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8231s! [10868.198739] Showing busy workqueues and worker pools: [10868.200590] workqueue events: flags=0x0 [10868.201819] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10868.203916] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10868.207632] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10868.209708] pending: cpu_latency_work [ptlrpc] [10868.211811] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10868.213665] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10868.217078] workqueue events_long: flags=0x0 [10868.219203] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10868.222295] pending: lu_objects_flush [obdclass] [10868.224511] workqueue events_unbound: flags=0x2 [10868.226463] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10868.229098] pending: flush_memcg_stats_dwork, idle_cull_fn [10868.231601] workqueue events_power_efficient: flags=0x80 [10868.233872] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10868.236725] pending: check_lifetime, do_cache_clean [10868.238777] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10868.240976] pending: neigh_periodic_work, neigh_periodic_work [10868.243128] workqueue mm_percpu_wq: flags=0x8 [10868.244536] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10868.246908] pending: vmstat_update, lru_add_drain_per_cpu [10868.248623] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10868.253230] pending: vmstat_update, lru_add_drain_per_cpu [10868.254796] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10868.257445] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10868.260486] workqueue xprtiod: flags=0xa [10868.262776] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10868.267415] pending: xprt_autoclose, xs_error_handle [10868.268887] workqueue ipv6_addrconf: flags=0x40008 [10868.270099] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10868.277930] pending: addrconf_verify_work [10868.282020] workqueue knbd0-recv: flags=0x1a [10868.283265] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10868.285577] in-flight: 651:recv_work [10868.286519] workqueue ptlrpc_pinger: flags=0x2 [10868.287721] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10868.297504] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10868.299180] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8231s workers=4 idle: 15979 31824 8 [10868.302548] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10858s workers=2 idle: 654 [10876.055051] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [10876.057034] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [10876.057537] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10876.057604] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10876.057610] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10876.057612] RIP: 0010:native_safe_halt+0x2e/0x50 [10876.057625] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10876.057629] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [10876.057633] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10876.057635] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10876.057637] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10876.057639] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10876.057642] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10876.057645] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10876.057647] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10876.057649] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10876.057658] Call Trace: [10876.057663] [10876.057669] ? show_regs.cold.9+0x22/0x2f [10876.057678] ? watchdog_timer_fn.cold.6+0x62/0x10a [10876.057683] ? __hrtimer_run_queues+0x1c2/0x4d0 [10876.057687] ? watchdog+0x40/0x40 [10876.057691] ? hrtimer_interrupt+0x128/0x2c0 [10876.057695] ? smp_apic_timer_interrupt+0xa6/0x270 [10876.057701] ? apic_timer_interrupt+0xf/0x20 [10876.057706] [10876.057709] ? native_safe_halt+0x2e/0x50 [10876.057712] kvm_wait+0xcd/0xe0 [10876.057720] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10876.057727] do_raw_spin_lock+0x177/0x190 [10876.057731] _raw_spin_lock+0x21/0x30 [10876.057734] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10876.058098] ? do_raw_spin_unlock+0x75/0x190 [10876.058106] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10876.060390] Modules linked in: lustre(O) [10876.093985] ? do_wait_intr+0xf0/0xf0 [10876.098692] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10876.098768] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10876.098776] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10876.098779] RIP: 0010:native_safe_halt+0x2e/0x50 [10876.098940] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10876.098944] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10876.098949] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10876.098951] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10876.098953] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10876.098956] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10876.098958] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10876.098961] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10876.098964] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10876.098966] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10876.098975] Call Trace: [10876.098982] [10876.098988] ? show_regs.cold.9+0x22/0x2f [10876.099059] ? watchdog_timer_fn.cold.6+0x62/0x10a [10876.099066] ? __hrtimer_run_queues+0x1c2/0x4d0 [10876.099070] ? watchdog+0x40/0x40 [10876.099085] ? hrtimer_interrupt+0x128/0x2c0 [10876.099089] ? smp_apic_timer_interrupt+0xa6/0x270 [10876.099096] ? apic_timer_interrupt+0xf/0x20 [10876.099101] [10876.099104] ? native_safe_halt+0x2e/0x50 [10876.099109] kvm_wait+0xcd/0xe0 [10876.099116] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10876.099124] do_raw_spin_lock+0x177/0x190 [10876.099128] _raw_spin_lock+0x21/0x30 [10876.099134] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10876.099409] obd_mod_free+0x46/0x140 [ptlrpc] [10876.099617] mdc_commit_open+0x91/0xb0 [mdc] [10876.099660] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10876.100064] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10876.100279] after_reply+0xe2c/0x1780 [ptlrpc] [10876.100480] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10876.100680] ? __next_timer_interrupt+0x160/0x160 [10876.100689] ptlrpcd+0xa82/0xe40 [ptlrpc] [10876.100910] ? do_wait_intr+0xf0/0xf0 [10876.100919] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10876.101133] kthread+0x1d1/0x200 [10876.101138] ? set_kthread_struct+0x70/0x70 [10876.101142] ret_from_fork+0x1f/0x30 [10876.393546] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10876.396149] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10876.398308] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10876.401401] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10876.402995] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10876.404995] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10876.406392] mdc_intent_lock+0x353/0x7e0 [mdc] [10876.409543] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10876.413547] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10876.415839] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10876.417933] lmv_intent_open+0x338/0x1150 [lmv] [10876.420023] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10876.421975] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10876.424198] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10876.426654] ll_intent_lock+0xc5/0xaa0 [lustre] [10876.428348] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10876.431408] ll_lookup_it+0xe0e/0x2270 [lustre] [10876.432761] ? do_raw_spin_unlock+0x75/0x190 [10876.435622] ? _raw_spin_unlock+0x12/0x30 [10876.436901] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10876.439946] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10876.444558] ll_atomic_open+0x36a/0xd70 [lustre] [10876.447288] lookup_open+0x356/0xb40 [10876.448279] do_last+0x313/0xfc0 [10876.449433] ? nd_jump_root+0xe5/0x160 [10876.451385] ? path_init+0x437/0x520 [10876.453348] path_openat+0xf7/0x500 [10876.454498] ? do_raw_spin_unlock+0x75/0x190 [10876.456556] do_filp_open+0x99/0x140 [10876.457934] ? getname_flags+0x6e/0x330 [10876.459826] ? __check_object_size+0xff/0x256 [10876.461365] ? do_raw_spin_unlock+0x75/0x190 [10876.463513] ? _raw_spin_unlock+0x12/0x30 [10876.465192] do_sys_openat2+0x2b4/0x410 [10876.466916] do_sys_open+0x73/0xa0 [10876.468192] __x64_sys_openat+0x24/0x30 [10876.469660] do_syscall_64+0xc1/0x440 [10876.470832] entry_SYSCALL_64_after_hwframe+0x49/0xae [10876.473733] RIP: 0033:0x7f53425a5332 [10876.474821] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10876.490181] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10876.492880] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10876.495743] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10876.498651] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10876.501662] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10876.505372] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10892.058030] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [10892.061013] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10892.078982] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10892.084235] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10892.088195] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10892.090741] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10892.098757] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10892.102568] RAX: 000000000000509b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10892.105044] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10892.107943] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10892.110961] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10892.114083] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10892.117106] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10892.120253] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10892.122242] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10892.124547] Call Trace: [10892.125544] [10892.126281] ? show_regs.cold.9+0x22/0x2f [10892.127722] ? watchdog_timer_fn.cold.6+0x62/0x10a [10892.129657] ? __hrtimer_run_queues+0x1c2/0x4d0 [10892.131636] ? watchdog+0x40/0x40 [10892.133026] ? hrtimer_interrupt+0x128/0x2c0 [10892.134498] ? smp_apic_timer_interrupt+0xa6/0x270 [10892.136344] ? apic_timer_interrupt+0xf/0x20 [10892.137651] [10892.138306] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10892.140497] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10892.142616] do_raw_spin_lock+0x177/0x190 [10892.144115] _raw_spin_lock+0x21/0x30 [10892.145633] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10892.148515] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10892.151089] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10892.153560] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10892.156081] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10892.158552] ? woken_wake_function+0x30/0x30 [10892.160399] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10892.162926] kthread+0x1d1/0x200 [10892.163906] ? set_kthread_struct+0x70/0x70 [10892.165475] ret_from_fork+0x1f/0x30 [10896.054198] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [10896.064979] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10896.097709] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10896.108284] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10896.123348] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10896.134768] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10896.138416] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10896.154138] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [10896.158906] RAX: 0000000000007c09 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10896.164370] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10896.169918] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10896.177371] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10896.193104] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10896.197073] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10896.200245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10896.203105] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10896.212408] Call Trace: [10896.213275] [10896.214040] ? show_regs.cold.9+0x22/0x2f [10896.217506] ? watchdog_timer_fn.cold.6+0x62/0x10a [10896.229042] ? __hrtimer_run_queues+0x1c2/0x4d0 [10896.234056] ? watchdog+0x40/0x40 [10896.239470] ? hrtimer_interrupt+0x128/0x2c0 [10896.242191] ? smp_apic_timer_interrupt+0xa6/0x270 [10896.245209] ? apic_timer_interrupt+0xf/0x20 [10896.247274] [10896.248678] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10896.252282] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10896.255971] ? newidle_balance.isra.93+0x207/0x580 [10896.258995] do_raw_spin_lock+0x177/0x190 [10896.262056] _raw_spin_lock+0x21/0x30 [10896.264207] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10896.268220] process_one_work+0x2c8/0x700 [10896.269686] worker_thread+0x296/0x6e0 [10896.272467] ? rescuer_thread+0x570/0x570 [10896.279700] kthread+0x1d1/0x200 [10896.281654] ? set_kthread_struct+0x70/0x70 [10896.287002] ret_from_fork+0x1f/0x30 [10898.401153] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8261s! [10898.410412] Showing busy workqueues and worker pools: [10898.414124] workqueue events: flags=0x0 [10898.418813] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10898.423116] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10898.435063] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10898.441837] pending: cpu_latency_work [ptlrpc] [10898.452970] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10898.461088] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10898.476087] workqueue events_long: flags=0x0 [10898.477364] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10898.486184] pending: lu_objects_flush [obdclass] [10898.502977] workqueue events_unbound: flags=0x2 [10898.504620] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10898.509489] pending: flush_memcg_stats_dwork, idle_cull_fn [10898.514401] workqueue events_power_efficient: flags=0x80 [10898.519816] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10898.523561] pending: check_lifetime, do_cache_clean [10898.525941] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10898.528574] pending: neigh_periodic_work, neigh_periodic_work [10898.531166] workqueue mm_percpu_wq: flags=0x8 [10898.533018] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10898.535695] pending: vmstat_update, lru_add_drain_per_cpu [10898.538063] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10898.540580] pending: vmstat_update, lru_add_drain_per_cpu [10898.542749] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10898.545168] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10898.547565] workqueue xprtiod: flags=0xa [10898.549044] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10898.551244] pending: xprt_autoclose, xs_error_handle [10898.553189] workqueue ipv6_addrconf: flags=0x40008 [10898.554952] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10898.557259] pending: addrconf_verify_work [10898.558944] workqueue knbd0-recv: flags=0x1a [10898.561277] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10898.564008] in-flight: 651:recv_work [10898.565848] workqueue ptlrpc_pinger: flags=0x2 [10898.567353] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10898.569760] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10898.572026] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8261s workers=4 idle: 15979 31824 8 [10898.574947] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10889s workers=2 idle: 654 [10904.056950] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [10904.059306] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [10904.064257] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10904.064338] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10904.064344] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10904.064347] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10904.064362] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10904.064366] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10904.064370] RAX: 00000000000065bf RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10904.064373] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10904.064375] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10904.064377] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10904.064379] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10904.064382] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10904.064384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10904.064386] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10904.064395] Call Trace: [10904.064402] [10904.064408] ? show_regs.cold.9+0x22/0x2f [10904.064417] ? watchdog_timer_fn.cold.6+0x62/0x10a [10904.064423] ? __hrtimer_run_queues+0x1c2/0x4d0 [10904.064428] ? watchdog+0x40/0x40 [10904.064432] ? hrtimer_interrupt+0x128/0x2c0 [10904.064437] ? smp_apic_timer_interrupt+0xa6/0x270 [10904.064460] ? apic_timer_interrupt+0xf/0x20 [10904.064465] [10904.064468] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10904.064474] do_raw_spin_lock+0x177/0x190 [10904.064479] _raw_spin_lock+0x21/0x30 [10904.064484] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10904.064814] ? do_raw_spin_unlock+0x75/0x190 [10904.064821] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10904.065039] ? do_wait_intr+0xf0/0xf0 [10904.065049] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10904.065445] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10904.065922] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10904.066232] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10904.066355] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10904.066397] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10904.066425] mdc_intent_lock+0x353/0x7e0 [mdc] [10904.066453] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10904.066543] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10904.067249] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10904.067283] lmv_intent_open+0x338/0x1150 [lmv] [10904.067315] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10904.067413] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10904.067563] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10904.067731] ll_intent_lock+0xc5/0xaa0 [lustre] [10904.067884] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10904.067968] ll_lookup_it+0xe0e/0x2270 [lustre] [10904.068310] ? do_raw_spin_unlock+0x75/0x190 [10904.068319] ? _raw_spin_unlock+0x12/0x30 [10904.068326] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10904.068353] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10904.068374] ll_atomic_open+0x36a/0xd70 [lustre] [10904.068569] lookup_open+0x356/0xb40 [10904.068595] do_last+0x313/0xfc0 [10904.068600] ? nd_jump_root+0xe5/0x160 [10904.068605] ? path_init+0x437/0x520 [10904.068611] path_openat+0xf7/0x500 [10904.068616] ? do_raw_spin_unlock+0x75/0x190 [10904.068621] do_filp_open+0x99/0x140 [10904.068627] ? getname_flags+0x6e/0x330 [10904.068632] ? __check_object_size+0xff/0x256 [10904.068640] ? do_raw_spin_unlock+0x75/0x190 [10904.068644] ? _raw_spin_unlock+0x12/0x30 [10904.068650] do_sys_openat2+0x2b4/0x410 [10904.068654] do_sys_open+0x73/0xa0 [10904.068658] __x64_sys_openat+0x24/0x30 [10904.068662] do_syscall_64+0xc1/0x440 [10904.068680] entry_SYSCALL_64_after_hwframe+0x49/0xae [10904.068686] RIP: 0033:0x7f53425a5332 [10904.068694] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10904.068698] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10904.068704] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10904.068707] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10904.068709] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10904.068711] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10904.068713] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10904.493414] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10904.542348] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10904.553349] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10904.561304] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [10904.564278] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [10904.587519] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10904.597432] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10904.606311] RDX: 0000000000007f01 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10904.609510] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10904.613199] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10904.616628] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10904.622106] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10904.628228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10904.632560] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10904.644007] Call Trace: [10904.647203] [10904.650362] ? show_regs.cold.9+0x22/0x2f [10904.651938] ? watchdog_timer_fn.cold.6+0x62/0x10a [10904.659878] ? __hrtimer_run_queues+0x1c2/0x4d0 [10904.663726] ? watchdog+0x40/0x40 [10904.666510] ? hrtimer_interrupt+0x128/0x2c0 [10904.672550] ? smp_apic_timer_interrupt+0xa6/0x270 [10904.675914] ? apic_timer_interrupt+0xf/0x20 [10904.678529] [10904.683265] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10904.686086] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10904.694524] do_raw_spin_lock+0x177/0x190 [10904.699383] _raw_spin_lock+0x21/0x30 [10904.704149] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10904.712289] obd_mod_free+0x46/0x140 [ptlrpc] [10904.717793] mdc_commit_open+0x91/0xb0 [mdc] [10904.725031] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10904.727303] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10904.731646] after_reply+0xe2c/0x1780 [ptlrpc] [10904.740288] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10904.747454] ? __next_timer_interrupt+0x160/0x160 [10904.753547] ptlrpcd+0xa82/0xe40 [ptlrpc] [10904.759191] ? do_wait_intr+0xf0/0xf0 [10904.760790] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10904.766859] kthread+0x1d1/0x200 [10904.768152] ? set_kthread_struct+0x70/0x70 [10904.773079] ret_from_fork+0x1f/0x30 [10920.058029] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [10920.065539] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10920.097606] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10920.110647] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10920.116003] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10920.124258] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10920.137631] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10920.149282] RAX: 0000000000007884 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10920.153392] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10920.161805] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10920.166081] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10920.171100] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10920.173550] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10920.180079] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10920.184159] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10920.193747] Call Trace: [10920.196395] [10920.198260] ? show_regs.cold.9+0x22/0x2f [10920.207754] ? watchdog_timer_fn.cold.6+0x62/0x10a [10920.211484] ? __hrtimer_run_queues+0x1c2/0x4d0 [10920.212996] ? watchdog+0x40/0x40 [10920.220652] ? hrtimer_interrupt+0x128/0x2c0 [10920.227279] ? smp_apic_timer_interrupt+0xa6/0x270 [10920.233549] ? apic_timer_interrupt+0xf/0x20 [10920.238333] [10920.240491] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10920.244265] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10920.249862] do_raw_spin_lock+0x177/0x190 [10920.254080] _raw_spin_lock+0x21/0x30 [10920.261677] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10920.270794] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10920.276644] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10920.285586] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10920.289929] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10920.293531] ? woken_wake_function+0x30/0x30 [10920.297857] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10920.302005] kthread+0x1d1/0x200 [10920.305111] ? set_kthread_struct+0x70/0x70 [10920.310169] ret_from_fork+0x1f/0x30 [10924.053878] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [10924.058773] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10924.110822] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10924.120080] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10924.130368] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10924.142777] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10924.149832] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10924.175386] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10924.183647] RAX: 0000000000006187 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10924.191756] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10924.199186] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10924.205658] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10924.212132] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10924.216253] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10924.221379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10924.223739] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10924.228870] Call Trace: [10924.229741] [10924.237485] ? show_regs.cold.9+0x22/0x2f [10924.241759] ? watchdog_timer_fn.cold.6+0x62/0x10a [10924.244472] ? __hrtimer_run_queues+0x1c2/0x4d0 [10924.246904] ? watchdog+0x40/0x40 [10924.256729] ? hrtimer_interrupt+0x128/0x2c0 [10924.263219] ? smp_apic_timer_interrupt+0xa6/0x270 [10924.269862] ? apic_timer_interrupt+0xf/0x20 [10924.280421] [10924.281109] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10924.288591] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10924.296029] ? newidle_balance.isra.93+0x207/0x580 [10924.297738] do_raw_spin_lock+0x177/0x190 [10924.307357] _raw_spin_lock+0x21/0x30 [10924.308821] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10924.313769] process_one_work+0x2c8/0x700 [10924.320808] worker_thread+0x296/0x6e0 [10924.322211] ? rescuer_thread+0x570/0x570 [10924.324912] kthread+0x1d1/0x200 [10924.327196] ? set_kthread_struct+0x70/0x70 [10924.328712] ret_from_fork+0x1f/0x30 [10928.607046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8291s! [10928.613087] Showing busy workqueues and worker pools: [10928.616578] workqueue events: flags=0x0 [10928.620591] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10928.623028] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10928.635771] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10928.639719] pending: cpu_latency_work [ptlrpc] [10928.649657] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10928.655727] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10928.660538] workqueue events_long: flags=0x0 [10928.661525] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10928.667735] pending: lu_objects_flush [obdclass] [10928.671067] workqueue events_unbound: flags=0x2 [10928.683129] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10928.687854] pending: flush_memcg_stats_dwork, idle_cull_fn [10928.691278] workqueue events_power_efficient: flags=0x80 [10928.693462] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10928.702504] pending: check_lifetime, do_cache_clean [10928.706675] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10928.711735] pending: neigh_periodic_work, neigh_periodic_work [10928.714732] workqueue mm_percpu_wq: flags=0x8 [10928.716036] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10928.718992] pending: vmstat_update, lru_add_drain_per_cpu [10928.722220] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10928.726465] pending: vmstat_update, lru_add_drain_per_cpu [10928.729365] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10928.733400] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10928.736463] workqueue xprtiod: flags=0xa [10928.738553] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10928.742587] pending: xprt_autoclose, xs_error_handle [10928.747898] workqueue ipv6_addrconf: flags=0x40008 [10928.749490] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10928.758946] pending: addrconf_verify_work [10928.760341] workqueue knbd0-recv: flags=0x1a [10928.762508] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10928.765139] in-flight: 651:recv_work [10928.767013] workqueue ptlrpc_pinger: flags=0x2 [10928.768931] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10928.771545] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10928.774407] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8291s workers=4 idle: 15979 31824 8 [10928.777507] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10919s workers=2 idle: 654 [10932.057016] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [10932.058887] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10932.059314] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [10932.070885] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10932.070902] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10932.070904] RIP: 0010:native_safe_halt+0x2e/0x50 [10932.070918] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10932.070922] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10932.070926] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10932.070928] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10932.070930] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10932.070931] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10932.070933] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10932.070936] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10932.070938] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10932.070940] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10932.070946] Call Trace: [10932.070951] [10932.070957] ? show_regs.cold.9+0x22/0x2f [10932.070966] ? watchdog_timer_fn.cold.6+0x62/0x10a [10932.070971] ? __hrtimer_run_queues+0x1c2/0x4d0 [10932.070974] ? watchdog+0x40/0x40 [10932.070977] ? hrtimer_interrupt+0x128/0x2c0 [10932.070980] ? smp_apic_timer_interrupt+0xa6/0x270 [10932.070985] ? apic_timer_interrupt+0xf/0x20 [10932.070988] [10932.070990] ? native_safe_halt+0x2e/0x50 [10932.070992] kvm_wait+0xcd/0xe0 [10932.070999] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10932.071004] do_raw_spin_lock+0x177/0x190 [10932.071007] _raw_spin_lock+0x21/0x30 [10932.071012] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10932.071482] obd_mod_free+0x46/0x140 [ptlrpc] [10932.074268] mdc_commit_open+0x91/0xb0 [mdc] [10932.074307] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10932.074526] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10932.074694] after_reply+0xe2c/0x1780 [ptlrpc] [10932.074835] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10932.074959] ? __next_timer_interrupt+0x160/0x160 [10932.074967] ptlrpcd+0xa82/0xe40 [ptlrpc] [10932.075149] ? do_wait_intr+0xf0/0xf0 [10932.075157] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10932.075352] kthread+0x1d1/0x200 [10932.075358] ? set_kthread_struct+0x70/0x70 [10932.075361] ret_from_fork+0x1f/0x30 [10932.281711] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10932.310651] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10932.317861] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10932.334293] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10932.339621] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10932.368335] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10932.374949] RAX: 0000000000005896 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10932.386299] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10932.391697] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10932.397985] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10932.403638] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10932.406857] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10932.410751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10932.413703] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10932.417500] Call Trace: [10932.418806] [10932.419835] ? show_regs.cold.9+0x22/0x2f [10932.424917] ? watchdog_timer_fn.cold.6+0x62/0x10a [10932.429510] ? __hrtimer_run_queues+0x1c2/0x4d0 [10932.432514] ? watchdog+0x40/0x40 [10932.433630] ? hrtimer_interrupt+0x128/0x2c0 [10932.440844] ? smp_apic_timer_interrupt+0xa6/0x270 [10932.442635] ? apic_timer_interrupt+0xf/0x20 [10932.445874] [10932.446739] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10932.452745] do_raw_spin_lock+0x177/0x190 [10932.456603] _raw_spin_lock+0x21/0x30 [10932.460969] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10932.463946] ? do_raw_spin_unlock+0x75/0x190 [10932.466711] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10932.470209] ? do_wait_intr+0xf0/0xf0 [10932.472755] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10932.477946] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10932.485543] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10932.490427] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10932.492733] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10932.497881] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10932.501118] mdc_intent_lock+0x353/0x7e0 [mdc] [10932.503692] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10932.510556] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10932.514358] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10932.518195] lmv_intent_open+0x338/0x1150 [lmv] [10932.525859] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10932.533184] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10932.535160] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10932.539556] ll_intent_lock+0xc5/0xaa0 [lustre] [10932.542713] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10932.545674] ll_lookup_it+0xe0e/0x2270 [lustre] [10932.550055] ? do_raw_spin_unlock+0x75/0x190 [10932.553701] ? _raw_spin_unlock+0x12/0x30 [10932.555452] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10932.557654] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10932.560327] ll_atomic_open+0x36a/0xd70 [lustre] [10932.562729] lookup_open+0x356/0xb40 [10932.565848] do_last+0x313/0xfc0 [10932.577882] ? nd_jump_root+0xe5/0x160 [10932.583022] ? path_init+0x437/0x520 [10932.587840] path_openat+0xf7/0x500 [10932.588964] ? do_raw_spin_unlock+0x75/0x190 [10932.605684] do_filp_open+0x99/0x140 [10932.606963] ? getname_flags+0x6e/0x330 [10932.608207] ? __check_object_size+0xff/0x256 [10932.620368] ? do_raw_spin_unlock+0x75/0x190 [10932.621757] ? _raw_spin_unlock+0x12/0x30 [10932.628047] do_sys_openat2+0x2b4/0x410 [10932.630660] do_sys_open+0x73/0xa0 [10932.634678] __x64_sys_openat+0x24/0x30 [10932.635929] do_syscall_64+0xc1/0x440 [10932.646834] entry_SYSCALL_64_after_hwframe+0x49/0xae [10932.648801] RIP: 0033:0x7f53425a5332 [10932.659425] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10932.679743] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10932.693470] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10932.701013] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10932.713887] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10932.719021] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10932.723830] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10948.058011] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [10948.063392] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10948.104217] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10948.115223] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10948.124572] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10948.128029] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10948.144926] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10948.149614] RAX: 0000000000006eba RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10948.153919] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10948.159670] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10948.164790] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10948.168178] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10948.171726] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10948.176026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10948.178050] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10948.180998] Call Trace: [10948.183909] [10948.184681] ? show_regs.cold.9+0x22/0x2f [10948.188495] ? watchdog_timer_fn.cold.6+0x62/0x10a [10948.191157] ? __hrtimer_run_queues+0x1c2/0x4d0 [10948.193148] ? watchdog+0x40/0x40 [10948.195243] ? hrtimer_interrupt+0x128/0x2c0 [10948.201971] ? smp_apic_timer_interrupt+0xa6/0x270 [10948.206010] ? apic_timer_interrupt+0xf/0x20 [10948.208755] [10948.209799] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10948.212577] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10948.216987] do_raw_spin_lock+0x177/0x190 [10948.220327] _raw_spin_lock+0x21/0x30 [10948.223038] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10948.227936] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10948.230257] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10948.233258] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10948.236213] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10948.239494] ? woken_wake_function+0x30/0x30 [10948.241185] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10948.246342] kthread+0x1d1/0x200 [10948.248619] ? set_kthread_struct+0x70/0x70 [10948.252279] ret_from_fork+0x1f/0x30 [10956.062473] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [10956.077526] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10956.130539] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10956.159325] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10956.175254] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10956.186872] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10956.193049] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10956.216651] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [10956.222556] RAX: 0000000000006884 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10956.227309] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10956.233206] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10956.238164] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10956.249217] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10956.258623] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10956.264267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10956.271250] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10956.276263] Call Trace: [10956.277680] [10956.278577] ? show_regs.cold.9+0x22/0x2f [10956.282407] ? watchdog_timer_fn.cold.6+0x62/0x10a [10956.287887] ? __hrtimer_run_queues+0x1c2/0x4d0 [10956.290122] ? watchdog+0x40/0x40 [10956.292015] ? hrtimer_interrupt+0x128/0x2c0 [10956.295627] ? smp_apic_timer_interrupt+0xa6/0x270 [10956.298659] ? apic_timer_interrupt+0xf/0x20 [10956.301237] [10956.302579] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10956.307095] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10956.313157] ? newidle_balance.isra.93+0x207/0x580 [10956.316064] do_raw_spin_lock+0x177/0x190 [10956.321140] _raw_spin_lock+0x21/0x30 [10956.326077] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10956.331412] process_one_work+0x2c8/0x700 [10956.335822] worker_thread+0x296/0x6e0 [10956.337109] ? rescuer_thread+0x570/0x570 [10956.345373] kthread+0x1d1/0x200 [10956.346535] ? set_kthread_struct+0x70/0x70 [10956.349256] ret_from_fork+0x1f/0x30 [10958.817850] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8321s! [10958.824178] Showing busy workqueues and worker pools: [10958.827765] workqueue events: flags=0x0 [10958.830703] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10958.834068] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10958.843930] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10958.849345] pending: cpu_latency_work [ptlrpc] [10958.852912] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10958.857596] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10958.861841] workqueue events_long: flags=0x0 [10958.863361] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10958.871654] pending: lu_objects_flush [obdclass] [10958.875316] workqueue events_unbound: flags=0x2 [10958.880429] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10958.885832] pending: flush_memcg_stats_dwork, idle_cull_fn [10958.889475] workqueue events_power_efficient: flags=0x80 [10958.893616] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10958.900775] pending: check_lifetime, do_cache_clean [10958.903752] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10958.913902] pending: neigh_periodic_work, neigh_periodic_work [10958.918822] workqueue mm_percpu_wq: flags=0x8 [10958.921907] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10958.926806] pending: vmstat_update, lru_add_drain_per_cpu [10958.931571] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10958.936945] pending: vmstat_update, lru_add_drain_per_cpu [10958.939895] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10958.943823] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10958.948725] workqueue xprtiod: flags=0xa [10958.952345] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10958.957996] pending: xprt_autoclose, xs_error_handle [10958.961616] workqueue ipv6_addrconf: flags=0x40008 [10958.965293] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10958.969439] pending: addrconf_verify_work [10958.971110] workqueue knbd0-recv: flags=0x1a [10958.972610] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10958.976147] in-flight: 651:recv_work [10958.979239] workqueue ptlrpc_pinger: flags=0x2 [10958.981168] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10958.987471] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10958.992693] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8321s workers=4 idle: 15979 31824 8 [10958.997267] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10949s workers=2 idle: 654 [10960.055031] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [10960.057185] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [10960.059730] Modules linked in: [10960.062371] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10960.062451] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10960.062458] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10960.062461] RIP: 0010:native_safe_halt+0x2e/0x50 [10960.062476] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10960.062481] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10960.062486] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [10960.062488] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [10960.062490] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [10960.062492] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [10960.062494] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [10960.062497] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [10960.062500] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10960.062502] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [10960.062510] Call Trace: [10960.062516] [10960.062522] ? show_regs.cold.9+0x22/0x2f [10960.062532] ? watchdog_timer_fn.cold.6+0x62/0x10a [10960.062539] ? __hrtimer_run_queues+0x1c2/0x4d0 [10960.062543] ? watchdog+0x40/0x40 [10960.062548] ? hrtimer_interrupt+0x128/0x2c0 [10960.062552] ? smp_apic_timer_interrupt+0xa6/0x270 [10960.062559] ? apic_timer_interrupt+0xf/0x20 [10960.062563] [10960.062566] ? native_safe_halt+0x2e/0x50 [10960.062569] kvm_wait+0xcd/0xe0 [10960.062576] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10960.062584] do_raw_spin_lock+0x177/0x190 [10960.062588] _raw_spin_lock+0x21/0x30 [10960.062593] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10960.064918] lustre(O) [10960.082836] obd_mod_free+0x46/0x140 [ptlrpc] [10960.096534] osp(O) [10960.100283] mdc_commit_open+0x91/0xb0 [mdc] [10960.100334] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10960.100654] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10960.103183] ofd(O) [10960.110281] after_reply+0xe2c/0x1780 [ptlrpc] [10960.117453] lod(O) [10960.120378] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10960.125990] mdt(O) [10960.128860] ? __next_timer_interrupt+0x160/0x160 [10960.128885] ptlrpcd+0xa82/0xe40 [ptlrpc] [10960.138016] mdd(O) [10960.141059] ? do_wait_intr+0xf0/0xf0 [10960.149225] mgs(O) [10960.152302] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10960.167522] lquota(O) [10960.168490] kthread+0x1d1/0x200 [10960.169190] lfsck(O) [10960.170717] ? set_kthread_struct+0x70/0x70 [10960.175725] obdecho(O) [10960.177848] ret_from_fork+0x1f/0x30 [10960.296671] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10960.331912] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10960.337577] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10960.340939] RIP: 0010:native_safe_halt+0x2e/0x50 [10960.343391] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [10960.350676] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10960.353850] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10960.357127] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10960.361267] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10960.366497] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10960.369495] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10960.375347] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10960.380342] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10960.387961] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10960.391585] Call Trace: [10960.393407] [10960.396337] ? show_regs.cold.9+0x22/0x2f [10960.398652] ? watchdog_timer_fn.cold.6+0x62/0x10a [10960.401837] ? __hrtimer_run_queues+0x1c2/0x4d0 [10960.407574] ? watchdog+0x40/0x40 [10960.413701] ? hrtimer_interrupt+0x128/0x2c0 [10960.421158] ? smp_apic_timer_interrupt+0xa6/0x270 [10960.425252] ? apic_timer_interrupt+0xf/0x20 [10960.427522] [10960.428339] ? native_safe_halt+0x2e/0x50 [10960.431504] kvm_wait+0xcd/0xe0 [10960.437506] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10960.444392] do_raw_spin_lock+0x177/0x190 [10960.447388] _raw_spin_lock+0x21/0x30 [10960.451056] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10960.460356] ? do_raw_spin_unlock+0x75/0x190 [10960.466546] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10960.472555] ? do_wait_intr+0xf0/0xf0 [10960.475521] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10960.482176] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10960.484711] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10960.488129] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10960.495037] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10960.500915] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10960.504161] mdc_intent_lock+0x353/0x7e0 [mdc] [10960.505390] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10960.515289] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10960.522832] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10960.528163] lmv_intent_open+0x338/0x1150 [lmv] [10960.533735] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10960.535687] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10960.547023] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10960.554180] ll_intent_lock+0xc5/0xaa0 [lustre] [10960.557056] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10960.561399] ll_lookup_it+0xe0e/0x2270 [lustre] [10960.564166] ? do_raw_spin_unlock+0x75/0x190 [10960.568232] ? _raw_spin_unlock+0x12/0x30 [10960.574019] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10960.575905] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10960.579162] ll_atomic_open+0x36a/0xd70 [lustre] [10960.585581] lookup_open+0x356/0xb40 [10960.589154] do_last+0x313/0xfc0 [10960.590956] ? nd_jump_root+0xe5/0x160 [10960.593526] ? path_init+0x437/0x520 [10960.596522] path_openat+0xf7/0x500 [10960.598173] ? do_raw_spin_unlock+0x75/0x190 [10960.600044] do_filp_open+0x99/0x140 [10960.601931] ? getname_flags+0x6e/0x330 [10960.606420] ? __check_object_size+0xff/0x256 [10960.609503] ? do_raw_spin_unlock+0x75/0x190 [10960.612119] ? _raw_spin_unlock+0x12/0x30 [10960.614336] do_sys_openat2+0x2b4/0x410 [10960.616491] do_sys_open+0x73/0xa0 [10960.618800] __x64_sys_openat+0x24/0x30 [10960.621388] do_syscall_64+0xc1/0x440 [10960.623514] entry_SYSCALL_64_after_hwframe+0x49/0xae [10960.626575] RIP: 0033:0x7f53425a5332 [10960.628303] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10960.638106] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10960.642110] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10960.645553] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10960.648566] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10960.652212] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10960.655264] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10974.637123] rcu: INFO: rcu_sched self-detected stall on CPU [10974.640248] rcu: 2-....: (4869848 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [10974.645166] (t=8880211 jiffies g=274185 q=66128) [10974.646994] rcu: rcu_sched kthread starved for 8338577 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [10974.651418] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [10974.655630] rcu: RCU grace-period kthread stack dump: [10974.657658] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [10974.661631] Call Trace: [10974.662997] __schedule+0x351/0xcb0 [10974.664521] schedule+0xc0/0x180 [10974.666246] schedule_timeout+0xb4/0x190 [10974.668044] ? do_raw_spin_unlock+0x75/0x190 [10974.669694] ? __next_timer_interrupt+0x160/0x160 [10974.671582] ? prepare_to_swait_event+0xf3/0x140 [10974.673271] rcu_gp_kthread+0x8dd/0x15b0 [10974.675001] ? force_qs_rnp+0x2c0/0x2c0 [10974.676452] kthread+0x1d1/0x200 [10974.678005] ? set_kthread_struct+0x70/0x70 [10974.680025] ret_from_fork+0x1f/0x30 [10974.682048] rcu: Stack dump where RCU GP kthread last ran: [10974.684309] Sending NMI from CPU 2 to CPUs 1: [10974.688578] NMI backtrace for cpu 1 [10974.688584] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10974.688586] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10974.688588] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10974.688590] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10974.688593] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [10974.688595] RAX: 0000000000005d4d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10974.688597] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10974.688598] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10974.688600] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10974.688605] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10974.688607] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10974.688608] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10974.688609] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10974.688611] Call Trace: [10974.688612] [10974.688613] ? show_regs.cold.9+0x22/0x2f [10974.688614] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10974.688615] ? nmi_cpu_backtrace_handler+0x15/0x40 [10974.688617] ? nmi_handle+0xbd/0x210 [10974.688618] ? default_do_nmi+0x4b/0x1b0 [10974.688619] ? do_nmi+0x280/0x380 [10974.688620] ? end_repeat_nmi+0x16/0x1f [10974.688621] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10974.688622] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10974.688624] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10974.688625] [10974.688626] do_raw_spin_lock+0x177/0x190 [10974.688628] _raw_spin_lock+0x21/0x30 [10974.688629] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10974.688630] ? do_raw_spin_unlock+0x75/0x190 [10974.688631] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10974.688632] ? do_wait_intr+0xf0/0xf0 [10974.688633] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10974.688634] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10974.688635] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10974.688637] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10974.688638] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10974.688639] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10974.688640] mdc_intent_lock+0x353/0x7e0 [mdc] [10974.688641] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10974.688642] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10974.688643] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10974.688645] lmv_intent_open+0x338/0x1150 [lmv] [10974.688646] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10974.688647] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10974.688648] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10974.688649] ll_intent_lock+0xc5/0xaa0 [lustre] [10974.688650] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10974.688651] ll_lookup_it+0xe0e/0x2270 [lustre] [10974.688653] ? do_raw_spin_unlock+0x75/0x190 [10974.688654] ? _raw_spin_unlock+0x12/0x30 [10974.688655] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10974.688656] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10974.688657] ll_atomic_open+0x36a/0xd70 [lustre] [10974.688658] lookup_open+0x356/0xb40 [10974.688659] do_last+0x313/0xfc0 [10974.688660] ? nd_jump_root+0xe5/0x160 [10974.688661] ? path_init+0x437/0x520 [10974.688662] path_openat+0xf7/0x500 [10974.688663] ? do_raw_spin_unlock+0x75/0x190 [10974.688664] do_filp_open+0x99/0x140 [10974.688665] ? getname_flags+0x6e/0x330 [10974.688666] ? __check_object_size+0xff/0x256 [10974.688667] ? do_raw_spin_unlock+0x75/0x190 [10974.688668] ? _raw_spin_unlock+0x12/0x30 [10974.688669] do_sys_openat2+0x2b4/0x410 [10974.688669] do_sys_open+0x73/0xa0 [10974.688670] __x64_sys_openat+0x24/0x30 [10974.688671] do_syscall_64+0xc1/0x440 [10974.688672] entry_SYSCALL_64_after_hwframe+0x49/0xae [10974.688673] RIP: 0033:0x7f53425a5332 [10974.688674] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10974.688676] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10974.688679] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10974.688680] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10974.688686] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10974.688687] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10974.688689] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10974.691485] Sending NMI from CPU 2 to CPUs 1: [10974.875629] NMI backtrace for cpu 1 [10974.875637] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10974.875639] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10974.875641] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10974.875642] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10974.875644] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [10974.875646] RAX: 0000000000002c12 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10974.875648] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [10974.875649] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [10974.875651] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [10974.875652] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [10974.875654] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [10974.875655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10974.875656] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [10974.875658] Call Trace: [10974.875658] [10974.875659] ? show_regs.cold.9+0x22/0x2f [10974.875660] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [10974.875661] ? nmi_cpu_backtrace_handler+0x15/0x40 [10974.875662] ? nmi_handle+0xbd/0x210 [10974.875663] ? default_do_nmi+0x4b/0x1b0 [10974.875664] ? do_nmi+0x280/0x380 [10974.875665] ? end_repeat_nmi+0x16/0x1f [10974.875667] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10974.875668] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10974.875670] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10974.875671] [10974.875671] do_raw_spin_lock+0x177/0x190 [10974.875672] _raw_spin_lock+0x21/0x30 [10974.875674] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [10974.875675] ? do_raw_spin_unlock+0x75/0x190 [10974.875676] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [10974.875678] ? do_wait_intr+0xf0/0xf0 [10974.875679] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [10974.875680] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [10974.875681] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10974.875682] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10974.875683] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10974.875685] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [10974.875686] mdc_intent_lock+0x353/0x7e0 [mdc] [10974.875687] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10974.875689] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [10974.875690] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [10974.875691] lmv_intent_open+0x338/0x1150 [lmv] [10974.875692] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10974.875694] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10974.875695] lmv_intent_lock+0x1dd/0x4c0 [lmv] [10974.875696] ll_intent_lock+0xc5/0xaa0 [lustre] [10974.875697] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [10974.875698] ll_lookup_it+0xe0e/0x2270 [lustre] [10974.875700] ? do_raw_spin_unlock+0x75/0x190 [10974.875701] ? _raw_spin_unlock+0x12/0x30 [10974.875702] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [10974.875703] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [10974.875705] ll_atomic_open+0x36a/0xd70 [lustre] [10974.875706] lookup_open+0x356/0xb40 [10974.875707] do_last+0x313/0xfc0 [10974.875708] ? nd_jump_root+0xe5/0x160 [10974.875709] ? path_init+0x437/0x520 [10974.875710] path_openat+0xf7/0x500 [10974.875711] ? do_raw_spin_unlock+0x75/0x190 [10974.875713] do_filp_open+0x99/0x140 [10974.875713] ? getname_flags+0x6e/0x330 [10974.875714] ? __check_object_size+0xff/0x256 [10974.875715] ? do_raw_spin_unlock+0x75/0x190 [10974.875716] ? _raw_spin_unlock+0x12/0x30 [10974.875717] do_sys_openat2+0x2b4/0x410 [10974.875718] do_sys_open+0x73/0xa0 [10974.875719] __x64_sys_openat+0x24/0x30 [10974.875720] do_syscall_64+0xc1/0x440 [10974.875721] entry_SYSCALL_64_after_hwframe+0x49/0xae [10974.875722] RIP: 0033:0x7f53425a5332 [10974.875724] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [10974.875726] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [10974.875729] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [10974.875730] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [10974.875732] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [10974.875733] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [10974.875734] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [10974.876930] NMI backtrace for cpu 2 [10975.065609] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10975.071081] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10975.074654] Call Trace: [10975.075625] [10975.076555] ? dump_stack+0xbb/0x10e [10975.078125] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [10975.080350] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [10975.081770] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [10975.083163] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [10975.085018] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [10975.087237] ? rcu_dump_cpu_stacks+0x11b/0x171 [10975.089225] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [10975.091245] ? update_process_times+0x6d/0xc0 [10975.092906] ? tick_sched_handle.isra.18+0x27/0x90 [10975.094273] ? tick_sched_timer+0x53/0xb0 [10975.095563] ? __hrtimer_run_queues+0x1c2/0x4d0 [10975.097587] ? tick_sched_handle.isra.18+0x90/0x90 [10975.099584] ? hrtimer_interrupt+0x128/0x2c0 [10975.101658] ? smp_apic_timer_interrupt+0xa6/0x270 [10975.103411] ? apic_timer_interrupt+0xf/0x20 [10975.105403] [10975.106172] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [10975.108525] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [10975.111029] ? do_raw_spin_lock+0x177/0x190 [10975.112732] ? _raw_spin_lock+0x21/0x30 [10975.114324] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [10975.116558] ? obd_mod_free+0x46/0x140 [ptlrpc] [10975.119248] ? mdc_commit_open+0x91/0xb0 [mdc] [10975.121030] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [10975.123715] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [10975.126315] ? after_reply+0xe2c/0x1780 [ptlrpc] [10975.128277] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [10975.130742] ? __next_timer_interrupt+0x160/0x160 [10975.132897] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [10975.134511] ? do_wait_intr+0xf0/0xf0 [10975.135670] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [10975.138098] ? kthread+0x1d1/0x200 [10975.139628] ? set_kthread_struct+0x70/0x70 [10975.141278] ? ret_from_fork+0x1f/0x30 [10976.058739] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [10976.070184] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10976.111188] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10976.122210] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10976.127169] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10976.131245] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10976.151547] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10976.156564] RAX: 00000000000024b4 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10976.163817] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [10976.175262] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [10976.180224] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [10976.186210] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [10976.193554] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [10976.199122] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10976.203307] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [10976.214620] Call Trace: [10976.219144] [10976.220784] ? show_regs.cold.9+0x22/0x2f [10976.227050] ? watchdog_timer_fn.cold.6+0x62/0x10a [10976.231841] ? __hrtimer_run_queues+0x1c2/0x4d0 [10976.239078] ? watchdog+0x40/0x40 [10976.241791] ? hrtimer_interrupt+0x128/0x2c0 [10976.248171] ? smp_apic_timer_interrupt+0xa6/0x270 [10976.255182] ? apic_timer_interrupt+0xf/0x20 [10976.258035] [10976.258795] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10976.265049] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10976.266928] do_raw_spin_lock+0x177/0x190 [10976.269435] _raw_spin_lock+0x21/0x30 [10976.273080] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [10976.278998] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [10976.288451] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [10976.293467] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [10976.299131] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [10976.302830] ? woken_wake_function+0x30/0x30 [10976.306348] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [10976.310252] kthread+0x1d1/0x200 [10976.312425] ? set_kthread_struct+0x70/0x70 [10976.314421] ret_from_fork+0x1f/0x30 [10984.052466] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [10984.061733] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [10984.099785] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [10984.113116] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [10984.118543] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [10984.121691] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10984.126500] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [10984.146466] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [10984.156895] RAX: 0000000000007c83 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [10984.160425] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [10984.167654] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [10984.170056] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [10984.177573] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [10984.181846] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [10984.195610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10984.201362] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [10984.203880] Call Trace: [10984.209072] [10984.210489] ? show_regs.cold.9+0x22/0x2f [10984.216487] ? watchdog_timer_fn.cold.6+0x62/0x10a [10984.220072] ? __hrtimer_run_queues+0x1c2/0x4d0 [10984.225605] ? watchdog+0x40/0x40 [10984.227412] ? hrtimer_interrupt+0x128/0x2c0 [10984.231953] ? smp_apic_timer_interrupt+0xa6/0x270 [10984.233587] ? apic_timer_interrupt+0xf/0x20 [10984.234802] [10984.241567] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [10984.245652] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [10984.256545] ? newidle_balance.isra.93+0x207/0x580 [10984.258118] do_raw_spin_lock+0x177/0x190 [10984.263422] _raw_spin_lock+0x21/0x30 [10984.269606] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [10984.273607] process_one_work+0x2c8/0x700 [10984.275774] worker_thread+0x296/0x6e0 [10984.279731] ? rescuer_thread+0x570/0x570 [10984.282487] kthread+0x1d1/0x200 [10984.289438] ? set_kthread_struct+0x70/0x70 [10984.290854] ret_from_fork+0x1f/0x30 [10989.025582] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8351s! [10989.035752] Showing busy workqueues and worker pools: [10989.040773] workqueue events: flags=0x0 [10989.044375] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [10989.054738] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [10989.061262] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10989.067737] pending: cpu_latency_work [ptlrpc] [10989.073036] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [10989.083359] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [10989.090994] workqueue events_long: flags=0x0 [10989.093996] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [10989.098544] pending: lu_objects_flush [obdclass] [10989.110212] workqueue events_unbound: flags=0x2 [10989.113706] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [10989.115822] pending: flush_memcg_stats_dwork, idle_cull_fn [10989.139366] workqueue events_power_efficient: flags=0x80 [10989.141327] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10989.157029] pending: check_lifetime, do_cache_clean [10989.168782] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10989.171073] pending: neigh_periodic_work, neigh_periodic_work [10989.200705] workqueue mm_percpu_wq: flags=0x8 [10989.205153] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10989.220816] pending: vmstat_update, lru_add_drain_per_cpu [10989.230352] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [10989.232716] pending: vmstat_update, lru_add_drain_per_cpu [10989.251304] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [10989.261804] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [10989.271014] workqueue xprtiod: flags=0xa [10989.284279] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [10989.291471] pending: xprt_autoclose, xs_error_handle [10989.299869] workqueue ipv6_addrconf: flags=0x40008 [10989.311563] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [10989.321085] pending: addrconf_verify_work [10989.322697] workqueue knbd0-recv: flags=0x1a [10989.328748] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [10989.338898] in-flight: 651:recv_work [10989.340347] workqueue ptlrpc_pinger: flags=0x2 [10989.350267] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [10989.362406] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [10989.380038] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8352s workers=4 idle: 15979 31824 8 [10989.396986] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=10979s workers=2 idle: 654 [11000.063476] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11000.063625] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11000.073108] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11000.073202] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11000.073209] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11000.073213] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11000.073227] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11000.073231] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11000.073235] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11000.073238] RDX: 000000000000030d RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11000.073241] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11000.073243] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11000.073245] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11000.073249] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11000.073252] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11000.073255] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11000.073264] Call Trace: [11000.073269] [11000.073276] ? show_regs.cold.9+0x22/0x2f [11000.073286] ? watchdog_timer_fn.cold.6+0x62/0x10a [11000.073292] ? __hrtimer_run_queues+0x1c2/0x4d0 [11000.073296] ? watchdog+0x40/0x40 [11000.073301] ? hrtimer_interrupt+0x128/0x2c0 [11000.073305] ? smp_apic_timer_interrupt+0xa6/0x270 [11000.073313] ? apic_timer_interrupt+0xf/0x20 [11000.073318] [11000.073320] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11000.073325] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11000.073330] do_raw_spin_lock+0x177/0x190 [11000.073335] _raw_spin_lock+0x21/0x30 [11000.073341] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11000.075838] obd_mod_free+0x46/0x140 [ptlrpc] [11000.076110] mdc_commit_open+0x91/0xb0 [mdc] [11000.076145] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11000.099972] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11000.100058] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11000.100065] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11000.100068] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11000.100082] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11000.100086] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11000.100090] RAX: 0000000000004389 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11000.100093] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11000.100095] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11000.100098] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11000.100100] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11000.100104] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11000.100106] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11000.100109] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11000.100119] Call Trace: [11000.100125] [11000.100131] ? show_regs.cold.9+0x22/0x2f [11000.100142] ? watchdog_timer_fn.cold.6+0x62/0x10a [11000.100148] ? __hrtimer_run_queues+0x1c2/0x4d0 [11000.100152] ? watchdog+0x40/0x40 [11000.100157] ? hrtimer_interrupt+0x128/0x2c0 [11000.100161] ? smp_apic_timer_interrupt+0xa6/0x270 [11000.100169] ? apic_timer_interrupt+0xf/0x20 [11000.100174] [11000.100177] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11000.100185] do_raw_spin_lock+0x177/0x190 [11000.100190] _raw_spin_lock+0x21/0x30 [11000.100195] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11000.171821] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11000.172130] after_reply+0xe2c/0x1780 [ptlrpc] [11000.172841] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11000.174108] ? __next_timer_interrupt+0x160/0x160 [11000.174121] ptlrpcd+0xa82/0xe40 [ptlrpc] [11000.174338] ? do_wait_intr+0xf0/0xf0 [11000.174349] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11000.174884] kthread+0x1d1/0x200 [11000.174892] ? set_kthread_struct+0x70/0x70 [11000.174969] ret_from_fork+0x1f/0x30 [11001.000743] ? do_raw_spin_unlock+0x75/0x190 [11001.002213] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11001.011552] ? do_wait_intr+0xf0/0xf0 [11001.012894] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11001.027564] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11001.030789] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11001.052798] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11001.056381] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11001.058328] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11001.071709] mdc_intent_lock+0x353/0x7e0 [mdc] [11001.073294] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11001.080162] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11001.095380] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11001.097283] lmv_intent_open+0x338/0x1150 [lmv] [11001.104264] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11001.117875] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11001.121428] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11001.128026] ll_intent_lock+0xc5/0xaa0 [lustre] [11001.129657] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11001.151531] ll_lookup_it+0xe0e/0x2270 [lustre] [11001.157667] ? do_raw_spin_unlock+0x75/0x190 [11001.159183] ? _raw_spin_unlock+0x12/0x30 [11001.175743] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11001.183341] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11001.194068] ll_atomic_open+0x36a/0xd70 [lustre] [11001.195766] lookup_open+0x356/0xb40 [11001.213057] do_last+0x313/0xfc0 [11001.214238] ? nd_jump_root+0xe5/0x160 [11001.222573] ? path_init+0x437/0x520 [11001.227169] path_openat+0xf7/0x500 [11001.235288] ? do_raw_spin_unlock+0x75/0x190 [11001.236778] do_filp_open+0x99/0x140 [11001.250972] ? getname_flags+0x6e/0x330 [11001.252300] ? __check_object_size+0xff/0x256 [11001.253811] ? do_raw_spin_unlock+0x75/0x190 [11001.269202] ? _raw_spin_unlock+0x12/0x30 [11001.270201] do_sys_openat2+0x2b4/0x410 [11001.287508] do_sys_open+0x73/0xa0 [11001.288752] __x64_sys_openat+0x24/0x30 [11001.295706] do_syscall_64+0xc1/0x440 [11001.297074] entry_SYSCALL_64_after_hwframe+0x49/0xae [11001.298776] RIP: 0033:0x7f53425a5332 [11001.308619] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11001.329705] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11001.342104] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11001.351305] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11001.358651] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11001.374371] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11001.384605] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11004.058245] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [11004.062009] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11004.081233] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11004.085814] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11004.089744] RIP: 0010:native_safe_halt+0x2e/0x50 [11004.091674] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11004.103121] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11004.111768] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11004.120104] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11004.124713] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11004.133729] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11004.140658] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11004.145216] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11004.149546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11004.158335] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11004.160782] Call Trace: [11004.162030] [11004.162710] ? show_regs.cold.9+0x22/0x2f [11004.165421] ? watchdog_timer_fn.cold.6+0x62/0x10a [11004.170726] ? __hrtimer_run_queues+0x1c2/0x4d0 [11004.173914] ? watchdog+0x40/0x40 [11004.176366] ? hrtimer_interrupt+0x128/0x2c0 [11004.178486] ? smp_apic_timer_interrupt+0xa6/0x270 [11004.180419] ? apic_timer_interrupt+0xf/0x20 [11004.182224] [11004.182824] ? native_safe_halt+0x2e/0x50 [11004.183846] kvm_wait+0xcd/0xe0 [11004.184670] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11004.186600] do_raw_spin_lock+0x177/0x190 [11004.188094] _raw_spin_lock+0x21/0x30 [11004.189373] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11004.191418] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11004.192990] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11004.194765] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11004.196168] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11004.197756] ? woken_wake_function+0x30/0x30 [11004.198980] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11004.200710] kthread+0x1d1/0x200 [11004.201611] ? set_kthread_struct+0x70/0x70 [11004.202695] ret_from_fork+0x1f/0x30 [11016.052170] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [11016.056077] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11016.098588] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11016.105273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11016.115144] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11016.122524] RIP: 0010:native_safe_halt+0x2e/0x50 [11016.126142] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11016.144304] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11016.150888] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11016.155419] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [11016.161029] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11016.164431] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11016.172886] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11016.177249] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11016.181751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11016.186755] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11016.190913] Call Trace: [11016.192487] [11016.193890] ? show_regs.cold.9+0x22/0x2f [11016.196598] ? watchdog_timer_fn.cold.6+0x62/0x10a [11016.200023] ? __hrtimer_run_queues+0x1c2/0x4d0 [11016.202415] ? watchdog+0x40/0x40 [11016.204350] ? hrtimer_interrupt+0x128/0x2c0 [11016.206426] ? smp_apic_timer_interrupt+0xa6/0x270 [11016.209139] ? apic_timer_interrupt+0xf/0x20 [11016.212316] [11016.213469] ? native_safe_halt+0x2e/0x50 [11016.216571] kvm_wait+0xcd/0xe0 [11016.218987] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11016.225469] ? newidle_balance.isra.93+0x207/0x580 [11016.230977] do_raw_spin_lock+0x177/0x190 [11016.236244] _raw_spin_lock+0x21/0x30 [11016.239544] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11016.245723] process_one_work+0x2c8/0x700 [11016.248320] worker_thread+0x296/0x6e0 [11016.250972] ? rescuer_thread+0x570/0x570 [11016.255482] kthread+0x1d1/0x200 [11016.259000] ? set_kthread_struct+0x70/0x70 [11016.264931] ret_from_fork+0x1f/0x30 [11019.744300] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8382s! [11019.750404] Showing busy workqueues and worker pools: [11019.755064] workqueue events: flags=0x0 [11019.756885] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11019.763881] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11019.768915] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11019.771796] pending: cpu_latency_work [ptlrpc] [11019.774370] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11019.777184] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11019.780989] workqueue events_long: flags=0x0 [11019.782521] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11019.785632] pending: lu_objects_flush [obdclass] [11019.790075] workqueue events_unbound: flags=0x2 [11019.792293] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11019.798164] pending: flush_memcg_stats_dwork, idle_cull_fn [11019.803318] workqueue events_power_efficient: flags=0x80 [11019.808015] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11019.817164] pending: check_lifetime, do_cache_clean [11019.820575] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11019.823454] pending: neigh_periodic_work, neigh_periodic_work [11019.830905] workqueue mm_percpu_wq: flags=0x8 [11019.836703] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11019.840499] pending: vmstat_update, lru_add_drain_per_cpu [11019.849957] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11019.857598] pending: vmstat_update, lru_add_drain_per_cpu [11019.863370] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11019.866455] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11019.871096] workqueue xprtiod: flags=0xa [11019.873513] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11019.875791] pending: xprt_autoclose, xs_error_handle [11019.880466] workqueue ipv6_addrconf: flags=0x40008 [11019.883333] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11019.887004] pending: addrconf_verify_work [11019.888716] workqueue knbd0-recv: flags=0x1a [11019.890282] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11019.893371] in-flight: 651:recv_work [11019.895078] workqueue ptlrpc_pinger: flags=0x2 [11019.896897] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11019.899569] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11019.902120] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8382s workers=4 idle: 15979 31824 8 [11019.905399] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11010s workers=2 idle: 654 [11028.055120] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11028.058297] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11028.061452] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11028.078633] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11028.078648] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11028.078651] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11028.078664] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11028.078669] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11028.078674] RAX: 00000000000007f7 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11028.078677] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11028.078679] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11028.078681] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11028.078684] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11028.078687] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11028.078690] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11028.078692] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11028.078700] Call Trace: [11028.078706] [11028.078712] ? show_regs.cold.9+0x22/0x2f [11028.078721] ? watchdog_timer_fn.cold.6+0x62/0x10a [11028.078728] ? __hrtimer_run_queues+0x1c2/0x4d0 [11028.078732] ? watchdog+0x40/0x40 [11028.078736] ? hrtimer_interrupt+0x128/0x2c0 [11028.078740] ? smp_apic_timer_interrupt+0xa6/0x270 [11028.078749] ? apic_timer_interrupt+0xf/0x20 [11028.078754] [11028.078756] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11028.078764] do_raw_spin_lock+0x177/0x190 [11028.078769] _raw_spin_lock+0x21/0x30 [11028.078774] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11028.079191] ? do_raw_spin_unlock+0x75/0x190 [11028.079198] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11028.079486] ? do_wait_intr+0xf0/0xf0 [11028.079498] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11028.079696] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11028.083826] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11028.083983] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11028.083991] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11028.083994] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11028.084007] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11028.084011] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11028.084017] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11028.084019] RDX: 0000000000004e1f RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11028.084021] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11028.084023] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11028.084026] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11028.084029] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11028.084032] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11028.084034] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11028.084041] Call Trace: [11028.084051] [11028.084057] ? show_regs.cold.9+0x22/0x2f [11028.084066] ? watchdog_timer_fn.cold.6+0x62/0x10a [11028.084072] ? __hrtimer_run_queues+0x1c2/0x4d0 [11028.084076] ? watchdog+0x40/0x40 [11028.084081] ? hrtimer_interrupt+0x128/0x2c0 [11028.084084] ? smp_apic_timer_interrupt+0xa6/0x270 [11028.084093] ? apic_timer_interrupt+0xf/0x20 [11028.084098] [11028.084101] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11028.084106] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11028.084110] do_raw_spin_lock+0x177/0x190 [11028.084114] _raw_spin_lock+0x21/0x30 [11028.084121] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11028.084659] obd_mod_free+0x46/0x140 [ptlrpc] [11028.090175] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11028.090650] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11028.090794] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11028.090856] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11028.091002] mdc_intent_lock+0x353/0x7e0 [mdc] [11028.091048] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11028.091156] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11028.091572] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11028.091614] lmv_intent_open+0x338/0x1150 [lmv] [11028.091645] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11028.091795] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11028.091888] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11028.091928] ll_intent_lock+0xc5/0xaa0 [lustre] [11028.092032] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11028.092120] ll_lookup_it+0xe0e/0x2270 [lustre] [11028.092207] ? do_raw_spin_unlock+0x75/0x190 [11028.092215] ? _raw_spin_unlock+0x12/0x30 [11028.092223] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11028.092252] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11028.092273] ll_atomic_open+0x36a/0xd70 [lustre] [11028.096064] mdc_commit_open+0x91/0xb0 [mdc] [11028.096120] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11028.096555] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11028.096769] after_reply+0xe2c/0x1780 [ptlrpc] [11028.096986] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11028.097287] ? __next_timer_interrupt+0x160/0x160 [11028.097296] ptlrpcd+0xa82/0xe40 [ptlrpc] [11028.097580] ? do_wait_intr+0xf0/0xf0 [11028.097594] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11028.097795] kthread+0x1d1/0x200 [11028.097801] ? set_kthread_struct+0x70/0x70 [11028.097804] ret_from_fork+0x1f/0x30 [11028.335580] lookup_open+0x356/0xb40 [11028.337209] do_last+0x313/0xfc0 [11028.338594] ? nd_jump_root+0xe5/0x160 [11028.340152] ? path_init+0x437/0x520 [11028.341751] path_openat+0xf7/0x500 [11028.344098] ? do_raw_spin_unlock+0x75/0x190 [11028.346086] do_filp_open+0x99/0x140 [11028.347501] ? getname_flags+0x6e/0x330 [11028.349866] ? __check_object_size+0xff/0x256 [11028.351756] ? do_raw_spin_unlock+0x75/0x190 [11028.353612] ? _raw_spin_unlock+0x12/0x30 [11028.355143] do_sys_openat2+0x2b4/0x410 [11028.356869] do_sys_open+0x73/0xa0 [11028.359091] __x64_sys_openat+0x24/0x30 [11028.360756] do_syscall_64+0xc1/0x440 [11028.362345] entry_SYSCALL_64_after_hwframe+0x49/0xae [11028.364833] RIP: 0033:0x7f53425a5332 [11028.366348] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11028.374715] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11028.378164] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11028.381829] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11028.385257] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11028.388507] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11028.391842] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11032.058764] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11032.065800] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11032.101810] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11032.109120] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11032.117876] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11032.125223] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11032.139719] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11032.145256] RAX: 0000000000004f89 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11032.154233] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11032.159367] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11032.163299] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11032.166473] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11032.171757] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11032.185316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11032.187398] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11032.196497] Call Trace: [11032.199416] [11032.200400] ? show_regs.cold.9+0x22/0x2f [11032.203803] ? watchdog_timer_fn.cold.6+0x62/0x10a [11032.208492] ? __hrtimer_run_queues+0x1c2/0x4d0 [11032.217985] ? watchdog+0x40/0x40 [11032.222101] ? hrtimer_interrupt+0x128/0x2c0 [11032.230517] ? smp_apic_timer_interrupt+0xa6/0x270 [11032.232251] ? apic_timer_interrupt+0xf/0x20 [11032.241275] [11032.243727] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11032.245147] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11032.251091] do_raw_spin_lock+0x177/0x190 [11032.253213] _raw_spin_lock+0x21/0x30 [11032.254734] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11032.257450] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11032.260536] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11032.264493] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11032.270131] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11032.274386] ? woken_wake_function+0x30/0x30 [11032.353443] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11032.356008] kthread+0x1d1/0x200 [11032.357275] ? set_kthread_struct+0x70/0x70 [11032.358996] ret_from_fork+0x1f/0x30 [11044.052286] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [11044.055438] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11044.071976] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11044.076723] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11044.079964] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11044.082577] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11044.084657] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11044.091532] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11044.094501] RAX: 0000000000004df3 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11044.097004] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11044.099642] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11044.102431] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11044.104881] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11044.107603] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11044.110118] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11044.112794] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11044.115430] Call Trace: [11044.116433] [11044.116966] ? show_regs.cold.9+0x22/0x2f [11044.118651] ? watchdog_timer_fn.cold.6+0x62/0x10a [11044.120523] ? __hrtimer_run_queues+0x1c2/0x4d0 [11044.122661] ? watchdog+0x40/0x40 [11044.124021] ? hrtimer_interrupt+0x128/0x2c0 [11044.125685] ? smp_apic_timer_interrupt+0xa6/0x270 [11044.128085] ? apic_timer_interrupt+0xf/0x20 [11044.129713] [11044.130653] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11044.132524] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11044.135715] ? newidle_balance.isra.93+0x207/0x580 [11044.137391] do_raw_spin_lock+0x177/0x190 [11044.139245] _raw_spin_lock+0x21/0x30 [11044.141253] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11044.143615] process_one_work+0x2c8/0x700 [11044.145284] worker_thread+0x296/0x6e0 [11044.146872] ? rescuer_thread+0x570/0x570 [11044.148409] kthread+0x1d1/0x200 [11044.149853] ? set_kthread_struct+0x70/0x70 [11044.151378] ret_from_fork+0x1f/0x30 [11049.953315] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8412s! [11049.963639] Showing busy workqueues and worker pools: [11049.970519] workqueue events: flags=0x0 [11049.974742] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11049.981120] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11049.989102] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11049.996942] pending: cpu_latency_work [ptlrpc] [11050.003251] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11050.008164] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11050.012068] workqueue events_long: flags=0x0 [11050.015868] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11050.018086] pending: lu_objects_flush [obdclass] [11050.020093] workqueue events_unbound: flags=0x2 [11050.021677] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11050.027896] pending: flush_memcg_stats_dwork, idle_cull_fn [11050.034134] workqueue events_power_efficient: flags=0x80 [11050.035982] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11050.041791] pending: check_lifetime, do_cache_clean [11050.044886] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11050.048235] pending: neigh_periodic_work, neigh_periodic_work [11050.051212] workqueue mm_percpu_wq: flags=0x8 [11050.057358] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11050.065920] pending: vmstat_update, lru_add_drain_per_cpu [11050.069913] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11050.074105] pending: vmstat_update, lru_add_drain_per_cpu [11050.078216] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11050.086737] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11050.090362] workqueue xprtiod: flags=0xa [11050.095362] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11050.103046] pending: xprt_autoclose, xs_error_handle [11050.106946] workqueue ipv6_addrconf: flags=0x40008 [11050.109446] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11050.112619] pending: addrconf_verify_work [11050.116033] workqueue knbd0-recv: flags=0x1a [11050.118887] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11050.123912] in-flight: 651:recv_work [11050.125927] workqueue ptlrpc_pinger: flags=0x2 [11050.128950] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11050.132517] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11050.137189] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8413s workers=4 idle: 15979 31824 8 [11050.146724] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11040s workers=2 idle: 654 [11056.056824] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11056.057012] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11056.061630] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11056.061745] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11056.061753] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11056.061756] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11056.061770] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11056.061775] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11056.061779] RAX: 0000000000003312 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11056.061781] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11056.061783] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11056.061786] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11056.061788] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11056.061792] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11056.061795] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11056.061797] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11056.061805] Call Trace: [11056.061810] [11056.061815] ? show_regs.cold.9+0x22/0x2f [11056.061823] ? watchdog_timer_fn.cold.6+0x62/0x10a [11056.061829] ? __hrtimer_run_queues+0x1c2/0x4d0 [11056.061833] ? watchdog+0x40/0x40 [11056.061837] ? hrtimer_interrupt+0x128/0x2c0 [11056.061842] ? smp_apic_timer_interrupt+0xa6/0x270 [11056.067625] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11056.067717] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11056.067728] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11056.067731] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11056.067757] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11056.067762] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11056.067766] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11056.067771] RDX: 0000000000007644 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11056.067774] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11056.067776] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11056.067778] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11056.067781] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11056.067784] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11056.067786] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11056.067795] Call Trace: [11056.067801] [11056.067806] ? show_regs.cold.9+0x22/0x2f [11056.067815] ? watchdog_timer_fn.cold.6+0x62/0x10a [11056.067823] ? __hrtimer_run_queues+0x1c2/0x4d0 [11056.067827] ? watchdog+0x40/0x40 [11056.067831] ? hrtimer_interrupt+0x128/0x2c0 [11056.067835] ? smp_apic_timer_interrupt+0xa6/0x270 [11056.067852] ? apic_timer_interrupt+0xf/0x20 [11056.067858] [11056.067861] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11056.067872] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11056.067878] do_raw_spin_lock+0x177/0x190 [11056.067883] _raw_spin_lock+0x21/0x30 [11056.067889] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11056.068445] obd_mod_free+0x46/0x140 [ptlrpc] [11056.068694] mdc_commit_open+0x91/0xb0 [mdc] [11056.068737] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11056.069665] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11056.070336] after_reply+0xe2c/0x1780 [ptlrpc] [11056.070696] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11056.071206] ? __next_timer_interrupt+0x160/0x160 [11056.071226] ptlrpcd+0xa82/0xe40 [ptlrpc] [11056.071987] ? do_wait_intr+0xf0/0xf0 [11056.071999] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11056.128496] ? apic_timer_interrupt+0xf/0x20 [11056.136732] kthread+0x1d1/0x200 [11056.136760] ? set_kthread_struct+0x70/0x70 [11056.136764] ret_from_fork+0x1f/0x30 [11056.350593] [11056.352084] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11056.356171] do_raw_spin_lock+0x177/0x190 [11056.359419] _raw_spin_lock+0x21/0x30 [11056.360815] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11056.363352] ? do_raw_spin_unlock+0x75/0x190 [11056.365322] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11056.367456] ? do_wait_intr+0xf0/0xf0 [11056.369308] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11056.371861] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11056.373880] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11056.376625] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11056.379030] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11056.381696] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11056.383136] mdc_intent_lock+0x353/0x7e0 [mdc] [11056.385043] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11056.390395] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11056.396033] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11056.398425] lmv_intent_open+0x338/0x1150 [lmv] [11056.400814] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11056.403455] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11056.405890] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11056.408173] ll_intent_lock+0xc5/0xaa0 [lustre] [11056.410535] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11056.412463] ll_lookup_it+0xe0e/0x2270 [lustre] [11056.421182] ? do_raw_spin_unlock+0x75/0x190 [11056.430887] ? _raw_spin_unlock+0x12/0x30 [11056.432465] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11056.439383] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11056.441237] ll_atomic_open+0x36a/0xd70 [lustre] [11056.445134] lookup_open+0x356/0xb40 [11056.446191] do_last+0x313/0xfc0 [11056.447170] ? nd_jump_root+0xe5/0x160 [11056.461261] ? path_init+0x437/0x520 [11056.468132] path_openat+0xf7/0x500 [11056.469406] ? do_raw_spin_unlock+0x75/0x190 [11056.475096] do_filp_open+0x99/0x140 [11056.476284] ? getname_flags+0x6e/0x330 [11056.477554] ? __check_object_size+0xff/0x256 [11056.482185] ? do_raw_spin_unlock+0x75/0x190 [11056.486153] ? _raw_spin_unlock+0x12/0x30 [11056.489101] do_sys_openat2+0x2b4/0x410 [11056.490384] do_sys_open+0x73/0xa0 [11056.491887] __x64_sys_openat+0x24/0x30 [11056.495121] do_syscall_64+0xc1/0x440 [11056.496287] entry_SYSCALL_64_after_hwframe+0x49/0xae [11056.500715] RIP: 0033:0x7f53425a5332 [11056.502530] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11056.512061] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11056.520780] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11056.526434] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11056.533196] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11056.535882] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11056.539296] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11060.058044] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11060.061077] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11060.093637] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11060.100467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11060.107707] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11060.110445] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11060.136765] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11060.149306] RAX: 0000000000005340 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11060.154681] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11060.170329] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11060.172751] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11060.198728] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11060.205446] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11060.215792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11060.225413] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11060.238320] Call Trace: [11060.244706] [11060.248661] ? show_regs.cold.9+0x22/0x2f [11060.254378] ? watchdog_timer_fn.cold.6+0x62/0x10a [11060.263381] ? __hrtimer_run_queues+0x1c2/0x4d0 [11060.267531] ? watchdog+0x40/0x40 [11060.279505] ? hrtimer_interrupt+0x128/0x2c0 [11060.283745] ? smp_apic_timer_interrupt+0xa6/0x270 [11060.292810] ? apic_timer_interrupt+0xf/0x20 [11060.294248] [11060.304803] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11060.309284] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11060.316611] do_raw_spin_lock+0x177/0x190 [11060.320404] _raw_spin_lock+0x21/0x30 [11060.323401] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11060.325475] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11060.331294] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11060.333559] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11060.341704] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11060.345078] ? woken_wake_function+0x30/0x30 [11060.349493] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11060.355224] kthread+0x1d1/0x200 [11060.357635] ? set_kthread_struct+0x70/0x70 [11060.359574] ret_from_fork+0x1f/0x30 [11076.054215] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [11076.061359] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11076.102424] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11076.107475] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11076.111029] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11076.115402] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11076.119361] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11076.129329] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11076.137414] RAX: 0000000000006be5 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11076.144900] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11076.152189] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11076.156327] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11076.164202] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11076.169328] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11076.174423] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11076.180230] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11076.184258] Call Trace: [11076.185104] [11076.186635] ? show_regs.cold.9+0x22/0x2f [11076.192650] ? watchdog_timer_fn.cold.6+0x62/0x10a [11076.195392] ? __hrtimer_run_queues+0x1c2/0x4d0 [11076.202877] ? watchdog+0x40/0x40 [11076.204379] ? hrtimer_interrupt+0x128/0x2c0 [11076.208514] ? smp_apic_timer_interrupt+0xa6/0x270 [11076.216281] ? apic_timer_interrupt+0xf/0x20 [11076.218396] [11076.220242] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11076.223425] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11076.230444] ? newidle_balance.isra.93+0x207/0x580 [11076.234624] do_raw_spin_lock+0x177/0x190 [11076.239806] _raw_spin_lock+0x21/0x30 [11076.242178] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11076.247116] process_one_work+0x2c8/0x700 [11076.248514] worker_thread+0x296/0x6e0 [11076.254892] ? rescuer_thread+0x570/0x570 [11076.258546] kthread+0x1d1/0x200 [11076.260555] ? set_kthread_struct+0x70/0x70 [11076.263966] ret_from_fork+0x1f/0x30 [11080.162986] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8443s! [11080.167137] Showing busy workqueues and worker pools: [11080.169705] workqueue events: flags=0x0 [11080.171941] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11080.174411] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11080.179183] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11080.186284] pending: cpu_latency_work [ptlrpc] [11080.192262] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11080.196042] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11080.204461] workqueue events_long: flags=0x0 [11080.208640] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11080.215413] pending: lu_objects_flush [obdclass] [11080.220824] workqueue events_unbound: flags=0x2 [11080.227645] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11080.235909] pending: flush_memcg_stats_dwork, idle_cull_fn [11080.241761] workqueue events_power_efficient: flags=0x80 [11080.246922] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11080.253457] pending: check_lifetime, do_cache_clean [11080.258688] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11080.265672] pending: neigh_periodic_work, neigh_periodic_work [11080.270773] workqueue mm_percpu_wq: flags=0x8 [11080.274206] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11080.279727] pending: vmstat_update, lru_add_drain_per_cpu [11080.288008] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11080.292933] pending: vmstat_update, lru_add_drain_per_cpu [11080.303224] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11080.313772] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11080.323283] workqueue xprtiod: flags=0xa [11080.326804] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11080.330946] pending: xprt_autoclose, xs_error_handle [11080.338838] workqueue ipv6_addrconf: flags=0x40008 [11080.349541] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11080.357912] pending: addrconf_verify_work [11080.359542] workqueue knbd0-recv: flags=0x1a [11080.361011] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11080.364254] in-flight: 651:recv_work [11080.370959] workqueue ptlrpc_pinger: flags=0x2 [11080.374905] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11080.385312] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11080.391977] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8443s workers=4 idle: 15979 31824 8 [11080.398681] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11070s workers=2 idle: 654 [11084.057963] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11084.059278] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11084.064614] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11084.064694] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11084.064702] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11084.064705] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11084.064721] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11084.064726] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11084.064731] RAX: 0000000000004e82 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11084.064734] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11084.064736] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11084.064738] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11084.064741] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11084.064745] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11084.064748] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11084.064751] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11084.064761] Call Trace: [11084.064767] [11084.064773] ? show_regs.cold.9+0x22/0x2f [11084.064784] ? watchdog_timer_fn.cold.6+0x62/0x10a [11084.064791] ? __hrtimer_run_queues+0x1c2/0x4d0 [11084.064794] ? watchdog+0x40/0x40 [11084.064799] ? hrtimer_interrupt+0x128/0x2c0 [11084.064803] ? smp_apic_timer_interrupt+0xa6/0x270 [11084.064811] ? apic_timer_interrupt+0xf/0x20 [11084.064816] [11084.064827] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11084.064835] do_raw_spin_lock+0x177/0x190 [11084.064852] _raw_spin_lock+0x21/0x30 [11084.064858] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11084.065337] ? do_raw_spin_unlock+0x75/0x190 [11084.065343] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11084.071203] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11084.071284] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11084.071291] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11084.071294] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11084.071308] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11084.071313] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11084.071317] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11084.071319] RDX: 0000000000007918 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11084.071322] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11084.071324] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11084.071326] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11084.071329] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11084.071332] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11084.071335] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11084.071364] Call Trace: [11084.071370] [11084.071375] ? show_regs.cold.9+0x22/0x2f [11084.071385] ? watchdog_timer_fn.cold.6+0x62/0x10a [11084.071391] ? __hrtimer_run_queues+0x1c2/0x4d0 [11084.071395] ? watchdog+0x40/0x40 [11084.071400] ? hrtimer_interrupt+0x128/0x2c0 [11084.071404] ? smp_apic_timer_interrupt+0xa6/0x270 [11084.071411] ? apic_timer_interrupt+0xf/0x20 [11084.071415] [11084.071418] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11084.071422] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11084.071441] do_raw_spin_lock+0x177/0x190 [11084.071446] _raw_spin_lock+0x21/0x30 [11084.071453] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11084.116415] ? do_wait_intr+0xf0/0xf0 [11084.120247] obd_mod_free+0x46/0x140 [ptlrpc] [11084.120592] mdc_commit_open+0x91/0xb0 [mdc] [11084.120629] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11084.120818] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11084.121015] after_reply+0xe2c/0x1780 [ptlrpc] [11084.121411] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11084.121639] ? __next_timer_interrupt+0x160/0x160 [11084.121650] ptlrpcd+0xa82/0xe40 [ptlrpc] [11084.121931] ? do_wait_intr+0xf0/0xf0 [11084.121956] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11084.125741] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11084.126043] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11084.126278] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11084.126576] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11084.126701] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11084.126746] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11084.126781] mdc_intent_lock+0x353/0x7e0 [mdc] [11084.126811] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11084.126900] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11084.127435] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11084.127468] lmv_intent_open+0x338/0x1150 [lmv] [11084.127501] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11084.127592] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11084.127692] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11084.127715] ll_intent_lock+0xc5/0xaa0 [lustre] [11084.127799] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11084.127881] ll_lookup_it+0xe0e/0x2270 [lustre] [11084.127969] ? do_raw_spin_unlock+0x75/0x190 [11084.127976] ? _raw_spin_unlock+0x12/0x30 [11084.127984] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11084.128011] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11084.128030] ll_atomic_open+0x36a/0xd70 [lustre] [11084.128392] lookup_open+0x356/0xb40 [11084.128407] do_last+0x313/0xfc0 [11084.128413] ? nd_jump_root+0xe5/0x160 [11084.128418] ? path_init+0x437/0x520 [11084.128483] path_openat+0xf7/0x500 [11084.128489] ? do_raw_spin_unlock+0x75/0x190 [11084.128494] do_filp_open+0x99/0x140 [11084.128500] ? getname_flags+0x6e/0x330 [11084.128505] ? __check_object_size+0xff/0x256 [11084.128512] ? do_raw_spin_unlock+0x75/0x190 [11084.128516] ? _raw_spin_unlock+0x12/0x30 [11084.128521] do_sys_openat2+0x2b4/0x410 [11084.128526] do_sys_open+0x73/0xa0 [11084.128530] __x64_sys_openat+0x24/0x30 [11084.128534] do_syscall_64+0xc1/0x440 [11084.128542] entry_SYSCALL_64_after_hwframe+0x49/0xae [11084.128548] RIP: 0033:0x7f53425a5332 [11084.128556] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11084.128561] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11084.128566] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11084.128568] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11084.128570] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11084.128572] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11084.128574] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11084.633106] kthread+0x1d1/0x200 [11084.634400] ? set_kthread_struct+0x70/0x70 [11084.637782] ret_from_fork+0x1f/0x30 [11088.058031] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11088.070652] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11088.122531] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11088.134680] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11088.143218] RIP: 0010:native_safe_halt+0x2e/0x50 [11088.147132] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11088.156677] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11088.169668] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11088.177793] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11088.182570] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11088.195695] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11088.201323] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11088.208091] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11088.218238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11088.224756] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11088.231319] Call Trace: [11088.233598] [11088.234521] ? show_regs.cold.9+0x22/0x2f [11088.236248] ? watchdog_timer_fn.cold.6+0x62/0x10a [11088.241658] ? __hrtimer_run_queues+0x1c2/0x4d0 [11088.246579] ? watchdog+0x40/0x40 [11088.249219] ? hrtimer_interrupt+0x128/0x2c0 [11088.252390] ? smp_apic_timer_interrupt+0xa6/0x270 [11088.255224] ? apic_timer_interrupt+0xf/0x20 [11088.258854] [11088.261661] ? native_safe_halt+0x2e/0x50 [11088.265606] kvm_wait+0xcd/0xe0 [11088.266607] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11088.272721] do_raw_spin_lock+0x177/0x190 [11088.275584] _raw_spin_lock+0x21/0x30 [11088.279606] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11088.286194] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11088.290882] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11088.297072] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11088.304129] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11088.308657] ? woken_wake_function+0x30/0x30 [11088.312771] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11088.318559] kthread+0x1d1/0x200 [11088.320754] ? set_kthread_struct+0x70/0x70 [11088.323787] ret_from_fork+0x1f/0x30 [11108.052321] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [11108.057145] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11108.100155] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11108.108560] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11108.112896] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11108.116739] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11108.121941] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11108.132105] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11108.136980] RAX: 00000000000033c1 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11108.139098] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11108.141264] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11108.143118] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11108.145287] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11108.147457] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11108.149678] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11108.151345] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11108.153321] Call Trace: [11108.153923] [11108.154666] ? show_regs.cold.9+0x22/0x2f [11108.155948] ? watchdog_timer_fn.cold.6+0x62/0x10a [11108.157593] ? __hrtimer_run_queues+0x1c2/0x4d0 [11108.158955] ? watchdog+0x40/0x40 [11108.160018] ? hrtimer_interrupt+0x128/0x2c0 [11108.161215] ? smp_apic_timer_interrupt+0xa6/0x270 [11108.162610] ? apic_timer_interrupt+0xf/0x20 [11108.163924] [11108.164679] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11108.166771] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11108.168642] ? newidle_balance.isra.93+0x207/0x580 [11108.170338] do_raw_spin_lock+0x177/0x190 [11108.171374] _raw_spin_lock+0x21/0x30 [11108.172616] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11108.174783] process_one_work+0x2c8/0x700 [11108.175880] worker_thread+0x296/0x6e0 [11108.176974] ? rescuer_thread+0x570/0x570 [11108.178102] kthread+0x1d1/0x200 [11108.179080] ? set_kthread_struct+0x70/0x70 [11108.180375] ret_from_fork+0x1f/0x30 [11110.880070] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8473s! [11110.887969] Showing busy workqueues and worker pools: [11110.891507] workqueue events: flags=0x0 [11110.897881] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11110.906860] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11110.917973] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11110.921902] pending: cpu_latency_work [ptlrpc] [11110.924883] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11110.928284] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11110.936255] workqueue events_long: flags=0x0 [11110.938931] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11110.943566] pending: lu_objects_flush [obdclass] [11110.950563] workqueue events_unbound: flags=0x2 [11110.955072] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11110.959515] pending: flush_memcg_stats_dwork, idle_cull_fn [11110.963562] workqueue events_power_efficient: flags=0x80 [11110.967585] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11110.975381] pending: check_lifetime, do_cache_clean [11110.981968] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11110.984751] pending: neigh_periodic_work, neigh_periodic_work [11110.993051] workqueue mm_percpu_wq: flags=0x8 [11110.996425] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11111.002622] pending: vmstat_update, lru_add_drain_per_cpu [11111.006023] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11111.009625] pending: vmstat_update, lru_add_drain_per_cpu [11111.018029] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11111.023097] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11111.026886] workqueue xprtiod: flags=0xa [11111.032108] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11111.047236] pending: xprt_autoclose, xs_error_handle [11111.050253] workqueue ipv6_addrconf: flags=0x40008 [11111.058145] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11111.060541] pending: addrconf_verify_work [11111.074070] workqueue knbd0-recv: flags=0x1a [11111.075644] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11111.081553] in-flight: 651:recv_work [11111.087945] workqueue ptlrpc_pinger: flags=0x2 [11111.092628] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11111.096503] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11111.100149] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8474s workers=4 idle: 15979 31824 8 [11111.107895] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11101s workers=2 idle: 654 [11112.055011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11112.058747] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11112.061606] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11112.061686] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11112.061693] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11112.061696] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11112.061710] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11112.061715] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11112.061719] RAX: 00000000000045e9 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11112.061721] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11112.061723] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11112.061725] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11112.061727] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11112.061730] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11112.061733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11112.061735] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11112.062219] Call Trace: [11112.062227] [11112.062234] ? show_regs.cold.9+0x22/0x2f [11112.062245] ? watchdog_timer_fn.cold.6+0x62/0x10a [11112.062251] ? __hrtimer_run_queues+0x1c2/0x4d0 [11112.062255] ? watchdog+0x40/0x40 [11112.062260] ? hrtimer_interrupt+0x128/0x2c0 [11112.062265] ? smp_apic_timer_interrupt+0xa6/0x270 [11112.062273] ? apic_timer_interrupt+0xf/0x20 [11112.062278] [11112.062281] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11112.062288] do_raw_spin_lock+0x177/0x190 [11112.062293] _raw_spin_lock+0x21/0x30 [11112.062299] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11112.062681] ? do_raw_spin_unlock+0x75/0x190 [11112.062687] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11112.070514] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11112.070685] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11112.070692] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11112.070695] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11112.070710] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11112.070738] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11112.070743] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11112.070746] RDX: 0000000000002948 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11112.070748] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11112.070750] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11112.070753] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11112.070756] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11112.070758] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11112.070761] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11112.070769] Call Trace: [11112.070774] [11112.070779] ? show_regs.cold.9+0x22/0x2f [11112.070789] ? watchdog_timer_fn.cold.6+0x62/0x10a [11112.070795] ? __hrtimer_run_queues+0x1c2/0x4d0 [11112.070800] ? watchdog+0x40/0x40 [11112.070819] ? hrtimer_interrupt+0x128/0x2c0 [11112.070824] ? smp_apic_timer_interrupt+0xa6/0x270 [11112.070837] ? apic_timer_interrupt+0xf/0x20 [11112.070841] [11112.070844] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11112.070849] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11112.070853] do_raw_spin_lock+0x177/0x190 [11112.070858] _raw_spin_lock+0x21/0x30 [11112.070864] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11112.071292] obd_mod_free+0x46/0x140 [ptlrpc] [11112.071417] mdc_commit_open+0x91/0xb0 [mdc] [11112.071445] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11112.071732] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11112.071918] after_reply+0xe2c/0x1780 [ptlrpc] [11112.072283] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11112.090972] ? do_wait_intr+0xf0/0xf0 [11112.090998] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11112.091332] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11112.091541] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11112.096683] ? __next_timer_interrupt+0x160/0x160 [11112.100198] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11112.100332] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11112.100378] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11112.100409] mdc_intent_lock+0x353/0x7e0 [mdc] [11112.100437] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11112.103742] ptlrpcd+0xa82/0xe40 [ptlrpc] [11112.104405] ? do_wait_intr+0xf0/0xf0 [11112.104624] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11112.105294] kthread+0x1d1/0x200 [11112.105302] ? set_kthread_struct+0x70/0x70 [11112.105307] ret_from_fork+0x1f/0x30 [11112.410807] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11112.414525] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11112.421139] lmv_intent_open+0x338/0x1150 [lmv] [11112.422878] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11112.426921] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11112.429426] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11112.434635] ll_intent_lock+0xc5/0xaa0 [lustre] [11112.439501] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11112.441484] ll_lookup_it+0xe0e/0x2270 [lustre] [11112.452576] ? do_raw_spin_unlock+0x75/0x190 [11112.454078] ? _raw_spin_unlock+0x12/0x30 [11112.458407] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11112.461434] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11112.464497] ll_atomic_open+0x36a/0xd70 [lustre] [11112.471490] lookup_open+0x356/0xb40 [11112.473413] do_last+0x313/0xfc0 [11112.475303] ? nd_jump_root+0xe5/0x160 [11112.478773] ? path_init+0x437/0x520 [11112.480805] path_openat+0xf7/0x500 [11112.482628] ? do_raw_spin_unlock+0x75/0x190 [11112.485454] do_filp_open+0x99/0x140 [11112.487474] ? getname_flags+0x6e/0x330 [11112.489488] ? __check_object_size+0xff/0x256 [11112.491484] ? do_raw_spin_unlock+0x75/0x190 [11112.493831] ? _raw_spin_unlock+0x12/0x30 [11112.497488] do_sys_openat2+0x2b4/0x410 [11112.500443] do_sys_open+0x73/0xa0 [11112.505625] __x64_sys_openat+0x24/0x30 [11112.508027] do_syscall_64+0xc1/0x440 [11112.509870] entry_SYSCALL_64_after_hwframe+0x49/0xae [11112.518829] RIP: 0033:0x7f53425a5332 [11112.522270] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11112.536489] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11112.538682] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11112.544048] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11112.552121] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11112.561622] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11112.569356] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11116.058007] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11116.062692] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11116.083439] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11116.096796] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11116.106372] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11116.110585] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11116.118647] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11116.125062] RAX: 000000000000764c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11116.129085] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11116.133394] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11116.137747] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11116.143347] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11116.146501] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11116.152415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11116.158250] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11116.166111] Call Trace: [11116.167096] [11116.167833] ? show_regs.cold.9+0x22/0x2f [11116.170715] ? watchdog_timer_fn.cold.6+0x62/0x10a [11116.176502] ? __hrtimer_run_queues+0x1c2/0x4d0 [11116.179301] ? watchdog+0x40/0x40 [11116.185283] ? hrtimer_interrupt+0x128/0x2c0 [11116.190943] ? smp_apic_timer_interrupt+0xa6/0x270 [11116.193539] ? apic_timer_interrupt+0xf/0x20 [11116.198999] [11116.200145] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11116.205821] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11116.210636] do_raw_spin_lock+0x177/0x190 [11116.216426] _raw_spin_lock+0x21/0x30 [11116.218146] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11116.227529] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11116.235057] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11116.241468] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11116.243959] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11116.249929] ? woken_wake_function+0x30/0x30 [11116.252645] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11116.261752] kthread+0x1d1/0x200 [11116.263283] ? set_kthread_struct+0x70/0x70 [11116.268155] ret_from_fork+0x1f/0x30 [11136.052037] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [11136.055594] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11136.072710] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11136.075722] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11136.079230] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11136.085412] RIP: 0010:native_safe_halt+0x2e/0x50 [11136.087959] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11136.102275] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11136.105597] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11136.108804] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [11136.111827] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11136.114707] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11136.117496] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11136.120588] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11136.123742] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11136.126215] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11136.129004] Call Trace: [11136.130107] [11136.130894] ? show_regs.cold.9+0x22/0x2f [11136.132331] ? watchdog_timer_fn.cold.6+0x62/0x10a [11136.134334] ? __hrtimer_run_queues+0x1c2/0x4d0 [11136.136147] ? watchdog+0x40/0x40 [11136.137576] ? hrtimer_interrupt+0x128/0x2c0 [11136.139213] ? smp_apic_timer_interrupt+0xa6/0x270 [11136.141054] ? apic_timer_interrupt+0xf/0x20 [11136.142173] [11136.142742] ? native_safe_halt+0x2e/0x50 [11136.149304] kvm_wait+0xcd/0xe0 [11136.150489] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11136.155471] ? newidle_balance.isra.93+0x207/0x580 [11136.161021] do_raw_spin_lock+0x177/0x190 [11136.162873] _raw_spin_lock+0x21/0x30 [11136.168363] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11136.170414] process_one_work+0x2c8/0x700 [11136.175950] worker_thread+0x296/0x6e0 [11136.177241] ? rescuer_thread+0x570/0x570 [11136.191233] kthread+0x1d1/0x200 [11136.192350] ? set_kthread_struct+0x70/0x70 [11136.193724] ret_from_fork+0x1f/0x30 [11140.057115] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11140.057922] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11140.061418] Modules linked in: [11140.069751] Modules linked in: lustre(O) osp(O) [11140.072788] lustre(O) osp(O) ofd(O) [11140.078718] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11140.078792] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11140.078801] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11140.078804] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11140.078819] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11140.078823] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11140.078828] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11140.078830] RDX: 00000000000006bb RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11140.078832] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11140.078834] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11140.078849] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11140.078852] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11140.078855] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11140.078857] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11140.078866] Call Trace: [11140.078872] [11140.078878] ? show_regs.cold.9+0x22/0x2f [11140.078889] ? watchdog_timer_fn.cold.6+0x62/0x10a [11140.078896] ? __hrtimer_run_queues+0x1c2/0x4d0 [11140.078900] ? watchdog+0x40/0x40 [11140.078905] ? hrtimer_interrupt+0x128/0x2c0 [11140.078909] ? smp_apic_timer_interrupt+0xa6/0x270 [11140.078918] ? apic_timer_interrupt+0xf/0x20 [11140.078923] [11140.078926] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11140.078932] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11140.078936] do_raw_spin_lock+0x177/0x190 [11140.078941] _raw_spin_lock+0x21/0x30 [11140.078948] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11140.079764] obd_mod_free+0x46/0x140 [ptlrpc] [11140.082253] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11140.082326] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11140.082334] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11140.082337] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11140.082350] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11140.082355] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11140.082359] RAX: 0000000000007ea9 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11140.082362] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11140.082364] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11140.082366] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11140.082369] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11140.082372] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11140.082375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11140.082378] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11140.082387] Call Trace: [11140.082392] [11140.082397] ? show_regs.cold.9+0x22/0x2f [11140.082406] ? watchdog_timer_fn.cold.6+0x62/0x10a [11140.082413] ? __hrtimer_run_queues+0x1c2/0x4d0 [11140.082418] ? watchdog+0x40/0x40 [11140.082423] ? hrtimer_interrupt+0x128/0x2c0 [11140.082427] ? smp_apic_timer_interrupt+0xa6/0x270 [11140.082435] ? apic_timer_interrupt+0xf/0x20 [11140.082440] [11140.082514] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11140.082524] do_raw_spin_lock+0x177/0x190 [11140.082529] _raw_spin_lock+0x21/0x30 [11140.082534] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11140.082810] ? do_raw_spin_unlock+0x75/0x190 [11140.082816] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11140.083029] ? do_wait_intr+0xf0/0xf0 [11140.083039] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11140.136196] mdc_commit_open+0x91/0xb0 [mdc] [11140.136571] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11140.136842] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11140.138133] after_reply+0xe2c/0x1780 [ptlrpc] [11140.139188] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11140.148036] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11140.148673] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11140.148933] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11140.149052] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11140.149277] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11140.149307] mdc_intent_lock+0x353/0x7e0 [mdc] [11140.149336] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11140.149423] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11140.149663] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11140.149694] lmv_intent_open+0x338/0x1150 [lmv] [11140.149733] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11140.149896] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11140.149987] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11140.150012] ll_intent_lock+0xc5/0xaa0 [lustre] [11140.150106] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11140.150267] ll_lookup_it+0xe0e/0x2270 [lustre] [11140.150404] ? do_raw_spin_unlock+0x75/0x190 [11140.150413] ? _raw_spin_unlock+0x12/0x30 [11140.150421] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11140.150458] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11140.150480] ll_atomic_open+0x36a/0xd70 [lustre] [11140.150579] lookup_open+0x356/0xb40 [11140.150588] do_last+0x313/0xfc0 [11140.150593] ? nd_jump_root+0xe5/0x160 [11140.150597] ? path_init+0x437/0x520 [11140.150602] path_openat+0xf7/0x500 [11140.150607] ? do_raw_spin_unlock+0x75/0x190 [11140.150611] do_filp_open+0x99/0x140 [11140.150618] ? getname_flags+0x6e/0x330 [11140.150622] ? __check_object_size+0xff/0x256 [11140.150629] ? do_raw_spin_unlock+0x75/0x190 [11140.150634] ? _raw_spin_unlock+0x12/0x30 [11140.150638] do_sys_openat2+0x2b4/0x410 [11140.150642] do_sys_open+0x73/0xa0 [11140.150646] __x64_sys_openat+0x24/0x30 [11140.150649] do_syscall_64+0xc1/0x440 [11140.150656] entry_SYSCALL_64_after_hwframe+0x49/0xae [11140.150663] RIP: 0033:0x7f53425a5332 [11140.150670] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11140.150675] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11140.150681] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11140.150753] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11140.150755] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11140.150757] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11140.150759] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11140.625815] ? __next_timer_interrupt+0x160/0x160 [11140.627040] ptlrpcd+0xa82/0xe40 [ptlrpc] [11140.633170] ? do_wait_intr+0xf0/0xf0 [11140.638353] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11140.642159] kthread+0x1d1/0x200 [11140.644178] ? set_kthread_struct+0x70/0x70 [11140.647605] ret_from_fork+0x1f/0x30 [11141.600102] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8504s! [11141.603614] Showing busy workqueues and worker pools: [11141.605303] workqueue events: flags=0x0 [11141.610184] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11141.615307] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11141.622254] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11141.628008] pending: cpu_latency_work [ptlrpc] [11141.635207] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11141.641741] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11141.649321] workqueue events_long: flags=0x0 [11141.651737] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11141.654859] pending: lu_objects_flush [obdclass] [11141.657490] workqueue events_unbound: flags=0x2 [11141.660184] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11141.663115] pending: flush_memcg_stats_dwork, idle_cull_fn [11141.666272] workqueue events_power_efficient: flags=0x80 [11141.669298] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11141.673254] pending: check_lifetime, do_cache_clean [11141.675592] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11141.679177] pending: neigh_periodic_work, neigh_periodic_work [11141.683595] workqueue mm_percpu_wq: flags=0x8 [11141.685480] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11141.687547] pending: vmstat_update, lru_add_drain_per_cpu [11141.689389] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11141.696302] pending: vmstat_update, lru_add_drain_per_cpu [11141.698790] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11141.702196] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11141.705326] workqueue xprtiod: flags=0xa [11141.707389] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11141.715626] pending: xprt_autoclose, xs_error_handle [11141.719407] workqueue ipv6_addrconf: flags=0x40008 [11141.721401] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11141.732060] pending: addrconf_verify_work [11141.738173] workqueue knbd0-recv: flags=0x1a [11141.742135] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11141.751132] in-flight: 651:recv_work [11141.753613] workqueue ptlrpc_pinger: flags=0x2 [11141.755634] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11141.758557] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11141.761698] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8504s workers=4 idle: 15979 31824 8 [11141.765825] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11132s workers=2 idle: 654 [11144.058024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11144.065261] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11144.101347] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11144.108294] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11144.111946] RIP: 0010:native_safe_halt+0x2e/0x50 [11144.115742] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11144.129508] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11144.133709] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11144.136768] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11144.141291] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11144.146026] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11144.153083] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11144.163623] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11144.172920] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11144.178319] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11144.181807] Call Trace: [11144.182647] [11144.183328] ? show_regs.cold.9+0x22/0x2f [11144.185694] ? watchdog_timer_fn.cold.6+0x62/0x10a [11144.187736] ? __hrtimer_run_queues+0x1c2/0x4d0 [11144.192103] ? watchdog+0x40/0x40 [11144.193300] ? hrtimer_interrupt+0x128/0x2c0 [11144.198986] ? smp_apic_timer_interrupt+0xa6/0x270 [11144.202018] ? apic_timer_interrupt+0xf/0x20 [11144.204648] [11144.205739] ? native_safe_halt+0x2e/0x50 [11144.208161] kvm_wait+0xcd/0xe0 [11144.211914] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11144.216916] do_raw_spin_lock+0x177/0x190 [11144.220563] _raw_spin_lock+0x21/0x30 [11144.224346] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11144.229230] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11144.234217] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11144.237680] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11144.244446] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11144.248834] ? woken_wake_function+0x30/0x30 [11144.253270] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11144.257589] kthread+0x1d1/0x200 [11144.259500] ? set_kthread_struct+0x70/0x70 [11144.262360] ret_from_fork+0x1f/0x30 [11154.640030] rcu: INFO: rcu_sched self-detected stall on CPU [11154.642330] rcu: 2-....: (4981282 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [11154.645841] (t=9060214 jiffies g=274185 q=66157) [11154.646941] rcu: rcu_sched kthread starved for 8518580 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [11154.656352] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11154.659820] rcu: RCU grace-period kthread stack dump: [11154.662219] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11154.667092] Call Trace: [11154.668132] __schedule+0x351/0xcb0 [11154.669820] schedule+0xc0/0x180 [11154.671357] schedule_timeout+0xb4/0x190 [11154.674018] ? do_raw_spin_unlock+0x75/0x190 [11154.675298] ? __next_timer_interrupt+0x160/0x160 [11154.678361] ? prepare_to_swait_event+0xf3/0x140 [11154.681379] rcu_gp_kthread+0x8dd/0x15b0 [11154.682653] ? force_qs_rnp+0x2c0/0x2c0 [11154.684586] kthread+0x1d1/0x200 [11154.685721] ? set_kthread_struct+0x70/0x70 [11154.687457] ret_from_fork+0x1f/0x30 [11154.688885] rcu: Stack dump where RCU GP kthread last ran: [11154.691126] Sending NMI from CPU 2 to CPUs 0: [11154.695858] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [11154.698381] Sending NMI from CPU 2 to CPUs 1: [11154.705036] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [11154.706798] NMI backtrace for cpu 2 [11154.713585] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11154.717484] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11154.720525] Call Trace: [11154.721452] [11154.722289] ? dump_stack+0xbb/0x10e [11154.723460] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11154.725286] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11154.727087] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11154.730081] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11154.731708] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11154.734666] ? rcu_dump_cpu_stacks+0x11b/0x171 [11154.735849] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11154.738153] ? update_process_times+0x6d/0xc0 [11154.740729] ? tick_sched_handle.isra.18+0x27/0x90 [11154.744285] ? tick_sched_timer+0x53/0xb0 [11154.747401] ? __hrtimer_run_queues+0x1c2/0x4d0 [11154.749305] ? tick_sched_handle.isra.18+0x90/0x90 [11154.750385] ? hrtimer_interrupt+0x128/0x2c0 [11154.752895] ? smp_apic_timer_interrupt+0xa6/0x270 [11154.754951] ? apic_timer_interrupt+0xf/0x20 [11154.756616] [11154.757464] ? native_safe_halt+0x2e/0x50 [11154.758799] ? kvm_wait+0xcd/0xe0 [11154.760167] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11154.761729] ? do_raw_spin_lock+0x177/0x190 [11154.763369] ? _raw_spin_lock+0x21/0x30 [11154.764523] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11154.772323] ? obd_mod_free+0x46/0x140 [ptlrpc] [11154.774525] ? mdc_commit_open+0x91/0xb0 [mdc] [11154.776484] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11154.778843] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11154.781019] ? after_reply+0xe2c/0x1780 [ptlrpc] [11154.783389] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11154.785628] ? __next_timer_interrupt+0x160/0x160 [11154.787558] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11154.789554] ? do_wait_intr+0xf0/0xf0 [11154.790890] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11154.793503] ? kthread+0x1d1/0x200 [11154.794840] ? set_kthread_struct+0x70/0x70 [11154.796642] ? ret_from_fork+0x1f/0x30 [11171.807185] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8534s! [11171.819626] Showing busy workqueues and worker pools: [11171.828189] workqueue events: flags=0x0 [11171.829507] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11171.843546] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11171.855349] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11171.857724] pending: cpu_latency_work [ptlrpc] [11171.867685] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11171.874651] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11171.882977] workqueue events_long: flags=0x0 [11171.888831] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11171.899759] pending: lu_objects_flush [obdclass] [11171.906331] workqueue events_unbound: flags=0x2 [11171.909533] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11171.912300] pending: flush_memcg_stats_dwork, idle_cull_fn [11171.923102] workqueue events_power_efficient: flags=0x80 [11171.926578] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11171.930712] pending: check_lifetime, do_cache_clean [11171.937023] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11171.948611] pending: neigh_periodic_work, neigh_periodic_work [11171.957691] workqueue mm_percpu_wq: flags=0x8 [11171.960581] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11171.963172] pending: vmstat_update, lru_add_drain_per_cpu [11171.968264] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11171.970637] pending: vmstat_update, lru_add_drain_per_cpu [11171.972368] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11171.974526] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11171.976752] workqueue xprtiod: flags=0xa [11171.978055] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11171.980268] pending: xprt_autoclose, xs_error_handle [11171.982104] workqueue ipv6_addrconf: flags=0x40008 [11171.983696] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11171.985602] pending: addrconf_verify_work [11171.986617] workqueue knbd0-recv: flags=0x1a [11171.987636] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11171.989123] in-flight: 651:recv_work [11171.989965] workqueue ptlrpc_pinger: flags=0x2 [11171.991683] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11171.994202] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11171.996499] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8534s workers=4 idle: 15979 31824 8 [11171.999403] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11162s workers=2 idle: 654 [11172.058073] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11172.060296] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11172.071619] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11172.075226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11172.077502] RIP: 0010:native_safe_halt+0x2e/0x50 [11172.079110] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11172.085454] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11172.087428] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11172.089232] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11172.091390] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11172.093454] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11172.095829] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11172.098387] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11172.101245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11172.103355] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11172.105959] Call Trace: [11172.106872] [11172.107665] ? show_regs.cold.9+0x22/0x2f [11172.109277] ? watchdog_timer_fn.cold.6+0x62/0x10a [11172.110839] ? __hrtimer_run_queues+0x1c2/0x4d0 [11172.112571] ? watchdog+0x40/0x40 [11172.113939] ? hrtimer_interrupt+0x128/0x2c0 [11172.115399] ? smp_apic_timer_interrupt+0xa6/0x270 [11172.117138] ? apic_timer_interrupt+0xf/0x20 [11172.118633] [11172.119423] ? native_safe_halt+0x2e/0x50 [11172.120759] kvm_wait+0xcd/0xe0 [11172.122060] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11172.123812] do_raw_spin_lock+0x177/0x190 [11172.125161] _raw_spin_lock+0x21/0x30 [11172.126303] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11172.128483] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11172.130420] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11172.132512] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11172.134544] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11172.136488] ? woken_wake_function+0x30/0x30 [11172.137996] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11172.140350] kthread+0x1d1/0x200 [11172.141457] ? set_kthread_struct+0x70/0x70 [11172.143183] ret_from_fork+0x1f/0x30 [11180.056259] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [11180.059955] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [11180.069730] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11180.070326] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11180.070335] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11180.070338] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11180.070515] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11180.070519] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11180.070523] RAX: 0000000000001d9c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11180.070525] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11180.070528] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11180.070530] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11180.070533] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11180.070537] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11180.070539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11180.070542] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11180.070551] Call Trace: [11180.070557] [11180.070563] ? show_regs.cold.9+0x22/0x2f [11180.070574] ? watchdog_timer_fn.cold.6+0x62/0x10a [11180.070585] ? __hrtimer_run_queues+0x1c2/0x4d0 [11180.070589] ? watchdog+0x40/0x40 [11180.070593] ? hrtimer_interrupt+0x128/0x2c0 [11180.070601] ? smp_apic_timer_interrupt+0xa6/0x270 [11180.070609] ? apic_timer_interrupt+0xf/0x20 [11180.070613] [11180.070616] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11180.070622] do_raw_spin_lock+0x177/0x190 [11180.070627] _raw_spin_lock+0x21/0x30 [11180.070636] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11180.072302] ? do_raw_spin_unlock+0x75/0x190 [11180.072312] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11180.078426] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr [11180.125432] ? do_wait_intr+0xf0/0xf0 [11180.125458] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11180.125872] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11180.126090] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11180.126307] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11180.135168] intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11180.135206] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11180.135214] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11180.135217] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11180.135231] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11180.135235] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11180.135239] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11180.135242] RDX: 0000000000000c4e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11180.135244] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11180.135247] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11180.135250] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11180.135253] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11180.135256] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11180.135258] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11180.135268] Call Trace: [11180.135273] [11180.135279] ? show_regs.cold.9+0x22/0x2f [11180.135288] ? watchdog_timer_fn.cold.6+0x62/0x10a [11180.135294] ? __hrtimer_run_queues+0x1c2/0x4d0 [11180.135298] ? watchdog+0x40/0x40 [11180.135302] ? hrtimer_interrupt+0x128/0x2c0 [11180.135305] ? smp_apic_timer_interrupt+0xa6/0x270 [11180.135313] ? apic_timer_interrupt+0xf/0x20 [11180.135318] [11180.135321] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11180.135326] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11180.135331] do_raw_spin_lock+0x177/0x190 [11180.135336] _raw_spin_lock+0x21/0x30 [11180.135344] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11180.135611] obd_mod_free+0x46/0x140 [ptlrpc] [11180.136157] mdc_commit_open+0x91/0xb0 [mdc] [11180.136264] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11180.136551] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11180.136906] after_reply+0xe2c/0x1780 [ptlrpc] [11180.137104] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11180.139457] ? __next_timer_interrupt+0x160/0x160 [11180.139468] ptlrpcd+0xa82/0xe40 [ptlrpc] [11180.140415] ? do_wait_intr+0xf0/0xf0 [11180.140425] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11180.151494] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11180.151549] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11180.151586] mdc_intent_lock+0x353/0x7e0 [mdc] [11180.151617] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11180.151731] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11180.152033] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11180.152065] lmv_intent_open+0x338/0x1150 [lmv] [11180.152095] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11180.152184] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11180.152273] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11180.152298] ll_intent_lock+0xc5/0xaa0 [lustre] [11180.152385] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11180.152465] ll_lookup_it+0xe0e/0x2270 [lustre] [11180.152542] ? do_raw_spin_unlock+0x75/0x190 [11180.152550] ? _raw_spin_unlock+0x12/0x30 [11180.152557] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11180.152583] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11180.152604] ll_atomic_open+0x36a/0xd70 [lustre] [11180.152687] lookup_open+0x356/0xb40 [11180.152698] do_last+0x313/0xfc0 [11180.152703] ? nd_jump_root+0xe5/0x160 [11180.152708] ? path_init+0x437/0x520 [11180.152712] path_openat+0xf7/0x500 [11180.152717] ? do_raw_spin_unlock+0x75/0x190 [11180.152722] do_filp_open+0x99/0x140 [11180.152728] ? getname_flags+0x6e/0x330 [11180.152733] ? __check_object_size+0xff/0x256 [11180.152740] ? do_raw_spin_unlock+0x75/0x190 [11180.157380] kthread+0x1d1/0x200 [11180.157401] ? set_kthread_struct+0x70/0x70 [11180.157406] ret_from_fork+0x1f/0x30 [11180.518298] ? _raw_spin_unlock+0x12/0x30 [11180.520199] do_sys_openat2+0x2b4/0x410 [11180.522081] do_sys_open+0x73/0xa0 [11180.523460] __x64_sys_openat+0x24/0x30 [11180.525180] do_syscall_64+0xc1/0x440 [11180.527522] entry_SYSCALL_64_after_hwframe+0x49/0xae [11180.534534] RIP: 0033:0x7f53425a5332 [11180.537376] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11180.553533] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11180.562069] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11180.569453] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11180.578488] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11180.582069] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11180.589691] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11196.052012] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [11196.054944] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11196.116419] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11196.125455] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11196.138607] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11196.146937] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11196.153261] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11196.177963] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11196.184884] RAX: 0000000000000cad RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11196.191954] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11196.200745] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11196.209347] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11196.215625] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11196.223283] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11196.225341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11196.230422] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11196.236520] Call Trace: [11196.239302] [11196.239825] ? show_regs.cold.9+0x22/0x2f [11196.241418] ? watchdog_timer_fn.cold.6+0x62/0x10a [11196.242535] ? __hrtimer_run_queues+0x1c2/0x4d0 [11196.245858] ? watchdog+0x40/0x40 [11196.250373] ? hrtimer_interrupt+0x128/0x2c0 [11196.252400] ? smp_apic_timer_interrupt+0xa6/0x270 [11196.256776] ? apic_timer_interrupt+0xf/0x20 [11196.258741] [11196.260017] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11196.262812] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11196.266214] ? newidle_balance.isra.93+0x207/0x580 [11196.267766] do_raw_spin_lock+0x177/0x190 [11196.273300] _raw_spin_lock+0x21/0x30 [11196.274411] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11196.279487] process_one_work+0x2c8/0x700 [11196.281553] worker_thread+0x296/0x6e0 [11196.283412] ? rescuer_thread+0x570/0x570 [11196.285194] kthread+0x1d1/0x200 [11196.286176] ? set_kthread_struct+0x70/0x70 [11196.287144] ret_from_fork+0x1f/0x30 [11200.058790] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11200.063114] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11200.083460] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11200.089526] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11200.094743] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11200.098940] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11200.112490] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11200.118768] RAX: 00000000000012a2 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11200.124566] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11200.128596] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11200.133091] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11200.135887] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11200.140292] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11200.144973] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11200.148593] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11200.153084] Call Trace: [11200.155221] [11200.156151] ? show_regs.cold.9+0x22/0x2f [11200.158335] ? watchdog_timer_fn.cold.6+0x62/0x10a [11200.162288] ? __hrtimer_run_queues+0x1c2/0x4d0 [11200.163630] ? watchdog+0x40/0x40 [11200.164592] ? hrtimer_interrupt+0x128/0x2c0 [11200.166133] ? smp_apic_timer_interrupt+0xa6/0x270 [11200.167603] ? apic_timer_interrupt+0xf/0x20 [11200.169564] [11200.170349] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11200.171890] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11200.173917] do_raw_spin_lock+0x177/0x190 [11200.175654] _raw_spin_lock+0x21/0x30 [11200.176946] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11200.180295] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11200.182999] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11200.186671] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11200.191637] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11200.196566] ? woken_wake_function+0x30/0x30 [11200.200053] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11200.206480] kthread+0x1d1/0x200 [11200.209153] ? set_kthread_struct+0x70/0x70 [11200.217680] ret_from_fork+0x1f/0x30 [11202.016041] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8564s! [11202.018602] Showing busy workqueues and worker pools: [11202.020063] workqueue events: flags=0x0 [11202.021502] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11202.023798] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11202.027723] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11202.030070] pending: cpu_latency_work [ptlrpc] [11202.032157] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11202.034881] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11202.038564] workqueue events_long: flags=0x0 [11202.040198] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11202.042784] pending: lu_objects_flush [obdclass] [11202.044835] workqueue events_unbound: flags=0x2 [11202.046466] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11202.048826] pending: flush_memcg_stats_dwork, idle_cull_fn [11202.050947] workqueue events_power_efficient: flags=0x80 [11202.052899] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11202.055153] pending: check_lifetime, do_cache_clean [11202.056896] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11202.059224] pending: neigh_periodic_work, neigh_periodic_work [11202.061305] workqueue mm_percpu_wq: flags=0x8 [11202.062871] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11202.065151] pending: vmstat_update, lru_add_drain_per_cpu [11202.066965] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11202.069008] pending: vmstat_update, lru_add_drain_per_cpu [11202.070988] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11202.073253] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11202.075446] workqueue xprtiod: flags=0xa [11202.076815] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11202.078807] pending: xprt_autoclose, xs_error_handle [11202.080561] workqueue ipv6_addrconf: flags=0x40008 [11202.082077] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11202.084622] pending: addrconf_verify_work [11202.086295] workqueue knbd0-recv: flags=0x1a [11202.087891] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11202.090072] in-flight: 651:recv_work [11202.091610] workqueue ptlrpc_pinger: flags=0x2 [11202.093226] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11202.095573] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11202.097862] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8565s workers=4 idle: 15979 31824 8 [11202.100573] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11192s workers=2 idle: 654 [11208.055032] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [11208.058051] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [11208.058372] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) [11208.065745] Modules linked in: [11208.065747] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 [11208.074587] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11208.074660] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11208.074667] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11208.074669] RIP: 0010:native_safe_halt+0x2e/0x50 [11208.074681] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11208.074684] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11208.074688] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11208.074690] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11208.074692] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11208.074694] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11208.074696] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11208.074699] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11208.074701] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11208.074703] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11208.074710] Call Trace: [11208.074715] [11208.074720] ? show_regs.cold.9+0x22/0x2f [11208.074728] ? watchdog_timer_fn.cold.6+0x62/0x10a [11208.074734] ? __hrtimer_run_queues+0x1c2/0x4d0 [11208.074737] ? watchdog+0x40/0x40 [11208.074741] ? hrtimer_interrupt+0x128/0x2c0 [11208.074744] ? smp_apic_timer_interrupt+0xa6/0x270 [11208.074750] ? apic_timer_interrupt+0xf/0x20 [11208.074754] [11208.074756] ? native_safe_halt+0x2e/0x50 [11208.074760] kvm_wait+0xcd/0xe0 [11208.074766] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11208.074772] do_raw_spin_lock+0x177/0x190 [11208.074775] _raw_spin_lock+0x21/0x30 [11208.074780] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11208.075990] dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11208.076022] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11208.076027] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11208.076030] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11208.076040] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11208.076044] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11208.076054] RAX: 000000000000736b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11208.076056] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11208.076058] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11208.076060] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11208.076063] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11208.076065] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11208.076069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11208.076072] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11208.076078] Call Trace: [11208.076081] [11208.076085] ? show_regs.cold.9+0x22/0x2f [11208.076094] ? watchdog_timer_fn.cold.6+0x62/0x10a [11208.078189] obd_mod_free+0x46/0x140 [ptlrpc] [11208.117294] ? __hrtimer_run_queues+0x1c2/0x4d0 [11208.117317] ? watchdog+0x40/0x40 [11208.117324] ? hrtimer_interrupt+0x128/0x2c0 [11208.117329] ? smp_apic_timer_interrupt+0xa6/0x270 [11208.117339] ? apic_timer_interrupt+0xf/0x20 [11208.117343] [11208.117349] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11208.117360] do_raw_spin_lock+0x177/0x190 [11208.117367] _raw_spin_lock+0x21/0x30 [11208.117372] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11208.117683] ? do_raw_spin_unlock+0x75/0x190 [11208.117689] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11208.117909] ? do_wait_intr+0xf0/0xf0 [11208.117919] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11208.118134] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11208.118914] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11208.119143] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11208.119387] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11208.119428] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11208.119460] mdc_intent_lock+0x353/0x7e0 [mdc] [11208.119487] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11208.119576] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11208.119798] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11208.119828] lmv_intent_open+0x338/0x1150 [lmv] [11208.119858] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11208.119948] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11208.120030] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11208.120053] ll_intent_lock+0xc5/0xaa0 [lustre] [11208.120709] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11208.120796] ll_lookup_it+0xe0e/0x2270 [lustre] [11208.120882] ? do_raw_spin_unlock+0x75/0x190 [11208.120891] ? _raw_spin_unlock+0x12/0x30 [11208.120899] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11208.120925] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11208.120943] ll_atomic_open+0x36a/0xd70 [lustre] [11208.121026] lookup_open+0x356/0xb40 [11208.121034] do_last+0x313/0xfc0 [11208.121038] ? nd_jump_root+0xe5/0x160 [11208.121042] ? path_init+0x437/0x520 [11208.121047] path_openat+0xf7/0x500 [11208.121051] ? do_raw_spin_unlock+0x75/0x190 [11208.121055] do_filp_open+0x99/0x140 [11208.121061] ? getname_flags+0x6e/0x330 [11208.121065] ? __check_object_size+0xff/0x256 [11208.121072] ? do_raw_spin_unlock+0x75/0x190 [11208.121076] ? _raw_spin_unlock+0x12/0x30 [11208.121080] do_sys_openat2+0x2b4/0x410 [11208.121083] do_sys_open+0x73/0xa0 [11208.121087] __x64_sys_openat+0x24/0x30 [11208.121091] do_syscall_64+0xc1/0x440 [11208.121098] entry_SYSCALL_64_after_hwframe+0x49/0xae [11208.121104] RIP: 0033:0x7f53425a5332 [11208.121111] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11208.121115] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11208.121119] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11208.121122] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11208.121124] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11208.121126] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11208.121128] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11208.575923] mdc_commit_open+0x91/0xb0 [mdc] [11208.577180] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11208.583097] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11208.589168] after_reply+0xe2c/0x1780 [ptlrpc] [11208.591904] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11208.594417] ? __next_timer_interrupt+0x160/0x160 [11208.602229] ptlrpcd+0xa82/0xe40 [ptlrpc] [11208.609176] ? do_wait_intr+0xf0/0xf0 [11208.614056] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11208.622811] kthread+0x1d1/0x200 [11208.623840] ? set_kthread_struct+0x70/0x70 [11208.625253] ret_from_fork+0x1f/0x30 [11228.052465] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [11228.058167] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11228.065103] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11228.126357] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11228.126369] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11228.126375] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11228.126733] RIP: 0010:native_safe_halt+0x2e/0x50 [11228.126743] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11228.126747] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11228.126751] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11228.128400] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [11228.128402] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11228.128405] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11228.128407] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11228.128409] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11228.128411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11228.128413] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11228.128426] Call Trace: [11228.128431] [11228.128435] ? show_regs.cold.9+0x22/0x2f [11228.128443] ? watchdog_timer_fn.cold.6+0x62/0x10a [11228.128449] ? __hrtimer_run_queues+0x1c2/0x4d0 [11228.128454] ? watchdog+0x40/0x40 [11228.128459] ? hrtimer_interrupt+0x128/0x2c0 [11228.128462] ? smp_apic_timer_interrupt+0xa6/0x270 [11228.128471] ? apic_timer_interrupt+0xf/0x20 [11228.128477] [11228.128480] ? native_safe_halt+0x2e/0x50 [11228.128484] kvm_wait+0xcd/0xe0 [11228.128491] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11228.128497] ? newidle_balance.isra.93+0x207/0x580 [11228.128503] do_raw_spin_lock+0x177/0x190 [11228.128507] _raw_spin_lock+0x21/0x30 [11228.128511] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11228.128750] process_one_work+0x2c8/0x700 [11228.129142] worker_thread+0x296/0x6e0 [11228.129145] ? rescuer_thread+0x570/0x570 [11228.129148] kthread+0x1d1/0x200 [11228.129151] ? set_kthread_struct+0x70/0x70 [11228.129153] ret_from_fork+0x1f/0x30 [11228.424797] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11228.472436] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11228.480111] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11228.483829] RIP: 0010:native_safe_halt+0x2e/0x50 [11228.486334] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11228.497591] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11228.502371] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11228.505844] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11228.507788] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11228.511950] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11228.515811] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11228.522110] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11228.525663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11228.527460] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11228.532406] Call Trace: [11228.534443] [11228.536347] ? show_regs.cold.9+0x22/0x2f [11228.538788] ? watchdog_timer_fn.cold.6+0x62/0x10a [11228.542080] ? __hrtimer_run_queues+0x1c2/0x4d0 [11228.545453] ? watchdog+0x40/0x40 [11228.549266] ? hrtimer_interrupt+0x128/0x2c0 [11228.556648] ? smp_apic_timer_interrupt+0xa6/0x270 [11228.562112] ? apic_timer_interrupt+0xf/0x20 [11228.567628] [11228.572112] ? native_safe_halt+0x2e/0x50 [11228.573519] kvm_wait+0xcd/0xe0 [11228.575062] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11228.580605] do_raw_spin_lock+0x177/0x190 [11228.584460] _raw_spin_lock+0x21/0x30 [11228.589611] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11228.596995] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11228.598876] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11228.610406] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11228.621431] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11228.631731] ? woken_wake_function+0x30/0x30 [11228.640408] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11228.648981] kthread+0x1d1/0x200 [11228.650133] ? set_kthread_struct+0x70/0x70 [11228.651534] ret_from_fork+0x1f/0x30 [11232.227927] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8595s! [11232.236048] Showing busy workqueues and worker pools: [11232.240668] workqueue events: flags=0x0 [11232.243986] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11232.252514] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11232.260303] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11232.264658] pending: cpu_latency_work [ptlrpc] [11232.272983] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11232.279911] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11232.291857] workqueue events_long: flags=0x0 [11232.295823] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11232.302920] pending: lu_objects_flush [obdclass] [11232.308012] workqueue events_unbound: flags=0x2 [11232.310956] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11232.316521] pending: flush_memcg_stats_dwork, idle_cull_fn [11232.320170] workqueue events_power_efficient: flags=0x80 [11232.326233] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11232.332846] pending: check_lifetime, do_cache_clean [11232.336785] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11232.340700] pending: neigh_periodic_work, neigh_periodic_work [11232.349285] workqueue mm_percpu_wq: flags=0x8 [11232.355086] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11232.364063] pending: vmstat_update, lru_add_drain_per_cpu [11232.370973] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11232.380610] pending: vmstat_update, lru_add_drain_per_cpu [11232.385134] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11232.389382] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11232.393228] workqueue xprtiod: flags=0xa [11232.396627] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11232.400766] pending: xprt_autoclose, xs_error_handle [11232.406756] workqueue ipv6_addrconf: flags=0x40008 [11232.412114] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11232.416849] pending: addrconf_verify_work [11232.418055] workqueue knbd0-recv: flags=0x1a [11232.424406] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11232.428577] in-flight: 651:recv_work [11232.430580] workqueue ptlrpc_pinger: flags=0x2 [11232.431943] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11232.442999] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11232.453434] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8595s workers=4 idle: 15979 31824 8 [11232.460813] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11222s workers=2 idle: 654 [11236.059041] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11236.059952] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11236.069854] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul [11236.076653] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11236.076736] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11236.076745] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11236.076748] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11236.076775] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11236.076780] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11236.076785] RAX: 0000000000007f08 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11236.076788] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11236.076791] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11236.076793] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11236.076795] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11236.076799] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11236.076802] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11236.076805] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11236.076814] Call Trace: [11236.076821] [11236.076828] ? show_regs.cold.9+0x22/0x2f [11236.076840] ? watchdog_timer_fn.cold.6+0x62/0x10a [11236.076848] ? __hrtimer_run_queues+0x1c2/0x4d0 [11236.076852] ? watchdog+0x40/0x40 [11236.076858] ? hrtimer_interrupt+0x128/0x2c0 [11236.076862] ? smp_apic_timer_interrupt+0xa6/0x270 [11236.076873] ? apic_timer_interrupt+0xf/0x20 [11236.076878] [11236.076881] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11236.076890] do_raw_spin_lock+0x177/0x190 [11236.076895] _raw_spin_lock+0x21/0x30 [11236.076901] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11236.077233] ? do_raw_spin_unlock+0x75/0x190 [11236.077239] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11236.077441] ? do_wait_intr+0xf0/0xf0 [11236.077452] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11236.077653] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11236.078176] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11236.078393] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11236.078535] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11236.078582] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11236.078614] mdc_intent_lock+0x353/0x7e0 [mdc] [11236.078646] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11236.078739] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11236.078996] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11236.079032] lmv_intent_open+0x338/0x1150 [lmv] [11236.079065] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11236.080020] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11236.080109] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11236.080139] ll_intent_lock+0xc5/0xaa0 [lustre] [11236.080228] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11236.080315] ll_lookup_it+0xe0e/0x2270 [lustre] [11236.080401] ? do_raw_spin_unlock+0x75/0x190 [11236.080409] ? _raw_spin_unlock+0x12/0x30 [11236.080417] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11236.080443] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11236.080463] ll_atomic_open+0x36a/0xd70 [lustre] [11236.080565] lookup_open+0x356/0xb40 [11236.080575] do_last+0x313/0xfc0 [11236.080581] ? nd_jump_root+0xe5/0x160 [11236.080586] ? path_init+0x437/0x520 [11236.080591] path_openat+0xf7/0x500 [11236.080597] ? do_raw_spin_unlock+0x75/0x190 [11236.080602] do_filp_open+0x99/0x140 [11236.080608] ? getname_flags+0x6e/0x330 [11236.080613] ? __check_object_size+0xff/0x256 [11236.080623] ? do_raw_spin_unlock+0x75/0x190 [11236.080627] ? _raw_spin_unlock+0x12/0x30 [11236.080632] do_sys_openat2+0x2b4/0x410 [11236.080637] do_sys_open+0x73/0xa0 [11236.080642] __x64_sys_openat+0x24/0x30 [11236.080646] do_syscall_64+0xc1/0x440 [11236.080652] entry_SYSCALL_64_after_hwframe+0x49/0xae [11236.080659] RIP: 0033:0x7f53425a5332 [11236.080666] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11236.080671] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11236.080676] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11236.080679] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11236.080682] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11236.080684] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11236.080687] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11236.567788] crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11236.584644] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11236.603059] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11236.609053] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11236.612218] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11236.628644] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11236.634031] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11236.643985] RDX: 0000000000006892 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11236.652877] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11236.668284] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11236.674048] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11236.676647] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11236.687029] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11236.697047] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11236.708973] Call Trace: [11236.715432] [11236.716596] ? show_regs.cold.9+0x22/0x2f [11236.721322] ? watchdog_timer_fn.cold.6+0x62/0x10a [11236.727562] ? __hrtimer_run_queues+0x1c2/0x4d0 [11236.733173] ? watchdog+0x40/0x40 [11236.738251] ? hrtimer_interrupt+0x128/0x2c0 [11236.742959] ? smp_apic_timer_interrupt+0xa6/0x270 [11236.746151] ? apic_timer_interrupt+0xf/0x20 [11236.749492] [11236.750319] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11236.753417] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11236.756163] do_raw_spin_lock+0x177/0x190 [11236.765881] _raw_spin_lock+0x21/0x30 [11236.769029] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11236.773417] obd_mod_free+0x46/0x140 [ptlrpc] [11236.781509] mdc_commit_open+0x91/0xb0 [mdc] [11236.783996] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11236.790557] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11236.798037] after_reply+0xe2c/0x1780 [ptlrpc] [11236.802618] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11236.805307] ? __next_timer_interrupt+0x160/0x160 [11236.809015] ptlrpcd+0xa82/0xe40 [ptlrpc] [11236.815115] ? do_wait_intr+0xf0/0xf0 [11236.816332] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11236.823081] kthread+0x1d1/0x200 [11236.828271] ? set_kthread_struct+0x70/0x70 [11236.835912] ret_from_fork+0x1f/0x30 [11256.062538] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [11256.064465] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11256.104170] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11256.115825] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11256.122831] RIP: 0010:native_safe_halt+0x2e/0x50 [11256.132263] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11256.169448] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11256.187966] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11256.200607] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11256.214105] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11256.224606] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11256.230764] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11256.240845] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11256.247018] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11256.256602] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11256.261142] Call Trace: [11256.262461] [11256.263290] ? show_regs.cold.9+0x22/0x2f [11256.266878] ? watchdog_timer_fn.cold.6+0x62/0x10a [11256.273979] ? __hrtimer_run_queues+0x1c2/0x4d0 [11256.281006] ? watchdog+0x40/0x40 [11256.282244] ? hrtimer_interrupt+0x128/0x2c0 [11256.286763] ? smp_apic_timer_interrupt+0xa6/0x270 [11256.288505] ? apic_timer_interrupt+0xf/0x20 [11256.294900] [11256.295872] ? native_safe_halt+0x2e/0x50 [11256.300771] kvm_wait+0xcd/0xe0 [11256.305179] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11256.314049] do_raw_spin_lock+0x177/0x190 [11256.319735] _raw_spin_lock+0x21/0x30 [11256.322199] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11256.326205] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11256.330873] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11256.337306] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11256.342272] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11256.345763] ? woken_wake_function+0x30/0x30 [11256.349987] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11256.352689] kthread+0x1d1/0x200 [11256.353794] ? set_kthread_struct+0x70/0x70 [11256.357931] ret_from_fork+0x1f/0x30 [11260.052786] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [11260.056392] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11260.077143] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11260.082657] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11260.086275] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11260.089568] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11260.092468] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11260.100842] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11260.103960] RAX: 0000000000001d88 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11260.107038] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11260.110152] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11260.113317] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11260.116277] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11260.119349] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11260.122325] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11260.124799] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11260.126806] Call Trace: [11260.127675] [11260.128442] ? show_regs.cold.9+0x22/0x2f [11260.130082] ? watchdog_timer_fn.cold.6+0x62/0x10a [11260.132008] ? __hrtimer_run_queues+0x1c2/0x4d0 [11260.134010] ? watchdog+0x40/0x40 [11260.135325] ? hrtimer_interrupt+0x128/0x2c0 [11260.137560] ? smp_apic_timer_interrupt+0xa6/0x270 [11260.139793] ? apic_timer_interrupt+0xf/0x20 [11260.141684] [11260.142452] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11260.144332] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11260.146780] ? newidle_balance.isra.93+0x207/0x580 [11260.148794] do_raw_spin_lock+0x177/0x190 [11260.150735] _raw_spin_lock+0x21/0x30 [11260.152332] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11260.154999] process_one_work+0x2c8/0x700 [11260.156894] worker_thread+0x296/0x6e0 [11260.158326] ? rescuer_thread+0x570/0x570 [11260.160121] kthread+0x1d1/0x200 [11260.161559] ? set_kthread_struct+0x70/0x70 [11260.162991] ret_from_fork+0x1f/0x30 [11262.944040] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8625s! [11262.946238] Showing busy workqueues and worker pools: [11262.947944] workqueue events: flags=0x0 [11262.949057] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11262.950888] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11262.955041] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11262.956665] pending: cpu_latency_work [ptlrpc] [11262.958442] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11262.960151] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11262.962748] workqueue events_long: flags=0x0 [11262.964103] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11262.966192] pending: lu_objects_flush [obdclass] [11262.968094] workqueue events_unbound: flags=0x2 [11262.969632] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11262.971553] pending: flush_memcg_stats_dwork, idle_cull_fn [11262.973491] workqueue events_power_efficient: flags=0x80 [11262.975130] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11262.977186] pending: check_lifetime, do_cache_clean [11262.978774] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11262.980766] pending: neigh_periodic_work, neigh_periodic_work [11262.982744] workqueue mm_percpu_wq: flags=0x8 [11262.984115] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11262.985963] pending: vmstat_update, lru_add_drain_per_cpu [11262.987721] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11262.989526] pending: vmstat_update, lru_add_drain_per_cpu [11262.991222] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11262.993488] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11262.995243] workqueue xprtiod: flags=0xa [11262.996543] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11262.998564] pending: xprt_autoclose, xs_error_handle [11263.000468] workqueue ipv6_addrconf: flags=0x40008 [11263.002047] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11263.004189] pending: addrconf_verify_work [11263.005677] workqueue knbd0-recv: flags=0x1a [11263.007068] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11263.009217] in-flight: 651:recv_work [11263.010652] workqueue ptlrpc_pinger: flags=0x2 [11263.012110] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11263.014112] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11263.016278] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8625s workers=4 idle: 15979 31824 8 [11263.018858] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11253s workers=2 idle: 654 [11264.057073] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11264.061440] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11264.062824] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11264.062921] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11264.062932] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11264.062936] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11264.062954] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11264.062961] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11264.062966] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11264.062969] RDX: 00000000000025b8 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11264.062971] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11264.062977] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11264.062979] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11264.062982] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11264.062985] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11264.062987] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11264.062997] Call Trace: [11264.063003] [11264.063013] ? show_regs.cold.9+0x22/0x2f [11264.063026] ? watchdog_timer_fn.cold.6+0x62/0x10a [11264.063033] ? __hrtimer_run_queues+0x1c2/0x4d0 [11264.063038] ? watchdog+0x40/0x40 [11264.063043] ? hrtimer_interrupt+0x128/0x2c0 [11264.063048] ? smp_apic_timer_interrupt+0xa6/0x270 [11264.063057] ? apic_timer_interrupt+0xf/0x20 [11264.063062] [11264.063066] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11264.063071] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11264.063076] do_raw_spin_lock+0x177/0x190 [11264.063081] _raw_spin_lock+0x21/0x30 [11264.063091] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11264.063641] obd_mod_free+0x46/0x140 [ptlrpc] [11264.063890] mdc_commit_open+0x91/0xb0 [mdc] [11264.063958] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11264.076873] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11264.076970] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11264.076980] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11264.076984] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11264.077004] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11264.077008] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11264.077024] RAX: 0000000000002d3c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11264.077027] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11264.077029] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11264.077031] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11264.077033] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11264.077037] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11264.077040] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11264.077043] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11264.077052] Call Trace: [11264.077058] [11264.077065] ? show_regs.cold.9+0x22/0x2f [11264.077076] ? watchdog_timer_fn.cold.6+0x62/0x10a [11264.077084] ? __hrtimer_run_queues+0x1c2/0x4d0 [11264.077088] ? watchdog+0x40/0x40 [11264.077093] ? hrtimer_interrupt+0x128/0x2c0 [11264.077098] ? smp_apic_timer_interrupt+0xa6/0x270 [11264.077106] ? apic_timer_interrupt+0xf/0x20 [11264.077111] [11264.077114] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11264.077121] do_raw_spin_lock+0x177/0x190 [11264.077126] _raw_spin_lock+0x21/0x30 [11264.077131] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11264.079078] ? do_raw_spin_unlock+0x75/0x190 [11264.079089] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11264.082560] ? do_wait_intr+0xf0/0xf0 [11264.082581] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11264.082961] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11264.083200] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11264.122053] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11264.127616] after_reply+0xe2c/0x1780 [ptlrpc] [11264.127822] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11264.128015] ? __next_timer_interrupt+0x160/0x160 [11264.128028] ptlrpcd+0xa82/0xe40 [ptlrpc] [11264.128231] ? do_wait_intr+0xf0/0xf0 [11264.128245] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11264.128453] kthread+0x1d1/0x200 [11264.128459] ? set_kthread_struct+0x70/0x70 [11264.128463] ret_from_fork+0x1f/0x30 [11264.616811] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11264.621933] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11264.628620] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11264.634490] mdc_intent_lock+0x353/0x7e0 [mdc] [11264.637081] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11264.643174] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11264.646237] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11264.648468] lmv_intent_open+0x338/0x1150 [lmv] [11264.650233] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11264.652896] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11264.655196] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11264.657754] ll_intent_lock+0xc5/0xaa0 [lustre] [11264.659706] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11264.661945] ll_lookup_it+0xe0e/0x2270 [lustre] [11264.666777] ? do_raw_spin_unlock+0x75/0x190 [11264.670729] ? _raw_spin_unlock+0x12/0x30 [11264.676230] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11264.682183] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11264.683963] ll_atomic_open+0x36a/0xd70 [lustre] [11264.690155] lookup_open+0x356/0xb40 [11264.693980] do_last+0x313/0xfc0 [11264.696887] ? nd_jump_root+0xe5/0x160 [11264.700171] ? path_init+0x437/0x520 [11264.705832] path_openat+0xf7/0x500 [11264.710688] ? do_raw_spin_unlock+0x75/0x190 [11264.714917] do_filp_open+0x99/0x140 [11264.719181] ? getname_flags+0x6e/0x330 [11264.723977] ? __check_object_size+0xff/0x256 [11264.730343] ? do_raw_spin_unlock+0x75/0x190 [11264.734584] ? _raw_spin_unlock+0x12/0x30 [11264.737253] do_sys_openat2+0x2b4/0x410 [11264.740774] do_sys_open+0x73/0xa0 [11264.746249] __x64_sys_openat+0x24/0x30 [11264.749744] do_syscall_64+0xc1/0x440 [11264.750965] entry_SYSCALL_64_after_hwframe+0x49/0xae [11264.753947] RIP: 0033:0x7f53425a5332 [11264.755726] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11264.764200] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11264.767378] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11264.770576] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11264.773083] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11264.776163] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11264.779211] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11284.059131] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [11284.061670] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11284.075222] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11284.079611] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11284.082799] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11284.085562] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11284.095934] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11284.098904] RAX: 0000000000007889 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11284.103683] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11284.107309] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11284.114991] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11284.122032] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11284.129216] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11284.138508] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11284.140320] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11284.146835] Call Trace: [11284.150257] [11284.153466] ? show_regs.cold.9+0x22/0x2f [11284.157477] ? watchdog_timer_fn.cold.6+0x62/0x10a [11284.162239] ? __hrtimer_run_queues+0x1c2/0x4d0 [11284.166851] ? watchdog+0x40/0x40 [11284.169483] ? hrtimer_interrupt+0x128/0x2c0 [11284.171109] ? smp_apic_timer_interrupt+0xa6/0x270 [11284.172592] ? apic_timer_interrupt+0xf/0x20 [11284.175255] [11284.176219] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11284.181317] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11284.186236] do_raw_spin_lock+0x177/0x190 [11284.188814] _raw_spin_lock+0x21/0x30 [11284.191862] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11284.200212] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11284.206836] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11284.214273] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11284.218831] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11284.224463] ? woken_wake_function+0x30/0x30 [11284.225665] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11284.231586] kthread+0x1d1/0x200 [11284.233908] ? set_kthread_struct+0x70/0x70 [11284.237771] ret_from_fork+0x1f/0x30 [11288.053633] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [11288.058341] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11288.089518] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11288.100984] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11288.106810] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11288.111844] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11288.115134] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11288.131174] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [11288.138509] RAX: 000000000000319f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11288.148276] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11288.157682] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11288.169134] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11288.176308] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11288.181560] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11288.186186] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11288.190160] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11288.194034] Call Trace: [11288.194939] [11288.195670] ? show_regs.cold.9+0x22/0x2f [11288.197552] ? watchdog_timer_fn.cold.6+0x62/0x10a [11288.202703] ? __hrtimer_run_queues+0x1c2/0x4d0 [11288.208208] ? watchdog+0x40/0x40 [11288.211835] ? hrtimer_interrupt+0x128/0x2c0 [11288.214250] ? smp_apic_timer_interrupt+0xa6/0x270 [11288.218630] ? apic_timer_interrupt+0xf/0x20 [11288.225087] [11288.226588] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11288.232298] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11288.236609] ? newidle_balance.isra.93+0x207/0x580 [11288.240488] do_raw_spin_lock+0x177/0x190 [11288.243006] _raw_spin_lock+0x21/0x30 [11288.245297] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11288.250181] process_one_work+0x2c8/0x700 [11288.253412] worker_thread+0x296/0x6e0 [11288.256487] ? rescuer_thread+0x570/0x570 [11288.259237] kthread+0x1d1/0x200 [11288.261552] ? set_kthread_struct+0x70/0x70 [11288.267417] ret_from_fork+0x1f/0x30 [11292.056795] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11292.057046] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11292.062399] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11292.062479] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11292.062486] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11292.062489] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11292.062505] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11292.062509] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11292.062513] RAX: 0000000000001489 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11292.062515] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11292.062517] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11292.062519] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11292.062521] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11292.062524] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11292.062527] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11292.062529] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11292.062539] Call Trace: [11292.062546] [11292.062551] ? show_regs.cold.9+0x22/0x2f [11292.062562] ? watchdog_timer_fn.cold.6+0x62/0x10a [11292.062568] ? __hrtimer_run_queues+0x1c2/0x4d0 [11292.062572] ? watchdog+0x40/0x40 [11292.062576] ? hrtimer_interrupt+0x128/0x2c0 [11292.062581] ? smp_apic_timer_interrupt+0xa6/0x270 [11292.062589] ? apic_timer_interrupt+0xf/0x20 [11292.062594] [11292.062597] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11292.062604] do_raw_spin_lock+0x177/0x190 [11292.062609] _raw_spin_lock+0x21/0x30 [11292.062614] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11292.062960] ? do_raw_spin_unlock+0x75/0x190 [11292.062966] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11292.063166] ? do_wait_intr+0xf0/0xf0 [11292.063220] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11292.063907] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11292.064847] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11292.076396] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11292.076481] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11292.076489] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11292.076492] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11292.106656] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11292.123141] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11292.123157] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11292.123167] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11292.123170] RDX: 00000000000018d2 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11292.123173] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11292.123175] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11292.123178] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11292.123181] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11292.123188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11292.123191] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11292.123199] Call Trace: [11292.123206] [11292.123212] ? show_regs.cold.9+0x22/0x2f [11292.123231] ? watchdog_timer_fn.cold.6+0x62/0x10a [11292.123238] ? __hrtimer_run_queues+0x1c2/0x4d0 [11292.123246] ? watchdog+0x40/0x40 [11292.123251] ? hrtimer_interrupt+0x128/0x2c0 [11292.123256] ? smp_apic_timer_interrupt+0xa6/0x270 [11292.123264] ? apic_timer_interrupt+0xf/0x20 [11292.123273] [11292.123280] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11292.123291] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11292.123296] do_raw_spin_lock+0x177/0x190 [11292.123300] _raw_spin_lock+0x21/0x30 [11292.123308] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11292.136815] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11292.150415] obd_mod_free+0x46/0x140 [ptlrpc] [11292.153081] mdc_commit_open+0x91/0xb0 [mdc] [11292.153126] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11292.154116] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11292.155368] after_reply+0xe2c/0x1780 [ptlrpc] [11292.157067] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11292.176772] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11292.180765] ? __next_timer_interrupt+0x160/0x160 [11292.180797] ptlrpcd+0xa82/0xe40 [ptlrpc] [11292.181088] ? do_wait_intr+0xf0/0xf0 [11292.181099] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11292.182060] kthread+0x1d1/0x200 [11292.182068] ? set_kthread_struct+0x70/0x70 [11292.182072] ret_from_fork+0x1f/0x30 [11292.467080] mdc_intent_lock+0x353/0x7e0 [mdc] [11292.468871] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11292.471594] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11292.473690] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11292.477087] lmv_intent_open+0x338/0x1150 [lmv] [11292.479167] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11292.485472] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11292.487735] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11292.490711] ll_intent_lock+0xc5/0xaa0 [lustre] [11292.492614] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11292.494364] ll_lookup_it+0xe0e/0x2270 [lustre] [11292.497589] ? do_raw_spin_unlock+0x75/0x190 [11292.499453] ? _raw_spin_unlock+0x12/0x30 [11292.501255] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11292.503904] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11292.506659] ll_atomic_open+0x36a/0xd70 [lustre] [11292.508637] lookup_open+0x356/0xb40 [11292.511113] do_last+0x313/0xfc0 [11292.512212] ? nd_jump_root+0xe5/0x160 [11292.513504] ? path_init+0x437/0x520 [11292.519205] path_openat+0xf7/0x500 [11292.520615] ? do_raw_spin_unlock+0x75/0x190 [11292.523241] do_filp_open+0x99/0x140 [11292.525567] ? getname_flags+0x6e/0x330 [11292.527493] ? __check_object_size+0xff/0x256 [11292.529627] ? do_raw_spin_unlock+0x75/0x190 [11292.532012] ? _raw_spin_unlock+0x12/0x30 [11292.533969] do_sys_openat2+0x2b4/0x410 [11292.536048] do_sys_open+0x73/0xa0 [11292.538843] __x64_sys_openat+0x24/0x30 [11292.541677] do_syscall_64+0xc1/0x440 [11292.544813] entry_SYSCALL_64_after_hwframe+0x49/0xae [11292.547619] RIP: 0033:0x7f53425a5332 [11292.549508] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11292.563468] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11292.571059] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11292.579505] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11292.589024] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11292.596576] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11292.600598] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11293.152044] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8656s! [11293.154747] Showing busy workqueues and worker pools: [11293.159939] workqueue events: flags=0x0 [11293.161424] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11293.167619] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11293.178537] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11293.184173] pending: cpu_latency_work [ptlrpc] [11293.189337] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11293.198742] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11293.209441] workqueue events_long: flags=0x0 [11293.219335] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11293.223620] pending: lu_objects_flush [obdclass] [11293.233516] workqueue events_unbound: flags=0x2 [11293.242053] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11293.248690] pending: flush_memcg_stats_dwork, idle_cull_fn [11293.250869] workqueue events_power_efficient: flags=0x80 [11293.257175] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11293.261719] pending: check_lifetime, do_cache_clean [11293.265712] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11293.273226] pending: neigh_periodic_work, neigh_periodic_work [11293.275742] workqueue mm_percpu_wq: flags=0x8 [11293.281329] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11293.288588] pending: vmstat_update, lru_add_drain_per_cpu [11293.295668] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11293.304371] pending: vmstat_update, lru_add_drain_per_cpu [11293.311537] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11293.321589] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11293.324237] workqueue xprtiod: flags=0xa [11293.326012] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11293.328420] pending: xprt_autoclose, xs_error_handle [11293.330817] workqueue ipv6_addrconf: flags=0x40008 [11293.333061] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11293.335619] pending: addrconf_verify_work [11293.337473] workqueue knbd0-recv: flags=0x1a [11293.341370] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11293.344949] in-flight: 651:recv_work [11293.346396] workqueue ptlrpc_pinger: flags=0x2 [11293.352683] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11293.358388] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11293.361722] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8656s workers=4 idle: 15979 31824 8 [11293.366663] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11283s workers=2 idle: 654 [11312.058709] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [11312.061475] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11312.100200] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11312.108084] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11312.115062] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11312.119389] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11312.133089] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11312.139242] RAX: 0000000000007d0b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11312.144746] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11312.148238] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11312.153172] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11312.156379] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11312.162028] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11312.171470] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11312.176142] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11312.181244] Call Trace: [11312.182127] [11312.184287] ? show_regs.cold.9+0x22/0x2f [11312.187172] ? watchdog_timer_fn.cold.6+0x62/0x10a [11312.190226] ? __hrtimer_run_queues+0x1c2/0x4d0 [11312.198024] ? watchdog+0x40/0x40 [11312.200137] ? hrtimer_interrupt+0x128/0x2c0 [11312.203155] ? smp_apic_timer_interrupt+0xa6/0x270 [11312.205280] ? apic_timer_interrupt+0xf/0x20 [11312.211443] [11312.212378] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11312.215979] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11312.219946] do_raw_spin_lock+0x177/0x190 [11312.224256] _raw_spin_lock+0x21/0x30 [11312.226058] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11312.230169] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11312.233207] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11312.239279] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11312.245641] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11312.252019] ? woken_wake_function+0x30/0x30 [11312.258230] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11312.265176] kthread+0x1d1/0x200 [11312.267338] ? set_kthread_struct+0x70/0x70 [11312.268788] ret_from_fork+0x1f/0x30 [11320.053170] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [11320.057025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11320.057977] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11320.057985] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11320.058060] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11320.058066] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11320.058069] RIP: 0010:native_safe_halt+0x2e/0x50 [11320.058090] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11320.058095] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11320.058099] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11320.058102] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11320.058104] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11320.058107] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11320.058109] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11320.058113] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11320.058116] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11320.058119] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11320.058128] Call Trace: [11320.058134] [11320.058140] ? show_regs.cold.9+0x22/0x2f [11320.058149] ? watchdog_timer_fn.cold.6+0x62/0x10a [11320.058155] ? __hrtimer_run_queues+0x1c2/0x4d0 [11320.058159] ? watchdog+0x40/0x40 [11320.058163] ? hrtimer_interrupt+0x128/0x2c0 [11320.058168] ? smp_apic_timer_interrupt+0xa6/0x270 [11320.058175] ? apic_timer_interrupt+0xf/0x20 [11320.058179] [11320.058182] ? native_safe_halt+0x2e/0x50 [11320.058185] kvm_wait+0xcd/0xe0 [11320.058193] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11320.058201] do_raw_spin_lock+0x177/0x190 [11320.058206] _raw_spin_lock+0x21/0x30 [11320.058210] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11320.058512] ? do_raw_spin_unlock+0x75/0x190 [11320.058519] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11320.058731] ? do_wait_intr+0xf0/0xf0 [11320.058740] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11320.059122] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11320.059355] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11320.059691] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11320.059802] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11320.059840] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11320.059869] mdc_intent_lock+0x353/0x7e0 [mdc] [11320.060060] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11320.060186] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11320.060424] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11320.060515] lmv_intent_open+0x338/0x1150 [lmv] [11320.060552] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11320.061016] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11320.061105] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11320.061130] ll_intent_lock+0xc5/0xaa0 [lustre] [11320.061217] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11320.061312] ll_lookup_it+0xe0e/0x2270 [lustre] [11320.061398] ? do_raw_spin_unlock+0x75/0x190 [11320.061406] ? _raw_spin_unlock+0x12/0x30 [11320.061411] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11320.061437] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11320.061605] ll_atomic_open+0x36a/0xd70 [lustre] [11320.061759] lookup_open+0x356/0xb40 [11320.061848] do_last+0x313/0xfc0 [11320.061853] ? nd_jump_root+0xe5/0x160 [11320.061876] ? path_init+0x437/0x520 [11320.061881] path_openat+0xf7/0x500 [11320.061964] ? do_raw_spin_unlock+0x75/0x190 [11320.061970] do_filp_open+0x99/0x140 [11320.061976] ? getname_flags+0x6e/0x330 [11320.061980] ? __check_object_size+0xff/0x256 [11320.061987] ? do_raw_spin_unlock+0x75/0x190 [11320.061991] ? _raw_spin_unlock+0x12/0x30 [11320.061996] do_sys_openat2+0x2b4/0x410 [11320.062000] do_sys_open+0x73/0xa0 [11320.062008] __x64_sys_openat+0x24/0x30 [11320.062013] do_syscall_64+0xc1/0x440 [11320.062020] entry_SYSCALL_64_after_hwframe+0x49/0xae [11320.062026] RIP: 0033:0x7f53425a5332 [11320.062034] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11320.062037] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11320.062042] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11320.062044] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11320.062046] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11320.062048] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11320.062050] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11320.066699] Modules linked in: [11320.072236] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11320.072447] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11320.072455] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11320.072459] RIP: 0010:native_safe_halt+0x2e/0x50 [11320.072472] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11320.072477] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [11320.072481] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11320.072483] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11320.072485] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11320.072487] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11320.072490] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11320.072593] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11320.072597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11320.072599] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11320.072608] Call Trace: [11320.072613] [11320.072619] ? show_regs.cold.9+0x22/0x2f [11320.072630] ? watchdog_timer_fn.cold.6+0x62/0x10a [11320.072636] ? __hrtimer_run_queues+0x1c2/0x4d0 [11320.072641] ? watchdog+0x40/0x40 [11320.072645] ? hrtimer_interrupt+0x128/0x2c0 [11320.072649] ? smp_apic_timer_interrupt+0xa6/0x270 [11320.072656] ? apic_timer_interrupt+0xf/0x20 [11320.072660] [11320.072663] ? native_safe_halt+0x2e/0x50 [11320.072667] kvm_wait+0xcd/0xe0 [11320.072675] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11320.072682] do_raw_spin_lock+0x177/0x190 [11320.072687] _raw_spin_lock+0x21/0x30 [11320.072693] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11320.076664] lustre(O) osp(O) [11320.097087] obd_mod_free+0x46/0x140 [ptlrpc] [11320.097595] mdc_commit_open+0x91/0xb0 [mdc] [11320.097643] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11320.097871] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11320.098086] after_reply+0xe2c/0x1780 [ptlrpc] [11320.103541] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11320.103613] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11320.103621] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11320.103627] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11320.103922] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11320.103933] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11320.103938] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11320.103943] RAX: 0000000000007e0a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11320.103945] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11320.103948] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11320.103950] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11320.103953] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11320.103956] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11320.103959] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11320.103962] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11320.103971] Call Trace: [11320.103977] [11320.103981] ? show_regs.cold.9+0x22/0x2f [11320.103990] ? watchdog_timer_fn.cold.6+0x62/0x10a [11320.103997] ? __hrtimer_run_queues+0x1c2/0x4d0 [11320.104001] ? watchdog+0x40/0x40 [11320.104006] ? hrtimer_interrupt+0x128/0x2c0 [11320.104010] ? smp_apic_timer_interrupt+0xa6/0x270 [11320.104021] ? apic_timer_interrupt+0xf/0x20 [11320.104026] [11320.104028] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11320.104033] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11320.104036] ? newidle_balance.isra.93+0x207/0x580 [11320.104043] do_raw_spin_lock+0x177/0x190 [11320.104048] _raw_spin_lock+0x21/0x30 [11320.104053] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11320.108828] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11320.114159] process_one_work+0x2c8/0x700 [11320.114190] worker_thread+0x296/0x6e0 [11320.114196] ? rescuer_thread+0x570/0x570 [11320.114200] kthread+0x1d1/0x200 [11320.114205] ? set_kthread_struct+0x70/0x70 [11320.114213] ret_from_fork+0x1f/0x30 [11320.652758] ? __next_timer_interrupt+0x160/0x160 [11320.656055] ptlrpcd+0xa82/0xe40 [ptlrpc] [11320.658255] ? do_wait_intr+0xf0/0xf0 [11320.661079] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11320.664631] kthread+0x1d1/0x200 [11320.666086] ? set_kthread_struct+0x70/0x70 [11320.667834] ret_from_fork+0x1f/0x30 [11323.873440] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8686s! [11323.875926] Showing busy workqueues and worker pools: [11323.877633] workqueue events: flags=0x0 [11323.878861] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11323.880909] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11323.885467] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11323.889002] pending: cpu_latency_work [ptlrpc] [11323.896172] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11323.907693] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11323.918628] workqueue events_long: flags=0x0 [11323.922472] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11323.928827] pending: lu_objects_flush [obdclass] [11323.932242] workqueue events_unbound: flags=0x2 [11323.934483] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11323.938905] pending: flush_memcg_stats_dwork, idle_cull_fn [11323.940518] workqueue events_power_efficient: flags=0x80 [11323.950346] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11323.954811] pending: check_lifetime, do_cache_clean [11323.961183] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11323.969888] pending: neigh_periodic_work, neigh_periodic_work [11323.974757] workqueue mm_percpu_wq: flags=0x8 [11323.980921] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11323.984737] pending: vmstat_update, lru_add_drain_per_cpu [11323.987730] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11323.990010] pending: vmstat_update, lru_add_drain_per_cpu [11323.993817] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11323.996888] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11324.000203] workqueue xprtiod: flags=0xa [11324.002136] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11324.005041] pending: xprt_autoclose, xs_error_handle [11324.006761] workqueue ipv6_addrconf: flags=0x40008 [11324.010791] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11324.013976] pending: addrconf_verify_work [11324.016136] workqueue knbd0-recv: flags=0x1a [11324.017773] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11324.020766] in-flight: 651:recv_work [11324.022881] workqueue ptlrpc_pinger: flags=0x2 [11324.027661] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11324.031560] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11324.034176] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8686s workers=4 idle: 15979 31824 8 [11324.040803] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11314s workers=2 idle: 654 [11334.645264] rcu: INFO: rcu_sched self-detected stall on CPU [11334.654442] rcu: 2-....: (5086370 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [11334.666369] (t=9240219 jiffies g=274185 q=66191) [11334.681364] rcu: rcu_sched kthread starved for 8698585 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [11334.692451] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11334.707509] rcu: RCU grace-period kthread stack dump: [11334.711750] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11334.725173] Call Trace: [11334.727009] __schedule+0x351/0xcb0 [11334.733355] schedule+0xc0/0x180 [11334.734473] schedule_timeout+0xb4/0x190 [11334.738216] ? do_raw_spin_unlock+0x75/0x190 [11334.743619] ? __next_timer_interrupt+0x160/0x160 [11334.752031] ? prepare_to_swait_event+0xf3/0x140 [11334.756755] rcu_gp_kthread+0x8dd/0x15b0 [11334.758779] ? force_qs_rnp+0x2c0/0x2c0 [11334.763422] kthread+0x1d1/0x200 [11334.764613] ? set_kthread_struct+0x70/0x70 [11334.771863] ret_from_fork+0x1f/0x30 [11334.778482] rcu: Stack dump where RCU GP kthread last ran: [11334.782179] NMI backtrace for cpu 2 [11334.783373] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11334.806815] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11334.816871] Call Trace: [11334.821298] [11334.822008] ? dump_stack+0xbb/0x10e [11334.828542] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11334.834952] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11334.842406] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11334.852710] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11334.863963] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11334.868990] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [11334.879498] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [11334.894894] ? update_process_times+0x6d/0xc0 [11334.902748] ? tick_sched_handle.isra.18+0x27/0x90 [11334.910603] ? tick_sched_timer+0x53/0xb0 [11334.917720] ? __hrtimer_run_queues+0x1c2/0x4d0 [11334.919334] ? tick_sched_handle.isra.18+0x90/0x90 [11334.926880] ? hrtimer_interrupt+0x128/0x2c0 [11334.928671] ? smp_apic_timer_interrupt+0xa6/0x270 [11334.938631] ? apic_timer_interrupt+0xf/0x20 [11334.940156] [11334.940858] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11334.953107] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11334.961770] ? do_raw_spin_lock+0x177/0x190 [11334.966224] ? _raw_spin_lock+0x21/0x30 [11334.969198] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11334.976878] ? obd_mod_free+0x46/0x140 [ptlrpc] [11334.982241] ? mdc_commit_open+0x91/0xb0 [mdc] [11334.988706] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11334.999864] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11335.007807] ? after_reply+0xe2c/0x1780 [ptlrpc] [11335.013015] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11335.020780] ? __next_timer_interrupt+0x160/0x160 [11335.027795] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11335.037774] ? do_wait_intr+0xf0/0xf0 [11335.039206] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11335.047421] ? kthread+0x1d1/0x200 [11335.052002] ? set_kthread_struct+0x70/0x70 [11335.059951] ? ret_from_fork+0x1f/0x30 [11335.063635] Sending NMI from CPU 2 to CPUs 1: [11335.068119] NMI backtrace for cpu 1 [11335.068268] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11335.068271] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11335.068273] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [11335.068275] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [11335.068278] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [11335.068281] RAX: 0000000000000e12 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11335.068282] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11335.068284] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11335.068285] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11335.068286] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11335.068287] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11335.068289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11335.068290] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11335.068291] Call Trace: [11335.068292] [11335.068293] ? show_regs.cold.9+0x22/0x2f [11335.068294] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [11335.068295] ? nmi_cpu_backtrace_handler+0x15/0x40 [11335.068296] ? nmi_handle+0xbd/0x210 [11335.068297] ? default_do_nmi+0x4b/0x1b0 [11335.068298] ? do_nmi+0x280/0x380 [11335.068299] ? end_repeat_nmi+0x16/0x1f [11335.068300] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [11335.068302] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [11335.068303] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [11335.068304] [11335.068305] do_raw_spin_lock+0x177/0x190 [11335.068306] _raw_spin_lock+0x21/0x30 [11335.068307] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11335.068308] ? do_raw_spin_unlock+0x75/0x190 [11335.068309] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11335.068310] ? do_wait_intr+0xf0/0xf0 [11335.068311] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11335.068312] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11335.068313] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11335.068314] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11335.068315] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11335.068316] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11335.068317] mdc_intent_lock+0x353/0x7e0 [mdc] [11335.068318] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11335.068320] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11335.068321] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11335.068322] lmv_intent_open+0x338/0x1150 [lmv] [11335.068323] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11335.068324] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11335.068325] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11335.068326] ll_intent_lock+0xc5/0xaa0 [lustre] [11335.068327] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11335.068329] ll_lookup_it+0xe0e/0x2270 [lustre] [11335.068330] ? do_raw_spin_unlock+0x75/0x190 [11335.068330] ? _raw_spin_unlock+0x12/0x30 [11335.068332] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11335.068333] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11335.068334] ll_atomic_open+0x36a/0xd70 [lustre] [11335.068335] lookup_open+0x356/0xb40 [11335.068335] do_last+0x313/0xfc0 [11335.068336] ? nd_jump_root+0xe5/0x160 [11335.068337] ? path_init+0x437/0x520 [11335.068338] path_openat+0xf7/0x500 [11335.068339] ? do_raw_spin_unlock+0x75/0x190 [11335.068340] do_filp_open+0x99/0x140 [11335.068341] ? getname_flags+0x6e/0x330 [11335.068342] ? __check_object_size+0xff/0x256 [11335.068343] ? do_raw_spin_unlock+0x75/0x190 [11335.068344] ? _raw_spin_unlock+0x12/0x30 [11335.068345] do_sys_openat2+0x2b4/0x410 [11335.068346] do_sys_open+0x73/0xa0 [11335.068348] __x64_sys_openat+0x24/0x30 [11335.068349] do_syscall_64+0xc1/0x440 [11335.068350] entry_SYSCALL_64_after_hwframe+0x49/0xae [11335.068351] RIP: 0033:0x7f53425a5332 [11335.068352] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11335.068354] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11335.068356] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11335.068358] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11335.068359] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11335.068360] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11335.068362] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11335.076497] NMI backtrace for cpu 2 [11335.499276] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11335.504966] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11335.510803] Call Trace: [11335.514111] [11335.517568] ? dump_stack+0xbb/0x10e [11335.519244] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11335.521611] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11335.523823] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11335.526556] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11335.530358] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11335.534422] ? rcu_dump_cpu_stacks+0x11b/0x171 [11335.538584] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11335.541392] ? update_process_times+0x6d/0xc0 [11335.545576] ? tick_sched_handle.isra.18+0x27/0x90 [11335.549638] ? tick_sched_timer+0x53/0xb0 [11335.555450] ? __hrtimer_run_queues+0x1c2/0x4d0 [11335.559281] ? tick_sched_handle.isra.18+0x90/0x90 [11335.560925] ? hrtimer_interrupt+0x128/0x2c0 [11335.567785] ? smp_apic_timer_interrupt+0xa6/0x270 [11335.569708] ? apic_timer_interrupt+0xf/0x20 [11335.574403] [11335.575295] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11335.579387] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11335.587079] ? do_raw_spin_lock+0x177/0x190 [11335.591101] ? _raw_spin_lock+0x21/0x30 [11335.595083] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11335.599908] ? obd_mod_free+0x46/0x140 [ptlrpc] [11335.604476] ? mdc_commit_open+0x91/0xb0 [mdc] [11335.607600] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11335.609813] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11335.614287] ? after_reply+0xe2c/0x1780 [ptlrpc] [11335.618279] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11335.623577] ? __next_timer_interrupt+0x160/0x160 [11335.625027] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11335.627961] ? do_wait_intr+0xf0/0xf0 [11335.629531] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11335.632561] ? kthread+0x1d1/0x200 [11335.634273] ? set_kthread_struct+0x70/0x70 [11335.636639] ? ret_from_fork+0x1f/0x30 [11340.061340] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [11340.069902] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11340.122142] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11340.136888] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11340.147881] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11340.154022] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11340.177681] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11340.188884] RAX: 000000000000788c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11340.201592] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11340.210557] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11340.217841] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11340.225134] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11340.235852] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11340.248759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11340.254070] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11340.265491] Call Trace: [11340.266546] [11340.268871] ? show_regs.cold.9+0x22/0x2f [11340.275155] ? watchdog_timer_fn.cold.6+0x62/0x10a [11340.281931] ? __hrtimer_run_queues+0x1c2/0x4d0 [11340.289661] ? watchdog+0x40/0x40 [11340.292554] ? hrtimer_interrupt+0x128/0x2c0 [11340.296383] ? smp_apic_timer_interrupt+0xa6/0x270 [11340.299660] ? apic_timer_interrupt+0xf/0x20 [11340.303573] [11340.305483] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11340.314356] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11340.319499] do_raw_spin_lock+0x177/0x190 [11340.327305] _raw_spin_lock+0x21/0x30 [11340.334811] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11340.338358] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11340.347832] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11340.358135] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11340.365585] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11340.371721] ? woken_wake_function+0x30/0x30 [11340.377635] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11340.397788] kthread+0x1d1/0x200 [11340.404911] ? set_kthread_struct+0x70/0x70 [11340.409601] ret_from_fork+0x1f/0x30 [11348.052031] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [11348.055332] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11348.075098] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11348.081896] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11348.086192] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11348.088938] RIP: 0010:native_safe_halt+0x2e/0x50 [11348.091019] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11348.099110] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11348.105748] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11348.109247] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [11348.111777] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11348.116031] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11348.118561] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11348.121290] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11348.124403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11348.126607] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11348.129337] Call Trace: [11348.130491] [11348.131159] ? show_regs.cold.9+0x22/0x2f [11348.132511] ? watchdog_timer_fn.cold.6+0x62/0x10a [11348.134531] ? __hrtimer_run_queues+0x1c2/0x4d0 [11348.136260] ? watchdog+0x40/0x40 [11348.137505] ? hrtimer_interrupt+0x128/0x2c0 [11348.139494] ? smp_apic_timer_interrupt+0xa6/0x270 [11348.143528] ? apic_timer_interrupt+0xf/0x20 [11348.146222] [11348.149380] ? native_safe_halt+0x2e/0x50 [11348.152349] kvm_wait+0xcd/0xe0 [11348.153380] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11348.155274] ? newidle_balance.isra.93+0x207/0x580 [11348.157778] do_raw_spin_lock+0x177/0x190 [11348.159283] _raw_spin_lock+0x21/0x30 [11348.160888] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11348.164043] process_one_work+0x2c8/0x700 [11348.165969] worker_thread+0x296/0x6e0 [11348.167856] ? rescuer_thread+0x570/0x570 [11348.170238] kthread+0x1d1/0x200 [11348.171495] ? set_kthread_struct+0x70/0x70 [11348.176333] ret_from_fork+0x1f/0x30 [11354.081339] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8716s! [11354.092637] Showing busy workqueues and worker pools: [11354.098453] workqueue events: flags=0x0 [11354.101916] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11354.108319] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11354.117236] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11354.126665] pending: cpu_latency_work [ptlrpc] [11354.136181] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11354.145880] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11354.153185] workqueue events_long: flags=0x0 [11354.164979] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11354.173288] pending: lu_objects_flush [obdclass] [11354.175231] workqueue events_unbound: flags=0x2 [11354.187944] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11354.195729] pending: flush_memcg_stats_dwork, idle_cull_fn [11354.208146] workqueue events_power_efficient: flags=0x80 [11354.210095] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11354.218574] pending: check_lifetime, do_cache_clean [11354.224164] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11354.232013] pending: neigh_periodic_work, neigh_periodic_work [11354.237938] workqueue mm_percpu_wq: flags=0x8 [11354.242226] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11354.249394] pending: vmstat_update, lru_add_drain_per_cpu [11354.258577] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11354.273110] pending: vmstat_update, lru_add_drain_per_cpu [11354.274976] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11354.283607] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11354.296086] workqueue xprtiod: flags=0xa [11354.301636] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11354.310884] pending: xprt_autoclose, xs_error_handle [11354.314755] workqueue ipv6_addrconf: flags=0x40008 [11354.320679] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11354.329614] pending: addrconf_verify_work [11354.336874] workqueue knbd0-recv: flags=0x1a [11354.339251] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11354.345460] in-flight: 651:recv_work [11354.353652] workqueue ptlrpc_pinger: flags=0x2 [11354.363777] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11354.369936] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11354.381817] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8717s workers=4 idle: 15979 31824 8 [11354.392257] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11344s workers=2 idle: 654 [11360.056160] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11360.062994] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11360.067723] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11360.067810] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11360.067817] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11360.067820] RIP: 0010:native_safe_halt+0x2e/0x50 [11360.067833] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11360.067837] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11360.067842] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11360.067845] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11360.067847] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11360.067849] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11360.067852] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11360.067855] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11360.067859] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11360.067861] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11360.067869] Call Trace: [11360.067874] [11360.067890] ? show_regs.cold.9+0x22/0x2f [11360.067899] ? watchdog_timer_fn.cold.6+0x62/0x10a [11360.067905] ? __hrtimer_run_queues+0x1c2/0x4d0 [11360.067909] ? watchdog+0x40/0x40 [11360.067913] ? hrtimer_interrupt+0x128/0x2c0 [11360.067917] ? smp_apic_timer_interrupt+0xa6/0x270 [11360.067924] ? apic_timer_interrupt+0xf/0x20 [11360.067928] [11360.067931] ? native_safe_halt+0x2e/0x50 [11360.067935] kvm_wait+0xcd/0xe0 [11360.067943] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11360.067951] do_raw_spin_lock+0x177/0x190 [11360.067955] _raw_spin_lock+0x21/0x30 [11360.067959] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11360.068699] ? do_raw_spin_unlock+0x75/0x190 [11360.068707] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11360.068918] ? do_wait_intr+0xf0/0xf0 [11360.068929] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11360.069126] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11360.080374] Modules linked in: lustre(O) [11360.107038] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11360.112968] osp(O) [11360.121383] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11360.121513] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11360.121554] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11360.121740] mdc_intent_lock+0x353/0x7e0 [mdc] [11360.121775] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11360.121867] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11360.125357] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11360.125438] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11360.125446] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11360.125449] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11360.125463] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11360.125467] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11360.125472] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11360.125475] RDX: 0000000000005ac9 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11360.125478] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11360.125480] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11360.125483] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11360.125486] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11360.125489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11360.125492] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11360.125500] Call Trace: [11360.125506] [11360.125512] ? show_regs.cold.9+0x22/0x2f [11360.125521] ? watchdog_timer_fn.cold.6+0x62/0x10a [11360.125528] ? __hrtimer_run_queues+0x1c2/0x4d0 [11360.125532] ? watchdog+0x40/0x40 [11360.125536] ? hrtimer_interrupt+0x128/0x2c0 [11360.125540] ? smp_apic_timer_interrupt+0xa6/0x270 [11360.125548] ? apic_timer_interrupt+0xf/0x20 [11360.125553] [11360.125556] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11360.125561] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11360.125566] do_raw_spin_lock+0x177/0x190 [11360.125570] _raw_spin_lock+0x21/0x30 [11360.125576] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11360.125890] obd_mod_free+0x46/0x140 [ptlrpc] [11360.126113] mdc_commit_open+0x91/0xb0 [mdc] [11360.126154] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11360.126354] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11360.126564] after_reply+0xe2c/0x1780 [ptlrpc] [11360.128653] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11360.128863] ? __next_timer_interrupt+0x160/0x160 [11360.128873] ptlrpcd+0xa82/0xe40 [ptlrpc] [11360.150120] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11360.158301] ? do_wait_intr+0xf0/0xf0 [11360.164296] lmv_intent_open+0x338/0x1150 [lmv] [11360.166184] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11360.173844] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11360.173998] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11360.174082] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11360.174110] ll_intent_lock+0xc5/0xaa0 [lustre] [11360.174194] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11360.174526] ll_lookup_it+0xe0e/0x2270 [lustre] [11360.175356] ? do_raw_spin_unlock+0x75/0x190 [11360.175365] ? _raw_spin_unlock+0x12/0x30 [11360.175372] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11360.175400] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11360.175420] ll_atomic_open+0x36a/0xd70 [lustre] [11360.175511] lookup_open+0x356/0xb40 [11360.175522] do_last+0x313/0xfc0 [11360.175528] ? nd_jump_root+0xe5/0x160 [11360.175533] ? path_init+0x437/0x520 [11360.175537] path_openat+0xf7/0x500 [11360.175543] ? do_raw_spin_unlock+0x75/0x190 [11360.175549] do_filp_open+0x99/0x140 [11360.175555] ? getname_flags+0x6e/0x330 [11360.175560] ? __check_object_size+0xff/0x256 [11360.175567] ? do_raw_spin_unlock+0x75/0x190 [11360.175571] ? _raw_spin_unlock+0x12/0x30 [11360.175576] do_sys_openat2+0x2b4/0x410 [11360.175581] do_sys_open+0x73/0xa0 [11360.176140] __x64_sys_openat+0x24/0x30 [11360.176148] do_syscall_64+0xc1/0x440 [11360.176155] entry_SYSCALL_64_after_hwframe+0x49/0xae [11360.176161] RIP: 0033:0x7f53425a5332 [11360.176307] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11360.176313] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11360.176318] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11360.176320] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11360.176323] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11360.176325] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11360.176327] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11360.740124] kthread+0x1d1/0x200 [11360.744443] ? set_kthread_struct+0x70/0x70 [11360.752565] ret_from_fork+0x1f/0x30 [11368.058793] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [11368.066471] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11368.088713] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11368.097185] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11368.101803] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [11368.104907] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [11368.111920] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11368.114727] RAX: 0000000000007177 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11368.117054] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11368.119711] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11368.122689] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11368.125874] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11368.128536] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11368.131686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11368.133775] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11368.136837] Call Trace: [11368.137869] [11368.138817] ? show_regs.cold.9+0x22/0x2f [11368.140205] ? watchdog_timer_fn.cold.6+0x62/0x10a [11368.142114] ? __hrtimer_run_queues+0x1c2/0x4d0 [11368.143995] ? watchdog+0x40/0x40 [11368.145025] ? hrtimer_interrupt+0x128/0x2c0 [11368.146888] ? smp_apic_timer_interrupt+0xa6/0x270 [11368.153336] ? apic_timer_interrupt+0xf/0x20 [11368.155351] [11368.156652] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [11368.159592] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11368.161559] do_raw_spin_lock+0x177/0x190 [11368.163263] _raw_spin_lock+0x21/0x30 [11368.164936] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11368.167643] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11368.170032] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11368.172847] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11368.175035] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11368.177182] ? woken_wake_function+0x30/0x30 [11368.179059] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11368.182085] kthread+0x1d1/0x200 [11368.183335] ? set_kthread_struct+0x70/0x70 [11368.185415] ret_from_fork+0x1f/0x30 [11380.056471] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [11380.074812] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11380.130842] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11380.137894] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11380.149755] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11380.152535] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11380.155408] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11380.171866] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11380.176789] RAX: 0000000000005f81 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11380.188425] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11380.198255] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11380.208402] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11380.213949] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11380.221638] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11380.227207] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11380.237112] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11380.241056] Call Trace: [11380.242177] [11380.247551] ? show_regs.cold.9+0x22/0x2f [11380.250776] ? watchdog_timer_fn.cold.6+0x62/0x10a [11380.252888] ? __hrtimer_run_queues+0x1c2/0x4d0 [11380.256419] ? watchdog+0x40/0x40 [11380.261671] ? hrtimer_interrupt+0x128/0x2c0 [11380.263170] ? smp_apic_timer_interrupt+0xa6/0x270 [11380.266795] ? apic_timer_interrupt+0xf/0x20 [11380.268818] [11380.269760] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11380.271874] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11380.274946] ? newidle_balance.isra.93+0x207/0x580 [11380.280599] do_raw_spin_lock+0x177/0x190 [11380.285501] _raw_spin_lock+0x21/0x30 [11380.288787] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11380.293247] process_one_work+0x2c8/0x700 [11380.296935] worker_thread+0x296/0x6e0 [11380.302983] ? rescuer_thread+0x570/0x570 [11380.306438] kthread+0x1d1/0x200 [11380.309005] ? set_kthread_struct+0x70/0x70 [11380.312676] ret_from_fork+0x1f/0x30 [11384.799113] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8747s! [11384.801773] Showing busy workqueues and worker pools: [11384.806054] workqueue events: flags=0x0 [11384.807793] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11384.814292] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11384.828713] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11384.834740] pending: cpu_latency_work [ptlrpc] [11384.837758] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11384.841080] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11384.849896] workqueue events_long: flags=0x0 [11384.851085] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11384.853562] pending: lu_objects_flush [obdclass] [11384.856353] workqueue events_unbound: flags=0x2 [11384.858305] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11384.861219] pending: flush_memcg_stats_dwork, idle_cull_fn [11384.866447] workqueue events_power_efficient: flags=0x80 [11384.869765] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11384.873734] pending: check_lifetime, do_cache_clean [11384.875527] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11384.885771] pending: neigh_periodic_work, neigh_periodic_work [11384.889865] workqueue mm_percpu_wq: flags=0x8 [11384.895241] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11384.900281] pending: vmstat_update, lru_add_drain_per_cpu [11384.903900] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11384.908663] pending: vmstat_update, lru_add_drain_per_cpu [11384.912577] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11384.918418] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11384.921106] workqueue xprtiod: flags=0xa [11384.923588] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11384.927765] pending: xprt_autoclose, xs_error_handle [11384.931789] workqueue ipv6_addrconf: flags=0x40008 [11384.935730] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11384.939035] pending: addrconf_verify_work [11384.941647] workqueue knbd0-recv: flags=0x1a [11384.943643] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11384.953821] in-flight: 651:recv_work [11384.958209] workqueue ptlrpc_pinger: flags=0x2 [11384.959785] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11384.963609] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11384.966648] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8747s workers=4 idle: 15979 31824 8 [11384.974010] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11375s workers=2 idle: 654 [11388.055628] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11388.059232] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11388.064813] Modules linked in: [11388.068625] Modules linked in: [11388.068629] lustre(O) osp(O) [11388.070199] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11388.070273] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11388.070280] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11388.070282] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11388.070295] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11388.070299] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11388.070304] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11388.070306] RDX: 0000000000005198 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11388.070308] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11388.070310] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11388.070312] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11388.070315] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11388.070318] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11388.070320] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11388.070329] Call Trace: [11388.070335] [11388.070341] ? show_regs.cold.9+0x22/0x2f [11388.070351] ? watchdog_timer_fn.cold.6+0x62/0x10a [11388.070358] ? __hrtimer_run_queues+0x1c2/0x4d0 [11388.070362] ? watchdog+0x40/0x40 [11388.070366] ? hrtimer_interrupt+0x128/0x2c0 [11388.070370] ? smp_apic_timer_interrupt+0xa6/0x270 [11388.070377] ? apic_timer_interrupt+0xf/0x20 [11388.070382] [11388.070385] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11388.070390] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11388.070395] do_raw_spin_lock+0x177/0x190 [11388.070401] _raw_spin_lock+0x21/0x30 [11388.070407] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11388.071158] obd_mod_free+0x46/0x140 [ptlrpc] [11388.071369] mdc_commit_open+0x91/0xb0 [mdc] [11388.071403] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11388.071613] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11388.071819] after_reply+0xe2c/0x1780 [ptlrpc] [11388.072024] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11388.072229] ? __next_timer_interrupt+0x160/0x160 [11388.072240] ptlrpcd+0xa82/0xe40 [ptlrpc] [11388.072448] ? do_wait_intr+0xf0/0xf0 [11388.072457] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11388.072667] kthread+0x1d1/0x200 [11388.072673] ? set_kthread_struct+0x70/0x70 [11388.072677] ret_from_fork+0x1f/0x30 [11388.365185] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11388.388685] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11388.391603] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11388.398085] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11388.402863] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11388.412209] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11388.416759] RAX: 0000000000003087 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11388.424790] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11388.429089] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11388.434570] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11388.442685] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11388.449890] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11388.455574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11388.462975] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11388.476152] Call Trace: [11388.477041] [11388.477819] ? show_regs.cold.9+0x22/0x2f [11388.485692] ? watchdog_timer_fn.cold.6+0x62/0x10a [11388.488605] ? __hrtimer_run_queues+0x1c2/0x4d0 [11388.490894] ? watchdog+0x40/0x40 [11388.493844] ? hrtimer_interrupt+0x128/0x2c0 [11388.500005] ? smp_apic_timer_interrupt+0xa6/0x270 [11388.502976] ? apic_timer_interrupt+0xf/0x20 [11388.505670] [11388.506626] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11388.510627] do_raw_spin_lock+0x177/0x190 [11388.514136] _raw_spin_lock+0x21/0x30 [11388.520929] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11388.527573] ? do_raw_spin_unlock+0x75/0x190 [11388.530245] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11388.534002] ? do_wait_intr+0xf0/0xf0 [11388.536029] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11388.540685] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11388.546738] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11388.550043] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11388.553803] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11388.560730] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11388.563844] mdc_intent_lock+0x353/0x7e0 [mdc] [11388.566303] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11388.568744] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11388.577235] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11388.579441] lmv_intent_open+0x338/0x1150 [lmv] [11388.581240] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11388.592302] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11388.594017] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11388.603753] ll_intent_lock+0xc5/0xaa0 [lustre] [11388.607846] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11388.611239] ll_lookup_it+0xe0e/0x2270 [lustre] [11388.615820] ? do_raw_spin_unlock+0x75/0x190 [11388.622072] ? _raw_spin_unlock+0x12/0x30 [11388.624782] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11388.627287] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11388.634376] ll_atomic_open+0x36a/0xd70 [lustre] [11388.640491] lookup_open+0x356/0xb40 [11388.642059] do_last+0x313/0xfc0 [11388.649986] ? nd_jump_root+0xe5/0x160 [11388.651836] ? path_init+0x437/0x520 [11388.654232] path_openat+0xf7/0x500 [11388.655392] ? do_raw_spin_unlock+0x75/0x190 [11388.659616] do_filp_open+0x99/0x140 [11388.665598] ? getname_flags+0x6e/0x330 [11388.676932] ? __check_object_size+0xff/0x256 [11388.679611] ? do_raw_spin_unlock+0x75/0x190 [11388.681900] ? _raw_spin_unlock+0x12/0x30 [11388.683841] do_sys_openat2+0x2b4/0x410 [11388.690809] do_sys_open+0x73/0xa0 [11388.692687] __x64_sys_openat+0x24/0x30 [11388.695312] do_syscall_64+0xc1/0x440 [11388.696938] entry_SYSCALL_64_after_hwframe+0x49/0xae [11388.703406] RIP: 0033:0x7f53425a5332 [11388.706844] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11388.726759] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11388.735308] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11388.743348] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11388.749878] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11388.759266] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11388.765687] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11396.059248] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11396.064483] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11396.131161] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11396.146568] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11396.161276] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11396.169977] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11396.182034] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11396.188419] RAX: 0000000000002282 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11396.191137] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11396.192857] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11396.197492] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11396.204047] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11396.207575] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11396.211585] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11396.213653] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11396.222639] Call Trace: [11396.223533] [11396.235051] ? show_regs.cold.9+0x22/0x2f [11396.237470] ? watchdog_timer_fn.cold.6+0x62/0x10a [11396.240765] ? __hrtimer_run_queues+0x1c2/0x4d0 [11396.249176] ? watchdog+0x40/0x40 [11396.251123] ? hrtimer_interrupt+0x128/0x2c0 [11396.258599] ? smp_apic_timer_interrupt+0xa6/0x270 [11396.260247] ? apic_timer_interrupt+0xf/0x20 [11396.269932] [11396.270715] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11396.272491] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11396.284456] do_raw_spin_lock+0x177/0x190 [11396.289490] _raw_spin_lock+0x21/0x30 [11396.292831] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11396.296939] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11396.299929] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11396.305860] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11396.310837] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11396.316729] ? woken_wake_function+0x30/0x30 [11396.318794] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11396.324622] kthread+0x1d1/0x200 [11396.328902] ? set_kthread_struct+0x70/0x70 [11396.332039] ret_from_fork+0x1f/0x30 [11412.054060] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [11412.058562] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11412.092219] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11412.103066] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11412.111351] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11412.115408] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11412.121127] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11412.132267] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11412.136260] RAX: 0000000000002ee6 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11412.139824] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11412.143467] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11412.147053] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11412.150783] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11412.155316] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11412.162327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11412.170153] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11412.178262] Call Trace: [11412.179255] [11412.182418] ? show_regs.cold.9+0x22/0x2f [11412.187617] ? watchdog_timer_fn.cold.6+0x62/0x10a [11412.191443] ? __hrtimer_run_queues+0x1c2/0x4d0 [11412.195499] ? watchdog+0x40/0x40 [11412.198599] ? hrtimer_interrupt+0x128/0x2c0 [11412.202309] ? smp_apic_timer_interrupt+0xa6/0x270 [11412.208538] ? apic_timer_interrupt+0xf/0x20 [11412.212516] [11412.214409] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11412.218415] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11412.221126] ? newidle_balance.isra.93+0x207/0x580 [11412.223534] do_raw_spin_lock+0x177/0x190 [11412.225636] _raw_spin_lock+0x21/0x30 [11412.227079] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11412.229651] process_one_work+0x2c8/0x700 [11412.231284] worker_thread+0x296/0x6e0 [11412.232972] ? rescuer_thread+0x570/0x570 [11412.234621] kthread+0x1d1/0x200 [11412.236047] ? set_kthread_struct+0x70/0x70 [11412.237410] ret_from_fork+0x1f/0x30 [11415.012976] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8777s! [11415.027065] Showing busy workqueues and worker pools: [11415.028669] workqueue events: flags=0x0 [11415.031135] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11415.034450] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11415.043379] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11415.048691] pending: cpu_latency_work [ptlrpc] [11415.050402] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11415.057432] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11415.067141] workqueue events_long: flags=0x0 [11415.089284] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11415.092570] pending: lu_objects_flush [obdclass] [11415.094609] workqueue events_unbound: flags=0x2 [11415.096240] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11415.100765] pending: flush_memcg_stats_dwork, idle_cull_fn [11415.109481] workqueue events_power_efficient: flags=0x80 [11415.117870] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11415.120533] pending: check_lifetime, do_cache_clean [11415.122664] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11415.124968] pending: neigh_periodic_work, neigh_periodic_work [11415.127824] workqueue mm_percpu_wq: flags=0x8 [11415.129645] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11415.135082] pending: vmstat_update, lru_add_drain_per_cpu [11415.139080] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11415.143339] pending: vmstat_update, lru_add_drain_per_cpu [11415.146499] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11415.149768] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11415.152236] workqueue xprtiod: flags=0xa [11415.153936] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11415.159900] pending: xprt_autoclose, xs_error_handle [11415.165326] workqueue ipv6_addrconf: flags=0x40008 [11415.167323] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11415.172161] pending: addrconf_verify_work [11415.175930] workqueue knbd0-recv: flags=0x1a [11415.181544] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11415.183862] in-flight: 651:recv_work [11415.186216] workqueue ptlrpc_pinger: flags=0x2 [11415.189056] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11415.197262] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11415.199345] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8778s workers=4 idle: 15979 31824 8 [11415.203678] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11405s workers=2 idle: 654 [11416.056272] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [11416.059931] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [11416.080925] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11416.081009] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11416.081017] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11416.081020] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11416.081034] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11416.081039] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11416.081044] RAX: 0000000000004584 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11416.081046] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11416.081049] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11416.081051] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11416.081054] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11416.081057] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11416.081060] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11416.081063] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11416.083203] Call Trace: [11416.090751] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11416.090833] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11416.090841] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11416.090844] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11416.090858] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11416.090863] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11416.090868] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11416.090871] RDX: 0000000000002dfa RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11416.090873] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11416.090876] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11416.090878] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11416.090881] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11416.090884] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11416.090887] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11416.090895] Call Trace: [11416.090900] [11416.090905] ? show_regs.cold.9+0x22/0x2f [11416.090914] ? watchdog_timer_fn.cold.6+0x62/0x10a [11416.090920] ? __hrtimer_run_queues+0x1c2/0x4d0 [11416.090925] ? watchdog+0x40/0x40 [11416.090930] ? hrtimer_interrupt+0x128/0x2c0 [11416.090934] ? smp_apic_timer_interrupt+0xa6/0x270 [11416.090942] ? apic_timer_interrupt+0xf/0x20 [11416.090947] [11416.090950] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11416.090955] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11416.090960] do_raw_spin_lock+0x177/0x190 [11416.090965] _raw_spin_lock+0x21/0x30 [11416.090971] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11416.091405] obd_mod_free+0x46/0x140 [ptlrpc] [11416.094189] mdc_commit_open+0x91/0xb0 [mdc] [11416.094294] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11416.094999] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11416.095227] after_reply+0xe2c/0x1780 [ptlrpc] [11416.095520] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11416.096048] ? __next_timer_interrupt+0x160/0x160 [11416.096059] ptlrpcd+0xa82/0xe40 [ptlrpc] [11416.096403] ? do_wait_intr+0xf0/0xf0 [11416.096414] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11416.118200] [11416.118213] ? show_regs.cold.9+0x22/0x2f [11416.118227] ? watchdog_timer_fn.cold.6+0x62/0x10a [11416.118233] ? __hrtimer_run_queues+0x1c2/0x4d0 [11416.126063] kthread+0x1d1/0x200 [11416.126084] ? set_kthread_struct+0x70/0x70 [11416.126089] ret_from_fork+0x1f/0x30 [11416.377096] ? watchdog+0x40/0x40 [11416.378172] ? hrtimer_interrupt+0x128/0x2c0 [11416.379759] ? smp_apic_timer_interrupt+0xa6/0x270 [11416.381316] ? apic_timer_interrupt+0xf/0x20 [11416.382931] [11416.383762] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11416.385932] do_raw_spin_lock+0x177/0x190 [11416.387180] _raw_spin_lock+0x21/0x30 [11416.388199] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11416.389687] ? do_raw_spin_unlock+0x75/0x190 [11416.391142] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11416.393025] ? do_wait_intr+0xf0/0xf0 [11416.394141] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11416.395890] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11416.397572] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11416.399662] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11416.401999] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11416.404303] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11416.406947] mdc_intent_lock+0x353/0x7e0 [mdc] [11416.408911] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11416.413279] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11416.416921] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11416.419676] lmv_intent_open+0x338/0x1150 [lmv] [11416.421824] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11416.424240] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11416.426958] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11416.428980] ll_intent_lock+0xc5/0xaa0 [lustre] [11416.431841] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11416.434768] ll_lookup_it+0xe0e/0x2270 [lustre] [11416.437098] ? do_raw_spin_unlock+0x75/0x190 [11416.439161] ? _raw_spin_unlock+0x12/0x30 [11416.441158] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11416.443879] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11416.446038] ll_atomic_open+0x36a/0xd70 [lustre] [11416.448307] lookup_open+0x356/0xb40 [11416.457392] do_last+0x313/0xfc0 [11416.459442] ? nd_jump_root+0xe5/0x160 [11416.461243] ? path_init+0x437/0x520 [11416.463060] path_openat+0xf7/0x500 [11416.466137] ? do_raw_spin_unlock+0x75/0x190 [11416.469255] do_filp_open+0x99/0x140 [11416.470863] ? getname_flags+0x6e/0x330 [11416.472214] ? __check_object_size+0xff/0x256 [11416.475103] ? do_raw_spin_unlock+0x75/0x190 [11416.477194] ? _raw_spin_unlock+0x12/0x30 [11416.478867] do_sys_openat2+0x2b4/0x410 [11416.480886] do_sys_open+0x73/0xa0 [11416.482301] __x64_sys_openat+0x24/0x30 [11416.484025] do_syscall_64+0xc1/0x440 [11416.485909] entry_SYSCALL_64_after_hwframe+0x49/0xae [11416.488067] RIP: 0033:0x7f53425a5332 [11416.489940] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11416.498998] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11416.503234] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11416.506372] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11416.516090] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11416.520144] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11416.529110] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11424.058060] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11424.064826] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11424.087239] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11424.096274] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11424.101503] RIP: 0010:native_safe_halt+0x2e/0x50 [11424.103458] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11424.115630] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11424.118431] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11424.120877] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11424.123638] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11424.126066] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11424.128444] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11424.130932] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11424.133353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11424.135165] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11424.137929] Call Trace: [11424.138780] [11424.139351] ? show_regs.cold.9+0x22/0x2f [11424.140559] ? watchdog_timer_fn.cold.6+0x62/0x10a [11424.142448] ? __hrtimer_run_queues+0x1c2/0x4d0 [11424.144112] ? watchdog+0x40/0x40 [11424.145524] ? hrtimer_interrupt+0x128/0x2c0 [11424.147043] ? smp_apic_timer_interrupt+0xa6/0x270 [11424.148902] ? apic_timer_interrupt+0xf/0x20 [11424.153920] [11424.154958] ? native_safe_halt+0x2e/0x50 [11424.156301] kvm_wait+0xcd/0xe0 [11424.158028] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11424.160768] do_raw_spin_lock+0x177/0x190 [11424.166798] _raw_spin_lock+0x21/0x30 [11424.171422] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11424.175479] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11424.183305] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11424.195822] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11424.203573] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11424.209413] ? woken_wake_function+0x30/0x30 [11424.217619] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11424.227132] kthread+0x1d1/0x200 [11424.229795] ? set_kthread_struct+0x70/0x70 [11424.234286] ret_from_fork+0x1f/0x30 [11440.055073] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [11440.069671] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11440.122292] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11440.142149] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11440.157787] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11440.163602] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11440.170306] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11440.208408] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11440.219131] RAX: 0000000000007887 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11440.233942] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11440.243240] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11440.256194] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11440.265759] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11440.279616] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11440.296332] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11440.305837] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11440.308570] Call Trace: [11440.315495] [11440.318848] ? show_regs.cold.9+0x22/0x2f [11440.331447] ? watchdog_timer_fn.cold.6+0x62/0x10a [11440.334154] ? __hrtimer_run_queues+0x1c2/0x4d0 [11440.345893] ? watchdog+0x40/0x40 [11440.351081] ? hrtimer_interrupt+0x128/0x2c0 [11440.358080] ? smp_apic_timer_interrupt+0xa6/0x270 [11440.370399] ? apic_timer_interrupt+0xf/0x20 [11440.377664] [11440.384371] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11440.399990] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11440.406069] ? newidle_balance.isra.93+0x207/0x580 [11440.419032] do_raw_spin_lock+0x177/0x190 [11440.423968] _raw_spin_lock+0x21/0x30 [11440.430297] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11440.439501] process_one_work+0x2c8/0x700 [11440.443311] worker_thread+0x296/0x6e0 [11440.448296] ? rescuer_thread+0x570/0x570 [11440.453681] kthread+0x1d1/0x200 [11440.457221] ? set_kthread_struct+0x70/0x70 [11440.459992] ret_from_fork+0x1f/0x30 [11444.055009] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [11444.058067] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11444.058522] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [11444.089583] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11444.089601] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11444.089605] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11444.089619] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11444.089623] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11444.089629] RAX: 0000000000001208 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11444.089631] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11444.089633] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11444.089636] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11444.089638] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11444.089641] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11444.089644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11444.089646] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11444.089655] Call Trace: [11444.089661] [11444.089666] ? show_regs.cold.9+0x22/0x2f [11444.089675] ? watchdog_timer_fn.cold.6+0x62/0x10a [11444.089682] ? __hrtimer_run_queues+0x1c2/0x4d0 [11444.089686] ? watchdog+0x40/0x40 [11444.089691] ? hrtimer_interrupt+0x128/0x2c0 [11444.089695] ? smp_apic_timer_interrupt+0xa6/0x270 [11444.089703] ? apic_timer_interrupt+0xf/0x20 [11444.089707] [11444.089710] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11444.089718] do_raw_spin_lock+0x177/0x190 [11444.089723] _raw_spin_lock+0x21/0x30 [11444.089728] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11444.090050] ? do_raw_spin_unlock+0x75/0x190 [11444.090057] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11444.102236] Modules linked in: [11444.110086] ? do_wait_intr+0xf0/0xf0 [11444.118800] lustre(O) [11444.125860] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11444.131758] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11444.131971] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11444.131978] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11444.131981] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11444.131996] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11444.132000] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11444.132005] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11444.132007] RDX: 0000000000003f77 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11444.132009] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11444.132011] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11444.132013] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11444.132016] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11444.132019] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11444.132021] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11444.132599] Call Trace: [11444.137766] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11444.138062] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11444.138594] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11444.138704] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11444.138739] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11444.138768] mdc_intent_lock+0x353/0x7e0 [mdc] [11444.138794] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11444.138878] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11444.139469] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11444.139501] lmv_intent_open+0x338/0x1150 [lmv] [11444.139534] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11444.139627] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11444.139710] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11444.139732] ll_intent_lock+0xc5/0xaa0 [lustre] [11444.139817] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11444.139896] ll_lookup_it+0xe0e/0x2270 [lustre] [11444.139976] ? do_raw_spin_unlock+0x75/0x190 [11444.139984] ? _raw_spin_unlock+0x12/0x30 [11444.139991] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11444.140016] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11444.140035] ll_atomic_open+0x36a/0xd70 [lustre] [11444.145529] [11444.149354] lookup_open+0x356/0xb40 [11444.149376] do_last+0x313/0xfc0 [11444.149381] ? nd_jump_root+0xe5/0x160 [11444.149387] ? path_init+0x437/0x520 [11444.149392] path_openat+0xf7/0x500 [11444.149397] ? do_raw_spin_unlock+0x75/0x190 [11444.155680] ? show_regs.cold.9+0x22/0x2f [11444.155703] ? watchdog_timer_fn.cold.6+0x62/0x10a [11444.155709] ? __hrtimer_run_queues+0x1c2/0x4d0 [11444.155713] ? watchdog+0x40/0x40 [11444.155717] ? hrtimer_interrupt+0x128/0x2c0 [11444.155721] ? smp_apic_timer_interrupt+0xa6/0x270 [11444.155728] ? apic_timer_interrupt+0xf/0x20 [11444.155733] [11444.155737] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11444.155744] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11444.155748] do_raw_spin_lock+0x177/0x190 [11444.155754] _raw_spin_lock+0x21/0x30 [11444.155761] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11444.156151] obd_mod_free+0x46/0x140 [ptlrpc] [11444.156416] mdc_commit_open+0x91/0xb0 [mdc] [11444.156525] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11444.159874] do_filp_open+0x99/0x140 [11444.159895] ? getname_flags+0x6e/0x330 [11444.159901] ? __check_object_size+0xff/0x256 [11444.159909] ? do_raw_spin_unlock+0x75/0x190 [11444.159914] ? _raw_spin_unlock+0x12/0x30 [11444.159921] do_sys_openat2+0x2b4/0x410 [11444.159926] do_sys_open+0x73/0xa0 [11444.159931] __x64_sys_openat+0x24/0x30 [11444.159935] do_syscall_64+0xc1/0x440 [11444.159942] entry_SYSCALL_64_after_hwframe+0x49/0xae [11444.159949] RIP: 0033:0x7f53425a5332 [11444.159958] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11444.159962] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11444.159967] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11444.159969] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11444.159971] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11444.159973] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11444.159975] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11444.481845] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11444.484781] after_reply+0xe2c/0x1780 [ptlrpc] [11444.486747] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11444.489824] ? __next_timer_interrupt+0x160/0x160 [11444.492774] ptlrpcd+0xa82/0xe40 [ptlrpc] [11444.494739] ? do_wait_intr+0xf0/0xf0 [11444.496240] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11444.498725] kthread+0x1d1/0x200 [11444.499928] ? set_kthread_struct+0x70/0x70 [11444.501833] ret_from_fork+0x1f/0x30 [11445.216504] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8808s! [11445.221221] Showing busy workqueues and worker pools: [11445.223743] workqueue events: flags=0x0 [11445.225365] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11445.228018] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11445.233588] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11445.236220] pending: cpu_latency_work [ptlrpc] [11445.238232] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11445.240895] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11445.244008] workqueue events_long: flags=0x0 [11445.245740] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11445.248367] pending: lu_objects_flush [obdclass] [11445.250686] workqueue events_unbound: flags=0x2 [11445.252393] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11445.255026] pending: flush_memcg_stats_dwork, idle_cull_fn [11445.258089] workqueue events_power_efficient: flags=0x80 [11445.260497] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11445.263361] pending: check_lifetime, do_cache_clean [11445.265683] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11445.269062] pending: neigh_periodic_work, neigh_periodic_work [11445.271649] workqueue mm_percpu_wq: flags=0x8 [11445.273643] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11445.276759] pending: vmstat_update, lru_add_drain_per_cpu [11445.279041] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11445.282001] pending: vmstat_update, lru_add_drain_per_cpu [11445.284665] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11445.287861] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11445.290405] workqueue xprtiod: flags=0xa [11445.292062] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11445.294457] pending: xprt_autoclose, xs_error_handle [11445.296809] workqueue ipv6_addrconf: flags=0x40008 [11445.298761] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11445.301578] pending: addrconf_verify_work [11445.303370] workqueue knbd0-recv: flags=0x1a [11445.305186] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11445.307996] in-flight: 651:recv_work [11445.309904] workqueue ptlrpc_pinger: flags=0x2 [11445.311790] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11445.314307] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11445.316476] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8808s workers=4 idle: 15979 31824 8 [11445.319944] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11435s workers=2 idle: 654 [11452.058336] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11452.062633] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11452.091229] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11452.097211] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11452.101533] RIP: 0010:native_safe_halt+0x2e/0x50 [11452.103494] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11452.112950] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11452.116664] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11452.119451] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11452.123191] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11452.127732] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11452.131892] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11452.137427] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11452.141334] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11452.144580] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11452.149455] Call Trace: [11452.151402] [11452.152441] ? show_regs.cold.9+0x22/0x2f [11452.153713] ? watchdog_timer_fn.cold.6+0x62/0x10a [11452.156207] ? __hrtimer_run_queues+0x1c2/0x4d0 [11452.157944] ? watchdog+0x40/0x40 [11452.160370] ? hrtimer_interrupt+0x128/0x2c0 [11452.162959] ? smp_apic_timer_interrupt+0xa6/0x270 [11452.164821] ? apic_timer_interrupt+0xf/0x20 [11452.166458] [11452.167344] ? native_safe_halt+0x2e/0x50 [11452.168884] kvm_wait+0xcd/0xe0 [11452.170200] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11452.172368] do_raw_spin_lock+0x177/0x190 [11452.173898] _raw_spin_lock+0x21/0x30 [11452.175280] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11452.178234] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11452.180239] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11452.182529] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11452.184806] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11452.187109] ? woken_wake_function+0x30/0x30 [11452.188714] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11452.191391] kthread+0x1d1/0x200 [11452.192678] ? set_kthread_struct+0x70/0x70 [11452.194340] ret_from_fork+0x1f/0x30 [11472.052047] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [11472.054934] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11472.057029] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [11472.057759] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [11472.057766] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11472.057840] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11472.057845] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11472.057849] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11472.057863] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11472.057867] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11472.057872] RAX: 000000000000790f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11472.057874] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11472.057876] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11472.057878] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11472.057880] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11472.057883] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11472.057886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11472.057888] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11472.057897] Call Trace: [11472.057903] [11472.057968] ? show_regs.cold.9+0x22/0x2f [11472.057979] ? watchdog_timer_fn.cold.6+0x62/0x10a [11472.057986] ? __hrtimer_run_queues+0x1c2/0x4d0 [11472.057991] ? watchdog+0x40/0x40 [11472.057995] ? hrtimer_interrupt+0x128/0x2c0 [11472.057999] ? smp_apic_timer_interrupt+0xa6/0x270 [11472.058007] ? apic_timer_interrupt+0xf/0x20 [11472.058012] [11472.058015] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11472.058022] do_raw_spin_lock+0x177/0x190 [11472.058028] _raw_spin_lock+0x21/0x30 [11472.058032] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11472.058367] ? do_raw_spin_unlock+0x75/0x190 [11472.058374] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11472.058674] ? do_wait_intr+0xf0/0xf0 [11472.058684] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11472.058886] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11472.059674] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11472.059896] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11472.061887] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11472.062394] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11472.062426] mdc_intent_lock+0x353/0x7e0 [mdc] [11472.062455] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11472.063182] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11472.063896] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11472.065857] lmv_intent_open+0x338/0x1150 [lmv] [11472.065889] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11472.068109] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11472.068194] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11472.068226] ll_intent_lock+0xc5/0xaa0 [lustre] [11472.068309] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11472.068390] ll_lookup_it+0xe0e/0x2270 [lustre] [11472.068470] ? do_raw_spin_unlock+0x75/0x190 [11472.068478] ? _raw_spin_unlock+0x12/0x30 [11472.068485] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11472.068512] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11472.068531] ll_atomic_open+0x36a/0xd70 [lustre] [11472.068619] lookup_open+0x356/0xb40 [11472.071060] do_last+0x313/0xfc0 [11472.071072] ? nd_jump_root+0xe5/0x160 [11472.071077] ? path_init+0x437/0x520 [11472.071082] path_openat+0xf7/0x500 [11472.071086] ? do_raw_spin_unlock+0x75/0x190 [11472.071093] do_filp_open+0x99/0x140 [11472.071099] ? getname_flags+0x6e/0x330 [11472.071104] ? __check_object_size+0xff/0x256 [11472.071112] ? do_raw_spin_unlock+0x75/0x190 [11472.071116] ? _raw_spin_unlock+0x12/0x30 [11472.071123] do_sys_openat2+0x2b4/0x410 [11472.071128] do_sys_open+0x73/0xa0 [11472.071133] __x64_sys_openat+0x24/0x30 [11472.071138] do_syscall_64+0xc1/0x440 [11472.071144] entry_SYSCALL_64_after_hwframe+0x49/0xae [11472.071151] RIP: 0033:0x7f53425a5332 [11472.071160] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11472.071164] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11472.071169] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11472.071172] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11472.071174] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11472.071177] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11472.071179] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11472.076326] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11472.079185] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11472.079265] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11472.079273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11472.079276] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11472.079290] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11472.079294] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11472.079299] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11472.079301] RDX: 0000000000007277 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11472.079303] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11472.079305] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11472.079308] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11472.079311] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11472.079314] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11472.079316] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11472.079325] Call Trace: [11472.079330] [11472.079337] ? show_regs.cold.9+0x22/0x2f [11472.079347] ? watchdog_timer_fn.cold.6+0x62/0x10a [11472.079354] ? __hrtimer_run_queues+0x1c2/0x4d0 [11472.079359] ? watchdog+0x40/0x40 [11472.079363] ? hrtimer_interrupt+0x128/0x2c0 [11472.079368] ? smp_apic_timer_interrupt+0xa6/0x270 [11472.079376] ? apic_timer_interrupt+0xf/0x20 [11472.079381] [11472.079386] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11472.083892] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11472.083907] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11472.084177] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11472.084189] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11472.084193] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11472.084198] RAX: 0000000000002986 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11472.084200] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11472.084202] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11472.084204] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11472.084206] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11472.084209] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11472.084211] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11472.084214] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11472.084223] Call Trace: [11472.084229] [11472.084234] ? show_regs.cold.9+0x22/0x2f [11472.084245] ? watchdog_timer_fn.cold.6+0x62/0x10a [11472.084251] ? __hrtimer_run_queues+0x1c2/0x4d0 [11472.084255] ? watchdog+0x40/0x40 [11472.084260] ? hrtimer_interrupt+0x128/0x2c0 [11472.084264] ? smp_apic_timer_interrupt+0xa6/0x270 [11472.084273] ? apic_timer_interrupt+0xf/0x20 [11472.084277] [11472.084280] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11472.084285] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11472.084288] ? newidle_balance.isra.93+0x207/0x580 [11472.084294] do_raw_spin_lock+0x177/0x190 [11472.084299] _raw_spin_lock+0x21/0x30 [11472.084303] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11472.123254] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11472.123275] do_raw_spin_lock+0x177/0x190 [11472.123282] _raw_spin_lock+0x21/0x30 [11472.123292] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11472.123573] obd_mod_free+0x46/0x140 [ptlrpc] [11472.124053] mdc_commit_open+0x91/0xb0 [mdc] [11472.124090] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11472.124836] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11472.124973] after_reply+0xe2c/0x1780 [ptlrpc] [11472.125149] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11472.125355] ? __next_timer_interrupt+0x160/0x160 [11472.125365] ptlrpcd+0xa82/0xe40 [ptlrpc] [11472.125588] ? do_wait_intr+0xf0/0xf0 [11472.125598] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11472.126164] kthread+0x1d1/0x200 [11472.126171] ? set_kthread_struct+0x70/0x70 [11472.126175] ret_from_fork+0x1f/0x30 [11472.737409] process_one_work+0x2c8/0x700 [11472.738951] worker_thread+0x296/0x6e0 [11472.740263] ? rescuer_thread+0x570/0x570 [11472.741802] kthread+0x1d1/0x200 [11472.742792] ? set_kthread_struct+0x70/0x70 [11472.744556] ret_from_fork+0x1f/0x30 [11475.424516] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8838s! [11475.427416] Showing busy workqueues and worker pools: [11475.446742] workqueue events: flags=0x0 [11475.450192] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11475.457932] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11475.474765] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11475.484903] pending: cpu_latency_work [ptlrpc] [11475.487812] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11475.491661] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11475.496366] workqueue events_long: flags=0x0 [11475.501520] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11475.509972] pending: lu_objects_flush [obdclass] [11475.514659] workqueue events_unbound: flags=0x2 [11475.525768] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11475.534940] pending: flush_memcg_stats_dwork, idle_cull_fn [11475.538851] workqueue events_power_efficient: flags=0x80 [11475.543286] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11475.551558] pending: check_lifetime, do_cache_clean [11475.554548] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11475.558401] pending: neigh_periodic_work, neigh_periodic_work [11475.561885] workqueue mm_percpu_wq: flags=0x8 [11475.564279] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11475.571876] pending: vmstat_update, lru_add_drain_per_cpu [11475.577128] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11475.581881] pending: vmstat_update, lru_add_drain_per_cpu [11475.587246] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11475.593402] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11475.602873] workqueue xprtiod: flags=0xa [11475.604745] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11475.607553] pending: xprt_autoclose, xs_error_handle [11475.610829] workqueue ipv6_addrconf: flags=0x40008 [11475.613335] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11475.615844] pending: addrconf_verify_work [11475.617171] workqueue knbd0-recv: flags=0x1a [11475.619239] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11475.622376] in-flight: 651:recv_work [11475.624406] workqueue ptlrpc_pinger: flags=0x2 [11475.626520] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11475.629113] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11475.631828] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8838s workers=4 idle: 15979 31824 8 [11475.634276] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11466s workers=2 idle: 654 [11480.061804] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11480.068204] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11480.129157] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11480.139330] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11480.156838] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11480.165520] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11480.206629] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11480.211656] RAX: 0000000000007f85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11480.218946] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11480.232107] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11480.242463] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11480.249813] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11480.255097] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11480.259235] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11480.263415] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11480.267744] Call Trace: [11480.271013] [11480.271591] ? show_regs.cold.9+0x22/0x2f [11480.272509] ? watchdog_timer_fn.cold.6+0x62/0x10a [11480.277149] ? __hrtimer_run_queues+0x1c2/0x4d0 [11480.279322] ? watchdog+0x40/0x40 [11480.285183] ? hrtimer_interrupt+0x128/0x2c0 [11480.288204] ? smp_apic_timer_interrupt+0xa6/0x270 [11480.295260] ? apic_timer_interrupt+0xf/0x20 [11480.296875] [11480.297633] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11480.301160] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11480.305990] do_raw_spin_lock+0x177/0x190 [11480.307965] _raw_spin_lock+0x21/0x30 [11480.309993] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11480.317278] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11480.320647] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11480.326094] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11480.328194] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11480.333235] ? woken_wake_function+0x30/0x30 [11480.338991] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11480.344027] kthread+0x1d1/0x200 [11480.345438] ? set_kthread_struct+0x70/0x70 [11480.352308] ret_from_fork+0x1f/0x30 [11500.052028] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [11500.055356] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [11500.055439] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11500.055514] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11500.055521] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11500.055529] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11500.055884] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11500.055895] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11500.055900] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11500.055904] RAX: 0000000000006585 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11500.056091] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11500.056094] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11500.056096] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11500.056098] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11500.056101] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11500.056104] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11500.056106] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11500.056114] Call Trace: [11500.056120] [11500.056125] ? show_regs.cold.9+0x22/0x2f [11500.057286] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [11500.057298] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11500.057372] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11500.057376] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11500.057379] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11500.057390] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11500.057394] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11500.057398] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11500.057400] RDX: 000000000000409d RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11500.057403] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11500.057406] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11500.057408] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11500.057411] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11500.057414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11500.057416] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11500.057423] Call Trace: [11500.059689] [11500.059699] ? show_regs.cold.9+0x22/0x2f [11500.059715] ? watchdog_timer_fn.cold.6+0x62/0x10a [11500.059722] ? __hrtimer_run_queues+0x1c2/0x4d0 [11500.059726] ? watchdog+0x40/0x40 [11500.059733] ? hrtimer_interrupt+0x128/0x2c0 [11500.059738] ? smp_apic_timer_interrupt+0xa6/0x270 [11500.059747] ? apic_timer_interrupt+0xf/0x20 [11500.059751] [11500.059754] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11500.059763] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11500.059768] do_raw_spin_lock+0x177/0x190 [11500.059773] _raw_spin_lock+0x21/0x30 [11500.059783] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11500.060181] obd_mod_free+0x46/0x140 [ptlrpc] [11500.060399] mdc_commit_open+0x91/0xb0 [mdc] [11500.060439] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11500.060843] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [11500.060647] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11500.062112] after_reply+0xe2c/0x1780 [ptlrpc] [11500.062975] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11500.063260] ? __next_timer_interrupt+0x160/0x160 [11500.063270] ptlrpcd+0xa82/0xe40 [ptlrpc] [11500.063641] ? do_wait_intr+0xf0/0xf0 [11500.063652] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11500.063853] kthread+0x1d1/0x200 [11500.063859] ? set_kthread_struct+0x70/0x70 [11500.063863] ret_from_fork+0x1f/0x30 [11500.082106] ? watchdog_timer_fn.cold.6+0x62/0x10a [11500.082131] ? __hrtimer_run_queues+0x1c2/0x4d0 [11500.082148] ? watchdog+0x40/0x40 [11500.082153] ? hrtimer_interrupt+0x128/0x2c0 [11500.082158] ? smp_apic_timer_interrupt+0xa6/0x270 [11500.082166] ? apic_timer_interrupt+0xf/0x20 [11500.082171] [11500.082176] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11500.082183] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11500.082187] ? newidle_balance.isra.93+0x207/0x580 [11500.082196] do_raw_spin_lock+0x177/0x190 [11500.082203] _raw_spin_lock+0x21/0x30 [11500.082209] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11500.082507] process_one_work+0x2c8/0x700 [11500.082515] worker_thread+0x296/0x6e0 [11500.082520] ? rescuer_thread+0x570/0x570 [11500.082524] kthread+0x1d1/0x200 [11500.082529] ? set_kthread_struct+0x70/0x70 [11500.082533] ret_from_fork+0x1f/0x30 [11500.709154] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11500.741370] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11500.749854] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11500.755070] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11500.764718] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11500.783755] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11500.792365] RAX: 0000000000007e0d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11500.802966] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11500.807938] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11500.813093] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11500.821368] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11500.829021] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11500.836621] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11500.840408] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11500.847300] Call Trace: [11500.848793] [11500.849652] ? show_regs.cold.9+0x22/0x2f [11500.856029] ? watchdog_timer_fn.cold.6+0x62/0x10a [11500.859814] ? __hrtimer_run_queues+0x1c2/0x4d0 [11500.863893] ? watchdog+0x40/0x40 [11500.869655] ? hrtimer_interrupt+0x128/0x2c0 [11500.872960] ? smp_apic_timer_interrupt+0xa6/0x270 [11500.881296] ? apic_timer_interrupt+0xf/0x20 [11500.886997] [11500.889905] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11500.893826] do_raw_spin_lock+0x177/0x190 [11500.896297] _raw_spin_lock+0x21/0x30 [11500.898384] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11500.910710] ? do_raw_spin_unlock+0x75/0x190 [11500.912207] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11500.914093] ? do_wait_intr+0xf0/0xf0 [11500.920959] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11500.922883] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11500.933497] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11500.942616] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11500.951383] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11500.962272] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11500.963773] mdc_intent_lock+0x353/0x7e0 [mdc] [11500.968292] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11500.971711] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11500.975715] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11500.978493] lmv_intent_open+0x338/0x1150 [lmv] [11500.982831] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11500.991699] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11500.999149] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11501.000801] ll_intent_lock+0xc5/0xaa0 [lustre] [11501.011317] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11501.023287] ll_lookup_it+0xe0e/0x2270 [lustre] [11501.032407] ? do_raw_spin_unlock+0x75/0x190 [11501.038899] ? _raw_spin_unlock+0x12/0x30 [11501.044837] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11501.047836] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11501.050574] ll_atomic_open+0x36a/0xd70 [lustre] [11501.053563] lookup_open+0x356/0xb40 [11501.055648] do_last+0x313/0xfc0 [11501.056950] ? nd_jump_root+0xe5/0x160 [11501.060639] ? path_init+0x437/0x520 [11501.061806] path_openat+0xf7/0x500 [11501.062829] ? do_raw_spin_unlock+0x75/0x190 [11501.069846] do_filp_open+0x99/0x140 [11501.071773] ? getname_flags+0x6e/0x330 [11501.072934] ? __check_object_size+0xff/0x256 [11501.074937] ? do_raw_spin_unlock+0x75/0x190 [11501.077363] ? _raw_spin_unlock+0x12/0x30 [11501.078452] do_sys_openat2+0x2b4/0x410 [11501.079282] do_sys_open+0x73/0xa0 [11501.079920] __x64_sys_openat+0x24/0x30 [11501.081854] do_syscall_64+0xc1/0x440 [11501.082920] entry_SYSCALL_64_after_hwframe+0x49/0xae [11501.088852] RIP: 0033:0x7f53425a5332 [11501.093676] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11501.109736] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11501.121101] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11501.124828] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11501.140671] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11501.150092] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11501.152738] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11506.144117] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8869s! [11506.149890] Showing busy workqueues and worker pools: [11506.155745] workqueue events: flags=0x0 [11506.159333] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11506.170757] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11506.187562] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11506.196944] pending: cpu_latency_work [ptlrpc] [11506.205648] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11506.212912] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11506.225707] workqueue events_long: flags=0x0 [11506.231351] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11506.235956] pending: lu_objects_flush [obdclass] [11506.242021] workqueue events_unbound: flags=0x2 [11506.246132] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11506.252806] pending: flush_memcg_stats_dwork, idle_cull_fn [11506.254886] workqueue events_power_efficient: flags=0x80 [11506.258407] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11506.261298] pending: check_lifetime, do_cache_clean [11506.264845] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11506.267977] pending: neigh_periodic_work, neigh_periodic_work [11506.272534] workqueue mm_percpu_wq: flags=0x8 [11506.274894] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11506.278157] pending: vmstat_update, lru_add_drain_per_cpu [11506.281765] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11506.284475] pending: vmstat_update, lru_add_drain_per_cpu [11506.287820] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11506.299926] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11506.306901] workqueue xprtiod: flags=0xa [11506.316631] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11506.329791] pending: xprt_autoclose, xs_error_handle [11506.337579] workqueue ipv6_addrconf: flags=0x40008 [11506.342073] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11506.343849] pending: addrconf_verify_work [11506.355327] workqueue knbd0-recv: flags=0x1a [11506.356410] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11506.358378] in-flight: 651:recv_work [11506.363861] workqueue ptlrpc_pinger: flags=0x2 [11506.370853] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11506.376826] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11506.386591] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8869s workers=4 idle: 15979 31824 8 [11506.392890] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11496s workers=2 idle: 654 [11508.058032] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11508.060567] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11508.099215] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11508.107178] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11508.111783] RIP: 0010:native_safe_halt+0x2e/0x50 [11508.113273] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11508.133244] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11508.145905] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11508.156249] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11508.165120] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11508.172560] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11508.186544] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11508.196548] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11508.204156] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11508.207375] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11508.212225] Call Trace: [11508.214421] [11508.216454] ? show_regs.cold.9+0x22/0x2f [11508.219267] ? watchdog_timer_fn.cold.6+0x62/0x10a [11508.223698] ? __hrtimer_run_queues+0x1c2/0x4d0 [11508.227988] ? watchdog+0x40/0x40 [11508.229161] ? hrtimer_interrupt+0x128/0x2c0 [11508.230631] ? smp_apic_timer_interrupt+0xa6/0x270 [11508.236399] ? apic_timer_interrupt+0xf/0x20 [11508.239512] [11508.241441] ? native_safe_halt+0x2e/0x50 [11508.244734] kvm_wait+0xcd/0xe0 [11508.248342] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11508.253260] do_raw_spin_lock+0x177/0x190 [11508.255988] _raw_spin_lock+0x21/0x30 [11508.257858] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11508.264369] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11508.270170] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11508.273279] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11508.278695] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11508.281495] ? woken_wake_function+0x30/0x30 [11508.284332] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11508.287461] kthread+0x1d1/0x200 [11508.289893] ? set_kthread_struct+0x70/0x70 [11508.293070] ret_from_fork+0x1f/0x30 [11514.651045] rcu: INFO: rcu_sched self-detected stall on CPU [11514.655149] rcu: 2-....: (5191665 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [11514.658755] (t=9420225 jiffies g=274185 q=66270) [11514.664776] rcu: rcu_sched kthread starved for 8878591 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [11514.670890] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11514.675126] rcu: RCU grace-period kthread stack dump: [11514.679122] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11514.683387] Call Trace: [11514.684105] __schedule+0x351/0xcb0 [11514.685018] schedule+0xc0/0x180 [11514.687235] schedule_timeout+0xb4/0x190 [11514.690011] ? do_raw_spin_unlock+0x75/0x190 [11514.691668] ? __next_timer_interrupt+0x160/0x160 [11514.694088] ? prepare_to_swait_event+0xf3/0x140 [11514.698317] rcu_gp_kthread+0x8dd/0x15b0 [11514.700856] ? force_qs_rnp+0x2c0/0x2c0 [11514.703115] kthread+0x1d1/0x200 [11514.703901] ? set_kthread_struct+0x70/0x70 [11514.705475] ret_from_fork+0x1f/0x30 [11514.708032] rcu: Stack dump where RCU GP kthread last ran: [11514.711866] NMI backtrace for cpu 2 [11514.714867] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11514.721865] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11514.730008] Call Trace: [11514.732212] [11514.733403] ? dump_stack+0xbb/0x10e [11514.735468] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11514.739398] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11514.741567] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11514.746383] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11514.751368] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11514.756813] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [11514.760313] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [11514.773568] ? update_process_times+0x6d/0xc0 [11514.779363] ? tick_sched_handle.isra.18+0x27/0x90 [11514.786510] ? tick_sched_timer+0x53/0xb0 [11514.789916] ? __hrtimer_run_queues+0x1c2/0x4d0 [11514.793754] ? tick_sched_handle.isra.18+0x90/0x90 [11514.796598] ? hrtimer_interrupt+0x128/0x2c0 [11514.801420] ? smp_apic_timer_interrupt+0xa6/0x270 [11514.806505] ? apic_timer_interrupt+0xf/0x20 [11514.816081] [11514.817423] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11514.820543] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11514.824771] ? do_raw_spin_lock+0x177/0x190 [11514.829571] ? _raw_spin_lock+0x21/0x30 [11514.830960] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11514.842536] ? obd_mod_free+0x46/0x140 [ptlrpc] [11514.848750] ? mdc_commit_open+0x91/0xb0 [mdc] [11514.854461] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11514.858604] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11514.861599] ? after_reply+0xe2c/0x1780 [ptlrpc] [11514.864440] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11514.867672] ? __next_timer_interrupt+0x160/0x160 [11514.869687] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11514.871853] ? do_wait_intr+0xf0/0xf0 [11514.873580] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11514.877694] ? kthread+0x1d1/0x200 [11514.879508] ? set_kthread_struct+0x70/0x70 [11514.881595] ? ret_from_fork+0x1f/0x30 [11514.883344] Sending NMI from CPU 2 to CPUs 1: [11514.888390] NMI backtrace for cpu 1 [11514.888398] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11514.888400] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11514.888401] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11514.888403] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11514.888406] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [11514.888408] RAX: 0000000000007b87 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11514.888410] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11514.888411] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11514.888413] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11514.888414] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11514.888415] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11514.888417] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11514.888418] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11514.888419] Call Trace: [11514.888420] [11514.888421] ? show_regs.cold.9+0x22/0x2f [11514.888422] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [11514.888423] ? nmi_cpu_backtrace_handler+0x15/0x40 [11514.888424] ? nmi_handle+0xbd/0x210 [11514.888426] ? default_do_nmi+0x4b/0x1b0 [11514.888426] ? do_nmi+0x280/0x380 [11514.888427] ? end_repeat_nmi+0x16/0x1f [11514.888429] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11514.888430] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11514.888431] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11514.888432] [11514.888433] do_raw_spin_lock+0x177/0x190 [11514.888434] _raw_spin_lock+0x21/0x30 [11514.888435] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11514.888436] ? do_raw_spin_unlock+0x75/0x190 [11514.888437] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11514.888439] ? do_wait_intr+0xf0/0xf0 [11514.888440] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11514.888441] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11514.888442] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11514.888444] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11514.888445] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11514.888446] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11514.888447] mdc_intent_lock+0x353/0x7e0 [mdc] [11514.888449] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11514.888450] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11514.888451] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11514.888452] lmv_intent_open+0x338/0x1150 [lmv] [11514.888454] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11514.888455] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11514.888456] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11514.888457] ll_intent_lock+0xc5/0xaa0 [lustre] [11514.888458] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11514.888459] ll_lookup_it+0xe0e/0x2270 [lustre] [11514.888460] ? do_raw_spin_unlock+0x75/0x190 [11514.888461] ? _raw_spin_unlock+0x12/0x30 [11514.888462] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11514.888463] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11514.888465] ll_atomic_open+0x36a/0xd70 [lustre] [11514.888466] lookup_open+0x356/0xb40 [11514.888467] do_last+0x313/0xfc0 [11514.888468] ? nd_jump_root+0xe5/0x160 [11514.888469] ? path_init+0x437/0x520 [11514.888470] path_openat+0xf7/0x500 [11514.888471] ? do_raw_spin_unlock+0x75/0x190 [11514.888472] do_filp_open+0x99/0x140 [11514.888473] ? getname_flags+0x6e/0x330 [11514.888474] ? __check_object_size+0xff/0x256 [11514.888475] ? do_raw_spin_unlock+0x75/0x190 [11514.888476] ? _raw_spin_unlock+0x12/0x30 [11514.888477] do_sys_openat2+0x2b4/0x410 [11514.888478] do_sys_open+0x73/0xa0 [11514.888478] __x64_sys_openat+0x24/0x30 [11514.888479] do_syscall_64+0xc1/0x440 [11514.888480] entry_SYSCALL_64_after_hwframe+0x49/0xae [11514.888481] RIP: 0033:0x7f53425a5332 [11514.888482] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11514.888484] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11514.888487] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11514.888488] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11514.888490] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11514.888491] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11514.888493] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11514.893046] NMI backtrace for cpu 2 [11515.164328] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11515.167336] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11515.170835] Call Trace: [11515.171569] [11515.172332] ? dump_stack+0xbb/0x10e [11515.176067] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11515.179858] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11515.181503] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11515.186437] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11515.189121] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11515.193104] ? rcu_dump_cpu_stacks+0x11b/0x171 [11515.195756] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11515.199249] ? update_process_times+0x6d/0xc0 [11515.202141] ? tick_sched_handle.isra.18+0x27/0x90 [11515.204587] ? tick_sched_timer+0x53/0xb0 [11515.209732] ? __hrtimer_run_queues+0x1c2/0x4d0 [11515.212092] ? tick_sched_handle.isra.18+0x90/0x90 [11515.216955] ? hrtimer_interrupt+0x128/0x2c0 [11515.220825] ? smp_apic_timer_interrupt+0xa6/0x270 [11515.224208] ? apic_timer_interrupt+0xf/0x20 [11515.227476] [11515.228596] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11515.233233] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11515.237454] ? do_raw_spin_lock+0x177/0x190 [11515.239863] ? _raw_spin_lock+0x21/0x30 [11515.242193] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11515.244241] ? obd_mod_free+0x46/0x140 [ptlrpc] [11515.249568] ? mdc_commit_open+0x91/0xb0 [mdc] [11515.253628] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11515.257897] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11515.264787] ? after_reply+0xe2c/0x1780 [ptlrpc] [11515.269998] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11515.273931] ? __next_timer_interrupt+0x160/0x160 [11515.275691] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11515.281306] ? do_wait_intr+0xf0/0xf0 [11515.282658] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11515.292618] ? kthread+0x1d1/0x200 [11515.294943] ? set_kthread_struct+0x70/0x70 [11515.298529] ? ret_from_fork+0x1f/0x30 [11532.062395] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [11532.069330] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11532.107494] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11532.119258] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11532.126191] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11532.129477] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11532.132953] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11532.155036] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11532.164384] RAX: 0000000000007e13 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11532.169330] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11532.187199] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11532.195027] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11532.202949] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11532.208525] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11532.215727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11532.222708] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11532.233991] Call Trace: [11532.234867] [11532.235551] ? show_regs.cold.9+0x22/0x2f [11532.240477] ? watchdog_timer_fn.cold.6+0x62/0x10a [11532.242100] ? __hrtimer_run_queues+0x1c2/0x4d0 [11532.244479] ? watchdog+0x40/0x40 [11532.247803] ? hrtimer_interrupt+0x128/0x2c0 [11532.256329] ? smp_apic_timer_interrupt+0xa6/0x270 [11532.263565] ? apic_timer_interrupt+0xf/0x20 [11532.267650] [11532.268359] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11532.272493] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11532.276110] ? newidle_balance.isra.93+0x207/0x580 [11532.278814] do_raw_spin_lock+0x177/0x190 [11532.281130] _raw_spin_lock+0x21/0x30 [11532.290494] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11532.299132] process_one_work+0x2c8/0x700 [11532.301748] worker_thread+0x296/0x6e0 [11532.305514] ? rescuer_thread+0x570/0x570 [11532.313544] kthread+0x1d1/0x200 [11532.314690] ? set_kthread_struct+0x70/0x70 [11532.318790] ret_from_fork+0x1f/0x30 [11536.059330] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11536.066834] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11536.097599] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11536.105689] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11536.112495] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11536.115831] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11536.124047] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11536.127114] RAX: 0000000000006d99 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11536.130129] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11536.133190] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11536.136418] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11536.139402] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11536.142516] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11536.145722] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11536.147921] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11536.151017] Call Trace: [11536.152113] [11536.152958] ? show_regs.cold.9+0x22/0x2f [11536.154687] ? watchdog_timer_fn.cold.6+0x62/0x10a [11536.156833] ? __hrtimer_run_queues+0x1c2/0x4d0 [11536.158885] ? watchdog+0x40/0x40 [11536.160422] ? hrtimer_interrupt+0x128/0x2c0 [11536.162402] ? smp_apic_timer_interrupt+0xa6/0x270 [11536.164936] ? apic_timer_interrupt+0xf/0x20 [11536.166898] [11536.168294] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11536.170906] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11536.173902] do_raw_spin_lock+0x177/0x190 [11536.175825] _raw_spin_lock+0x21/0x30 [11536.177747] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11536.180739] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11536.183674] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11536.186386] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11536.190954] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11536.194233] ? woken_wake_function+0x30/0x30 [11536.196185] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11536.198787] kthread+0x1d1/0x200 [11536.200657] ? set_kthread_struct+0x70/0x70 [11536.202351] ret_from_fork+0x1f/0x30 [11536.864810] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8899s! [11536.872153] Showing busy workqueues and worker pools: [11536.878456] workqueue events: flags=0x0 [11536.882526] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11536.885422] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11536.890397] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11536.892960] pending: cpu_latency_work [ptlrpc] [11536.895635] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11536.898257] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11536.902207] workqueue events_long: flags=0x0 [11536.903815] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11536.906004] pending: lu_objects_flush [obdclass] [11536.907379] workqueue events_unbound: flags=0x2 [11536.908765] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11536.911180] pending: flush_memcg_stats_dwork, idle_cull_fn [11536.913440] workqueue events_power_efficient: flags=0x80 [11536.915657] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11536.918279] pending: check_lifetime, do_cache_clean [11536.920487] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11536.923142] pending: neigh_periodic_work, neigh_periodic_work [11536.925819] workqueue mm_percpu_wq: flags=0x8 [11536.927487] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11536.930459] pending: vmstat_update, lru_add_drain_per_cpu [11536.933084] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11536.935916] pending: vmstat_update, lru_add_drain_per_cpu [11536.938445] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11536.941084] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11536.943798] workqueue xprtiod: flags=0xa [11536.945399] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11536.948097] pending: xprt_autoclose, xs_error_handle [11536.950109] workqueue ipv6_addrconf: flags=0x40008 [11536.952113] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11536.955045] pending: addrconf_verify_work [11536.957300] workqueue knbd0-recv: flags=0x1a [11536.959430] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11536.962335] in-flight: 651:recv_work [11536.963448] workqueue ptlrpc_pinger: flags=0x2 [11536.964875] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11536.967142] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11536.969805] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8899s workers=4 idle: 15979 31824 8 [11536.973312] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11527s workers=2 idle: 654 [11540.057618] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11540.057859] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11540.062733] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11540.062811] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11540.062818] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11540.062821] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11540.062835] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11540.062839] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11540.062844] RAX: 000000000000720d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11540.062846] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11540.062848] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11540.062851] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11540.062853] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11540.062857] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11540.062860] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11540.062862] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11540.062871] Call Trace: [11540.062877] [11540.062882] ? show_regs.cold.9+0x22/0x2f [11540.062891] ? watchdog_timer_fn.cold.6+0x62/0x10a [11540.062899] ? __hrtimer_run_queues+0x1c2/0x4d0 [11540.062903] ? watchdog+0x40/0x40 [11540.062907] ? hrtimer_interrupt+0x128/0x2c0 [11540.062911] ? smp_apic_timer_interrupt+0xa6/0x270 [11540.062920] ? apic_timer_interrupt+0xf/0x20 [11540.062925] [11540.062928] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11540.062935] do_raw_spin_lock+0x177/0x190 [11540.062941] _raw_spin_lock+0x21/0x30 [11540.062946] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11540.063284] ? do_raw_spin_unlock+0x75/0x190 [11540.063291] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11540.063500] ? do_wait_intr+0xf0/0xf0 [11540.063509] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11540.063833] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11540.064085] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11540.066333] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11540.066768] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11540.066809] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11540.066838] mdc_intent_lock+0x353/0x7e0 [mdc] [11540.066865] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11540.066952] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11540.068468] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11540.069054] lmv_intent_open+0x338/0x1150 [lmv] [11540.069085] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11540.069203] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11540.069283] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11540.069306] ll_intent_lock+0xc5/0xaa0 [lustre] [11540.069393] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11540.070809] ll_lookup_it+0xe0e/0x2270 [lustre] [11540.078723] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11540.078801] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11540.078810] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11540.078813] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11540.078828] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11540.078833] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11540.078837] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11540.078840] RDX: 0000000000007721 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11540.078842] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11540.078844] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11540.078847] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11540.078850] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11540.078853] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11540.078856] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11540.078866] Call Trace: [11540.078873] [11540.078879] ? show_regs.cold.9+0x22/0x2f [11540.078890] ? watchdog_timer_fn.cold.6+0x62/0x10a [11540.078897] ? __hrtimer_run_queues+0x1c2/0x4d0 [11540.078901] ? watchdog+0x40/0x40 [11540.078906] ? hrtimer_interrupt+0x128/0x2c0 [11540.078910] ? smp_apic_timer_interrupt+0xa6/0x270 [11540.078918] ? apic_timer_interrupt+0xf/0x20 [11540.078923] [11540.078926] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11540.078931] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11540.078936] do_raw_spin_lock+0x177/0x190 [11540.078941] _raw_spin_lock+0x21/0x30 [11540.078947] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11540.079219] obd_mod_free+0x46/0x140 [ptlrpc] [11540.079425] mdc_commit_open+0x91/0xb0 [mdc] [11540.081270] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11540.083005] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11540.083259] after_reply+0xe2c/0x1780 [ptlrpc] [11540.084168] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11540.126023] ? do_raw_spin_unlock+0x75/0x190 [11540.126044] ? _raw_spin_unlock+0x12/0x30 [11540.126053] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11540.126082] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11540.126103] ll_atomic_open+0x36a/0xd70 [lustre] [11540.126387] lookup_open+0x356/0xb40 [11540.126399] do_last+0x313/0xfc0 [11540.126405] ? nd_jump_root+0xe5/0x160 [11540.126409] ? path_init+0x437/0x520 [11540.126413] path_openat+0xf7/0x500 [11540.126418] ? do_raw_spin_unlock+0x75/0x190 [11540.126423] do_filp_open+0x99/0x140 [11540.126429] ? getname_flags+0x6e/0x330 [11540.126498] ? __check_object_size+0xff/0x256 [11540.126508] ? do_raw_spin_unlock+0x75/0x190 [11540.126512] ? _raw_spin_unlock+0x12/0x30 [11540.126517] do_sys_openat2+0x2b4/0x410 [11540.126590] do_sys_open+0x73/0xa0 [11540.126597] __x64_sys_openat+0x24/0x30 [11540.126601] do_syscall_64+0xc1/0x440 [11540.126608] entry_SYSCALL_64_after_hwframe+0x49/0xae [11540.126615] RIP: 0033:0x7f53425a5332 [11540.126624] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11540.126628] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11540.126633] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11540.126636] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11540.126638] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11540.126640] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11540.126643] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11540.486088] ? __next_timer_interrupt+0x160/0x160 [11540.488400] ptlrpcd+0xa82/0xe40 [ptlrpc] [11540.492880] ? do_wait_intr+0xf0/0xf0 [11540.496863] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11540.501675] kthread+0x1d1/0x200 [11540.504941] ? set_kthread_struct+0x70/0x70 [11540.507331] ret_from_fork+0x1f/0x30 [11564.052738] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [11564.058456] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11564.059030] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11564.098975] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11564.098987] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11564.098994] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11564.099681] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [11564.099694] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [11564.099700] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11564.099704] RAX: 0000000000001031 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11564.099707] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11564.099709] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11564.099712] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11564.099714] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11564.099718] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11564.099721] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11564.099724] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11564.099733] Call Trace: [11564.099739] [11564.099746] ? show_regs.cold.9+0x22/0x2f [11564.099755] ? watchdog_timer_fn.cold.6+0x62/0x10a [11564.099762] ? __hrtimer_run_queues+0x1c2/0x4d0 [11564.099766] ? watchdog+0x40/0x40 [11564.099770] ? hrtimer_interrupt+0x128/0x2c0 [11564.099775] ? smp_apic_timer_interrupt+0xa6/0x270 [11564.099783] ? apic_timer_interrupt+0xf/0x20 [11564.099788] [11564.099791] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [11564.099796] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11564.099800] ? newidle_balance.isra.93+0x207/0x580 [11564.099806] do_raw_spin_lock+0x177/0x190 [11564.099812] _raw_spin_lock+0x21/0x30 [11564.099817] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11564.100380] process_one_work+0x2c8/0x700 [11564.100388] worker_thread+0x296/0x6e0 [11564.100393] ? rescuer_thread+0x570/0x570 [11564.100397] kthread+0x1d1/0x200 [11564.100402] ? set_kthread_struct+0x70/0x70 [11564.100406] ret_from_fork+0x1f/0x30 [11564.267602] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11564.317712] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11564.325720] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11564.330630] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11564.338053] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11564.349664] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11564.357880] RAX: 0000000000006e16 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11564.364366] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11564.370403] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11564.376393] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11564.383974] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11564.391879] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11564.398703] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11564.404497] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11564.412587] Call Trace: [11564.413846] [11564.414692] ? show_regs.cold.9+0x22/0x2f [11564.420274] ? watchdog_timer_fn.cold.6+0x62/0x10a [11564.427340] ? __hrtimer_run_queues+0x1c2/0x4d0 [11564.432521] ? watchdog+0x40/0x40 [11564.436959] ? hrtimer_interrupt+0x128/0x2c0 [11564.441586] ? smp_apic_timer_interrupt+0xa6/0x270 [11564.446974] ? apic_timer_interrupt+0xf/0x20 [11564.451587] [11564.453585] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11564.459676] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11564.466270] do_raw_spin_lock+0x177/0x190 [11564.468925] _raw_spin_lock+0x21/0x30 [11564.472196] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11564.479461] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11564.485577] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11564.491588] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11564.497369] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11564.503683] ? woken_wake_function+0x30/0x30 [11564.508647] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11564.515999] kthread+0x1d1/0x200 [11564.520717] ? set_kthread_struct+0x70/0x70 [11564.522773] ret_from_fork+0x1f/0x30 [11567.072078] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8929s! [11567.078180] Showing busy workqueues and worker pools: [11567.083656] workqueue events: flags=0x0 [11567.087117] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11567.089360] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11567.110619] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11567.112798] pending: cpu_latency_work [ptlrpc] [11567.127738] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11567.135909] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11567.139735] workqueue events_long: flags=0x0 [11567.146018] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11567.152662] pending: lu_objects_flush [obdclass] [11567.155618] workqueue events_unbound: flags=0x2 [11567.158116] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11567.160985] pending: flush_memcg_stats_dwork, idle_cull_fn [11567.163941] workqueue events_power_efficient: flags=0x80 [11567.168944] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11567.186687] pending: check_lifetime, do_cache_clean [11567.194258] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11567.197979] pending: neigh_periodic_work, neigh_periodic_work [11567.199736] workqueue mm_percpu_wq: flags=0x8 [11567.203287] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11567.209374] pending: vmstat_update, lru_add_drain_per_cpu [11567.214770] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11567.217247] pending: vmstat_update, lru_add_drain_per_cpu [11567.227929] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11567.229856] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11567.235793] workqueue xprtiod: flags=0xa [11567.238102] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11567.245332] pending: xprt_autoclose, xs_error_handle [11567.250279] workqueue ipv6_addrconf: flags=0x40008 [11567.259089] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11567.264245] pending: addrconf_verify_work [11567.265604] workqueue knbd0-recv: flags=0x1a [11567.266836] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11567.288816] in-flight: 651:recv_work [11567.297876] workqueue ptlrpc_pinger: flags=0x2 [11567.299478] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11567.311564] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11567.318429] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8930s workers=4 idle: 15979 31824 8 [11567.325933] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11557s workers=2 idle: 654 [11568.057954] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11568.060685] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11568.071317] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11568.071401] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11568.071409] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11568.071412] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11568.071493] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11568.071498] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11568.071507] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11568.071510] RDX: 0000000000001567 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11568.071513] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11568.071515] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11568.071517] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11568.071520] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11568.071523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11568.071525] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11568.071550] Call Trace: [11568.071567] [11568.071574] ? show_regs.cold.9+0x22/0x2f [11568.071586] ? watchdog_timer_fn.cold.6+0x62/0x10a [11568.071598] ? __hrtimer_run_queues+0x1c2/0x4d0 [11568.071602] ? watchdog+0x40/0x40 [11568.071607] ? hrtimer_interrupt+0x128/0x2c0 [11568.071610] ? smp_apic_timer_interrupt+0xa6/0x270 [11568.071619] ? apic_timer_interrupt+0xf/0x20 [11568.071623] [11568.071626] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11568.071635] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11568.071640] do_raw_spin_lock+0x177/0x190 [11568.071645] _raw_spin_lock+0x21/0x30 [11568.071651] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11568.071932] obd_mod_free+0x46/0x140 [ptlrpc] [11568.075866] Modules linked in: lustre(O) [11568.121509] mdc_commit_open+0x91/0xb0 [mdc] [11568.121559] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11568.121819] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11568.122028] after_reply+0xe2c/0x1780 [ptlrpc] [11568.122233] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11568.122437] ? __next_timer_interrupt+0x160/0x160 [11568.122448] ptlrpcd+0xa82/0xe40 [ptlrpc] [11568.122658] ? do_wait_intr+0xf0/0xf0 [11568.122668] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11568.137673] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11568.137746] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11568.137754] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11568.137757] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11568.137772] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11568.137776] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11568.137780] RAX: 0000000000003b11 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11568.137783] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11568.137785] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11568.137787] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11568.137790] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11568.137793] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11568.137797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11568.137799] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11568.137807] Call Trace: [11568.137812] [11568.137818] ? show_regs.cold.9+0x22/0x2f [11568.137827] ? watchdog_timer_fn.cold.6+0x62/0x10a [11568.137833] ? __hrtimer_run_queues+0x1c2/0x4d0 [11568.137839] ? watchdog+0x40/0x40 [11568.137843] ? hrtimer_interrupt+0x128/0x2c0 [11568.137847] ? smp_apic_timer_interrupt+0xa6/0x270 [11568.137855] ? apic_timer_interrupt+0xf/0x20 [11568.137859] [11568.137862] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11568.137868] do_raw_spin_lock+0x177/0x190 [11568.137873] _raw_spin_lock+0x21/0x30 [11568.137878] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11568.138174] ? do_raw_spin_unlock+0x75/0x190 [11568.138180] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11568.147661] kthread+0x1d1/0x200 [11568.147681] ? set_kthread_struct+0x70/0x70 [11568.147686] ret_from_fork+0x1f/0x30 [11568.494760] ? do_wait_intr+0xf0/0xf0 [11568.496648] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11568.499379] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11568.503517] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11568.506777] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11568.516094] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11568.520434] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11568.522704] mdc_intent_lock+0x353/0x7e0 [mdc] [11568.527873] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11568.531443] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11568.535603] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11568.538165] lmv_intent_open+0x338/0x1150 [lmv] [11568.540583] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11568.544795] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11568.547080] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11568.549364] ll_intent_lock+0xc5/0xaa0 [lustre] [11568.551610] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11568.555447] ll_lookup_it+0xe0e/0x2270 [lustre] [11568.557198] ? do_raw_spin_unlock+0x75/0x190 [11568.559557] ? _raw_spin_unlock+0x12/0x30 [11568.561624] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11568.564617] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11568.570455] ll_atomic_open+0x36a/0xd70 [lustre] [11568.573791] lookup_open+0x356/0xb40 [11568.581410] do_last+0x313/0xfc0 [11568.583321] ? nd_jump_root+0xe5/0x160 [11568.586507] ? path_init+0x437/0x520 [11568.589146] path_openat+0xf7/0x500 [11568.591381] ? do_raw_spin_unlock+0x75/0x190 [11568.596667] do_filp_open+0x99/0x140 [11568.598393] ? getname_flags+0x6e/0x330 [11568.600880] ? __check_object_size+0xff/0x256 [11568.603247] ? do_raw_spin_unlock+0x75/0x190 [11568.605352] ? _raw_spin_unlock+0x12/0x30 [11568.607931] do_sys_openat2+0x2b4/0x410 [11568.609421] do_sys_open+0x73/0xa0 [11568.611315] __x64_sys_openat+0x24/0x30 [11568.613883] do_syscall_64+0xc1/0x440 [11568.615694] entry_SYSCALL_64_after_hwframe+0x49/0xae [11568.618723] RIP: 0033:0x7f53425a5332 [11568.623789] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11568.639342] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11568.644692] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11568.648822] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11568.651284] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11568.653230] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11568.656096] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11592.053127] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [11592.058030] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11592.062266] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11592.062431] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11592.062438] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11592.062444] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11592.062769] RIP: 0010:native_safe_halt+0x2e/0x50 [11592.062779] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11592.062784] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11592.062788] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11592.062791] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [11592.062793] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11592.062795] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11592.062797] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11592.062801] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11592.062804] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11592.062806] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11592.062815] Call Trace: [11592.062820] [11592.062826] ? show_regs.cold.9+0x22/0x2f [11592.062836] ? watchdog_timer_fn.cold.6+0x62/0x10a [11592.062842] ? __hrtimer_run_queues+0x1c2/0x4d0 [11592.062846] ? watchdog+0x40/0x40 [11592.062850] ? hrtimer_interrupt+0x128/0x2c0 [11592.062854] ? smp_apic_timer_interrupt+0xa6/0x270 [11592.062860] ? apic_timer_interrupt+0xf/0x20 [11592.062865] [11592.062867] ? native_safe_halt+0x2e/0x50 [11592.062871] kvm_wait+0xcd/0xe0 [11592.062879] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11592.062886] ? newidle_balance.isra.93+0x207/0x580 [11592.062892] do_raw_spin_lock+0x177/0x190 [11592.062896] _raw_spin_lock+0x21/0x30 [11592.062900] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11592.063110] process_one_work+0x2c8/0x700 [11592.063117] worker_thread+0x296/0x6e0 [11592.063121] ? rescuer_thread+0x570/0x570 [11592.063126] kthread+0x1d1/0x200 [11592.063129] ? set_kthread_struct+0x70/0x70 [11592.063133] ret_from_fork+0x1f/0x30 [11592.259432] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11592.311775] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11592.321406] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11592.332258] RIP: 0010:native_safe_halt+0x2e/0x50 [11592.336743] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11592.358971] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11592.365363] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11592.370176] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11592.375601] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11592.378884] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11592.385664] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11592.393474] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11592.397454] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11592.400283] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11592.402762] Call Trace: [11592.403677] [11592.410434] ? show_regs.cold.9+0x22/0x2f [11592.413170] ? watchdog_timer_fn.cold.6+0x62/0x10a [11592.416312] ? __hrtimer_run_queues+0x1c2/0x4d0 [11592.422064] ? watchdog+0x40/0x40 [11592.426038] ? hrtimer_interrupt+0x128/0x2c0 [11592.428221] ? smp_apic_timer_interrupt+0xa6/0x270 [11592.433477] ? apic_timer_interrupt+0xf/0x20 [11592.436497] [11592.440954] ? native_safe_halt+0x2e/0x50 [11592.445952] kvm_wait+0xcd/0xe0 [11592.449249] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11592.457314] do_raw_spin_lock+0x177/0x190 [11592.463974] _raw_spin_lock+0x21/0x30 [11592.469411] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11592.474862] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11592.478922] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11592.482375] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11592.485294] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11592.488020] ? woken_wake_function+0x30/0x30 [11592.490187] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11592.493348] kthread+0x1d1/0x200 [11592.494808] ? set_kthread_struct+0x70/0x70 [11592.496415] ret_from_fork+0x1f/0x30 [11596.056593] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11596.058348] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11596.074606] Modules linked in: [11596.077900] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11596.077986] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11596.077995] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11596.077998] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11596.078014] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11596.078018] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11596.078023] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11596.078025] RDX: 00000000000037ce RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11596.078028] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11596.078030] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11596.078033] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11596.078036] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11596.078039] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11596.078042] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11596.078051] Call Trace: [11596.078058] [11596.078064] ? show_regs.cold.9+0x22/0x2f [11596.078075] ? watchdog_timer_fn.cold.6+0x62/0x10a [11596.078082] ? __hrtimer_run_queues+0x1c2/0x4d0 [11596.078086] ? watchdog+0x40/0x40 [11596.078091] ? hrtimer_interrupt+0x128/0x2c0 [11596.078095] ? smp_apic_timer_interrupt+0xa6/0x270 [11596.078104] ? apic_timer_interrupt+0xf/0x20 [11596.078110] [11596.078113] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11596.078118] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11596.078123] do_raw_spin_lock+0x177/0x190 [11596.078127] _raw_spin_lock+0x21/0x30 [11596.078134] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11596.078401] obd_mod_free+0x46/0x140 [ptlrpc] [11596.151919] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11596.152000] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11596.152008] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11596.152011] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11596.152028] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11596.152032] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11596.152037] RAX: 0000000000007f84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11596.152039] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11596.152042] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11596.152044] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11596.152046] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11596.152050] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11596.152053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11596.152055] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11596.152064] Call Trace: [11596.152070] [11596.152077] ? show_regs.cold.9+0x22/0x2f [11596.152086] ? watchdog_timer_fn.cold.6+0x62/0x10a [11596.152092] ? __hrtimer_run_queues+0x1c2/0x4d0 [11596.152097] ? watchdog+0x40/0x40 [11596.152101] ? hrtimer_interrupt+0x128/0x2c0 [11596.152106] ? smp_apic_timer_interrupt+0xa6/0x270 [11596.152115] ? apic_timer_interrupt+0xf/0x20 [11596.152119] [11596.152122] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11596.152129] do_raw_spin_lock+0x177/0x190 [11596.152134] _raw_spin_lock+0x21/0x30 [11596.152138] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11596.152418] ? do_raw_spin_unlock+0x75/0x190 [11596.152424] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11596.153698] ? do_wait_intr+0xf0/0xf0 [11596.153708] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11596.153898] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11596.154101] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11596.154307] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11596.154417] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11596.155262] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11596.155297] mdc_intent_lock+0x353/0x7e0 [mdc] [11596.155322] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11596.155415] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11596.172875] mdc_commit_open+0x91/0xb0 [mdc] [11596.172925] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11596.173238] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11596.173431] after_reply+0xe2c/0x1780 [ptlrpc] [11596.173646] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11596.173905] ? __next_timer_interrupt+0x160/0x160 [11596.173916] ptlrpcd+0xa82/0xe40 [ptlrpc] [11596.174140] ? do_wait_intr+0xf0/0xf0 [11596.174149] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11596.174347] kthread+0x1d1/0x200 [11596.174354] ? set_kthread_struct+0x70/0x70 [11596.174358] ret_from_fork+0x1f/0x30 [11596.605916] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11596.613771] lmv_intent_open+0x338/0x1150 [lmv] [11596.623151] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11596.626237] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11596.635799] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11596.639234] ll_intent_lock+0xc5/0xaa0 [lustre] [11596.649623] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11596.660196] ll_lookup_it+0xe0e/0x2270 [lustre] [11596.665696] ? do_raw_spin_unlock+0x75/0x190 [11596.672939] ? _raw_spin_unlock+0x12/0x30 [11596.679395] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11596.690150] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11596.695050] ll_atomic_open+0x36a/0xd70 [lustre] [11596.698043] lookup_open+0x356/0xb40 [11596.704822] do_last+0x313/0xfc0 [11596.707549] ? nd_jump_root+0xe5/0x160 [11596.717731] ? path_init+0x437/0x520 [11596.721614] path_openat+0xf7/0x500 [11596.729219] ? do_raw_spin_unlock+0x75/0x190 [11596.734424] do_filp_open+0x99/0x140 [11596.742155] ? getname_flags+0x6e/0x330 [11596.746542] ? __check_object_size+0xff/0x256 [11596.749364] ? do_raw_spin_unlock+0x75/0x190 [11596.751119] ? _raw_spin_unlock+0x12/0x30 [11596.758646] do_sys_openat2+0x2b4/0x410 [11596.761276] do_sys_open+0x73/0xa0 [11596.767650] __x64_sys_openat+0x24/0x30 [11596.770031] do_syscall_64+0xc1/0x440 [11596.773555] entry_SYSCALL_64_after_hwframe+0x49/0xae [11596.777361] RIP: 0033:0x7f53425a5332 [11596.779559] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11596.804381] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11596.814847] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11596.817322] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11596.826796] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11596.841234] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11596.852082] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11597.803661] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8960s! [11597.823492] Showing busy workqueues and worker pools: [11597.833737] workqueue events: flags=0x0 [11597.840062] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11597.848007] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11597.868144] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11597.879123] pending: cpu_latency_work [ptlrpc] [11597.883043] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11597.893785] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11597.901734] workqueue events_long: flags=0x0 [11597.904911] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11597.909733] pending: lu_objects_flush [obdclass] [11597.913950] workqueue events_unbound: flags=0x2 [11597.916335] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11597.919998] pending: flush_memcg_stats_dwork, idle_cull_fn [11597.923898] workqueue events_power_efficient: flags=0x80 [11597.926334] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11597.931374] pending: check_lifetime, do_cache_clean [11597.935092] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11597.937950] pending: neigh_periodic_work, neigh_periodic_work [11597.942482] workqueue mm_percpu_wq: flags=0x8 [11597.944712] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11597.948412] pending: vmstat_update, lru_add_drain_per_cpu [11597.950671] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11597.953138] pending: vmstat_update, lru_add_drain_per_cpu [11597.955683] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11597.957998] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11597.960868] workqueue xprtiod: flags=0xa [11597.962677] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11597.967240] pending: xprt_autoclose, xs_error_handle [11597.973181] workqueue ipv6_addrconf: flags=0x40008 [11597.979615] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11597.984945] pending: addrconf_verify_work [11597.988946] workqueue knbd0-recv: flags=0x1a [11597.994992] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11597.997368] in-flight: 651:recv_work [11598.001112] workqueue ptlrpc_pinger: flags=0x2 [11598.003134] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11598.010613] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11598.013357] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8960s workers=4 idle: 15979 31824 8 [11598.016810] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11588s workers=2 idle: 654 [11620.058115] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [11620.065591] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11620.108464] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11620.120511] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11620.128715] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11620.132582] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11620.161578] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [11620.169008] RAX: 0000000000001555 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11620.178685] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11620.192568] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11620.204629] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11620.213116] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11620.223978] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11620.230785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11620.236685] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11620.243047] Call Trace: [11620.244054] [11620.251158] ? show_regs.cold.9+0x22/0x2f [11620.254237] ? watchdog_timer_fn.cold.6+0x62/0x10a [11620.260377] ? __hrtimer_run_queues+0x1c2/0x4d0 [11620.265663] ? watchdog+0x40/0x40 [11620.275963] ? hrtimer_interrupt+0x128/0x2c0 [11620.292222] ? smp_apic_timer_interrupt+0xa6/0x270 [11620.297670] ? apic_timer_interrupt+0xf/0x20 [11620.304692] [11620.305386] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11620.316101] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11620.319249] do_raw_spin_lock+0x177/0x190 [11620.324601] _raw_spin_lock+0x21/0x30 [11620.327227] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11620.337294] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11620.343694] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11620.346260] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11620.351790] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11620.353802] ? woken_wake_function+0x30/0x30 [11620.358728] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11620.360738] kthread+0x1d1/0x200 [11620.362257] ? set_kthread_struct+0x70/0x70 [11620.363225] ret_from_fork+0x1f/0x30 [11624.052034] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [11624.055055] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11624.057007] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11624.057019] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11624.057093] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11624.057100] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11624.057154] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11624.057159] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11624.057165] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11624.057447] RIP: 0010:native_safe_halt+0x2e/0x50 [11624.057456] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11624.057460] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11624.057464] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11624.057466] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [11624.057469] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11624.057471] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11624.057473] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11624.057476] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11624.057479] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11624.057481] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11624.057489] Call Trace: [11624.057506] [11624.057511] ? show_regs.cold.9+0x22/0x2f [11624.058744] ? watchdog_timer_fn.cold.6+0x62/0x10a [11624.058754] ? __hrtimer_run_queues+0x1c2/0x4d0 [11624.058758] ? watchdog+0x40/0x40 [11624.058762] ? hrtimer_interrupt+0x128/0x2c0 [11624.058765] ? smp_apic_timer_interrupt+0xa6/0x270 [11624.058772] ? apic_timer_interrupt+0xf/0x20 [11624.058776] [11624.058779] ? native_safe_halt+0x2e/0x50 [11624.058783] kvm_wait+0xcd/0xe0 [11624.058788] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11624.058794] ? newidle_balance.isra.93+0x207/0x580 [11624.058800] do_raw_spin_lock+0x177/0x190 [11624.058804] _raw_spin_lock+0x21/0x30 [11624.058808] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11624.059187] process_one_work+0x2c8/0x700 [11624.059193] worker_thread+0x296/0x6e0 [11624.059198] ? rescuer_thread+0x570/0x570 [11624.059201] kthread+0x1d1/0x200 [11624.059205] ? set_kthread_struct+0x70/0x70 [11624.059208] ret_from_fork+0x1f/0x30 [11624.060582] Modules linked in: [11624.065833] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11624.065846] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11624.065862] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11624.065866] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11624.065872] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11624.065874] RDX: 0000000000003006 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11624.065876] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11624.065878] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11624.065880] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11624.065883] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11624.065885] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11624.065887] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11624.065896] Call Trace: [11624.065903] [11624.065910] ? show_regs.cold.9+0x22/0x2f [11624.065921] ? watchdog_timer_fn.cold.6+0x62/0x10a [11624.065928] ? __hrtimer_run_queues+0x1c2/0x4d0 [11624.065932] ? watchdog+0x40/0x40 [11624.065937] ? hrtimer_interrupt+0x128/0x2c0 [11624.065941] ? smp_apic_timer_interrupt+0xa6/0x270 [11624.065949] ? apic_timer_interrupt+0xf/0x20 [11624.065953] [11624.065956] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11624.065960] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11624.066073] do_raw_spin_lock+0x177/0x190 [11624.066081] _raw_spin_lock+0x21/0x30 [11624.066089] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11624.092715] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11624.092796] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11624.092804] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11624.092808] RIP: 0010:native_safe_halt+0x2e/0x50 [11624.092823] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11624.092827] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11624.092831] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11624.092833] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11624.092836] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11624.092838] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11624.092840] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11624.092843] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11624.092846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11624.092848] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11624.092857] Call Trace: [11624.092861] [11624.092867] ? show_regs.cold.9+0x22/0x2f [11624.092876] ? watchdog_timer_fn.cold.6+0x62/0x10a [11624.092883] ? __hrtimer_run_queues+0x1c2/0x4d0 [11624.092887] ? watchdog+0x40/0x40 [11624.092891] ? hrtimer_interrupt+0x128/0x2c0 [11624.092895] ? smp_apic_timer_interrupt+0xa6/0x270 [11624.092902] ? apic_timer_interrupt+0xf/0x20 [11624.092906] [11624.092909] ? native_safe_halt+0x2e/0x50 [11624.092913] kvm_wait+0xcd/0xe0 [11624.092921] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11624.092930] do_raw_spin_lock+0x177/0x190 [11624.092935] _raw_spin_lock+0x21/0x30 [11624.092940] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11624.100260] obd_mod_free+0x46/0x140 [ptlrpc] [11624.100552] mdc_commit_open+0x91/0xb0 [mdc] [11624.100599] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11624.100789] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11624.128599] ? do_raw_spin_unlock+0x75/0x190 [11624.134572] after_reply+0xe2c/0x1780 [ptlrpc] [11624.134858] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11624.138557] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11624.138802] ? do_wait_intr+0xf0/0xf0 [11624.138813] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11624.138955] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11624.140853] ? __next_timer_interrupt+0x160/0x160 [11624.140868] ptlrpcd+0xa82/0xe40 [ptlrpc] [11624.141167] ? do_wait_intr+0xf0/0xf0 [11624.141175] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11624.141366] kthread+0x1d1/0x200 [11624.141373] ? set_kthread_struct+0x70/0x70 [11624.141377] ret_from_fork+0x1f/0x30 [11624.530745] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11624.534842] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11624.537533] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11624.541367] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11624.545097] mdc_intent_lock+0x353/0x7e0 [mdc] [11624.547393] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11624.551566] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11624.555935] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11624.558804] lmv_intent_open+0x338/0x1150 [lmv] [11624.562177] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11624.567504] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11624.572059] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11624.574656] ll_intent_lock+0xc5/0xaa0 [lustre] [11624.576174] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11624.577865] ll_lookup_it+0xe0e/0x2270 [lustre] [11624.580187] ? do_raw_spin_unlock+0x75/0x190 [11624.581313] ? _raw_spin_unlock+0x12/0x30 [11624.582910] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11624.584382] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11624.586517] ll_atomic_open+0x36a/0xd70 [lustre] [11624.587902] lookup_open+0x356/0xb40 [11624.588890] do_last+0x313/0xfc0 [11624.590561] ? nd_jump_root+0xe5/0x160 [11624.591566] ? path_init+0x437/0x520 [11624.593852] path_openat+0xf7/0x500 [11624.594825] ? do_raw_spin_unlock+0x75/0x190 [11624.595922] do_filp_open+0x99/0x140 [11624.596816] ? getname_flags+0x6e/0x330 [11624.597856] ? __check_object_size+0xff/0x256 [11624.599106] ? do_raw_spin_unlock+0x75/0x190 [11624.600187] ? _raw_spin_unlock+0x12/0x30 [11624.601678] do_sys_openat2+0x2b4/0x410 [11624.602681] do_sys_open+0x73/0xa0 [11624.603850] __x64_sys_openat+0x24/0x30 [11624.605180] do_syscall_64+0xc1/0x440 [11624.606498] entry_SYSCALL_64_after_hwframe+0x49/0xae [11624.608172] RIP: 0033:0x7f53425a5332 [11624.609334] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11624.616522] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11624.619218] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11624.621640] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11624.624736] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11624.627488] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11624.629772] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11628.512769] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 8991s! [11628.520632] Showing busy workqueues and worker pools: [11628.523478] workqueue events: flags=0x0 [11628.525613] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11628.533898] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11628.539806] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11628.542778] pending: cpu_latency_work [ptlrpc] [11628.545625] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11628.549724] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11628.556370] workqueue events_long: flags=0x0 [11628.558789] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11628.562098] pending: lu_objects_flush [obdclass] [11628.564290] workqueue events_unbound: flags=0x2 [11628.566029] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11628.568103] pending: flush_memcg_stats_dwork, idle_cull_fn [11628.570492] workqueue events_power_efficient: flags=0x80 [11628.572629] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11628.575383] pending: check_lifetime, do_cache_clean [11628.577985] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11628.580469] pending: neigh_periodic_work, neigh_periodic_work [11628.583356] workqueue mm_percpu_wq: flags=0x8 [11628.585129] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11628.587785] pending: vmstat_update, lru_add_drain_per_cpu [11628.590496] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11628.593517] pending: vmstat_update, lru_add_drain_per_cpu [11628.596040] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11628.599529] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11628.602616] workqueue xprtiod: flags=0xa [11628.604132] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11628.606678] pending: xprt_autoclose, xs_error_handle [11628.608909] workqueue ipv6_addrconf: flags=0x40008 [11628.611210] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11628.614345] pending: addrconf_verify_work [11628.616425] workqueue knbd0-recv: flags=0x1a [11628.618351] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11628.621116] in-flight: 651:recv_work [11628.622843] workqueue ptlrpc_pinger: flags=0x2 [11628.624823] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11628.627735] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11628.630613] pool 8: cpus=0-3 flags=0x4 nice=0 hung=8991s workers=4 idle: 15979 31824 8 [11628.634455] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11619s workers=2 idle: 654 [11648.062569] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [11648.066073] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11648.101189] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11648.106810] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11648.111155] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11648.114185] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11648.128095] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11648.135075] RAX: 0000000000007a36 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11648.142258] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11648.149250] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11648.153387] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11648.157670] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11648.161349] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11648.165760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11648.168833] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11648.175589] Call Trace: [11648.176652] [11648.177376] ? show_regs.cold.9+0x22/0x2f [11648.181824] ? watchdog_timer_fn.cold.6+0x62/0x10a [11648.188515] ? __hrtimer_run_queues+0x1c2/0x4d0 [11648.191004] ? watchdog+0x40/0x40 [11648.192811] ? hrtimer_interrupt+0x128/0x2c0 [11648.194729] ? smp_apic_timer_interrupt+0xa6/0x270 [11648.196965] ? apic_timer_interrupt+0xf/0x20 [11648.199054] [11648.199982] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11648.202767] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11648.205667] do_raw_spin_lock+0x177/0x190 [11648.207253] _raw_spin_lock+0x21/0x30 [11648.209308] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11648.212660] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11648.215532] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11648.218254] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11648.221273] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11648.223923] ? woken_wake_function+0x30/0x30 [11648.226016] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11648.228789] kthread+0x1d1/0x200 [11648.230033] ? set_kthread_struct+0x70/0x70 [11648.231916] ret_from_fork+0x1f/0x30 [11652.055459] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11652.062107] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod [11652.062541] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11652.110098] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11652.110219] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11652.110228] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11652.110231] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11652.110247] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11652.110251] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11652.110255] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11652.110257] RDX: 0000000000005607 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11652.110259] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11652.110261] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11652.110264] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11652.110267] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11652.110270] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11652.110273] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11652.110281] Call Trace: [11652.110286] [11652.110292] ? show_regs.cold.9+0x22/0x2f [11652.110302] ? watchdog_timer_fn.cold.6+0x62/0x10a [11652.110308] ? __hrtimer_run_queues+0x1c2/0x4d0 [11652.110313] ? watchdog+0x40/0x40 [11652.110317] ? hrtimer_interrupt+0x128/0x2c0 [11652.110320] ? smp_apic_timer_interrupt+0xa6/0x270 [11652.110397] ? apic_timer_interrupt+0xf/0x20 [11652.110402] [11652.110405] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11652.110410] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11652.110414] do_raw_spin_lock+0x177/0x190 [11652.110419] _raw_spin_lock+0x21/0x30 [11652.110425] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11652.110788] obd_mod_free+0x46/0x140 [ptlrpc] [11652.110980] mdc_commit_open+0x91/0xb0 [mdc] [11652.111019] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11652.111216] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11652.119618] sha512_ssse3 sha512_generic [11652.119644] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11652.119713] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11652.119717] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11652.119733] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11652.119738] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11652.119743] RAX: 0000000000003031 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11652.119746] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11652.119749] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11652.119750] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11652.119753] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11652.119756] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11652.120125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11652.120128] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11652.120136] Call Trace: [11652.120142] [11652.120148] ? show_regs.cold.9+0x22/0x2f [11652.120162] ? watchdog_timer_fn.cold.6+0x62/0x10a [11652.120169] ? __hrtimer_run_queues+0x1c2/0x4d0 [11652.120173] ? watchdog+0x40/0x40 [11652.120177] ? hrtimer_interrupt+0x128/0x2c0 [11652.120182] ? smp_apic_timer_interrupt+0xa6/0x270 [11652.120189] ? apic_timer_interrupt+0xf/0x20 [11652.120194] [11652.120197] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11652.120206] do_raw_spin_lock+0x177/0x190 [11652.120211] _raw_spin_lock+0x21/0x30 [11652.120217] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11652.120691] ? do_raw_spin_unlock+0x75/0x190 [11652.120698] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11652.121288] ? do_wait_intr+0xf0/0xf0 [11652.121298] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11652.121516] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11652.121731] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11652.155442] after_reply+0xe2c/0x1780 [ptlrpc] [11652.166463] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11652.166670] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11652.166707] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11652.166740] mdc_intent_lock+0x353/0x7e0 [mdc] [11652.166769] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11652.167153] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11652.167532] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11652.167573] lmv_intent_open+0x338/0x1150 [lmv] [11652.167610] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11652.167715] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11652.167809] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11652.167832] ll_intent_lock+0xc5/0xaa0 [lustre] [11652.167918] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11652.167999] ll_lookup_it+0xe0e/0x2270 [lustre] [11652.168225] ? do_raw_spin_unlock+0x75/0x190 [11652.168233] ? _raw_spin_unlock+0x12/0x30 [11652.168246] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11652.168275] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11652.168301] ll_atomic_open+0x36a/0xd70 [lustre] [11652.168388] lookup_open+0x356/0xb40 [11652.168402] do_last+0x313/0xfc0 [11652.168407] ? nd_jump_root+0xe5/0x160 [11652.168412] ? path_init+0x437/0x520 [11652.168417] path_openat+0xf7/0x500 [11652.168423] ? do_raw_spin_unlock+0x75/0x190 [11652.168427] do_filp_open+0x99/0x140 [11652.168434] ? getname_flags+0x6e/0x330 [11652.168439] ? __check_object_size+0xff/0x256 [11652.168447] ? do_raw_spin_unlock+0x75/0x190 [11652.168451] ? _raw_spin_unlock+0x12/0x30 [11652.168459] do_sys_openat2+0x2b4/0x410 [11652.168464] do_sys_open+0x73/0xa0 [11652.168483] __x64_sys_openat+0x24/0x30 [11652.168487] do_syscall_64+0xc1/0x440 [11652.168495] entry_SYSCALL_64_after_hwframe+0x49/0xae [11652.168502] RIP: 0033:0x7f53425a5332 [11652.168510] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11652.168513] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11652.168519] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11652.168521] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11652.168524] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11652.168526] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11652.168528] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11652.569150] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11652.575834] ? __next_timer_interrupt+0x160/0x160 [11652.579310] ptlrpcd+0xa82/0xe40 [ptlrpc] [11652.585258] ? do_wait_intr+0xf0/0xf0 [11652.589212] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11652.594635] kthread+0x1d1/0x200 [11652.596007] ? set_kthread_struct+0x70/0x70 [11652.599047] ret_from_fork+0x1f/0x30 [11656.059926] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [11656.074587] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11656.124222] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11656.144909] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11656.149296] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11656.153067] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11656.157311] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11656.172146] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11656.178011] RAX: 000000000000692c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11656.189068] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11656.195855] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11656.205493] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11656.208371] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11656.217155] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11656.227987] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11656.232965] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11656.244986] Call Trace: [11656.250353] [11656.253248] ? show_regs.cold.9+0x22/0x2f [11656.257980] ? watchdog_timer_fn.cold.6+0x62/0x10a [11656.266323] ? __hrtimer_run_queues+0x1c2/0x4d0 [11656.269384] ? watchdog+0x40/0x40 [11656.272864] ? hrtimer_interrupt+0x128/0x2c0 [11656.277145] ? smp_apic_timer_interrupt+0xa6/0x270 [11656.282080] ? apic_timer_interrupt+0xf/0x20 [11656.289267] [11656.294295] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11656.300956] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11656.306249] ? newidle_balance.isra.93+0x207/0x580 [11656.308075] do_raw_spin_lock+0x177/0x190 [11656.309544] _raw_spin_lock+0x21/0x30 [11656.311220] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11656.313929] process_one_work+0x2c8/0x700 [11656.318032] worker_thread+0x296/0x6e0 [11656.320755] ? rescuer_thread+0x570/0x570 [11656.323313] kthread+0x1d1/0x200 [11656.325181] ? set_kthread_struct+0x70/0x70 [11656.327377] ret_from_fork+0x1f/0x30 [11658.723459] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9021s! [11658.729115] Showing busy workqueues and worker pools: [11658.731457] workqueue events: flags=0x0 [11658.734292] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11658.739459] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11658.751240] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11658.754411] pending: cpu_latency_work [ptlrpc] [11658.756237] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11658.758206] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11658.766182] workqueue events_long: flags=0x0 [11658.768029] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11658.773971] pending: lu_objects_flush [obdclass] [11658.785916] workqueue events_unbound: flags=0x2 [11658.793827] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11658.805302] pending: flush_memcg_stats_dwork, idle_cull_fn [11658.811691] workqueue events_power_efficient: flags=0x80 [11658.816811] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11658.827034] pending: check_lifetime, do_cache_clean [11658.831470] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11658.840561] pending: neigh_periodic_work, neigh_periodic_work [11658.844234] workqueue mm_percpu_wq: flags=0x8 [11658.845798] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11658.850476] pending: vmstat_update, lru_add_drain_per_cpu [11658.856851] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11658.862325] pending: vmstat_update, lru_add_drain_per_cpu [11658.864480] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11658.867378] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11658.871222] workqueue xprtiod: flags=0xa [11658.875319] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11658.883516] pending: xprt_autoclose, xs_error_handle [11658.888265] workqueue ipv6_addrconf: flags=0x40008 [11658.890460] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11658.894418] pending: addrconf_verify_work [11658.898448] workqueue knbd0-recv: flags=0x1a [11658.902652] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11658.906710] in-flight: 651:recv_work [11658.911491] workqueue ptlrpc_pinger: flags=0x2 [11658.913866] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11658.921886] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11658.926464] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9021s workers=4 idle: 15979 31824 8 [11658.935969] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11649s workers=2 idle: 654 [11676.058125] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [11676.060107] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11676.078756] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11676.083228] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11676.085876] RIP: 0010:native_safe_halt+0x2e/0x50 [11676.087289] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11676.095549] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11676.099337] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11676.107004] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11676.111194] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11676.115418] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11676.119183] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11676.127123] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11676.133738] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11676.135713] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11676.142507] Call Trace: [11676.143596] [11676.144931] ? show_regs.cold.9+0x22/0x2f [11676.149479] ? watchdog_timer_fn.cold.6+0x62/0x10a [11676.155527] ? __hrtimer_run_queues+0x1c2/0x4d0 [11676.159847] ? watchdog+0x40/0x40 [11676.161038] ? hrtimer_interrupt+0x128/0x2c0 [11676.164383] ? smp_apic_timer_interrupt+0xa6/0x270 [11676.169173] ? apic_timer_interrupt+0xf/0x20 [11676.178777] [11676.179551] ? native_safe_halt+0x2e/0x50 [11676.183691] kvm_wait+0xcd/0xe0 [11676.185584] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11676.192121] do_raw_spin_lock+0x177/0x190 [11676.193588] _raw_spin_lock+0x21/0x30 [11676.195673] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11676.201904] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11676.203641] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11676.208730] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11676.211180] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11676.213874] ? woken_wake_function+0x30/0x30 [11676.215351] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11676.218047] kthread+0x1d1/0x200 [11676.219695] ? set_kthread_struct+0x70/0x70 [11676.222277] ret_from_fork+0x1f/0x30 [11680.056693] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11680.061401] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11680.064054] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11680.088322] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11680.088337] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11680.088340] RIP: 0010:native_safe_halt+0x2e/0x50 [11680.088353] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11680.088358] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11680.088364] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11680.088366] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11680.088369] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11680.088371] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11680.088374] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11680.088377] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11680.088381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11680.088383] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11680.088391] Call Trace: [11680.088397] [11680.088404] ? show_regs.cold.9+0x22/0x2f [11680.088415] ? watchdog_timer_fn.cold.6+0x62/0x10a [11680.088421] ? __hrtimer_run_queues+0x1c2/0x4d0 [11680.088425] ? watchdog+0x40/0x40 [11680.088430] ? hrtimer_interrupt+0x128/0x2c0 [11680.088434] ? smp_apic_timer_interrupt+0xa6/0x270 [11680.088441] ? apic_timer_interrupt+0xf/0x20 [11680.088446] [11680.088448] ? native_safe_halt+0x2e/0x50 [11680.088453] kvm_wait+0xcd/0xe0 [11680.088460] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11680.088470] do_raw_spin_lock+0x177/0x190 [11680.088475] _raw_spin_lock+0x21/0x30 [11680.088479] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11680.088773] ? do_raw_spin_unlock+0x75/0x190 [11680.088780] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11680.089200] ? do_wait_intr+0xf0/0xf0 [11680.089210] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11680.089416] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11680.089633] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11680.090394] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11680.090524] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11680.090569] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11680.090600] mdc_intent_lock+0x353/0x7e0 [mdc] [11680.090626] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11680.093605] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11680.093684] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11680.093691] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11680.093695] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11680.093709] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11680.093714] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11680.093719] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11680.093721] RDX: 0000000000006a72 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11680.093724] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11680.093726] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11680.093728] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11680.093731] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11680.093735] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11680.093737] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11680.093746] Call Trace: [11680.093752] [11680.093758] ? show_regs.cold.9+0x22/0x2f [11680.093769] ? watchdog_timer_fn.cold.6+0x62/0x10a [11680.093777] ? __hrtimer_run_queues+0x1c2/0x4d0 [11680.093924] ? watchdog+0x40/0x40 [11680.093929] ? hrtimer_interrupt+0x128/0x2c0 [11680.093933] ? smp_apic_timer_interrupt+0xa6/0x270 [11680.093943] ? apic_timer_interrupt+0xf/0x20 [11680.093950] [11680.093953] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11680.093958] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11680.093962] do_raw_spin_lock+0x177/0x190 [11680.093966] _raw_spin_lock+0x21/0x30 [11680.093972] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11680.094315] obd_mod_free+0x46/0x140 [ptlrpc] [11680.094526] mdc_commit_open+0x91/0xb0 [mdc] [11680.094567] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11680.094777] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11680.103410] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11680.103716] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11680.103759] lmv_intent_open+0x338/0x1150 [lmv] [11680.103792] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11680.111620] after_reply+0xe2c/0x1780 [ptlrpc] [11680.119258] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11680.119384] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11680.119421] ll_intent_lock+0xc5/0xaa0 [lustre] [11680.119505] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11680.119586] ll_lookup_it+0xe0e/0x2270 [lustre] [11680.119667] ? do_raw_spin_unlock+0x75/0x190 [11680.119675] ? _raw_spin_unlock+0x12/0x30 [11680.119682] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11680.119710] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11680.119730] ll_atomic_open+0x36a/0xd70 [lustre] [11680.119812] lookup_open+0x356/0xb40 [11680.119821] do_last+0x313/0xfc0 [11680.119825] ? nd_jump_root+0xe5/0x160 [11680.119830] ? path_init+0x437/0x520 [11680.119834] path_openat+0xf7/0x500 [11680.119840] ? do_raw_spin_unlock+0x75/0x190 [11680.119844] do_filp_open+0x99/0x140 [11680.120194] ? getname_flags+0x6e/0x330 [11680.120199] ? __check_object_size+0xff/0x256 [11680.120208] ? do_raw_spin_unlock+0x75/0x190 [11680.120212] ? _raw_spin_unlock+0x12/0x30 [11680.120217] do_sys_openat2+0x2b4/0x410 [11680.120222] do_sys_open+0x73/0xa0 [11680.120227] __x64_sys_openat+0x24/0x30 [11680.120232] do_syscall_64+0xc1/0x440 [11680.120240] entry_SYSCALL_64_after_hwframe+0x49/0xae [11680.120247] RIP: 0033:0x7f53425a5332 [11680.120255] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11680.120259] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11680.120265] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11680.120268] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11680.120270] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11680.120272] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11680.120276] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11680.726690] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11680.729181] ? __next_timer_interrupt+0x160/0x160 [11680.730906] ptlrpcd+0xa82/0xe40 [ptlrpc] [11680.733116] ? do_wait_intr+0xf0/0xf0 [11680.734599] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11680.736923] kthread+0x1d1/0x200 [11680.738527] ? set_kthread_struct+0x70/0x70 [11680.740228] ret_from_fork+0x1f/0x30 [11684.053120] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [11684.065676] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11684.111290] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11684.121742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11684.126111] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11684.132647] RIP: 0010:native_safe_halt+0x2e/0x50 [11684.137636] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11684.149521] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11684.154399] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11684.159407] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [11684.163456] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11684.168367] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11684.172712] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11684.177072] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11684.193060] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11684.199112] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11684.203447] Call Trace: [11684.206592] [11684.209003] ? show_regs.cold.9+0x22/0x2f [11684.211092] ? watchdog_timer_fn.cold.6+0x62/0x10a [11684.224214] ? __hrtimer_run_queues+0x1c2/0x4d0 [11684.232489] ? watchdog+0x40/0x40 [11684.238350] ? hrtimer_interrupt+0x128/0x2c0 [11684.242086] ? smp_apic_timer_interrupt+0xa6/0x270 [11684.250898] ? apic_timer_interrupt+0xf/0x20 [11684.254508] [11684.255620] ? native_safe_halt+0x2e/0x50 [11684.261393] kvm_wait+0xcd/0xe0 [11684.265620] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11684.268592] ? newidle_balance.isra.93+0x207/0x580 [11684.272934] do_raw_spin_lock+0x177/0x190 [11684.274400] _raw_spin_lock+0x21/0x30 [11684.284057] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11684.290465] process_one_work+0x2c8/0x700 [11684.296600] worker_thread+0x296/0x6e0 [11684.299490] ? rescuer_thread+0x570/0x570 [11684.311052] kthread+0x1d1/0x200 [11684.312589] ? set_kthread_struct+0x70/0x70 [11684.318691] ret_from_fork+0x1f/0x30 [11689.440051] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9052s! [11689.444255] Showing busy workqueues and worker pools: [11689.450540] workqueue events: flags=0x0 [11689.457369] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11689.466049] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11689.476201] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11689.481854] pending: cpu_latency_work [ptlrpc] [11689.495366] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11689.509157] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11689.517276] workqueue events_long: flags=0x0 [11689.520349] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11689.526225] pending: lu_objects_flush [obdclass] [11689.530134] workqueue events_unbound: flags=0x2 [11689.534271] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11689.540553] pending: flush_memcg_stats_dwork, idle_cull_fn [11689.546885] workqueue events_power_efficient: flags=0x80 [11689.550317] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11689.555105] pending: check_lifetime, do_cache_clean [11689.560594] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11689.569797] pending: neigh_periodic_work, neigh_periodic_work [11689.576183] workqueue mm_percpu_wq: flags=0x8 [11689.578419] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11689.580594] pending: vmstat_update, lru_add_drain_per_cpu [11689.587309] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11689.597163] pending: vmstat_update, lru_add_drain_per_cpu [11689.600527] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11689.606186] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11689.611789] workqueue xprtiod: flags=0xa [11689.614274] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11689.622085] pending: xprt_autoclose, xs_error_handle [11689.624027] workqueue ipv6_addrconf: flags=0x40008 [11689.625148] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11689.626723] pending: addrconf_verify_work [11689.634419] workqueue knbd0-recv: flags=0x1a [11689.640284] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11689.648359] in-flight: 651:recv_work [11689.649952] workqueue ptlrpc_pinger: flags=0x2 [11689.653462] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11689.656693] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11689.660508] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9052s workers=4 idle: 15979 31824 8 [11689.664648] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11680s workers=2 idle: 654 [11694.654474] rcu: INFO: rcu_sched self-detected stall on CPU [11694.660000] rcu: 2-....: (5298734 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [11694.668989] (t=9600228 jiffies g=274185 q=66287) [11694.676426] rcu: rcu_sched kthread starved for 9058594 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [11694.685495] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11694.690570] rcu: RCU grace-period kthread stack dump: [11694.692708] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11694.703110] Call Trace: [11694.704458] __schedule+0x351/0xcb0 [11694.708341] schedule+0xc0/0x180 [11694.710147] schedule_timeout+0xb4/0x190 [11694.711606] ? do_raw_spin_unlock+0x75/0x190 [11694.713841] ? __next_timer_interrupt+0x160/0x160 [11694.715518] ? prepare_to_swait_event+0xf3/0x140 [11694.720300] rcu_gp_kthread+0x8dd/0x15b0 [11694.722037] ? force_qs_rnp+0x2c0/0x2c0 [11694.724579] kthread+0x1d1/0x200 [11694.730081] ? set_kthread_struct+0x70/0x70 [11694.733491] ret_from_fork+0x1f/0x30 [11694.738236] rcu: Stack dump where RCU GP kthread last ran: [11694.744608] Sending NMI from CPU 2 to CPUs 1: [11694.755702] NMI backtrace for cpu 1 [11694.755709] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11694.755712] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11694.755714] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11694.755715] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11694.755718] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [11694.755721] RAX: 0000000000005f0a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11694.755722] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11694.755724] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11694.755725] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11694.755736] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11694.755737] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11694.755738] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11694.755739] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11694.755741] Call Trace: [11694.755742] [11694.755743] ? show_regs.cold.9+0x22/0x2f [11694.755744] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [11694.755745] ? nmi_cpu_backtrace_handler+0x15/0x40 [11694.755746] ? nmi_handle+0xbd/0x210 [11694.755748] ? default_do_nmi+0x4b/0x1b0 [11694.755749] ? do_nmi+0x280/0x380 [11694.755750] ? end_repeat_nmi+0x16/0x1f [11694.755751] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11694.755752] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11694.755753] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11694.755754] [11694.755755] do_raw_spin_lock+0x177/0x190 [11694.755756] _raw_spin_lock+0x21/0x30 [11694.755757] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11694.755759] ? do_raw_spin_unlock+0x75/0x190 [11694.755760] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11694.755761] ? do_wait_intr+0xf0/0xf0 [11694.755762] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11694.755763] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11694.755764] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11694.755765] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11694.755766] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11694.755767] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11694.755768] mdc_intent_lock+0x353/0x7e0 [mdc] [11694.755769] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11694.755770] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11694.755771] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11694.755772] lmv_intent_open+0x338/0x1150 [lmv] [11694.755773] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11694.755774] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11694.755775] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11694.755776] ll_intent_lock+0xc5/0xaa0 [lustre] [11694.755777] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11694.755778] ll_lookup_it+0xe0e/0x2270 [lustre] [11694.755779] ? do_raw_spin_unlock+0x75/0x190 [11694.755780] ? _raw_spin_unlock+0x12/0x30 [11694.755781] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11694.755782] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11694.755783] ll_atomic_open+0x36a/0xd70 [lustre] [11694.755784] lookup_open+0x356/0xb40 [11694.755785] do_last+0x313/0xfc0 [11694.755786] ? nd_jump_root+0xe5/0x160 [11694.755787] ? path_init+0x437/0x520 [11694.755788] path_openat+0xf7/0x500 [11694.755789] ? do_raw_spin_unlock+0x75/0x190 [11694.755790] do_filp_open+0x99/0x140 [11694.755791] ? getname_flags+0x6e/0x330 [11694.755792] ? __check_object_size+0xff/0x256 [11694.755793] ? do_raw_spin_unlock+0x75/0x190 [11694.755794] ? _raw_spin_unlock+0x12/0x30 [11694.755795] do_sys_openat2+0x2b4/0x410 [11694.755796] do_sys_open+0x73/0xa0 [11694.755797] __x64_sys_openat+0x24/0x30 [11694.755798] do_syscall_64+0xc1/0x440 [11694.755799] entry_SYSCALL_64_after_hwframe+0x49/0xae [11694.755801] RIP: 0033:0x7f53425a5332 [11694.755802] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11694.755804] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11694.755807] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11694.755808] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11694.755809] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11694.755811] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11694.755812] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11694.756632] Sending NMI from CPU 2 to CPUs 1: [11694.996051] NMI backtrace for cpu 1 [11694.996058] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11694.996061] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11694.996062] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11694.996064] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11694.996066] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [11694.996070] RAX: 0000000000005fbb RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11694.996071] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11694.996073] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11694.996075] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11694.996076] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11694.996079] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11694.996081] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11694.996082] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11694.996084] Call Trace: [11694.996085] [11694.996086] ? show_regs.cold.9+0x22/0x2f [11694.996088] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [11694.996089] ? nmi_cpu_backtrace_handler+0x15/0x40 [11694.996090] ? nmi_handle+0xbd/0x210 [11694.996092] ? default_do_nmi+0x4b/0x1b0 [11694.996093] ? do_nmi+0x280/0x380 [11694.996094] ? end_repeat_nmi+0x16/0x1f [11694.996095] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11694.996096] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11694.996098] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11694.996099] [11694.996100] do_raw_spin_lock+0x177/0x190 [11694.996101] _raw_spin_lock+0x21/0x30 [11694.996103] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11694.996104] ? do_raw_spin_unlock+0x75/0x190 [11694.996105] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11694.996106] ? do_wait_intr+0xf0/0xf0 [11694.996108] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11694.996109] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11694.996110] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11694.996111] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11694.996112] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11694.996114] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11694.996115] mdc_intent_lock+0x353/0x7e0 [mdc] [11694.996116] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11694.996117] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11694.996118] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11694.996119] lmv_intent_open+0x338/0x1150 [lmv] [11694.996120] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11694.996121] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11694.996123] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11694.996124] ll_intent_lock+0xc5/0xaa0 [lustre] [11694.996125] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11694.996126] ll_lookup_it+0xe0e/0x2270 [lustre] [11694.996127] ? do_raw_spin_unlock+0x75/0x190 [11694.996128] ? _raw_spin_unlock+0x12/0x30 [11694.996130] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11694.996131] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11694.996132] ll_atomic_open+0x36a/0xd70 [lustre] [11694.996134] lookup_open+0x356/0xb40 [11694.996135] do_last+0x313/0xfc0 [11694.996136] ? nd_jump_root+0xe5/0x160 [11694.996137] ? path_init+0x437/0x520 [11694.996138] path_openat+0xf7/0x500 [11694.996139] ? do_raw_spin_unlock+0x75/0x190 [11694.996140] do_filp_open+0x99/0x140 [11694.996141] ? getname_flags+0x6e/0x330 [11694.996143] ? __check_object_size+0xff/0x256 [11694.996144] ? do_raw_spin_unlock+0x75/0x190 [11694.996145] ? _raw_spin_unlock+0x12/0x30 [11694.996146] do_sys_openat2+0x2b4/0x410 [11694.996148] do_sys_open+0x73/0xa0 [11694.996149] __x64_sys_openat+0x24/0x30 [11694.996150] do_syscall_64+0xc1/0x440 [11694.996151] entry_SYSCALL_64_after_hwframe+0x49/0xae [11694.996153] RIP: 0033:0x7f53425a5332 [11694.996154] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11694.996157] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11694.996160] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11694.996161] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11694.996163] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11694.996164] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11694.996165] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11694.999685] NMI backtrace for cpu 2 [11695.287198] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11695.294076] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11695.296358] Call Trace: [11695.304397] [11695.305678] ? dump_stack+0xbb/0x10e [11695.307414] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [11695.310351] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [11695.312830] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [11695.316302] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [11695.318376] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [11695.320987] ? rcu_dump_cpu_stacks+0x11b/0x171 [11695.322503] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [11695.324386] ? update_process_times+0x6d/0xc0 [11695.325752] ? tick_sched_handle.isra.18+0x27/0x90 [11695.328439] ? tick_sched_timer+0x53/0xb0 [11695.330230] ? __hrtimer_run_queues+0x1c2/0x4d0 [11695.332438] ? tick_sched_handle.isra.18+0x90/0x90 [11695.335054] ? hrtimer_interrupt+0x128/0x2c0 [11695.337044] ? smp_apic_timer_interrupt+0xa6/0x270 [11695.339131] ? apic_timer_interrupt+0xf/0x20 [11695.341053] [11695.342044] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11695.344500] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11695.347147] ? do_raw_spin_lock+0x177/0x190 [11695.349923] ? _raw_spin_lock+0x21/0x30 [11695.351376] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11695.354578] ? obd_mod_free+0x46/0x140 [ptlrpc] [11695.357465] ? mdc_commit_open+0x91/0xb0 [mdc] [11695.360323] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11695.362466] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11695.366916] ? after_reply+0xe2c/0x1780 [ptlrpc] [11695.370040] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11695.373145] ? __next_timer_interrupt+0x160/0x160 [11695.376274] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [11695.383175] ? do_wait_intr+0xf0/0xf0 [11695.385322] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11695.392931] ? kthread+0x1d1/0x200 [11695.395377] ? set_kthread_struct+0x70/0x70 [11695.399463] ? ret_from_fork+0x1f/0x30 [11704.060598] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [11704.063969] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11704.103406] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11704.111434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11704.114472] RIP: 0010:native_safe_halt+0x2e/0x50 [11704.118451] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11704.130912] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11704.135255] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11704.138053] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11704.140497] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11704.143021] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11704.145480] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11704.148777] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11704.152275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11704.157098] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11704.167661] Call Trace: [11704.169089] [11704.171111] ? show_regs.cold.9+0x22/0x2f [11704.173207] ? watchdog_timer_fn.cold.6+0x62/0x10a [11704.176511] ? __hrtimer_run_queues+0x1c2/0x4d0 [11704.180302] ? watchdog+0x40/0x40 [11704.181463] ? hrtimer_interrupt+0x128/0x2c0 [11704.185172] ? smp_apic_timer_interrupt+0xa6/0x270 [11704.189354] ? apic_timer_interrupt+0xf/0x20 [11704.193887] [11704.194643] ? native_safe_halt+0x2e/0x50 [11704.195992] kvm_wait+0xcd/0xe0 [11704.198254] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11704.201464] do_raw_spin_lock+0x177/0x190 [11704.203482] _raw_spin_lock+0x21/0x30 [11704.205739] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11704.211523] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11704.215443] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11704.220047] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11704.223519] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11704.230873] ? woken_wake_function+0x30/0x30 [11704.236798] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11704.238874] kthread+0x1d1/0x200 [11704.242265] ? set_kthread_struct+0x70/0x70 [11704.245553] ret_from_fork+0x1f/0x30 [11716.054962] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [11716.058305] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11716.083214] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11716.093514] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11716.101277] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11716.106305] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11716.108208] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11716.118308] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11716.122938] RAX: 000000000000611f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11716.125861] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11716.132898] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11716.136227] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11716.142510] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11716.147299] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11716.158232] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11716.164616] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11716.177557] Call Trace: [11716.182312] [11716.183051] ? show_regs.cold.9+0x22/0x2f [11716.192170] ? watchdog_timer_fn.cold.6+0x62/0x10a [11716.198097] ? __hrtimer_run_queues+0x1c2/0x4d0 [11716.203962] ? watchdog+0x40/0x40 [11716.208632] ? hrtimer_interrupt+0x128/0x2c0 [11716.211175] ? smp_apic_timer_interrupt+0xa6/0x270 [11716.221591] ? apic_timer_interrupt+0xf/0x20 [11716.227714] [11716.228887] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11716.233747] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11716.240600] ? newidle_balance.isra.93+0x207/0x580 [11716.244696] do_raw_spin_lock+0x177/0x190 [11716.251951] _raw_spin_lock+0x21/0x30 [11716.254049] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11716.259455] process_one_work+0x2c8/0x700 [11716.260857] worker_thread+0x296/0x6e0 [11716.262416] ? rescuer_thread+0x570/0x570 [11716.263896] kthread+0x1d1/0x200 [11716.265240] ? set_kthread_struct+0x70/0x70 [11716.267244] ret_from_fork+0x1f/0x30 [11720.058190] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [11720.058301] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [11720.070009] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11720.070092] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11720.070100] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11720.070103] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11720.070118] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11720.070123] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11720.070128] RAX: 0000000000005c2b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11720.070130] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11720.070132] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11720.070134] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11720.070137] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11720.070140] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11720.070143] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11720.070146] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11720.070155] Call Trace: [11720.070161] [11720.070167] ? show_regs.cold.9+0x22/0x2f [11720.070177] ? watchdog_timer_fn.cold.6+0x62/0x10a [11720.070184] ? __hrtimer_run_queues+0x1c2/0x4d0 [11720.070188] ? watchdog+0x40/0x40 [11720.070192] ? hrtimer_interrupt+0x128/0x2c0 [11720.070196] ? smp_apic_timer_interrupt+0xa6/0x270 [11720.070205] ? apic_timer_interrupt+0xf/0x20 [11720.070210] [11720.070213] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11720.070220] do_raw_spin_lock+0x177/0x190 [11720.070225] _raw_spin_lock+0x21/0x30 [11720.070230] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11720.077426] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11720.077511] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11720.077519] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11720.077522] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11720.077539] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11720.077544] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11720.077553] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11720.077555] RDX: 00000000000058de RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11720.077558] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11720.077560] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11720.077562] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11720.077566] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11720.077569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11720.077571] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11720.077579] Call Trace: [11720.077585] [11720.077590] ? show_regs.cold.9+0x22/0x2f [11720.077609] ? watchdog_timer_fn.cold.6+0x62/0x10a [11720.077615] ? __hrtimer_run_queues+0x1c2/0x4d0 [11720.077621] ? watchdog+0x40/0x40 [11720.077629] ? hrtimer_interrupt+0x128/0x2c0 [11720.077633] ? smp_apic_timer_interrupt+0xa6/0x270 [11720.077643] ? apic_timer_interrupt+0xf/0x20 [11720.077648] [11720.077655] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11720.077661] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11720.077666] do_raw_spin_lock+0x177/0x190 [11720.077672] _raw_spin_lock+0x21/0x30 [11720.077679] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11720.078090] obd_mod_free+0x46/0x140 [ptlrpc] [11720.078340] mdc_commit_open+0x91/0xb0 [mdc] [11720.078390] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11720.078667] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11720.078853] after_reply+0xe2c/0x1780 [ptlrpc] [11720.079213] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11720.079442] ? __next_timer_interrupt+0x160/0x160 [11720.079461] ptlrpcd+0xa82/0xe40 [ptlrpc] [11720.079680] ? do_wait_intr+0xf0/0xf0 [11720.079690] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11720.080482] kthread+0x1d1/0x200 [11720.080489] ? set_kthread_struct+0x70/0x70 [11720.080494] ret_from_fork+0x1f/0x30 [11720.160105] BUG: workqueue lockup - pool [11720.168584] ? do_raw_spin_unlock+0x75/0x190 [11720.168704] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11720.171071] cpus=0-3 flags=0x4 nice=0 stuck for 9083s! [11720.171090] Showing busy workqueues and worker pools: [11720.171092] workqueue events: flags=0x0 [11720.171096] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11720.171103] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4] [11720.181905] ? do_wait_intr+0xf0/0xf0 [11720.181927] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11720.182202] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11720.182854] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11720.183062] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11720.183202] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11720.183259] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11720.183550] mdc_intent_lock+0x353/0x7e0 [mdc] [11720.183659] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11720.183799] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11720.184248] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11720.184292] lmv_intent_open+0x338/0x1150 [lmv] [11720.184337] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11720.184434] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11720.184660] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11720.184689] ll_intent_lock+0xc5/0xaa0 [lustre] [11720.184788] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11720.184873] ll_lookup_it+0xe0e/0x2270 [lustre] [11720.184957] ? do_raw_spin_unlock+0x75/0x190 [11720.184966] ? _raw_spin_unlock+0x12/0x30 [11720.184975] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11720.185003] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11720.185032] ll_atomic_open+0x36a/0xd70 [lustre] [11720.185130] lookup_open+0x356/0xb40 [11720.185143] do_last+0x313/0xfc0 [11720.185148] ? nd_jump_root+0xe5/0x160 [11720.185157] ? path_init+0x437/0x520 [11720.185165] path_openat+0xf7/0x500 [11720.185170] ? do_raw_spin_unlock+0x75/0x190 [11720.185175] do_filp_open+0x99/0x140 [11720.185185] ? getname_flags+0x6e/0x330 [11720.185190] ? __check_object_size+0xff/0x256 [11720.185198] ? do_raw_spin_unlock+0x75/0x190 [11720.185202] ? _raw_spin_unlock+0x12/0x30 [11720.185206] do_sys_openat2+0x2b4/0x410 [11720.185210] do_sys_open+0x73/0xa0 [11720.185215] __x64_sys_openat+0x24/0x30 [11720.185219] do_syscall_64+0xc1/0x440 [11720.185225] entry_SYSCALL_64_after_hwframe+0x49/0xae [11720.185231] RIP: 0033:0x7f53425a5332 [11720.185239] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11720.185243] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11720.185248] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11720.185251] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11720.185253] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11720.185255] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11720.185257] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11721.022775] , push_to_pool [11721.025556] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11721.030194] pending: cpu_latency_work [ptlrpc] [11721.035050] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11721.043935] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11721.049743] workqueue events_long: flags=0x0 [11721.054438] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11721.059989] pending: lu_objects_flush [obdclass] [11721.062659] workqueue events_unbound: flags=0x2 [11721.064050] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11721.067592] pending: flush_memcg_stats_dwork, idle_cull_fn [11721.070748] workqueue events_power_efficient: flags=0x80 [11721.073968] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11721.077151] pending: check_lifetime, do_cache_clean [11721.079682] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11721.082906] pending: neigh_periodic_work, neigh_periodic_work [11721.085752] workqueue mm_percpu_wq: flags=0x8 [11721.087572] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11721.090370] pending: vmstat_update, lru_add_drain_per_cpu [11721.092667] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11721.095379] pending: vmstat_update, lru_add_drain_per_cpu [11721.097679] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11721.101237] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11721.104098] workqueue xprtiod: flags=0xa [11721.105718] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11721.108249] pending: xprt_autoclose, xs_error_handle [11721.110045] workqueue ipv6_addrconf: flags=0x40008 [11721.111609] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11721.114178] pending: addrconf_verify_work [11721.115942] workqueue knbd0-recv: flags=0x1a [11721.117706] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11721.120439] in-flight: 651:recv_work [11721.122056] workqueue ptlrpc_pinger: flags=0x2 [11721.123884] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11721.126394] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11721.129052] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9084s workers=4 idle: 15979 31824 8 [11721.132169] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11711s workers=2 idle: 654 [11732.060351] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [11732.072661] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11732.122067] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11732.131496] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11732.137505] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11732.140448] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11732.166088] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11732.176226] RAX: 0000000000002b0c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11732.184960] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11732.192051] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11732.203907] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11732.210610] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11732.217794] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11732.230949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11732.233064] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11732.238737] Call Trace: [11732.241907] [11732.243258] ? show_regs.cold.9+0x22/0x2f [11732.245607] ? watchdog_timer_fn.cold.6+0x62/0x10a [11732.251066] ? __hrtimer_run_queues+0x1c2/0x4d0 [11732.257449] ? watchdog+0x40/0x40 [11732.263605] ? hrtimer_interrupt+0x128/0x2c0 [11732.267044] ? smp_apic_timer_interrupt+0xa6/0x270 [11732.272044] ? apic_timer_interrupt+0xf/0x20 [11732.277040] [11732.279932] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11732.282825] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11732.290419] do_raw_spin_lock+0x177/0x190 [11732.295607] _raw_spin_lock+0x21/0x30 [11732.299954] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11732.303409] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11732.306398] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11732.310316] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11732.314263] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11732.317589] ? woken_wake_function+0x30/0x30 [11732.319754] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11732.322949] kthread+0x1d1/0x200 [11732.324575] ? set_kthread_struct+0x70/0x70 [11732.328705] ret_from_fork+0x1f/0x30 [11748.052009] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [11748.055010] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [11748.056242] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11748.056314] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11748.056321] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11748.056327] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11748.056602] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [11748.056613] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [11748.056616] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11748.056620] RAX: 000000000000242f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11748.056622] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11748.056624] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11748.056626] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11748.056628] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11748.056631] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11748.056634] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11748.056636] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11748.056646] Call Trace: [11748.056651] [11748.056657] ? show_regs.cold.9+0x22/0x2f [11748.061089] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11748.061337] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11748.061412] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [11748.061420] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11748.061497] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11748.061507] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11748.061512] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11748.061645] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11748.061654] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11748.062176] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11748.062179] RDX: 00000000000076be RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11748.062181] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11748.062183] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11748.062185] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11748.062188] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11748.062192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11748.062195] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11748.062202] Call Trace: [11748.062208] [11748.062214] ? show_regs.cold.9+0x22/0x2f [11748.062223] ? watchdog_timer_fn.cold.6+0x62/0x10a [11748.062230] ? __hrtimer_run_queues+0x1c2/0x4d0 [11748.062240] ? watchdog+0x40/0x40 [11748.062245] ? hrtimer_interrupt+0x128/0x2c0 [11748.062249] ? smp_apic_timer_interrupt+0xa6/0x270 [11748.062257] ? apic_timer_interrupt+0xf/0x20 [11748.062262] [11748.062265] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11748.062274] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11748.062280] do_raw_spin_lock+0x177/0x190 [11748.062285] _raw_spin_lock+0x21/0x30 [11748.062291] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11748.062586] obd_mod_free+0x46/0x140 [ptlrpc] [11748.062807] mdc_commit_open+0x91/0xb0 [mdc] [11748.063377] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11748.063701] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11748.064054] after_reply+0xe2c/0x1780 [ptlrpc] [11748.064908] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11748.065910] ? __next_timer_interrupt+0x160/0x160 [11748.065923] ptlrpcd+0xa82/0xe40 [ptlrpc] [11748.066222] ? do_wait_intr+0xf0/0xf0 [11748.066231] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11748.066433] kthread+0x1d1/0x200 [11748.066438] ? set_kthread_struct+0x70/0x70 [11748.066442] ret_from_fork+0x1f/0x30 [11748.135648] ? watchdog_timer_fn.cold.6+0x62/0x10a [11748.135673] ? __hrtimer_run_queues+0x1c2/0x4d0 [11748.135678] ? watchdog+0x40/0x40 [11748.135682] ? hrtimer_interrupt+0x128/0x2c0 [11748.135687] ? smp_apic_timer_interrupt+0xa6/0x270 [11748.135696] ? apic_timer_interrupt+0xf/0x20 [11748.135700] [11748.135705] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [11748.135713] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11748.135717] ? newidle_balance.isra.93+0x207/0x580 [11748.135725] do_raw_spin_lock+0x177/0x190 [11748.135731] _raw_spin_lock+0x21/0x30 [11748.135736] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11748.138707] process_one_work+0x2c8/0x700 [11748.138718] worker_thread+0x296/0x6e0 [11748.138723] ? rescuer_thread+0x570/0x570 [11748.138727] kthread+0x1d1/0x200 [11748.138732] ? set_kthread_struct+0x70/0x70 [11748.138736] ret_from_fork+0x1f/0x30 [11748.600698] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11748.603670] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11748.606069] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11748.613228] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11748.621639] RAX: 0000000000005889 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11748.629085] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11748.636456] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11748.642594] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11748.646521] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11748.650046] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11748.652721] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11748.654880] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11748.658073] Call Trace: [11748.658771] [11748.659843] ? show_regs.cold.9+0x22/0x2f [11748.661376] ? watchdog_timer_fn.cold.6+0x62/0x10a [11748.663095] ? __hrtimer_run_queues+0x1c2/0x4d0 [11748.665038] ? watchdog+0x40/0x40 [11748.666326] ? hrtimer_interrupt+0x128/0x2c0 [11748.667767] ? smp_apic_timer_interrupt+0xa6/0x270 [11748.669569] ? apic_timer_interrupt+0xf/0x20 [11748.671308] [11748.672436] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11748.674595] do_raw_spin_lock+0x177/0x190 [11748.676010] _raw_spin_lock+0x21/0x30 [11748.677437] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11748.679588] ? do_raw_spin_unlock+0x75/0x190 [11748.681511] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11748.685357] ? do_wait_intr+0xf0/0xf0 [11748.687661] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11748.690739] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11748.693558] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11748.695964] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11748.698920] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11748.700973] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11748.702904] mdc_intent_lock+0x353/0x7e0 [mdc] [11748.704591] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11748.706721] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11748.709302] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11748.711423] lmv_intent_open+0x338/0x1150 [lmv] [11748.713083] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11748.715388] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11748.717297] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11748.718771] ll_intent_lock+0xc5/0xaa0 [lustre] [11748.720563] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11748.722523] ll_lookup_it+0xe0e/0x2270 [lustre] [11748.724155] ? do_raw_spin_unlock+0x75/0x190 [11748.725744] ? _raw_spin_unlock+0x12/0x30 [11748.727117] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11748.728726] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11748.730615] ll_atomic_open+0x36a/0xd70 [lustre] [11748.732205] lookup_open+0x356/0xb40 [11748.733303] do_last+0x313/0xfc0 [11748.734517] ? nd_jump_root+0xe5/0x160 [11748.735636] ? path_init+0x437/0x520 [11748.737096] path_openat+0xf7/0x500 [11748.738349] ? do_raw_spin_unlock+0x75/0x190 [11748.739852] do_filp_open+0x99/0x140 [11748.742647] ? getname_flags+0x6e/0x330 [11748.746080] ? __check_object_size+0xff/0x256 [11748.747662] ? do_raw_spin_unlock+0x75/0x190 [11748.749456] ? _raw_spin_unlock+0x12/0x30 [11748.752833] do_sys_openat2+0x2b4/0x410 [11748.756275] do_sys_open+0x73/0xa0 [11748.757994] __x64_sys_openat+0x24/0x30 [11748.759359] do_syscall_64+0xc1/0x440 [11748.760776] entry_SYSCALL_64_after_hwframe+0x49/0xae [11748.762531] RIP: 0033:0x7f53425a5332 [11748.763898] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11748.770762] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11748.773127] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11748.774945] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11748.777153] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11748.779317] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11748.781203] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11751.392114] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9114s! [11751.395091] Showing busy workqueues and worker pools: [11751.397693] workqueue events: flags=0x0 [11751.405856] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11751.409045] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11751.416318] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11751.420239] pending: cpu_latency_work [ptlrpc] [11751.423455] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11751.427220] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11751.431724] workqueue events_long: flags=0x0 [11751.434191] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11751.437789] pending: lu_objects_flush [obdclass] [11751.440720] workqueue events_unbound: flags=0x2 [11751.446305] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11751.450216] pending: flush_memcg_stats_dwork, idle_cull_fn [11751.455144] workqueue events_power_efficient: flags=0x80 [11751.459099] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11751.463275] pending: check_lifetime, do_cache_clean [11751.466445] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11751.470713] pending: neigh_periodic_work, neigh_periodic_work [11751.474660] workqueue mm_percpu_wq: flags=0x8 [11751.477148] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11751.480791] pending: vmstat_update, lru_add_drain_per_cpu [11751.484258] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11751.490089] pending: vmstat_update, lru_add_drain_per_cpu [11751.494645] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11751.499320] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11751.503165] workqueue xprtiod: flags=0xa [11751.505264] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11751.508924] pending: xprt_autoclose, xs_error_handle [11751.512234] workqueue ipv6_addrconf: flags=0x40008 [11751.515454] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11751.520214] pending: addrconf_verify_work [11751.523302] workqueue knbd0-recv: flags=0x1a [11751.532183] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11751.537765] in-flight: 651:recv_work [11751.540016] workqueue ptlrpc_pinger: flags=0x2 [11751.543349] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11751.548514] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11751.552293] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9114s workers=4 idle: 15979 31824 8 [11751.558509] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11742s workers=2 idle: 654 [11760.058009] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11760.061121] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11760.082733] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11760.086972] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11760.091692] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11760.093460] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11760.101090] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11760.107975] RAX: 0000000000004f2a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11760.115110] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11760.121242] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11760.125480] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11760.129241] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11760.132357] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11760.135711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11760.144701] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11760.148088] Call Trace: [11760.150009] [11760.152973] ? show_regs.cold.9+0x22/0x2f [11760.156398] ? watchdog_timer_fn.cold.6+0x62/0x10a [11760.158968] ? __hrtimer_run_queues+0x1c2/0x4d0 [11760.165535] ? watchdog+0x40/0x40 [11760.171535] ? hrtimer_interrupt+0x128/0x2c0 [11760.175176] ? smp_apic_timer_interrupt+0xa6/0x270 [11760.178303] ? apic_timer_interrupt+0xf/0x20 [11760.180461] [11760.188472] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11760.192331] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11760.198267] do_raw_spin_lock+0x177/0x190 [11760.199842] _raw_spin_lock+0x21/0x30 [11760.201255] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11760.205085] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11760.206849] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11760.213004] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11760.214590] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11760.218297] ? woken_wake_function+0x30/0x30 [11760.220398] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11760.223801] kthread+0x1d1/0x200 [11760.229123] ? set_kthread_struct+0x70/0x70 [11760.230399] ret_from_fork+0x1f/0x30 [11776.052014] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [11776.054953] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11776.056843] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [11776.057293] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [11776.057300] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11776.057384] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11776.057389] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11776.057392] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11776.057405] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11776.057409] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11776.057414] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11776.057416] RDX: 00000000000042ed RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11776.057419] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11776.057421] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11776.057424] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11776.057427] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11776.057430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11776.057432] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11776.057440] Call Trace: [11776.057445] [11776.057450] ? show_regs.cold.9+0x22/0x2f [11776.057458] ? watchdog_timer_fn.cold.6+0x62/0x10a [11776.057465] ? __hrtimer_run_queues+0x1c2/0x4d0 [11776.057469] ? watchdog+0x40/0x40 [11776.057473] ? hrtimer_interrupt+0x128/0x2c0 [11776.057477] ? smp_apic_timer_interrupt+0xa6/0x270 [11776.057485] ? apic_timer_interrupt+0xf/0x20 [11776.057490] [11776.057493] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11776.057497] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11776.057502] do_raw_spin_lock+0x177/0x190 [11776.057507] _raw_spin_lock+0x21/0x30 [11776.057512] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11776.057802] obd_mod_free+0x46/0x140 [ptlrpc] [11776.058007] mdc_commit_open+0x91/0xb0 [mdc] [11776.058102] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11776.058415] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11776.058655] after_reply+0xe2c/0x1780 [ptlrpc] [11776.058869] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11776.059060] ? __next_timer_interrupt+0x160/0x160 [11776.059069] ptlrpcd+0xa82/0xe40 [ptlrpc] [11776.059478] ? do_wait_intr+0xf0/0xf0 [11776.059543] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11776.059747] kthread+0x1d1/0x200 [11776.059753] ? set_kthread_struct+0x70/0x70 [11776.059757] ret_from_fork+0x1f/0x30 [11776.075855] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11776.075869] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11776.075876] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11776.076292] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11776.076316] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11776.076320] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11776.076324] RAX: 000000000000550a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11776.076326] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11776.076328] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11776.076329] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11776.076332] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11776.076339] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11776.076342] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11776.076344] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11776.076351] Call Trace: [11776.076357] [11776.076362] ? show_regs.cold.9+0x22/0x2f [11776.076376] ? watchdog_timer_fn.cold.6+0x62/0x10a [11776.076384] ? __hrtimer_run_queues+0x1c2/0x4d0 [11776.076389] ? watchdog+0x40/0x40 [11776.076394] ? hrtimer_interrupt+0x128/0x2c0 [11776.076399] ? smp_apic_timer_interrupt+0xa6/0x270 [11776.076538] ? apic_timer_interrupt+0xf/0x20 [11776.076543] [11776.076545] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11776.076550] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11776.076554] ? newidle_balance.isra.93+0x207/0x580 [11776.076559] do_raw_spin_lock+0x177/0x190 [11776.076564] _raw_spin_lock+0x21/0x30 [11776.076568] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11776.080688] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11776.080890] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11776.080897] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11776.080899] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11776.080913] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11776.080917] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11776.080921] RAX: 0000000000004391 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11776.080923] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11776.080926] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11776.080927] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11776.080929] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11776.080932] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11776.080935] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11776.080937] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11776.080947] Call Trace: [11776.080952] [11776.080962] ? show_regs.cold.9+0x22/0x2f [11776.080971] ? watchdog_timer_fn.cold.6+0x62/0x10a [11776.080981] ? __hrtimer_run_queues+0x1c2/0x4d0 [11776.080985] ? watchdog+0x40/0x40 [11776.080989] ? hrtimer_interrupt+0x128/0x2c0 [11776.080996] ? smp_apic_timer_interrupt+0xa6/0x270 [11776.081004] ? apic_timer_interrupt+0xf/0x20 [11776.081008] [11776.081011] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11776.081018] do_raw_spin_lock+0x177/0x190 [11776.081023] _raw_spin_lock+0x21/0x30 [11776.081028] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11776.081435] ? do_raw_spin_unlock+0x75/0x190 [11776.081442] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11776.081638] ? do_wait_intr+0xf0/0xf0 [11776.081648] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11776.081864] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11776.082280] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11776.082595] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11776.082702] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11776.082738] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11776.082783] mdc_intent_lock+0x353/0x7e0 [mdc] [11776.082808] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11776.082898] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11776.083205] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11776.083235] lmv_intent_open+0x338/0x1150 [lmv] [11776.083270] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11776.087159] process_one_work+0x2c8/0x700 [11776.087181] worker_thread+0x296/0x6e0 [11776.087186] ? rescuer_thread+0x570/0x570 [11776.087189] kthread+0x1d1/0x200 [11776.087194] ? set_kthread_struct+0x70/0x70 [11776.087198] ret_from_fork+0x1f/0x30 [11776.535212] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11776.537528] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11776.539248] ll_intent_lock+0xc5/0xaa0 [lustre] [11776.541265] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11776.543585] ll_lookup_it+0xe0e/0x2270 [lustre] [11776.545219] ? do_raw_spin_unlock+0x75/0x190 [11776.547239] ? _raw_spin_unlock+0x12/0x30 [11776.549235] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11776.551086] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11776.553110] ll_atomic_open+0x36a/0xd70 [lustre] [11776.556407] lookup_open+0x356/0xb40 [11776.558426] do_last+0x313/0xfc0 [11776.559761] ? nd_jump_root+0xe5/0x160 [11776.561137] ? path_init+0x437/0x520 [11776.563815] path_openat+0xf7/0x500 [11776.565354] ? do_raw_spin_unlock+0x75/0x190 [11776.567013] do_filp_open+0x99/0x140 [11776.568818] ? getname_flags+0x6e/0x330 [11776.570191] ? __check_object_size+0xff/0x256 [11776.572054] ? do_raw_spin_unlock+0x75/0x190 [11776.573776] ? _raw_spin_unlock+0x12/0x30 [11776.575277] do_sys_openat2+0x2b4/0x410 [11776.576919] do_sys_open+0x73/0xa0 [11776.578241] __x64_sys_openat+0x24/0x30 [11776.580167] do_syscall_64+0xc1/0x440 [11776.581496] entry_SYSCALL_64_after_hwframe+0x49/0xae [11776.583270] RIP: 0033:0x7f53425a5332 [11776.584808] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11776.592525] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11776.594982] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11776.597648] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11776.600115] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11776.602870] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11776.605234] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11781.609062] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9144s! [11781.611049] Showing busy workqueues and worker pools: [11781.613529] workqueue events: flags=0x0 [11781.615252] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11781.620532] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11781.631619] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11781.634179] pending: cpu_latency_work [ptlrpc] [11781.638047] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11781.641633] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11781.646761] workqueue events_long: flags=0x0 [11781.648610] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11781.651994] pending: lu_objects_flush [obdclass] [11781.654557] workqueue events_unbound: flags=0x2 [11781.657167] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11781.660219] pending: flush_memcg_stats_dwork, idle_cull_fn [11781.663056] workqueue events_power_efficient: flags=0x80 [11781.665643] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11781.669837] pending: check_lifetime, do_cache_clean [11781.672699] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11781.675843] pending: neigh_periodic_work, neigh_periodic_work [11781.679120] workqueue mm_percpu_wq: flags=0x8 [11781.681519] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11781.685345] pending: vmstat_update, lru_add_drain_per_cpu [11781.688116] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11781.691604] pending: vmstat_update, lru_add_drain_per_cpu [11781.694691] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11781.697601] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11781.700114] workqueue xprtiod: flags=0xa [11781.702189] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11781.706057] pending: xprt_autoclose, xs_error_handle [11781.709778] workqueue ipv6_addrconf: flags=0x40008 [11781.712193] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11781.715479] pending: addrconf_verify_work [11781.717626] workqueue knbd0-recv: flags=0x1a [11781.720116] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11781.723636] in-flight: 651:recv_work [11781.725972] workqueue ptlrpc_pinger: flags=0x2 [11781.728184] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11781.734946] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11781.738369] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9144s workers=4 idle: 15979 31824 8 [11781.743966] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11772s workers=2 idle: 654 [11788.059313] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11788.065267] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11788.094159] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11788.104483] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11788.117612] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11788.126047] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11788.148276] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11788.156066] RAX: 0000000000002a0d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11788.158189] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11788.164694] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11788.168562] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11788.172429] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11788.175509] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11788.180083] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11788.182885] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11788.185831] Call Trace: [11788.186710] [11788.187466] ? show_regs.cold.9+0x22/0x2f [11788.189390] ? watchdog_timer_fn.cold.6+0x62/0x10a [11788.191512] ? __hrtimer_run_queues+0x1c2/0x4d0 [11788.194565] ? watchdog+0x40/0x40 [11788.196467] ? hrtimer_interrupt+0x128/0x2c0 [11788.198993] ? smp_apic_timer_interrupt+0xa6/0x270 [11788.202947] ? apic_timer_interrupt+0xf/0x20 [11788.204815] [11788.205616] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11788.214447] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11788.222932] do_raw_spin_lock+0x177/0x190 [11788.224422] _raw_spin_lock+0x21/0x30 [11788.230722] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11788.233997] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11788.236989] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11788.240448] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11788.243506] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11788.257627] ? woken_wake_function+0x30/0x30 [11788.260514] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11788.264264] kthread+0x1d1/0x200 [11788.265081] ? set_kthread_struct+0x70/0x70 [11788.266954] ret_from_fork+0x1f/0x30 [11804.058032] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11804.060666] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11804.064066] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11804.064143] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11804.064150] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11804.064153] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11804.064199] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11804.064205] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11804.064210] RAX: 0000000000005d6e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11804.064212] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11804.064214] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11804.064217] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11804.064219] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11804.064222] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11804.072257] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11804.072333] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11804.072340] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11804.072343] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11804.072357] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11804.072362] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11804.072366] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11804.072369] RDX: 00000000000019ba RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11804.072371] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11804.072373] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11804.072376] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11804.072379] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11804.072382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11804.072385] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11804.072395] Call Trace: [11804.073505] [11804.073517] ? show_regs.cold.9+0x22/0x2f [11804.073531] ? watchdog_timer_fn.cold.6+0x62/0x10a [11804.073537] ? __hrtimer_run_queues+0x1c2/0x4d0 [11804.073541] ? watchdog+0x40/0x40 [11804.073545] ? hrtimer_interrupt+0x128/0x2c0 [11804.073549] ? smp_apic_timer_interrupt+0xa6/0x270 [11804.073558] ? apic_timer_interrupt+0xf/0x20 [11804.073562] [11804.073565] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11804.073570] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11804.073574] do_raw_spin_lock+0x177/0x190 [11804.073579] _raw_spin_lock+0x21/0x30 [11804.073585] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11804.074202] obd_mod_free+0x46/0x140 [ptlrpc] [11804.074405] mdc_commit_open+0x91/0xb0 [mdc] [11804.074981] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11804.076229] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11804.076505] after_reply+0xe2c/0x1780 [ptlrpc] [11804.076700] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11804.111265] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11804.111275] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11804.111284] Call Trace: [11804.111291] [11804.111296] ? show_regs.cold.9+0x22/0x2f [11804.111310] ? watchdog_timer_fn.cold.6+0x62/0x10a [11804.111317] ? __hrtimer_run_queues+0x1c2/0x4d0 [11804.111322] ? watchdog+0x40/0x40 [11804.111325] ? hrtimer_interrupt+0x128/0x2c0 [11804.111328] ? smp_apic_timer_interrupt+0xa6/0x270 [11804.111337] ? apic_timer_interrupt+0xf/0x20 [11804.111341] [11804.111343] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11804.111353] do_raw_spin_lock+0x177/0x190 [11804.111358] _raw_spin_lock+0x21/0x30 [11804.111362] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11804.111948] ? do_raw_spin_unlock+0x75/0x190 [11804.111954] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11804.112237] ? do_wait_intr+0xf0/0xf0 [11804.112246] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11804.112523] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11804.112953] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11804.123347] ? __next_timer_interrupt+0x160/0x160 [11804.123372] ptlrpcd+0xa82/0xe40 [ptlrpc] [11804.123670] ? do_wait_intr+0xf0/0xf0 [11804.123680] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11804.123892] kthread+0x1d1/0x200 [11804.123898] ? set_kthread_struct+0x70/0x70 [11804.123902] ret_from_fork+0x1f/0x30 [11804.665505] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11804.679445] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11804.687343] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11804.693619] mdc_intent_lock+0x353/0x7e0 [mdc] [11804.697546] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11804.701576] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11804.706550] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11804.708979] lmv_intent_open+0x338/0x1150 [lmv] [11804.713480] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11804.717414] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11804.722358] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11804.724510] ll_intent_lock+0xc5/0xaa0 [lustre] [11804.727311] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11804.730419] ll_lookup_it+0xe0e/0x2270 [lustre] [11804.737311] ? do_raw_spin_unlock+0x75/0x190 [11804.738818] ? _raw_spin_unlock+0x12/0x30 [11804.739929] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11804.742794] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11804.746021] ll_atomic_open+0x36a/0xd70 [lustre] [11804.748409] lookup_open+0x356/0xb40 [11804.755292] do_last+0x313/0xfc0 [11804.757338] ? nd_jump_root+0xe5/0x160 [11804.760287] ? path_init+0x437/0x520 [11804.761753] path_openat+0xf7/0x500 [11804.762889] ? do_raw_spin_unlock+0x75/0x190 [11804.773846] do_filp_open+0x99/0x140 [11804.775660] ? getname_flags+0x6e/0x330 [11804.778286] ? __check_object_size+0xff/0x256 [11804.780678] ? do_raw_spin_unlock+0x75/0x190 [11804.785077] ? _raw_spin_unlock+0x12/0x30 [11804.788299] do_sys_openat2+0x2b4/0x410 [11804.789976] do_sys_open+0x73/0xa0 [11804.797171] __x64_sys_openat+0x24/0x30 [11804.799233] do_syscall_64+0xc1/0x440 [11804.800959] entry_SYSCALL_64_after_hwframe+0x49/0xae [11804.803782] RIP: 0033:0x7f53425a5332 [11804.806162] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11804.818872] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11804.823935] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11804.825570] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11804.842517] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11804.848712] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11804.856616] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11808.052011] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [11808.055733] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11808.079058] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11808.085886] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11808.091535] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11808.094891] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11808.098070] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11808.106222] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11808.110065] RAX: 0000000000006d9d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11808.113842] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11808.116984] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11808.120265] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11808.123817] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11808.127373] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11808.131711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11808.134614] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11808.137844] Call Trace: [11808.138995] [11808.140146] ? show_regs.cold.9+0x22/0x2f [11808.142132] ? watchdog_timer_fn.cold.6+0x62/0x10a [11808.144985] ? __hrtimer_run_queues+0x1c2/0x4d0 [11808.147426] ? watchdog+0x40/0x40 [11808.149178] ? hrtimer_interrupt+0x128/0x2c0 [11808.151348] ? smp_apic_timer_interrupt+0xa6/0x270 [11808.153333] ? apic_timer_interrupt+0xf/0x20 [11808.155628] [11808.156797] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11808.159216] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11808.161891] ? newidle_balance.isra.93+0x207/0x580 [11808.165287] do_raw_spin_lock+0x177/0x190 [11808.166886] _raw_spin_lock+0x21/0x30 [11808.168725] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11808.188957] process_one_work+0x2c8/0x700 [11808.192963] worker_thread+0x296/0x6e0 [11808.195009] ? rescuer_thread+0x570/0x570 [11808.197630] kthread+0x1d1/0x200 [11808.200221] ? set_kthread_struct+0x70/0x70 [11808.205848] ret_from_fork+0x1f/0x30 [11811.809734] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9174s! [11811.812489] Showing busy workqueues and worker pools: [11811.825366] workqueue events: flags=0x0 [11811.828321] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11811.832394] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11811.841163] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11811.845240] pending: cpu_latency_work [ptlrpc] [11811.853071] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11811.855155] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11811.859252] workqueue events_long: flags=0x0 [11811.863245] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11811.869340] pending: lu_objects_flush [obdclass] [11811.872893] workqueue events_unbound: flags=0x2 [11811.875154] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11811.883411] pending: flush_memcg_stats_dwork, idle_cull_fn [11811.885413] workqueue events_power_efficient: flags=0x80 [11811.889074] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11811.895697] pending: check_lifetime, do_cache_clean [11811.897429] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11811.901261] pending: neigh_periodic_work, neigh_periodic_work [11811.904252] workqueue mm_percpu_wq: flags=0x8 [11811.908460] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11811.911928] pending: vmstat_update, lru_add_drain_per_cpu [11811.914382] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11811.921509] pending: vmstat_update, lru_add_drain_per_cpu [11811.928251] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11811.932007] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11811.937925] workqueue xprtiod: flags=0xa [11811.939393] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11811.945420] pending: xprt_autoclose, xs_error_handle [11811.947456] workqueue ipv6_addrconf: flags=0x40008 [11811.954235] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11811.958286] pending: addrconf_verify_work [11811.960226] workqueue knbd0-recv: flags=0x1a [11811.962877] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11811.965620] in-flight: 651:recv_work [11811.967361] workqueue ptlrpc_pinger: flags=0x2 [11811.973143] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11811.977597] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11811.981841] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9174s workers=4 idle: 15979 31824 8 [11811.992237] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11802s workers=2 idle: 654 [11816.058609] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11816.067366] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11816.118104] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11816.129134] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11816.137035] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11816.145062] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11816.164298] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11816.175167] RAX: 0000000000007b3e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11816.183050] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11816.192791] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11816.201358] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11816.206469] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11816.216749] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11816.227455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11816.233961] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11816.239554] Call Trace: [11816.241935] [11816.242688] ? show_regs.cold.9+0x22/0x2f [11816.245163] ? watchdog_timer_fn.cold.6+0x62/0x10a [11816.248244] ? __hrtimer_run_queues+0x1c2/0x4d0 [11816.250388] ? watchdog+0x40/0x40 [11816.251766] ? hrtimer_interrupt+0x128/0x2c0 [11816.254271] ? smp_apic_timer_interrupt+0xa6/0x270 [11816.256396] ? apic_timer_interrupt+0xf/0x20 [11816.258519] [11816.259678] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11816.262375] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11816.264247] do_raw_spin_lock+0x177/0x190 [11816.267728] _raw_spin_lock+0x21/0x30 [11816.269099] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11816.280425] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11816.286428] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11816.293009] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11816.297536] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11816.299459] ? woken_wake_function+0x30/0x30 [11816.308054] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11816.310366] kthread+0x1d1/0x200 [11816.317432] ? set_kthread_struct+0x70/0x70 [11816.323791] ret_from_fork+0x1f/0x30 [11832.055020] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11832.057646] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11832.059124] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11832.090488] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11832.090508] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11832.090511] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11832.090526] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11832.090530] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11832.090535] RAX: 000000000000598b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11832.090538] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11832.090540] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11832.090542] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11832.090544] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11832.090547] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11832.090551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11832.090553] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11832.090561] Call Trace: [11832.090567] [11832.090573] ? show_regs.cold.9+0x22/0x2f [11832.090583] ? watchdog_timer_fn.cold.6+0x62/0x10a [11832.090589] ? __hrtimer_run_queues+0x1c2/0x4d0 [11832.090594] ? watchdog+0x40/0x40 [11832.090598] ? hrtimer_interrupt+0x128/0x2c0 [11832.090601] ? smp_apic_timer_interrupt+0xa6/0x270 [11832.090609] ? apic_timer_interrupt+0xf/0x20 [11832.090613] [11832.090615] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11832.090623] do_raw_spin_lock+0x177/0x190 [11832.090628] _raw_spin_lock+0x21/0x30 [11832.090632] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11832.091049] ? do_raw_spin_unlock+0x75/0x190 [11832.091056] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11832.091415] ? do_wait_intr+0xf0/0xf0 [11832.091425] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11832.093629] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11832.093928] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11832.097833] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11832.098051] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11832.098060] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11832.098063] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11832.098081] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11832.098085] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11832.098090] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11832.098093] RDX: 00000000000045fe RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11832.098095] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11832.098098] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11832.098101] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11832.098104] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11832.098107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11832.098109] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11832.098118] Call Trace: [11832.098124] [11832.098131] ? show_regs.cold.9+0x22/0x2f [11832.098142] ? watchdog_timer_fn.cold.6+0x62/0x10a [11832.098148] ? __hrtimer_run_queues+0x1c2/0x4d0 [11832.098232] ? watchdog+0x40/0x40 [11832.098236] ? hrtimer_interrupt+0x128/0x2c0 [11832.098240] ? smp_apic_timer_interrupt+0xa6/0x270 [11832.098248] ? apic_timer_interrupt+0xf/0x20 [11832.098257] [11832.098261] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11832.098265] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11832.098270] do_raw_spin_lock+0x177/0x190 [11832.098274] _raw_spin_lock+0x21/0x30 [11832.098280] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11832.104699] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11832.108092] obd_mod_free+0x46/0x140 [ptlrpc] [11832.110689] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11832.110751] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11832.110782] mdc_intent_lock+0x353/0x7e0 [mdc] [11832.110811] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11832.110934] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11832.118048] mdc_commit_open+0x91/0xb0 [mdc] [11832.118101] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11832.118564] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11832.118771] after_reply+0xe2c/0x1780 [ptlrpc] [11832.118975] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11832.119436] ? __next_timer_interrupt+0x160/0x160 [11832.119448] ptlrpcd+0xa82/0xe40 [ptlrpc] [11832.119683] ? do_wait_intr+0xf0/0xf0 [11832.119693] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11832.119914] kthread+0x1d1/0x200 [11832.119919] ? set_kthread_struct+0x70/0x70 [11832.119923] ret_from_fork+0x1f/0x30 [11832.480149] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11832.483182] lmv_intent_open+0x338/0x1150 [lmv] [11832.486001] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11832.492152] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11832.496632] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11832.500136] ll_intent_lock+0xc5/0xaa0 [lustre] [11832.505529] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11832.515176] ll_lookup_it+0xe0e/0x2270 [lustre] [11832.519204] ? do_raw_spin_unlock+0x75/0x190 [11832.522113] ? _raw_spin_unlock+0x12/0x30 [11832.523851] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11832.532488] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11832.537259] ll_atomic_open+0x36a/0xd70 [lustre] [11832.539884] lookup_open+0x356/0xb40 [11832.541026] do_last+0x313/0xfc0 [11832.542750] ? nd_jump_root+0xe5/0x160 [11832.545194] ? path_init+0x437/0x520 [11832.551839] path_openat+0xf7/0x500 [11832.554712] ? do_raw_spin_unlock+0x75/0x190 [11832.558124] do_filp_open+0x99/0x140 [11832.561874] ? getname_flags+0x6e/0x330 [11832.564871] ? __check_object_size+0xff/0x256 [11832.567489] ? do_raw_spin_unlock+0x75/0x190 [11832.569820] ? _raw_spin_unlock+0x12/0x30 [11832.574693] do_sys_openat2+0x2b4/0x410 [11832.576171] do_sys_open+0x73/0xa0 [11832.577651] __x64_sys_openat+0x24/0x30 [11832.581132] do_syscall_64+0xc1/0x440 [11832.582671] entry_SYSCALL_64_after_hwframe+0x49/0xae [11832.586846] RIP: 0033:0x7f53425a5332 [11832.589039] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11832.603837] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11832.608025] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11832.618650] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11832.626959] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11832.631809] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11832.636063] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11836.055008] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [11836.065306] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11836.088102] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11836.093045] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11836.096946] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11836.099837] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11836.102622] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11836.109656] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [11836.112246] RAX: 0000000000002306 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11836.114801] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11836.117903] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11836.120878] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11836.124059] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11836.127042] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11836.129852] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11836.131844] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11836.135202] Call Trace: [11836.136401] [11836.137234] ? show_regs.cold.9+0x22/0x2f [11836.139012] ? watchdog_timer_fn.cold.6+0x62/0x10a [11836.140814] ? __hrtimer_run_queues+0x1c2/0x4d0 [11836.142093] ? watchdog+0x40/0x40 [11836.143502] ? hrtimer_interrupt+0x128/0x2c0 [11836.145009] ? smp_apic_timer_interrupt+0xa6/0x270 [11836.146915] ? apic_timer_interrupt+0xf/0x20 [11836.148723] [11836.149624] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11836.152025] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11836.154218] ? newidle_balance.isra.93+0x207/0x580 [11836.155970] do_raw_spin_lock+0x177/0x190 [11836.157699] _raw_spin_lock+0x21/0x30 [11836.158977] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11836.162000] process_one_work+0x2c8/0x700 [11836.163783] worker_thread+0x296/0x6e0 [11836.165715] ? rescuer_thread+0x570/0x570 [11836.167997] kthread+0x1d1/0x200 [11836.169366] ? set_kthread_struct+0x70/0x70 [11836.171348] ret_from_fork+0x1f/0x30 [11842.018909] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9204s! [11842.025451] Showing busy workqueues and worker pools: [11842.027440] workqueue events: flags=0x0 [11842.035352] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11842.040750] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11842.057223] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11842.060124] pending: cpu_latency_work [ptlrpc] [11842.065872] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11842.068622] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11842.077298] workqueue events_long: flags=0x0 [11842.079536] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11842.081377] pending: lu_objects_flush [obdclass] [11842.086959] workqueue events_unbound: flags=0x2 [11842.089866] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11842.093923] pending: flush_memcg_stats_dwork, idle_cull_fn [11842.097630] workqueue events_power_efficient: flags=0x80 [11842.099502] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11842.102050] pending: check_lifetime, do_cache_clean [11842.103392] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11842.113541] pending: neigh_periodic_work, neigh_periodic_work [11842.118233] workqueue mm_percpu_wq: flags=0x8 [11842.121531] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11842.130579] pending: vmstat_update, lru_add_drain_per_cpu [11842.142736] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11842.157823] pending: vmstat_update, lru_add_drain_per_cpu [11842.159888] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11842.176417] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11842.187721] workqueue xprtiod: flags=0xa [11842.192264] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11842.196802] pending: xprt_autoclose, xs_error_handle [11842.201836] workqueue ipv6_addrconf: flags=0x40008 [11842.208258] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11842.212545] pending: addrconf_verify_work [11842.215683] workqueue knbd0-recv: flags=0x1a [11842.221785] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11842.229398] in-flight: 651:recv_work [11842.231477] workqueue ptlrpc_pinger: flags=0x2 [11842.235395] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11842.241054] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11842.247339] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9205s workers=4 idle: 15979 31824 8 [11842.251591] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11832s workers=2 idle: 654 [11844.059370] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11844.063718] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11844.086283] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11844.090690] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11844.094454] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11844.096351] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11844.102903] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11844.107183] RAX: 0000000000003791 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11844.111013] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11844.113836] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11844.116551] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11844.119325] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11844.124764] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11844.127972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11844.130280] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11844.132875] Call Trace: [11844.134928] [11844.136309] ? show_regs.cold.9+0x22/0x2f [11844.137782] ? watchdog_timer_fn.cold.6+0x62/0x10a [11844.139667] ? __hrtimer_run_queues+0x1c2/0x4d0 [11844.143349] ? watchdog+0x40/0x40 [11844.144633] ? hrtimer_interrupt+0x128/0x2c0 [11844.145917] ? smp_apic_timer_interrupt+0xa6/0x270 [11844.148304] ? apic_timer_interrupt+0xf/0x20 [11844.149681] [11844.150482] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11844.152491] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11844.154125] do_raw_spin_lock+0x177/0x190 [11844.155757] _raw_spin_lock+0x21/0x30 [11844.157081] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11844.159978] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11844.161936] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11844.164011] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11844.166535] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11844.168703] ? woken_wake_function+0x30/0x30 [11844.170193] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11844.172329] kthread+0x1d1/0x200 [11844.173760] ? set_kthread_struct+0x70/0x70 [11844.175364] ret_from_fork+0x1f/0x30 [11860.056673] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11860.063095] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11860.069029] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11860.069107] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11860.069114] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11860.069117] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11860.069130] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11860.069135] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11860.069139] RAX: 00000000000066a9 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11860.069141] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11860.069143] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11860.069145] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11860.069147] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11860.069150] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11860.069153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11860.070149] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11860.070169] Call Trace: [11860.070175] [11860.070182] ? show_regs.cold.9+0x22/0x2f [11860.070193] ? watchdog_timer_fn.cold.6+0x62/0x10a [11860.070199] ? __hrtimer_run_queues+0x1c2/0x4d0 [11860.070202] ? watchdog+0x40/0x40 [11860.070207] ? hrtimer_interrupt+0x128/0x2c0 [11860.070211] ? smp_apic_timer_interrupt+0xa6/0x270 [11860.070219] ? apic_timer_interrupt+0xf/0x20 [11860.070224] [11860.070227] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11860.070236] do_raw_spin_lock+0x177/0x190 [11860.070242] _raw_spin_lock+0x21/0x30 [11860.070247] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11860.070677] ? do_raw_spin_unlock+0x75/0x190 [11860.071105] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11860.071984] ? do_wait_intr+0xf0/0xf0 [11860.071995] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11860.072209] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11860.073086] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11860.073517] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11860.073778] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11860.073815] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11860.073916] mdc_intent_lock+0x353/0x7e0 [mdc] [11860.073952] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11860.074094] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11860.076004] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11860.076116] lmv_intent_open+0x338/0x1150 [lmv] [11860.076146] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11860.112715] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11860.112795] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11860.112803] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11860.112806] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11860.112820] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11860.112825] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11860.112840] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11860.112843] RDX: 00000000000015dd RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11860.112845] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11860.112847] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11860.112849] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11860.112852] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11860.112855] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11860.112857] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11860.112866] Call Trace: [11860.112872] [11860.112877] ? show_regs.cold.9+0x22/0x2f [11860.112887] ? watchdog_timer_fn.cold.6+0x62/0x10a [11860.112895] ? __hrtimer_run_queues+0x1c2/0x4d0 [11860.112899] ? watchdog+0x40/0x40 [11860.112904] ? hrtimer_interrupt+0x128/0x2c0 [11860.112908] ? smp_apic_timer_interrupt+0xa6/0x270 [11860.112929] ? apic_timer_interrupt+0xf/0x20 [11860.112934] [11860.112937] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11860.112942] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11860.112947] do_raw_spin_lock+0x177/0x190 [11860.112952] _raw_spin_lock+0x21/0x30 [11860.112959] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11860.113256] obd_mod_free+0x46/0x140 [ptlrpc] [11860.113463] mdc_commit_open+0x91/0xb0 [mdc] [11860.113499] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11860.113710] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11860.113927] after_reply+0xe2c/0x1780 [ptlrpc] [11860.114121] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11860.114310] ? __next_timer_interrupt+0x160/0x160 [11860.114319] ptlrpcd+0xa82/0xe40 [ptlrpc] [11860.114519] ? do_wait_intr+0xf0/0xf0 [11860.114528] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11860.114727] kthread+0x1d1/0x200 [11860.114732] ? set_kthread_struct+0x70/0x70 [11860.114736] ret_from_fork+0x1f/0x30 [11860.810430] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11860.814412] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11860.815735] ll_intent_lock+0xc5/0xaa0 [lustre] [11860.820613] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11860.826368] ll_lookup_it+0xe0e/0x2270 [lustre] [11860.828440] ? do_raw_spin_unlock+0x75/0x190 [11860.832402] ? _raw_spin_unlock+0x12/0x30 [11860.835404] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11860.839475] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11860.842157] ll_atomic_open+0x36a/0xd70 [lustre] [11860.844054] lookup_open+0x356/0xb40 [11860.845733] do_last+0x313/0xfc0 [11860.848556] ? nd_jump_root+0xe5/0x160 [11860.850640] ? path_init+0x437/0x520 [11860.852122] path_openat+0xf7/0x500 [11860.855589] ? do_raw_spin_unlock+0x75/0x190 [11860.863186] do_filp_open+0x99/0x140 [11860.864435] ? getname_flags+0x6e/0x330 [11860.869973] ? __check_object_size+0xff/0x256 [11860.871886] ? do_raw_spin_unlock+0x75/0x190 [11860.875479] ? _raw_spin_unlock+0x12/0x30 [11860.877247] do_sys_openat2+0x2b4/0x410 [11860.879219] do_sys_open+0x73/0xa0 [11860.880570] __x64_sys_openat+0x24/0x30 [11860.882260] do_syscall_64+0xc1/0x440 [11860.889216] entry_SYSCALL_64_after_hwframe+0x49/0xae [11860.898493] RIP: 0033:0x7f53425a5332 [11860.899644] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11860.916187] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11860.921143] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11860.929359] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11860.936586] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11860.941410] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11860.946615] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11868.055560] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [11868.070058] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11868.099604] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11868.115778] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11868.123901] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11868.130872] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11868.137267] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11868.152160] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11868.156760] RAX: 000000000000751c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11868.159884] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11868.163004] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11868.166275] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11868.169564] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11868.172752] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11868.176243] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11868.178773] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11868.181995] Call Trace: [11868.183248] [11868.184228] ? show_regs.cold.9+0x22/0x2f [11868.186129] ? watchdog_timer_fn.cold.6+0x62/0x10a [11868.188160] ? __hrtimer_run_queues+0x1c2/0x4d0 [11868.190177] ? watchdog+0x40/0x40 [11868.191547] ? hrtimer_interrupt+0x128/0x2c0 [11868.193555] ? smp_apic_timer_interrupt+0xa6/0x270 [11868.195890] ? apic_timer_interrupt+0xf/0x20 [11868.197911] [11868.198751] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11868.201197] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11868.203602] ? newidle_balance.isra.93+0x207/0x580 [11868.205880] do_raw_spin_lock+0x177/0x190 [11868.210961] _raw_spin_lock+0x21/0x30 [11868.214676] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11868.220554] process_one_work+0x2c8/0x700 [11868.222289] worker_thread+0x296/0x6e0 [11868.224228] ? rescuer_thread+0x570/0x570 [11868.226346] kthread+0x1d1/0x200 [11868.227750] ? set_kthread_struct+0x70/0x70 [11868.231614] ret_from_fork+0x1f/0x30 [11872.061189] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11872.067404] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11872.126381] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11872.131196] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11872.150075] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11872.155006] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11872.175779] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11872.179495] RAX: 000000000000740e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11872.184559] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11872.187687] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11872.191527] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11872.194766] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11872.197561] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11872.201237] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11872.208145] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11872.210204] Call Trace: [11872.217439] [11872.218315] ? show_regs.cold.9+0x22/0x2f [11872.220918] ? watchdog_timer_fn.cold.6+0x62/0x10a [11872.226763] ? __hrtimer_run_queues+0x1c2/0x4d0 [11872.229740] ? watchdog+0x40/0x40 [11872.231125] ? hrtimer_interrupt+0x128/0x2c0 [11872.235650] ? smp_apic_timer_interrupt+0xa6/0x270 [11872.238730] ? apic_timer_interrupt+0xf/0x20 [11872.240512] [11872.241245] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11872.243816] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11872.246442] do_raw_spin_lock+0x177/0x190 [11872.248123] _raw_spin_lock+0x21/0x30 [11872.249890] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11872.252861] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11872.255377] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11872.257896] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11872.260491] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11872.263067] ? woken_wake_function+0x30/0x30 [11872.266166] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11872.270476] kthread+0x1d1/0x200 [11872.272434] ? set_kthread_struct+0x70/0x70 [11872.279022] ret_from_fork+0x1f/0x30 [11872.754140] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9235s! [11872.761369] Showing busy workqueues and worker pools: [11872.769384] workqueue events: flags=0x0 [11872.778753] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11872.781111] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11872.789477] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11872.793461] pending: cpu_latency_work [ptlrpc] [11872.796351] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11872.798249] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11872.801570] workqueue events_long: flags=0x0 [11872.802997] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11872.805139] pending: lu_objects_flush [obdclass] [11872.806937] workqueue events_unbound: flags=0x2 [11872.808402] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11872.811149] pending: flush_memcg_stats_dwork, idle_cull_fn [11872.813467] workqueue events_power_efficient: flags=0x80 [11872.815841] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11872.818004] pending: check_lifetime, do_cache_clean [11872.820142] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11872.823492] pending: neigh_periodic_work, neigh_periodic_work [11872.833997] workqueue mm_percpu_wq: flags=0x8 [11872.836684] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11872.840798] pending: vmstat_update, lru_add_drain_per_cpu [11872.845277] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11872.847572] pending: vmstat_update, lru_add_drain_per_cpu [11872.851305] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11872.857192] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11872.866290] workqueue xprtiod: flags=0xa [11872.868123] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11872.871429] pending: xprt_autoclose, xs_error_handle [11872.878162] workqueue ipv6_addrconf: flags=0x40008 [11872.882979] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11872.887235] pending: addrconf_verify_work [11872.890993] workqueue knbd0-recv: flags=0x1a [11872.895460] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11872.908420] in-flight: 651:recv_work [11872.917432] workqueue ptlrpc_pinger: flags=0x2 [11872.925896] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11872.932297] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11872.937202] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9235s workers=4 idle: 15979 31824 8 [11872.942119] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11863s workers=2 idle: 654 [11874.661009] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [11874.663566] rcu: 1-....: (5270207 ticks this GP) idle=732/1/0x4000000000000000 softirq=131051/131051 fqs=96983 [11874.668738] rcu: 2-....: (5398705 ticks this GP) idle=b22/1/0x4000000000000000 softirq=137717/137717 fqs=96983 [11874.672882] (detected by 0, t=9780246 jiffies, g=274185, q=66312) [11874.675091] Sending NMI from CPU 0 to CPUs 1: [11874.688420] NMI backtrace for cpu 1 [11874.688427] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11874.688429] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11874.688431] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11874.688433] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11874.688435] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [11874.688438] RAX: 000000000000538f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11874.688439] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11874.688441] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11874.688443] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11874.688444] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11874.688445] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11874.688447] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11874.688449] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11874.688450] Call Trace: [11874.688451] [11874.688452] ? show_regs.cold.9+0x22/0x2f [11874.688453] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [11874.688455] ? nmi_cpu_backtrace_handler+0x15/0x40 [11874.688456] ? nmi_handle+0xbd/0x210 [11874.688457] ? default_do_nmi+0x4b/0x1b0 [11874.688458] ? do_nmi+0x280/0x380 [11874.688459] ? end_repeat_nmi+0x16/0x1f [11874.688460] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11874.688462] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11874.688463] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11874.688464] [11874.688465] do_raw_spin_lock+0x177/0x190 [11874.688466] _raw_spin_lock+0x21/0x30 [11874.688467] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11874.688469] ? do_raw_spin_unlock+0x75/0x190 [11874.688470] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11874.688471] ? do_wait_intr+0xf0/0xf0 [11874.688472] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11874.688473] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11874.688475] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11874.688476] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11874.688477] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11874.688478] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11874.688479] mdc_intent_lock+0x353/0x7e0 [mdc] [11874.688481] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11874.688483] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11874.688484] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11874.688485] lmv_intent_open+0x338/0x1150 [lmv] [11874.688486] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11874.688487] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11874.688489] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11874.688490] ll_intent_lock+0xc5/0xaa0 [lustre] [11874.688492] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11874.688493] ll_lookup_it+0xe0e/0x2270 [lustre] [11874.688494] ? do_raw_spin_unlock+0x75/0x190 [11874.688495] ? _raw_spin_unlock+0x12/0x30 [11874.688497] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11874.688498] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11874.688499] ll_atomic_open+0x36a/0xd70 [lustre] [11874.688501] lookup_open+0x356/0xb40 [11874.688502] do_last+0x313/0xfc0 [11874.688503] ? nd_jump_root+0xe5/0x160 [11874.688504] ? path_init+0x437/0x520 [11874.688505] path_openat+0xf7/0x500 [11874.688506] ? do_raw_spin_unlock+0x75/0x190 [11874.688507] do_filp_open+0x99/0x140 [11874.688508] ? getname_flags+0x6e/0x330 [11874.688509] ? __check_object_size+0xff/0x256 [11874.688510] ? do_raw_spin_unlock+0x75/0x190 [11874.688511] ? _raw_spin_unlock+0x12/0x30 [11874.688512] do_sys_openat2+0x2b4/0x410 [11874.688514] do_sys_open+0x73/0xa0 [11874.688515] __x64_sys_openat+0x24/0x30 [11874.688516] do_syscall_64+0xc1/0x440 [11874.688517] entry_SYSCALL_64_after_hwframe+0x49/0xae [11874.688518] RIP: 0033:0x7f53425a5332 [11874.688520] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11874.688522] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11874.688525] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11874.688527] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11874.688529] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11874.688530] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11874.688532] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11874.688878] Sending NMI from CPU 0 to CPUs 2: [11874.929525] NMI backtrace for cpu 2 [11874.929532] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11874.929535] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11874.929537] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11874.929538] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11874.929540] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 [11874.929543] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11874.929545] RDX: 0000000000007f80 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11874.929546] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11874.929548] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11874.929550] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11874.929551] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11874.929553] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11874.929555] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11874.929557] Call Trace: [11874.929557] [11874.929558] ? show_regs.cold.9+0x22/0x2f [11874.929560] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [11874.929560] ? nmi_cpu_backtrace_handler+0x15/0x40 [11874.929562] ? nmi_handle+0xbd/0x210 [11874.929562] ? default_do_nmi+0x4b/0x1b0 [11874.929563] ? do_nmi+0x280/0x380 [11874.929564] ? end_repeat_nmi+0x16/0x1f [11874.929565] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11874.929566] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11874.929567] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11874.929569] [11874.929580] do_raw_spin_lock+0x177/0x190 [11874.929581] _raw_spin_lock+0x21/0x30 [11874.929582] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11874.929583] obd_mod_free+0x46/0x140 [ptlrpc] [11874.929585] mdc_commit_open+0x91/0xb0 [mdc] [11874.929586] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11874.929587] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11874.929589] after_reply+0xe2c/0x1780 [ptlrpc] [11874.929590] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11874.929591] ? __next_timer_interrupt+0x160/0x160 [11874.929593] ptlrpcd+0xa82/0xe40 [ptlrpc] [11874.929594] ? do_wait_intr+0xf0/0xf0 [11874.929595] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11874.929596] kthread+0x1d1/0x200 [11874.929597] ? set_kthread_struct+0x70/0x70 [11874.929599] ret_from_fork+0x1f/0x30 [11874.930837] rcu: rcu_sched kthread starved for 9238862 jiffies! g274185 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [11875.066216] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [11875.070916] rcu: RCU grace-period kthread stack dump: [11875.072186] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [11875.084209] Call Trace: [11875.086267] __schedule+0x351/0xcb0 [11875.088825] schedule+0xc0/0x180 [11875.090775] schedule_timeout+0xb4/0x190 [11875.092697] ? do_raw_spin_unlock+0x75/0x190 [11875.096284] ? __next_timer_interrupt+0x160/0x160 [11875.097918] ? prepare_to_swait_event+0xf3/0x140 [11875.102728] rcu_gp_kthread+0x8dd/0x15b0 [11875.105348] ? force_qs_rnp+0x2c0/0x2c0 [11875.108658] kthread+0x1d1/0x200 [11875.109796] ? set_kthread_struct+0x70/0x70 [11875.111212] ret_from_fork+0x1f/0x30 [11875.120475] rcu: Stack dump where RCU GP kthread last ran: [11875.124868] Sending NMI from CPU 0 to CPUs 2: [11875.133088] NMI backtrace for cpu 2 [11875.133096] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11875.133098] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11875.133099] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11875.133101] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11875.133103] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 [11875.133106] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11875.133107] RDX: 00000000000053e1 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11875.133108] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11875.133110] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11875.133111] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11875.133113] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11875.133114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11875.133116] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11875.133117] Call Trace: [11875.133118] [11875.133119] ? show_regs.cold.9+0x22/0x2f [11875.133120] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [11875.133122] ? nmi_cpu_backtrace_handler+0x15/0x40 [11875.133123] ? nmi_handle+0xbd/0x210 [11875.133124] ? default_do_nmi+0x4b/0x1b0 [11875.133125] ? do_nmi+0x280/0x380 [11875.133126] ? end_repeat_nmi+0x16/0x1f [11875.133127] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11875.133128] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11875.133130] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11875.133131] [11875.133132] do_raw_spin_lock+0x177/0x190 [11875.133133] _raw_spin_lock+0x21/0x30 [11875.133134] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11875.133135] obd_mod_free+0x46/0x140 [ptlrpc] [11875.133136] mdc_commit_open+0x91/0xb0 [mdc] [11875.133137] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11875.133138] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11875.133139] after_reply+0xe2c/0x1780 [ptlrpc] [11875.133140] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11875.133142] ? __next_timer_interrupt+0x160/0x160 [11875.133143] ptlrpcd+0xa82/0xe40 [ptlrpc] [11875.133144] ? do_wait_intr+0xf0/0xf0 [11875.133145] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11875.133146] kthread+0x1d1/0x200 [11875.133147] ? set_kthread_struct+0x70/0x70 [11875.133148] ret_from_fork+0x1f/0x30 [11900.052152] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [11900.055061] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11900.057192] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11900.057203] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11900.057289] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11900.057294] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11900.057297] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11900.057311] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11900.057315] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11900.057319] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11900.057322] RDX: 0000000000003060 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11900.057324] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11900.057326] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11900.057329] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11900.057333] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11900.057336] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11900.057339] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11900.057348] Call Trace: [11900.057353] [11900.057359] ? show_regs.cold.9+0x22/0x2f [11900.057367] ? watchdog_timer_fn.cold.6+0x62/0x10a [11900.057372] ? __hrtimer_run_queues+0x1c2/0x4d0 [11900.057375] ? watchdog+0x40/0x40 [11900.057377] ? hrtimer_interrupt+0x128/0x2c0 [11900.057380] ? smp_apic_timer_interrupt+0xa6/0x270 [11900.057387] ? apic_timer_interrupt+0xf/0x20 [11900.057390] [11900.057392] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11900.057395] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11900.057397] do_raw_spin_lock+0x177/0x190 [11900.057400] _raw_spin_lock+0x21/0x30 [11900.057405] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11900.058029] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11900.058036] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11900.058092] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11900.058096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11900.058098] RIP: 0010:native_safe_halt+0x2e/0x50 [11900.058106] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11900.058109] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11900.058113] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11900.058115] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11900.058117] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11900.058119] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11900.058121] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11900.058123] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11900.058126] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11900.058128] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11900.058143] Call Trace: [11900.058146] [11900.058149] ? show_regs.cold.9+0x22/0x2f [11900.058156] ? watchdog_timer_fn.cold.6+0x62/0x10a [11900.058160] ? __hrtimer_run_queues+0x1c2/0x4d0 [11900.058164] ? watchdog+0x40/0x40 [11900.058168] ? hrtimer_interrupt+0x128/0x2c0 [11900.058171] ? smp_apic_timer_interrupt+0xa6/0x270 [11900.058176] ? apic_timer_interrupt+0xf/0x20 [11900.058181] [11900.058184] ? native_safe_halt+0x2e/0x50 [11900.058188] kvm_wait+0xcd/0xe0 [11900.058195] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11900.058201] do_raw_spin_lock+0x177/0x190 [11900.058206] _raw_spin_lock+0x21/0x30 [11900.058285] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl [11900.058210] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11900.058325] i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11900.058347] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11900.058352] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11900.058357] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11900.058025] obd_mod_free+0x46/0x140 [ptlrpc] [11900.059736] mdc_commit_open+0x91/0xb0 [mdc] [11900.059771] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11900.060533] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11900.059588] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11900.061052] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11900.061287] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11900.060812] after_reply+0xe2c/0x1780 [ptlrpc] [11900.063338] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11900.063608] ? woken_wake_function+0x30/0x30 [11900.063618] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11900.065083] kthread+0x1d1/0x200 [11900.065091] ? set_kthread_struct+0x70/0x70 [11900.065095] ret_from_fork+0x1f/0x30 [11900.065861] [11900.065866] RIP: 0010:native_safe_halt+0x2e/0x50 [11900.065875] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11900.065878] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11900.065883] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11900.065885] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11900.065887] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11900.065889] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11900.065891] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11900.065895] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11900.065897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11900.065909] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11900.065916] Call Trace: [11900.065920] [11900.065924] ? show_regs.cold.9+0x22/0x2f [11900.065933] ? watchdog_timer_fn.cold.6+0x62/0x10a [11900.065939] ? __hrtimer_run_queues+0x1c2/0x4d0 [11900.065943] ? watchdog+0x40/0x40 [11900.065947] ? hrtimer_interrupt+0x128/0x2c0 [11900.065951] ? smp_apic_timer_interrupt+0xa6/0x270 [11900.065957] ? apic_timer_interrupt+0xf/0x20 [11900.065962] [11900.065965] ? native_safe_halt+0x2e/0x50 [11900.065969] kvm_wait+0xcd/0xe0 [11900.065976] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11900.065982] ? newidle_balance.isra.93+0x207/0x580 [11900.065988] do_raw_spin_lock+0x177/0x190 [11900.065992] _raw_spin_lock+0x21/0x30 [11900.065996] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11900.066259] process_one_work+0x2c8/0x700 [11900.066267] worker_thread+0x296/0x6e0 [11900.066272] ? rescuer_thread+0x570/0x570 [11900.066275] kthread+0x1d1/0x200 [11900.066279] ? set_kthread_struct+0x70/0x70 [11900.066283] ret_from_fork+0x1f/0x30 [11900.066437] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11900.066893] ? __next_timer_interrupt+0x160/0x160 [11900.066903] ptlrpcd+0xa82/0xe40 [ptlrpc] [11900.067422] ? do_wait_intr+0xf0/0xf0 [11900.067431] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11900.067700] kthread+0x1d1/0x200 [11900.067706] ? set_kthread_struct+0x70/0x70 [11900.067710] ret_from_fork+0x1f/0x30 [11901.034746] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11901.070709] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11901.075894] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11901.086695] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11901.097918] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11901.116185] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11901.127534] RAX: 0000000000006892 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11901.132579] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11901.136093] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11901.156098] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11901.164723] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11901.172810] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11901.181594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11901.189765] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11901.197207] Call Trace: [11901.199421] [11901.202416] ? show_regs.cold.9+0x22/0x2f [11901.204180] ? watchdog_timer_fn.cold.6+0x62/0x10a [11901.208757] ? __hrtimer_run_queues+0x1c2/0x4d0 [11901.215480] ? watchdog+0x40/0x40 [11901.217807] ? hrtimer_interrupt+0x128/0x2c0 [11901.219636] ? smp_apic_timer_interrupt+0xa6/0x270 [11901.220812] ? apic_timer_interrupt+0xf/0x20 [11901.225273] [11901.226433] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11901.228902] do_raw_spin_lock+0x177/0x190 [11901.233259] _raw_spin_lock+0x21/0x30 [11901.235474] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11901.238654] ? do_raw_spin_unlock+0x75/0x190 [11901.243417] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11901.246319] ? do_wait_intr+0xf0/0xf0 [11901.248292] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11901.254642] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11901.258045] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11901.264735] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11901.267230] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11901.273718] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11901.278526] mdc_intent_lock+0x353/0x7e0 [mdc] [11901.280950] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11901.286756] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11901.295769] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11901.299857] lmv_intent_open+0x338/0x1150 [lmv] [11901.304691] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11901.315322] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11901.317706] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11901.321376] ll_intent_lock+0xc5/0xaa0 [lustre] [11901.323833] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11901.329691] ll_lookup_it+0xe0e/0x2270 [lustre] [11901.331235] ? do_raw_spin_unlock+0x75/0x190 [11901.339878] ? _raw_spin_unlock+0x12/0x30 [11901.342594] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11901.345563] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11901.353526] ll_atomic_open+0x36a/0xd70 [lustre] [11901.359137] lookup_open+0x356/0xb40 [11901.361630] do_last+0x313/0xfc0 [11901.367615] ? nd_jump_root+0xe5/0x160 [11901.371856] ? path_init+0x437/0x520 [11901.372918] path_openat+0xf7/0x500 [11901.376710] ? do_raw_spin_unlock+0x75/0x190 [11901.381058] do_filp_open+0x99/0x140 [11901.384933] ? getname_flags+0x6e/0x330 [11901.387668] ? __check_object_size+0xff/0x256 [11901.390790] ? do_raw_spin_unlock+0x75/0x190 [11901.392019] ? _raw_spin_unlock+0x12/0x30 [11901.394728] do_sys_openat2+0x2b4/0x410 [11901.395833] do_sys_open+0x73/0xa0 [11901.398879] __x64_sys_openat+0x24/0x30 [11901.401587] do_syscall_64+0xc1/0x440 [11901.402553] entry_SYSCALL_64_after_hwframe+0x49/0xae [11901.404418] RIP: 0033:0x7f53425a5332 [11901.406542] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11901.416602] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11901.420590] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11901.424000] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11901.429387] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11901.431985] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11901.436388] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11903.456459] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9266s! [11903.463793] Showing busy workqueues and worker pools: [11903.470766] workqueue events: flags=0x0 [11903.473968] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11903.476800] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11903.488921] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11903.493538] pending: cpu_latency_work [ptlrpc] [11903.495772] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11903.498575] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11903.504261] workqueue events_long: flags=0x0 [11903.505883] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11903.508813] pending: lu_objects_flush [obdclass] [11903.511323] workqueue events_unbound: flags=0x2 [11903.513197] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11903.515894] pending: flush_memcg_stats_dwork, idle_cull_fn [11903.518662] workqueue events_power_efficient: flags=0x80 [11903.521383] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11903.524439] pending: check_lifetime, do_cache_clean [11903.526670] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11903.529468] pending: neigh_periodic_work, neigh_periodic_work [11903.531799] workqueue mm_percpu_wq: flags=0x8 [11903.533587] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11903.538278] pending: vmstat_update, lru_add_drain_per_cpu [11903.553744] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11903.569849] pending: vmstat_update, lru_add_drain_per_cpu [11903.577802] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11903.593453] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11903.599810] workqueue xprtiod: flags=0xa [11903.606714] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11903.613249] pending: xprt_autoclose, xs_error_handle [11903.617142] workqueue ipv6_addrconf: flags=0x40008 [11903.620432] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11903.626284] pending: addrconf_verify_work [11903.628335] workqueue knbd0-recv: flags=0x1a [11903.630467] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11903.632817] in-flight: 651:recv_work [11903.634696] workqueue ptlrpc_pinger: flags=0x2 [11903.637530] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11903.646605] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11903.654669] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9266s workers=4 idle: 15979 31824 8 [11903.667281] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11894s workers=2 idle: 654 [11928.054797] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [11928.057222] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11928.058700] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11928.058709] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11928.058789] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11928.058795] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11928.058797] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11928.058811] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11928.058815] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11928.058819] RAX: 0000000000007e0e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11928.058821] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11928.058823] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11928.058825] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11928.058827] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11928.058831] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11928.058833] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11928.058835] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11928.058844] Call Trace: [11928.058850] [11928.058856] ? show_regs.cold.9+0x22/0x2f [11928.058865] ? watchdog_timer_fn.cold.6+0x62/0x10a [11928.058871] ? __hrtimer_run_queues+0x1c2/0x4d0 [11928.058875] ? watchdog+0x40/0x40 [11928.058880] ? hrtimer_interrupt+0x128/0x2c0 [11928.058884] ? smp_apic_timer_interrupt+0xa6/0x270 [11928.058892] ? apic_timer_interrupt+0xf/0x20 [11928.058897] [11928.058900] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11928.058905] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11928.058910] do_raw_spin_lock+0x177/0x190 [11928.058916] _raw_spin_lock+0x21/0x30 [11928.058921] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11928.059211] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11928.059418] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11928.062752] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11928.063010] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11928.063236] ? woken_wake_function+0x30/0x30 [11928.063289] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11928.063297] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11928.063245] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11928.063366] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11928.063373] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11928.063376] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11928.063386] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11928.063390] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11928.063395] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11928.063398] RDX: 0000000000007e1e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11928.063400] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11928.063403] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11928.063406] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11928.063409] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11928.063412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11928.063415] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11928.063423] Call Trace: [11928.063428] [11928.063430] ? show_regs.cold.9+0x22/0x2f [11928.063441] ? watchdog_timer_fn.cold.6+0x62/0x10a [11928.063446] ? __hrtimer_run_queues+0x1c2/0x4d0 [11928.063451] ? watchdog+0x40/0x40 [11928.063455] ? hrtimer_interrupt+0x128/0x2c0 [11928.063459] ? smp_apic_timer_interrupt+0xa6/0x270 [11928.063466] ? apic_timer_interrupt+0xf/0x20 [11928.063472] [11928.063474] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11928.063480] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11928.063483] kthread+0x1d1/0x200 [11928.063488] do_raw_spin_lock+0x177/0x190 [11928.063493] ? set_kthread_struct+0x70/0x70 [11928.063497] _raw_spin_lock+0x21/0x30 [11928.063502] ret_from_fork+0x1f/0x30 [11928.063507] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11928.063715] obd_mod_free+0x46/0x140 [ptlrpc] [11928.064029] mdc_commit_open+0x91/0xb0 [mdc] [11928.064055] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11928.064122] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11928.064127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11928.064133] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11928.064071] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11928.064627] [11928.064629] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11928.064651] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11928.064656] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11928.064660] RAX: 0000000000001d0d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11928.064662] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11928.064665] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11928.064667] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11928.064670] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11928.064673] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11928.064676] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11928.064678] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11928.064685] Call Trace: [11928.064759] [11928.064763] ? show_regs.cold.9+0x22/0x2f [11928.064770] ? watchdog_timer_fn.cold.6+0x62/0x10a [11928.064775] ? __hrtimer_run_queues+0x1c2/0x4d0 [11928.064779] ? watchdog+0x40/0x40 [11928.064783] ? hrtimer_interrupt+0x128/0x2c0 [11928.064787] ? smp_apic_timer_interrupt+0xa6/0x270 [11928.064794] ? apic_timer_interrupt+0xf/0x20 [11928.064799] [11928.064801] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11928.064805] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11928.064809] ? newidle_balance.isra.93+0x207/0x580 [11928.064814] do_raw_spin_lock+0x177/0x190 [11928.064818] _raw_spin_lock+0x21/0x30 [11928.064823] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11928.065283] process_one_work+0x2c8/0x700 [11928.065292] worker_thread+0x296/0x6e0 [11928.065296] ? rescuer_thread+0x570/0x570 [11928.065301] kthread+0x1d1/0x200 [11928.065305] ? set_kthread_struct+0x70/0x70 [11928.065309] ret_from_fork+0x1f/0x30 [11928.065416] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11928.065672] after_reply+0xe2c/0x1780 [ptlrpc] [11928.065863] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11928.066054] ? __next_timer_interrupt+0x160/0x160 [11928.066062] ptlrpcd+0xa82/0xe40 [ptlrpc] [11928.069258] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11928.069329] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11928.069337] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11928.069340] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11928.069353] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11928.069357] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11928.069362] RAX: 0000000000007b86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11928.069365] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11928.069368] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11928.069370] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11928.069372] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11928.069375] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11928.069378] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11928.069380] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11928.069390] Call Trace: [11928.069396] [11928.069402] ? show_regs.cold.9+0x22/0x2f [11928.069413] ? watchdog_timer_fn.cold.6+0x62/0x10a [11928.069420] ? __hrtimer_run_queues+0x1c2/0x4d0 [11928.069425] ? watchdog+0x40/0x40 [11928.069429] ? hrtimer_interrupt+0x128/0x2c0 [11928.069432] ? smp_apic_timer_interrupt+0xa6/0x270 [11928.069440] ? apic_timer_interrupt+0xf/0x20 [11928.069445] [11928.069448] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11928.069456] do_raw_spin_lock+0x177/0x190 [11928.069461] _raw_spin_lock+0x21/0x30 [11928.069466] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11928.073889] ? do_raw_spin_unlock+0x75/0x190 [11928.073903] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11928.079914] ? do_wait_intr+0xf0/0xf0 [11928.109332] ? do_wait_intr+0xf0/0xf0 [11928.116054] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11928.116416] kthread+0x1d1/0x200 [11928.116425] ? set_kthread_struct+0x70/0x70 [11928.116430] ret_from_fork+0x1f/0x30 [11928.679250] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11928.683769] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11928.686206] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11928.688222] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11928.696574] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11928.699387] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11928.705701] mdc_intent_lock+0x353/0x7e0 [mdc] [11928.711441] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11928.713481] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11928.722347] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11928.726700] lmv_intent_open+0x338/0x1150 [lmv] [11928.729593] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11928.734442] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11928.741117] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11928.744215] ll_intent_lock+0xc5/0xaa0 [lustre] [11928.751581] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11928.755862] ll_lookup_it+0xe0e/0x2270 [lustre] [11928.759518] ? do_raw_spin_unlock+0x75/0x190 [11928.765203] ? _raw_spin_unlock+0x12/0x30 [11928.767571] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11928.774013] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11928.779725] ll_atomic_open+0x36a/0xd70 [lustre] [11928.782446] lookup_open+0x356/0xb40 [11928.785621] do_last+0x313/0xfc0 [11928.791043] ? nd_jump_root+0xe5/0x160 [11928.797896] ? path_init+0x437/0x520 [11928.801920] path_openat+0xf7/0x500 [11928.804409] ? do_raw_spin_unlock+0x75/0x190 [11928.808418] do_filp_open+0x99/0x140 [11928.814707] ? getname_flags+0x6e/0x330 [11928.817606] ? __check_object_size+0xff/0x256 [11928.821448] ? do_raw_spin_unlock+0x75/0x190 [11928.827121] ? _raw_spin_unlock+0x12/0x30 [11928.832657] do_sys_openat2+0x2b4/0x410 [11928.833862] do_sys_open+0x73/0xa0 [11928.840618] __x64_sys_openat+0x24/0x30 [11928.842862] do_syscall_64+0xc1/0x440 [11928.846133] entry_SYSCALL_64_after_hwframe+0x49/0xae [11928.848193] RIP: 0033:0x7f53425a5332 [11928.850577] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11928.859964] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11928.864129] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11928.867972] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11928.870973] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11928.874369] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11928.877454] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11934.176252] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9297s! [11934.188222] Showing busy workqueues and worker pools: [11934.193778] workqueue events: flags=0x0 [11934.197320] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11934.203056] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11934.207446] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11934.209762] pending: cpu_latency_work [ptlrpc] [11934.215128] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11934.222354] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11934.233010] workqueue events_long: flags=0x0 [11934.234398] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11934.239632] pending: lu_objects_flush [obdclass] [11934.241018] workqueue events_unbound: flags=0x2 [11934.243380] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11934.247956] pending: flush_memcg_stats_dwork, idle_cull_fn [11934.250176] workqueue events_power_efficient: flags=0x80 [11934.255089] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11934.261518] pending: check_lifetime, do_cache_clean [11934.265160] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11934.273872] pending: neigh_periodic_work, neigh_periodic_work [11934.281281] workqueue mm_percpu_wq: flags=0x8 [11934.287279] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11934.291991] pending: vmstat_update, lru_add_drain_per_cpu [11934.298265] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11934.307363] pending: vmstat_update, lru_add_drain_per_cpu [11934.309410] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11934.331182] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11934.344900] workqueue xprtiod: flags=0xa [11934.351299] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11934.362145] pending: xprt_autoclose, xs_error_handle [11934.368894] workqueue ipv6_addrconf: flags=0x40008 [11934.376340] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11934.385193] pending: addrconf_verify_work [11934.393679] workqueue knbd0-recv: flags=0x1a [11934.398616] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11934.417141] in-flight: 651:recv_work [11934.422641] workqueue ptlrpc_pinger: flags=0x2 [11934.430155] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11934.438927] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11934.446169] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9297s workers=4 idle: 15979 31824 8 [11934.458048] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11924s workers=2 idle: 654 [11956.057364] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [11956.058422] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [11956.058504] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [11956.058531] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11956.058610] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11956.058616] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11956.058620] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11956.058639] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11956.058645] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11956.058649] RAX: 0000000000007a8b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11956.058652] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11956.058654] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11956.058657] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11956.058660] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11956.058663] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11956.058666] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11956.058668] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11956.058677] Call Trace: [11956.058684] [11956.058693] ? show_regs.cold.9+0x22/0x2f [11956.058706] ? watchdog_timer_fn.cold.6+0x62/0x10a [11956.058715] ? __hrtimer_run_queues+0x1c2/0x4d0 [11956.058720] ? watchdog+0x40/0x40 [11956.058725] ? hrtimer_interrupt+0x128/0x2c0 [11956.058729] ? smp_apic_timer_interrupt+0xa6/0x270 [11956.058740] ? apic_timer_interrupt+0xf/0x20 [11956.058745] [11956.058749] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11956.058753] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11956.058760] do_raw_spin_lock+0x177/0x190 [11956.058764] _raw_spin_lock+0x21/0x30 [11956.058769] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11956.059119] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11956.059326] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11956.059544] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11956.059755] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11956.059954] ? woken_wake_function+0x30/0x30 [11956.059962] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11956.060133] kthread+0x1d1/0x200 [11956.060140] ? set_kthread_struct+0x70/0x70 [11956.060144] ret_from_fork+0x1f/0x30 [11956.062880] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11956.062956] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11956.062964] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11956.062967] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11956.062981] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11956.062985] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11956.062989] RAX: 0000000000000e81 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11956.062991] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11956.062993] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11956.062995] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11956.062998] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11956.063001] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11956.063004] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11956.063006] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11956.063017] Call Trace: [11956.063022] [11956.063028] ? show_regs.cold.9+0x22/0x2f [11956.063037] ? watchdog_timer_fn.cold.6+0x62/0x10a [11956.063043] ? __hrtimer_run_queues+0x1c2/0x4d0 [11956.063047] ? watchdog+0x40/0x40 [11956.063051] ? hrtimer_interrupt+0x128/0x2c0 [11956.063056] ? smp_apic_timer_interrupt+0xa6/0x270 [11956.063065] ? apic_timer_interrupt+0xf/0x20 [11956.063071] [11956.063073] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11956.063080] do_raw_spin_lock+0x177/0x190 [11956.063085] _raw_spin_lock+0x21/0x30 [11956.063090] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11956.065202] ? do_raw_spin_unlock+0x75/0x190 [11956.065211] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11956.065770] ? do_wait_intr+0xf0/0xf0 [11956.065780] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11956.065984] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11956.066210] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11956.074985] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11956.075376] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11956.075385] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11956.075388] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11956.075401] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11956.075405] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11956.075410] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11956.075413] RDX: 0000000000004691 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11956.075416] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11956.075418] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11956.075421] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11956.075424] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11956.075427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11956.075429] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11956.075438] Call Trace: [11956.075444] [11956.075450] ? show_regs.cold.9+0x22/0x2f [11956.075460] ? watchdog_timer_fn.cold.6+0x62/0x10a [11956.075470] ? __hrtimer_run_queues+0x1c2/0x4d0 [11956.075475] ? watchdog+0x40/0x40 [11956.075480] ? hrtimer_interrupt+0x128/0x2c0 [11956.075484] ? smp_apic_timer_interrupt+0xa6/0x270 [11956.075493] ? apic_timer_interrupt+0xf/0x20 [11956.075498] [11956.075501] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11956.075506] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11956.075510] do_raw_spin_lock+0x177/0x190 [11956.075514] _raw_spin_lock+0x21/0x30 [11956.075520] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11956.075779] obd_mod_free+0x46/0x140 [ptlrpc] [11956.075972] mdc_commit_open+0x91/0xb0 [mdc] [11956.078361] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11956.078603] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11956.078800] after_reply+0xe2c/0x1780 [ptlrpc] [11956.078994] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11956.079203] ? __next_timer_interrupt+0x160/0x160 [11956.079214] ptlrpcd+0xa82/0xe40 [ptlrpc] [11956.079415] ? do_wait_intr+0xf0/0xf0 [11956.079425] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11956.079631] kthread+0x1d1/0x200 [11956.079637] ? set_kthread_struct+0x70/0x70 [11956.079641] ret_from_fork+0x1f/0x30 [11956.573544] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11956.575131] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11956.577114] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11956.578660] mdc_intent_lock+0x353/0x7e0 [mdc] [11956.580071] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11956.581927] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11956.583879] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11956.585772] lmv_intent_open+0x338/0x1150 [lmv] [11956.587016] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11956.588815] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11956.590373] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11956.591935] ll_intent_lock+0xc5/0xaa0 [lustre] [11956.593465] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11956.594840] ll_lookup_it+0xe0e/0x2270 [lustre] [11956.599239] ? do_raw_spin_unlock+0x75/0x190 [11956.603118] ? _raw_spin_unlock+0x12/0x30 [11956.605087] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11956.607250] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11956.609592] ll_atomic_open+0x36a/0xd70 [lustre] [11956.612120] lookup_open+0x356/0xb40 [11956.613787] do_last+0x313/0xfc0 [11956.615096] ? nd_jump_root+0xe5/0x160 [11956.617036] ? path_init+0x437/0x520 [11956.619083] path_openat+0xf7/0x500 [11956.620806] ? do_raw_spin_unlock+0x75/0x190 [11956.622100] do_filp_open+0x99/0x140 [11956.623488] ? getname_flags+0x6e/0x330 [11956.624954] ? __check_object_size+0xff/0x256 [11956.627502] ? do_raw_spin_unlock+0x75/0x190 [11956.629601] ? _raw_spin_unlock+0x12/0x30 [11956.631580] do_sys_openat2+0x2b4/0x410 [11956.633537] do_sys_open+0x73/0xa0 [11956.635283] __x64_sys_openat+0x24/0x30 [11956.636858] do_syscall_64+0xc1/0x440 [11956.638323] entry_SYSCALL_64_after_hwframe+0x49/0xae [11956.640471] RIP: 0033:0x7f53425a5332 [11956.641980] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11956.656074] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11956.660919] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11956.665947] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11956.669808] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11956.679471] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11956.685405] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11960.052110] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [11960.057445] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11960.093236] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11960.102798] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11960.110952] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11960.115657] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11960.134260] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11960.157395] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11960.162266] RAX: 0000000000007507 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11960.170149] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11960.172591] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11960.175861] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11960.181333] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11960.186676] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11960.190821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11960.193620] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11960.197816] Call Trace: [11960.198786] [11960.200276] ? show_regs.cold.9+0x22/0x2f [11960.202415] ? watchdog_timer_fn.cold.6+0x62/0x10a [11960.207469] ? __hrtimer_run_queues+0x1c2/0x4d0 [11960.208782] ? watchdog+0x40/0x40 [11960.209974] ? hrtimer_interrupt+0x128/0x2c0 [11960.214385] ? smp_apic_timer_interrupt+0xa6/0x270 [11960.216371] ? apic_timer_interrupt+0xf/0x20 [11960.218231] [11960.218960] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11960.224726] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11960.227169] ? newidle_balance.isra.93+0x207/0x580 [11960.230626] do_raw_spin_lock+0x177/0x190 [11960.232594] _raw_spin_lock+0x21/0x30 [11960.238532] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11960.241044] process_one_work+0x2c8/0x700 [11960.243376] worker_thread+0x296/0x6e0 [11960.245526] ? rescuer_thread+0x570/0x570 [11960.248313] kthread+0x1d1/0x200 [11960.249918] ? set_kthread_struct+0x70/0x70 [11960.252055] ret_from_fork+0x1f/0x30 [11964.908702] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9327s! [11964.916941] Showing busy workqueues and worker pools: [11964.918758] workqueue events: flags=0x0 [11964.921094] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11964.925074] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11964.930438] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11964.934151] pending: cpu_latency_work [ptlrpc] [11964.936537] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11964.939530] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11964.943660] workqueue events_long: flags=0x0 [11964.945788] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11964.948505] pending: lu_objects_flush [obdclass] [11964.950393] workqueue events_unbound: flags=0x2 [11964.951755] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11964.953519] pending: flush_memcg_stats_dwork, idle_cull_fn [11964.955578] workqueue events_power_efficient: flags=0x80 [11964.957527] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11964.959579] pending: check_lifetime, do_cache_clean [11964.961277] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11964.963684] pending: neigh_periodic_work, neigh_periodic_work [11964.966525] workqueue mm_percpu_wq: flags=0x8 [11964.968424] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11964.970937] pending: vmstat_update, lru_add_drain_per_cpu [11964.973356] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11964.975538] pending: vmstat_update, lru_add_drain_per_cpu [11964.977249] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11964.979695] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11964.982331] workqueue xprtiod: flags=0xa [11964.983757] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11964.985957] pending: xprt_autoclose, xs_error_handle [11964.987738] workqueue ipv6_addrconf: flags=0x40008 [11964.989805] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11964.992064] pending: addrconf_verify_work [11964.993347] workqueue knbd0-recv: flags=0x1a [11964.994730] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11964.996466] in-flight: 651:recv_work [11964.998126] workqueue ptlrpc_pinger: flags=0x2 [11964.999423] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11965.001122] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11965.003956] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9327s workers=4 idle: 15979 31824 8 [11965.008344] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11955s workers=2 idle: 654 [11984.056934] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [11984.057094] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [11984.063996] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [11984.064010] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw [11984.064086] Modules linked in: [11984.064087] libata dm_mirror [11984.064092] lustre(O) [11984.064096] dm_region_hash [11984.064099] osp(O) [11984.064100] dm_log [11984.064104] ofd(O) [11984.064106] dm_mod [11984.064110] lod(O) [11984.064111] sha512_ssse3 [11984.064115] mdt(O) [11984.064117] sha512_generic [11984.064119] mdd(O) [11984.064123] [11984.064124] mgs(O) lquota(O) [11984.064129] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11984.064132] lfsck(O) obdecho(O) [11984.064141] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11984.064142] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) [11984.064153] RIP: 0010:native_safe_halt+0x2e/0x50 [11984.064163] fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) [11984.064171] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [11984.064174] obdclass(O) ksocklnd(O) [11984.064180] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 [11984.064181] lnet(O) libcfs(O) [11984.064186] ORIG_RAX: ffffffffffffff13 [11984.064190] rpcsec_gss_krb5 [11984.064192] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11984.064194] auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac [11984.064198] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [11984.064205] rapl i2c_piix4 [11984.064212] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [11984.064213] pcspkr squashfs [11984.064218] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [11984.064221] ata_generic crct10dif_pclmul [11984.064226] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [11984.064229] crc32_pclmul crc32c_intel [11984.064234] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [11984.064236] ghash_clmulni_intel ata_piix [11984.064241] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11984.064243] serio_raw libata [11984.064248] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [11984.064251] dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11984.064261] Call Trace: [11984.064264] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11984.064270] [11984.064272] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11984.064277] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11984.064279] ? show_regs.cold.9+0x22/0x2f [11984.064290] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11984.064297] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [11984.064295] ? watchdog_timer_fn.cold.6+0x62/0x10a [11984.064301] ORIG_RAX: ffffffffffffff13 [11984.064307] RAX: 0000000000003f83 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11984.064307] ? __hrtimer_run_queues+0x1c2/0x4d0 [11984.064312] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [11984.064318] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [11984.064318] ? watchdog+0x40/0x40 [11984.064322] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [11984.064329] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [11984.064327] ? hrtimer_interrupt+0x128/0x2c0 [11984.064333] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [11984.064340] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11984.064338] ? smp_apic_timer_interrupt+0xa6/0x270 [11984.064343] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [11984.064351] ? apic_timer_interrupt+0xf/0x20 [11984.064355] Call Trace: [11984.064361] [11984.064362] [11984.064368] ? native_safe_halt+0x2e/0x50 [11984.064369] ? show_regs.cold.9+0x22/0x2f [11984.064381] ? watchdog_timer_fn.cold.6+0x62/0x10a [11984.064378] kvm_wait+0xcd/0xe0 [11984.064389] ? __hrtimer_run_queues+0x1c2/0x4d0 [11984.064392] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11984.064397] ? watchdog+0x40/0x40 [11984.064404] do_raw_spin_lock+0x177/0x190 [11984.064406] ? hrtimer_interrupt+0x128/0x2c0 [11984.064416] _raw_spin_lock+0x21/0x30 [11984.064417] ? smp_apic_timer_interrupt+0xa6/0x270 [11984.064429] ? apic_timer_interrupt+0xf/0x20 [11984.064434] [11984.064437] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11984.064443] do_raw_spin_lock+0x177/0x190 [11984.064448] _raw_spin_lock+0x21/0x30 [11984.064426] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [11984.065692] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [11984.064453] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [11984.068548] ? do_raw_spin_unlock+0x75/0x190 [11984.068558] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [11984.071112] ? do_wait_intr+0xf0/0xf0 [11984.071123] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [11984.071315] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [11984.071528] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11984.071733] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11984.071836] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11984.071868] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [11984.071897] Modules linked in: lustre(O) osp(O) [11984.071895] mdc_intent_lock+0x353/0x7e0 [mdc] [11984.071908] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata [11984.071924] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11984.071965] dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11984.071976] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11984.071981] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11984.071984] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [11984.071994] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [11984.071998] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [11984.072003] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [11984.072006] RDX: 0000000000001faa RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [11984.072011] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [11984.072013] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [11984.072016] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [11984.072019] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [11984.072022] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11984.072024] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [11984.072031] Call Trace: [11984.072035] [11984.072038] ? show_regs.cold.9+0x22/0x2f [11984.072047] ? watchdog_timer_fn.cold.6+0x62/0x10a [11984.072055] ? __hrtimer_run_queues+0x1c2/0x4d0 [11984.072059] ? watchdog+0x40/0x40 [11984.072062] ? hrtimer_interrupt+0x128/0x2c0 [11984.072066] ? smp_apic_timer_interrupt+0xa6/0x270 [11984.072076] ? apic_timer_interrupt+0xf/0x20 [11984.072080] [11984.072083] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [11984.072088] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [11984.072092] do_raw_spin_lock+0x177/0x190 [11984.072097] _raw_spin_lock+0x21/0x30 [11984.072009] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [11984.072526] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [11984.072556] lmv_intent_open+0x338/0x1150 [lmv] [11984.072584] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11984.072681] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11984.072771] lmv_intent_lock+0x1dd/0x4c0 [lmv] [11984.072794] ll_intent_lock+0xc5/0xaa0 [lustre] [11984.072880] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [11984.072962] ll_lookup_it+0xe0e/0x2270 [lustre] [11984.073034] ? do_raw_spin_unlock+0x75/0x190 [11984.073038] ? _raw_spin_unlock+0x12/0x30 [11984.073042] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [11984.073061] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [11984.073074] ll_atomic_open+0x36a/0xd70 [lustre] [11984.073129] lookup_open+0x356/0xb40 [11984.073136] do_last+0x313/0xfc0 [11984.073140] ? nd_jump_root+0xe5/0x160 [11984.073143] ? path_init+0x437/0x520 [11984.073147] path_openat+0xf7/0x500 [11984.073150] ? do_raw_spin_unlock+0x75/0x190 [11984.073153] do_filp_open+0x99/0x140 [11984.073160] ? getname_flags+0x6e/0x330 [11984.073165] ? __check_object_size+0xff/0x256 [11984.073173] ? do_raw_spin_unlock+0x75/0x190 [11984.073178] ? _raw_spin_unlock+0x12/0x30 [11984.073183] do_sys_openat2+0x2b4/0x410 [11984.073187] do_sys_open+0x73/0xa0 [11984.073192] __x64_sys_openat+0x24/0x30 [11984.073197] do_syscall_64+0xc1/0x440 [11984.073203] entry_SYSCALL_64_after_hwframe+0x49/0xae [11984.073209] RIP: 0033:0x7f53425a5332 [11984.073216] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [11984.073220] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [11984.073225] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [11984.073227] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [11984.073228] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [11984.073230] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [11984.073231] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [11984.072103] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [11984.079085] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [11984.079381] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [11984.140137] obd_mod_free+0x46/0x140 [ptlrpc] [11984.140445] mdc_commit_open+0x91/0xb0 [mdc] [11984.142275] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [11984.144804] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [11984.145064] after_reply+0xe2c/0x1780 [ptlrpc] [11984.145279] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [11984.147324] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [11984.147598] ? woken_wake_function+0x30/0x30 [11984.147609] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [11984.147824] kthread+0x1d1/0x200 [11984.147830] ? set_kthread_struct+0x70/0x70 [11984.147834] ret_from_fork+0x1f/0x30 [11985.104971] ? __next_timer_interrupt+0x160/0x160 [11985.106975] ptlrpcd+0xa82/0xe40 [ptlrpc] [11985.110148] ? do_wait_intr+0xf0/0xf0 [11985.112948] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [11985.116566] kthread+0x1d1/0x200 [11985.118010] ? set_kthread_struct+0x70/0x70 [11985.119748] ret_from_fork+0x1f/0x30 [11992.052847] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [11992.059604] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [11992.104901] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [11992.121594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [11992.131651] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [11992.135636] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11992.138334] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [11992.155717] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [11992.160297] RAX: 0000000000002516 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [11992.166850] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [11992.172996] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [11992.181584] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [11992.186255] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [11992.189490] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [11992.194372] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11992.196999] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [11992.203419] Call Trace: [11992.204901] [11992.207599] ? show_regs.cold.9+0x22/0x2f [11992.210661] ? watchdog_timer_fn.cold.6+0x62/0x10a [11992.215747] ? __hrtimer_run_queues+0x1c2/0x4d0 [11992.217393] ? watchdog+0x40/0x40 [11992.221490] ? hrtimer_interrupt+0x128/0x2c0 [11992.226075] ? smp_apic_timer_interrupt+0xa6/0x270 [11992.228247] ? apic_timer_interrupt+0xf/0x20 [11992.229697] [11992.230244] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [11992.238451] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [11992.243859] ? newidle_balance.isra.93+0x207/0x580 [11992.246245] do_raw_spin_lock+0x177/0x190 [11992.249480] _raw_spin_lock+0x21/0x30 [11992.251745] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [11992.258096] process_one_work+0x2c8/0x700 [11992.261572] worker_thread+0x296/0x6e0 [11992.264009] ? rescuer_thread+0x570/0x570 [11992.268412] kthread+0x1d1/0x200 [11992.270606] ? set_kthread_struct+0x70/0x70 [11992.274980] ret_from_fork+0x1f/0x30 [11995.110539] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9358s! [11995.125718] Showing busy workqueues and worker pools: [11995.127496] workqueue events: flags=0x0 [11995.133611] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [11995.143207] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [11995.152261] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11995.158359] pending: cpu_latency_work [ptlrpc] [11995.166402] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [11995.171945] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [11995.176202] workqueue events_long: flags=0x0 [11995.178091] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [11995.180936] pending: lu_objects_flush [obdclass] [11995.183302] workqueue events_unbound: flags=0x2 [11995.184911] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [11995.187132] pending: flush_memcg_stats_dwork, idle_cull_fn [11995.191993] workqueue events_power_efficient: flags=0x80 [11995.194704] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11995.198057] pending: check_lifetime, do_cache_clean [11995.200679] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11995.204075] pending: neigh_periodic_work, neigh_periodic_work [11995.207669] workqueue mm_percpu_wq: flags=0x8 [11995.209645] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11995.212873] pending: vmstat_update, lru_add_drain_per_cpu [11995.215603] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [11995.223818] pending: vmstat_update, lru_add_drain_per_cpu [11995.229048] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [11995.236470] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [11995.240283] workqueue xprtiod: flags=0xa [11995.243643] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [11995.247355] pending: xprt_autoclose, xs_error_handle [11995.250594] workqueue ipv6_addrconf: flags=0x40008 [11995.253287] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [11995.256953] pending: addrconf_verify_work [11995.259222] workqueue knbd0-recv: flags=0x1a [11995.261093] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [11995.264347] in-flight: 651:recv_work [11995.266382] workqueue ptlrpc_pinger: flags=0x2 [11995.268645] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [11995.271663] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [11995.274931] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9358s workers=4 idle: 15979 31824 8 [11995.277943] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=11985s workers=2 idle: 654 [12012.055011] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [12012.057028] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [12012.058192] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [12012.058201] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12012.058268] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12012.058273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12012.058276] RIP: 0010:native_safe_halt+0x2e/0x50 [12012.058288] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12012.058293] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12012.058299] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12012.058301] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12012.058303] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12012.058305] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12012.058308] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12012.058311] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12012.058313] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12012.058315] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12012.058324] Call Trace: [12012.058330] [12012.058335] ? show_regs.cold.9+0x22/0x2f [12012.058345] ? watchdog_timer_fn.cold.6+0x62/0x10a [12012.058352] ? __hrtimer_run_queues+0x1c2/0x4d0 [12012.058356] ? watchdog+0x40/0x40 [12012.058361] ? hrtimer_interrupt+0x128/0x2c0 [12012.058365] ? smp_apic_timer_interrupt+0xa6/0x270 [12012.058371] ? apic_timer_interrupt+0xf/0x20 [12012.058375] [12012.058378] ? native_safe_halt+0x2e/0x50 [12012.058383] kvm_wait+0xcd/0xe0 [12012.058390] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12012.058399] do_raw_spin_lock+0x177/0x190 [12012.058404] _raw_spin_lock+0x21/0x30 [12012.058408] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12012.058690] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12012.058889] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12012.059013] Modules linked in: lustre(O) [12012.059167] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12012.059401] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12012.059613] ? woken_wake_function+0x30/0x30 [12012.059622] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12012.059833] kthread+0x1d1/0x200 [12012.059840] ? set_kthread_struct+0x70/0x70 [12012.059843] ret_from_fork+0x1f/0x30 [12012.061890] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul [12012.064871] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12012.065000] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12012.065007] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12012.065010] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12012.065023] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12012.065027] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12012.065031] RAX: 00000000000060cc RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12012.065034] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12012.065036] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12012.065039] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12012.065042] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12012.065045] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12012.065049] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12012.065052] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12012.065061] Call Trace: [12012.065066] [12012.065073] ? show_regs.cold.9+0x22/0x2f [12012.065082] ? watchdog_timer_fn.cold.6+0x62/0x10a [12012.065088] ? __hrtimer_run_queues+0x1c2/0x4d0 [12012.065093] ? watchdog+0x40/0x40 [12012.065098] ? hrtimer_interrupt+0x128/0x2c0 [12012.065102] ? smp_apic_timer_interrupt+0xa6/0x270 [12012.065109] ? apic_timer_interrupt+0xf/0x20 [12012.065114] [12012.065117] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12012.065135] do_raw_spin_lock+0x177/0x190 [12012.065141] _raw_spin_lock+0x21/0x30 [12012.065146] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12012.065729] ? do_raw_spin_unlock+0x75/0x190 [12012.065736] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12012.066025] ? do_wait_intr+0xf0/0xf0 [12012.066034] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12012.066322] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12012.066692] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12012.067049] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12012.067255] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12012.067298] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12012.067329] mdc_intent_lock+0x353/0x7e0 [mdc] [12012.067359] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12012.067580] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12012.067828] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12012.067859] lmv_intent_open+0x338/0x1150 [lmv] [12012.067956] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12012.068052] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12012.068221] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12012.068246] ll_intent_lock+0xc5/0xaa0 [lustre] [12012.101006] crc32c_intel [12012.111956] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12012.124278] ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12012.124306] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12012.124313] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12012.124317] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12012.124330] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12012.124335] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12012.124340] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12012.124342] RDX: 0000000000004114 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12012.124345] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12012.124348] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12012.124350] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12012.124354] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12012.124358] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12012.124360] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12012.124370] Call Trace: [12012.124375] [12012.124382] ? show_regs.cold.9+0x22/0x2f [12012.124392] ? watchdog_timer_fn.cold.6+0x62/0x10a [12012.124400] ? __hrtimer_run_queues+0x1c2/0x4d0 [12012.124404] ? watchdog+0x40/0x40 [12012.124409] ? hrtimer_interrupt+0x128/0x2c0 [12012.124413] ? smp_apic_timer_interrupt+0xa6/0x270 [12012.124421] ? apic_timer_interrupt+0xf/0x20 [12012.124426] [12012.124429] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12012.124434] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12012.125242] do_raw_spin_lock+0x177/0x190 [12012.125248] _raw_spin_lock+0x21/0x30 [12012.125256] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12012.127386] obd_mod_free+0x46/0x140 [ptlrpc] [12012.127604] mdc_commit_open+0x91/0xb0 [mdc] [12012.127640] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12012.127840] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12012.128040] after_reply+0xe2c/0x1780 [ptlrpc] [12012.128940] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12012.129139] ? __next_timer_interrupt+0x160/0x160 [12012.129150] ptlrpcd+0xa82/0xe40 [ptlrpc] [12012.129561] ? do_wait_intr+0xf0/0xf0 [12012.129572] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12012.129784] kthread+0x1d1/0x200 [12012.129791] ? set_kthread_struct+0x70/0x70 [12012.129795] ret_from_fork+0x1f/0x30 [12012.841921] ll_lookup_it+0xe0e/0x2270 [lustre] [12012.843522] ? do_raw_spin_unlock+0x75/0x190 [12012.844786] ? _raw_spin_unlock+0x12/0x30 [12012.852180] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12012.857880] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12012.865370] ll_atomic_open+0x36a/0xd70 [lustre] [12012.870687] lookup_open+0x356/0xb40 [12012.873716] do_last+0x313/0xfc0 [12012.880016] ? nd_jump_root+0xe5/0x160 [12012.885464] ? path_init+0x437/0x520 [12012.897472] path_openat+0xf7/0x500 [12012.898867] ? do_raw_spin_unlock+0x75/0x190 [12012.906542] do_filp_open+0x99/0x140 [12012.909574] ? getname_flags+0x6e/0x330 [12012.916974] ? __check_object_size+0xff/0x256 [12012.925446] ? do_raw_spin_unlock+0x75/0x190 [12012.928640] ? _raw_spin_unlock+0x12/0x30 [12012.934005] do_sys_openat2+0x2b4/0x410 [12012.936392] do_sys_open+0x73/0xa0 [12012.941315] __x64_sys_openat+0x24/0x30 [12012.945079] do_syscall_64+0xc1/0x440 [12012.954126] entry_SYSCALL_64_after_hwframe+0x49/0xae [12012.966443] RIP: 0033:0x7f53425a5332 [12012.973381] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12012.996551] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12013.007584] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12013.020051] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12013.029476] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12013.038635] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12013.046889] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12020.053474] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [12020.066840] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12020.121568] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12020.130896] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12020.147543] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12020.165874] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12020.171108] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12020.198351] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12020.214198] RAX: 0000000000004806 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12020.215886] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12020.220157] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12020.229923] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12020.235442] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12020.237805] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12020.250259] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12020.257861] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12020.264842] Call Trace: [12020.265752] [12020.266749] ? show_regs.cold.9+0x22/0x2f [12020.273073] ? watchdog_timer_fn.cold.6+0x62/0x10a [12020.277853] ? __hrtimer_run_queues+0x1c2/0x4d0 [12020.281745] ? watchdog+0x40/0x40 [12020.285741] ? hrtimer_interrupt+0x128/0x2c0 [12020.291996] ? smp_apic_timer_interrupt+0xa6/0x270 [12020.298925] ? apic_timer_interrupt+0xf/0x20 [12020.307194] [12020.309601] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12020.312667] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12020.320861] ? newidle_balance.isra.93+0x207/0x580 [12020.324768] do_raw_spin_lock+0x177/0x190 [12020.326986] _raw_spin_lock+0x21/0x30 [12020.328283] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12020.332937] process_one_work+0x2c8/0x700 [12020.334343] worker_thread+0x296/0x6e0 [12020.342004] ? rescuer_thread+0x570/0x570 [12020.351402] kthread+0x1d1/0x200 [12020.354880] ? set_kthread_struct+0x70/0x70 [12020.356787] ret_from_fork+0x1f/0x30 [12025.317834] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9388s! [12025.326417] Showing busy workqueues and worker pools: [12025.328931] workqueue events: flags=0x0 [12025.335085] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12025.341158] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12025.358033] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12025.371856] pending: cpu_latency_work [ptlrpc] [12025.373560] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12025.383472] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12025.394037] workqueue events_long: flags=0x0 [12025.399621] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12025.408684] pending: lu_objects_flush [obdclass] [12025.411713] workqueue events_unbound: flags=0x2 [12025.419319] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12025.422901] pending: flush_memcg_stats_dwork, idle_cull_fn [12025.431428] workqueue events_power_efficient: flags=0x80 [12025.436817] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12025.445629] pending: check_lifetime, do_cache_clean [12025.448748] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12025.457270] pending: neigh_periodic_work, neigh_periodic_work [12025.461497] workqueue mm_percpu_wq: flags=0x8 [12025.466012] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12025.469113] pending: vmstat_update, lru_add_drain_per_cpu [12025.475168] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12025.479468] pending: vmstat_update, lru_add_drain_per_cpu [12025.482420] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12025.494047] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12025.498862] workqueue xprtiod: flags=0xa [12025.500235] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12025.513865] pending: xprt_autoclose, xs_error_handle [12025.521342] workqueue ipv6_addrconf: flags=0x40008 [12025.525361] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12025.533243] pending: addrconf_verify_work [12025.537761] workqueue knbd0-recv: flags=0x1a [12025.542782] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12025.549202] in-flight: 651:recv_work [12025.554515] workqueue ptlrpc_pinger: flags=0x2 [12025.561349] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12025.566677] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12025.570854] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9388s workers=4 idle: 15979 31824 8 [12025.581906] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12016s workers=2 idle: 654 [12040.063073] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [12040.064924] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [12040.065026] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [12040.065032] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12040.065096] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12040.065103] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12040.065105] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12040.065119] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12040.065123] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12040.065127] RAX: 0000000000006685 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12040.065129] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12040.065131] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12040.065133] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12040.065135] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12040.065138] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12040.065140] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12040.065142] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12040.065150] Call Trace: [12040.065157] [12040.065162] ? show_regs.cold.9+0x22/0x2f [12040.065174] ? watchdog_timer_fn.cold.6+0x62/0x10a [12040.065179] ? __hrtimer_run_queues+0x1c2/0x4d0 [12040.065184] ? watchdog+0x40/0x40 [12040.065188] ? hrtimer_interrupt+0x128/0x2c0 [12040.065192] ? smp_apic_timer_interrupt+0xa6/0x270 [12040.065201] ? apic_timer_interrupt+0xf/0x20 [12040.065205] [12040.065208] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12040.065212] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12040.065217] do_raw_spin_lock+0x177/0x190 [12040.065222] _raw_spin_lock+0x21/0x30 [12040.065227] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12040.065750] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12040.066772] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12040.067667] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12040.069234] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12040.072841] ? woken_wake_function+0x30/0x30 [12040.072851] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12040.073071] kthread+0x1d1/0x200 [12040.073077] ? set_kthread_struct+0x70/0x70 [12040.073081] ret_from_fork+0x1f/0x30 [12040.078801] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12040.078881] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12040.078889] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12040.078892] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12040.078907] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12040.078911] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12040.078915] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12040.078918] RDX: 0000000000003253 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12040.078920] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12040.078922] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12040.078924] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12040.078927] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12040.078930] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12040.078932] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12040.078941] Call Trace: [12040.078946] [12040.078952] ? show_regs.cold.9+0x22/0x2f [12040.078963] ? watchdog_timer_fn.cold.6+0x62/0x10a [12040.078969] ? __hrtimer_run_queues+0x1c2/0x4d0 [12040.078974] ? watchdog+0x40/0x40 [12040.078978] ? hrtimer_interrupt+0x128/0x2c0 [12040.078982] ? smp_apic_timer_interrupt+0xa6/0x270 [12040.078990] ? apic_timer_interrupt+0xf/0x20 [12040.078994] [12040.078998] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12040.079003] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12040.079007] do_raw_spin_lock+0x177/0x190 [12040.079012] _raw_spin_lock+0x21/0x30 [12040.079017] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12040.079319] obd_mod_free+0x46/0x140 [ptlrpc] [12040.079520] mdc_commit_open+0x91/0xb0 [mdc] [12040.079559] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12040.079784] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12040.079976] after_reply+0xe2c/0x1780 [ptlrpc] [12040.088122] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12040.088206] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12040.088215] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12040.088218] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12040.088233] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12040.088237] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12040.088241] RAX: 0000000000005615 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12040.088244] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12040.088247] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12040.088249] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12040.088252] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12040.088255] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12040.088259] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12040.088261] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12040.088272] Call Trace: [12040.088278] [12040.088284] ? show_regs.cold.9+0x22/0x2f [12040.088311] ? watchdog_timer_fn.cold.6+0x62/0x10a [12040.088318] ? __hrtimer_run_queues+0x1c2/0x4d0 [12040.088322] ? watchdog+0x40/0x40 [12040.088327] ? hrtimer_interrupt+0x128/0x2c0 [12040.088331] ? smp_apic_timer_interrupt+0xa6/0x270 [12040.088340] ? apic_timer_interrupt+0xf/0x20 [12040.088345] [12040.088348] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12040.088356] do_raw_spin_lock+0x177/0x190 [12040.088362] _raw_spin_lock+0x21/0x30 [12040.088367] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12040.091633] ? do_raw_spin_unlock+0x75/0x190 [12040.091645] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12040.092207] ? do_wait_intr+0xf0/0xf0 [12040.092218] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12040.100003] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12040.100326] ? __next_timer_interrupt+0x160/0x160 [12040.100337] ptlrpcd+0xa82/0xe40 [ptlrpc] [12040.100551] ? do_wait_intr+0xf0/0xf0 [12040.100561] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12040.100912] kthread+0x1d1/0x200 [12040.100919] ? set_kthread_struct+0x70/0x70 [12040.100924] ret_from_fork+0x1f/0x30 [12040.978641] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12040.981862] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12040.988659] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12040.999845] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12041.001860] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12041.012498] mdc_intent_lock+0x353/0x7e0 [mdc] [12041.015178] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12041.018283] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12041.020202] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12041.030502] lmv_intent_open+0x338/0x1150 [lmv] [12041.035930] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12041.040461] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12041.052054] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12041.061052] ll_intent_lock+0xc5/0xaa0 [lustre] [12041.062890] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12041.074254] ll_lookup_it+0xe0e/0x2270 [lustre] [12041.081155] ? do_raw_spin_unlock+0x75/0x190 [12041.091029] ? _raw_spin_unlock+0x12/0x30 [12041.092353] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12041.096977] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12041.104287] ll_atomic_open+0x36a/0xd70 [lustre] [12041.108312] lookup_open+0x356/0xb40 [12041.111125] do_last+0x313/0xfc0 [12041.113524] ? nd_jump_root+0xe5/0x160 [12041.119339] ? path_init+0x437/0x520 [12041.122405] path_openat+0xf7/0x500 [12041.123643] ? do_raw_spin_unlock+0x75/0x190 [12041.125539] do_filp_open+0x99/0x140 [12041.129232] ? getname_flags+0x6e/0x330 [12041.135434] ? __check_object_size+0xff/0x256 [12041.139865] ? do_raw_spin_unlock+0x75/0x190 [12041.144142] ? _raw_spin_unlock+0x12/0x30 [12041.145489] do_sys_openat2+0x2b4/0x410 [12041.153878] do_sys_open+0x73/0xa0 [12041.166433] __x64_sys_openat+0x24/0x30 [12041.175502] do_syscall_64+0xc1/0x440 [12041.178181] entry_SYSCALL_64_after_hwframe+0x49/0xae [12041.188052] RIP: 0033:0x7f53425a5332 [12041.189420] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12041.223987] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12041.233099] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12041.237239] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12041.249770] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12041.269907] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12041.272230] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12052.053804] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [12052.059894] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12052.084733] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12052.089535] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12052.092989] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12052.095974] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12052.099201] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12052.109604] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12052.114284] RAX: 0000000000006e1d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12052.119169] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12052.123224] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12052.126191] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12052.129199] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12052.132119] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12052.135292] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12052.137526] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12052.140311] Call Trace: [12052.141390] [12052.142133] ? show_regs.cold.9+0x22/0x2f [12052.143893] ? watchdog_timer_fn.cold.6+0x62/0x10a [12052.146443] ? __hrtimer_run_queues+0x1c2/0x4d0 [12052.150103] ? watchdog+0x40/0x40 [12052.152214] ? hrtimer_interrupt+0x128/0x2c0 [12052.154182] ? smp_apic_timer_interrupt+0xa6/0x270 [12052.157412] ? apic_timer_interrupt+0xf/0x20 [12052.159613] [12052.162726] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12052.165002] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12052.167211] ? newidle_balance.isra.93+0x207/0x580 [12052.171050] do_raw_spin_lock+0x177/0x190 [12052.174880] _raw_spin_lock+0x21/0x30 [12052.178223] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12052.181850] process_one_work+0x2c8/0x700 [12052.183634] worker_thread+0x296/0x6e0 [12052.185368] ? rescuer_thread+0x570/0x570 [12052.187053] kthread+0x1d1/0x200 [12052.188482] ? set_kthread_struct+0x70/0x70 [12052.190382] ret_from_fork+0x1f/0x30 [12054.664051] rcu: INFO: rcu_sched self-detected stall on CPU [12054.666162] rcu: 2-....: (5502329 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [12054.669979] (t=9960238 jiffies g=274185 q=66374) [12054.675046] rcu: rcu_sched kthread starved for 9418604 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [12054.691488] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12054.703281] rcu: RCU grace-period kthread stack dump: [12054.706884] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12054.718661] Call Trace: [12054.721758] __schedule+0x351/0xcb0 [12054.726831] schedule+0xc0/0x180 [12054.730412] schedule_timeout+0xb4/0x190 [12054.733100] ? do_raw_spin_unlock+0x75/0x190 [12054.735588] ? __next_timer_interrupt+0x160/0x160 [12054.738550] ? prepare_to_swait_event+0xf3/0x140 [12054.740401] rcu_gp_kthread+0x8dd/0x15b0 [12054.747488] ? force_qs_rnp+0x2c0/0x2c0 [12054.752111] kthread+0x1d1/0x200 [12054.754180] ? set_kthread_struct+0x70/0x70 [12054.756773] ret_from_fork+0x1f/0x30 [12054.763022] rcu: Stack dump where RCU GP kthread last ran: [12054.773229] NMI backtrace for cpu 2 [12054.776933] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12054.785708] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12054.793141] Call Trace: [12054.797000] [12054.799802] ? dump_stack+0xbb/0x10e [12054.803144] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12054.808392] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12054.810627] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12054.819038] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12054.825084] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12054.826855] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [12054.831175] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [12054.837028] ? update_process_times+0x6d/0xc0 [12054.843079] ? tick_sched_handle.isra.18+0x27/0x90 [12054.850146] ? tick_sched_timer+0x53/0xb0 [12054.854020] ? __hrtimer_run_queues+0x1c2/0x4d0 [12054.857004] ? tick_sched_handle.isra.18+0x90/0x90 [12054.859765] ? hrtimer_interrupt+0x128/0x2c0 [12054.863747] ? smp_apic_timer_interrupt+0xa6/0x270 [12054.865899] ? apic_timer_interrupt+0xf/0x20 [12054.867257] [12054.869332] ? native_safe_halt+0x2e/0x50 [12054.873045] ? kvm_wait+0xcd/0xe0 [12054.877151] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12054.887336] ? do_raw_spin_lock+0x177/0x190 [12054.891067] ? _raw_spin_lock+0x21/0x30 [12054.892310] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12054.896569] ? obd_mod_free+0x46/0x140 [ptlrpc] [12054.900984] ? mdc_commit_open+0x91/0xb0 [mdc] [12054.903366] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12054.908362] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12054.920016] ? after_reply+0xe2c/0x1780 [ptlrpc] [12054.924876] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12054.930302] ? __next_timer_interrupt+0x160/0x160 [12054.935966] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12054.938695] ? do_wait_intr+0xf0/0xf0 [12054.939908] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12054.945536] ? kthread+0x1d1/0x200 [12054.947987] ? set_kthread_struct+0x70/0x70 [12054.951638] ? ret_from_fork+0x1f/0x30 [12054.954002] Sending NMI from CPU 2 to CPUs 1: [12054.957447] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [12054.960791] NMI backtrace for cpu 2 [12054.965341] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12054.973494] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12054.980834] Call Trace: [12054.987978] [12054.990821] ? dump_stack+0xbb/0x10e [12054.996769] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12055.003399] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12055.011719] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12055.016129] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12055.023020] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12055.026649] ? rcu_dump_cpu_stacks+0x11b/0x171 [12055.028041] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12055.037719] ? update_process_times+0x6d/0xc0 [12055.041310] ? tick_sched_handle.isra.18+0x27/0x90 [12055.045578] ? tick_sched_timer+0x53/0xb0 [12055.046927] ? __hrtimer_run_queues+0x1c2/0x4d0 [12055.058986] ? tick_sched_handle.isra.18+0x90/0x90 [12055.066573] ? hrtimer_interrupt+0x128/0x2c0 [12055.072507] ? smp_apic_timer_interrupt+0xa6/0x270 [12055.075392] ? apic_timer_interrupt+0xf/0x20 [12055.078306] [12055.079785] ? native_safe_halt+0x2e/0x50 [12055.082340] ? kvm_wait+0xcd/0xe0 [12055.084771] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12055.090392] ? do_raw_spin_lock+0x177/0x190 [12055.094823] ? _raw_spin_lock+0x21/0x30 [12055.097364] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12055.101953] ? obd_mod_free+0x46/0x140 [ptlrpc] [12055.105440] ? mdc_commit_open+0x91/0xb0 [mdc] [12055.109132] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12055.114293] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12055.120266] ? after_reply+0xe2c/0x1780 [ptlrpc] [12055.124885] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12055.128815] ? __next_timer_interrupt+0x160/0x160 [12055.132915] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12055.141769] ? do_wait_intr+0xf0/0xf0 [12055.144749] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12055.151508] ? kthread+0x1d1/0x200 [12055.152907] ? set_kthread_struct+0x70/0x70 [12055.162320] ? ret_from_fork+0x1f/0x30 [12056.039502] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9418s! [12056.051121] Showing busy workqueues and worker pools: [12056.053180] workqueue events: flags=0x0 [12056.054880] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12056.057502] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12056.061250] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12056.064866] pending: cpu_latency_work [ptlrpc] [12056.067540] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12056.070834] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12056.074970] workqueue events_long: flags=0x0 [12056.076710] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12056.079841] pending: lu_objects_flush [obdclass] [12056.082158] workqueue events_unbound: flags=0x2 [12056.083810] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12056.086257] pending: flush_memcg_stats_dwork, idle_cull_fn [12056.088553] workqueue events_power_efficient: flags=0x80 [12056.090922] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12056.093799] pending: check_lifetime, do_cache_clean [12056.096312] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12056.099545] pending: neigh_periodic_work, neigh_periodic_work [12056.101986] workqueue mm_percpu_wq: flags=0x8 [12056.103923] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12056.107681] pending: vmstat_update, lru_add_drain_per_cpu [12056.110743] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12056.113619] pending: vmstat_update, lru_add_drain_per_cpu [12056.115947] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12056.118881] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12056.121666] workqueue xprtiod: flags=0xa [12056.123145] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12056.125905] pending: xprt_autoclose, xs_error_handle [12056.128204] workqueue ipv6_addrconf: flags=0x40008 [12056.130501] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12056.134952] pending: addrconf_verify_work [12056.136846] workqueue knbd0-recv: flags=0x1a [12056.138674] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12056.141364] in-flight: 651:recv_work [12056.143151] workqueue ptlrpc_pinger: flags=0x2 [12056.144985] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12056.147479] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12056.150534] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9419s workers=4 idle: 15979 31824 8 [12056.153889] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12046s workers=2 idle: 654 [12068.059234] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [12068.063219] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12068.115296] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12068.121698] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12068.134579] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12068.142905] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12068.171077] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12068.182922] RAX: 0000000000005f85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12068.188669] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12068.196095] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12068.205649] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12068.212753] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12068.221735] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12068.234724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12068.243591] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12068.258172] Call Trace: [12068.259078] [12068.259779] ? show_regs.cold.9+0x22/0x2f [12068.266203] ? watchdog_timer_fn.cold.6+0x62/0x10a [12068.275680] ? __hrtimer_run_queues+0x1c2/0x4d0 [12068.280804] ? watchdog+0x40/0x40 [12068.283812] ? hrtimer_interrupt+0x128/0x2c0 [12068.289882] ? smp_apic_timer_interrupt+0xa6/0x270 [12068.296023] ? apic_timer_interrupt+0xf/0x20 [12068.306627] [12068.310686] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12068.317176] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12068.331325] do_raw_spin_lock+0x177/0x190 [12068.332768] _raw_spin_lock+0x21/0x30 [12068.342036] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12068.353605] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12068.361117] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12068.367456] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12068.381946] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12068.385444] ? woken_wake_function+0x30/0x30 [12068.386968] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12068.408649] kthread+0x1d1/0x200 [12068.413531] ? set_kthread_struct+0x70/0x70 [12068.428728] ret_from_fork+0x1f/0x30 [12080.061217] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [12080.061541] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [12080.067726] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12080.067801] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12080.067808] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12080.067811] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12080.067824] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12080.067828] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12080.067832] RAX: 0000000000007a19 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12080.067834] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12080.067836] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12080.067838] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12080.067840] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12080.067843] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12080.067846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12080.067848] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12080.067856] Call Trace: [12080.067861] [12080.067867] ? show_regs.cold.9+0x22/0x2f [12080.067876] ? watchdog_timer_fn.cold.6+0x62/0x10a [12080.067882] ? __hrtimer_run_queues+0x1c2/0x4d0 [12080.067886] ? watchdog+0x40/0x40 [12080.067890] ? hrtimer_interrupt+0x128/0x2c0 [12080.067893] ? smp_apic_timer_interrupt+0xa6/0x270 [12080.067901] ? apic_timer_interrupt+0xf/0x20 [12080.067906] [12080.067909] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12080.067917] do_raw_spin_lock+0x177/0x190 [12080.067921] _raw_spin_lock+0x21/0x30 [12080.067926] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12080.086406] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12080.086492] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12080.086500] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12080.086503] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12080.086516] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12080.086534] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12080.086618] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12080.086621] RDX: 0000000000007256 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12080.086623] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12080.086625] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12080.086627] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12080.086631] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12080.086634] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12080.086636] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12080.086644] Call Trace: [12080.086649] [12080.086660] ? show_regs.cold.9+0x22/0x2f [12080.086670] ? watchdog_timer_fn.cold.6+0x62/0x10a [12080.086676] ? __hrtimer_run_queues+0x1c2/0x4d0 [12080.086680] ? watchdog+0x40/0x40 [12080.086684] ? hrtimer_interrupt+0x128/0x2c0 [12080.086687] ? smp_apic_timer_interrupt+0xa6/0x270 [12080.086695] ? apic_timer_interrupt+0xf/0x20 [12080.086699] [12080.086701] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12080.086706] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12080.086710] do_raw_spin_lock+0x177/0x190 [12080.086714] _raw_spin_lock+0x21/0x30 [12080.086720] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12080.088136] obd_mod_free+0x46/0x140 [ptlrpc] [12080.136414] ? do_raw_spin_unlock+0x75/0x190 [12080.136436] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12080.136884] ? do_wait_intr+0xf0/0xf0 [12080.136894] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12080.137098] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12080.137317] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12080.137887] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12080.138019] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12080.138072] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12080.138100] mdc_intent_lock+0x353/0x7e0 [mdc] [12080.138129] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12080.138212] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12080.138433] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12080.138461] lmv_intent_open+0x338/0x1150 [lmv] [12080.138496] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12080.139170] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12080.139257] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12080.139285] ll_intent_lock+0xc5/0xaa0 [lustre] [12080.139378] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12080.139467] ll_lookup_it+0xe0e/0x2270 [lustre] [12080.147397] mdc_commit_open+0x91/0xb0 [mdc] [12080.147453] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12080.149142] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12080.149397] after_reply+0xe2c/0x1780 [ptlrpc] [12080.149610] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12080.149816] ? __next_timer_interrupt+0x160/0x160 [12080.149827] ptlrpcd+0xa82/0xe40 [ptlrpc] [12080.159251] ? do_raw_spin_unlock+0x75/0x190 [12080.159272] ? _raw_spin_unlock+0x12/0x30 [12080.159280] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12080.159310] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12080.159332] ll_atomic_open+0x36a/0xd70 [lustre] [12080.163545] ? do_wait_intr+0xf0/0xf0 [12080.163570] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12080.163843] kthread+0x1d1/0x200 [12080.163849] ? set_kthread_struct+0x70/0x70 [12080.163853] ret_from_fork+0x1f/0x30 [12080.899696] lookup_open+0x356/0xb40 [12080.900927] do_last+0x313/0xfc0 [12080.905159] ? nd_jump_root+0xe5/0x160 [12080.908421] ? path_init+0x437/0x520 [12080.915434] path_openat+0xf7/0x500 [12080.916446] ? do_raw_spin_unlock+0x75/0x190 [12080.917500] do_filp_open+0x99/0x140 [12080.922816] ? getname_flags+0x6e/0x330 [12080.924811] ? __check_object_size+0xff/0x256 [12080.929414] ? do_raw_spin_unlock+0x75/0x190 [12080.932122] ? _raw_spin_unlock+0x12/0x30 [12080.939500] do_sys_openat2+0x2b4/0x410 [12080.940889] do_sys_open+0x73/0xa0 [12080.944351] __x64_sys_openat+0x24/0x30 [12080.946878] do_syscall_64+0xc1/0x440 [12080.950343] entry_SYSCALL_64_after_hwframe+0x49/0xae [12080.960740] RIP: 0033:0x7f53425a5332 [12080.962069] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12080.980904] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12080.988757] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12080.990930] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12080.998508] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12081.006339] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12081.009282] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12084.052767] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [12084.058762] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12084.094329] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12084.100768] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12084.104258] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12084.107927] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12084.111262] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12084.118644] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [12084.121625] RAX: 0000000000002a83 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12084.123816] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12084.127293] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12084.130148] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12084.133013] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12084.135984] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12084.139330] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12084.142253] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12084.144692] Call Trace: [12084.145466] [12084.146340] ? show_regs.cold.9+0x22/0x2f [12084.147969] ? watchdog_timer_fn.cold.6+0x62/0x10a [12084.150826] ? __hrtimer_run_queues+0x1c2/0x4d0 [12084.152285] ? watchdog+0x40/0x40 [12084.153662] ? hrtimer_interrupt+0x128/0x2c0 [12084.155881] ? smp_apic_timer_interrupt+0xa6/0x270 [12084.158205] ? apic_timer_interrupt+0xf/0x20 [12084.160235] [12084.161175] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12084.163257] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12084.165298] ? newidle_balance.isra.93+0x207/0x580 [12084.166946] do_raw_spin_lock+0x177/0x190 [12084.168397] _raw_spin_lock+0x21/0x30 [12084.169921] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12084.171829] process_one_work+0x2c8/0x700 [12084.175680] worker_thread+0x296/0x6e0 [12084.178788] ? rescuer_thread+0x570/0x570 [12084.180849] kthread+0x1d1/0x200 [12084.183878] ? set_kthread_struct+0x70/0x70 [12084.185750] ret_from_fork+0x1f/0x30 [12086.240037] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9449s! [12086.243955] Showing busy workqueues and worker pools: [12086.246355] workqueue events: flags=0x0 [12086.248107] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12086.251990] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12086.257614] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12086.259861] pending: cpu_latency_work [ptlrpc] [12086.262352] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12086.265129] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12086.268357] workqueue events_long: flags=0x0 [12086.270403] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12086.273189] pending: lu_objects_flush [obdclass] [12086.275648] workqueue events_unbound: flags=0x2 [12086.277833] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12086.280591] pending: flush_memcg_stats_dwork, idle_cull_fn [12086.283375] workqueue events_power_efficient: flags=0x80 [12086.286013] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12086.289007] pending: check_lifetime, do_cache_clean [12086.292393] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12086.297097] pending: neigh_periodic_work, neigh_periodic_work [12086.301329] workqueue mm_percpu_wq: flags=0x8 [12086.304058] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12086.309086] pending: vmstat_update, lru_add_drain_per_cpu [12086.312487] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12086.316808] pending: vmstat_update, lru_add_drain_per_cpu [12086.320385] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12086.324524] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12086.328776] workqueue xprtiod: flags=0xa [12086.331182] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12086.334023] pending: xprt_autoclose, xs_error_handle [12086.336351] workqueue ipv6_addrconf: flags=0x40008 [12086.338385] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12086.342326] pending: addrconf_verify_work [12086.345060] workqueue knbd0-recv: flags=0x1a [12086.347341] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12086.350990] in-flight: 651:recv_work [12086.353139] workqueue ptlrpc_pinger: flags=0x2 [12086.355514] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12086.358929] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12086.362316] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9449s workers=4 idle: 15979 31824 8 [12086.366640] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12076s workers=2 idle: 654 [12096.058679] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [12096.061179] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12096.075958] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12096.079260] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12096.081828] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12096.083420] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12096.088518] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12096.090791] RAX: 0000000000007f85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12096.092987] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12096.095240] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12096.097489] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12096.099603] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12096.101592] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12096.103646] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12096.105550] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12096.107520] Call Trace: [12096.108224] [12096.108756] ? show_regs.cold.9+0x22/0x2f [12096.109949] ? watchdog_timer_fn.cold.6+0x62/0x10a [12096.111197] ? __hrtimer_run_queues+0x1c2/0x4d0 [12096.112390] ? watchdog+0x40/0x40 [12096.113476] ? hrtimer_interrupt+0x128/0x2c0 [12096.114754] ? smp_apic_timer_interrupt+0xa6/0x270 [12096.115986] ? apic_timer_interrupt+0xf/0x20 [12096.117189] [12096.117813] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12096.119207] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12096.120798] do_raw_spin_lock+0x177/0x190 [12096.121985] _raw_spin_lock+0x21/0x30 [12096.122966] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12096.124916] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12096.127069] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12096.128499] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12096.130341] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12096.131941] ? woken_wake_function+0x30/0x30 [12096.133378] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12096.134915] kthread+0x1d1/0x200 [12096.135636] ? set_kthread_struct+0x70/0x70 [12096.136638] ret_from_fork+0x1f/0x30 [12108.055257] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [12108.059067] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [12108.061247] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12108.061323] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12108.061330] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12108.061333] RIP: 0010:native_safe_halt+0x2e/0x50 [12108.061345] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12108.061350] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12108.061356] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12108.061358] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12108.061360] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12108.061363] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12108.061366] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12108.061369] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12108.061372] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12108.061374] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12108.061384] Call Trace: [12108.061390] [12108.061396] ? show_regs.cold.9+0x22/0x2f [12108.061405] ? watchdog_timer_fn.cold.6+0x62/0x10a [12108.061413] ? __hrtimer_run_queues+0x1c2/0x4d0 [12108.061417] ? watchdog+0x40/0x40 [12108.061421] ? hrtimer_interrupt+0x128/0x2c0 [12108.061426] ? smp_apic_timer_interrupt+0xa6/0x270 [12108.061433] ? apic_timer_interrupt+0xf/0x20 [12108.061438] [12108.061441] ? native_safe_halt+0x2e/0x50 [12108.061446] kvm_wait+0xcd/0xe0 [12108.061453] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12108.061461] do_raw_spin_lock+0x177/0x190 [12108.061466] _raw_spin_lock+0x21/0x30 [12108.061470] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12108.062063] ? do_raw_spin_unlock+0x75/0x190 [12108.062071] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12108.062349] ? do_wait_intr+0xf0/0xf0 [12108.062359] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12108.062564] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12108.073828] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12108.073907] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12108.073914] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12108.073918] RIP: 0010:native_safe_halt+0x2e/0x50 [12108.073931] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12108.073935] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12108.073939] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12108.073941] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12108.073944] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12108.073946] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12108.073947] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12108.073950] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12108.073953] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12108.073955] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12108.073963] Call Trace: [12108.073969] [12108.073974] ? show_regs.cold.9+0x22/0x2f [12108.073984] ? watchdog_timer_fn.cold.6+0x62/0x10a [12108.073992] ? __hrtimer_run_queues+0x1c2/0x4d0 [12108.073997] ? watchdog+0x40/0x40 [12108.074001] ? hrtimer_interrupt+0x128/0x2c0 [12108.074004] ? smp_apic_timer_interrupt+0xa6/0x270 [12108.074697] ? apic_timer_interrupt+0xf/0x20 [12108.074703] [12108.074706] ? native_safe_halt+0x2e/0x50 [12108.074710] kvm_wait+0xcd/0xe0 [12108.074718] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12108.074724] do_raw_spin_lock+0x177/0x190 [12108.074728] _raw_spin_lock+0x21/0x30 [12108.074733] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12108.075004] obd_mod_free+0x46/0x140 [ptlrpc] [12108.075416] mdc_commit_open+0x91/0xb0 [mdc] [12108.075456] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12108.075729] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12108.077312] after_reply+0xe2c/0x1780 [ptlrpc] [12108.077579] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12108.118687] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12108.130720] ? __next_timer_interrupt+0x160/0x160 [12108.143249] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12108.147211] ptlrpcd+0xa82/0xe40 [ptlrpc] [12108.169683] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12108.169737] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12108.169769] mdc_intent_lock+0x353/0x7e0 [mdc] [12108.169799] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12108.170443] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12108.170935] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12108.170969] lmv_intent_open+0x338/0x1150 [lmv] [12108.170999] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12108.171089] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12108.171173] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12108.171193] ll_intent_lock+0xc5/0xaa0 [lustre] [12108.171452] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12108.171605] ll_lookup_it+0xe0e/0x2270 [lustre] [12108.171688] ? do_raw_spin_unlock+0x75/0x190 [12108.171696] ? _raw_spin_unlock+0x12/0x30 [12108.171704] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12108.171732] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12108.171751] ll_atomic_open+0x36a/0xd70 [lustre] [12108.178024] ? do_wait_intr+0xf0/0xf0 [12108.178060] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12108.188961] lookup_open+0x356/0xb40 [12108.188984] do_last+0x313/0xfc0 [12108.188989] ? nd_jump_root+0xe5/0x160 [12108.188994] ? path_init+0x437/0x520 [12108.188998] path_openat+0xf7/0x500 [12108.189003] ? do_raw_spin_unlock+0x75/0x190 [12108.189010] do_filp_open+0x99/0x140 [12108.189016] ? getname_flags+0x6e/0x330 [12108.189021] ? __check_object_size+0xff/0x256 [12108.189027] ? do_raw_spin_unlock+0x75/0x190 [12108.189032] ? _raw_spin_unlock+0x12/0x30 [12108.189039] do_sys_openat2+0x2b4/0x410 [12108.189044] do_sys_open+0x73/0xa0 [12108.189048] __x64_sys_openat+0x24/0x30 [12108.189052] do_syscall_64+0xc1/0x440 [12108.189058] entry_SYSCALL_64_after_hwframe+0x49/0xae [12108.189065] RIP: 0033:0x7f53425a5332 [12108.189073] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12108.189077] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12108.189083] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12108.189086] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12108.189088] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12108.189090] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12108.189092] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12108.636936] kthread+0x1d1/0x200 [12108.638550] ? set_kthread_struct+0x70/0x70 [12108.640995] ret_from_fork+0x1f/0x30 [12112.052012] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [12112.055058] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12112.087032] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12112.091877] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12112.096396] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12112.100371] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12112.103977] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12112.114765] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12112.117698] RAX: 00000000000032ff RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12112.128726] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12112.133088] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12112.146015] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12112.151685] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12112.155199] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12112.158868] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12112.164955] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12112.174166] Call Trace: [12112.175133] [12112.175999] ? show_regs.cold.9+0x22/0x2f [12112.181807] ? watchdog_timer_fn.cold.6+0x62/0x10a [12112.191645] ? __hrtimer_run_queues+0x1c2/0x4d0 [12112.196334] ? watchdog+0x40/0x40 [12112.197860] ? hrtimer_interrupt+0x128/0x2c0 [12112.204913] ? smp_apic_timer_interrupt+0xa6/0x270 [12112.210230] ? apic_timer_interrupt+0xf/0x20 [12112.213531] [12112.214183] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12112.218778] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12112.222355] ? newidle_balance.isra.93+0x207/0x580 [12112.225280] do_raw_spin_lock+0x177/0x190 [12112.234101] _raw_spin_lock+0x21/0x30 [12112.236149] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12112.286019] process_one_work+0x2c8/0x700 [12112.291764] worker_thread+0x296/0x6e0 [12112.294097] ? rescuer_thread+0x570/0x570 [12112.301094] kthread+0x1d1/0x200 [12112.304037] ? set_kthread_struct+0x70/0x70 [12112.307406] ret_from_fork+0x1f/0x30 [12116.457049] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9479s! [12116.462378] Showing busy workqueues and worker pools: [12116.465278] workqueue events: flags=0x0 [12116.467072] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12116.470978] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12116.483498] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12116.494876] pending: cpu_latency_work [ptlrpc] [12116.502917] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12116.506175] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12116.510103] workqueue events_long: flags=0x0 [12116.516736] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12116.520010] pending: lu_objects_flush [obdclass] [12116.522554] workqueue events_unbound: flags=0x2 [12116.524773] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12116.527352] pending: flush_memcg_stats_dwork, idle_cull_fn [12116.532184] workqueue events_power_efficient: flags=0x80 [12116.533983] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12116.536809] pending: check_lifetime, do_cache_clean [12116.539281] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12116.545439] pending: neigh_periodic_work, neigh_periodic_work [12116.548816] workqueue mm_percpu_wq: flags=0x8 [12116.551644] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12116.555810] pending: vmstat_update, lru_add_drain_per_cpu [12116.559063] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12116.561249] pending: vmstat_update, lru_add_drain_per_cpu [12116.563557] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12116.566655] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12116.571109] workqueue xprtiod: flags=0xa [12116.573395] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12116.575891] pending: xprt_autoclose, xs_error_handle [12116.582603] workqueue ipv6_addrconf: flags=0x40008 [12116.587590] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12116.595017] pending: addrconf_verify_work [12116.597038] workqueue knbd0-recv: flags=0x1a [12116.601207] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12116.606563] in-flight: 651:recv_work [12116.610691] workqueue ptlrpc_pinger: flags=0x2 [12116.613013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12116.617908] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12116.622757] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9479s workers=4 idle: 15979 31824 8 [12116.631868] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12107s workers=2 idle: 654 [12124.062330] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [12124.065746] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12124.083846] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12124.087590] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12124.090790] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12124.093509] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12124.100664] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12124.103658] RAX: 000000000000425e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12124.106579] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12124.108883] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12124.111754] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12124.114625] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12124.117354] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12124.120935] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12124.123136] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12124.126442] Call Trace: [12124.127502] [12124.128444] ? show_regs.cold.9+0x22/0x2f [12124.129890] ? watchdog_timer_fn.cold.6+0x62/0x10a [12124.131930] ? __hrtimer_run_queues+0x1c2/0x4d0 [12124.134414] ? watchdog+0x40/0x40 [12124.135668] ? hrtimer_interrupt+0x128/0x2c0 [12124.137540] ? smp_apic_timer_interrupt+0xa6/0x270 [12124.139453] ? apic_timer_interrupt+0xf/0x20 [12124.141398] [12124.142481] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12124.144532] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12124.146741] do_raw_spin_lock+0x177/0x190 [12124.148687] _raw_spin_lock+0x21/0x30 [12124.150372] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12124.153150] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12124.155429] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12124.157729] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12124.160146] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12124.162630] ? woken_wake_function+0x30/0x30 [12124.164280] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12124.166353] kthread+0x1d1/0x200 [12124.168114] ? set_kthread_struct+0x70/0x70 [12124.169838] ret_from_fork+0x1f/0x30 [12136.061012] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [12136.061364] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [12136.071238] Modules linked in: [12136.074833] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12136.074909] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12136.074917] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12136.074919] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12136.074934] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12136.074939] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12136.074943] RAX: 0000000000007e85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12136.074945] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12136.074948] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12136.074950] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12136.074953] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12136.074956] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12136.074959] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12136.074961] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12136.074971] Call Trace: [12136.074977] [12136.074982] ? show_regs.cold.9+0x22/0x2f [12136.074992] ? watchdog_timer_fn.cold.6+0x62/0x10a [12136.075013] ? __hrtimer_run_queues+0x1c2/0x4d0 [12136.075018] ? watchdog+0x40/0x40 [12136.075022] ? hrtimer_interrupt+0x128/0x2c0 [12136.075027] ? smp_apic_timer_interrupt+0xa6/0x270 [12136.075036] ? apic_timer_interrupt+0xf/0x20 [12136.075041] [12136.075044] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12136.075051] do_raw_spin_lock+0x177/0x190 [12136.075056] _raw_spin_lock+0x21/0x30 [12136.075062] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12136.075348] ? do_raw_spin_unlock+0x75/0x190 [12136.075354] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12136.075552] ? do_wait_intr+0xf0/0xf0 [12136.075562] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12136.075753] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12136.075964] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12136.076179] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12136.076292] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12136.076329] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12136.076358] mdc_intent_lock+0x353/0x7e0 [mdc] [12136.076383] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12136.076439] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12136.078626] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12136.078833] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12136.078840] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12136.078843] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12136.078864] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12136.078868] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12136.078872] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12136.078875] RDX: 0000000000000752 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12136.078878] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12136.078880] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12136.078882] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12136.078885] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12136.078888] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12136.078890] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12136.078897] Call Trace: [12136.078903] [12136.078908] ? show_regs.cold.9+0x22/0x2f [12136.078918] ? watchdog_timer_fn.cold.6+0x62/0x10a [12136.078924] ? __hrtimer_run_queues+0x1c2/0x4d0 [12136.078928] ? watchdog+0x40/0x40 [12136.078932] ? hrtimer_interrupt+0x128/0x2c0 [12136.078936] ? smp_apic_timer_interrupt+0xa6/0x270 [12136.078944] ? apic_timer_interrupt+0xf/0x20 [12136.078949] [12136.078952] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12136.078957] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12136.078961] do_raw_spin_lock+0x177/0x190 [12136.078966] _raw_spin_lock+0x21/0x30 [12136.078972] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12136.079332] obd_mod_free+0x46/0x140 [ptlrpc] [12136.079527] mdc_commit_open+0x91/0xb0 [mdc] [12136.079682] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12136.143577] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12136.143635] lmv_intent_open+0x338/0x1150 [lmv] [12136.143667] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12136.143778] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12136.143862] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12136.143884] ll_intent_lock+0xc5/0xaa0 [lustre] [12136.145182] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12136.145558] ll_lookup_it+0xe0e/0x2270 [lustre] [12136.145638] ? do_raw_spin_unlock+0x75/0x190 [12136.145647] ? _raw_spin_unlock+0x12/0x30 [12136.145654] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12136.145682] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12136.145703] ll_atomic_open+0x36a/0xd70 [lustre] [12136.145787] lookup_open+0x356/0xb40 [12136.145795] do_last+0x313/0xfc0 [12136.145800] ? nd_jump_root+0xe5/0x160 [12136.145805] ? path_init+0x437/0x520 [12136.145809] path_openat+0xf7/0x500 [12136.145814] ? do_raw_spin_unlock+0x75/0x190 [12136.145819] do_filp_open+0x99/0x140 [12136.145825] ? getname_flags+0x6e/0x330 [12136.145830] ? __check_object_size+0xff/0x256 [12136.145838] ? do_raw_spin_unlock+0x75/0x190 [12136.145842] ? _raw_spin_unlock+0x12/0x30 [12136.145847] do_sys_openat2+0x2b4/0x410 [12136.145852] do_sys_open+0x73/0xa0 [12136.145856] __x64_sys_openat+0x24/0x30 [12136.145860] do_syscall_64+0xc1/0x440 [12136.145866] entry_SYSCALL_64_after_hwframe+0x49/0xae [12136.145873] RIP: 0033:0x7f53425a5332 [12136.145881] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12136.145886] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12136.145892] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12136.145894] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12136.145897] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12136.145899] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12136.145901] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12136.771609] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12136.775069] after_reply+0xe2c/0x1780 [ptlrpc] [12136.776906] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12136.783707] ? __next_timer_interrupt+0x160/0x160 [12136.788967] ptlrpcd+0xa82/0xe40 [ptlrpc] [12136.796565] ? do_wait_intr+0xf0/0xf0 [12136.798838] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12136.810859] kthread+0x1d1/0x200 [12136.812843] ? set_kthread_struct+0x70/0x70 [12136.816581] ret_from_fork+0x1f/0x30 [12144.053784] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [12144.062314] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12144.105492] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12144.115795] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12144.122757] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12144.125023] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12144.134787] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12144.154427] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12144.159325] RAX: 0000000000007987 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12144.164949] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12144.184437] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12144.194401] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12144.199802] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12144.203264] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12144.209124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12144.213142] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12144.219566] Call Trace: [12144.221334] [12144.222494] ? show_regs.cold.9+0x22/0x2f [12144.228992] ? watchdog_timer_fn.cold.6+0x62/0x10a [12144.231778] ? __hrtimer_run_queues+0x1c2/0x4d0 [12144.237798] ? watchdog+0x40/0x40 [12144.241062] ? hrtimer_interrupt+0x128/0x2c0 [12144.248018] ? smp_apic_timer_interrupt+0xa6/0x270 [12144.253428] ? apic_timer_interrupt+0xf/0x20 [12144.257699] [12144.258575] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12144.262132] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12144.266275] ? newidle_balance.isra.93+0x207/0x580 [12144.268282] do_raw_spin_lock+0x177/0x190 [12144.269675] _raw_spin_lock+0x21/0x30 [12144.271107] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12144.273382] process_one_work+0x2c8/0x700 [12144.274714] worker_thread+0x296/0x6e0 [12144.276239] ? rescuer_thread+0x570/0x570 [12144.277742] kthread+0x1d1/0x200 [12144.282758] ? set_kthread_struct+0x70/0x70 [12144.284383] ret_from_fork+0x1f/0x30 [12146.662969] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9509s! [12146.671031] Showing busy workqueues and worker pools: [12146.677017] workqueue events: flags=0x0 [12146.685834] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12146.702118] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12146.718347] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12146.728191] pending: cpu_latency_work [ptlrpc] [12146.733359] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12146.741366] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12146.753016] workqueue events_long: flags=0x0 [12146.755551] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12146.761319] pending: lu_objects_flush [obdclass] [12146.765920] workqueue events_unbound: flags=0x2 [12146.768250] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12146.772575] pending: flush_memcg_stats_dwork, idle_cull_fn [12146.780235] workqueue events_power_efficient: flags=0x80 [12146.786479] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12146.792526] pending: check_lifetime, do_cache_clean [12146.799010] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12146.804217] pending: neigh_periodic_work, neigh_periodic_work [12146.808193] workqueue mm_percpu_wq: flags=0x8 [12146.810568] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12146.826480] pending: vmstat_update, lru_add_drain_per_cpu [12146.833222] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12146.836756] pending: vmstat_update, lru_add_drain_per_cpu [12146.840039] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12146.842265] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12146.850961] workqueue xprtiod: flags=0xa [12146.852304] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12146.864029] pending: xprt_autoclose, xs_error_handle [12146.868622] workqueue ipv6_addrconf: flags=0x40008 [12146.876567] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12146.881643] pending: addrconf_verify_work [12146.885059] workqueue knbd0-recv: flags=0x1a [12146.890252] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12146.897248] in-flight: 651:recv_work [12146.902366] workqueue ptlrpc_pinger: flags=0x2 [12146.910272] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12146.913431] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12146.922996] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9509s workers=4 idle: 15979 31824 8 [12146.930706] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12137s workers=2 idle: 654 [12152.067367] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [12152.076234] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12152.118317] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12152.126172] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12152.130400] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12152.133114] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12152.141098] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12152.144835] RAX: 000000000000780d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12152.148608] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12152.155953] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12152.159981] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12152.166670] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12152.172951] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12152.177604] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12152.180316] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12152.184960] Call Trace: [12152.186877] [12152.187555] ? show_regs.cold.9+0x22/0x2f [12152.189309] ? watchdog_timer_fn.cold.6+0x62/0x10a [12152.191965] ? __hrtimer_run_queues+0x1c2/0x4d0 [12152.194495] ? watchdog+0x40/0x40 [12152.196344] ? hrtimer_interrupt+0x128/0x2c0 [12152.198593] ? smp_apic_timer_interrupt+0xa6/0x270 [12152.202192] ? apic_timer_interrupt+0xf/0x20 [12152.204875] [12152.205679] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12152.208223] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12152.210710] do_raw_spin_lock+0x177/0x190 [12152.213368] _raw_spin_lock+0x21/0x30 [12152.216871] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12152.220484] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12152.222376] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12152.225384] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12152.228174] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12152.233394] ? woken_wake_function+0x30/0x30 [12152.239442] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12152.243579] kthread+0x1d1/0x200 [12152.244867] ? set_kthread_struct+0x70/0x70 [12152.246665] ret_from_fork+0x1f/0x30 [12164.055083] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [12164.057214] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [12164.062328] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12164.062412] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12164.062419] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12164.062422] RIP: 0010:native_safe_halt+0x2e/0x50 [12164.062436] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12164.062441] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12164.062445] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12164.062447] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12164.062449] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12164.062451] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12164.062453] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12164.062456] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12164.062459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12164.062461] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12164.062470] Call Trace: [12164.062475] [12164.062481] ? show_regs.cold.9+0x22/0x2f [12164.062492] ? watchdog_timer_fn.cold.6+0x62/0x10a [12164.062498] ? __hrtimer_run_queues+0x1c2/0x4d0 [12164.062502] ? watchdog+0x40/0x40 [12164.062518] ? hrtimer_interrupt+0x128/0x2c0 [12164.062524] ? smp_apic_timer_interrupt+0xa6/0x270 [12164.062532] ? apic_timer_interrupt+0xf/0x20 [12164.062537] [12164.062540] ? native_safe_halt+0x2e/0x50 [12164.062544] kvm_wait+0xcd/0xe0 [12164.062553] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12164.062565] do_raw_spin_lock+0x177/0x190 [12164.062571] _raw_spin_lock+0x21/0x30 [12164.062576] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12164.063222] ? do_raw_spin_unlock+0x75/0x190 [12164.063230] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12164.063597] ? do_wait_intr+0xf0/0xf0 [12164.063606] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12164.067811] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12164.067896] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12164.067905] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12164.067908] RIP: 0010:native_safe_halt+0x2e/0x50 [12164.067923] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12164.067928] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [12164.067934] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12164.067937] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12164.067939] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12164.067942] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12164.067945] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12164.067949] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12164.067952] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12164.067954] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12164.067965] Call Trace: [12164.067970] [12164.067977] ? show_regs.cold.9+0x22/0x2f [12164.067989] ? watchdog_timer_fn.cold.6+0x62/0x10a [12164.068208] ? __hrtimer_run_queues+0x1c2/0x4d0 [12164.091154] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12164.097405] ? watchdog+0x40/0x40 [12164.097426] ? hrtimer_interrupt+0x128/0x2c0 [12164.097432] ? smp_apic_timer_interrupt+0xa6/0x270 [12164.097627] ? apic_timer_interrupt+0xf/0x20 [12164.097633] [12164.097638] ? native_safe_halt+0x2e/0x50 [12164.097644] kvm_wait+0xcd/0xe0 [12164.097654] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12164.097661] do_raw_spin_lock+0x177/0x190 [12164.097667] _raw_spin_lock+0x21/0x30 [12164.097675] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12164.097945] obd_mod_free+0x46/0x140 [ptlrpc] [12164.099217] mdc_commit_open+0x91/0xb0 [mdc] [12164.103118] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12164.110000] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12164.122614] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12164.130359] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12164.134173] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12164.138339] after_reply+0xe2c/0x1780 [ptlrpc] [12164.142443] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12164.147140] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12164.150240] mdc_intent_lock+0x353/0x7e0 [mdc] [12164.154692] ? __next_timer_interrupt+0x160/0x160 [12164.154715] ptlrpcd+0xa82/0xe40 [ptlrpc] [12164.157235] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12164.161578] ? do_wait_intr+0xf0/0xf0 [12164.162849] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12164.163937] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12164.165498] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12164.168311] kthread+0x1d1/0x200 [12164.170380] lmv_intent_open+0x338/0x1150 [lmv] [12164.172275] ? set_kthread_struct+0x70/0x70 [12164.174216] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12164.176795] ret_from_fork+0x1f/0x30 [12164.178821] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12164.392304] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12164.394360] ll_intent_lock+0xc5/0xaa0 [lustre] [12164.396696] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12164.399168] ll_lookup_it+0xe0e/0x2270 [lustre] [12164.401369] ? do_raw_spin_unlock+0x75/0x190 [12164.404775] ? _raw_spin_unlock+0x12/0x30 [12164.406488] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12164.409376] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12164.411984] ll_atomic_open+0x36a/0xd70 [lustre] [12164.414813] lookup_open+0x356/0xb40 [12164.416998] do_last+0x313/0xfc0 [12164.418099] ? nd_jump_root+0xe5/0x160 [12164.421124] ? path_init+0x437/0x520 [12164.426421] path_openat+0xf7/0x500 [12164.433242] ? do_raw_spin_unlock+0x75/0x190 [12164.443933] do_filp_open+0x99/0x140 [12164.448596] ? getname_flags+0x6e/0x330 [12164.450422] ? __check_object_size+0xff/0x256 [12164.454445] ? do_raw_spin_unlock+0x75/0x190 [12164.461374] ? _raw_spin_unlock+0x12/0x30 [12164.465313] do_sys_openat2+0x2b4/0x410 [12164.471494] do_sys_open+0x73/0xa0 [12164.476280] __x64_sys_openat+0x24/0x30 [12164.479915] do_syscall_64+0xc1/0x440 [12164.484268] entry_SYSCALL_64_after_hwframe+0x49/0xae [12164.488337] RIP: 0033:0x7f53425a5332 [12164.495392] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12164.509891] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12164.516680] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12164.522782] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12164.530552] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12164.542573] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12164.544964] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12172.052087] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [12172.055758] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12172.092775] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12172.100156] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12172.123727] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12172.129083] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12172.134993] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12172.152439] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12172.158483] RAX: 000000000000623e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12172.162812] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12172.166208] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12172.173314] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12172.181183] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12172.188067] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12172.192196] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12172.199956] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12172.206545] Call Trace: [12172.207851] [12172.209433] ? show_regs.cold.9+0x22/0x2f [12172.211305] ? watchdog_timer_fn.cold.6+0x62/0x10a [12172.212971] ? __hrtimer_run_queues+0x1c2/0x4d0 [12172.216473] ? watchdog+0x40/0x40 [12172.218867] ? hrtimer_interrupt+0x128/0x2c0 [12172.227192] ? smp_apic_timer_interrupt+0xa6/0x270 [12172.231821] ? apic_timer_interrupt+0xf/0x20 [12172.236094] [12172.238138] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12172.244308] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12172.256120] ? newidle_balance.isra.93+0x207/0x580 [12172.260838] do_raw_spin_lock+0x177/0x190 [12172.265047] _raw_spin_lock+0x21/0x30 [12172.266194] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12172.276198] process_one_work+0x2c8/0x700 [12172.282022] worker_thread+0x296/0x6e0 [12172.283738] ? rescuer_thread+0x570/0x570 [12172.288469] kthread+0x1d1/0x200 [12172.292284] ? set_kthread_struct+0x70/0x70 [12172.296894] ret_from_fork+0x1f/0x30 [12177.377471] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9540s! [12177.387051] Showing busy workqueues and worker pools: [12177.389388] workqueue events: flags=0x0 [12177.395899] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12177.404392] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12177.414694] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12177.421807] pending: cpu_latency_work [ptlrpc] [12177.428413] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12177.433187] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12177.445872] workqueue events_long: flags=0x0 [12177.449222] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12177.457206] pending: lu_objects_flush [obdclass] [12177.462171] workqueue events_unbound: flags=0x2 [12177.467356] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12177.470123] pending: flush_memcg_stats_dwork, idle_cull_fn [12177.475413] workqueue events_power_efficient: flags=0x80 [12177.484648] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12177.491267] pending: check_lifetime, do_cache_clean [12177.495219] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12177.504169] pending: neigh_periodic_work, neigh_periodic_work [12177.511472] workqueue mm_percpu_wq: flags=0x8 [12177.514151] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12177.527170] pending: vmstat_update, lru_add_drain_per_cpu [12177.538113] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12177.547997] pending: vmstat_update, lru_add_drain_per_cpu [12177.556418] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12177.562848] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12177.568935] workqueue xprtiod: flags=0xa [12177.571062] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12177.577127] pending: xprt_autoclose, xs_error_handle [12177.580729] workqueue ipv6_addrconf: flags=0x40008 [12177.582954] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12177.586258] pending: addrconf_verify_work [12177.588500] workqueue knbd0-recv: flags=0x1a [12177.590772] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12177.594118] in-flight: 651:recv_work [12177.595964] workqueue ptlrpc_pinger: flags=0x2 [12177.598251] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12177.600930] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12177.604027] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9540s workers=4 idle: 15979 31824 8 [12177.608730] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12168s workers=2 idle: 654 [12180.058027] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [12180.060900] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12180.089894] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12180.095688] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12180.102994] RIP: 0010:native_safe_halt+0x2e/0x50 [12180.104575] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12180.116754] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12180.120919] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12180.123498] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12180.125673] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12180.129256] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12180.136472] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12180.139189] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12180.144053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12180.145790] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12180.153883] Call Trace: [12180.155775] [12180.156911] ? show_regs.cold.9+0x22/0x2f [12180.159553] ? watchdog_timer_fn.cold.6+0x62/0x10a [12180.161752] ? __hrtimer_run_queues+0x1c2/0x4d0 [12180.163299] ? watchdog+0x40/0x40 [12180.166320] ? hrtimer_interrupt+0x128/0x2c0 [12180.167614] ? smp_apic_timer_interrupt+0xa6/0x270 [12180.169190] ? apic_timer_interrupt+0xf/0x20 [12180.172305] [12180.173000] ? native_safe_halt+0x2e/0x50 [12180.174672] kvm_wait+0xcd/0xe0 [12180.180514] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12180.184871] do_raw_spin_lock+0x177/0x190 [12180.186401] _raw_spin_lock+0x21/0x30 [12180.187439] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12180.191017] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12180.194128] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12180.201559] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12180.206114] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12180.212139] ? woken_wake_function+0x30/0x30 [12180.214092] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12180.217987] kthread+0x1d1/0x200 [12180.219999] ? set_kthread_struct+0x70/0x70 [12180.224205] ret_from_fork+0x1f/0x30 [12192.055027] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [12192.057432] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [12192.057458] Modules linked in: [12192.059542] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12192.059605] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12192.059612] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12192.059615] RIP: 0010:native_safe_halt+0x2e/0x50 [12192.059629] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12192.059632] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [12192.059635] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12192.059637] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12192.059639] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12192.059640] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12192.059642] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12192.059644] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12192.059646] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12192.059647] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12192.059655] Call Trace: [12192.059659] [12192.059664] ? show_regs.cold.9+0x22/0x2f [12192.059672] ? watchdog_timer_fn.cold.6+0x62/0x10a [12192.059676] ? __hrtimer_run_queues+0x1c2/0x4d0 [12192.059679] ? watchdog+0x40/0x40 [12192.059682] ? hrtimer_interrupt+0x128/0x2c0 [12192.059684] ? smp_apic_timer_interrupt+0xa6/0x270 [12192.059689] ? apic_timer_interrupt+0xf/0x20 [12192.059692] [12192.059693] ? native_safe_halt+0x2e/0x50 [12192.059696] kvm_wait+0xcd/0xe0 [12192.059701] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12192.059706] do_raw_spin_lock+0x177/0x190 [12192.059709] _raw_spin_lock+0x21/0x30 [12192.059713] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12192.060435] obd_mod_free+0x46/0x140 [ptlrpc] [12192.060560] mdc_commit_open+0x91/0xb0 [mdc] [12192.060594] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12192.060731] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12192.063095] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12192.063162] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12192.063168] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12192.063171] RIP: 0010:native_safe_halt+0x2e/0x50 [12192.063183] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12192.063187] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12192.063191] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12192.063193] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12192.063195] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12192.063197] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12192.063200] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12192.063204] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12192.063207] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12192.063209] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12192.063379] Call Trace: [12192.063385] [12192.063389] ? show_regs.cold.9+0x22/0x2f [12192.063398] ? watchdog_timer_fn.cold.6+0x62/0x10a [12192.063402] ? __hrtimer_run_queues+0x1c2/0x4d0 [12192.063405] ? watchdog+0x40/0x40 [12192.063408] ? hrtimer_interrupt+0x128/0x2c0 [12192.063410] ? smp_apic_timer_interrupt+0xa6/0x270 [12192.063415] ? apic_timer_interrupt+0xf/0x20 [12192.063419] [12192.063421] ? native_safe_halt+0x2e/0x50 [12192.063423] kvm_wait+0xcd/0xe0 [12192.063428] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12192.063434] do_raw_spin_lock+0x177/0x190 [12192.063437] _raw_spin_lock+0x21/0x30 [12192.063440] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12192.063644] ? do_raw_spin_unlock+0x75/0x190 [12192.063648] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12192.063773] ? do_wait_intr+0xf0/0xf0 [12192.063779] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12192.094415] after_reply+0xe2c/0x1780 [ptlrpc] [12192.094707] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12192.094911] ? __next_timer_interrupt+0x160/0x160 [12192.094921] ptlrpcd+0xa82/0xe40 [ptlrpc] [12192.096445] ? do_wait_intr+0xf0/0xf0 [12192.096456] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12192.096673] kthread+0x1d1/0x200 [12192.096680] ? set_kthread_struct+0x70/0x70 [12192.096683] ret_from_fork+0x1f/0x30 [12192.544963] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12192.547279] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12192.550761] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12192.552909] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12192.556443] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12192.558302] mdc_intent_lock+0x353/0x7e0 [mdc] [12192.559983] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12192.562377] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12192.565203] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12192.567592] lmv_intent_open+0x338/0x1150 [lmv] [12192.569505] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12192.571833] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12192.573745] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12192.576405] ll_intent_lock+0xc5/0xaa0 [lustre] [12192.578366] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12192.580613] ll_lookup_it+0xe0e/0x2270 [lustre] [12192.582782] ? do_raw_spin_unlock+0x75/0x190 [12192.584077] ? _raw_spin_unlock+0x12/0x30 [12192.585577] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12192.587703] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12192.593216] ll_atomic_open+0x36a/0xd70 [lustre] [12192.596790] lookup_open+0x356/0xb40 [12192.599524] do_last+0x313/0xfc0 [12192.600627] ? nd_jump_root+0xe5/0x160 [12192.601948] ? path_init+0x437/0x520 [12192.607613] path_openat+0xf7/0x500 [12192.608818] ? do_raw_spin_unlock+0x75/0x190 [12192.616847] do_filp_open+0x99/0x140 [12192.618140] ? getname_flags+0x6e/0x330 [12192.619471] ? __check_object_size+0xff/0x256 [12192.625461] ? do_raw_spin_unlock+0x75/0x190 [12192.626990] ? _raw_spin_unlock+0x12/0x30 [12192.629506] do_sys_openat2+0x2b4/0x410 [12192.635279] do_sys_open+0x73/0xa0 [12192.639358] __x64_sys_openat+0x24/0x30 [12192.640418] do_syscall_64+0xc1/0x440 [12192.641344] entry_SYSCALL_64_after_hwframe+0x49/0xae [12192.649892] RIP: 0033:0x7f53425a5332 [12192.652604] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12192.678644] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12192.690822] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12192.698889] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12192.703207] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12192.710908] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12192.718841] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12204.055589] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [12204.061090] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12204.084307] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12204.089591] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12204.093464] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12204.096746] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12204.100183] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12204.109441] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12204.113309] RAX: 0000000000007f84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12204.117242] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12204.121192] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12204.124656] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12204.129368] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12204.133158] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12204.137168] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12204.139953] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12204.143142] Call Trace: [12204.144669] [12204.145767] ? show_regs.cold.9+0x22/0x2f [12204.147693] ? watchdog_timer_fn.cold.6+0x62/0x10a [12204.149636] ? __hrtimer_run_queues+0x1c2/0x4d0 [12204.151735] ? watchdog+0x40/0x40 [12204.152947] ? hrtimer_interrupt+0x128/0x2c0 [12204.155066] ? smp_apic_timer_interrupt+0xa6/0x270 [12204.157257] ? apic_timer_interrupt+0xf/0x20 [12204.159611] [12204.160553] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12204.163090] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12204.165693] ? newidle_balance.isra.93+0x207/0x580 [12204.167796] do_raw_spin_lock+0x177/0x190 [12204.169971] _raw_spin_lock+0x21/0x30 [12204.171869] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12204.174469] process_one_work+0x2c8/0x700 [12204.176339] worker_thread+0x296/0x6e0 [12204.178128] ? rescuer_thread+0x570/0x570 [12204.179700] kthread+0x1d1/0x200 [12204.181666] ? set_kthread_struct+0x70/0x70 [12204.183516] ret_from_fork+0x1f/0x30 [12208.058291] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [12208.062886] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12208.102238] BUG: workqueue lockup - pool [12208.106219] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12208.106233] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12208.106236] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12208.106249] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12208.106254] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12208.106259] RAX: 0000000000007c90 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12208.106262] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12208.106264] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12208.106266] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12208.106268] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12208.106272] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12208.106275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12208.106277] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12208.106286] Call Trace: [12208.106292] [12208.106299] ? show_regs.cold.9+0x22/0x2f [12208.110560] cpus=0-3 flags=0x4 nice=0 stuck for 9571s! [12208.110582] Showing busy workqueues and worker pools: [12208.110585] workqueue events: flags=0x0 [12208.110588] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12208.110596] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [12208.115239] ? watchdog_timer_fn.cold.6+0x62/0x10a [12208.115258] ? __hrtimer_run_queues+0x1c2/0x4d0 [12208.115262] ? watchdog+0x40/0x40 [12208.115266] ? hrtimer_interrupt+0x128/0x2c0 [12208.115270] ? smp_apic_timer_interrupt+0xa6/0x270 [12208.115278] ? apic_timer_interrupt+0xf/0x20 [12208.115284] [12208.115289] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12208.115296] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12208.115303] do_raw_spin_lock+0x177/0x190 [12208.115309] _raw_spin_lock+0x21/0x30 [12208.115314] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12208.115580] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12208.115931] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12208.116244] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12208.116456] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12208.116670] ? woken_wake_function+0x30/0x30 [12208.116678] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12208.116908] kthread+0x1d1/0x200 [12208.116914] ? set_kthread_struct+0x70/0x70 [12208.116918] ret_from_fork+0x1f/0x30 [12208.217018] , nfs4_renew_state [nfsv4], push_to_pool [12208.219090] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12208.222090] pending: cpu_latency_work [ptlrpc] [12208.224608] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12208.227639] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12208.231081] workqueue events_long: flags=0x0 [12208.233128] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12208.236641] pending: lu_objects_flush [obdclass] [12208.239304] workqueue events_unbound: flags=0x2 [12208.241489] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12208.243811] pending: flush_memcg_stats_dwork, idle_cull_fn [12208.246044] workqueue events_power_efficient: flags=0x80 [12208.248549] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12208.251602] pending: check_lifetime, do_cache_clean [12208.254181] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12208.256984] pending: neigh_periodic_work, neigh_periodic_work [12208.261021] workqueue mm_percpu_wq: flags=0x8 [12208.262892] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12208.265696] pending: vmstat_update, lru_add_drain_per_cpu [12208.267583] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12208.269694] pending: vmstat_update, lru_add_drain_per_cpu [12208.271486] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12208.273553] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12208.276161] workqueue xprtiod: flags=0xa [12208.277563] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12208.279682] pending: xprt_autoclose, xs_error_handle [12208.281445] workqueue ipv6_addrconf: flags=0x40008 [12208.282966] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12208.284893] pending: addrconf_verify_work [12208.286138] workqueue knbd0-recv: flags=0x1a [12208.287476] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12208.289358] in-flight: 651:recv_work [12208.290431] workqueue ptlrpc_pinger: flags=0x2 [12208.292105] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12208.294049] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12208.296140] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9571s workers=4 idle: 15979 31824 8 [12208.298376] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12198s workers=2 idle: 654 [12220.055010] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [12220.057160] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [12220.061287] Modules linked in: [12220.067038] Modules linked in: [12220.067043] lustre(O) osp(O) [12220.069098] lustre(O) [12220.070537] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12220.070610] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12220.070617] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12220.070620] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12220.070635] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12220.070639] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12220.070643] RAX: 0000000000001970 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12220.070646] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12220.070648] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12220.070650] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12220.070652] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12220.070655] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12220.070658] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12220.070660] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12220.070670] Call Trace: [12220.070675] [12220.070681] ? show_regs.cold.9+0x22/0x2f [12220.070691] ? watchdog_timer_fn.cold.6+0x62/0x10a [12220.070697] ? __hrtimer_run_queues+0x1c2/0x4d0 [12220.070702] ? watchdog+0x40/0x40 [12220.070706] ? hrtimer_interrupt+0x128/0x2c0 [12220.070710] ? smp_apic_timer_interrupt+0xa6/0x270 [12220.070716] ? apic_timer_interrupt+0xf/0x20 [12220.070721] [12220.070724] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12220.070730] do_raw_spin_lock+0x177/0x190 [12220.070734] _raw_spin_lock+0x21/0x30 [12220.070739] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12220.071020] ? do_raw_spin_unlock+0x75/0x190 [12220.071025] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12220.071221] ? do_wait_intr+0xf0/0xf0 [12220.071229] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12220.071581] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12220.071793] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12220.072227] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12220.072421] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12220.072455] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12220.072480] mdc_intent_lock+0x353/0x7e0 [mdc] [12220.072504] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12220.072601] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12220.072817] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12220.072845] lmv_intent_open+0x338/0x1150 [lmv] [12220.072874] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12220.072960] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12220.073040] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12220.073064] ll_intent_lock+0xc5/0xaa0 [lustre] [12220.073149] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12220.073229] ll_lookup_it+0xe0e/0x2270 [lustre] [12220.073389] ? do_raw_spin_unlock+0x75/0x190 [12220.073395] ? _raw_spin_unlock+0x12/0x30 [12220.073402] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12220.073427] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12220.073449] ll_atomic_open+0x36a/0xd70 [lustre] [12220.073548] lookup_open+0x356/0xb40 [12220.073557] do_last+0x313/0xfc0 [12220.073563] ? nd_jump_root+0xe5/0x160 [12220.073568] ? path_init+0x437/0x520 [12220.073572] path_openat+0xf7/0x500 [12220.073578] ? do_raw_spin_unlock+0x75/0x190 [12220.073582] do_filp_open+0x99/0x140 [12220.073589] ? getname_flags+0x6e/0x330 [12220.073594] ? __check_object_size+0xff/0x256 [12220.073601] ? do_raw_spin_unlock+0x75/0x190 [12220.073606] ? _raw_spin_unlock+0x12/0x30 [12220.073610] do_sys_openat2+0x2b4/0x410 [12220.073615] do_sys_open+0x73/0xa0 [12220.073619] __x64_sys_openat+0x24/0x30 [12220.073623] do_syscall_64+0xc1/0x440 [12220.073631] entry_SYSCALL_64_after_hwframe+0x49/0xae [12220.073637] RIP: 0033:0x7f53425a5332 [12220.073644] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12220.073648] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12220.073653] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12220.073655] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12220.073658] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12220.073659] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12220.073661] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12220.720135] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12220.773470] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12220.783036] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12220.790136] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12220.796281] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12220.808688] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12220.818070] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12220.829067] RDX: 0000000000005f54 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12220.839045] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12220.847065] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12220.857991] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12220.860262] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12220.873263] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12220.878047] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12220.889855] Call Trace: [12220.891937] [12220.893919] ? show_regs.cold.9+0x22/0x2f [12220.897268] ? watchdog_timer_fn.cold.6+0x62/0x10a [12220.902984] ? __hrtimer_run_queues+0x1c2/0x4d0 [12220.905978] ? watchdog+0x40/0x40 [12220.911385] ? hrtimer_interrupt+0x128/0x2c0 [12220.914059] ? smp_apic_timer_interrupt+0xa6/0x270 [12220.919285] ? apic_timer_interrupt+0xf/0x20 [12220.922039] [12220.922917] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12220.928882] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12220.934890] do_raw_spin_lock+0x177/0x190 [12220.938102] _raw_spin_lock+0x21/0x30 [12220.940931] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12220.948390] obd_mod_free+0x46/0x140 [ptlrpc] [12220.954079] mdc_commit_open+0x91/0xb0 [mdc] [12220.955209] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12220.970666] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12220.975507] after_reply+0xe2c/0x1780 [ptlrpc] [12220.988253] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12220.990270] ? __next_timer_interrupt+0x160/0x160 [12221.009280] ptlrpcd+0xa82/0xe40 [ptlrpc] [12221.030115] ? do_wait_intr+0xf0/0xf0 [12221.031573] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12221.051530] kthread+0x1d1/0x200 [12221.052807] ? set_kthread_struct+0x70/0x70 [12221.054236] ret_from_fork+0x1f/0x30 [12234.674328] rcu: INFO: rcu_sched self-detected stall on CPU [12234.676419] rcu: 2-....: (5600697 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [12234.689284] (t=10140248 jiffies g=274185 q=66436) [12234.695919] rcu: rcu_sched kthread starved for 9598614 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [12234.706739] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12234.714527] rcu: RCU grace-period kthread stack dump: [12234.721509] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12234.726477] Call Trace: [12234.727306] __schedule+0x351/0xcb0 [12234.736303] schedule+0xc0/0x180 [12234.737986] schedule_timeout+0xb4/0x190 [12234.742547] ? do_raw_spin_unlock+0x75/0x190 [12234.744114] ? __next_timer_interrupt+0x160/0x160 [12234.747955] ? prepare_to_swait_event+0xf3/0x140 [12234.749520] rcu_gp_kthread+0x8dd/0x15b0 [12234.750890] ? force_qs_rnp+0x2c0/0x2c0 [12234.764926] kthread+0x1d1/0x200 [12234.767905] ? set_kthread_struct+0x70/0x70 [12234.774552] ret_from_fork+0x1f/0x30 [12234.775681] rcu: Stack dump where RCU GP kthread last ran: [12234.782451] Sending NMI from CPU 2 to CPUs 3: [12234.787032] NMI backtrace for cpu 3 [12234.787039] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12234.787041] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12234.787043] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12234.787044] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12234.787047] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 [12234.787049] RAX: 0000000000003a53 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12234.787050] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12234.787052] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12234.787054] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12234.787055] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12234.787057] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12234.787059] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12234.787060] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12234.787062] Call Trace: [12234.787063] [12234.787063] ? show_regs.cold.9+0x22/0x2f [12234.787065] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [12234.787066] ? nmi_cpu_backtrace_handler+0x15/0x40 [12234.787067] ? nmi_handle+0xbd/0x210 [12234.787068] ? default_do_nmi+0x4b/0x1b0 [12234.787069] ? do_nmi+0x280/0x380 [12234.787070] ? end_repeat_nmi+0x16/0x1f [12234.787071] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12234.787073] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12234.787074] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12234.787075] [12234.787076] do_raw_spin_lock+0x177/0x190 [12234.787077] _raw_spin_lock+0x21/0x30 [12234.787078] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12234.787079] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12234.787080] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12234.787081] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12234.787082] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12234.787083] ? woken_wake_function+0x30/0x30 [12234.787202] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12234.787204] kthread+0x1d1/0x200 [12234.787205] ? set_kthread_struct+0x70/0x70 [12234.787206] ret_from_fork+0x1f/0x30 [12234.790456] Sending NMI from CPU 2 to CPUs 1: [12234.894494] NMI backtrace for cpu 1 [12234.894502] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12234.894508] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12234.894510] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12234.894511] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12234.894514] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [12234.894516] RAX: 0000000000005217 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12234.894518] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12234.894523] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12234.894524] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12234.894526] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12234.894528] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12234.894529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12234.894533] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12234.894535] Call Trace: [12234.894536] [12234.894537] ? show_regs.cold.9+0x22/0x2f [12234.894538] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [12234.894540] ? nmi_cpu_backtrace_handler+0x15/0x40 [12234.894541] ? nmi_handle+0xbd/0x210 [12234.894542] ? default_do_nmi+0x4b/0x1b0 [12234.894543] ? do_nmi+0x280/0x380 [12234.894544] ? end_repeat_nmi+0x16/0x1f [12234.894545] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12234.894549] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12234.894550] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12234.894551] [12234.894552] do_raw_spin_lock+0x177/0x190 [12234.894553] _raw_spin_lock+0x21/0x30 [12234.894554] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12234.894556] ? do_raw_spin_unlock+0x75/0x190 [12234.894557] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12234.894675] ? do_wait_intr+0xf0/0xf0 [12234.894677] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12234.894678] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12234.894679] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12234.894681] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12234.894682] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12234.894683] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12234.894685] mdc_intent_lock+0x353/0x7e0 [mdc] [12234.894686] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12234.894690] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12234.894691] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12234.894693] lmv_intent_open+0x338/0x1150 [lmv] [12234.894694] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12234.894695] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12234.894696] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12234.894700] ll_intent_lock+0xc5/0xaa0 [lustre] [12234.894702] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12234.894703] ll_lookup_it+0xe0e/0x2270 [lustre] [12234.894704] ? do_raw_spin_unlock+0x75/0x190 [12234.894706] ? _raw_spin_unlock+0x12/0x30 [12234.894707] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12234.894708] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12234.894709] ll_atomic_open+0x36a/0xd70 [lustre] [12234.894710] lookup_open+0x356/0xb40 [12234.894712] do_last+0x313/0xfc0 [12234.894713] ? nd_jump_root+0xe5/0x160 [12234.894716] ? path_init+0x437/0x520 [12234.894717] path_openat+0xf7/0x500 [12234.894718] ? do_raw_spin_unlock+0x75/0x190 [12234.894720] do_filp_open+0x99/0x140 [12234.894721] ? getname_flags+0x6e/0x330 [12234.894722] ? __check_object_size+0xff/0x256 [12234.894723] ? do_raw_spin_unlock+0x75/0x190 [12234.894724] ? _raw_spin_unlock+0x12/0x30 [12234.894728] do_sys_openat2+0x2b4/0x410 [12234.894729] do_sys_open+0x73/0xa0 [12234.894731] __x64_sys_openat+0x24/0x30 [12234.894732] do_syscall_64+0xc1/0x440 [12234.894733] entry_SYSCALL_64_after_hwframe+0x49/0xae [12234.894734] RIP: 0033:0x7f53425a5332 [12234.894736] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12234.894738] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12234.894740] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12234.894742] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12234.894743] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12234.894748] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12234.894749] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12234.895484] NMI backtrace for cpu 2 [12235.059061] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12235.062925] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12235.066352] Call Trace: [12235.066898] [12235.067592] ? dump_stack+0xbb/0x10e [12235.069008] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12235.070544] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12235.072962] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12235.074775] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12235.076555] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12235.078707] ? rcu_dump_cpu_stacks+0x11b/0x171 [12235.080428] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12235.082236] ? update_process_times+0x6d/0xc0 [12235.083973] ? tick_sched_handle.isra.18+0x27/0x90 [12235.085751] ? tick_sched_timer+0x53/0xb0 [12235.087600] ? __hrtimer_run_queues+0x1c2/0x4d0 [12235.089219] ? tick_sched_handle.isra.18+0x90/0x90 [12235.091181] ? hrtimer_interrupt+0x128/0x2c0 [12235.092869] ? smp_apic_timer_interrupt+0xa6/0x270 [12235.094576] ? apic_timer_interrupt+0xf/0x20 [12235.095704] [12235.096216] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12235.097688] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12235.099065] ? do_raw_spin_lock+0x177/0x190 [12235.100554] ? _raw_spin_lock+0x21/0x30 [12235.101949] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12235.104221] ? obd_mod_free+0x46/0x140 [ptlrpc] [12235.106812] ? mdc_commit_open+0x91/0xb0 [mdc] [12235.108677] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12235.111058] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12235.113517] ? after_reply+0xe2c/0x1780 [ptlrpc] [12235.116258] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12235.118818] ? __next_timer_interrupt+0x160/0x160 [12235.120846] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12235.123083] ? do_wait_intr+0xf0/0xf0 [12235.124643] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12235.127412] ? kthread+0x1d1/0x200 [12235.128726] ? set_kthread_struct+0x70/0x70 [12235.130468] ? ret_from_fork+0x1f/0x30 [12236.052010] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [12236.060419] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12236.107514] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12236.114752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12236.122233] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12236.127521] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12236.132049] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12236.149961] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12236.156016] RAX: 0000000000001f15 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12236.166449] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12236.176965] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12236.184585] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12236.189073] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12236.199748] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12236.209745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12236.218021] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12236.224854] Call Trace: [12236.226453] [12236.227152] ? show_regs.cold.9+0x22/0x2f [12236.233783] ? watchdog_timer_fn.cold.6+0x62/0x10a [12236.235424] ? __hrtimer_run_queues+0x1c2/0x4d0 [12236.239509] ? watchdog+0x40/0x40 [12236.240658] ? hrtimer_interrupt+0x128/0x2c0 [12236.245516] ? smp_apic_timer_interrupt+0xa6/0x270 [12236.250794] ? apic_timer_interrupt+0xf/0x20 [12236.254071] [12236.255418] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12236.259759] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12236.270207] ? newidle_balance.isra.93+0x207/0x580 [12236.280542] do_raw_spin_lock+0x177/0x190 [12236.281472] _raw_spin_lock+0x21/0x30 [12236.282269] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12236.286590] process_one_work+0x2c8/0x700 [12236.287739] worker_thread+0x296/0x6e0 [12236.289709] ? rescuer_thread+0x570/0x570 [12236.291211] kthread+0x1d1/0x200 [12236.292755] ? set_kthread_struct+0x70/0x70 [12236.294735] ret_from_fork+0x1f/0x30 [12238.304087] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9601s! [12238.307641] Showing busy workqueues and worker pools: [12238.310709] workqueue events: flags=0x0 [12238.313081] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12238.316265] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12238.322217] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12238.325648] pending: cpu_latency_work [ptlrpc] [12238.328632] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12238.331869] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12238.336610] workqueue events_long: flags=0x0 [12238.339328] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12238.342747] pending: lu_objects_flush [obdclass] [12238.345217] workqueue events_unbound: flags=0x2 [12238.347147] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12238.352870] pending: flush_memcg_stats_dwork, idle_cull_fn [12238.356979] workqueue events_power_efficient: flags=0x80 [12238.361476] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12238.368535] pending: check_lifetime, do_cache_clean [12238.373476] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12238.377537] pending: neigh_periodic_work, neigh_periodic_work [12238.381463] workqueue mm_percpu_wq: flags=0x8 [12238.382927] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12238.388279] pending: vmstat_update, lru_add_drain_per_cpu [12238.390914] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12238.393923] pending: vmstat_update, lru_add_drain_per_cpu [12238.396468] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12238.401497] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12238.404089] workqueue xprtiod: flags=0xa [12238.406712] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12238.409796] pending: xprt_autoclose, xs_error_handle [12238.413564] workqueue ipv6_addrconf: flags=0x40008 [12238.416341] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12238.422397] pending: addrconf_verify_work [12238.425654] workqueue knbd0-recv: flags=0x1a [12238.427741] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12238.431166] in-flight: 651:recv_work [12238.432889] workqueue ptlrpc_pinger: flags=0x2 [12238.435265] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12238.438110] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12238.440673] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9601s workers=4 idle: 15979 31824 8 [12238.444703] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12228s workers=2 idle: 654 [12260.056870] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [12260.057056] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [12260.058010] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [12260.058018] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12260.058097] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12260.058101] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12260.058104] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12260.058116] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12260.058119] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12260.058122] RAX: 0000000000003d3b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12260.058124] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12260.058125] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12260.058127] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12260.058129] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12260.058131] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12260.058133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12260.058134] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12260.058141] Call Trace: [12260.058145] [12260.058149] ? show_regs.cold.9+0x22/0x2f [12260.058156] ? watchdog_timer_fn.cold.6+0x62/0x10a [12260.058160] ? __hrtimer_run_queues+0x1c2/0x4d0 [12260.058163] ? watchdog+0x40/0x40 [12260.058166] ? hrtimer_interrupt+0x128/0x2c0 [12260.058168] ? smp_apic_timer_interrupt+0xa6/0x270 [12260.058175] ? apic_timer_interrupt+0xf/0x20 [12260.058178] [12260.058180] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12260.058183] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12260.058187] do_raw_spin_lock+0x177/0x190 [12260.058190] _raw_spin_lock+0x21/0x30 [12260.058193] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12260.059795] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12260.059870] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12260.059876] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12260.059879] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12260.059947] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12260.059951] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12260.059956] RAX: 000000000000774b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12260.059959] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12260.059961] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12260.059963] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12260.059966] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12260.059969] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12260.059973] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12260.059975] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12260.059983] Call Trace: [12260.059986] [12260.059990] ? show_regs.cold.9+0x22/0x2f [12260.059998] ? watchdog_timer_fn.cold.6+0x62/0x10a [12260.060004] ? __hrtimer_run_queues+0x1c2/0x4d0 [12260.060009] ? watchdog+0x40/0x40 [12260.060014] ? hrtimer_interrupt+0x128/0x2c0 [12260.060018] ? smp_apic_timer_interrupt+0xa6/0x270 [12260.060026] ? apic_timer_interrupt+0xf/0x20 [12260.060031] [12260.060034] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12260.060041] do_raw_spin_lock+0x177/0x190 [12260.060046] _raw_spin_lock+0x21/0x30 [12260.060050] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12260.060295] ? do_raw_spin_unlock+0x75/0x190 [12260.060301] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12260.061049] ? do_wait_intr+0xf0/0xf0 [12260.061058] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12260.061287] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12260.062233] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12260.062976] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12260.063093] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12260.063130] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12260.063158] mdc_intent_lock+0x353/0x7e0 [mdc] [12260.063184] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12260.063269] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12260.058801] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12260.063791] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12260.063986] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12260.064288] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12260.064325] lmv_intent_open+0x338/0x1150 [lmv] [12260.064353] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12260.064743] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12260.064913] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12260.065105] ? woken_wake_function+0x30/0x30 [12260.065523] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12260.065565] ll_intent_lock+0xc5/0xaa0 [lustre] [12260.065113] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12260.065820] kthread+0x1d1/0x200 [12260.065836] ? set_kthread_struct+0x70/0x70 [12260.065840] ret_from_fork+0x1f/0x30 [12260.073603] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12260.073680] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12260.073687] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12260.073697] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12260.073714] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12260.073717] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12260.073720] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12260.073722] RDX: 0000000000007693 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12260.073724] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12260.073725] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12260.073727] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12260.073729] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12260.073731] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12260.073733] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12260.073739] Call Trace: [12260.073743] [12260.073747] ? show_regs.cold.9+0x22/0x2f [12260.073755] ? watchdog_timer_fn.cold.6+0x62/0x10a [12260.073760] ? __hrtimer_run_queues+0x1c2/0x4d0 [12260.073762] ? watchdog+0x40/0x40 [12260.073765] ? hrtimer_interrupt+0x128/0x2c0 [12260.073767] ? smp_apic_timer_interrupt+0xa6/0x270 [12260.073773] ? apic_timer_interrupt+0xf/0x20 [12260.073777] [12260.073778] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12260.073782] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12260.073785] do_raw_spin_lock+0x177/0x190 [12260.073788] _raw_spin_lock+0x21/0x30 [12260.073792] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12260.074025] obd_mod_free+0x46/0x140 [ptlrpc] [12260.074195] mdc_commit_open+0x91/0xb0 [mdc] [12260.074223] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12260.074415] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12260.074673] after_reply+0xe2c/0x1780 [ptlrpc] [12260.074815] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12260.074965] ? __next_timer_interrupt+0x160/0x160 [12260.074972] ptlrpcd+0xa82/0xe40 [ptlrpc] [12260.075160] ? do_wait_intr+0xf0/0xf0 [12260.075167] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12260.084781] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12260.084930] ll_lookup_it+0xe0e/0x2270 [lustre] [12260.085032] ? do_raw_spin_unlock+0x75/0x190 [12260.085041] ? _raw_spin_unlock+0x12/0x30 [12260.085047] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12260.085085] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12260.085106] ll_atomic_open+0x36a/0xd70 [lustre] [12260.085199] lookup_open+0x356/0xb40 [12260.085208] do_last+0x313/0xfc0 [12260.085217] ? nd_jump_root+0xe5/0x160 [12260.085222] ? path_init+0x437/0x520 [12260.085227] path_openat+0xf7/0x500 [12260.085235] ? do_raw_spin_unlock+0x75/0x190 [12260.085240] do_filp_open+0x99/0x140 [12260.085251] ? getname_flags+0x6e/0x330 [12260.085339] ? __check_object_size+0xff/0x256 [12260.085347] ? do_raw_spin_unlock+0x75/0x190 [12260.085351] ? _raw_spin_unlock+0x12/0x30 [12260.085356] do_sys_openat2+0x2b4/0x410 [12260.085361] do_sys_open+0x73/0xa0 [12260.085366] __x64_sys_openat+0x24/0x30 [12260.085370] do_syscall_64+0xc1/0x440 [12260.085377] entry_SYSCALL_64_after_hwframe+0x49/0xae [12260.085389] RIP: 0033:0x7f53425a5332 [12260.085397] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12260.085404] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12260.085409] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12260.085412] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12260.085415] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12260.085417] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12260.085420] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12260.470211] kthread+0x1d1/0x200 [12260.471086] ? set_kthread_struct+0x70/0x70 [12260.472173] ret_from_fork+0x1f/0x30 [12264.054580] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [12264.064370] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12264.097126] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12264.102234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12264.106080] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12264.109399] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [12264.111803] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [12264.122694] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12264.125842] RAX: 00000000000009f9 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12264.129735] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12264.132311] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12264.135711] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12264.143157] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12264.149281] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12264.159156] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12264.165296] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12264.173594] Call Trace: [12264.174481] [12264.175852] ? show_regs.cold.9+0x22/0x2f [12264.180268] ? watchdog_timer_fn.cold.6+0x62/0x10a [12264.182747] ? __hrtimer_run_queues+0x1c2/0x4d0 [12264.190937] ? watchdog+0x40/0x40 [12264.193044] ? hrtimer_interrupt+0x128/0x2c0 [12264.200681] ? smp_apic_timer_interrupt+0xa6/0x270 [12264.202988] ? apic_timer_interrupt+0xf/0x20 [12264.210613] [12264.212855] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [12264.215016] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12264.221179] ? newidle_balance.isra.93+0x207/0x580 [12264.231107] do_raw_spin_lock+0x177/0x190 [12264.232951] _raw_spin_lock+0x21/0x30 [12264.234247] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12264.247027] process_one_work+0x2c8/0x700 [12264.250326] worker_thread+0x296/0x6e0 [12264.253715] ? rescuer_thread+0x570/0x570 [12264.256660] kthread+0x1d1/0x200 [12264.257589] ? set_kthread_struct+0x70/0x70 [12264.262001] ret_from_fork+0x1f/0x30 [12268.511101] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9631s! [12268.514509] Showing busy workqueues and worker pools: [12268.518569] workqueue events: flags=0x0 [12268.533970] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12268.546499] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12268.551250] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12268.556591] pending: cpu_latency_work [ptlrpc] [12268.559293] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12268.564134] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12268.569581] workqueue events_long: flags=0x0 [12268.572266] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12268.575923] pending: lu_objects_flush [obdclass] [12268.582401] workqueue events_unbound: flags=0x2 [12268.585183] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12268.594344] pending: flush_memcg_stats_dwork, idle_cull_fn [12268.600365] workqueue events_power_efficient: flags=0x80 [12268.606883] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12268.612361] pending: check_lifetime, do_cache_clean [12268.617274] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12268.626081] pending: neigh_periodic_work, neigh_periodic_work [12268.632763] workqueue mm_percpu_wq: flags=0x8 [12268.636296] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12268.642349] pending: vmstat_update, lru_add_drain_per_cpu [12268.645971] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12268.648343] pending: vmstat_update, lru_add_drain_per_cpu [12268.652500] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12268.661596] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12268.669469] workqueue xprtiod: flags=0xa [12268.674495] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12268.678868] pending: xprt_autoclose, xs_error_handle [12268.684393] workqueue ipv6_addrconf: flags=0x40008 [12268.690535] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12268.694135] pending: addrconf_verify_work [12268.698360] workqueue knbd0-recv: flags=0x1a [12268.703748] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12268.709308] in-flight: 651:recv_work [12268.711399] workqueue ptlrpc_pinger: flags=0x2 [12268.713463] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12268.717388] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12268.720301] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9631s workers=4 idle: 15979 31824 8 [12268.722152] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12259s workers=2 idle: 654 [12288.055011] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [12288.058068] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [12288.058279] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12288.058365] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12288.058372] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12288.058375] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12288.058389] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12288.058393] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12288.058398] RAX: 000000000000514d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12288.058400] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12288.058403] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12288.058406] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12288.058408] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12288.058412] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12288.058415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12288.058417] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12288.058427] Call Trace: [12288.058433] [12288.058439] ? show_regs.cold.9+0x22/0x2f [12288.058448] ? watchdog_timer_fn.cold.6+0x62/0x10a [12288.058454] ? __hrtimer_run_queues+0x1c2/0x4d0 [12288.058459] ? watchdog+0x40/0x40 [12288.058463] ? hrtimer_interrupt+0x128/0x2c0 [12288.058468] ? smp_apic_timer_interrupt+0xa6/0x270 [12288.058476] ? apic_timer_interrupt+0xf/0x20 [12288.058481] [12288.058485] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12288.058796] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [12288.058803] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12288.058861] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12288.058866] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12288.058869] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12288.058877] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12288.058881] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12288.058885] RAX: 0000000000007f86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12288.058887] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12288.058889] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12288.058892] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12288.058894] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12288.058896] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12288.058899] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12288.058901] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12288.058908] Call Trace: [12288.058911] [12288.058915] ? show_regs.cold.9+0x22/0x2f [12288.058921] ? watchdog_timer_fn.cold.6+0x62/0x10a [12288.058926] ? __hrtimer_run_queues+0x1c2/0x4d0 [12288.058930] ? watchdog+0x40/0x40 [12288.058934] ? hrtimer_interrupt+0x128/0x2c0 [12288.058938] ? smp_apic_timer_interrupt+0xa6/0x270 [12288.058944] ? apic_timer_interrupt+0xf/0x20 [12288.059015] [12288.059019] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12288.059024] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12288.059030] do_raw_spin_lock+0x177/0x190 [12288.059036] _raw_spin_lock+0x21/0x30 [12288.059040] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12288.059482] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12288.059922] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12288.060307] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12288.060640] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12288.060925] ? woken_wake_function+0x30/0x30 [12288.062331] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12288.062783] kthread+0x1d1/0x200 [12288.062791] ? set_kthread_struct+0x70/0x70 [12288.062794] ret_from_fork+0x1f/0x30 [12288.065129] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12288.065946] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12288.065953] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12288.065956] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12288.065970] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12288.065973] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12288.065978] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12288.065981] RDX: 0000000000007510 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12288.065983] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12288.065986] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12288.065988] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12288.065991] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12288.065995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12288.065997] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12288.066006] Call Trace: [12288.066012] [12288.066019] ? show_regs.cold.9+0x22/0x2f [12288.066029] ? watchdog_timer_fn.cold.6+0x62/0x10a [12288.066035] ? __hrtimer_run_queues+0x1c2/0x4d0 [12288.066039] ? watchdog+0x40/0x40 [12288.066044] ? hrtimer_interrupt+0x128/0x2c0 [12288.066048] ? smp_apic_timer_interrupt+0xa6/0x270 [12288.066056] ? apic_timer_interrupt+0xf/0x20 [12288.066061] [12288.066064] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12288.066069] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12288.066074] do_raw_spin_lock+0x177/0x190 [12288.066079] _raw_spin_lock+0x21/0x30 [12288.066086] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12288.066720] obd_mod_free+0x46/0x140 [ptlrpc] [12288.066941] mdc_commit_open+0x91/0xb0 [mdc] [12288.066983] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12288.067188] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12288.096643] do_raw_spin_lock+0x177/0x190 [12288.096666] _raw_spin_lock+0x21/0x30 [12288.096673] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12288.096949] ? do_raw_spin_unlock+0x75/0x190 [12288.096955] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12288.097151] ? do_wait_intr+0xf0/0xf0 [12288.097161] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12288.099171] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12288.099632] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12288.099846] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12288.099958] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12288.099998] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12288.100027] mdc_intent_lock+0x353/0x7e0 [mdc] [12288.100052] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12288.100138] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12288.114151] after_reply+0xe2c/0x1780 [ptlrpc] [12288.114506] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12288.114703] ? __next_timer_interrupt+0x160/0x160 [12288.114714] ptlrpcd+0xa82/0xe40 [ptlrpc] [12288.114918] ? do_wait_intr+0xf0/0xf0 [12288.114926] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12288.120857] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12288.120909] lmv_intent_open+0x338/0x1150 [lmv] [12288.120939] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12288.121046] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12288.121135] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12288.121157] ll_intent_lock+0xc5/0xaa0 [lustre] [12288.121241] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12288.121323] ll_lookup_it+0xe0e/0x2270 [lustre] [12288.123520] ? do_raw_spin_unlock+0x75/0x190 [12288.123530] ? _raw_spin_unlock+0x12/0x30 [12288.123538] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12288.123566] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12288.123589] ll_atomic_open+0x36a/0xd70 [lustre] [12288.123680] lookup_open+0x356/0xb40 [12288.123688] do_last+0x313/0xfc0 [12288.123694] ? nd_jump_root+0xe5/0x160 [12288.123698] ? path_init+0x437/0x520 [12288.123703] path_openat+0xf7/0x500 [12288.123708] ? do_raw_spin_unlock+0x75/0x190 [12288.123713] do_filp_open+0x99/0x140 [12288.123720] ? getname_flags+0x6e/0x330 [12288.123724] ? __check_object_size+0xff/0x256 [12288.123732] ? do_raw_spin_unlock+0x75/0x190 [12288.123736] ? _raw_spin_unlock+0x12/0x30 [12288.123741] do_sys_openat2+0x2b4/0x410 [12288.123745] do_sys_open+0x73/0xa0 [12288.123750] __x64_sys_openat+0x24/0x30 [12288.123754] do_syscall_64+0xc1/0x440 [12288.123761] entry_SYSCALL_64_after_hwframe+0x49/0xae [12288.123768] RIP: 0033:0x7f53425a5332 [12288.123776] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12288.123781] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12288.123786] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12288.123788] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12288.123791] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12288.123793] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12288.123795] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12288.744839] kthread+0x1d1/0x200 [12288.747529] ? set_kthread_struct+0x70/0x70 [12288.748886] ret_from_fork+0x1f/0x30 [12296.052008] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [12296.058100] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12296.100893] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12296.110224] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12296.116807] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12296.122723] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12296.128973] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12296.149143] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12296.153844] RAX: 000000000000645f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12296.157928] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12296.162053] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12296.166829] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12296.171180] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12296.178404] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12296.183438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12296.186961] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12296.189920] Call Trace: [12296.192566] [12296.195206] ? show_regs.cold.9+0x22/0x2f [12296.196527] ? watchdog_timer_fn.cold.6+0x62/0x10a [12296.199059] ? __hrtimer_run_queues+0x1c2/0x4d0 [12296.201394] ? watchdog+0x40/0x40 [12296.202800] ? hrtimer_interrupt+0x128/0x2c0 [12296.205080] ? smp_apic_timer_interrupt+0xa6/0x270 [12296.207856] ? apic_timer_interrupt+0xf/0x20 [12296.210748] [12296.211834] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12296.215494] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12296.219263] ? newidle_balance.isra.93+0x207/0x580 [12296.221011] do_raw_spin_lock+0x177/0x190 [12296.223807] _raw_spin_lock+0x21/0x30 [12296.229895] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12296.233603] process_one_work+0x2c8/0x700 [12296.235881] worker_thread+0x296/0x6e0 [12296.237679] ? rescuer_thread+0x570/0x570 [12296.244464] kthread+0x1d1/0x200 [12296.246676] ? set_kthread_struct+0x70/0x70 [12296.250833] ret_from_fork+0x1f/0x30 [12299.232899] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9662s! [12299.237499] Showing busy workqueues and worker pools: [12299.240263] workqueue events: flags=0x0 [12299.243364] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12299.246834] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12299.253254] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12299.259798] pending: cpu_latency_work [ptlrpc] [12299.263934] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12299.267511] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12299.271891] workqueue events_long: flags=0x0 [12299.274286] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12299.278311] pending: lu_objects_flush [obdclass] [12299.281264] workqueue events_unbound: flags=0x2 [12299.283733] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12299.287293] pending: flush_memcg_stats_dwork, idle_cull_fn [12299.290339] workqueue events_power_efficient: flags=0x80 [12299.293792] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12299.297566] pending: check_lifetime, do_cache_clean [12299.300351] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12299.303611] pending: neigh_periodic_work, neigh_periodic_work [12299.306758] workqueue mm_percpu_wq: flags=0x8 [12299.309214] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12299.312495] pending: vmstat_update, lru_add_drain_per_cpu [12299.315555] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12299.319270] pending: vmstat_update, lru_add_drain_per_cpu [12299.323741] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12299.332935] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12299.339313] workqueue xprtiod: flags=0xa [12299.341979] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12299.346329] pending: xprt_autoclose, xs_error_handle [12299.349381] workqueue ipv6_addrconf: flags=0x40008 [12299.353637] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12299.359148] pending: addrconf_verify_work [12299.362904] workqueue knbd0-recv: flags=0x1a [12299.365771] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12299.370595] in-flight: 651:recv_work [12299.372935] workqueue ptlrpc_pinger: flags=0x2 [12299.376367] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12299.379980] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12299.385149] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9662s workers=4 idle: 15979 31824 8 [12299.390692] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12289s workers=2 idle: 654 [12316.055572] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [12316.057008] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [12316.058027] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [12316.058035] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12316.058121] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12316.058127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12316.058130] RIP: 0010:native_safe_halt+0x2e/0x50 [12316.058143] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12316.058147] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [12316.058152] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12316.058155] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12316.058157] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12316.058160] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12316.058163] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12316.058166] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12316.058170] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12316.058172] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12316.058182] Call Trace: [12316.058188] [12316.058194] ? show_regs.cold.9+0x22/0x2f [12316.058206] ? watchdog_timer_fn.cold.6+0x62/0x10a [12316.058213] ? __hrtimer_run_queues+0x1c2/0x4d0 [12316.058218] ? watchdog+0x40/0x40 [12316.058223] ? hrtimer_interrupt+0x128/0x2c0 [12316.058227] ? smp_apic_timer_interrupt+0xa6/0x270 [12316.058235] ? apic_timer_interrupt+0xf/0x20 [12316.058241] [12316.058244] ? native_safe_halt+0x2e/0x50 [12316.058249] kvm_wait+0xcd/0xe0 [12316.058256] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12316.058265] do_raw_spin_lock+0x177/0x190 [12316.058270] _raw_spin_lock+0x21/0x30 [12316.058274] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12316.058561] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12316.058756] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12316.058989] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12316.059213] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12316.059439] ? woken_wake_function+0x30/0x30 [12316.059448] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12316.059666] kthread+0x1d1/0x200 [12316.059673] ? set_kthread_struct+0x70/0x70 [12316.059676] ret_from_fork+0x1f/0x30 [12316.061484] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12316.061559] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12316.061566] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12316.061570] RIP: 0010:native_safe_halt+0x2e/0x50 [12316.061581] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12316.061585] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12316.061590] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12316.061592] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12316.061594] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12316.061596] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12316.061599] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12316.061602] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12316.061605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12316.061607] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12316.061616] Call Trace: [12316.061621] [12316.061624] ? show_regs.cold.9+0x22/0x2f [12316.061633] ? watchdog_timer_fn.cold.6+0x62/0x10a [12316.061639] ? __hrtimer_run_queues+0x1c2/0x4d0 [12316.061644] ? watchdog+0x40/0x40 [12316.061650] ? hrtimer_interrupt+0x128/0x2c0 [12316.061654] ? smp_apic_timer_interrupt+0xa6/0x270 [12316.061660] ? apic_timer_interrupt+0xf/0x20 [12316.061664] [12316.061667] ? native_safe_halt+0x2e/0x50 [12316.061670] kvm_wait+0xcd/0xe0 [12316.061677] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12316.061685] do_raw_spin_lock+0x177/0x190 [12316.061690] _raw_spin_lock+0x21/0x30 [12316.061694] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12316.067069] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12316.067276] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12316.067284] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12316.067287] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12316.067301] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12316.067306] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12316.067310] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12316.067313] RDX: 0000000000001a50 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12316.067315] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12316.067318] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12316.067320] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12316.067323] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12316.067327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12316.067329] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12316.067337] Call Trace: [12316.067342] [12316.067347] ? show_regs.cold.9+0x22/0x2f [12316.067356] ? watchdog_timer_fn.cold.6+0x62/0x10a [12316.067362] ? __hrtimer_run_queues+0x1c2/0x4d0 [12316.067366] ? watchdog+0x40/0x40 [12316.067371] ? hrtimer_interrupt+0x128/0x2c0 [12316.067375] ? smp_apic_timer_interrupt+0xa6/0x270 [12316.067383] ? apic_timer_interrupt+0xf/0x20 [12316.067388] [12316.067391] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12316.067395] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12316.067400] do_raw_spin_lock+0x177/0x190 [12316.067404] _raw_spin_lock+0x21/0x30 [12316.067410] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12316.076747] ? do_raw_spin_unlock+0x75/0x190 [12316.100481] obd_mod_free+0x46/0x140 [ptlrpc] [12316.105262] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12316.105562] ? do_wait_intr+0xf0/0xf0 [12316.105573] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12316.105801] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12316.106167] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12316.106382] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12316.106495] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12316.106536] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12316.106565] mdc_intent_lock+0x353/0x7e0 [mdc] [12316.106590] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12316.106672] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12316.115258] mdc_commit_open+0x91/0xb0 [mdc] [12316.115308] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12316.115556] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12316.115743] after_reply+0xe2c/0x1780 [ptlrpc] [12316.116105] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12316.116301] ? __next_timer_interrupt+0x160/0x160 [12316.116312] ptlrpcd+0xa82/0xe40 [ptlrpc] [12316.116922] ? do_wait_intr+0xf0/0xf0 [12316.116934] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12316.117204] kthread+0x1d1/0x200 [12316.117210] ? set_kthread_struct+0x70/0x70 [12316.117214] ret_from_fork+0x1f/0x30 [12316.443670] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12316.445195] lmv_intent_open+0x338/0x1150 [lmv] [12316.446571] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12316.448693] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12316.450650] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12316.451947] ll_intent_lock+0xc5/0xaa0 [lustre] [12316.453317] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12316.455218] ll_lookup_it+0xe0e/0x2270 [lustre] [12316.457112] ? do_raw_spin_unlock+0x75/0x190 [12316.458910] ? _raw_spin_unlock+0x12/0x30 [12316.460251] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12316.462043] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12316.463320] ll_atomic_open+0x36a/0xd70 [lustre] [12316.464745] lookup_open+0x356/0xb40 [12316.466238] do_last+0x313/0xfc0 [12316.467591] ? nd_jump_root+0xe5/0x160 [12316.469232] ? path_init+0x437/0x520 [12316.470690] path_openat+0xf7/0x500 [12316.472307] ? do_raw_spin_unlock+0x75/0x190 [12316.473464] do_filp_open+0x99/0x140 [12316.474498] ? getname_flags+0x6e/0x330 [12316.476277] ? __check_object_size+0xff/0x256 [12316.478344] ? do_raw_spin_unlock+0x75/0x190 [12316.480098] ? _raw_spin_unlock+0x12/0x30 [12316.481758] do_sys_openat2+0x2b4/0x410 [12316.483256] do_sys_open+0x73/0xa0 [12316.484746] __x64_sys_openat+0x24/0x30 [12316.486319] do_syscall_64+0xc1/0x440 [12316.487755] entry_SYSCALL_64_after_hwframe+0x49/0xae [12316.489301] RIP: 0033:0x7f53425a5332 [12316.490632] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12316.497239] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12316.501207] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12316.503761] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12316.506972] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12316.510083] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12316.512660] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12324.052056] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [12324.055761] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12324.076440] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12324.080687] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12324.084118] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12324.086658] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12324.089193] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12324.096157] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12324.099038] RAX: 0000000000007485 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12324.101699] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12324.104665] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12324.107849] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12324.110615] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12324.113403] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12324.117165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12324.124310] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12324.127808] Call Trace: [12324.129538] [12324.130373] ? show_regs.cold.9+0x22/0x2f [12324.132640] ? watchdog_timer_fn.cold.6+0x62/0x10a [12324.135445] ? __hrtimer_run_queues+0x1c2/0x4d0 [12324.138313] ? watchdog+0x40/0x40 [12324.141040] ? hrtimer_interrupt+0x128/0x2c0 [12324.142929] ? smp_apic_timer_interrupt+0xa6/0x270 [12324.145061] ? apic_timer_interrupt+0xf/0x20 [12324.146870] [12324.147809] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12324.151433] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12324.154549] ? newidle_balance.isra.93+0x207/0x580 [12324.156267] do_raw_spin_lock+0x177/0x190 [12324.157655] _raw_spin_lock+0x21/0x30 [12324.159344] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12324.161396] process_one_work+0x2c8/0x700 [12324.162604] worker_thread+0x296/0x6e0 [12324.163561] ? rescuer_thread+0x570/0x570 [12324.165482] kthread+0x1d1/0x200 [12324.166834] ? set_kthread_struct+0x70/0x70 [12324.169774] ret_from_fork+0x1f/0x30 [12329.440272] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9692s! [12329.442485] Showing busy workqueues and worker pools: [12329.448153] workqueue events: flags=0x0 [12329.452044] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12329.457073] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12329.463278] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12329.470213] pending: cpu_latency_work [ptlrpc] [12329.475347] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12329.479062] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12329.484639] workqueue events_long: flags=0x0 [12329.486167] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12329.488607] pending: lu_objects_flush [obdclass] [12329.490230] workqueue events_unbound: flags=0x2 [12329.492528] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12329.495594] pending: flush_memcg_stats_dwork, idle_cull_fn [12329.497263] workqueue events_power_efficient: flags=0x80 [12329.498514] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12329.500638] pending: check_lifetime, do_cache_clean [12329.502213] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12329.505386] pending: neigh_periodic_work, neigh_periodic_work [12329.508139] workqueue mm_percpu_wq: flags=0x8 [12329.509997] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12329.513603] pending: vmstat_update, lru_add_drain_per_cpu [12329.515206] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12329.522909] pending: vmstat_update, lru_add_drain_per_cpu [12329.527310] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12329.530546] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12329.533793] workqueue xprtiod: flags=0xa [12329.535010] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12329.538676] pending: xprt_autoclose, xs_error_handle [12329.540470] workqueue ipv6_addrconf: flags=0x40008 [12329.544721] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12329.547399] pending: addrconf_verify_work [12329.549011] workqueue knbd0-recv: flags=0x1a [12329.550558] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12329.559996] in-flight: 651:recv_work [12329.562178] workqueue ptlrpc_pinger: flags=0x2 [12329.563714] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12329.569214] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12329.572065] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9692s workers=4 idle: 15979 31824 8 [12329.575598] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12320s workers=2 idle: 654 [12344.055015] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [12344.057804] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12344.057972] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [12344.058041] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [12344.058047] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12344.058090] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12344.058094] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12344.058096] RIP: 0010:native_safe_halt+0x2e/0x50 [12344.058106] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12344.058109] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12344.058113] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12344.058115] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12344.058116] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12344.058118] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12344.058120] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12344.058123] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12344.058125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12344.058126] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12344.058133] Call Trace: [12344.058138] [12344.058142] ? show_regs.cold.9+0x22/0x2f [12344.058149] ? watchdog_timer_fn.cold.6+0x62/0x10a [12344.058154] ? __hrtimer_run_queues+0x1c2/0x4d0 [12344.058157] ? watchdog+0x40/0x40 [12344.058161] ? hrtimer_interrupt+0x128/0x2c0 [12344.058163] ? smp_apic_timer_interrupt+0xa6/0x270 [12344.058168] ? apic_timer_interrupt+0xf/0x20 [12344.058172] [12344.058173] ? native_safe_halt+0x2e/0x50 [12344.058176] kvm_wait+0xcd/0xe0 [12344.058182] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12344.058187] do_raw_spin_lock+0x177/0x190 [12344.058190] _raw_spin_lock+0x21/0x30 [12344.058193] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12344.060070] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12344.060213] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12344.060395] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12344.061132] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12344.061275] ? woken_wake_function+0x30/0x30 [12344.061282] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12344.062637] kthread+0x1d1/0x200 [12344.062645] ? set_kthread_struct+0x70/0x70 [12344.062648] ret_from_fork+0x1f/0x30 [12344.102370] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12344.102383] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12344.102386] RIP: 0010:native_safe_halt+0x2e/0x50 [12344.102399] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12344.102403] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12344.102408] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12344.102411] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12344.102413] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12344.102415] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12344.102417] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12344.102421] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12344.102424] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12344.102426] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12344.102434] Call Trace: [12344.102440] [12344.102444] ? show_regs.cold.9+0x22/0x2f [12344.102453] ? watchdog_timer_fn.cold.6+0x62/0x10a [12344.102472] ? __hrtimer_run_queues+0x1c2/0x4d0 [12344.102477] ? watchdog+0x40/0x40 [12344.102481] ? hrtimer_interrupt+0x128/0x2c0 [12344.102486] ? smp_apic_timer_interrupt+0xa6/0x270 [12344.102493] ? apic_timer_interrupt+0xf/0x20 [12344.102498] [12344.102500] ? native_safe_halt+0x2e/0x50 [12344.102504] kvm_wait+0xcd/0xe0 [12344.102512] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12344.102522] do_raw_spin_lock+0x177/0x190 [12344.111080] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) [12344.119123] _raw_spin_lock+0x21/0x30 [12344.119147] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12344.119801] ? do_raw_spin_unlock+0x75/0x190 [12344.119810] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12344.120763] ? do_wait_intr+0xf0/0xf0 [12344.120771] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12344.120964] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12344.121168] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12344.121324] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12344.121777] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12344.121820] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12344.121850] mdc_intent_lock+0x353/0x7e0 [mdc] [12344.121959] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12344.122049] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12344.122278] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12344.122308] lmv_intent_open+0x338/0x1150 [lmv] [12344.122340] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12344.122507] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12344.122810] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12344.122838] ll_intent_lock+0xc5/0xaa0 [lustre] [12344.122999] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12344.123082] ll_lookup_it+0xe0e/0x2270 [lustre] [12344.123164] ? do_raw_spin_unlock+0x75/0x190 [12344.123172] ? _raw_spin_unlock+0x12/0x30 [12344.123179] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12344.143378] lquota(O) [12344.147418] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12344.147468] ll_atomic_open+0x36a/0xd70 [lustre] [12344.147597] lookup_open+0x356/0xb40 [12344.147610] do_last+0x313/0xfc0 [12344.147616] ? nd_jump_root+0xe5/0x160 [12344.147621] ? path_init+0x437/0x520 [12344.147626] path_openat+0xf7/0x500 [12344.147631] ? do_raw_spin_unlock+0x75/0x190 [12344.147638] do_filp_open+0x99/0x140 [12344.147646] ? getname_flags+0x6e/0x330 [12344.147650] ? __check_object_size+0xff/0x256 [12344.147658] ? do_raw_spin_unlock+0x75/0x190 [12344.147662] ? _raw_spin_unlock+0x12/0x30 [12344.147668] do_sys_openat2+0x2b4/0x410 [12344.147672] do_sys_open+0x73/0xa0 [12344.147676] __x64_sys_openat+0x24/0x30 [12344.147680] do_syscall_64+0xc1/0x440 [12344.147686] entry_SYSCALL_64_after_hwframe+0x49/0xae [12344.147693] RIP: 0033:0x7f53425a5332 [12344.147701] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12344.147705] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12344.147710] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12344.147712] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12344.147714] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12344.147717] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12344.147719] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12344.155467] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12344.521377] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12344.526910] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12344.530465] RIP: 0010:native_safe_halt+0x2e/0x50 [12344.532753] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12344.543289] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12344.547986] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12344.551654] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12344.555313] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12344.558718] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12344.562792] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12344.566334] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12344.571017] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12344.574201] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12344.577505] Call Trace: [12344.578836] [12344.579764] ? show_regs.cold.9+0x22/0x2f [12344.581828] ? watchdog_timer_fn.cold.6+0x62/0x10a [12344.584164] ? __hrtimer_run_queues+0x1c2/0x4d0 [12344.586380] ? watchdog+0x40/0x40 [12344.587959] ? hrtimer_interrupt+0x128/0x2c0 [12344.590229] ? smp_apic_timer_interrupt+0xa6/0x270 [12344.592299] ? apic_timer_interrupt+0xf/0x20 [12344.594259] [12344.595404] ? native_safe_halt+0x2e/0x50 [12344.597682] kvm_wait+0xcd/0xe0 [12344.599169] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12344.601400] do_raw_spin_lock+0x177/0x190 [12344.603316] _raw_spin_lock+0x21/0x30 [12344.604989] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12344.607963] obd_mod_free+0x46/0x140 [ptlrpc] [12344.610610] mdc_commit_open+0x91/0xb0 [mdc] [12344.612956] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12344.615323] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12344.619223] after_reply+0xe2c/0x1780 [ptlrpc] [12344.621531] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12344.624601] ? __next_timer_interrupt+0x160/0x160 [12344.626813] ptlrpcd+0xa82/0xe40 [ptlrpc] [12344.632242] ? do_wait_intr+0xf0/0xf0 [12344.634980] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12344.637463] kthread+0x1d1/0x200 [12344.639137] ? set_kthread_struct+0x70/0x70 [12344.640909] ret_from_fork+0x1f/0x30 [12356.056944] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [12356.061874] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12356.083257] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12356.089912] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12356.093902] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12356.097271] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12356.100212] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12356.110336] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12356.113649] RAX: 0000000000005411 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12356.116970] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12356.120118] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12356.123397] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12356.126621] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12356.129918] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12356.133919] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12356.136640] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12356.139863] Call Trace: [12356.140949] [12356.141808] ? show_regs.cold.9+0x22/0x2f [12356.143720] ? watchdog_timer_fn.cold.6+0x62/0x10a [12356.145947] ? __hrtimer_run_queues+0x1c2/0x4d0 [12356.148050] ? watchdog+0x40/0x40 [12356.149884] ? hrtimer_interrupt+0x128/0x2c0 [12356.151626] ? smp_apic_timer_interrupt+0xa6/0x270 [12356.153676] ? apic_timer_interrupt+0xf/0x20 [12356.155594] [12356.156482] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12356.159247] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12356.162106] ? newidle_balance.isra.93+0x207/0x580 [12356.164077] do_raw_spin_lock+0x177/0x190 [12356.166231] _raw_spin_lock+0x21/0x30 [12356.167792] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12356.170285] process_one_work+0x2c8/0x700 [12356.171876] worker_thread+0x296/0x6e0 [12356.173461] ? rescuer_thread+0x570/0x570 [12356.174968] kthread+0x1d1/0x200 [12356.176401] ? set_kthread_struct+0x70/0x70 [12356.178093] ret_from_fork+0x1f/0x30 [12359.648228] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9722s! [12359.657251] Showing busy workqueues and worker pools: [12359.661671] workqueue events: flags=0x0 [12359.666569] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12359.677616] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12359.689047] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12359.695651] pending: cpu_latency_work [ptlrpc] [12359.697649] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12359.709149] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12359.715476] workqueue events_long: flags=0x0 [12359.724231] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12359.729405] pending: lu_objects_flush [obdclass] [12359.732359] workqueue events_unbound: flags=0x2 [12359.734456] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12359.737617] pending: flush_memcg_stats_dwork, idle_cull_fn [12359.740447] workqueue events_power_efficient: flags=0x80 [12359.745018] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12359.750858] pending: check_lifetime, do_cache_clean [12359.753874] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12359.764501] pending: neigh_periodic_work, neigh_periodic_work [12359.767779] workqueue mm_percpu_wq: flags=0x8 [12359.771660] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12359.776492] pending: vmstat_update, lru_add_drain_per_cpu [12359.781569] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12359.783196] pending: vmstat_update, lru_add_drain_per_cpu [12359.784678] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12359.791226] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12359.796722] workqueue xprtiod: flags=0xa [12359.798108] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12359.803487] pending: xprt_autoclose, xs_error_handle [12359.805708] workqueue ipv6_addrconf: flags=0x40008 [12359.807473] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12359.813332] pending: addrconf_verify_work [12359.814906] workqueue knbd0-recv: flags=0x1a [12359.816750] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12359.819926] in-flight: 651:recv_work [12359.823307] workqueue ptlrpc_pinger: flags=0x2 [12359.831061] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12359.833945] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12359.837454] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9722s workers=4 idle: 15979 31824 8 [12359.841710] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12350s workers=2 idle: 654 [12372.055014] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [12372.057416] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [12372.060898] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [12372.060917] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12372.061217] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12372.061223] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12372.061226] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12372.061239] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12372.061243] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12372.061248] RAX: 0000000000002090 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12372.061251] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12372.061253] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12372.061256] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12372.061259] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12372.061263] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12372.061266] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12372.061269] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12372.061278] Call Trace: [12372.061284] [12372.061289] ? show_regs.cold.9+0x22/0x2f [12372.061298] ? watchdog_timer_fn.cold.6+0x62/0x10a [12372.061304] ? __hrtimer_run_queues+0x1c2/0x4d0 [12372.061309] ? watchdog+0x40/0x40 [12372.061313] ? hrtimer_interrupt+0x128/0x2c0 [12372.061317] ? smp_apic_timer_interrupt+0xa6/0x270 [12372.061326] ? apic_timer_interrupt+0xf/0x20 [12372.061331] [12372.061335] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12372.061340] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12372.061346] do_raw_spin_lock+0x177/0x190 [12372.061351] _raw_spin_lock+0x21/0x30 [12372.061355] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12372.061667] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12372.062738] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12372.063297] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12372.063790] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl [12372.063521] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12372.064258] ? woken_wake_function+0x30/0x30 [12372.064267] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12372.064488] kthread+0x1d1/0x200 [12372.064494] ? set_kthread_struct+0x70/0x70 [12372.064499] ret_from_fork+0x1f/0x30 [12372.067813] Modules linked in: [12372.070687] i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12372.070719] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12372.094174] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [12372.099331] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12372.099342] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12372.099357] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12372.099362] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12372.099367] RAX: 000000000000045c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12372.099369] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12372.099372] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12372.099374] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12372.099377] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12372.099380] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12372.099383] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12372.099386] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12372.099395] Call Trace: [12372.099399] [12372.099405] ? show_regs.cold.9+0x22/0x2f [12372.104229] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12372.104304] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12372.104312] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12372.104314] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12372.104329] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12372.104334] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12372.104338] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12372.104341] RDX: 000000000000031e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12372.104343] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12372.104346] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12372.104348] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12372.104352] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12372.104355] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12372.104357] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12372.104367] Call Trace: [12372.104372] [12372.104381] ? show_regs.cold.9+0x22/0x2f [12372.104390] ? watchdog_timer_fn.cold.6+0x62/0x10a [12372.104396] ? __hrtimer_run_queues+0x1c2/0x4d0 [12372.104401] ? watchdog+0x40/0x40 [12372.104405] ? hrtimer_interrupt+0x128/0x2c0 [12372.104408] ? smp_apic_timer_interrupt+0xa6/0x270 [12372.104416] ? apic_timer_interrupt+0xf/0x20 [12372.104420] [12372.104424] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12372.104429] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12372.104433] do_raw_spin_lock+0x177/0x190 [12372.104437] _raw_spin_lock+0x21/0x30 [12372.104443] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12372.108092] ? watchdog_timer_fn.cold.6+0x62/0x10a [12372.116825] obd_mod_free+0x46/0x140 [ptlrpc] [12372.120241] ? __hrtimer_run_queues+0x1c2/0x4d0 [12372.120264] ? watchdog+0x40/0x40 [12372.124251] mdc_commit_open+0x91/0xb0 [mdc] [12372.128274] ? hrtimer_interrupt+0x128/0x2c0 [12372.128293] ? smp_apic_timer_interrupt+0xa6/0x270 [12372.128303] ? apic_timer_interrupt+0xf/0x20 [12372.128308] [12372.128313] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12372.128323] do_raw_spin_lock+0x177/0x190 [12372.128330] _raw_spin_lock+0x21/0x30 [12372.128335] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12372.132113] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12372.135633] ? do_raw_spin_unlock+0x75/0x190 [12372.139825] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12372.143882] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12372.147182] after_reply+0xe2c/0x1780 [ptlrpc] [12372.150883] ? do_wait_intr+0xf0/0xf0 [12372.150906] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12372.152689] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12372.153765] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12372.157247] ? __next_timer_interrupt+0x160/0x160 [12372.163210] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12372.166794] ptlrpcd+0xa82/0xe40 [ptlrpc] [12372.168274] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12372.169624] ? do_wait_intr+0xf0/0xf0 [12372.171630] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12372.174074] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12372.175122] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12372.178090] kthread+0x1d1/0x200 [12372.180849] mdc_intent_lock+0x353/0x7e0 [mdc] [12372.183270] ? set_kthread_struct+0x70/0x70 [12372.184876] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12372.189403] ret_from_fork+0x1f/0x30 [12372.191683] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12372.436826] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12372.439448] lmv_intent_open+0x338/0x1150 [lmv] [12372.441297] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12372.443722] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12372.446242] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12372.448128] ll_intent_lock+0xc5/0xaa0 [lustre] [12372.450271] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12372.452596] ll_lookup_it+0xe0e/0x2270 [lustre] [12372.454665] ? do_raw_spin_unlock+0x75/0x190 [12372.456694] ? _raw_spin_unlock+0x12/0x30 [12372.458367] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12372.460523] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12372.462982] ll_atomic_open+0x36a/0xd70 [lustre] [12372.465134] lookup_open+0x356/0xb40 [12372.466758] do_last+0x313/0xfc0 [12372.467848] ? nd_jump_root+0xe5/0x160 [12372.468914] ? path_init+0x437/0x520 [12372.470717] path_openat+0xf7/0x500 [12372.471894] ? do_raw_spin_unlock+0x75/0x190 [12372.473803] do_filp_open+0x99/0x140 [12372.475142] ? getname_flags+0x6e/0x330 [12372.477139] ? __check_object_size+0xff/0x256 [12372.478731] ? do_raw_spin_unlock+0x75/0x190 [12372.480322] ? _raw_spin_unlock+0x12/0x30 [12372.481794] do_sys_openat2+0x2b4/0x410 [12372.483342] do_sys_open+0x73/0xa0 [12372.484578] __x64_sys_openat+0x24/0x30 [12372.486381] do_syscall_64+0xc1/0x440 [12372.488005] entry_SYSCALL_64_after_hwframe+0x49/0xae [12372.490838] RIP: 0033:0x7f53425a5332 [12372.492480] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12372.501225] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12372.504398] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12372.507327] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12372.510687] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12372.513422] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12372.516148] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12384.052964] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [12384.063984] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12384.107818] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12384.113842] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12384.119231] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12384.125832] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12384.131302] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12384.159805] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12384.170969] RAX: 0000000000006928 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12384.180910] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12384.196906] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12384.206913] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12384.213774] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12384.223878] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12384.234241] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12384.244743] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12384.255832] Call Trace: [12384.257417] [12384.260255] ? show_regs.cold.9+0x22/0x2f [12384.265880] ? watchdog_timer_fn.cold.6+0x62/0x10a [12384.271142] ? __hrtimer_run_queues+0x1c2/0x4d0 [12384.275151] ? watchdog+0x40/0x40 [12384.279117] ? hrtimer_interrupt+0x128/0x2c0 [12384.285703] ? smp_apic_timer_interrupt+0xa6/0x270 [12384.288064] ? apic_timer_interrupt+0xf/0x20 [12384.293339] [12384.294846] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12384.297144] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12384.300337] ? newidle_balance.isra.93+0x207/0x580 [12384.305546] do_raw_spin_lock+0x177/0x190 [12384.311186] _raw_spin_lock+0x21/0x30 [12384.312751] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12384.323857] process_one_work+0x2c8/0x700 [12384.325880] worker_thread+0x296/0x6e0 [12384.328178] ? rescuer_thread+0x570/0x570 [12384.334437] kthread+0x1d1/0x200 [12384.337971] ? set_kthread_struct+0x70/0x70 [12384.341449] ret_from_fork+0x1f/0x30 [12389.859112] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9752s! [12389.868271] Showing busy workqueues and worker pools: [12389.870160] workqueue events: flags=0x0 [12389.872289] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12389.877051] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12389.885936] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12389.891606] pending: cpu_latency_work [ptlrpc] [12389.895200] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12389.905071] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12389.911257] workqueue events_long: flags=0x0 [12389.916241] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12389.922855] pending: lu_objects_flush [obdclass] [12389.933415] workqueue events_unbound: flags=0x2 [12389.939788] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12389.945682] pending: flush_memcg_stats_dwork, idle_cull_fn [12389.950831] workqueue events_power_efficient: flags=0x80 [12389.953659] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12389.957393] pending: check_lifetime, do_cache_clean [12389.959996] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12389.963333] pending: neigh_periodic_work, neigh_periodic_work [12389.970414] workqueue mm_percpu_wq: flags=0x8 [12389.976151] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12389.982216] pending: vmstat_update, lru_add_drain_per_cpu [12389.985433] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12389.989824] pending: vmstat_update, lru_add_drain_per_cpu [12389.992889] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12389.998051] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12390.000885] workqueue xprtiod: flags=0xa [12390.003908] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12390.013358] pending: xprt_autoclose, xs_error_handle [12390.020863] workqueue ipv6_addrconf: flags=0x40008 [12390.027716] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12390.032924] pending: addrconf_verify_work [12390.035442] workqueue knbd0-recv: flags=0x1a [12390.037030] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12390.043744] in-flight: 651:recv_work [12390.050767] workqueue ptlrpc_pinger: flags=0x2 [12390.055895] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12390.060786] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12390.069239] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9752s workers=4 idle: 15979 31824 8 [12390.075966] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12380s workers=2 idle: 654 [12400.055267] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [12400.058699] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12400.059939] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [12400.059994] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [12400.060001] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12400.060062] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12400.060067] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12400.060070] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12400.060085] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12400.060089] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12400.060094] RAX: 0000000000007d3c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12400.060097] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12400.060099] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12400.060101] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12400.060103] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12400.060106] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12400.060109] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12400.060111] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12400.060121] Call Trace: [12400.060126] [12400.060132] ? show_regs.cold.9+0x22/0x2f [12400.060142] ? watchdog_timer_fn.cold.6+0x62/0x10a [12400.060147] ? __hrtimer_run_queues+0x1c2/0x4d0 [12400.060151] ? watchdog+0x40/0x40 [12400.060155] ? hrtimer_interrupt+0x128/0x2c0 [12400.060159] ? smp_apic_timer_interrupt+0xa6/0x270 [12400.060168] ? apic_timer_interrupt+0xf/0x20 [12400.060173] [12400.060175] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12400.060180] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12400.060186] do_raw_spin_lock+0x177/0x190 [12400.060190] _raw_spin_lock+0x21/0x30 [12400.060196] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12400.061256] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12400.062207] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12400.062441] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12400.062787] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12400.063630] ? woken_wake_function+0x30/0x30 [12400.063640] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12400.064102] kthread+0x1d1/0x200 [12400.064108] ? set_kthread_struct+0x70/0x70 [12400.064112] ret_from_fork+0x1f/0x30 [12400.088310] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12400.088325] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12400.088328] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12400.088342] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12400.088346] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12400.088352] RAX: 0000000000004c0f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12400.088354] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12400.088357] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12400.088359] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12400.088361] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12400.088364] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12400.088366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12400.088369] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12400.088386] Call Trace: [12400.088959] [12400.088966] ? show_regs.cold.9+0x22/0x2f [12400.088975] ? watchdog_timer_fn.cold.6+0x62/0x10a [12400.088980] ? __hrtimer_run_queues+0x1c2/0x4d0 [12400.088983] ? watchdog+0x40/0x40 [12400.088986] ? hrtimer_interrupt+0x128/0x2c0 [12400.088988] ? smp_apic_timer_interrupt+0xa6/0x270 [12400.088995] ? apic_timer_interrupt+0xf/0x20 [12400.088998] [12400.089000] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12400.089006] do_raw_spin_lock+0x177/0x190 [12400.089009] _raw_spin_lock+0x21/0x30 [12400.089012] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12400.089232] ? do_raw_spin_unlock+0x75/0x190 [12400.089249] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12400.094321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) [12400.097632] ? do_wait_intr+0xf0/0xf0 [12400.097656] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12400.097931] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12400.140179] mdd(O) [12400.152575] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12400.152863] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12400.153085] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12400.153120] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12400.153148] mdc_intent_lock+0x353/0x7e0 [mdc] [12400.153295] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12400.153440] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12400.153717] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12400.153747] lmv_intent_open+0x338/0x1150 [lmv] [12400.153781] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12400.153869] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12400.154069] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12400.154093] ll_intent_lock+0xc5/0xaa0 [lustre] [12400.154192] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12400.154290] ll_lookup_it+0xe0e/0x2270 [lustre] [12400.154393] ? do_raw_spin_unlock+0x75/0x190 [12400.154400] ? _raw_spin_unlock+0x12/0x30 [12400.154407] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12400.154437] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12400.154517] ll_atomic_open+0x36a/0xd70 [lustre] [12400.154611] lookup_open+0x356/0xb40 [12400.154621] do_last+0x313/0xfc0 [12400.154626] ? nd_jump_root+0xe5/0x160 [12400.154631] ? path_init+0x437/0x520 [12400.154636] path_openat+0xf7/0x500 [12400.154641] ? do_raw_spin_unlock+0x75/0x190 [12400.154646] do_filp_open+0x99/0x140 [12400.154653] ? getname_flags+0x6e/0x330 [12400.154657] ? __check_object_size+0xff/0x256 [12400.154664] ? do_raw_spin_unlock+0x75/0x190 [12400.154669] ? _raw_spin_unlock+0x12/0x30 [12400.154674] do_sys_openat2+0x2b4/0x410 [12400.154679] do_sys_open+0x73/0xa0 [12400.154683] __x64_sys_openat+0x24/0x30 [12400.154687] do_syscall_64+0xc1/0x440 [12400.154693] entry_SYSCALL_64_after_hwframe+0x49/0xae [12400.154700] RIP: 0033:0x7f53425a5332 [12400.154707] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12400.154711] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12400.154717] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12400.154719] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12400.154721] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12400.154724] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12400.154726] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12400.687073] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12400.709917] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12400.720160] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12400.725825] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12400.728615] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12400.740782] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12400.743707] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12400.749009] RDX: 0000000000006fce RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12400.758503] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12400.766369] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12400.773123] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12400.782788] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12400.791939] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12400.799054] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12400.804997] Call Trace: [12400.807190] [12400.809213] ? show_regs.cold.9+0x22/0x2f [12400.812227] ? watchdog_timer_fn.cold.6+0x62/0x10a [12400.815119] ? __hrtimer_run_queues+0x1c2/0x4d0 [12400.817294] ? watchdog+0x40/0x40 [12400.818695] ? hrtimer_interrupt+0x128/0x2c0 [12400.820582] ? smp_apic_timer_interrupt+0xa6/0x270 [12400.822787] ? apic_timer_interrupt+0xf/0x20 [12400.824691] [12400.828648] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12400.830315] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12400.832834] do_raw_spin_lock+0x177/0x190 [12400.834564] _raw_spin_lock+0x21/0x30 [12400.842855] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12400.846184] obd_mod_free+0x46/0x140 [ptlrpc] [12400.848530] mdc_commit_open+0x91/0xb0 [mdc] [12400.850710] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12400.855035] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12400.861042] after_reply+0xe2c/0x1780 [ptlrpc] [12400.868421] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12400.870472] ? __next_timer_interrupt+0x160/0x160 [12400.873839] ptlrpcd+0xa82/0xe40 [ptlrpc] [12400.877866] ? do_wait_intr+0xf0/0xf0 [12400.883253] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12400.891752] kthread+0x1d1/0x200 [12400.898738] ? set_kthread_struct+0x70/0x70 [12400.904922] ret_from_fork+0x1f/0x30 [12414.677027] rcu: INFO: rcu_sched self-detected stall on CPU [12414.679793] rcu: 2-....: (5713711 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [12414.685316] (t=10320251 jiffies g=274185 q=66483) [12414.688446] rcu: rcu_sched kthread starved for 9778617 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [12414.693782] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12414.698588] rcu: RCU grace-period kthread stack dump: [12414.700804] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12414.705640] Call Trace: [12414.706921] __schedule+0x351/0xcb0 [12414.708763] schedule+0xc0/0x180 [12414.710310] schedule_timeout+0xb4/0x190 [12414.714047] ? do_raw_spin_unlock+0x75/0x190 [12414.718516] ? __next_timer_interrupt+0x160/0x160 [12414.724722] ? prepare_to_swait_event+0xf3/0x140 [12414.728925] rcu_gp_kthread+0x8dd/0x15b0 [12414.734965] ? force_qs_rnp+0x2c0/0x2c0 [12414.742160] kthread+0x1d1/0x200 [12414.744926] ? set_kthread_struct+0x70/0x70 [12414.746312] ret_from_fork+0x1f/0x30 [12414.749947] rcu: Stack dump where RCU GP kthread last ran: [12414.756042] Sending NMI from CPU 2 to CPUs 1: [12414.760787] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [12414.761596] Sending NMI from CPU 2 to CPUs 1: [12414.775562] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [12414.778930] NMI backtrace for cpu 2 [12414.786339] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12414.792539] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12414.798906] Call Trace: [12414.800058] [12414.801436] ? dump_stack+0xbb/0x10e [12414.803100] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12414.804614] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12414.805959] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12414.808735] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12414.812212] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12414.814786] ? rcu_dump_cpu_stacks+0x11b/0x171 [12414.817107] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12414.819935] ? update_process_times+0x6d/0xc0 [12414.822778] ? tick_sched_handle.isra.18+0x27/0x90 [12414.827452] ? tick_sched_timer+0x53/0xb0 [12414.833297] ? __hrtimer_run_queues+0x1c2/0x4d0 [12414.835902] ? tick_sched_handle.isra.18+0x90/0x90 [12414.846393] ? hrtimer_interrupt+0x128/0x2c0 [12414.852235] ? smp_apic_timer_interrupt+0xa6/0x270 [12414.863874] ? apic_timer_interrupt+0xf/0x20 [12414.867386] [12414.874429] ? native_safe_halt+0x2e/0x50 [12414.875716] ? kvm_wait+0xcd/0xe0 [12414.876824] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12414.886636] ? do_raw_spin_lock+0x177/0x190 [12414.887795] ? _raw_spin_lock+0x21/0x30 [12414.910431] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12414.912194] ? obd_mod_free+0x46/0x140 [ptlrpc] [12414.919357] ? mdc_commit_open+0x91/0xb0 [mdc] [12414.928263] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12414.933102] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12414.945049] ? after_reply+0xe2c/0x1780 [ptlrpc] [12414.946836] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12414.957022] ? __next_timer_interrupt+0x160/0x160 [12414.962319] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12414.964794] ? do_wait_intr+0xf0/0xf0 [12414.966732] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12414.969718] ? kthread+0x1d1/0x200 [12414.971055] ? set_kthread_struct+0x70/0x70 [12414.973712] ? ret_from_fork+0x1f/0x30 [12416.056228] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [12416.064556] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12416.115560] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12416.137598] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12416.155663] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12416.167230] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12416.178860] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12416.203876] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12416.219504] RAX: 0000000000002787 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12416.230851] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12416.240717] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12416.253199] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12416.267913] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12416.274879] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12416.285518] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12416.294880] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12416.306481] Call Trace: [12416.308470] [12416.311875] ? show_regs.cold.9+0x22/0x2f [12416.317201] ? watchdog_timer_fn.cold.6+0x62/0x10a [12416.322114] ? __hrtimer_run_queues+0x1c2/0x4d0 [12416.324645] ? watchdog+0x40/0x40 [12416.325787] ? hrtimer_interrupt+0x128/0x2c0 [12416.327777] ? smp_apic_timer_interrupt+0xa6/0x270 [12416.329540] ? apic_timer_interrupt+0xf/0x20 [12416.335630] [12416.336338] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12416.338001] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12416.344016] ? newidle_balance.isra.93+0x207/0x580 [12416.348428] do_raw_spin_lock+0x177/0x190 [12416.349654] _raw_spin_lock+0x21/0x30 [12416.351465] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12416.353797] process_one_work+0x2c8/0x700 [12416.355275] worker_thread+0x296/0x6e0 [12416.356579] ? rescuer_thread+0x570/0x570 [12416.358958] kthread+0x1d1/0x200 [12416.360148] ? set_kthread_struct+0x70/0x70 [12416.361911] ret_from_fork+0x1f/0x30 [12420.578332] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9783s! [12420.588107] Showing busy workqueues and worker pools: [12420.590888] workqueue events: flags=0x0 [12420.600253] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12420.609007] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12420.622708] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12420.628324] pending: cpu_latency_work [ptlrpc] [12420.634130] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12420.643633] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12420.653227] workqueue events_long: flags=0x0 [12420.658768] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12420.663702] pending: lu_objects_flush [obdclass] [12420.667779] workqueue events_unbound: flags=0x2 [12420.673150] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12420.682521] pending: flush_memcg_stats_dwork, idle_cull_fn [12420.692234] workqueue events_power_efficient: flags=0x80 [12420.699387] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12420.707932] pending: check_lifetime, do_cache_clean [12420.713462] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12420.719784] pending: neigh_periodic_work, neigh_periodic_work [12420.723133] workqueue mm_percpu_wq: flags=0x8 [12420.725526] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12420.731840] pending: vmstat_update, lru_add_drain_per_cpu [12420.744618] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12420.751958] pending: vmstat_update, lru_add_drain_per_cpu [12420.756300] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12420.759935] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12420.765145] workqueue xprtiod: flags=0xa [12420.771217] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12420.776789] pending: xprt_autoclose, xs_error_handle [12420.781418] workqueue ipv6_addrconf: flags=0x40008 [12420.783172] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12420.785747] pending: addrconf_verify_work [12420.790275] workqueue knbd0-recv: flags=0x1a [12420.792791] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12420.798716] in-flight: 651:recv_work [12420.800266] workqueue ptlrpc_pinger: flags=0x2 [12420.802943] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12420.805895] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12420.813267] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9783s workers=4 idle: 15979 31824 8 [12420.820597] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12411s workers=2 idle: 654 [12428.058027] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [12428.060952] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12428.086925] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12428.092194] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12428.108608] RIP: 0010:native_safe_halt+0x2e/0x50 [12428.110842] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12428.135247] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12428.139386] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12428.142974] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12428.146661] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12428.149613] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12428.152633] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12428.157062] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12428.162287] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12428.165743] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12428.169744] Call Trace: [12428.170858] [12428.172058] ? show_regs.cold.9+0x22/0x2f [12428.173717] ? watchdog_timer_fn.cold.6+0x62/0x10a [12428.175831] ? __hrtimer_run_queues+0x1c2/0x4d0 [12428.178319] ? watchdog+0x40/0x40 [12428.179508] ? hrtimer_interrupt+0x128/0x2c0 [12428.181564] ? smp_apic_timer_interrupt+0xa6/0x270 [12428.184642] ? apic_timer_interrupt+0xf/0x20 [12428.186853] [12428.187636] ? native_safe_halt+0x2e/0x50 [12428.189108] kvm_wait+0xcd/0xe0 [12428.190872] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12428.193592] do_raw_spin_lock+0x177/0x190 [12428.195793] _raw_spin_lock+0x21/0x30 [12428.197623] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12428.202649] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12428.206117] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12428.210304] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12428.215587] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12428.221632] ? woken_wake_function+0x30/0x30 [12428.223711] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12428.230407] kthread+0x1d1/0x200 [12428.231600] ? set_kthread_struct+0x70/0x70 [12428.234601] ret_from_fork+0x1f/0x30 [12440.055599] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [12440.060796] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12440.061225] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [12440.080525] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12440.080539] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12440.080542] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12440.080556] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12440.080561] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12440.080567] RAX: 0000000000002dec RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12440.080570] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12440.080573] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12440.080576] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12440.080578] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12440.080582] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12440.080585] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12440.080587] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12440.080595] Call Trace: [12440.080600] [12440.080605] ? show_regs.cold.9+0x22/0x2f [12440.080614] ? watchdog_timer_fn.cold.6+0x62/0x10a [12440.080619] ? __hrtimer_run_queues+0x1c2/0x4d0 [12440.080624] ? watchdog+0x40/0x40 [12440.080628] ? hrtimer_interrupt+0x128/0x2c0 [12440.080632] ? smp_apic_timer_interrupt+0xa6/0x270 [12440.080640] ? apic_timer_interrupt+0xf/0x20 [12440.080644] [12440.080647] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12440.090066] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12440.090150] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12440.090156] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12440.090159] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12440.090173] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12440.090177] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12440.090181] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12440.090183] RDX: 0000000000000a3f RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12440.090185] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12440.090187] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12440.090189] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12440.090192] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12440.090194] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12440.090196] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12440.090205] Call Trace: [12440.090211] [12440.090216] ? show_regs.cold.9+0x22/0x2f [12440.090226] ? watchdog_timer_fn.cold.6+0x62/0x10a [12440.090232] ? __hrtimer_run_queues+0x1c2/0x4d0 [12440.090237] ? watchdog+0x40/0x40 [12440.090241] ? hrtimer_interrupt+0x128/0x2c0 [12440.090245] ? smp_apic_timer_interrupt+0xa6/0x270 [12440.090254] ? apic_timer_interrupt+0xf/0x20 [12440.090259] [12440.090262] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12440.090267] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12440.090271] do_raw_spin_lock+0x177/0x190 [12440.090276] _raw_spin_lock+0x21/0x30 [12440.090283] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12440.090563] obd_mod_free+0x46/0x140 [ptlrpc] [12440.091023] mdc_commit_open+0x91/0xb0 [mdc] [12440.091063] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12440.096389] do_raw_spin_lock+0x177/0x190 [12440.104584] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12440.107932] _raw_spin_lock+0x21/0x30 [12440.107958] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12440.123067] after_reply+0xe2c/0x1780 [ptlrpc] [12440.127509] ? do_raw_spin_unlock+0x75/0x190 [12440.132495] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12440.135932] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12440.141996] ? __next_timer_interrupt+0x160/0x160 [12440.145548] ? do_wait_intr+0xf0/0xf0 [12440.145572] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12440.146042] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12440.146620] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12440.147080] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12440.147270] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12440.147373] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12440.147411] mdc_intent_lock+0x353/0x7e0 [mdc] [12440.147449] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12440.147548] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12440.147972] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12440.147999] lmv_intent_open+0x338/0x1150 [lmv] [12440.148031] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12440.148184] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12440.148269] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12440.148565] ll_intent_lock+0xc5/0xaa0 [lustre] [12440.149233] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12440.149384] ll_lookup_it+0xe0e/0x2270 [lustre] [12440.149476] ? do_raw_spin_unlock+0x75/0x190 [12440.149484] ? _raw_spin_unlock+0x12/0x30 [12440.149491] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12440.149518] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12440.149538] ll_atomic_open+0x36a/0xd70 [lustre] [12440.149623] lookup_open+0x356/0xb40 [12440.149632] do_last+0x313/0xfc0 [12440.149637] ? nd_jump_root+0xe5/0x160 [12440.149642] ? path_init+0x437/0x520 [12440.149648] path_openat+0xf7/0x500 [12440.149653] ? do_raw_spin_unlock+0x75/0x190 [12440.149658] do_filp_open+0x99/0x140 [12440.149664] ? getname_flags+0x6e/0x330 [12440.149773] ? __check_object_size+0xff/0x256 [12440.149964] ? do_raw_spin_unlock+0x75/0x190 [12440.149970] ? _raw_spin_unlock+0x12/0x30 [12440.149974] do_sys_openat2+0x2b4/0x410 [12440.149979] do_sys_open+0x73/0xa0 [12440.149983] __x64_sys_openat+0x24/0x30 [12440.149987] do_syscall_64+0xc1/0x440 [12440.149993] entry_SYSCALL_64_after_hwframe+0x49/0xae [12440.150000] RIP: 0033:0x7f53425a5332 [12440.150007] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12440.150011] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12440.150017] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12440.150019] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12440.150022] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12440.150024] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12440.150027] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12440.472530] ptlrpcd+0xa82/0xe40 [ptlrpc] [12440.475215] ? do_wait_intr+0xf0/0xf0 [12440.476907] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12440.479331] kthread+0x1d1/0x200 [12440.486428] ? set_kthread_struct+0x70/0x70 [12440.487708] ret_from_fork+0x1f/0x30 [12448.052313] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [12448.056412] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12448.082870] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12448.096871] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12448.101382] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12448.104697] RIP: 0010:native_safe_halt+0x2e/0x50 [12448.107666] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12448.121129] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12448.127426] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12448.135061] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [12448.143837] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12448.152749] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12448.156694] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12448.160514] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12448.165026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12448.171369] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12448.176138] Call Trace: [12448.177499] [12448.178646] ? show_regs.cold.9+0x22/0x2f [12448.182370] ? watchdog_timer_fn.cold.6+0x62/0x10a [12448.189116] ? __hrtimer_run_queues+0x1c2/0x4d0 [12448.194928] ? watchdog+0x40/0x40 [12448.198026] ? hrtimer_interrupt+0x128/0x2c0 [12448.202014] ? smp_apic_timer_interrupt+0xa6/0x270 [12448.207291] ? apic_timer_interrupt+0xf/0x20 [12448.209889] [12448.212264] ? native_safe_halt+0x2e/0x50 [12448.216000] kvm_wait+0xcd/0xe0 [12448.219067] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12448.224985] ? newidle_balance.isra.93+0x207/0x580 [12448.230047] do_raw_spin_lock+0x177/0x190 [12448.232192] _raw_spin_lock+0x21/0x30 [12448.234921] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12448.240069] process_one_work+0x2c8/0x700 [12448.244097] worker_thread+0x296/0x6e0 [12448.247915] ? rescuer_thread+0x570/0x570 [12448.251661] kthread+0x1d1/0x200 [12448.254426] ? set_kthread_struct+0x70/0x70 [12448.258453] ret_from_fork+0x1f/0x30 [12451.300319] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9814s! [12451.314873] Showing busy workqueues and worker pools: [12451.324522] workqueue events: flags=0x0 [12451.325885] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12451.339663] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12451.370988] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12451.378986] pending: cpu_latency_work [ptlrpc] [12451.385672] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12451.394704] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12451.405858] workqueue events_long: flags=0x0 [12451.407327] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12451.421221] pending: lu_objects_flush [obdclass] [12451.427886] workqueue events_unbound: flags=0x2 [12451.429520] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12451.440896] pending: flush_memcg_stats_dwork, idle_cull_fn [12451.443376] workqueue events_power_efficient: flags=0x80 [12451.446051] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12451.449711] pending: check_lifetime, do_cache_clean [12451.452150] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12451.455452] pending: neigh_periodic_work, neigh_periodic_work [12451.458262] workqueue mm_percpu_wq: flags=0x8 [12451.460224] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12451.463418] pending: vmstat_update, lru_add_drain_per_cpu [12451.467100] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12451.470354] pending: vmstat_update, lru_add_drain_per_cpu [12451.476129] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12451.479258] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12451.482827] workqueue xprtiod: flags=0xa [12451.484890] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12451.488463] pending: xprt_autoclose, xs_error_handle [12451.490724] workqueue ipv6_addrconf: flags=0x40008 [12451.492631] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12451.496174] pending: addrconf_verify_work [12451.498550] workqueue knbd0-recv: flags=0x1a [12451.500382] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12451.503427] in-flight: 651:recv_work [12451.506433] workqueue ptlrpc_pinger: flags=0x2 [12451.508537] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12451.511867] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12451.516094] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9814s workers=4 idle: 15979 31824 8 [12451.523867] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12441s workers=2 idle: 654 [12456.058026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [12456.062237] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12456.100892] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12456.114232] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12456.126313] RIP: 0010:native_safe_halt+0x2e/0x50 [12456.129341] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12456.140931] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12456.145929] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12456.155381] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12456.159066] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12456.161324] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12456.165062] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12456.174168] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12456.179539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12456.187051] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12456.194145] Call Trace: [12456.197880] [12456.201551] ? show_regs.cold.9+0x22/0x2f [12456.203543] ? watchdog_timer_fn.cold.6+0x62/0x10a [12456.212171] ? __hrtimer_run_queues+0x1c2/0x4d0 [12456.219279] ? watchdog+0x40/0x40 [12456.220933] ? hrtimer_interrupt+0x128/0x2c0 [12456.223721] ? smp_apic_timer_interrupt+0xa6/0x270 [12456.227195] ? apic_timer_interrupt+0xf/0x20 [12456.233175] [12456.234151] ? native_safe_halt+0x2e/0x50 [12456.237043] kvm_wait+0xcd/0xe0 [12456.243203] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12456.247140] do_raw_spin_lock+0x177/0x190 [12456.253081] _raw_spin_lock+0x21/0x30 [12456.254590] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12456.261910] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12456.269032] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12456.276874] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12456.286157] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12456.293741] ? woken_wake_function+0x30/0x30 [12456.296354] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12456.303290] kthread+0x1d1/0x200 [12456.308458] ? set_kthread_struct+0x70/0x70 [12456.311212] ret_from_fork+0x1f/0x30 [12468.057072] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [12468.059566] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12468.060406] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [12468.082306] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12468.082321] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12468.082324] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12468.082338] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12468.082343] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12468.082349] RAX: 0000000000001ba3 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12468.082351] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12468.082354] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12468.082356] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12468.082359] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12468.082363] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12468.082366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12468.082368] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12468.082377] Call Trace: [12468.082382] [12468.082388] ? show_regs.cold.9+0x22/0x2f [12468.082398] ? watchdog_timer_fn.cold.6+0x62/0x10a [12468.082404] ? __hrtimer_run_queues+0x1c2/0x4d0 [12468.082409] ? watchdog+0x40/0x40 [12468.082414] ? hrtimer_interrupt+0x128/0x2c0 [12468.082418] ? smp_apic_timer_interrupt+0xa6/0x270 [12468.082427] ? apic_timer_interrupt+0xf/0x20 [12468.082432] [12468.082435] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12468.082443] do_raw_spin_lock+0x177/0x190 [12468.082449] _raw_spin_lock+0x21/0x30 [12468.082453] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12468.082997] ? do_raw_spin_unlock+0x75/0x190 [12468.083004] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12468.083282] ? do_wait_intr+0xf0/0xf0 [12468.083291] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12468.083604] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12468.089231] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12468.089302] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12468.089309] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12468.089311] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12468.089324] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12468.089328] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12468.089332] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12468.089335] RDX: 00000000000002c1 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12468.089337] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12468.089339] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12468.089341] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12468.089344] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12468.089347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12468.089350] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12468.089358] Call Trace: [12468.089363] [12468.089369] ? show_regs.cold.9+0x22/0x2f [12468.089377] ? watchdog_timer_fn.cold.6+0x62/0x10a [12468.089383] ? __hrtimer_run_queues+0x1c2/0x4d0 [12468.089388] ? watchdog+0x40/0x40 [12468.089391] ? hrtimer_interrupt+0x128/0x2c0 [12468.089395] ? smp_apic_timer_interrupt+0xa6/0x270 [12468.089402] ? apic_timer_interrupt+0xf/0x20 [12468.089406] [12468.089409] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12468.089413] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12468.089417] do_raw_spin_lock+0x177/0x190 [12468.089422] _raw_spin_lock+0x21/0x30 [12468.089427] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12468.089689] obd_mod_free+0x46/0x140 [ptlrpc] [12468.089883] mdc_commit_open+0x91/0xb0 [mdc] [12468.089920] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12468.090112] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12468.090873] after_reply+0xe2c/0x1780 [ptlrpc] [12468.091067] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12468.091460] ? __next_timer_interrupt+0x160/0x160 [12468.091470] ptlrpcd+0xa82/0xe40 [ptlrpc] [12468.103498] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12468.104470] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12468.105033] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12468.105075] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12468.105105] mdc_intent_lock+0x353/0x7e0 [mdc] [12468.105132] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12468.114113] ? do_wait_intr+0xf0/0xf0 [12468.114461] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12468.114809] kthread+0x1d1/0x200 [12468.114816] ? set_kthread_struct+0x70/0x70 [12468.114820] ret_from_fork+0x1f/0x30 [12468.380886] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12468.383517] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12468.386018] lmv_intent_open+0x338/0x1150 [lmv] [12468.388519] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12468.390648] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12468.392980] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12468.394604] ll_intent_lock+0xc5/0xaa0 [lustre] [12468.396585] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12468.398869] ll_lookup_it+0xe0e/0x2270 [lustre] [12468.400842] ? do_raw_spin_unlock+0x75/0x190 [12468.402531] ? _raw_spin_unlock+0x12/0x30 [12468.404314] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12468.405960] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12468.407430] ll_atomic_open+0x36a/0xd70 [lustre] [12468.409318] lookup_open+0x356/0xb40 [12468.410649] do_last+0x313/0xfc0 [12468.411873] ? nd_jump_root+0xe5/0x160 [12468.413104] ? path_init+0x437/0x520 [12468.414626] path_openat+0xf7/0x500 [12468.416286] ? do_raw_spin_unlock+0x75/0x190 [12468.417957] do_filp_open+0x99/0x140 [12468.419480] ? getname_flags+0x6e/0x330 [12468.420778] ? __check_object_size+0xff/0x256 [12468.422578] ? do_raw_spin_unlock+0x75/0x190 [12468.424307] ? _raw_spin_unlock+0x12/0x30 [12468.425672] do_sys_openat2+0x2b4/0x410 [12468.427033] do_sys_open+0x73/0xa0 [12468.428256] __x64_sys_openat+0x24/0x30 [12468.429813] do_syscall_64+0xc1/0x440 [12468.431120] entry_SYSCALL_64_after_hwframe+0x49/0xae [12468.433232] RIP: 0033:0x7f53425a5332 [12468.434580] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12468.441689] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12468.444458] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12468.447008] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12468.449437] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12468.452055] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12468.454380] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12476.052421] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [12476.054618] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12476.095009] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12476.102022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12476.110613] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12476.117817] RIP: 0010:native_safe_halt+0x2e/0x50 [12476.119453] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12476.142730] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12476.147716] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12476.153244] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [12476.159800] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12476.179903] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12476.185894] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12476.200360] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12476.203238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12476.207609] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12476.215560] Call Trace: [12476.216882] [12476.217840] ? show_regs.cold.9+0x22/0x2f [12476.219445] ? watchdog_timer_fn.cold.6+0x62/0x10a [12476.221495] ? __hrtimer_run_queues+0x1c2/0x4d0 [12476.225446] ? watchdog+0x40/0x40 [12476.227018] ? hrtimer_interrupt+0x128/0x2c0 [12476.231884] ? smp_apic_timer_interrupt+0xa6/0x270 [12476.235831] ? apic_timer_interrupt+0xf/0x20 [12476.237883] [12476.239216] ? native_safe_halt+0x2e/0x50 [12476.243359] kvm_wait+0xcd/0xe0 [12476.245275] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12476.247929] ? newidle_balance.isra.93+0x207/0x580 [12476.250682] do_raw_spin_lock+0x177/0x190 [12476.252550] _raw_spin_lock+0x21/0x30 [12476.255427] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12476.258807] process_one_work+0x2c8/0x700 [12476.260471] worker_thread+0x296/0x6e0 [12476.263274] ? rescuer_thread+0x570/0x570 [12476.264955] kthread+0x1d1/0x200 [12476.266706] ? set_kthread_struct+0x70/0x70 [12476.268406] ret_from_fork+0x1f/0x30 [12482.021224] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9844s! [12482.029456] Showing busy workqueues and worker pools: [12482.034788] workqueue events: flags=0x0 [12482.041355] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12482.050424] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12482.072189] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12482.081923] pending: cpu_latency_work [ptlrpc] [12482.089111] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12482.097793] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12482.111366] workqueue events_long: flags=0x0 [12482.116277] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12482.120379] pending: lu_objects_flush [obdclass] [12482.125813] workqueue events_unbound: flags=0x2 [12482.132601] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12482.136201] pending: flush_memcg_stats_dwork, idle_cull_fn [12482.140887] workqueue events_power_efficient: flags=0x80 [12482.145781] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12482.155882] pending: check_lifetime, do_cache_clean [12482.164071] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12482.171953] pending: neigh_periodic_work, neigh_periodic_work [12482.177443] workqueue mm_percpu_wq: flags=0x8 [12482.179905] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12482.184950] pending: vmstat_update, lru_add_drain_per_cpu [12482.196372] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12482.204891] pending: vmstat_update, lru_add_drain_per_cpu [12482.209275] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12482.216336] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12482.224044] workqueue xprtiod: flags=0xa [12482.225524] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12482.228550] pending: xprt_autoclose, xs_error_handle [12482.231114] workqueue ipv6_addrconf: flags=0x40008 [12482.233179] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12482.235696] pending: addrconf_verify_work [12482.237580] workqueue knbd0-recv: flags=0x1a [12482.239336] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12482.242005] in-flight: 651:recv_work [12482.249369] workqueue ptlrpc_pinger: flags=0x2 [12482.254232] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12482.256197] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12482.263479] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9845s workers=4 idle: 15979 31824 8 [12482.269760] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12472s workers=2 idle: 654 [12484.058486] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [12484.067882] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12484.101703] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12484.109494] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12484.115846] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12484.123532] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12484.143885] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12484.154864] RAX: 0000000000007a06 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12484.166643] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12484.177296] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12484.187336] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12484.191667] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12484.196786] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12484.214695] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12484.228438] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12484.238021] Call Trace: [12484.238880] [12484.239574] ? show_regs.cold.9+0x22/0x2f [12484.245567] ? watchdog_timer_fn.cold.6+0x62/0x10a [12484.255100] ? __hrtimer_run_queues+0x1c2/0x4d0 [12484.260551] ? watchdog+0x40/0x40 [12484.261613] ? hrtimer_interrupt+0x128/0x2c0 [12484.269339] ? smp_apic_timer_interrupt+0xa6/0x270 [12484.274710] ? apic_timer_interrupt+0xf/0x20 [12484.289307] [12484.295142] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12484.299387] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12484.309298] do_raw_spin_lock+0x177/0x190 [12484.312239] _raw_spin_lock+0x21/0x30 [12484.318846] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12484.326098] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12484.330310] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12484.334087] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12484.337097] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12484.340088] ? woken_wake_function+0x30/0x30 [12484.342206] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12484.346048] kthread+0x1d1/0x200 [12484.348815] ? set_kthread_struct+0x70/0x70 [12484.351347] ret_from_fork+0x1f/0x30 [12496.055058] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [12496.057049] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [12496.058087] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [12496.062882] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12496.063386] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12496.063394] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12496.063397] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12496.063545] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12496.063551] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12496.063556] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12496.063558] RDX: 0000000000002ec3 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12496.063560] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12496.063562] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12496.063564] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12496.063567] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12496.063570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12496.063572] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12496.063580] Call Trace: [12496.063598] [12496.063604] ? show_regs.cold.9+0x22/0x2f [12496.063613] ? watchdog_timer_fn.cold.6+0x62/0x10a [12496.063619] ? __hrtimer_run_queues+0x1c2/0x4d0 [12496.063623] ? watchdog+0x40/0x40 [12496.063627] ? hrtimer_interrupt+0x128/0x2c0 [12496.063631] ? smp_apic_timer_interrupt+0xa6/0x270 [12496.063638] ? apic_timer_interrupt+0xf/0x20 [12496.063643] [12496.063645] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12496.063649] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12496.063653] do_raw_spin_lock+0x177/0x190 [12496.063657] _raw_spin_lock+0x21/0x30 [12496.063664] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12496.068216] mdt(O) [12496.094013] obd_mod_free+0x46/0x140 [ptlrpc] [12496.098287] mdd(O) mgs(O) lquota(O) [12496.113862] mdc_commit_open+0x91/0xb0 [mdc] [12496.113915] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12496.114177] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12496.115308] after_reply+0xe2c/0x1780 [ptlrpc] [12496.115510] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12496.116077] ? __next_timer_interrupt+0x160/0x160 [12496.116089] ptlrpcd+0xa82/0xe40 [ptlrpc] [12496.116291] ? do_wait_intr+0xf0/0xf0 [12496.116301] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12496.116498] kthread+0x1d1/0x200 [12496.116504] ? set_kthread_struct+0x70/0x70 [12496.116507] ret_from_fork+0x1f/0x30 [12496.394654] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12496.448635] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12496.465423] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12496.476222] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12496.481913] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12496.498522] RSP: 0018:ffffbb5381503500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [12496.507171] RAX: 00000000000014d9 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12496.512008] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12496.520744] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12496.529652] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12496.535408] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12496.541659] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12496.548099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12496.550890] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12496.555655] Call Trace: [12496.557923] [12496.558647] ? show_regs.cold.9+0x22/0x2f [12496.560299] ? watchdog_timer_fn.cold.6+0x62/0x10a [12496.562210] ? __hrtimer_run_queues+0x1c2/0x4d0 [12496.565802] ? watchdog+0x40/0x40 [12496.568924] ? hrtimer_interrupt+0x128/0x2c0 [12496.571225] ? smp_apic_timer_interrupt+0xa6/0x270 [12496.575441] ? apic_timer_interrupt+0xf/0x20 [12496.578107] [12496.578871] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12496.586588] do_raw_spin_lock+0x177/0x190 [12496.591664] _raw_spin_lock+0x21/0x30 [12496.604677] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12496.606593] ? do_raw_spin_unlock+0x75/0x190 [12496.609111] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12496.612235] ? do_wait_intr+0xf0/0xf0 [12496.614746] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12496.629325] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12496.633754] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12496.644927] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12496.653401] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12496.659102] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12496.660590] mdc_intent_lock+0x353/0x7e0 [mdc] [12496.666988] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12496.676154] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12496.683522] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12496.687854] lmv_intent_open+0x338/0x1150 [lmv] [12496.689436] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12496.696999] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12496.701191] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12496.708973] ll_intent_lock+0xc5/0xaa0 [lustre] [12496.715500] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12496.722573] ll_lookup_it+0xe0e/0x2270 [lustre] [12496.725625] ? do_raw_spin_unlock+0x75/0x190 [12496.727495] ? _raw_spin_unlock+0x12/0x30 [12496.729753] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12496.731811] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12496.734540] ll_atomic_open+0x36a/0xd70 [lustre] [12496.741082] lookup_open+0x356/0xb40 [12496.743087] do_last+0x313/0xfc0 [12496.744852] ? nd_jump_root+0xe5/0x160 [12496.749657] ? path_init+0x437/0x520 [12496.751644] path_openat+0xf7/0x500 [12496.759496] ? do_raw_spin_unlock+0x75/0x190 [12496.766999] do_filp_open+0x99/0x140 [12496.775553] ? getname_flags+0x6e/0x330 [12496.777417] ? __check_object_size+0xff/0x256 [12496.785180] ? do_raw_spin_unlock+0x75/0x190 [12496.786759] ? _raw_spin_unlock+0x12/0x30 [12496.790551] do_sys_openat2+0x2b4/0x410 [12496.792906] do_sys_open+0x73/0xa0 [12496.795073] __x64_sys_openat+0x24/0x30 [12496.801375] do_syscall_64+0xc1/0x440 [12496.803527] entry_SYSCALL_64_after_hwframe+0x49/0xae [12496.806640] RIP: 0033:0x7f53425a5332 [12496.816494] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12496.834005] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12496.838669] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12496.842617] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12496.847695] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12496.853613] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12496.859591] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12508.052111] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [12508.060553] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12508.104608] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12508.118405] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12508.123356] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12508.129508] RIP: 0010:native_safe_halt+0x2e/0x50 [12508.134548] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12508.146796] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12508.153401] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12508.157475] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [12508.167797] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12508.171310] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12508.178421] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12508.185044] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12508.188760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12508.190781] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12508.193878] Call Trace: [12508.195309] [12508.196390] ? show_regs.cold.9+0x22/0x2f [12508.198563] ? watchdog_timer_fn.cold.6+0x62/0x10a [12508.201897] ? __hrtimer_run_queues+0x1c2/0x4d0 [12508.204456] ? watchdog+0x40/0x40 [12508.206704] ? hrtimer_interrupt+0x128/0x2c0 [12508.210604] ? smp_apic_timer_interrupt+0xa6/0x270 [12508.213045] ? apic_timer_interrupt+0xf/0x20 [12508.215734] [12508.227120] ? native_safe_halt+0x2e/0x50 [12508.228445] kvm_wait+0xcd/0xe0 [12508.230841] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12508.238311] ? newidle_balance.isra.93+0x207/0x580 [12508.239960] do_raw_spin_lock+0x177/0x190 [12508.241017] _raw_spin_lock+0x21/0x30 [12508.246997] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12508.251960] process_one_work+0x2c8/0x700 [12508.254410] worker_thread+0x296/0x6e0 [12508.256914] ? rescuer_thread+0x570/0x570 [12508.259727] kthread+0x1d1/0x200 [12508.261592] ? set_kthread_struct+0x70/0x70 [12508.271325] ret_from_fork+0x1f/0x30 [12512.058028] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [12512.060482] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12512.113245] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12512.126314] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12512.132339] RIP: 0010:native_safe_halt+0x2e/0x50 [12512.135210] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12512.152262] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12512.161081] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12512.170043] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12512.175173] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12512.183911] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12512.190125] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12512.200246] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12512.203840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12512.207784] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12512.216185] Call Trace: [12512.218006] [12512.220607] ? show_regs.cold.9+0x22/0x2f [12512.222261] ? watchdog_timer_fn.cold.6+0x62/0x10a [12512.227263] ? __hrtimer_run_queues+0x1c2/0x4d0 [12512.231216] ? watchdog+0x40/0x40 [12512.232834] ? hrtimer_interrupt+0x128/0x2c0 [12512.237184] ? smp_apic_timer_interrupt+0xa6/0x270 [12512.247570] ? apic_timer_interrupt+0xf/0x20 [12512.253799] [12512.254518] ? native_safe_halt+0x2e/0x50 [12512.255759] kvm_wait+0xcd/0xe0 [12512.258317] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12512.261887] do_raw_spin_lock+0x177/0x190 [12512.266319] _raw_spin_lock+0x21/0x30 [12512.270086] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12512.273881] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12512.283063] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12512.289457] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12512.296540] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12512.302352] ? woken_wake_function+0x30/0x30 [12512.305192] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12512.312120] kthread+0x1d1/0x200 [12512.315394] ? set_kthread_struct+0x70/0x70 [12512.320906] ret_from_fork+0x1f/0x30 [12512.735084] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9875s! [12512.737227] Showing busy workqueues and worker pools: [12512.740293] workqueue events: flags=0x0 [12512.742686] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12512.747310] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12512.756550] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12512.760150] pending: cpu_latency_work [ptlrpc] [12512.764534] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12512.767099] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12512.772537] workqueue events_long: flags=0x0 [12512.773815] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12512.776776] pending: lu_objects_flush [obdclass] [12512.781584] workqueue events_unbound: flags=0x2 [12512.786528] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12512.789848] pending: flush_memcg_stats_dwork, idle_cull_fn [12512.793549] workqueue events_power_efficient: flags=0x80 [12512.795480] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12512.799755] pending: check_lifetime, do_cache_clean [12512.801626] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12512.806550] pending: neigh_periodic_work, neigh_periodic_work [12512.810792] workqueue mm_percpu_wq: flags=0x8 [12512.815940] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12512.820900] pending: vmstat_update, lru_add_drain_per_cpu [12512.828065] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12512.831134] pending: vmstat_update, lru_add_drain_per_cpu [12512.834090] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12512.838508] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12512.841633] workqueue xprtiod: flags=0xa [12512.843747] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12512.850719] pending: xprt_autoclose, xs_error_handle [12512.852666] workqueue ipv6_addrconf: flags=0x40008 [12512.856577] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12512.858802] pending: addrconf_verify_work [12512.867928] workqueue knbd0-recv: flags=0x1a [12512.870245] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12512.876184] in-flight: 651:recv_work [12512.878682] workqueue ptlrpc_pinger: flags=0x2 [12512.880584] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12512.885520] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12512.893383] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9875s workers=4 idle: 15979 31824 8 [12512.899151] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12503s workers=2 idle: 654 [12524.055015] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [12524.057023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [12524.058559] Modules linked in: [12524.062750] Modules linked in: [12524.062754] lustre(O) osp(O) [12524.064114] lustre(O) [12524.065558] ofd(O) [12524.066728] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) [12524.067729] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12524.067798] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12524.067805] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12524.067808] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12524.067822] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12524.067825] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12524.067830] RAX: 000000000000510d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12524.067832] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12524.067835] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12524.067837] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12524.067839] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12524.067842] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12524.067846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12524.067848] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12524.067856] Call Trace: [12524.067863] [12524.067868] ? show_regs.cold.9+0x22/0x2f [12524.067878] ? watchdog_timer_fn.cold.6+0x62/0x10a [12524.067883] ? __hrtimer_run_queues+0x1c2/0x4d0 [12524.067887] ? watchdog+0x40/0x40 [12524.067891] ? hrtimer_interrupt+0x128/0x2c0 [12524.067895] ? smp_apic_timer_interrupt+0xa6/0x270 [12524.067903] ? apic_timer_interrupt+0xf/0x20 [12524.067913] [12524.067916] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12524.067923] do_raw_spin_lock+0x177/0x190 [12524.067927] _raw_spin_lock+0x21/0x30 [12524.067933] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12524.068991] lfsck(O) [12524.070589] ? do_raw_spin_unlock+0x75/0x190 [12524.088147] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12524.088266] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12524.088273] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12524.088277] RIP: 0010:native_safe_halt+0x2e/0x50 [12524.088289] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12524.088293] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12524.088298] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12524.088300] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12524.088302] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12524.088304] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12524.088307] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12524.088310] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12524.088313] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12524.088315] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12524.088322] Call Trace: [12524.088327] [12524.088331] ? show_regs.cold.9+0x22/0x2f [12524.088340] ? watchdog_timer_fn.cold.6+0x62/0x10a [12524.088346] ? __hrtimer_run_queues+0x1c2/0x4d0 [12524.088350] ? watchdog+0x40/0x40 [12524.088354] ? hrtimer_interrupt+0x128/0x2c0 [12524.088357] ? smp_apic_timer_interrupt+0xa6/0x270 [12524.088364] ? apic_timer_interrupt+0xf/0x20 [12524.088368] [12524.088371] ? native_safe_halt+0x2e/0x50 [12524.088374] kvm_wait+0xcd/0xe0 [12524.088381] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12524.088387] do_raw_spin_lock+0x177/0x190 [12524.088391] _raw_spin_lock+0x21/0x30 [12524.088396] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12524.088695] obd_mod_free+0x46/0x140 [ptlrpc] [12524.088821] mdc_commit_open+0x91/0xb0 [mdc] [12524.088850] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12524.088980] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12524.089152] after_reply+0xe2c/0x1780 [ptlrpc] [12524.089403] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12524.089652] ? __next_timer_interrupt+0x160/0x160 [12524.089660] ptlrpcd+0xa82/0xe40 [ptlrpc] [12524.089793] ? do_wait_intr+0xf0/0xf0 [12524.089800] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12524.089931] kthread+0x1d1/0x200 [12524.089935] ? set_kthread_struct+0x70/0x70 [12524.089938] ret_from_fork+0x1f/0x30 [12524.247372] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12524.249548] ? do_wait_intr+0xf0/0xf0 [12524.251054] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12524.253282] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12524.255574] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12524.258004] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12524.260140] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12524.262369] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12524.264167] mdc_intent_lock+0x353/0x7e0 [mdc] [12524.266010] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12524.268021] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12524.270574] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12524.272759] lmv_intent_open+0x338/0x1150 [lmv] [12524.274665] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12524.276938] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12524.279498] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12524.282169] ll_intent_lock+0xc5/0xaa0 [lustre] [12524.284304] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12524.288083] ll_lookup_it+0xe0e/0x2270 [lustre] [12524.289961] ? do_raw_spin_unlock+0x75/0x190 [12524.294049] ? _raw_spin_unlock+0x12/0x30 [12524.297140] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12524.303165] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12524.305545] ll_atomic_open+0x36a/0xd70 [lustre] [12524.309770] lookup_open+0x356/0xb40 [12524.311168] do_last+0x313/0xfc0 [12524.315059] ? nd_jump_root+0xe5/0x160 [12524.316908] ? path_init+0x437/0x520 [12524.321747] path_openat+0xf7/0x500 [12524.323120] ? do_raw_spin_unlock+0x75/0x190 [12524.329173] do_filp_open+0x99/0x140 [12524.331549] ? getname_flags+0x6e/0x330 [12524.333133] ? __check_object_size+0xff/0x256 [12524.334571] ? do_raw_spin_unlock+0x75/0x190 [12524.338584] ? _raw_spin_unlock+0x12/0x30 [12524.341377] do_sys_openat2+0x2b4/0x410 [12524.343011] do_sys_open+0x73/0xa0 [12524.344644] __x64_sys_openat+0x24/0x30 [12524.346593] do_syscall_64+0xc1/0x440 [12524.348328] entry_SYSCALL_64_after_hwframe+0x49/0xae [12524.351925] RIP: 0033:0x7f53425a5332 [12524.354910] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12524.370330] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12524.373972] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12524.377737] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12524.381033] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12524.389930] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12524.392155] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12540.053809] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [12540.059517] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12540.060148] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [12540.123016] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12540.123029] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12540.123035] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12540.123314] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12540.123324] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12540.123328] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12540.123332] RAX: 0000000000007c94 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12540.123335] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12540.123337] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12540.123339] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12540.123341] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12540.123345] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12540.123348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12540.123350] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12540.123359] Call Trace: [12540.123365] [12540.123370] ? show_regs.cold.9+0x22/0x2f [12540.123379] ? watchdog_timer_fn.cold.6+0x62/0x10a [12540.123386] ? __hrtimer_run_queues+0x1c2/0x4d0 [12540.123391] ? watchdog+0x40/0x40 [12540.123395] ? hrtimer_interrupt+0x128/0x2c0 [12540.123399] ? smp_apic_timer_interrupt+0xa6/0x270 [12540.123408] ? apic_timer_interrupt+0xf/0x20 [12540.123413] [12540.123416] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12540.123421] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12540.123425] ? newidle_balance.isra.93+0x207/0x580 [12540.123431] do_raw_spin_lock+0x177/0x190 [12540.123436] _raw_spin_lock+0x21/0x30 [12540.123442] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12540.123651] process_one_work+0x2c8/0x700 [12540.123659] worker_thread+0x296/0x6e0 [12540.123663] ? rescuer_thread+0x570/0x570 [12540.123667] kthread+0x1d1/0x200 [12540.123671] ? set_kthread_struct+0x70/0x70 [12540.123675] ret_from_fork+0x1f/0x30 [12540.272825] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12540.308294] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12540.312773] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12540.316220] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12540.318495] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12540.338803] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12540.343597] RAX: 0000000000007635 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12540.348188] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12540.352571] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12540.354770] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12540.364724] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12540.375163] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12540.384324] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12540.390947] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12540.399991] Call Trace: [12540.400788] [12540.401889] ? show_regs.cold.9+0x22/0x2f [12540.407682] ? watchdog_timer_fn.cold.6+0x62/0x10a [12540.409650] ? __hrtimer_run_queues+0x1c2/0x4d0 [12540.411315] ? watchdog+0x40/0x40 [12540.414189] ? hrtimer_interrupt+0x128/0x2c0 [12540.415747] ? smp_apic_timer_interrupt+0xa6/0x270 [12540.417803] ? apic_timer_interrupt+0xf/0x20 [12540.422885] [12540.424021] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12540.429266] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12540.431474] do_raw_spin_lock+0x177/0x190 [12540.433303] _raw_spin_lock+0x21/0x30 [12540.435130] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12540.438483] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12540.445873] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12540.450461] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12540.455674] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12540.461252] ? woken_wake_function+0x30/0x30 [12540.464424] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12540.471578] kthread+0x1d1/0x200 [12540.472531] ? set_kthread_struct+0x70/0x70 [12540.476857] ret_from_fork+0x1f/0x30 [12542.944052] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9905s! [12542.948712] Showing busy workqueues and worker pools: [12542.950854] workqueue events: flags=0x0 [12542.953060] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12542.956232] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12542.961165] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12542.964055] pending: cpu_latency_work [ptlrpc] [12542.966452] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12542.969307] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12542.973843] workqueue events_long: flags=0x0 [12542.975542] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12542.978285] pending: lu_objects_flush [obdclass] [12542.980628] workqueue events_unbound: flags=0x2 [12542.982588] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12542.985123] pending: flush_memcg_stats_dwork, idle_cull_fn [12542.987357] workqueue events_power_efficient: flags=0x80 [12542.989799] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12542.992587] pending: check_lifetime, do_cache_clean [12542.994540] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12542.997436] pending: neigh_periodic_work, neigh_periodic_work [12542.999952] workqueue mm_percpu_wq: flags=0x8 [12543.001772] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12543.004810] pending: vmstat_update, lru_add_drain_per_cpu [12543.007342] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12543.010373] pending: vmstat_update, lru_add_drain_per_cpu [12543.012628] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12543.015378] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12543.017853] workqueue xprtiod: flags=0xa [12543.020373] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12543.024195] pending: xprt_autoclose, xs_error_handle [12543.026612] workqueue ipv6_addrconf: flags=0x40008 [12543.028762] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12543.031726] pending: addrconf_verify_work [12543.033608] workqueue knbd0-recv: flags=0x1a [12543.035694] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12543.038252] in-flight: 651:recv_work [12543.040291] workqueue ptlrpc_pinger: flags=0x2 [12543.041849] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12543.044449] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12543.047163] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9905s workers=4 idle: 15979 31824 8 [12543.050370] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12533s workers=2 idle: 654 [12552.055025] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [12552.058755] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12552.059769] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [12552.095617] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12552.095632] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12552.095635] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12552.095649] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12552.095659] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12552.095665] RAX: 0000000000007588 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12552.095668] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12552.095670] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12552.095673] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12552.095675] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12552.095678] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12552.095681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12552.095684] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12552.095692] Call Trace: [12552.095698] [12552.095703] ? show_regs.cold.9+0x22/0x2f [12552.095731] ? watchdog_timer_fn.cold.6+0x62/0x10a [12552.095737] ? __hrtimer_run_queues+0x1c2/0x4d0 [12552.095741] ? watchdog+0x40/0x40 [12552.095746] ? hrtimer_interrupt+0x128/0x2c0 [12552.095749] ? smp_apic_timer_interrupt+0xa6/0x270 [12552.095757] ? apic_timer_interrupt+0xf/0x20 [12552.095764] [12552.095767] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12552.095775] do_raw_spin_lock+0x177/0x190 [12552.095783] _raw_spin_lock+0x21/0x30 [12552.095787] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12552.096231] ? do_raw_spin_unlock+0x75/0x190 [12552.096237] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12552.096435] ? do_wait_intr+0xf0/0xf0 [12552.096443] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12552.096654] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12552.097202] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12552.097413] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12552.097537] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12552.097625] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12552.097655] mdc_intent_lock+0x353/0x7e0 [mdc] [12552.097683] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12552.097946] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12552.098424] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12552.098456] lmv_intent_open+0x338/0x1150 [lmv] [12552.098562] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12552.099302] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12552.099621] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12552.099647] ll_intent_lock+0xc5/0xaa0 [lustre] [12552.105374] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12552.105459] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12552.105467] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12552.105469] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12552.106271] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12552.106283] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12552.106288] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12552.106291] RDX: 0000000000000bd6 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12552.106293] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12552.106295] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12552.106298] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12552.106304] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12552.106310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12552.106313] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12552.106321] Call Trace: [12552.106327] [12552.106333] ? show_regs.cold.9+0x22/0x2f [12552.106343] ? watchdog_timer_fn.cold.6+0x62/0x10a [12552.106354] ? __hrtimer_run_queues+0x1c2/0x4d0 [12552.106362] ? watchdog+0x40/0x40 [12552.106366] ? hrtimer_interrupt+0x128/0x2c0 [12552.106371] ? smp_apic_timer_interrupt+0xa6/0x270 [12552.106383] ? apic_timer_interrupt+0xf/0x20 [12552.106388] [12552.106392] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12552.106397] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12552.106403] do_raw_spin_lock+0x177/0x190 [12552.106414] _raw_spin_lock+0x21/0x30 [12552.106421] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12552.106728] obd_mod_free+0x46/0x140 [ptlrpc] [12552.107043] mdc_commit_open+0x91/0xb0 [mdc] [12552.107080] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12552.107297] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12552.107510] after_reply+0xe2c/0x1780 [ptlrpc] [12552.107720] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12552.109560] ? __next_timer_interrupt+0x160/0x160 [12552.109571] ptlrpcd+0xa82/0xe40 [ptlrpc] [12552.111197] ? do_wait_intr+0xf0/0xf0 [12552.111210] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12552.111427] kthread+0x1d1/0x200 [12552.111434] ? set_kthread_struct+0x70/0x70 [12552.111438] ret_from_fork+0x1f/0x30 [12552.610568] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12552.617479] ll_lookup_it+0xe0e/0x2270 [lustre] [12552.624189] ? do_raw_spin_unlock+0x75/0x190 [12552.627518] ? _raw_spin_unlock+0x12/0x30 [12552.637836] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12552.641049] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12552.642871] ll_atomic_open+0x36a/0xd70 [lustre] [12552.649457] lookup_open+0x356/0xb40 [12552.651350] do_last+0x313/0xfc0 [12552.653017] ? nd_jump_root+0xe5/0x160 [12552.654467] ? path_init+0x437/0x520 [12552.657281] path_openat+0xf7/0x500 [12552.659113] ? do_raw_spin_unlock+0x75/0x190 [12552.661201] do_filp_open+0x99/0x140 [12552.668101] ? getname_flags+0x6e/0x330 [12552.676547] ? __check_object_size+0xff/0x256 [12552.681359] ? do_raw_spin_unlock+0x75/0x190 [12552.690925] ? _raw_spin_unlock+0x12/0x30 [12552.698101] do_sys_openat2+0x2b4/0x410 [12552.702419] do_sys_open+0x73/0xa0 [12552.709052] __x64_sys_openat+0x24/0x30 [12552.717446] do_syscall_64+0xc1/0x440 [12552.720981] entry_SYSCALL_64_after_hwframe+0x49/0xae [12552.726559] RIP: 0033:0x7f53425a5332 [12552.737487] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12552.764150] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12552.767962] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12552.773455] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12552.780598] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12552.787297] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12552.793105] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12568.052356] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [12568.057274] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12568.058925] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [12568.077778] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12568.077792] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12568.077799] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12568.078086] RIP: 0010:native_safe_halt+0x2e/0x50 [12568.078096] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12568.078100] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12568.078105] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12568.078107] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [12568.078110] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12568.078112] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12568.078114] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12568.078118] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12568.078121] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12568.078123] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12568.078131] Call Trace: [12568.078141] [12568.078146] ? show_regs.cold.9+0x22/0x2f [12568.078157] ? watchdog_timer_fn.cold.6+0x62/0x10a [12568.078163] ? __hrtimer_run_queues+0x1c2/0x4d0 [12568.078168] ? watchdog+0x40/0x40 [12568.078173] ? hrtimer_interrupt+0x128/0x2c0 [12568.078177] ? smp_apic_timer_interrupt+0xa6/0x270 [12568.078184] ? apic_timer_interrupt+0xf/0x20 [12568.078188] [12568.078191] ? native_safe_halt+0x2e/0x50 [12568.078195] kvm_wait+0xcd/0xe0 [12568.078202] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12568.078208] ? newidle_balance.isra.93+0x207/0x580 [12568.078214] do_raw_spin_lock+0x177/0x190 [12568.078219] _raw_spin_lock+0x21/0x30 [12568.078223] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12568.078516] process_one_work+0x2c8/0x700 [12568.078528] worker_thread+0x296/0x6e0 [12568.078533] ? rescuer_thread+0x570/0x570 [12568.078537] kthread+0x1d1/0x200 [12568.078541] ? set_kthread_struct+0x70/0x70 [12568.078545] ret_from_fork+0x1f/0x30 [12568.212278] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12568.230172] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12568.234618] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12568.237901] RIP: 0010:native_safe_halt+0x2e/0x50 [12568.239960] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12568.246934] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12568.249764] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12568.252406] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12568.255357] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12568.258099] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12568.260740] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12568.263536] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12568.266600] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12568.269806] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12568.274700] Call Trace: [12568.275888] [12568.278657] ? show_regs.cold.9+0x22/0x2f [12568.281240] ? watchdog_timer_fn.cold.6+0x62/0x10a [12568.282927] ? __hrtimer_run_queues+0x1c2/0x4d0 [12568.285997] ? watchdog+0x40/0x40 [12568.287026] ? hrtimer_interrupt+0x128/0x2c0 [12568.291497] ? smp_apic_timer_interrupt+0xa6/0x270 [12568.293251] ? apic_timer_interrupt+0xf/0x20 [12568.295332] [12568.297224] ? native_safe_halt+0x2e/0x50 [12568.299645] kvm_wait+0xcd/0xe0 [12568.300854] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12568.302519] do_raw_spin_lock+0x177/0x190 [12568.304761] _raw_spin_lock+0x21/0x30 [12568.307222] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12568.310655] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12568.314249] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12568.320015] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12568.327963] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12568.330626] ? woken_wake_function+0x30/0x30 [12568.332725] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12568.334942] kthread+0x1d1/0x200 [12568.337105] ? set_kthread_struct+0x70/0x70 [12568.342408] ret_from_fork+0x1f/0x30 [12573.158271] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9936s! [12573.165088] Showing busy workqueues and worker pools: [12573.170114] workqueue events: flags=0x0 [12573.171534] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12573.177489] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12573.185628] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12573.189436] pending: cpu_latency_work [ptlrpc] [12573.191833] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12573.194909] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12573.197891] workqueue events_long: flags=0x0 [12573.199491] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12573.201513] pending: lu_objects_flush [obdclass] [12573.204057] workqueue events_unbound: flags=0x2 [12573.205263] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12573.207841] pending: flush_memcg_stats_dwork, idle_cull_fn [12573.210420] workqueue events_power_efficient: flags=0x80 [12573.212673] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12573.215536] pending: check_lifetime, do_cache_clean [12573.217753] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12573.220429] pending: neigh_periodic_work, neigh_periodic_work [12573.222777] workqueue mm_percpu_wq: flags=0x8 [12573.224141] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12573.226229] pending: vmstat_update, lru_add_drain_per_cpu [12573.227838] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12573.229498] pending: vmstat_update, lru_add_drain_per_cpu [12573.230835] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12573.232808] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12573.235297] workqueue xprtiod: flags=0xa [12573.236917] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12573.239425] pending: xprt_autoclose, xs_error_handle [12573.241567] workqueue ipv6_addrconf: flags=0x40008 [12573.243445] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12573.246049] pending: addrconf_verify_work [12573.247828] workqueue knbd0-recv: flags=0x1a [12573.249680] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12573.251438] in-flight: 651:recv_work [12573.252445] workqueue ptlrpc_pinger: flags=0x2 [12573.253752] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12573.256104] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12573.258691] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9936s workers=4 idle: 15979 31824 8 [12573.261303] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12563s workers=2 idle: 654 [12580.057453] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [12580.058961] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [12580.061378] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12580.061455] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12580.061555] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12580.061558] RIP: 0010:native_safe_halt+0x2e/0x50 [12580.061571] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12580.061575] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12580.061580] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12580.061583] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12580.061585] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12580.061588] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12580.061590] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12580.061593] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12580.061596] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12580.061598] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12580.061607] Call Trace: [12580.061613] [12580.061619] ? show_regs.cold.9+0x22/0x2f [12580.061629] ? watchdog_timer_fn.cold.6+0x62/0x10a [12580.061636] ? __hrtimer_run_queues+0x1c2/0x4d0 [12580.061640] ? watchdog+0x40/0x40 [12580.061644] ? hrtimer_interrupt+0x128/0x2c0 [12580.061648] ? smp_apic_timer_interrupt+0xa6/0x270 [12580.061654] ? apic_timer_interrupt+0xf/0x20 [12580.061659] [12580.061662] ? native_safe_halt+0x2e/0x50 [12580.061666] kvm_wait+0xcd/0xe0 [12580.061673] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12580.061680] do_raw_spin_lock+0x177/0x190 [12580.061684] _raw_spin_lock+0x21/0x30 [12580.061690] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12580.061985] obd_mod_free+0x46/0x140 [ptlrpc] [12580.062181] mdc_commit_open+0x91/0xb0 [mdc] [12580.062229] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12580.062421] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12580.062622] after_reply+0xe2c/0x1780 [ptlrpc] [12580.062815] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12580.063001] ? __next_timer_interrupt+0x160/0x160 [12580.063011] ptlrpcd+0xa82/0xe40 [ptlrpc] [12580.063209] ? do_wait_intr+0xf0/0xf0 [12580.063232] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12580.063429] kthread+0x1d1/0x200 [12580.063434] ? set_kthread_struct+0x70/0x70 [12580.063438] ret_from_fork+0x1f/0x30 [12580.282446] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12580.318649] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12580.329156] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12580.339291] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12580.346020] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12580.366788] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12580.373242] RAX: 00000000000078eb RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12580.376513] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12580.380549] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12580.384150] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12580.387584] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12580.392074] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12580.406024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12580.412932] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12580.422617] Call Trace: [12580.423474] [12580.424196] ? show_regs.cold.9+0x22/0x2f [12580.432591] ? watchdog_timer_fn.cold.6+0x62/0x10a [12580.434586] ? __hrtimer_run_queues+0x1c2/0x4d0 [12580.442918] ? watchdog+0x40/0x40 [12580.445662] ? hrtimer_interrupt+0x128/0x2c0 [12580.447096] ? smp_apic_timer_interrupt+0xa6/0x270 [12580.454202] ? apic_timer_interrupt+0xf/0x20 [12580.458002] [12580.462146] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12580.466128] do_raw_spin_lock+0x177/0x190 [12580.468559] _raw_spin_lock+0x21/0x30 [12580.471042] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12580.475059] ? do_raw_spin_unlock+0x75/0x190 [12580.480178] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12580.483713] ? do_wait_intr+0xf0/0xf0 [12580.487157] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12580.492638] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12580.496065] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12580.507725] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12580.509339] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12580.515733] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12580.522645] mdc_intent_lock+0x353/0x7e0 [mdc] [12580.524279] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12580.531372] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12580.535579] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12580.537771] lmv_intent_open+0x338/0x1150 [lmv] [12580.540487] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12580.542450] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12580.549302] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12580.552301] ll_intent_lock+0xc5/0xaa0 [lustre] [12580.563097] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12580.569337] ll_lookup_it+0xe0e/0x2270 [lustre] [12580.575054] ? do_raw_spin_unlock+0x75/0x190 [12580.577554] ? _raw_spin_unlock+0x12/0x30 [12580.584967] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12580.586691] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12580.593117] ll_atomic_open+0x36a/0xd70 [lustre] [12580.607683] lookup_open+0x356/0xb40 [12580.608867] do_last+0x313/0xfc0 [12580.609857] ? nd_jump_root+0xe5/0x160 [12580.619393] ? path_init+0x437/0x520 [12580.620656] path_openat+0xf7/0x500 [12580.623745] ? do_raw_spin_unlock+0x75/0x190 [12580.627483] do_filp_open+0x99/0x140 [12580.630004] ? getname_flags+0x6e/0x330 [12580.632510] ? __check_object_size+0xff/0x256 [12580.635019] ? do_raw_spin_unlock+0x75/0x190 [12580.639177] ? _raw_spin_unlock+0x12/0x30 [12580.641398] do_sys_openat2+0x2b4/0x410 [12580.642789] do_sys_open+0x73/0xa0 [12580.645675] __x64_sys_openat+0x24/0x30 [12580.647451] do_syscall_64+0xc1/0x440 [12580.649124] entry_SYSCALL_64_after_hwframe+0x49/0xae [12580.652738] RIP: 0033:0x7f53425a5332 [12580.656584] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12580.668767] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12580.676898] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12580.680502] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12580.683669] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12580.688089] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12580.696039] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12594.680858] rcu: INFO: rcu_sched self-detected stall on CPU [12594.687429] rcu: 2-....: (5826296 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [12594.698932] (t=10500254 jiffies g=274185 q=66535) [12594.705726] rcu: rcu_sched kthread starved for 9958620 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [12594.713058] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12594.718303] rcu: RCU grace-period kthread stack dump: [12594.719701] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12594.725739] Call Trace: [12594.727666] __schedule+0x351/0xcb0 [12594.729529] schedule+0xc0/0x180 [12594.731947] schedule_timeout+0xb4/0x190 [12594.733679] ? do_raw_spin_unlock+0x75/0x190 [12594.735157] ? __next_timer_interrupt+0x160/0x160 [12594.741259] ? prepare_to_swait_event+0xf3/0x140 [12594.744538] rcu_gp_kthread+0x8dd/0x15b0 [12594.747896] ? force_qs_rnp+0x2c0/0x2c0 [12594.754455] kthread+0x1d1/0x200 [12594.755928] ? set_kthread_struct+0x70/0x70 [12594.763265] ret_from_fork+0x1f/0x30 [12594.764823] rcu: Stack dump where RCU GP kthread last ran: [12594.775030] Sending NMI from CPU 2 to CPUs 3: [12594.778563] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [12594.780163] Sending NMI from CPU 2 to CPUs 1: [12594.787699] NMI backtrace for cpu 1 [12594.787708] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12594.787710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12594.787712] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12594.787713] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12594.787715] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [12594.787718] RAX: 0000000000005c74 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12594.787719] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12594.787721] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12594.787722] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12594.787724] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12594.787725] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12594.787727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12594.787728] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12594.787730] Call Trace: [12594.787730] [12594.787731] ? show_regs.cold.9+0x22/0x2f [12594.787733] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [12594.787734] ? nmi_cpu_backtrace_handler+0x15/0x40 [12594.787735] ? nmi_handle+0xbd/0x210 [12594.787736] ? default_do_nmi+0x4b/0x1b0 [12594.787737] ? do_nmi+0x280/0x380 [12594.787738] ? end_repeat_nmi+0x16/0x1f [12594.787739] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12594.787741] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12594.787742] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12594.787743] [12594.787743] do_raw_spin_lock+0x177/0x190 [12594.787744] _raw_spin_lock+0x21/0x30 [12594.787745] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12594.787746] ? do_raw_spin_unlock+0x75/0x190 [12594.787747] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12594.787748] ? do_wait_intr+0xf0/0xf0 [12594.787749] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12594.787750] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12594.787751] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12594.787752] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12594.787753] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12594.787754] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12594.787755] mdc_intent_lock+0x353/0x7e0 [mdc] [12594.787757] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12594.787758] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12594.787759] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12594.787760] lmv_intent_open+0x338/0x1150 [lmv] [12594.787761] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12594.787763] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12594.787764] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12594.787766] ll_intent_lock+0xc5/0xaa0 [lustre] [12594.787767] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12594.787768] ll_lookup_it+0xe0e/0x2270 [lustre] [12594.787769] ? do_raw_spin_unlock+0x75/0x190 [12594.787770] ? _raw_spin_unlock+0x12/0x30 [12594.787771] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12594.787772] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12594.787773] ll_atomic_open+0x36a/0xd70 [lustre] [12594.787774] lookup_open+0x356/0xb40 [12594.787775] do_last+0x313/0xfc0 [12594.787776] ? nd_jump_root+0xe5/0x160 [12594.787777] ? path_init+0x437/0x520 [12594.787778] path_openat+0xf7/0x500 [12594.787779] ? do_raw_spin_unlock+0x75/0x190 [12594.787780] do_filp_open+0x99/0x140 [12594.787781] ? getname_flags+0x6e/0x330 [12594.787782] ? __check_object_size+0xff/0x256 [12594.787783] ? do_raw_spin_unlock+0x75/0x190 [12594.787785] ? _raw_spin_unlock+0x12/0x30 [12594.787786] do_sys_openat2+0x2b4/0x410 [12594.787787] do_sys_open+0x73/0xa0 [12594.787788] __x64_sys_openat+0x24/0x30 [12594.787789] do_syscall_64+0xc1/0x440 [12594.787790] entry_SYSCALL_64_after_hwframe+0x49/0xae [12594.787792] RIP: 0033:0x7f53425a5332 [12594.787793] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12594.787796] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12594.787798] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12594.787800] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12594.787801] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12594.787803] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12594.787805] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12594.792916] NMI backtrace for cpu 2 [12595.011742] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12595.016654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12595.019880] Call Trace: [12595.020865] [12595.021689] ? dump_stack+0xbb/0x10e [12595.023355] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12595.025362] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12595.027279] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12595.029182] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12595.031125] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12595.033250] ? rcu_dump_cpu_stacks+0x11b/0x171 [12595.034871] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12595.036675] ? update_process_times+0x6d/0xc0 [12595.038268] ? tick_sched_handle.isra.18+0x27/0x90 [12595.040603] ? tick_sched_timer+0x53/0xb0 [12595.042189] ? __hrtimer_run_queues+0x1c2/0x4d0 [12595.044226] ? tick_sched_handle.isra.18+0x90/0x90 [12595.046179] ? hrtimer_interrupt+0x128/0x2c0 [12595.048020] ? smp_apic_timer_interrupt+0xa6/0x270 [12595.049894] ? apic_timer_interrupt+0xf/0x20 [12595.052098] [12595.052786] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12595.054558] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12595.056506] ? do_raw_spin_lock+0x177/0x190 [12595.057607] ? _raw_spin_lock+0x21/0x30 [12595.058663] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12595.060509] ? obd_mod_free+0x46/0x140 [ptlrpc] [12595.062322] ? mdc_commit_open+0x91/0xb0 [mdc] [12595.063652] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12595.065483] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12595.067404] ? after_reply+0xe2c/0x1780 [ptlrpc] [12595.069219] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12595.071198] ? __next_timer_interrupt+0x160/0x160 [12595.072919] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12595.074595] ? do_wait_intr+0xf0/0xf0 [12595.075698] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12595.077577] ? kthread+0x1d1/0x200 [12595.078578] ? set_kthread_struct+0x70/0x70 [12595.079775] ? ret_from_fork+0x1f/0x30 [12600.052016] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [12600.056729] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12600.110842] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12600.117689] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12600.126950] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12600.132743] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12600.135659] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12600.153512] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12600.168541] RAX: 0000000000006ebe RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12600.170762] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12600.174672] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12600.178206] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12600.186576] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12600.194834] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12600.204069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12600.207897] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12600.211712] Call Trace: [12600.213565] [12600.214626] ? show_regs.cold.9+0x22/0x2f [12600.220534] ? watchdog_timer_fn.cold.6+0x62/0x10a [12600.224304] ? __hrtimer_run_queues+0x1c2/0x4d0 [12600.225756] ? watchdog+0x40/0x40 [12600.236271] ? hrtimer_interrupt+0x128/0x2c0 [12600.243488] ? smp_apic_timer_interrupt+0xa6/0x270 [12600.249219] ? apic_timer_interrupt+0xf/0x20 [12600.253158] [12600.255705] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12600.258670] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12600.263171] ? newidle_balance.isra.93+0x207/0x580 [12600.269725] do_raw_spin_lock+0x177/0x190 [12600.271979] _raw_spin_lock+0x21/0x30 [12600.274059] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12600.279762] process_one_work+0x2c8/0x700 [12600.283032] worker_thread+0x296/0x6e0 [12600.284347] ? rescuer_thread+0x570/0x570 [12600.287262] kthread+0x1d1/0x200 [12600.288895] ? set_kthread_struct+0x70/0x70 [12600.302519] ret_from_fork+0x1f/0x30 [12603.359874] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9966s! [12603.365121] Showing busy workqueues and worker pools: [12603.367354] workqueue events: flags=0x0 [12603.369725] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12603.378059] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12603.385434] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12603.392084] pending: cpu_latency_work [ptlrpc] [12603.400142] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12603.409984] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12603.420283] workqueue events_long: flags=0x0 [12603.422114] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12603.426395] pending: lu_objects_flush [obdclass] [12603.428550] workqueue events_unbound: flags=0x2 [12603.429694] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12603.432718] pending: flush_memcg_stats_dwork, idle_cull_fn [12603.436697] workqueue events_power_efficient: flags=0x80 [12603.446894] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12603.453722] pending: check_lifetime, do_cache_clean [12603.458765] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12603.464784] pending: neigh_periodic_work, neigh_periodic_work [12603.473626] workqueue mm_percpu_wq: flags=0x8 [12603.476378] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12603.480768] pending: vmstat_update, lru_add_drain_per_cpu [12603.484100] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12603.488010] pending: vmstat_update, lru_add_drain_per_cpu [12603.491226] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12603.494639] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12603.498201] workqueue xprtiod: flags=0xa [12603.500739] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12603.510881] pending: xprt_autoclose, xs_error_handle [12603.516993] workqueue ipv6_addrconf: flags=0x40008 [12603.520244] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12603.526392] pending: addrconf_verify_work [12603.532455] workqueue knbd0-recv: flags=0x1a [12603.535273] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12603.539998] in-flight: 651:recv_work [12603.544293] workqueue ptlrpc_pinger: flags=0x2 [12603.546260] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12603.549195] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12603.552039] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9966s workers=4 idle: 15979 31824 8 [12603.554713] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12594s workers=2 idle: 654 [12620.058462] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [12620.060404] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [12620.061893] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [12620.061903] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12620.061997] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12620.062002] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12620.062005] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12620.062019] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12620.062023] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12620.062028] RAX: 000000000000250e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12620.062031] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12620.062034] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12620.062037] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12620.062040] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12620.062044] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12620.062047] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12620.062050] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12620.062058] Call Trace: [12620.062064] [12620.062069] ? show_regs.cold.9+0x22/0x2f [12620.062079] ? watchdog_timer_fn.cold.6+0x62/0x10a [12620.062086] ? __hrtimer_run_queues+0x1c2/0x4d0 [12620.062090] ? watchdog+0x40/0x40 [12620.062095] ? hrtimer_interrupt+0x128/0x2c0 [12620.062100] ? smp_apic_timer_interrupt+0xa6/0x270 [12620.062118] ? apic_timer_interrupt+0xf/0x20 [12620.062124] [12620.062127] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12620.062134] do_raw_spin_lock+0x177/0x190 [12620.062140] _raw_spin_lock+0x21/0x30 [12620.062145] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12620.062415] ? do_raw_spin_unlock+0x75/0x190 [12620.062421] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12620.062633] ? do_wait_intr+0xf0/0xf0 [12620.062642] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12620.062870] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12620.063095] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12620.063162] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12620.063168] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12620.063169] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12620.063177] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12620.063181] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [12620.063185] RAX: 0000000000007f85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12620.063187] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12620.063189] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12620.063192] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12620.063194] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12620.063197] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12620.063200] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12620.063202] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12620.063208] Call Trace: [12620.063211] [12620.063214] ? show_regs.cold.9+0x22/0x2f [12620.063220] ? watchdog_timer_fn.cold.6+0x62/0x10a [12620.063225] ? __hrtimer_run_queues+0x1c2/0x4d0 [12620.063133] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12620.063361] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12620.063473] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12620.063533] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12620.063567] mdc_intent_lock+0x353/0x7e0 [mdc] [12620.063605] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12620.063689] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12620.063904] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12620.063933] lmv_intent_open+0x338/0x1150 [lmv] [12620.063963] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12620.064062] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12620.064163] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12620.064188] ll_intent_lock+0xc5/0xaa0 [lustre] [12620.064280] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12620.064371] ll_lookup_it+0xe0e/0x2270 [lustre] [12620.064455] ? do_raw_spin_unlock+0x75/0x190 [12620.064461] ? _raw_spin_unlock+0x12/0x30 [12620.064467] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12620.064490] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12620.064508] ll_atomic_open+0x36a/0xd70 [lustre] [12620.064601] lookup_open+0x356/0xb40 [12620.064611] do_last+0x313/0xfc0 [12620.064616] ? nd_jump_root+0xe5/0x160 [12620.064621] ? path_init+0x437/0x520 [12620.064626] path_openat+0xf7/0x500 [12620.064631] ? do_raw_spin_unlock+0x75/0x190 [12620.064636] do_filp_open+0x99/0x140 [12620.064643] ? getname_flags+0x6e/0x330 [12620.064647] ? __check_object_size+0xff/0x256 [12620.064654] ? do_raw_spin_unlock+0x75/0x190 [12620.064659] ? _raw_spin_unlock+0x12/0x30 [12620.064664] do_sys_openat2+0x2b4/0x410 [12620.064669] do_sys_open+0x73/0xa0 [12620.064673] __x64_sys_openat+0x24/0x30 [12620.064677] do_syscall_64+0xc1/0x440 [12620.064684] entry_SYSCALL_64_after_hwframe+0x49/0xae [12620.064690] RIP: 0033:0x7f53425a5332 [12620.064697] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12620.064701] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12620.064706] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12620.064709] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12620.064711] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12620.064714] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12620.064717] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12620.064737] ? watchdog+0x40/0x40 [12620.064745] ? hrtimer_interrupt+0x128/0x2c0 [12620.064750] ? smp_apic_timer_interrupt+0xa6/0x270 [12620.064755] ? apic_timer_interrupt+0xf/0x20 [12620.064759] [12620.064763] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12620.064768] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12620.064773] do_raw_spin_lock+0x177/0x190 [12620.064778] _raw_spin_lock+0x21/0x30 [12620.064783] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12620.065011] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12620.065222] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12620.071096] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12620.071174] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12620.071183] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12620.071185] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12620.071201] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12620.071204] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12620.071209] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12620.071212] RDX: 0000000000007f95 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12620.071214] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12620.071216] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12620.071218] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12620.071221] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12620.071224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12620.071226] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12620.071235] Call Trace: [12620.071241] [12620.071247] ? show_regs.cold.9+0x22/0x2f [12620.071256] ? watchdog_timer_fn.cold.6+0x62/0x10a [12620.071263] ? __hrtimer_run_queues+0x1c2/0x4d0 [12620.071267] ? watchdog+0x40/0x40 [12620.071271] ? hrtimer_interrupt+0x128/0x2c0 [12620.071275] ? smp_apic_timer_interrupt+0xa6/0x270 [12620.071283] ? apic_timer_interrupt+0xf/0x20 [12620.071287] [12620.071290] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12620.071294] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12620.071299] do_raw_spin_lock+0x177/0x190 [12620.071303] _raw_spin_lock+0x21/0x30 [12620.071309] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12620.071617] obd_mod_free+0x46/0x140 [ptlrpc] [12620.071813] mdc_commit_open+0x91/0xb0 [mdc] [12620.071855] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12620.072047] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12620.072251] after_reply+0xe2c/0x1780 [ptlrpc] [12620.072458] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12620.090891] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12620.139994] ? __next_timer_interrupt+0x160/0x160 [12620.140018] ptlrpcd+0xa82/0xe40 [ptlrpc] [12620.145033] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12620.145315] ? woken_wake_function+0x30/0x30 [12620.145326] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12620.145784] kthread+0x1d1/0x200 [12620.145792] ? set_kthread_struct+0x70/0x70 [12620.145796] ret_from_fork+0x1f/0x30 [12621.065148] ? do_wait_intr+0xf0/0xf0 [12621.067719] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12621.080067] kthread+0x1d1/0x200 [12621.088491] ? set_kthread_struct+0x70/0x70 [12621.095713] ret_from_fork+0x1f/0x30 [12628.052373] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [12628.064126] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12628.096715] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12628.102960] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12628.107246] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12628.110849] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12628.114141] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12628.132837] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12628.141934] RAX: 0000000000007d9b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12628.144966] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12628.158243] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12628.161200] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12628.171310] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12628.174979] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12628.180203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12628.181939] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12628.188960] Call Trace: [12628.193231] [12628.196733] ? show_regs.cold.9+0x22/0x2f [12628.198054] ? watchdog_timer_fn.cold.6+0x62/0x10a [12628.203858] ? __hrtimer_run_queues+0x1c2/0x4d0 [12628.206106] ? watchdog+0x40/0x40 [12628.207127] ? hrtimer_interrupt+0x128/0x2c0 [12628.209147] ? smp_apic_timer_interrupt+0xa6/0x270 [12628.211758] ? apic_timer_interrupt+0xf/0x20 [12628.215046] [12628.215750] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12628.221783] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12628.225046] ? newidle_balance.isra.93+0x207/0x580 [12628.230989] do_raw_spin_lock+0x177/0x190 [12628.233102] _raw_spin_lock+0x21/0x30 [12628.237457] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12628.242964] process_one_work+0x2c8/0x700 [12628.245288] worker_thread+0x296/0x6e0 [12628.246912] ? rescuer_thread+0x570/0x570 [12628.251862] kthread+0x1d1/0x200 [12628.253966] ? set_kthread_struct+0x70/0x70 [12628.257140] ret_from_fork+0x1f/0x30 [12633.572851] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 9996s! [12633.580707] Showing busy workqueues and worker pools: [12633.585750] workqueue events: flags=0x0 [12633.590949] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12633.599811] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12633.610007] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12633.614293] pending: cpu_latency_work [ptlrpc] [12633.620676] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12633.628717] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12633.633730] workqueue events_long: flags=0x0 [12633.636136] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12633.639687] pending: lu_objects_flush [obdclass] [12633.642333] workqueue events_unbound: flags=0x2 [12633.647814] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12633.654653] pending: flush_memcg_stats_dwork, idle_cull_fn [12633.659649] workqueue events_power_efficient: flags=0x80 [12633.664706] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12633.669843] pending: check_lifetime, do_cache_clean [12633.672745] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12633.676172] pending: neigh_periodic_work, neigh_periodic_work [12633.679851] workqueue mm_percpu_wq: flags=0x8 [12633.685533] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12633.692062] pending: vmstat_update, lru_add_drain_per_cpu [12633.698896] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12633.701169] pending: vmstat_update, lru_add_drain_per_cpu [12633.712026] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12633.719319] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12633.725769] workqueue xprtiod: flags=0xa [12633.729053] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12633.736101] pending: xprt_autoclose, xs_error_handle [12633.742092] workqueue ipv6_addrconf: flags=0x40008 [12633.746723] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12633.753988] pending: addrconf_verify_work [12633.758533] workqueue knbd0-recv: flags=0x1a [12633.763742] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12633.770477] in-flight: 651:recv_work [12633.772567] workqueue ptlrpc_pinger: flags=0x2 [12633.780072] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12633.784674] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12633.789619] pool 8: cpus=0-3 flags=0x4 nice=0 hung=9996s workers=4 idle: 15979 31824 8 [12633.798317] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12624s workers=2 idle: 654 [12648.055054] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [12648.057130] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [12648.058071] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [12648.058080] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12648.058158] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12648.058165] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12648.058168] RIP: 0010:native_safe_halt+0x2e/0x50 [12648.058181] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12648.058186] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12648.058190] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12648.058192] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12648.058194] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12648.058196] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12648.058199] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12648.058203] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12648.058206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12648.058208] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12648.058217] Call Trace: [12648.058223] [12648.058229] ? show_regs.cold.9+0x22/0x2f [12648.058240] ? watchdog_timer_fn.cold.6+0x62/0x10a [12648.058247] ? __hrtimer_run_queues+0x1c2/0x4d0 [12648.058251] ? watchdog+0x40/0x40 [12648.058255] ? hrtimer_interrupt+0x128/0x2c0 [12648.058259] ? smp_apic_timer_interrupt+0xa6/0x270 [12648.058264] ? apic_timer_interrupt+0xf/0x20 [12648.058270] [12648.058273] ? native_safe_halt+0x2e/0x50 [12648.058277] kvm_wait+0xcd/0xe0 [12648.058285] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12648.058294] do_raw_spin_lock+0x177/0x190 [12648.058299] _raw_spin_lock+0x21/0x30 [12648.058304] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12648.058574] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12648.058785] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12648.059018] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12648.059256] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12648.059485] ? woken_wake_function+0x30/0x30 [12648.059495] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12648.059727] kthread+0x1d1/0x200 [12648.059733] ? set_kthread_struct+0x70/0x70 [12648.059738] ret_from_fork+0x1f/0x30 [12648.062181] Modules linked in: lustre(O) [12648.065655] Modules linked in: [12648.069278] osp(O) ofd(O) [12648.089893] lustre(O) [12648.108946] lod(O) [12648.113876] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12648.113944] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12648.113951] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12648.113954] RIP: 0010:native_safe_halt+0x2e/0x50 [12648.113967] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12648.113972] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [12648.113977] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12648.113979] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12648.113982] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12648.113984] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12648.113986] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12648.113989] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12648.113993] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12648.113995] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12648.114004] Call Trace: [12648.114011] [12648.114017] ? show_regs.cold.9+0x22/0x2f [12648.114088] ? watchdog_timer_fn.cold.6+0x62/0x10a [12648.114095] ? __hrtimer_run_queues+0x1c2/0x4d0 [12648.114099] ? watchdog+0x40/0x40 [12648.114103] ? hrtimer_interrupt+0x128/0x2c0 [12648.114107] ? smp_apic_timer_interrupt+0xa6/0x270 [12648.114114] ? apic_timer_interrupt+0xf/0x20 [12648.114119] [12648.114122] ? native_safe_halt+0x2e/0x50 [12648.114126] kvm_wait+0xcd/0xe0 [12648.114134] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12648.114141] do_raw_spin_lock+0x177/0x190 [12648.114145] _raw_spin_lock+0x21/0x30 [12648.114151] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12648.114550] obd_mod_free+0x46/0x140 [ptlrpc] [12648.114857] mdc_commit_open+0x91/0xb0 [mdc] [12648.114894] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12648.115167] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12648.115620] after_reply+0xe2c/0x1780 [ptlrpc] [12648.115825] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12648.116035] ? __next_timer_interrupt+0x160/0x160 [12648.116047] ptlrpcd+0xa82/0xe40 [ptlrpc] [12648.116256] ? do_wait_intr+0xf0/0xf0 [12648.116265] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12648.116646] kthread+0x1d1/0x200 [12648.116653] ? set_kthread_struct+0x70/0x70 [12648.116657] ret_from_fork+0x1f/0x30 [12648.535900] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12648.563751] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12648.570340] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12648.574236] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12648.576610] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12648.585427] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12648.589526] RAX: 00000000000065fb RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12648.593441] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12648.597872] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12648.602757] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12648.606578] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12648.609375] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12648.612370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12648.614695] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12648.617963] Call Trace: [12648.618844] [12648.620171] ? show_regs.cold.9+0x22/0x2f [12648.622050] ? watchdog_timer_fn.cold.6+0x62/0x10a [12648.623990] ? __hrtimer_run_queues+0x1c2/0x4d0 [12648.626998] ? watchdog+0x40/0x40 [12648.629343] ? hrtimer_interrupt+0x128/0x2c0 [12648.630879] ? smp_apic_timer_interrupt+0xa6/0x270 [12648.634086] ? apic_timer_interrupt+0xf/0x20 [12648.636384] [12648.638345] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12648.639974] do_raw_spin_lock+0x177/0x190 [12648.642603] _raw_spin_lock+0x21/0x30 [12648.645416] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12648.649571] ? do_raw_spin_unlock+0x75/0x190 [12648.652003] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12648.653981] ? do_wait_intr+0xf0/0xf0 [12648.656845] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12648.658996] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12648.662875] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12648.666409] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12648.670266] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12648.672989] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12648.674857] mdc_intent_lock+0x353/0x7e0 [mdc] [12648.676782] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12648.679193] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12648.681978] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12648.684898] lmv_intent_open+0x338/0x1150 [lmv] [12648.686917] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12648.689495] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12648.691851] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12648.693457] ll_intent_lock+0xc5/0xaa0 [lustre] [12648.695008] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12648.697203] ll_lookup_it+0xe0e/0x2270 [lustre] [12648.699475] ? do_raw_spin_unlock+0x75/0x190 [12648.701409] ? _raw_spin_unlock+0x12/0x30 [12648.703270] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12648.706669] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12648.710858] ll_atomic_open+0x36a/0xd70 [lustre] [12648.713546] lookup_open+0x356/0xb40 [12648.715679] do_last+0x313/0xfc0 [12648.717472] ? nd_jump_root+0xe5/0x160 [12648.719770] ? path_init+0x437/0x520 [12648.721550] path_openat+0xf7/0x500 [12648.723557] ? do_raw_spin_unlock+0x75/0x190 [12648.725930] do_filp_open+0x99/0x140 [12648.728418] ? getname_flags+0x6e/0x330 [12648.730212] ? __check_object_size+0xff/0x256 [12648.731749] ? do_raw_spin_unlock+0x75/0x190 [12648.733456] ? _raw_spin_unlock+0x12/0x30 [12648.736261] do_sys_openat2+0x2b4/0x410 [12648.738428] do_sys_open+0x73/0xa0 [12648.740494] __x64_sys_openat+0x24/0x30 [12648.742730] do_syscall_64+0xc1/0x440 [12648.745094] entry_SYSCALL_64_after_hwframe+0x49/0xae [12648.749278] RIP: 0033:0x7f53425a5332 [12648.751265] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12648.761298] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12648.765376] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12648.768659] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12648.772209] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12648.775962] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12648.779272] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12660.052181] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [12660.058120] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12660.080074] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12660.086638] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12660.091552] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12660.096147] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12660.099111] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12660.112682] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [12660.116384] RAX: 0000000000007f05 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12660.120480] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12660.127020] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12660.131690] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12660.141354] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12660.154705] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12660.165119] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12660.173504] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12660.180542] Call Trace: [12660.181369] [12660.182073] ? show_regs.cold.9+0x22/0x2f [12660.185149] ? watchdog_timer_fn.cold.6+0x62/0x10a [12660.187863] ? __hrtimer_run_queues+0x1c2/0x4d0 [12660.189743] ? watchdog+0x40/0x40 [12660.192371] ? hrtimer_interrupt+0x128/0x2c0 [12660.202967] ? smp_apic_timer_interrupt+0xa6/0x270 [12660.211266] ? apic_timer_interrupt+0xf/0x20 [12660.223512] [12660.225604] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12660.232200] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12660.236606] ? newidle_balance.isra.93+0x207/0x580 [12660.242880] do_raw_spin_lock+0x177/0x190 [12660.247102] _raw_spin_lock+0x21/0x30 [12660.251070] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12660.257595] process_one_work+0x2c8/0x700 [12660.263764] worker_thread+0x296/0x6e0 [12660.267298] ? rescuer_thread+0x570/0x570 [12660.270029] kthread+0x1d1/0x200 [12660.271388] ? set_kthread_struct+0x70/0x70 [12660.273877] ret_from_fork+0x1f/0x30 [12664.290000] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10027s! [12664.296685] Showing busy workqueues and worker pools: [12664.299663] workqueue events: flags=0x0 [12664.302530] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12664.309379] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12664.319563] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12664.327508] pending: cpu_latency_work [ptlrpc] [12664.330350] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12664.336372] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12664.340417] workqueue events_long: flags=0x0 [12664.342855] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12664.346070] pending: lu_objects_flush [obdclass] [12664.348320] workqueue events_unbound: flags=0x2 [12664.350039] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12664.352612] pending: flush_memcg_stats_dwork, idle_cull_fn [12664.355022] workqueue events_power_efficient: flags=0x80 [12664.357196] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12664.362798] pending: check_lifetime, do_cache_clean [12664.368729] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12664.378080] pending: neigh_periodic_work, neigh_periodic_work [12664.385007] workqueue mm_percpu_wq: flags=0x8 [12664.388223] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12664.395940] pending: vmstat_update, lru_add_drain_per_cpu [12664.398401] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12664.406350] pending: vmstat_update, lru_add_drain_per_cpu [12664.409331] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12664.414192] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12664.417109] workqueue xprtiod: flags=0xa [12664.422383] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12664.430020] pending: xprt_autoclose, xs_error_handle [12664.438988] workqueue ipv6_addrconf: flags=0x40008 [12664.441231] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12664.444507] pending: addrconf_verify_work [12664.448584] workqueue knbd0-recv: flags=0x1a [12664.450532] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12664.453627] in-flight: 651:recv_work [12664.456334] workqueue ptlrpc_pinger: flags=0x2 [12664.458270] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12664.464894] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12664.471234] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10027s workers=4 idle: 15979 31824 8 [12664.474880] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12654s workers=2 idle: 654 [12676.055578] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [12676.058380] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [12676.058868] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [12676.058875] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12676.058951] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12676.058962] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12676.058965] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12676.058982] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12676.058987] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12676.058991] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12676.058994] RDX: 0000000000007b71 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12676.058996] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12676.058998] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12676.059000] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12676.059004] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12676.059007] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12676.059014] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12676.059021] Call Trace: [12676.059027] [12676.059033] ? show_regs.cold.9+0x22/0x2f [12676.059699] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [12676.059707] ? watchdog_timer_fn.cold.6+0x62/0x10a [12676.059715] mdt(O) mdd(O) mgs(O) [12676.059722] ? __hrtimer_run_queues+0x1c2/0x4d0 [12676.059726] lquota(O) lfsck(O) [12676.059732] ? watchdog+0x40/0x40 [12676.059734] obdecho(O) mgc(O) mdc(O) [12676.059742] ? hrtimer_interrupt+0x128/0x2c0 [12676.059745] lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) [12676.059753] ? smp_apic_timer_interrupt+0xa6/0x270 [12676.059760] ptlrpc_gss(O) ptlrpc(O) obdclass(O) [12676.059767] ? apic_timer_interrupt+0xf/0x20 [12676.059773] ksocklnd(O) [12676.059778] [12676.059778] lnet(O) libcfs(O) rpcsec_gss_krb5 [12676.059785] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12676.059790] auth_rpcgss nfsv4 dns_resolver [12676.059798] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12676.059802] intel_rapl_msr intel_rapl_common sb_edac [12676.059809] do_raw_spin_lock+0x177/0x190 [12676.059813] rapl i2c_piix4 pcspkr squashfs [12676.059823] _raw_spin_lock+0x21/0x30 [12676.059825] ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12676.059853] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12676.059858] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12676.059860] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12676.059870] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12676.059874] RSP: 0018:ffffbb5381503500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [12676.059878] RAX: 0000000000007619 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12676.059881] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12676.059883] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12676.059885] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12676.059888] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12676.059890] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12676.059893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12676.059895] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12676.059902] Call Trace: [12676.059906] [12676.059909] ? show_regs.cold.9+0x22/0x2f [12676.059916] ? watchdog_timer_fn.cold.6+0x62/0x10a [12676.059921] ? __hrtimer_run_queues+0x1c2/0x4d0 [12676.059924] ? watchdog+0x40/0x40 [12676.059928] ? hrtimer_interrupt+0x128/0x2c0 [12676.059932] ? smp_apic_timer_interrupt+0xa6/0x270 [12676.059938] ? apic_timer_interrupt+0xf/0x20 [12676.059943] [12676.059946] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12676.059952] do_raw_spin_lock+0x177/0x190 [12676.059957] _raw_spin_lock+0x21/0x30 [12676.059836] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12676.059962] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12676.060502] ? do_raw_spin_unlock+0x75/0x190 [12676.062269] obd_mod_free+0x46/0x140 [ptlrpc] [12676.060508] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12676.062790] ? do_wait_intr+0xf0/0xf0 [12676.063289] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12676.063515] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12676.063753] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12676.064607] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12676.064718] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12676.064763] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12676.064795] mdc_intent_lock+0x353/0x7e0 [mdc] [12676.064906] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12676.064990] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12676.065211] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12676.065256] lmv_intent_open+0x338/0x1150 [lmv] [12676.065286] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12676.065370] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12676.065457] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12676.065482] ll_intent_lock+0xc5/0xaa0 [lustre] [12676.065582] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12676.065674] ll_lookup_it+0xe0e/0x2270 [lustre] [12676.065761] ? do_raw_spin_unlock+0x75/0x190 [12676.065770] ? _raw_spin_unlock+0x12/0x30 [12676.065776] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12676.066908] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12676.066981] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12676.066989] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12676.066992] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12676.067004] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12676.067008] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12676.067012] RAX: 000000000000707a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12676.067014] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12676.067016] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12676.067018] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12676.067021] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12676.067024] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12676.067027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12676.067030] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12676.067038] Call Trace: [12676.067042] [12676.067047] ? show_regs.cold.9+0x22/0x2f [12676.067057] ? watchdog_timer_fn.cold.6+0x62/0x10a [12676.067064] ? __hrtimer_run_queues+0x1c2/0x4d0 [12676.067068] ? watchdog+0x40/0x40 [12676.067073] ? hrtimer_interrupt+0x128/0x2c0 [12676.067077] ? smp_apic_timer_interrupt+0xa6/0x270 [12676.067083] ? apic_timer_interrupt+0xf/0x20 [12676.067087] [12676.067090] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12676.067094] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12676.067100] do_raw_spin_lock+0x177/0x190 [12676.067105] _raw_spin_lock+0x21/0x30 [12676.067109] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12676.068400] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12676.068434] ll_atomic_open+0x36a/0xd70 [lustre] [12676.068552] lookup_open+0x356/0xb40 [12676.068561] do_last+0x313/0xfc0 [12676.068566] ? nd_jump_root+0xe5/0x160 [12676.068570] ? path_init+0x437/0x520 [12676.068575] path_openat+0xf7/0x500 [12676.068580] ? do_raw_spin_unlock+0x75/0x190 [12676.068586] do_filp_open+0x99/0x140 [12676.068592] ? getname_flags+0x6e/0x330 [12676.068595] ? __check_object_size+0xff/0x256 [12676.068601] ? do_raw_spin_unlock+0x75/0x190 [12676.068604] ? _raw_spin_unlock+0x12/0x30 [12676.068607] do_sys_openat2+0x2b4/0x410 [12676.068610] do_sys_open+0x73/0xa0 [12676.068613] __x64_sys_openat+0x24/0x30 [12676.068615] do_syscall_64+0xc1/0x440 [12676.068621] entry_SYSCALL_64_after_hwframe+0x49/0xae [12676.068625] RIP: 0033:0x7f53425a5332 [12676.068631] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12676.068634] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12676.068637] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12676.068639] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12676.068641] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12676.068642] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12676.068644] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12676.072265] mdc_commit_open+0x91/0xb0 [mdc] [12676.072316] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12676.103138] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12676.103439] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12676.111148] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12676.111456] after_reply+0xe2c/0x1780 [ptlrpc] [12676.111721] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12676.111976] ? __next_timer_interrupt+0x160/0x160 [12676.111993] ptlrpcd+0xa82/0xe40 [ptlrpc] [12676.112222] ? do_wait_intr+0xf0/0xf0 [12676.112237] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12676.112481] kthread+0x1d1/0x200 [12676.112487] ? set_kthread_struct+0x70/0x70 [12676.112495] ret_from_fork+0x1f/0x30 [12676.537584] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12676.539367] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12676.553125] ? woken_wake_function+0x30/0x30 [12676.556300] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12676.559276] kthread+0x1d1/0x200 [12676.567945] ? set_kthread_struct+0x70/0x70 [12676.571187] ret_from_fork+0x1f/0x30 [12692.062957] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [12692.072070] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12692.119821] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12692.134030] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12692.142865] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12692.151157] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12692.155406] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12692.173578] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12692.184052] RAX: 000000000000638f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12692.192975] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12692.195982] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12692.200924] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12692.208891] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12692.213977] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12692.220581] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12692.226295] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12692.228763] Call Trace: [12692.239056] [12692.240974] ? show_regs.cold.9+0x22/0x2f [12692.245081] ? watchdog_timer_fn.cold.6+0x62/0x10a [12692.252419] ? __hrtimer_run_queues+0x1c2/0x4d0 [12692.254469] ? watchdog+0x40/0x40 [12692.257398] ? hrtimer_interrupt+0x128/0x2c0 [12692.260638] ? smp_apic_timer_interrupt+0xa6/0x270 [12692.262782] ? apic_timer_interrupt+0xf/0x20 [12692.265654] [12692.267540] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12692.273114] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12692.283723] ? newidle_balance.isra.93+0x207/0x580 [12692.290080] do_raw_spin_lock+0x177/0x190 [12692.298952] _raw_spin_lock+0x21/0x30 [12692.309902] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12692.321473] process_one_work+0x2c8/0x700 [12692.327905] worker_thread+0x296/0x6e0 [12692.329188] ? rescuer_thread+0x570/0x570 [12692.334669] kthread+0x1d1/0x200 [12692.335771] ? set_kthread_struct+0x70/0x70 [12692.337079] ret_from_fork+0x1f/0x30 [12694.515826] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10057s! [12694.534974] Showing busy workqueues and worker pools: [12694.538904] workqueue events: flags=0x0 [12694.542540] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12694.552677] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12694.567953] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12694.577040] pending: cpu_latency_work [ptlrpc] [12694.589092] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12694.600842] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12694.615231] workqueue events_long: flags=0x0 [12694.624065] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12694.630793] pending: lu_objects_flush [obdclass] [12694.645028] workqueue events_unbound: flags=0x2 [12694.650873] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12694.660656] pending: flush_memcg_stats_dwork, idle_cull_fn [12694.668975] workqueue events_power_efficient: flags=0x80 [12694.675030] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12694.686533] pending: check_lifetime, do_cache_clean [12694.692120] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12694.699150] pending: neigh_periodic_work, neigh_periodic_work [12694.710645] workqueue mm_percpu_wq: flags=0x8 [12694.719026] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12694.733496] pending: vmstat_update, lru_add_drain_per_cpu [12694.739037] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12694.760901] pending: vmstat_update, lru_add_drain_per_cpu [12694.768872] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12694.779736] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12694.790587] workqueue xprtiod: flags=0xa [12694.806082] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12694.829527] pending: xprt_autoclose, xs_error_handle [12694.834856] workqueue ipv6_addrconf: flags=0x40008 [12694.846021] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12694.863163] pending: addrconf_verify_work [12694.873705] workqueue knbd0-recv: flags=0x1a [12694.875155] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12694.890186] in-flight: 651:recv_work [12694.895780] workqueue ptlrpc_pinger: flags=0x2 [12694.898613] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12694.904716] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12694.912692] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10057s workers=4 idle: 15979 31824 8 [12694.924787] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12685s workers=2 idle: 654 [12704.057783] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [12704.058082] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [12704.065986] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12704.066183] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12704.066885] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12704.066888] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12704.066902] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12704.066906] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12704.066911] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12704.066913] RDX: 0000000000005926 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12704.066916] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12704.066918] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12704.066920] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12704.066923] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12704.066926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12704.066928] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12704.066936] Call Trace: [12704.066943] [12704.066949] ? show_regs.cold.9+0x22/0x2f [12704.066958] ? watchdog_timer_fn.cold.6+0x62/0x10a [12704.066964] ? __hrtimer_run_queues+0x1c2/0x4d0 [12704.066969] ? watchdog+0x40/0x40 [12704.066973] ? hrtimer_interrupt+0x128/0x2c0 [12704.066977] ? smp_apic_timer_interrupt+0xa6/0x270 [12704.066985] ? apic_timer_interrupt+0xf/0x20 [12704.066990] [12704.066992] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12704.066997] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12704.067002] do_raw_spin_lock+0x177/0x190 [12704.067006] _raw_spin_lock+0x21/0x30 [12704.067014] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12704.067561] obd_mod_free+0x46/0x140 [ptlrpc] [12704.067792] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [12704.067799] Modules linked in: lustre(O) [12704.067779] mdc_commit_open+0x91/0xb0 [mdc] [12704.067804] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12704.067865] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12704.067869] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12704.067871] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12704.067879] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12704.067884] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12704.067891] RAX: 000000000000670a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12704.067894] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12704.067896] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12704.067898] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12704.067900] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12704.067903] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12704.067906] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12704.067908] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12704.067916] Call Trace: [12704.067919] [12704.067922] ? show_regs.cold.9+0x22/0x2f [12704.067928] ? watchdog_timer_fn.cold.6+0x62/0x10a [12704.067933] ? __hrtimer_run_queues+0x1c2/0x4d0 [12704.067937] ? watchdog+0x40/0x40 [12704.067942] ? hrtimer_interrupt+0x128/0x2c0 [12704.067946] ? smp_apic_timer_interrupt+0xa6/0x270 [12704.067952] ? apic_timer_interrupt+0xf/0x20 [12704.067956] [12704.067959] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12704.067965] do_raw_spin_lock+0x177/0x190 [12704.067969] _raw_spin_lock+0x21/0x30 [12704.067973] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12704.068276] ? do_raw_spin_unlock+0x75/0x190 [12704.067890] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12704.068447] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12704.068715] after_reply+0xe2c/0x1780 [ptlrpc] [12704.069425] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12704.069637] ? __next_timer_interrupt+0x160/0x160 [12704.069647] ptlrpcd+0xa82/0xe40 [ptlrpc] [12704.071482] ? do_wait_intr+0xf0/0xf0 [12704.071494] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12704.071736] kthread+0x1d1/0x200 [12704.071743] ? set_kthread_struct+0x70/0x70 [12704.071747] ret_from_fork+0x1f/0x30 [12704.068281] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12704.072081] ? do_wait_intr+0xf0/0xf0 [12704.072090] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12704.075753] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12704.075994] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12704.076252] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12704.076364] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12704.076401] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12704.076432] mdc_intent_lock+0x353/0x7e0 [mdc] [12704.076461] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12704.076554] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12704.076791] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12704.076821] lmv_intent_open+0x338/0x1150 [lmv] [12704.076853] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12704.076946] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12704.077039] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12704.077075] ll_intent_lock+0xc5/0xaa0 [lustre] [12704.077166] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12704.077255] ll_lookup_it+0xe0e/0x2270 [lustre] [12704.077332] ? do_raw_spin_unlock+0x75/0x190 [12704.077340] ? _raw_spin_unlock+0x12/0x30 [12704.077346] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12704.077372] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12704.077391] ll_atomic_open+0x36a/0xd70 [lustre] [12704.077472] lookup_open+0x356/0xb40 [12704.077481] do_last+0x313/0xfc0 [12704.077486] ? nd_jump_root+0xe5/0x160 [12704.077491] ? path_init+0x437/0x520 [12704.077495] path_openat+0xf7/0x500 [12704.077500] ? do_raw_spin_unlock+0x75/0x190 [12704.077505] do_filp_open+0x99/0x140 [12704.077511] ? getname_flags+0x6e/0x330 [12704.077516] ? __check_object_size+0xff/0x256 [12704.077523] ? do_raw_spin_unlock+0x75/0x190 [12704.077527] ? _raw_spin_unlock+0x12/0x30 [12704.077531] do_sys_openat2+0x2b4/0x410 [12704.077536] do_sys_open+0x73/0xa0 [12704.077540] __x64_sys_openat+0x24/0x30 [12704.077544] do_syscall_64+0xc1/0x440 [12704.077550] entry_SYSCALL_64_after_hwframe+0x49/0xae [12704.077556] RIP: 0033:0x7f53425a5332 [12704.077564] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12704.077567] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12704.077572] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12704.077574] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12704.077576] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12704.077578] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12704.077580] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12704.967753] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12705.059647] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12705.082897] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12705.101443] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12705.116765] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12705.152390] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12705.165751] RAX: 0000000000006287 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12705.201172] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12705.210651] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12705.224460] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12705.230604] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12705.243077] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12705.255954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12705.260383] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12705.265918] Call Trace: [12705.267017] [12705.270652] ? show_regs.cold.9+0x22/0x2f [12705.279613] ? watchdog_timer_fn.cold.6+0x62/0x10a [12705.294344] ? __hrtimer_run_queues+0x1c2/0x4d0 [12705.295736] ? watchdog+0x40/0x40 [12705.297712] ? hrtimer_interrupt+0x128/0x2c0 [12705.306299] ? smp_apic_timer_interrupt+0xa6/0x270 [12705.311349] ? apic_timer_interrupt+0xf/0x20 [12705.315721] [12705.317626] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12705.324675] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12705.330513] do_raw_spin_lock+0x177/0x190 [12705.335587] _raw_spin_lock+0x21/0x30 [12705.339489] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12705.352352] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12705.358925] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12705.361944] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12705.369299] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12705.374688] ? woken_wake_function+0x30/0x30 [12705.376707] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12705.382885] kthread+0x1d1/0x200 [12705.384585] ? set_kthread_struct+0x70/0x70 [12705.387763] ret_from_fork+0x1f/0x30 [12720.052229] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [12720.056481] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12720.082714] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12720.088903] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12720.093437] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12720.096349] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12720.100344] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12720.110247] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12720.114336] RAX: 0000000000003863 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12720.117724] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12720.121186] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12720.124426] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12720.127586] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12720.130658] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12720.136283] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12720.139658] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12720.142251] Call Trace: [12720.143012] [12720.143546] ? show_regs.cold.9+0x22/0x2f [12720.144741] ? watchdog_timer_fn.cold.6+0x62/0x10a [12720.146293] ? __hrtimer_run_queues+0x1c2/0x4d0 [12720.147508] ? watchdog+0x40/0x40 [12720.148497] ? hrtimer_interrupt+0x128/0x2c0 [12720.149636] ? smp_apic_timer_interrupt+0xa6/0x270 [12720.151022] ? apic_timer_interrupt+0xf/0x20 [12720.152253] [12720.152723] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12720.154541] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12720.156077] ? newidle_balance.isra.93+0x207/0x580 [12720.163050] do_raw_spin_lock+0x177/0x190 [12720.173342] _raw_spin_lock+0x21/0x30 [12720.177477] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12720.184383] process_one_work+0x2c8/0x700 [12720.188368] worker_thread+0x296/0x6e0 [12720.194201] ? rescuer_thread+0x570/0x570 [12720.199215] kthread+0x1d1/0x200 [12720.201554] ? set_kthread_struct+0x70/0x70 [12720.204516] ret_from_fork+0x1f/0x30 [12725.217088] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10088s! [12725.222440] Showing busy workqueues and worker pools: [12725.245567] workqueue events: flags=0x0 [12725.248440] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12725.258281] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12725.266080] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12725.268326] pending: cpu_latency_work [ptlrpc] [12725.285862] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12725.290944] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12725.296361] workqueue events_long: flags=0x0 [12725.299439] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12725.303964] pending: lu_objects_flush [obdclass] [12725.306916] workqueue events_unbound: flags=0x2 [12725.315920] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12725.321080] pending: flush_memcg_stats_dwork, idle_cull_fn [12725.324838] workqueue events_power_efficient: flags=0x80 [12725.327136] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12725.330029] pending: check_lifetime, do_cache_clean [12725.332320] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12725.338661] pending: neigh_periodic_work, neigh_periodic_work [12725.342487] workqueue mm_percpu_wq: flags=0x8 [12725.349742] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12725.358302] pending: vmstat_update, lru_add_drain_per_cpu [12725.361155] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12725.365366] pending: vmstat_update, lru_add_drain_per_cpu [12725.369034] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12725.372883] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12725.380428] workqueue xprtiod: flags=0xa [12725.383400] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12725.390115] pending: xprt_autoclose, xs_error_handle [12725.393815] workqueue ipv6_addrconf: flags=0x40008 [12725.396534] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12725.401459] pending: addrconf_verify_work [12725.403063] workqueue knbd0-recv: flags=0x1a [12725.404619] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12725.415124] in-flight: 651:recv_work [12725.421987] workqueue ptlrpc_pinger: flags=0x2 [12725.430338] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12725.434964] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12725.439192] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10088s workers=4 idle: 15979 31824 8 [12725.460104] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12715s workers=2 idle: 654 [12732.057104] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [12732.059537] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [12732.059773] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12732.059843] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12732.059849] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12732.059852] RIP: 0010:native_safe_halt+0x2e/0x50 [12732.059863] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12732.059867] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12732.059871] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12732.059874] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12732.059876] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12732.059878] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12732.059881] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12732.059884] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12732.059886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12732.059889] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12732.059896] Call Trace: [12732.059901] [12732.059905] ? show_regs.cold.9+0x22/0x2f [12732.059914] ? watchdog_timer_fn.cold.6+0x62/0x10a [12732.059919] ? __hrtimer_run_queues+0x1c2/0x4d0 [12732.059923] ? watchdog+0x40/0x40 [12732.059927] ? hrtimer_interrupt+0x128/0x2c0 [12732.059930] ? smp_apic_timer_interrupt+0xa6/0x270 [12732.059936] ? apic_timer_interrupt+0xf/0x20 [12732.059940] [12732.059943] ? native_safe_halt+0x2e/0x50 [12732.059946] kvm_wait+0xcd/0xe0 [12732.059952] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12732.059958] do_raw_spin_lock+0x177/0x190 [12732.059962] _raw_spin_lock+0x21/0x30 [12732.059966] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12732.060222] obd_mod_free+0x46/0x140 [ptlrpc] [12732.060414] mdc_commit_open+0x91/0xb0 [mdc] [12732.060452] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12732.060593] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [12732.060598] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12732.060711] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12732.060715] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12732.060717] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12732.060724] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12732.060728] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12732.060732] RAX: 0000000000007d8d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12732.060734] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12732.060736] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12732.060738] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12732.060740] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12732.060743] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12732.060745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12732.060747] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12732.060754] Call Trace: [12732.060757] [12732.060759] ? show_regs.cold.9+0x22/0x2f [12732.060664] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12732.060766] ? watchdog_timer_fn.cold.6+0x62/0x10a [12732.060770] ? __hrtimer_run_queues+0x1c2/0x4d0 [12732.060773] ? watchdog+0x40/0x40 [12732.060777] ? hrtimer_interrupt+0x128/0x2c0 [12732.060780] ? smp_apic_timer_interrupt+0xa6/0x270 [12732.060785] ? apic_timer_interrupt+0xf/0x20 [12732.060790] [12732.060792] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12732.060796] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12732.060801] do_raw_spin_lock+0x177/0x190 [12732.060805] _raw_spin_lock+0x21/0x30 [12732.060809] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12732.060997] after_reply+0xe2c/0x1780 [ptlrpc] [12732.061655] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12732.061858] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12732.062080] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12732.061421] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12732.062682] ? __next_timer_interrupt+0x160/0x160 [12732.062633] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12732.062693] ptlrpcd+0xa82/0xe40 [ptlrpc] [12732.063120] ? do_wait_intr+0xf0/0xf0 [12732.063130] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12732.063495] ? woken_wake_function+0x30/0x30 [12732.063651] kthread+0x1d1/0x200 [12732.063657] ? set_kthread_struct+0x70/0x70 [12732.063660] ret_from_fork+0x1f/0x30 [12732.063501] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12732.064633] kthread+0x1d1/0x200 [12732.064710] ? set_kthread_struct+0x70/0x70 [12732.064714] ret_from_fork+0x1f/0x30 [12732.373027] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12732.407300] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12732.413776] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12732.419586] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12732.422650] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12732.439249] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12732.448037] RAX: 0000000000007d8c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12732.458361] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12732.469740] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12732.476113] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12732.478939] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12732.485705] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12732.493317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12732.498713] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12732.502718] Call Trace: [12732.504710] [12732.507485] ? show_regs.cold.9+0x22/0x2f [12732.510501] ? watchdog_timer_fn.cold.6+0x62/0x10a [12732.513051] ? __hrtimer_run_queues+0x1c2/0x4d0 [12732.518819] ? watchdog+0x40/0x40 [12732.520051] ? hrtimer_interrupt+0x128/0x2c0 [12732.525618] ? smp_apic_timer_interrupt+0xa6/0x270 [12732.527213] ? apic_timer_interrupt+0xf/0x20 [12732.530981] [12732.531798] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12732.537730] do_raw_spin_lock+0x177/0x190 [12732.542423] _raw_spin_lock+0x21/0x30 [12732.544935] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12732.549118] ? do_raw_spin_unlock+0x75/0x190 [12732.554036] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12732.559284] ? do_wait_intr+0xf0/0xf0 [12732.560684] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12732.569720] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12732.572605] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12732.582088] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12732.587081] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12732.590467] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12732.593446] mdc_intent_lock+0x353/0x7e0 [mdc] [12732.597271] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12732.604977] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12732.609748] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12732.614820] lmv_intent_open+0x338/0x1150 [lmv] [12732.617865] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12732.621005] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12732.624046] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12732.626101] ll_intent_lock+0xc5/0xaa0 [lustre] [12732.628526] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12732.633205] ll_lookup_it+0xe0e/0x2270 [lustre] [12732.635984] ? do_raw_spin_unlock+0x75/0x190 [12732.639749] ? _raw_spin_unlock+0x12/0x30 [12732.641429] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12732.643732] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12732.646204] ll_atomic_open+0x36a/0xd70 [lustre] [12732.649799] lookup_open+0x356/0xb40 [12732.651605] do_last+0x313/0xfc0 [12732.653324] ? nd_jump_root+0xe5/0x160 [12732.654639] ? path_init+0x437/0x520 [12732.658157] path_openat+0xf7/0x500 [12732.659417] ? do_raw_spin_unlock+0x75/0x190 [12732.664306] do_filp_open+0x99/0x140 [12732.671595] ? getname_flags+0x6e/0x330 [12732.677648] ? __check_object_size+0xff/0x256 [12732.679866] ? do_raw_spin_unlock+0x75/0x190 [12732.682435] ? _raw_spin_unlock+0x12/0x30 [12732.688361] do_sys_openat2+0x2b4/0x410 [12732.689778] do_sys_open+0x73/0xa0 [12732.690842] __x64_sys_openat+0x24/0x30 [12732.693036] do_syscall_64+0xc1/0x440 [12732.695524] entry_SYSCALL_64_after_hwframe+0x49/0xae [12732.698336] RIP: 0033:0x7f53425a5332 [12732.700060] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12732.711642] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12732.717108] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12732.721064] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12732.728640] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12732.731226] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12732.735503] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12752.052012] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [12752.055278] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12752.075543] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12752.080743] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12752.084763] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12752.088292] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12752.091734] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12752.100100] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12752.104165] RAX: 0000000000003cc1 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12752.108883] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12752.112391] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12752.116638] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12752.119588] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12752.122879] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12752.126829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12752.129584] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12752.133097] Call Trace: [12752.134363] [12752.135313] ? show_regs.cold.9+0x22/0x2f [12752.139589] ? watchdog_timer_fn.cold.6+0x62/0x10a [12752.143226] ? __hrtimer_run_queues+0x1c2/0x4d0 [12752.148339] ? watchdog+0x40/0x40 [12752.149827] ? hrtimer_interrupt+0x128/0x2c0 [12752.153927] ? smp_apic_timer_interrupt+0xa6/0x270 [12752.157241] ? apic_timer_interrupt+0xf/0x20 [12752.159383] [12752.160575] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12752.166561] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12752.169128] ? newidle_balance.isra.93+0x207/0x580 [12752.171741] do_raw_spin_lock+0x177/0x190 [12752.174371] _raw_spin_lock+0x21/0x30 [12752.185007] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12752.188717] process_one_work+0x2c8/0x700 [12752.190595] worker_thread+0x296/0x6e0 [12752.196869] ? rescuer_thread+0x570/0x570 [12752.198345] kthread+0x1d1/0x200 [12752.199756] ? set_kthread_struct+0x70/0x70 [12752.202671] ret_from_fork+0x1f/0x30 [12755.938945] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10118s! [12755.945083] Showing busy workqueues and worker pools: [12755.948792] workqueue events: flags=0x0 [12755.952356] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12755.956034] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12755.961660] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12755.964694] pending: cpu_latency_work [ptlrpc] [12755.967640] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12755.971194] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12755.975642] workqueue events_long: flags=0x0 [12755.977708] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12755.980982] pending: lu_objects_flush [obdclass] [12755.984416] workqueue events_unbound: flags=0x2 [12755.986672] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12755.990061] pending: flush_memcg_stats_dwork, idle_cull_fn [12755.993038] workqueue events_power_efficient: flags=0x80 [12755.995841] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12755.999572] pending: check_lifetime, do_cache_clean [12756.002115] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12756.004790] pending: neigh_periodic_work, neigh_periodic_work [12756.007064] workqueue mm_percpu_wq: flags=0x8 [12756.008703] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12756.012391] pending: vmstat_update, lru_add_drain_per_cpu [12756.015265] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12756.018587] pending: vmstat_update, lru_add_drain_per_cpu [12756.021365] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12756.024613] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12756.031389] workqueue xprtiod: flags=0xa [12756.034447] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12756.054504] pending: xprt_autoclose, xs_error_handle [12756.063363] workqueue ipv6_addrconf: flags=0x40008 [12756.069372] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12756.074407] pending: addrconf_verify_work [12756.081738] workqueue knbd0-recv: flags=0x1a [12756.087976] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12756.095153] in-flight: 651:recv_work [12756.098860] workqueue ptlrpc_pinger: flags=0x2 [12756.105549] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12756.113175] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12756.119194] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10119s workers=4 idle: 15979 31824 8 [12756.133839] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12746s workers=2 idle: 654 [12760.055126] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [12760.057132] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [12760.058019] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [12760.058028] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12760.058112] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12760.058118] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12760.058121] RIP: 0010:native_safe_halt+0x2e/0x50 [12760.058134] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12760.058139] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12760.058143] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12760.058146] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12760.058148] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12760.058150] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12760.058153] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12760.058157] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12760.060992] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12760.060996] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12760.061006] Call Trace: [12760.061015] [12760.061023] ? show_regs.cold.9+0x22/0x2f [12760.061038] ? watchdog_timer_fn.cold.6+0x62/0x10a [12760.061045] ? __hrtimer_run_queues+0x1c2/0x4d0 [12760.061050] ? watchdog+0x40/0x40 [12760.061055] ? hrtimer_interrupt+0x128/0x2c0 [12760.061070] ? smp_apic_timer_interrupt+0xa6/0x270 [12760.061078] ? apic_timer_interrupt+0xf/0x20 [12760.061083] [12760.061086] ? native_safe_halt+0x2e/0x50 [12760.061090] kvm_wait+0xcd/0xe0 [12760.061098] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12760.061107] do_raw_spin_lock+0x177/0x190 [12760.061112] _raw_spin_lock+0x21/0x30 [12760.061116] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12760.061660] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12760.061784] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12760.061941] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12760.062154] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12760.062366] ? woken_wake_function+0x30/0x30 [12760.062374] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12760.062581] kthread+0x1d1/0x200 [12760.062586] ? set_kthread_struct+0x70/0x70 [12760.062590] ret_from_fork+0x1f/0x30 [12760.066207] Modules linked in: lustre(O) [12760.073347] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12760.073425] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12760.073434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12760.073438] RIP: 0010:native_safe_halt+0x2e/0x50 [12760.073452] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12760.073457] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12760.073462] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12760.073465] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12760.073467] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12760.073470] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12760.073472] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12760.073475] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12760.073478] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12760.073481] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12760.073490] Call Trace: [12760.073497] [12760.073503] ? show_regs.cold.9+0x22/0x2f [12760.073514] ? watchdog_timer_fn.cold.6+0x62/0x10a [12760.073522] ? __hrtimer_run_queues+0x1c2/0x4d0 [12760.073527] ? watchdog+0x40/0x40 [12760.073531] ? hrtimer_interrupt+0x128/0x2c0 [12760.073536] ? smp_apic_timer_interrupt+0xa6/0x270 [12760.073732] ? apic_timer_interrupt+0xf/0x20 [12760.073737] [12760.073741] ? native_safe_halt+0x2e/0x50 [12760.073745] kvm_wait+0xcd/0xe0 [12760.073753] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12760.073762] do_raw_spin_lock+0x177/0x190 [12760.073767] _raw_spin_lock+0x21/0x30 [12760.073773] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12760.075160] obd_mod_free+0x46/0x140 [ptlrpc] [12760.076140] mdc_commit_open+0x91/0xb0 [mdc] [12760.076178] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12760.076953] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12760.086131] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12760.086209] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12760.086216] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12760.086219] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12760.086234] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12760.086238] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12760.086242] RAX: 0000000000007f16 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12760.086245] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12760.086247] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12760.086249] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12760.086251] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12760.086254] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12760.086257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12760.086259] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12760.086268] Call Trace: [12760.086273] [12760.086279] ? show_regs.cold.9+0x22/0x2f [12760.086288] ? watchdog_timer_fn.cold.6+0x62/0x10a [12760.086294] ? __hrtimer_run_queues+0x1c2/0x4d0 [12760.086299] ? watchdog+0x40/0x40 [12760.086302] ? hrtimer_interrupt+0x128/0x2c0 [12760.086306] ? smp_apic_timer_interrupt+0xa6/0x270 [12760.086314] ? apic_timer_interrupt+0xf/0x20 [12760.086318] [12760.086321] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12760.086327] do_raw_spin_lock+0x177/0x190 [12760.086332] _raw_spin_lock+0x21/0x30 [12760.086337] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12760.086634] ? do_raw_spin_unlock+0x75/0x190 [12760.086641] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12760.133804] after_reply+0xe2c/0x1780 [ptlrpc] [12760.143580] ? do_wait_intr+0xf0/0xf0 [12760.148913] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12760.149487] ? __next_timer_interrupt+0x160/0x160 [12760.149497] ptlrpcd+0xa82/0xe40 [ptlrpc] [12760.156395] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12760.181418] ? do_wait_intr+0xf0/0xf0 [12760.181443] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12760.190553] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12760.197872] kthread+0x1d1/0x200 [12760.197897] ? set_kthread_struct+0x70/0x70 [12760.197903] ret_from_fork+0x1f/0x30 [12760.842200] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12760.844399] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12760.855794] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12760.858756] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12760.867809] mdc_intent_lock+0x353/0x7e0 [mdc] [12760.872147] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12760.878248] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12760.884467] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12760.892420] lmv_intent_open+0x338/0x1150 [lmv] [12760.893698] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12760.897763] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12760.905195] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12760.909899] ll_intent_lock+0xc5/0xaa0 [lustre] [12760.912445] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12760.917000] ll_lookup_it+0xe0e/0x2270 [lustre] [12760.918728] ? do_raw_spin_unlock+0x75/0x190 [12760.927727] ? _raw_spin_unlock+0x12/0x30 [12760.930442] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12760.938180] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12760.943951] ll_atomic_open+0x36a/0xd70 [lustre] [12760.946302] lookup_open+0x356/0xb40 [12760.952197] do_last+0x313/0xfc0 [12760.953364] ? nd_jump_root+0xe5/0x160 [12760.966398] ? path_init+0x437/0x520 [12760.967695] path_openat+0xf7/0x500 [12760.968857] ? do_raw_spin_unlock+0x75/0x190 [12760.979978] do_filp_open+0x99/0x140 [12760.986155] ? getname_flags+0x6e/0x330 [12760.991498] ? __check_object_size+0xff/0x256 [12760.993001] ? do_raw_spin_unlock+0x75/0x190 [12761.006158] ? _raw_spin_unlock+0x12/0x30 [12761.007534] do_sys_openat2+0x2b4/0x410 [12761.013032] do_sys_open+0x73/0xa0 [12761.018767] __x64_sys_openat+0x24/0x30 [12761.030695] do_syscall_64+0xc1/0x440 [12761.031986] entry_SYSCALL_64_after_hwframe+0x49/0xae [12761.044217] RIP: 0033:0x7f53425a5332 [12761.050806] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12761.076908] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12761.082082] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12761.097065] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12761.101151] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12761.104600] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12761.108229] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12774.683545] rcu: INFO: rcu_sched self-detected stall on CPU [12774.685505] rcu: 2-....: (5928051 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [12774.690570] (t=10680257 jiffies g=274185 q=66620) [12774.693143] rcu: rcu_sched kthread starved for 10138623 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [12774.697967] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12774.701875] rcu: RCU grace-period kthread stack dump: [12774.704508] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12774.709132] Call Trace: [12774.710184] __schedule+0x351/0xcb0 [12774.711923] schedule+0xc0/0x180 [12774.713112] schedule_timeout+0xb4/0x190 [12774.714368] ? do_raw_spin_unlock+0x75/0x190 [12774.720092] ? __next_timer_interrupt+0x160/0x160 [12774.725473] ? prepare_to_swait_event+0xf3/0x140 [12774.729539] rcu_gp_kthread+0x8dd/0x15b0 [12774.732250] ? force_qs_rnp+0x2c0/0x2c0 [12774.734470] kthread+0x1d1/0x200 [12774.735937] ? set_kthread_struct+0x70/0x70 [12774.737972] ret_from_fork+0x1f/0x30 [12774.739436] rcu: Stack dump where RCU GP kthread last ran: [12774.745902] NMI backtrace for cpu 2 [12774.747144] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12774.758496] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12774.765121] Call Trace: [12774.767088] [12774.768143] ? dump_stack+0xbb/0x10e [12774.769628] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12774.772116] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12774.777881] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12774.779663] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12774.790358] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12774.795105] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [12774.799138] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [12774.802903] ? update_process_times+0x6d/0xc0 [12774.807747] ? tick_sched_handle.isra.18+0x27/0x90 [12774.810791] ? tick_sched_timer+0x53/0xb0 [12774.813060] ? __hrtimer_run_queues+0x1c2/0x4d0 [12774.822082] ? tick_sched_handle.isra.18+0x90/0x90 [12774.828745] ? hrtimer_interrupt+0x128/0x2c0 [12774.834115] ? smp_apic_timer_interrupt+0xa6/0x270 [12774.837403] ? apic_timer_interrupt+0xf/0x20 [12774.844545] [12774.847175] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12774.854570] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12774.861523] ? do_raw_spin_lock+0x177/0x190 [12774.866085] ? _raw_spin_lock+0x21/0x30 [12774.871109] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12774.878415] ? obd_mod_free+0x46/0x140 [ptlrpc] [12774.885462] ? mdc_commit_open+0x91/0xb0 [mdc] [12774.889421] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12774.895195] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12774.898076] ? after_reply+0xe2c/0x1780 [ptlrpc] [12774.904167] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12774.906119] ? __next_timer_interrupt+0x160/0x160 [12774.912578] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12774.918139] ? do_wait_intr+0xf0/0xf0 [12774.919504] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12774.927310] ? kthread+0x1d1/0x200 [12774.932637] ? set_kthread_struct+0x70/0x70 [12774.934051] ? ret_from_fork+0x1f/0x30 [12774.936790] Sending NMI from CPU 2 to CPUs 1: [12774.943111] NMI backtrace for cpu 1 [12774.943118] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12774.943120] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12774.943122] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [12774.943124] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [12774.943126] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [12774.943129] RAX: 00000000000006cc RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12774.943130] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12774.943132] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12774.943133] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12774.943135] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12774.943136] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12774.943138] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12774.943139] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12774.943140] Call Trace: [12774.943141] [12774.943142] ? show_regs.cold.9+0x22/0x2f [12774.943143] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [12774.943144] ? nmi_cpu_backtrace_handler+0x15/0x40 [12774.943145] ? nmi_handle+0xbd/0x210 [12774.943146] ? default_do_nmi+0x4b/0x1b0 [12774.943146] ? do_nmi+0x280/0x380 [12774.943147] ? end_repeat_nmi+0x16/0x1f [12774.943149] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [12774.943150] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [12774.943151] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [12774.943151] [12774.943153] do_raw_spin_lock+0x177/0x190 [12774.943153] _raw_spin_lock+0x21/0x30 [12774.943154] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12774.943155] ? do_raw_spin_unlock+0x75/0x190 [12774.943156] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12774.943157] ? do_wait_intr+0xf0/0xf0 [12774.943163] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12774.943164] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12774.943165] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12774.943166] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12774.943167] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12774.943168] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12774.943169] mdc_intent_lock+0x353/0x7e0 [mdc] [12774.943170] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12774.943171] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12774.943172] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12774.943173] lmv_intent_open+0x338/0x1150 [lmv] [12774.943174] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12774.943175] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12774.943177] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12774.943178] ll_intent_lock+0xc5/0xaa0 [lustre] [12774.943179] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12774.943180] ll_lookup_it+0xe0e/0x2270 [lustre] [12774.943180] ? do_raw_spin_unlock+0x75/0x190 [12774.943182] ? _raw_spin_unlock+0x12/0x30 [12774.943183] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12774.943184] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12774.943185] ll_atomic_open+0x36a/0xd70 [lustre] [12774.943186] lookup_open+0x356/0xb40 [12774.943187] do_last+0x313/0xfc0 [12774.943188] ? nd_jump_root+0xe5/0x160 [12774.943189] ? path_init+0x437/0x520 [12774.943190] path_openat+0xf7/0x500 [12774.943190] ? do_raw_spin_unlock+0x75/0x190 [12774.943192] do_filp_open+0x99/0x140 [12774.943193] ? getname_flags+0x6e/0x330 [12774.943194] ? __check_object_size+0xff/0x256 [12774.943198] ? do_raw_spin_unlock+0x75/0x190 [12774.943199] ? _raw_spin_unlock+0x12/0x30 [12774.943200] do_sys_openat2+0x2b4/0x410 [12774.943201] do_sys_open+0x73/0xa0 [12774.943202] __x64_sys_openat+0x24/0x30 [12774.943203] do_syscall_64+0xc1/0x440 [12774.943204] entry_SYSCALL_64_after_hwframe+0x49/0xae [12774.943205] RIP: 0033:0x7f53425a5332 [12774.943206] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12774.943208] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12774.943210] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12774.943212] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12774.943213] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12774.943214] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12774.943216] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12774.945351] NMI backtrace for cpu 2 [12775.265526] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12775.271387] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12775.279024] Call Trace: [12775.281339] [12775.285738] ? dump_stack+0xbb/0x10e [12775.287921] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12775.296236] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12775.299170] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12775.305090] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12775.315548] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12775.324516] ? rcu_dump_cpu_stacks+0x11b/0x171 [12775.331171] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12775.343709] ? update_process_times+0x6d/0xc0 [12775.345612] ? tick_sched_handle.isra.18+0x27/0x90 [12775.350158] ? tick_sched_timer+0x53/0xb0 [12775.353960] ? __hrtimer_run_queues+0x1c2/0x4d0 [12775.358590] ? tick_sched_handle.isra.18+0x90/0x90 [12775.365748] ? hrtimer_interrupt+0x128/0x2c0 [12775.376951] ? smp_apic_timer_interrupt+0xa6/0x270 [12775.379094] ? apic_timer_interrupt+0xf/0x20 [12775.390300] [12775.393035] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12775.400549] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12775.407664] ? do_raw_spin_lock+0x177/0x190 [12775.411204] ? _raw_spin_lock+0x21/0x30 [12775.416632] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12775.423251] ? obd_mod_free+0x46/0x140 [ptlrpc] [12775.429622] ? mdc_commit_open+0x91/0xb0 [mdc] [12775.432834] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12775.439230] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12775.443801] ? after_reply+0xe2c/0x1780 [ptlrpc] [12775.451296] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12775.455731] ? __next_timer_interrupt+0x160/0x160 [12775.457738] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12775.463020] ? do_wait_intr+0xf0/0xf0 [12775.469821] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12775.480144] ? kthread+0x1d1/0x200 [12775.482760] ? set_kthread_struct+0x70/0x70 [12775.485437] ? ret_from_fork+0x1f/0x30 [12784.052046] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [12784.057366] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12784.100127] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12784.120626] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12784.130087] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12784.135980] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12784.140813] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12784.161038] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12784.163234] RAX: 0000000000005265 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12784.174888] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12784.179437] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12784.187749] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12784.195335] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12784.202425] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12784.213577] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12784.224607] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12784.228404] Call Trace: [12784.234775] [12784.240099] ? show_regs.cold.9+0x22/0x2f [12784.241515] ? watchdog_timer_fn.cold.6+0x62/0x10a [12784.248995] ? __hrtimer_run_queues+0x1c2/0x4d0 [12784.257646] ? watchdog+0x40/0x40 [12784.258784] ? hrtimer_interrupt+0x128/0x2c0 [12784.260167] ? smp_apic_timer_interrupt+0xa6/0x270 [12784.268980] ? apic_timer_interrupt+0xf/0x20 [12784.272604] [12784.277078] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12784.284165] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12784.290174] ? newidle_balance.isra.93+0x207/0x580 [12784.294372] do_raw_spin_lock+0x177/0x190 [12784.301116] _raw_spin_lock+0x21/0x30 [12784.307095] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12784.312441] process_one_work+0x2c8/0x700 [12784.313842] worker_thread+0x296/0x6e0 [12784.318549] ? rescuer_thread+0x570/0x570 [12784.322259] kthread+0x1d1/0x200 [12784.324863] ? set_kthread_struct+0x70/0x70 [12784.326213] ret_from_fork+0x1f/0x30 [12786.656032] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10149s! [12786.662017] Showing busy workqueues and worker pools: [12786.666856] workqueue events: flags=0x0 [12786.673700] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12786.684122] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12786.697374] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12786.706890] pending: cpu_latency_work [ptlrpc] [12786.709701] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12786.715290] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12786.726745] workqueue events_long: flags=0x0 [12786.729398] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12786.736488] pending: lu_objects_flush [obdclass] [12786.745550] workqueue events_unbound: flags=0x2 [12786.751182] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12786.754544] pending: flush_memcg_stats_dwork, idle_cull_fn [12786.757942] workqueue events_power_efficient: flags=0x80 [12786.760532] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12786.763885] pending: check_lifetime, do_cache_clean [12786.767281] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12786.770073] pending: neigh_periodic_work, neigh_periodic_work [12786.772216] workqueue mm_percpu_wq: flags=0x8 [12786.775495] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12786.781657] pending: vmstat_update, lru_add_drain_per_cpu [12786.786886] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12786.791803] pending: vmstat_update, lru_add_drain_per_cpu [12786.793199] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12786.799777] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12786.807819] workqueue xprtiod: flags=0xa [12786.815162] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12786.821026] pending: xprt_autoclose, xs_error_handle [12786.832327] workqueue ipv6_addrconf: flags=0x40008 [12786.836083] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12786.840275] pending: addrconf_verify_work [12786.845516] workqueue knbd0-recv: flags=0x1a [12786.847997] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12786.855373] in-flight: 651:recv_work [12786.858417] workqueue ptlrpc_pinger: flags=0x2 [12786.862766] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12786.867622] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12786.870962] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10149s workers=4 idle: 15979 31824 8 [12786.876253] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12777s workers=2 idle: 654 [12788.059582] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [12788.065388] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12788.105054] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12788.117672] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12788.121697] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12788.125173] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12788.138914] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12788.144954] RAX: 0000000000005b86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12788.148888] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12788.151077] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12788.162710] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12788.165953] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12788.169586] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12788.173669] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12788.178125] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12788.180538] Call Trace: [12788.181479] [12788.182387] ? show_regs.cold.9+0x22/0x2f [12788.185039] ? watchdog_timer_fn.cold.6+0x62/0x10a [12788.187052] ? __hrtimer_run_queues+0x1c2/0x4d0 [12788.189792] ? watchdog+0x40/0x40 [12788.192275] ? hrtimer_interrupt+0x128/0x2c0 [12788.194051] ? smp_apic_timer_interrupt+0xa6/0x270 [12788.198043] ? apic_timer_interrupt+0xf/0x20 [12788.204646] [12788.205186] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12788.209678] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12788.212442] do_raw_spin_lock+0x177/0x190 [12788.214063] _raw_spin_lock+0x21/0x30 [12788.218191] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12788.224033] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12788.227163] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12788.231994] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12788.234964] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12788.237648] ? woken_wake_function+0x30/0x30 [12788.239253] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12788.242532] kthread+0x1d1/0x200 [12788.243812] ? set_kthread_struct+0x70/0x70 [12788.247911] ret_from_fork+0x1f/0x30 [12800.055137] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [12800.057888] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [12800.059426] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12800.059503] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12800.059511] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12800.059514] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12800.059790] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12800.059795] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12800.059799] RAX: 0000000000006734 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12800.059802] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12800.059804] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12800.059806] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12800.059808] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12800.059812] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12800.059814] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12800.059817] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12800.059827] Call Trace: [12800.059833] [12800.059839] ? show_regs.cold.9+0x22/0x2f [12800.059850] ? watchdog_timer_fn.cold.6+0x62/0x10a [12800.059856] ? __hrtimer_run_queues+0x1c2/0x4d0 [12800.059861] ? watchdog+0x40/0x40 [12800.059865] ? hrtimer_interrupt+0x128/0x2c0 [12800.059869] ? smp_apic_timer_interrupt+0xa6/0x270 [12800.059880] ? apic_timer_interrupt+0xf/0x20 [12800.059884] [12800.059888] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12800.059895] do_raw_spin_lock+0x177/0x190 [12800.059901] _raw_spin_lock+0x21/0x30 [12800.059906] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12800.060661] ? do_raw_spin_unlock+0x75/0x190 [12800.060669] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12800.060868] ? do_wait_intr+0xf0/0xf0 [12800.060876] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12800.061419] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12800.070384] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12800.070458] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12800.070466] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12800.070469] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12800.070482] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12800.070486] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12800.070491] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12800.070493] RDX: 0000000000004409 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12800.070502] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12800.070505] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12800.097643] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12800.097951] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12800.098304] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12800.098349] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12800.098448] mdc_intent_lock+0x353/0x7e0 [mdc] [12800.098476] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12800.098707] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12800.099286] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12800.099321] lmv_intent_open+0x338/0x1150 [lmv] [12800.099425] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12800.099640] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12800.099724] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12800.099748] ll_intent_lock+0xc5/0xaa0 [lustre] [12800.099840] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12800.099929] ll_lookup_it+0xe0e/0x2270 [lustre] [12800.100492] ? do_raw_spin_unlock+0x75/0x190 [12800.100501] ? _raw_spin_unlock+0x12/0x30 [12800.100509] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12800.100556] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12800.100577] ll_atomic_open+0x36a/0xd70 [lustre] [12800.100668] lookup_open+0x356/0xb40 [12800.100677] do_last+0x313/0xfc0 [12800.100682] ? nd_jump_root+0xe5/0x160 [12800.100686] ? path_init+0x437/0x520 [12800.100691] path_openat+0xf7/0x500 [12800.100696] ? do_raw_spin_unlock+0x75/0x190 [12800.100701] do_filp_open+0x99/0x140 [12800.100707] ? getname_flags+0x6e/0x330 [12800.100712] ? __check_object_size+0xff/0x256 [12800.100719] ? do_raw_spin_unlock+0x75/0x190 [12800.100722] ? _raw_spin_unlock+0x12/0x30 [12800.100732] do_sys_openat2+0x2b4/0x410 [12800.100738] do_sys_open+0x73/0xa0 [12800.100742] __x64_sys_openat+0x24/0x30 [12800.100745] do_syscall_64+0xc1/0x440 [12800.100752] entry_SYSCALL_64_after_hwframe+0x49/0xae [12800.100759] RIP: 0033:0x7f53425a5332 [12800.100766] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12800.100770] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12800.100775] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12800.100778] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12800.100780] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12800.100782] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12800.100786] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12800.453678] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12800.459133] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12800.464647] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12800.467638] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12800.471090] Call Trace: [12800.472079] [12800.473106] ? show_regs.cold.9+0x22/0x2f [12800.475705] ? watchdog_timer_fn.cold.6+0x62/0x10a [12800.478426] ? __hrtimer_run_queues+0x1c2/0x4d0 [12800.480632] ? watchdog+0x40/0x40 [12800.482333] ? hrtimer_interrupt+0x128/0x2c0 [12800.484245] ? smp_apic_timer_interrupt+0xa6/0x270 [12800.486249] ? apic_timer_interrupt+0xf/0x20 [12800.489941] [12800.492584] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12800.498424] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12800.505578] do_raw_spin_lock+0x177/0x190 [12800.509449] _raw_spin_lock+0x21/0x30 [12800.511745] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12800.516962] obd_mod_free+0x46/0x140 [ptlrpc] [12800.519199] mdc_commit_open+0x91/0xb0 [mdc] [12800.521019] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12800.524017] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12800.526849] after_reply+0xe2c/0x1780 [ptlrpc] [12800.530150] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12800.536777] ? __next_timer_interrupt+0x160/0x160 [12800.538738] ptlrpcd+0xa82/0xe40 [ptlrpc] [12800.541938] ? do_wait_intr+0xf0/0xf0 [12800.544876] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12800.549866] kthread+0x1d1/0x200 [12800.551957] ? set_kthread_struct+0x70/0x70 [12800.553790] ret_from_fork+0x1f/0x30 [12812.054273] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [12812.066300] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12812.096791] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12812.104946] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12812.109095] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12812.112716] RIP: 0010:native_safe_halt+0x2e/0x50 [12812.120128] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12812.131842] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12812.140188] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12812.143771] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [12812.152699] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12812.162468] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12812.168019] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12812.178821] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12812.187696] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12812.194430] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12812.203774] Call Trace: [12812.206072] [12812.206605] ? show_regs.cold.9+0x22/0x2f [12812.207966] ? watchdog_timer_fn.cold.6+0x62/0x10a [12812.210329] ? __hrtimer_run_queues+0x1c2/0x4d0 [12812.211830] ? watchdog+0x40/0x40 [12812.213079] ? hrtimer_interrupt+0x128/0x2c0 [12812.214514] ? smp_apic_timer_interrupt+0xa6/0x270 [12812.215936] ? apic_timer_interrupt+0xf/0x20 [12812.217751] [12812.218757] ? native_safe_halt+0x2e/0x50 [12812.220021] kvm_wait+0xcd/0xe0 [12812.220878] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12812.222550] ? newidle_balance.isra.93+0x207/0x580 [12812.228428] do_raw_spin_lock+0x177/0x190 [12812.230919] _raw_spin_lock+0x21/0x30 [12812.232708] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12812.238088] process_one_work+0x2c8/0x700 [12812.242375] worker_thread+0x296/0x6e0 [12812.252813] ? rescuer_thread+0x570/0x570 [12812.254718] kthread+0x1d1/0x200 [12812.255937] ? set_kthread_struct+0x70/0x70 [12812.258356] ret_from_fork+0x1f/0x30 [12816.058107] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [12816.061879] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12816.123796] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12816.144577] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12816.151862] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12816.154477] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12816.173718] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12816.179603] RAX: 0000000000006709 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12816.188017] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12816.195042] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12816.201882] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12816.209869] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12816.222611] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12816.233258] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12816.244299] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12816.251503] Call Trace: [12816.254061] [12816.256769] ? show_regs.cold.9+0x22/0x2f [12816.260952] ? watchdog_timer_fn.cold.6+0x62/0x10a [12816.265320] ? __hrtimer_run_queues+0x1c2/0x4d0 [12816.275303] ? watchdog+0x40/0x40 [12816.278940] ? hrtimer_interrupt+0x128/0x2c0 [12816.284555] ? smp_apic_timer_interrupt+0xa6/0x270 [12816.289727] ? apic_timer_interrupt+0xf/0x20 [12816.298384] [12816.300330] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12816.309756] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12816.320909] do_raw_spin_lock+0x177/0x190 [12816.325818] _raw_spin_lock+0x21/0x30 [12816.337454] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12816.350515] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12816.363032] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12816.373701] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12816.385325] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12816.390808] ? woken_wake_function+0x30/0x30 [12816.398540] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12816.409993] kthread+0x1d1/0x200 [12816.411715] ? set_kthread_struct+0x70/0x70 [12816.412966] ret_from_fork+0x1f/0x30 [12817.386067] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10180s! [12817.397484] Showing busy workqueues and worker pools: [12817.405030] workqueue events: flags=0x0 [12817.406459] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12817.419786] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12817.445158] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12817.450634] pending: cpu_latency_work [ptlrpc] [12817.453457] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12817.456691] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12817.461188] workqueue events_long: flags=0x0 [12817.462729] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12817.466174] pending: lu_objects_flush [obdclass] [12817.469805] workqueue events_unbound: flags=0x2 [12817.475733] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12817.480958] pending: flush_memcg_stats_dwork, idle_cull_fn [12817.486177] workqueue events_power_efficient: flags=0x80 [12817.493027] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12817.495284] pending: check_lifetime, do_cache_clean [12817.502960] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12817.511895] pending: neigh_periodic_work, neigh_periodic_work [12817.516633] workqueue mm_percpu_wq: flags=0x8 [12817.519770] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12817.527957] pending: vmstat_update, lru_add_drain_per_cpu [12817.534824] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12817.537162] pending: vmstat_update, lru_add_drain_per_cpu [12817.542911] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12817.549949] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12817.553528] workqueue xprtiod: flags=0xa [12817.556443] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12817.559399] pending: xprt_autoclose, xs_error_handle [12817.566332] workqueue ipv6_addrconf: flags=0x40008 [12817.569441] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12817.576830] pending: addrconf_verify_work [12817.581685] workqueue knbd0-recv: flags=0x1a [12817.586333] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12817.592565] in-flight: 651:recv_work [12817.598163] workqueue ptlrpc_pinger: flags=0x2 [12817.610533] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12817.626408] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12817.628256] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10180s workers=4 idle: 15979 31824 8 [12817.637703] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12808s workers=2 idle: 654 [12828.057051] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [12828.061428] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12828.062565] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [12828.084501] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12828.084516] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12828.084519] RIP: 0010:native_safe_halt+0x2e/0x50 [12828.084602] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12828.084607] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12828.084613] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12828.084615] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12828.084616] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12828.084619] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12828.084621] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12828.084624] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12828.084627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12828.084630] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12828.084641] Call Trace: [12828.084660] [12828.084673] ? show_regs.cold.9+0x22/0x2f [12828.084682] ? watchdog_timer_fn.cold.6+0x62/0x10a [12828.084688] ? __hrtimer_run_queues+0x1c2/0x4d0 [12828.084692] ? watchdog+0x40/0x40 [12828.084696] ? hrtimer_interrupt+0x128/0x2c0 [12828.084704] ? smp_apic_timer_interrupt+0xa6/0x270 [12828.084710] ? apic_timer_interrupt+0xf/0x20 [12828.084715] [12828.084718] ? native_safe_halt+0x2e/0x50 [12828.084726] kvm_wait+0xcd/0xe0 [12828.084737] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12828.084744] do_raw_spin_lock+0x177/0x190 [12828.084748] _raw_spin_lock+0x21/0x30 [12828.084754] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12828.085111] obd_mod_free+0x46/0x140 [ptlrpc] [12828.088608] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [12828.093686] mdc_commit_open+0x91/0xb0 [mdc] [12828.093755] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12828.102837] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) [12828.106547] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12828.118735] obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12828.118788] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12828.118796] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12828.118799] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12828.118814] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12828.118818] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12828.118823] RAX: 0000000000007c6f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12828.118826] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12828.118828] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12828.118830] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12828.118833] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12828.118837] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12828.118840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12828.118842] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12828.118851] Call Trace: [12828.118857] [12828.119541] ? show_regs.cold.9+0x22/0x2f [12828.124016] after_reply+0xe2c/0x1780 [ptlrpc] [12828.124363] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12828.124621] ? __next_timer_interrupt+0x160/0x160 [12828.124631] ptlrpcd+0xa82/0xe40 [ptlrpc] [12828.125569] ? do_wait_intr+0xf0/0xf0 [12828.125579] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12828.125786] kthread+0x1d1/0x200 [12828.125793] ? set_kthread_struct+0x70/0x70 [12828.125797] ret_from_fork+0x1f/0x30 [12828.319367] ? watchdog_timer_fn.cold.6+0x62/0x10a [12828.321121] ? __hrtimer_run_queues+0x1c2/0x4d0 [12828.322720] ? watchdog+0x40/0x40 [12828.323841] ? hrtimer_interrupt+0x128/0x2c0 [12828.325809] ? smp_apic_timer_interrupt+0xa6/0x270 [12828.329544] ? apic_timer_interrupt+0xf/0x20 [12828.331109] [12828.332380] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12828.334572] do_raw_spin_lock+0x177/0x190 [12828.337059] _raw_spin_lock+0x21/0x30 [12828.338508] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12828.342572] ? do_raw_spin_unlock+0x75/0x190 [12828.344456] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12828.346436] ? do_wait_intr+0xf0/0xf0 [12828.349148] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12828.351891] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12828.353787] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12828.357554] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12828.360074] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12828.361529] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12828.363116] mdc_intent_lock+0x353/0x7e0 [mdc] [12828.365263] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12828.368321] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12828.370748] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12828.377413] lmv_intent_open+0x338/0x1150 [lmv] [12828.382116] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12828.384086] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12828.389586] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12828.391734] ll_intent_lock+0xc5/0xaa0 [lustre] [12828.393923] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12828.396479] ll_lookup_it+0xe0e/0x2270 [lustre] [12828.398928] ? do_raw_spin_unlock+0x75/0x190 [12828.400516] ? _raw_spin_unlock+0x12/0x30 [12828.402545] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12828.404402] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12828.406948] ll_atomic_open+0x36a/0xd70 [lustre] [12828.408636] lookup_open+0x356/0xb40 [12828.413630] do_last+0x313/0xfc0 [12828.417951] ? nd_jump_root+0xe5/0x160 [12828.425358] ? path_init+0x437/0x520 [12828.426530] path_openat+0xf7/0x500 [12828.433921] ? do_raw_spin_unlock+0x75/0x190 [12828.438125] do_filp_open+0x99/0x140 [12828.440955] ? getname_flags+0x6e/0x330 [12828.445400] ? __check_object_size+0xff/0x256 [12828.447536] ? do_raw_spin_unlock+0x75/0x190 [12828.449950] ? _raw_spin_unlock+0x12/0x30 [12828.452150] do_sys_openat2+0x2b4/0x410 [12828.454015] do_sys_open+0x73/0xa0 [12828.455439] __x64_sys_openat+0x24/0x30 [12828.457343] do_syscall_64+0xc1/0x440 [12828.458965] entry_SYSCALL_64_after_hwframe+0x49/0xae [12828.461144] RIP: 0033:0x7f53425a5332 [12828.462752] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12828.473331] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12828.478094] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12828.482801] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12828.487583] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12828.495989] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12828.498571] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12844.054851] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [12844.058018] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [12844.070868] Modules linked in: lustre(O) [12844.073575] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) [12844.077449] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) [12844.085283] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12844.085404] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12844.085411] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12844.085414] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12844.085428] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12844.085433] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12844.085437] RAX: 0000000000002134 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12844.085439] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12844.085442] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12844.085443] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12844.085446] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12844.085449] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12844.085452] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12844.085454] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12844.085463] Call Trace: [12844.085469] [12844.085475] ? show_regs.cold.9+0x22/0x2f [12844.085484] ? watchdog_timer_fn.cold.6+0x62/0x10a [12844.085490] ? __hrtimer_run_queues+0x1c2/0x4d0 [12844.085494] ? watchdog+0x40/0x40 [12844.085498] ? hrtimer_interrupt+0x128/0x2c0 [12844.085503] ? smp_apic_timer_interrupt+0xa6/0x270 [12844.085511] ? apic_timer_interrupt+0xf/0x20 [12844.085516] [12844.085519] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12844.085524] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12844.085531] do_raw_spin_lock+0x177/0x190 [12844.085536] _raw_spin_lock+0x21/0x30 [12844.085542] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12844.086014] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12844.086200] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12844.086416] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12844.086767] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12844.086984] ? woken_wake_function+0x30/0x30 [12844.086994] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12844.087192] kthread+0x1d1/0x200 [12844.087199] ? set_kthread_struct+0x70/0x70 [12844.087204] ret_from_fork+0x1f/0x30 [12844.202831] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12844.212401] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12844.217068] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12844.221956] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12844.224505] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12844.226443] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12844.231773] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12844.234879] RAX: 0000000000006ae5 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12844.238213] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12844.241300] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12844.243671] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12844.245885] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12844.248664] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12844.252448] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12844.254873] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12844.257665] Call Trace: [12844.258861] [12844.259649] ? show_regs.cold.9+0x22/0x2f [12844.261859] ? watchdog_timer_fn.cold.6+0x62/0x10a [12844.264073] ? __hrtimer_run_queues+0x1c2/0x4d0 [12844.266029] ? watchdog+0x40/0x40 [12844.267497] ? hrtimer_interrupt+0x128/0x2c0 [12844.269953] ? smp_apic_timer_interrupt+0xa6/0x270 [12844.272307] ? apic_timer_interrupt+0xf/0x20 [12844.274171] [12844.275218] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12844.277965] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12844.280711] ? newidle_balance.isra.93+0x207/0x580 [12844.282850] do_raw_spin_lock+0x177/0x190 [12844.285043] _raw_spin_lock+0x21/0x30 [12844.286971] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12844.289081] process_one_work+0x2c8/0x700 [12844.290095] worker_thread+0x296/0x6e0 [12844.291095] ? rescuer_thread+0x570/0x570 [12844.292261] kthread+0x1d1/0x200 [12844.293764] ? set_kthread_struct+0x70/0x70 [12844.295960] ret_from_fork+0x1f/0x30 [12848.096076] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10211s! [12848.102412] Showing busy workqueues and worker pools: [12848.106568] workqueue events: flags=0x0 [12848.111062] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12848.113662] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12848.120742] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12848.124408] pending: cpu_latency_work [ptlrpc] [12848.126178] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12848.132056] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12848.135713] workqueue events_long: flags=0x0 [12848.137378] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12848.140535] pending: lu_objects_flush [obdclass] [12848.142497] workqueue events_unbound: flags=0x2 [12848.145462] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12848.148159] pending: flush_memcg_stats_dwork, idle_cull_fn [12848.150882] workqueue events_power_efficient: flags=0x80 [12848.153022] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12848.157087] pending: check_lifetime, do_cache_clean [12848.161657] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12848.166852] pending: neigh_periodic_work, neigh_periodic_work [12848.169350] workqueue mm_percpu_wq: flags=0x8 [12848.173074] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12848.176943] pending: vmstat_update, lru_add_drain_per_cpu [12848.180379] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12848.184213] pending: vmstat_update, lru_add_drain_per_cpu [12848.186429] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12848.188792] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12848.193354] workqueue xprtiod: flags=0xa [12848.195467] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12848.197887] pending: xprt_autoclose, xs_error_handle [12848.200240] workqueue ipv6_addrconf: flags=0x40008 [12848.201826] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12848.204088] pending: addrconf_verify_work [12848.205574] workqueue knbd0-recv: flags=0x1a [12848.207190] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12848.210881] in-flight: 651:recv_work [12848.212440] workqueue ptlrpc_pinger: flags=0x2 [12848.214266] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12848.216636] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12848.219250] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10211s workers=4 idle: 15979 31824 8 [12848.225746] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12838s workers=2 idle: 654 [12856.055143] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [12856.057512] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [12856.061296] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12856.061612] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12856.061620] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12856.061623] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12856.061637] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12856.061642] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12856.061646] RAX: 000000000000230f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12856.061649] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12856.061651] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12856.061653] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12856.061656] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12856.061660] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12856.061663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12856.061665] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12856.061674] Call Trace: [12856.061681] [12856.061687] ? show_regs.cold.9+0x22/0x2f [12856.061696] ? watchdog_timer_fn.cold.6+0x62/0x10a [12856.061702] ? __hrtimer_run_queues+0x1c2/0x4d0 [12856.061707] ? watchdog+0x40/0x40 [12856.061711] ? hrtimer_interrupt+0x128/0x2c0 [12856.061716] ? smp_apic_timer_interrupt+0xa6/0x270 [12856.061723] ? apic_timer_interrupt+0xf/0x20 [12856.061728] [12856.061731] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12856.061738] do_raw_spin_lock+0x177/0x190 [12856.061743] _raw_spin_lock+0x21/0x30 [12856.061748] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12856.062030] ? do_raw_spin_unlock+0x75/0x190 [12856.062036] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12856.062246] ? do_wait_intr+0xf0/0xf0 [12856.062255] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12856.076002] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12856.076094] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12856.076102] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12856.076106] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12856.076122] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12856.076126] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12856.076131] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12856.076134] RDX: 0000000000005030 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12856.076136] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12856.076138] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12856.076141] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12856.076144] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12856.076147] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12856.076149] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12856.076160] Call Trace: [12856.076167] [12856.076173] ? show_regs.cold.9+0x22/0x2f [12856.076183] ? watchdog_timer_fn.cold.6+0x62/0x10a [12856.076190] ? __hrtimer_run_queues+0x1c2/0x4d0 [12856.076195] ? watchdog+0x40/0x40 [12856.076200] ? hrtimer_interrupt+0x128/0x2c0 [12856.076204] ? smp_apic_timer_interrupt+0xa6/0x270 [12856.076212] ? apic_timer_interrupt+0xf/0x20 [12856.076218] [12856.076220] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12856.076226] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12856.076231] do_raw_spin_lock+0x177/0x190 [12856.076235] _raw_spin_lock+0x21/0x30 [12856.076240] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12856.076521] obd_mod_free+0x46/0x140 [ptlrpc] [12856.076734] mdc_commit_open+0x91/0xb0 [mdc] [12856.076769] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12856.076968] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12856.077163] after_reply+0xe2c/0x1780 [ptlrpc] [12856.077356] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12856.077562] ? __next_timer_interrupt+0x160/0x160 [12856.077572] ptlrpcd+0xa82/0xe40 [ptlrpc] [12856.077772] ? do_wait_intr+0xf0/0xf0 [12856.077782] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12856.077985] kthread+0x1d1/0x200 [12856.077992] ? set_kthread_struct+0x70/0x70 [12856.077996] ret_from_fork+0x1f/0x30 [12856.448170] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12856.450771] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12856.453871] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12856.457124] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12856.460075] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12856.462504] mdc_intent_lock+0x353/0x7e0 [mdc] [12856.465274] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12856.467905] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12856.472850] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12856.478732] lmv_intent_open+0x338/0x1150 [lmv] [12856.483740] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12856.489634] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12856.494791] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12856.500939] ll_intent_lock+0xc5/0xaa0 [lustre] [12856.503233] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12856.506899] ll_lookup_it+0xe0e/0x2270 [lustre] [12856.513103] ? do_raw_spin_unlock+0x75/0x190 [12856.517501] ? _raw_spin_unlock+0x12/0x30 [12856.521500] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12856.528319] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12856.534030] ll_atomic_open+0x36a/0xd70 [lustre] [12856.539115] lookup_open+0x356/0xb40 [12856.542737] do_last+0x313/0xfc0 [12856.545168] ? nd_jump_root+0xe5/0x160 [12856.547494] ? path_init+0x437/0x520 [12856.548627] path_openat+0xf7/0x500 [12856.554199] ? do_raw_spin_unlock+0x75/0x190 [12856.556247] do_filp_open+0x99/0x140 [12856.560028] ? getname_flags+0x6e/0x330 [12856.563097] ? __check_object_size+0xff/0x256 [12856.568725] ? do_raw_spin_unlock+0x75/0x190 [12856.571853] ? _raw_spin_unlock+0x12/0x30 [12856.575276] do_sys_openat2+0x2b4/0x410 [12856.578960] do_sys_open+0x73/0xa0 [12856.581264] __x64_sys_openat+0x24/0x30 [12856.584292] do_syscall_64+0xc1/0x440 [12856.588483] entry_SYSCALL_64_after_hwframe+0x49/0xae [12856.591289] RIP: 0033:0x7f53425a5332 [12856.595324] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12856.611472] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12856.621592] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12856.629206] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12856.635696] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12856.647238] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12856.649939] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12872.059593] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [12872.071123] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12872.128444] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12872.139801] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12872.145537] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12872.157171] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12872.182084] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12872.190304] RAX: 000000000000728a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12872.195374] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12872.198079] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12872.201070] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12872.204411] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12872.206733] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12872.210203] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12872.213057] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12872.216530] Call Trace: [12872.217635] [12872.218590] ? show_regs.cold.9+0x22/0x2f [12872.220534] ? watchdog_timer_fn.cold.6+0x62/0x10a [12872.222618] ? __hrtimer_run_queues+0x1c2/0x4d0 [12872.224987] ? watchdog+0x40/0x40 [12872.226625] ? hrtimer_interrupt+0x128/0x2c0 [12872.228303] ? smp_apic_timer_interrupt+0xa6/0x270 [12872.230558] ? apic_timer_interrupt+0xf/0x20 [12872.232706] [12872.235517] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12872.240319] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12872.242214] do_raw_spin_lock+0x177/0x190 [12872.244587] _raw_spin_lock+0x21/0x30 [12872.246652] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12872.250787] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12872.260992] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12872.266158] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12872.275645] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12872.278690] ? woken_wake_function+0x30/0x30 [12872.283152] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12872.289832] kthread+0x1d1/0x200 [12872.291588] ? set_kthread_struct+0x70/0x70 [12872.299031] ret_from_fork+0x1f/0x30 [12876.052060] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [12876.058263] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12876.089741] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12876.098929] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12876.109374] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12876.113081] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12876.120706] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12876.146720] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12876.154841] RAX: 00000000000050ed RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12876.157456] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12876.164015] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12876.175599] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12876.183884] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12876.190442] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12876.194728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12876.198596] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12876.207613] Call Trace: [12876.210801] [12876.214508] ? show_regs.cold.9+0x22/0x2f [12876.215931] ? watchdog_timer_fn.cold.6+0x62/0x10a [12876.221159] ? __hrtimer_run_queues+0x1c2/0x4d0 [12876.226502] ? watchdog+0x40/0x40 [12876.229294] ? hrtimer_interrupt+0x128/0x2c0 [12876.233791] ? smp_apic_timer_interrupt+0xa6/0x270 [12876.235994] ? apic_timer_interrupt+0xf/0x20 [12876.238791] [12876.244207] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12876.247495] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12876.251531] ? newidle_balance.isra.93+0x207/0x580 [12876.255978] do_raw_spin_lock+0x177/0x190 [12876.260244] _raw_spin_lock+0x21/0x30 [12876.264534] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12876.267057] process_one_work+0x2c8/0x700 [12876.270403] worker_thread+0x296/0x6e0 [12876.272994] ? rescuer_thread+0x570/0x570 [12876.275294] kthread+0x1d1/0x200 [12876.277295] ? set_kthread_struct+0x70/0x70 [12876.282541] ret_from_fork+0x1f/0x30 [12878.311462] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10241s! [12878.317628] Showing busy workqueues and worker pools: [12878.333848] workqueue events: flags=0x0 [12878.335747] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12878.339622] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12878.345805] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12878.348836] pending: cpu_latency_work [ptlrpc] [12878.351449] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12878.354876] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12878.359258] workqueue events_long: flags=0x0 [12878.362241] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12878.366457] pending: lu_objects_flush [obdclass] [12878.368770] workqueue events_unbound: flags=0x2 [12878.371371] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12878.374701] pending: flush_memcg_stats_dwork, idle_cull_fn [12878.377337] workqueue events_power_efficient: flags=0x80 [12878.380065] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12878.383099] pending: check_lifetime, do_cache_clean [12878.385660] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12878.388912] pending: neigh_periodic_work, neigh_periodic_work [12878.391444] workqueue mm_percpu_wq: flags=0x8 [12878.393680] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12878.397120] pending: vmstat_update, lru_add_drain_per_cpu [12878.399948] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12878.403071] pending: vmstat_update, lru_add_drain_per_cpu [12878.405943] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12878.409229] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12878.412895] workqueue xprtiod: flags=0xa [12878.414842] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12878.417945] pending: xprt_autoclose, xs_error_handle [12878.420973] workqueue ipv6_addrconf: flags=0x40008 [12878.423542] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12878.427447] pending: addrconf_verify_work [12878.429904] workqueue knbd0-recv: flags=0x1a [12878.432507] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12878.436462] in-flight: 651:recv_work [12878.438528] workqueue ptlrpc_pinger: flags=0x2 [12878.440951] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12878.443140] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12878.445620] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10241s workers=4 idle: 15979 31824 8 [12878.448660] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12868s workers=2 idle: 654 [12884.056290] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [12884.057025] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [12884.059400] Modules linked in: [12884.062763] Modules linked in: [12884.062765] lustre(O) osp(O) ofd(O) lod(O) [12884.064319] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12884.064397] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12884.064404] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12884.064407] RIP: 0010:native_safe_halt+0x2e/0x50 [12884.064423] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12884.064428] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12884.064432] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12884.064435] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12884.064437] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12884.064440] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12884.064443] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12884.064446] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12884.064449] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12884.064452] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12884.064461] Call Trace: [12884.064467] [12884.064473] ? show_regs.cold.9+0x22/0x2f [12884.064483] ? watchdog_timer_fn.cold.6+0x62/0x10a [12884.064489] ? __hrtimer_run_queues+0x1c2/0x4d0 [12884.064492] ? watchdog+0x40/0x40 [12884.064496] ? hrtimer_interrupt+0x128/0x2c0 [12884.064500] ? smp_apic_timer_interrupt+0xa6/0x270 [12884.064679] ? apic_timer_interrupt+0xf/0x20 [12884.064685] [12884.064689] ? native_safe_halt+0x2e/0x50 [12884.064693] kvm_wait+0xcd/0xe0 [12884.064701] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12884.064710] do_raw_spin_lock+0x177/0x190 [12884.064714] _raw_spin_lock+0x21/0x30 [12884.064720] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12884.065029] obd_mod_free+0x46/0x140 [ptlrpc] [12884.065379] mdc_commit_open+0x91/0xb0 [mdc] [12884.065418] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12884.065643] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12884.070309] mdt(O) [12884.072363] after_reply+0xe2c/0x1780 [ptlrpc] [12884.110583] mdd(O) [12884.116795] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12884.128266] mgs(O) [12884.130089] ? __next_timer_interrupt+0x160/0x160 [12884.140279] lquota(O) [12884.143439] ptlrpcd+0xa82/0xe40 [ptlrpc] [12884.148215] lfsck(O) [12884.150736] ? do_wait_intr+0xf0/0xf0 [12884.154351] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12884.154412] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12884.154419] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12884.154422] RIP: 0010:native_safe_halt+0x2e/0x50 [12884.154435] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12884.154439] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12884.154444] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12884.154446] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12884.154449] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12884.154451] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12884.154454] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12884.154457] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12884.154461] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12884.154463] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12884.154472] Call Trace: [12884.154478] [12884.154484] ? show_regs.cold.9+0x22/0x2f [12884.154494] ? watchdog_timer_fn.cold.6+0x62/0x10a [12884.154500] ? __hrtimer_run_queues+0x1c2/0x4d0 [12884.154505] ? watchdog+0x40/0x40 [12884.154509] ? hrtimer_interrupt+0x128/0x2c0 [12884.154513] ? smp_apic_timer_interrupt+0xa6/0x270 [12884.154520] ? apic_timer_interrupt+0xf/0x20 [12884.154525] [12884.154527] ? native_safe_halt+0x2e/0x50 [12884.154530] kvm_wait+0xcd/0xe0 [12884.154538] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12884.154547] do_raw_spin_lock+0x177/0x190 [12884.154552] _raw_spin_lock+0x21/0x30 [12884.154557] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12884.158560] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12884.161170] ? do_raw_spin_unlock+0x75/0x190 [12884.161190] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12884.172791] kthread+0x1d1/0x200 [12884.172812] ? set_kthread_struct+0x70/0x70 [12884.172817] ret_from_fork+0x1f/0x30 [12884.437104] ? do_wait_intr+0xf0/0xf0 [12884.438552] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12884.442114] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12884.445147] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12884.449440] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12884.452173] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12884.453948] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12884.455351] mdc_intent_lock+0x353/0x7e0 [mdc] [12884.458286] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12884.459983] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12884.468863] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12884.474382] lmv_intent_open+0x338/0x1150 [lmv] [12884.478163] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12884.480530] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12884.483268] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12884.485422] ll_intent_lock+0xc5/0xaa0 [lustre] [12884.490841] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12884.494420] ll_lookup_it+0xe0e/0x2270 [lustre] [12884.502207] ? do_raw_spin_unlock+0x75/0x190 [12884.503636] ? _raw_spin_unlock+0x12/0x30 [12884.507610] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12884.511038] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12884.519252] ll_atomic_open+0x36a/0xd70 [lustre] [12884.526151] lookup_open+0x356/0xb40 [12884.527569] do_last+0x313/0xfc0 [12884.531918] ? nd_jump_root+0xe5/0x160 [12884.539960] ? path_init+0x437/0x520 [12884.544620] path_openat+0xf7/0x500 [12884.546333] ? do_raw_spin_unlock+0x75/0x190 [12884.547574] do_filp_open+0x99/0x140 [12884.549343] ? getname_flags+0x6e/0x330 [12884.551524] ? __check_object_size+0xff/0x256 [12884.553218] ? do_raw_spin_unlock+0x75/0x190 [12884.554526] ? _raw_spin_unlock+0x12/0x30 [12884.557231] do_sys_openat2+0x2b4/0x410 [12884.558349] do_sys_open+0x73/0xa0 [12884.560503] __x64_sys_openat+0x24/0x30 [12884.563744] do_syscall_64+0xc1/0x440 [12884.571528] entry_SYSCALL_64_after_hwframe+0x49/0xae [12884.574449] RIP: 0033:0x7f53425a5332 [12884.576548] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12884.586735] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12884.590362] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12884.594191] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12884.601478] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12884.607196] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12884.619254] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12900.059306] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [12900.067439] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12900.108177] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12900.114303] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12900.124428] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12900.132059] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12900.159658] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [12900.162415] RAX: 0000000000003e00 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12900.165409] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12900.168407] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12900.171510] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12900.177131] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12900.181926] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12900.186225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12900.189425] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12900.194552] Call Trace: [12900.195979] [12900.199079] ? show_regs.cold.9+0x22/0x2f [12900.202580] ? watchdog_timer_fn.cold.6+0x62/0x10a [12900.205972] ? __hrtimer_run_queues+0x1c2/0x4d0 [12900.208522] ? watchdog+0x40/0x40 [12900.210243] ? hrtimer_interrupt+0x128/0x2c0 [12900.211594] ? smp_apic_timer_interrupt+0xa6/0x270 [12900.215013] ? apic_timer_interrupt+0xf/0x20 [12900.217521] [12900.218936] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12900.223440] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12900.225632] do_raw_spin_lock+0x177/0x190 [12900.232348] _raw_spin_lock+0x21/0x30 [12900.234451] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12900.239212] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12900.242286] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12900.245420] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12900.253089] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12900.257187] ? woken_wake_function+0x30/0x30 [12900.260425] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12900.264588] kthread+0x1d1/0x200 [12900.268012] ? set_kthread_struct+0x70/0x70 [12900.270254] ret_from_fork+0x1f/0x30 [12904.060054] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [12904.073431] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12904.109444] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12904.124668] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12904.131162] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12904.137673] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12904.139679] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12904.158653] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12904.163279] RAX: 0000000000005376 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12904.177337] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12904.185015] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12904.194407] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12904.200247] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12904.209353] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12904.216145] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12904.218495] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12904.221281] Call Trace: [12904.222261] [12904.223084] ? show_regs.cold.9+0x22/0x2f [12904.224592] ? watchdog_timer_fn.cold.6+0x62/0x10a [12904.226488] ? __hrtimer_run_queues+0x1c2/0x4d0 [12904.228186] ? watchdog+0x40/0x40 [12904.229501] ? hrtimer_interrupt+0x128/0x2c0 [12904.231200] ? smp_apic_timer_interrupt+0xa6/0x270 [12904.232892] ? apic_timer_interrupt+0xf/0x20 [12904.234516] [12904.235371] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12904.239490] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12904.242405] ? newidle_balance.isra.93+0x207/0x580 [12904.246297] do_raw_spin_lock+0x177/0x190 [12904.250214] _raw_spin_lock+0x21/0x30 [12904.253067] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12904.261692] process_one_work+0x2c8/0x700 [12904.267441] worker_thread+0x296/0x6e0 [12904.269965] ? rescuer_thread+0x570/0x570 [12904.271694] kthread+0x1d1/0x200 [12904.275716] ? set_kthread_struct+0x70/0x70 [12904.281182] ret_from_fork+0x1f/0x30 [12908.512200] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10271s! [12908.518103] Showing busy workqueues and worker pools: [12908.520704] workqueue events: flags=0x0 [12908.523217] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12908.529467] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12908.539240] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12908.545788] pending: cpu_latency_work [ptlrpc] [12908.552091] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12908.556189] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12908.563511] workqueue events_long: flags=0x0 [12908.564926] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12908.570296] pending: lu_objects_flush [obdclass] [12908.572174] workqueue events_unbound: flags=0x2 [12908.580878] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12908.582340] pending: flush_memcg_stats_dwork, idle_cull_fn [12908.586321] workqueue events_power_efficient: flags=0x80 [12908.588053] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12908.594086] pending: check_lifetime, do_cache_clean [12908.596347] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12908.599304] pending: neigh_periodic_work, neigh_periodic_work [12908.601801] workqueue mm_percpu_wq: flags=0x8 [12908.606926] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12908.610658] pending: vmstat_update, lru_add_drain_per_cpu [12908.613113] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12908.616677] pending: vmstat_update, lru_add_drain_per_cpu [12908.619234] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12908.621712] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12908.624749] workqueue xprtiod: flags=0xa [12908.626690] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12908.629204] pending: xprt_autoclose, xs_error_handle [12908.631093] workqueue ipv6_addrconf: flags=0x40008 [12908.632344] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12908.635092] pending: addrconf_verify_work [12908.636655] workqueue knbd0-recv: flags=0x1a [12908.638377] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12908.640095] in-flight: 651:recv_work [12908.641442] workqueue ptlrpc_pinger: flags=0x2 [12908.642898] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12908.645100] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12908.647143] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10271s workers=4 idle: 15979 31824 8 [12908.649754] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12899s workers=2 idle: 654 [12912.055230] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [12912.057089] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [12912.058071] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) [12912.061089] Modules linked in: [12912.061091] lquota(O) lfsck(O) obdecho(O) [12912.064415] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs [12912.065768] mgc(O) [12912.070274] ata_generic crct10dif_pclmul crc32_pclmul [12912.083873] mdc(O) [12912.086180] crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12912.086209] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12912.086216] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12912.086219] RIP: 0010:native_safe_halt+0x2e/0x50 [12912.086233] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12912.086238] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12912.086242] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12912.086245] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12912.086247] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12912.086249] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12912.086253] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12912.086256] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12912.086259] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12912.086262] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12912.086272] Call Trace: [12912.086279] [12912.086285] ? show_regs.cold.9+0x22/0x2f [12912.086296] ? watchdog_timer_fn.cold.6+0x62/0x10a [12912.086304] ? __hrtimer_run_queues+0x1c2/0x4d0 [12912.086309] ? watchdog+0x40/0x40 [12912.086313] ? hrtimer_interrupt+0x128/0x2c0 [12912.086317] ? smp_apic_timer_interrupt+0xa6/0x270 [12912.086323] ? apic_timer_interrupt+0xf/0x20 [12912.086327] [12912.086330] ? native_safe_halt+0x2e/0x50 [12912.086334] kvm_wait+0xcd/0xe0 [12912.086342] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12912.086350] do_raw_spin_lock+0x177/0x190 [12912.086354] _raw_spin_lock+0x21/0x30 [12912.086361] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12912.095379] lov(O) [12912.105507] obd_mod_free+0x46/0x140 [ptlrpc] [12912.114869] osc(O) [12912.121631] mdc_commit_open+0x91/0xb0 [mdc] [12912.121686] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12912.122181] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12912.122418] after_reply+0xe2c/0x1780 [ptlrpc] [12912.122759] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12912.122958] ? __next_timer_interrupt+0x160/0x160 [12912.123039] ptlrpcd+0xa82/0xe40 [ptlrpc] [12912.123257] ? do_wait_intr+0xf0/0xf0 [12912.123266] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12912.123704] kthread+0x1d1/0x200 [12912.123711] ? set_kthread_struct+0x70/0x70 [12912.123715] ret_from_fork+0x1f/0x30 [12912.249499] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12912.263165] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12912.267669] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12912.270485] RIP: 0010:native_safe_halt+0x2e/0x50 [12912.272192] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [12912.279222] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12912.282538] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12912.286036] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12912.289190] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12912.293145] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12912.297662] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12912.300222] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12912.304908] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12912.306775] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12912.309307] Call Trace: [12912.310375] [12912.311678] ? show_regs.cold.9+0x22/0x2f [12912.313121] ? watchdog_timer_fn.cold.6+0x62/0x10a [12912.315005] ? __hrtimer_run_queues+0x1c2/0x4d0 [12912.317027] ? watchdog+0x40/0x40 [12912.318254] ? hrtimer_interrupt+0x128/0x2c0 [12912.319453] ? smp_apic_timer_interrupt+0xa6/0x270 [12912.321003] ? apic_timer_interrupt+0xf/0x20 [12912.322669] [12912.323846] ? native_safe_halt+0x2e/0x50 [12912.324985] kvm_wait+0xcd/0xe0 [12912.325927] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12912.327740] do_raw_spin_lock+0x177/0x190 [12912.332859] _raw_spin_lock+0x21/0x30 [12912.334166] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12912.338282] ? do_raw_spin_unlock+0x75/0x190 [12912.341404] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12912.347552] ? do_wait_intr+0xf0/0xf0 [12912.349134] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12912.355605] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12912.359042] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12912.364264] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12912.367049] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12912.372109] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12912.373771] mdc_intent_lock+0x353/0x7e0 [mdc] [12912.379341] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12912.381250] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12912.389739] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12912.392705] lmv_intent_open+0x338/0x1150 [lmv] [12912.394287] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12912.398246] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12912.402190] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12912.404699] ll_intent_lock+0xc5/0xaa0 [lustre] [12912.406850] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12912.409974] ll_lookup_it+0xe0e/0x2270 [lustre] [12912.413333] ? do_raw_spin_unlock+0x75/0x190 [12912.415993] ? _raw_spin_unlock+0x12/0x30 [12912.417536] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12912.420487] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12912.428122] ll_atomic_open+0x36a/0xd70 [lustre] [12912.431365] lookup_open+0x356/0xb40 [12912.432764] do_last+0x313/0xfc0 [12912.435606] ? nd_jump_root+0xe5/0x160 [12912.436730] ? path_init+0x437/0x520 [12912.438742] path_openat+0xf7/0x500 [12912.440754] ? do_raw_spin_unlock+0x75/0x190 [12912.443373] do_filp_open+0x99/0x140 [12912.446756] ? getname_flags+0x6e/0x330 [12912.450315] ? __check_object_size+0xff/0x256 [12912.452760] ? do_raw_spin_unlock+0x75/0x190 [12912.454623] ? _raw_spin_unlock+0x12/0x30 [12912.457203] do_sys_openat2+0x2b4/0x410 [12912.459351] do_sys_open+0x73/0xa0 [12912.462432] __x64_sys_openat+0x24/0x30 [12912.464893] do_syscall_64+0xc1/0x440 [12912.466372] entry_SYSCALL_64_after_hwframe+0x49/0xae [12912.469580] RIP: 0033:0x7f53425a5332 [12912.476806] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12912.494231] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12912.500193] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12912.507062] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12912.519726] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12912.523352] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12912.526097] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12928.058163] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [12928.062704] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12928.092715] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12928.100949] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12928.105921] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12928.110745] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12928.125907] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12928.132093] RAX: 0000000000007738 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12928.148511] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12928.154907] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12928.160305] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12928.170083] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12928.172663] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12928.178641] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12928.182797] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12928.189397] Call Trace: [12928.190237] [12928.192510] ? show_regs.cold.9+0x22/0x2f [12928.194853] ? watchdog_timer_fn.cold.6+0x62/0x10a [12928.197938] ? __hrtimer_run_queues+0x1c2/0x4d0 [12928.201059] ? watchdog+0x40/0x40 [12928.202026] ? hrtimer_interrupt+0x128/0x2c0 [12928.205971] ? smp_apic_timer_interrupt+0xa6/0x270 [12928.208590] ? apic_timer_interrupt+0xf/0x20 [12928.209971] [12928.211234] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12928.213087] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12928.215704] do_raw_spin_lock+0x177/0x190 [12928.216777] _raw_spin_lock+0x21/0x30 [12928.217760] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12928.219917] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12928.221688] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12928.223449] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12928.225069] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12928.226920] ? woken_wake_function+0x30/0x30 [12928.228305] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12928.229993] kthread+0x1d1/0x200 [12928.230966] ? set_kthread_struct+0x70/0x70 [12928.232176] ret_from_fork+0x1f/0x30 [12936.055957] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [12936.068143] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12936.121950] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12936.132366] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12936.135823] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12936.141561] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12936.143675] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12936.163695] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12936.166208] RAX: 0000000000000d84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12936.172623] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12936.180183] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12936.191176] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12936.200701] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12936.208799] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12936.213213] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12936.216262] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12936.226916] Call Trace: [12936.227671] [12936.228306] ? show_regs.cold.9+0x22/0x2f [12936.233148] ? watchdog_timer_fn.cold.6+0x62/0x10a [12936.236798] ? __hrtimer_run_queues+0x1c2/0x4d0 [12936.242734] ? watchdog+0x40/0x40 [12936.244193] ? hrtimer_interrupt+0x128/0x2c0 [12936.247765] ? smp_apic_timer_interrupt+0xa6/0x270 [12936.252226] ? apic_timer_interrupt+0xf/0x20 [12936.253923] [12936.257747] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12936.260991] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12936.267179] ? newidle_balance.isra.93+0x207/0x580 [12936.269674] do_raw_spin_lock+0x177/0x190 [12936.272182] _raw_spin_lock+0x21/0x30 [12936.276349] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12936.279791] process_one_work+0x2c8/0x700 [12936.283190] worker_thread+0x296/0x6e0 [12936.285191] ? rescuer_thread+0x570/0x570 [12936.289763] kthread+0x1d1/0x200 [12936.292220] ? set_kthread_struct+0x70/0x70 [12936.295824] ret_from_fork+0x1f/0x30 [12938.720542] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10301s! [12938.726755] Showing busy workqueues and worker pools: [12938.732301] workqueue events: flags=0x0 [12938.737346] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12938.742873] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12938.755809] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12938.761787] pending: cpu_latency_work [ptlrpc] [12938.767728] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12938.770878] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12938.775251] workqueue events_long: flags=0x0 [12938.776843] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12938.779630] pending: lu_objects_flush [obdclass] [12938.781845] workqueue events_unbound: flags=0x2 [12938.784099] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12938.786539] pending: flush_memcg_stats_dwork, idle_cull_fn [12938.788488] workqueue events_power_efficient: flags=0x80 [12938.790735] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12938.793220] pending: check_lifetime, do_cache_clean [12938.795149] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12938.797542] pending: neigh_periodic_work, neigh_periodic_work [12938.799634] workqueue mm_percpu_wq: flags=0x8 [12938.801235] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12938.803378] pending: vmstat_update, lru_add_drain_per_cpu [12938.805261] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12938.807531] pending: vmstat_update, lru_add_drain_per_cpu [12938.809643] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12938.812094] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12938.814492] workqueue xprtiod: flags=0xa [12938.815851] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12938.818299] pending: xprt_autoclose, xs_error_handle [12938.820196] workqueue ipv6_addrconf: flags=0x40008 [12938.821666] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12938.823779] pending: addrconf_verify_work [12938.825327] workqueue knbd0-recv: flags=0x1a [12938.826692] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12938.828592] in-flight: 651:recv_work [12938.830014] workqueue ptlrpc_pinger: flags=0x2 [12938.831654] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12938.834092] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12938.836584] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10301s workers=4 idle: 15979 31824 8 [12938.839718] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12929s workers=2 idle: 654 [12940.055071] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [12940.058694] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12940.060378] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [12940.079866] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12940.079880] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12940.079883] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12940.080347] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12940.080353] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12940.080359] RAX: 0000000000003595 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12940.080361] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12940.080363] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12940.080365] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12940.080367] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12940.080370] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12940.080373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12940.080375] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12940.080383] Call Trace: [12940.080388] [12940.080394] ? show_regs.cold.9+0x22/0x2f [12940.080403] ? watchdog_timer_fn.cold.6+0x62/0x10a [12940.080408] ? __hrtimer_run_queues+0x1c2/0x4d0 [12940.080413] ? watchdog+0x40/0x40 [12940.080417] ? hrtimer_interrupt+0x128/0x2c0 [12940.080420] ? smp_apic_timer_interrupt+0xa6/0x270 [12940.080428] ? apic_timer_interrupt+0xf/0x20 [12940.080566] [12940.080569] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12940.080577] do_raw_spin_lock+0x177/0x190 [12940.080581] _raw_spin_lock+0x21/0x30 [12940.080585] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12940.080858] ? do_raw_spin_unlock+0x75/0x190 [12940.080864] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12940.081385] ? do_wait_intr+0xf0/0xf0 [12940.081396] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12940.081590] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12940.081797] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12940.082400] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12940.082518] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12940.082558] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12940.082589] mdc_intent_lock+0x353/0x7e0 [mdc] [12940.082619] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12940.082703] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12940.083397] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12940.083433] lmv_intent_open+0x338/0x1150 [lmv] [12940.083459] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12940.083562] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12940.083648] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12940.083675] ll_intent_lock+0xc5/0xaa0 [lustre] [12940.083773] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12940.083859] ll_lookup_it+0xe0e/0x2270 [lustre] [12940.084450] ? do_raw_spin_unlock+0x75/0x190 [12940.084459] ? _raw_spin_unlock+0x12/0x30 [12940.084466] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12940.084497] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12940.084518] ll_atomic_open+0x36a/0xd70 [lustre] [12940.084602] lookup_open+0x356/0xb40 [12940.084611] do_last+0x313/0xfc0 [12940.084616] ? nd_jump_root+0xe5/0x160 [12940.084621] ? path_init+0x437/0x520 [12940.084626] path_openat+0xf7/0x500 [12940.084631] ? do_raw_spin_unlock+0x75/0x190 [12940.084636] do_filp_open+0x99/0x140 [12940.084642] ? getname_flags+0x6e/0x330 [12940.084646] ? __check_object_size+0xff/0x256 [12940.084654] ? do_raw_spin_unlock+0x75/0x190 [12940.084658] ? _raw_spin_unlock+0x12/0x30 [12940.084664] do_sys_openat2+0x2b4/0x410 [12940.084668] do_sys_open+0x73/0xa0 [12940.084672] __x64_sys_openat+0x24/0x30 [12940.084675] do_syscall_64+0xc1/0x440 [12940.084682] entry_SYSCALL_64_after_hwframe+0x49/0xae [12940.084688] RIP: 0033:0x7f53425a5332 [12940.084697] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12940.084701] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12940.084706] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12940.084708] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12940.084710] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12940.084712] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12940.084714] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12940.381111] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12940.420544] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12940.428798] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12940.434782] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12940.442502] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12940.456732] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12940.460869] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12940.468970] RDX: 00000000000048d6 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12940.479325] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12940.481274] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12940.485397] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12940.487403] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12940.493972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12940.497122] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12940.503445] Call Trace: [12940.505023] [12940.505827] ? show_regs.cold.9+0x22/0x2f [12940.508667] ? watchdog_timer_fn.cold.6+0x62/0x10a [12940.519522] ? __hrtimer_run_queues+0x1c2/0x4d0 [12940.534133] ? watchdog+0x40/0x40 [12940.535105] ? hrtimer_interrupt+0x128/0x2c0 [12940.536269] ? smp_apic_timer_interrupt+0xa6/0x270 [12940.543965] ? apic_timer_interrupt+0xf/0x20 [12940.545380] [12940.546014] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12940.554490] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12940.556149] do_raw_spin_lock+0x177/0x190 [12940.563818] _raw_spin_lock+0x21/0x30 [12940.567535] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12940.570689] obd_mod_free+0x46/0x140 [ptlrpc] [12940.574485] mdc_commit_open+0x91/0xb0 [mdc] [12940.580260] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12940.585296] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12940.587035] after_reply+0xe2c/0x1780 [ptlrpc] [12940.588467] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12940.600119] ? __next_timer_interrupt+0x160/0x160 [12940.604544] ptlrpcd+0xa82/0xe40 [ptlrpc] [12940.608481] ? do_wait_intr+0xf0/0xf0 [12940.612421] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12940.616642] kthread+0x1d1/0x200 [12940.620137] ? set_kthread_struct+0x70/0x70 [12940.621480] ret_from_fork+0x1f/0x30 [12954.687255] rcu: INFO: rcu_sched self-detected stall on CPU [12954.695028] rcu: 2-....: (6035186 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [12954.705099] (t=10860261 jiffies g=274185 q=66637) [12954.710174] rcu: rcu_sched kthread starved for 10318627 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [12954.720994] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [12954.730906] rcu: RCU grace-period kthread stack dump: [12954.742036] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [12954.750264] Call Trace: [12954.752239] __schedule+0x351/0xcb0 [12954.757318] schedule+0xc0/0x180 [12954.759251] schedule_timeout+0xb4/0x190 [12954.761772] ? do_raw_spin_unlock+0x75/0x190 [12954.773218] ? __next_timer_interrupt+0x160/0x160 [12954.788346] ? prepare_to_swait_event+0xf3/0x140 [12954.798049] rcu_gp_kthread+0x8dd/0x15b0 [12954.801995] ? force_qs_rnp+0x2c0/0x2c0 [12954.809077] kthread+0x1d1/0x200 [12954.811464] ? set_kthread_struct+0x70/0x70 [12954.816777] ret_from_fork+0x1f/0x30 [12954.819953] rcu: Stack dump where RCU GP kthread last ran: [12954.829631] Sending NMI from CPU 2 to CPUs 0: [12954.835517] NMI backtrace for cpu 0 [12954.835525] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12954.835527] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12954.835529] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12954.835532] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12954.835533] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12954.835536] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 [12954.835538] RAX: 000000000000346a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12954.835540] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12954.835541] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12954.835543] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12954.835545] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12954.835546] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12954.835548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12954.835549] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12954.835550] Call Trace: [12954.835551] [12954.835552] ? show_regs.cold.9+0x22/0x2f [12954.835553] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [12954.835554] ? nmi_cpu_backtrace_handler+0x15/0x40 [12954.835556] ? nmi_handle+0xbd/0x210 [12954.835557] ? default_do_nmi+0x4b/0x1b0 [12954.835558] ? do_nmi+0x280/0x380 [12954.835559] ? end_repeat_nmi+0x16/0x1f [12954.835560] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12954.835561] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12954.835562] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12954.835563] [12954.835564] ? newidle_balance.isra.93+0x207/0x580 [12954.835565] do_raw_spin_lock+0x177/0x190 [12954.835566] _raw_spin_lock+0x21/0x30 [12954.835568] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12954.835569] process_one_work+0x2c8/0x700 [12954.835570] worker_thread+0x296/0x6e0 [12954.835572] ? rescuer_thread+0x570/0x570 [12954.835573] kthread+0x1d1/0x200 [12954.835574] ? set_kthread_struct+0x70/0x70 [12954.835575] ret_from_fork+0x1f/0x30 [12954.837114] Sending NMI from CPU 2 to CPUs 1: [12955.009137] NMI backtrace for cpu 1 [12955.009144] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12955.009146] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12955.009148] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12955.009150] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12955.009153] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [12955.009155] RAX: 00000000000017be RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12955.009157] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12955.009159] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12955.009160] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12955.009162] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12955.009163] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12955.009165] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12955.009167] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12955.009168] Call Trace: [12955.009169] [12955.009169] ? show_regs.cold.9+0x22/0x2f [12955.009171] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [12955.009172] ? nmi_cpu_backtrace_handler+0x15/0x40 [12955.009173] ? nmi_handle+0xbd/0x210 [12955.009174] ? default_do_nmi+0x4b/0x1b0 [12955.009175] ? do_nmi+0x280/0x380 [12955.009176] ? end_repeat_nmi+0x16/0x1f [12955.009177] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12955.009179] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12955.009180] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12955.009193] [12955.009195] do_raw_spin_lock+0x177/0x190 [12955.009196] _raw_spin_lock+0x21/0x30 [12955.009197] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12955.009198] ? do_raw_spin_unlock+0x75/0x190 [12955.009200] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12955.009201] ? do_wait_intr+0xf0/0xf0 [12955.009202] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12955.009204] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12955.009205] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12955.009206] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12955.009208] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12955.009210] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12955.009211] mdc_intent_lock+0x353/0x7e0 [mdc] [12955.009212] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12955.009214] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12955.009215] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12955.009217] lmv_intent_open+0x338/0x1150 [lmv] [12955.009218] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12955.009219] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12955.009221] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12955.009222] ll_intent_lock+0xc5/0xaa0 [lustre] [12955.009223] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12955.009225] ll_lookup_it+0xe0e/0x2270 [lustre] [12955.009226] ? do_raw_spin_unlock+0x75/0x190 [12955.009227] ? _raw_spin_unlock+0x12/0x30 [12955.009228] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12955.009230] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12955.009231] ll_atomic_open+0x36a/0xd70 [lustre] [12955.009233] lookup_open+0x356/0xb40 [12955.009234] do_last+0x313/0xfc0 [12955.009235] ? nd_jump_root+0xe5/0x160 [12955.009236] ? path_init+0x437/0x520 [12955.009237] path_openat+0xf7/0x500 [12955.009238] ? do_raw_spin_unlock+0x75/0x190 [12955.009240] do_filp_open+0x99/0x140 [12955.009241] ? getname_flags+0x6e/0x330 [12955.009242] ? __check_object_size+0xff/0x256 [12955.009244] ? do_raw_spin_unlock+0x75/0x190 [12955.009245] ? _raw_spin_unlock+0x12/0x30 [12955.009246] do_sys_openat2+0x2b4/0x410 [12955.009247] do_sys_open+0x73/0xa0 [12955.009248] __x64_sys_openat+0x24/0x30 [12955.009249] do_syscall_64+0xc1/0x440 [12955.009250] entry_SYSCALL_64_after_hwframe+0x49/0xae [12955.009252] RIP: 0033:0x7f53425a5332 [12955.009253] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12955.009256] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12955.009259] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12955.009260] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12955.009262] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12955.009263] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12955.009265] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12955.009271] NMI backtrace for cpu 2 [12955.407179] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12955.415247] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12955.431358] Call Trace: [12955.433238] [12955.434055] ? dump_stack+0xbb/0x10e [12955.436188] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [12955.439519] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [12955.443710] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [12955.449445] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [12955.452434] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [12955.460046] ? rcu_dump_cpu_stacks+0x11b/0x171 [12955.462035] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [12955.467166] ? update_process_times+0x6d/0xc0 [12955.469407] ? tick_sched_handle.isra.18+0x27/0x90 [12955.472019] ? tick_sched_timer+0x53/0xb0 [12955.473971] ? __hrtimer_run_queues+0x1c2/0x4d0 [12955.480475] ? tick_sched_handle.isra.18+0x90/0x90 [12955.487223] ? hrtimer_interrupt+0x128/0x2c0 [12955.490718] ? smp_apic_timer_interrupt+0xa6/0x270 [12955.494045] ? apic_timer_interrupt+0xf/0x20 [12955.496067] [12955.499432] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12955.506195] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12955.509078] ? do_raw_spin_lock+0x177/0x190 [12955.510615] ? _raw_spin_lock+0x21/0x30 [12955.511953] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12955.514156] ? obd_mod_free+0x46/0x140 [ptlrpc] [12955.516205] ? mdc_commit_open+0x91/0xb0 [mdc] [12955.517790] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12955.520010] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12955.522077] ? after_reply+0xe2c/0x1780 [ptlrpc] [12955.524104] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12955.526132] ? __next_timer_interrupt+0x160/0x160 [12955.527961] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [12955.529939] ? do_wait_intr+0xf0/0xf0 [12955.531293] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12955.533333] ? kthread+0x1d1/0x200 [12955.534477] ? set_kthread_struct+0x70/0x70 [12955.536082] ? ret_from_fork+0x1f/0x30 [12956.062852] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [12956.069864] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12956.125187] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12956.134526] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12956.145499] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12956.155516] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12956.175496] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [12956.184896] RAX: 0000000000002487 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12956.190673] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12956.202192] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12956.213838] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12956.222525] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12956.230562] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12956.237685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12956.245277] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12956.256801] Call Trace: [12956.257691] [12956.258294] ? show_regs.cold.9+0x22/0x2f [12956.270518] ? watchdog_timer_fn.cold.6+0x62/0x10a [12956.278297] ? __hrtimer_run_queues+0x1c2/0x4d0 [12956.284133] ? watchdog+0x40/0x40 [12956.289585] ? hrtimer_interrupt+0x128/0x2c0 [12956.300111] ? smp_apic_timer_interrupt+0xa6/0x270 [12956.309403] ? apic_timer_interrupt+0xf/0x20 [12956.310949] [12956.316200] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12956.327019] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12956.332685] do_raw_spin_lock+0x177/0x190 [12956.336951] _raw_spin_lock+0x21/0x30 [12956.342696] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12956.352103] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12956.362876] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12956.373127] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12956.378013] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12956.389116] ? woken_wake_function+0x30/0x30 [12956.390598] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12956.395490] kthread+0x1d1/0x200 [12956.396495] ? set_kthread_struct+0x70/0x70 [12956.403584] ret_from_fork+0x1f/0x30 [12968.935274] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10331s! [12968.939349] Showing busy workqueues and worker pools: [12968.941931] workqueue events: flags=0x0 [12968.943823] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12968.947307] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12968.953596] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12968.957597] pending: cpu_latency_work [ptlrpc] [12968.960934] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12968.964821] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12968.970090] workqueue events_long: flags=0x0 [12968.972207] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12968.976038] pending: lu_objects_flush [obdclass] [12968.979941] workqueue events_unbound: flags=0x2 [12968.982715] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12968.986142] pending: flush_memcg_stats_dwork, idle_cull_fn [12968.989775] workqueue events_power_efficient: flags=0x80 [12968.992735] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12968.996009] pending: check_lifetime, do_cache_clean [12968.998656] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12969.002369] pending: neigh_periodic_work, neigh_periodic_work [12969.005907] workqueue mm_percpu_wq: flags=0x8 [12969.008490] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12969.011115] pending: vmstat_update, lru_add_drain_per_cpu [12969.014075] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12969.017370] pending: vmstat_update, lru_add_drain_per_cpu [12969.020681] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12969.023612] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12969.027759] workqueue xprtiod: flags=0xa [12969.029783] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12969.033768] pending: xprt_autoclose, xs_error_handle [12969.037019] workqueue ipv6_addrconf: flags=0x40008 [12969.039234] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12969.045361] pending: addrconf_verify_work [12969.049034] workqueue knbd0-recv: flags=0x1a [12969.050108] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12969.053264] in-flight: 651:recv_work [12969.058136] workqueue ptlrpc_pinger: flags=0x2 [12969.061638] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12969.064961] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12969.066977] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10331s workers=4 idle: 15979 31824 8 [12969.070157] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12959s workers=2 idle: 654 [12980.061798] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [12980.062430] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [12980.065161] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12980.065327] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12980.065334] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12980.065338] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [12980.065350] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [12980.065354] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12980.065359] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [12980.065361] RDX: 00000000000078a2 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [12980.065363] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [12980.065365] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [12980.065367] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [12980.065371] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [12980.065379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12980.065381] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [12980.065389] Call Trace: [12980.065395] [12980.065401] ? show_regs.cold.9+0x22/0x2f [12980.065412] ? watchdog_timer_fn.cold.6+0x62/0x10a [12980.065419] ? __hrtimer_run_queues+0x1c2/0x4d0 [12980.065423] ? watchdog+0x40/0x40 [12980.065426] ? hrtimer_interrupt+0x128/0x2c0 [12980.065430] ? smp_apic_timer_interrupt+0xa6/0x270 [12980.065437] ? apic_timer_interrupt+0xf/0x20 [12980.065442] [12980.065445] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [12980.065449] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [12980.065454] do_raw_spin_lock+0x177/0x190 [12980.065458] _raw_spin_lock+0x21/0x30 [12980.065464] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [12980.065748] obd_mod_free+0x46/0x140 [ptlrpc] [12980.065964] mdc_commit_open+0x91/0xb0 [mdc] [12980.066001] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [12980.066701] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [12980.066994] after_reply+0xe2c/0x1780 [ptlrpc] [12980.067437] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [12980.067633] ? __next_timer_interrupt+0x160/0x160 [12980.067643] ptlrpcd+0xa82/0xe40 [ptlrpc] [12980.068015] ? do_wait_intr+0xf0/0xf0 [12980.068025] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [12980.068341] kthread+0x1d1/0x200 [12980.068349] ? set_kthread_struct+0x70/0x70 [12980.068352] ret_from_fork+0x1f/0x30 [12980.349938] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12980.384264] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12980.391038] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12980.395804] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12980.400941] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12980.412498] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12980.417491] RAX: 0000000000003985 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12980.419874] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [12980.426843] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [12980.434669] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [12980.440987] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [12980.443345] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [12980.448297] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12980.451431] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [12980.453566] Call Trace: [12980.454400] [12980.455257] ? show_regs.cold.9+0x22/0x2f [12980.456336] ? watchdog_timer_fn.cold.6+0x62/0x10a [12980.458044] ? __hrtimer_run_queues+0x1c2/0x4d0 [12980.460015] ? watchdog+0x40/0x40 [12980.461586] ? hrtimer_interrupt+0x128/0x2c0 [12980.463526] ? smp_apic_timer_interrupt+0xa6/0x270 [12980.464883] ? apic_timer_interrupt+0xf/0x20 [12980.466465] [12980.467002] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12980.468608] do_raw_spin_lock+0x177/0x190 [12980.469916] _raw_spin_lock+0x21/0x30 [12980.470983] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [12980.473667] ? do_raw_spin_unlock+0x75/0x190 [12980.475478] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [12980.483946] ? do_wait_intr+0xf0/0xf0 [12980.488290] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [12980.495973] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [12980.498016] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12980.500948] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12980.506953] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12980.511363] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [12980.514181] mdc_intent_lock+0x353/0x7e0 [mdc] [12980.518120] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12980.520889] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [12980.524103] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [12980.526913] lmv_intent_open+0x338/0x1150 [lmv] [12980.529093] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12980.531686] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12980.534509] lmv_intent_lock+0x1dd/0x4c0 [lmv] [12980.539585] ll_intent_lock+0xc5/0xaa0 [lustre] [12980.546385] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [12980.550846] ll_lookup_it+0xe0e/0x2270 [lustre] [12980.556006] ? do_raw_spin_unlock+0x75/0x190 [12980.564108] ? _raw_spin_unlock+0x12/0x30 [12980.569128] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [12980.573348] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [12980.575972] ll_atomic_open+0x36a/0xd70 [lustre] [12980.579045] lookup_open+0x356/0xb40 [12980.582135] do_last+0x313/0xfc0 [12980.585078] ? nd_jump_root+0xe5/0x160 [12980.587153] ? path_init+0x437/0x520 [12980.594478] path_openat+0xf7/0x500 [12980.597086] ? do_raw_spin_unlock+0x75/0x190 [12980.600666] do_filp_open+0x99/0x140 [12980.602054] ? getname_flags+0x6e/0x330 [12980.604845] ? __check_object_size+0xff/0x256 [12980.606906] ? do_raw_spin_unlock+0x75/0x190 [12980.608867] ? _raw_spin_unlock+0x12/0x30 [12980.613134] do_sys_openat2+0x2b4/0x410 [12980.625431] do_sys_open+0x73/0xa0 [12980.626601] __x64_sys_openat+0x24/0x30 [12980.631928] do_syscall_64+0xc1/0x440 [12980.636154] entry_SYSCALL_64_after_hwframe+0x49/0xae [12980.639464] RIP: 0033:0x7f53425a5332 [12980.643711] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [12980.682267] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [12980.696928] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [12980.698816] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [12980.701296] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [12980.708926] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [12980.713015] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [12984.061951] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [12984.070929] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12984.124329] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12984.139375] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12984.152945] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12984.161358] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12984.188350] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12984.197642] RAX: 0000000000007a85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12984.205759] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [12984.218370] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [12984.233059] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [12984.242482] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [12984.250448] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [12984.256710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12984.268496] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [12984.277119] Call Trace: [12984.282501] [12984.283610] ? show_regs.cold.9+0x22/0x2f [12984.290112] ? watchdog_timer_fn.cold.6+0x62/0x10a [12984.295173] ? __hrtimer_run_queues+0x1c2/0x4d0 [12984.296640] ? watchdog+0x40/0x40 [12984.297727] ? hrtimer_interrupt+0x128/0x2c0 [12984.302531] ? smp_apic_timer_interrupt+0xa6/0x270 [12984.305421] ? apic_timer_interrupt+0xf/0x20 [12984.312366] [12984.312929] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12984.315778] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12984.326811] do_raw_spin_lock+0x177/0x190 [12984.331064] _raw_spin_lock+0x21/0x30 [12984.338961] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [12984.352769] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [12984.355558] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [12984.363174] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [12984.370705] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [12984.385789] ? woken_wake_function+0x30/0x30 [12984.393406] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [12984.402431] kthread+0x1d1/0x200 [12984.405412] ? set_kthread_struct+0x70/0x70 [12984.410737] ret_from_fork+0x1f/0x30 [12996.056258] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [12996.063504] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [12996.104799] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [12996.113377] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [12996.120649] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [12996.129194] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12996.136112] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [12996.150181] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [12996.154039] RAX: 0000000000007c1f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [12996.156168] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [12996.162987] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [12996.166258] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [12996.172444] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [12996.177776] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [12996.179910] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [12996.185635] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [12996.190896] Call Trace: [12996.191942] [12996.192469] ? show_regs.cold.9+0x22/0x2f [12996.194244] ? watchdog_timer_fn.cold.6+0x62/0x10a [12996.201340] ? __hrtimer_run_queues+0x1c2/0x4d0 [12996.205806] ? watchdog+0x40/0x40 [12996.209022] ? hrtimer_interrupt+0x128/0x2c0 [12996.213392] ? smp_apic_timer_interrupt+0xa6/0x270 [12996.217264] ? apic_timer_interrupt+0xf/0x20 [12996.221455] [12996.222462] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [12996.225194] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [12996.237067] ? newidle_balance.isra.93+0x207/0x580 [12996.238346] do_raw_spin_lock+0x177/0x190 [12996.252441] _raw_spin_lock+0x21/0x30 [12996.254737] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [12996.268366] process_one_work+0x2c8/0x700 [12996.271995] worker_thread+0x296/0x6e0 [12996.281829] ? rescuer_thread+0x570/0x570 [12996.288225] kthread+0x1d1/0x200 [12996.291491] ? set_kthread_struct+0x70/0x70 [12996.298952] ret_from_fork+0x1f/0x30 [12999.140751] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10362s! [12999.149103] Showing busy workqueues and worker pools: [12999.157432] workqueue events: flags=0x0 [12999.159671] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [12999.163661] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [12999.170012] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12999.171968] pending: cpu_latency_work [ptlrpc] [12999.173751] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [12999.176050] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [12999.178684] workqueue events_long: flags=0x0 [12999.180385] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [12999.182922] pending: lu_objects_flush [obdclass] [12999.184458] workqueue events_unbound: flags=0x2 [12999.185612] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [12999.187571] pending: flush_memcg_stats_dwork, idle_cull_fn [12999.189636] workqueue events_power_efficient: flags=0x80 [12999.191549] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12999.194087] pending: check_lifetime, do_cache_clean [12999.195705] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12999.198058] pending: neigh_periodic_work, neigh_periodic_work [12999.200129] workqueue mm_percpu_wq: flags=0x8 [12999.201737] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12999.204451] pending: vmstat_update, lru_add_drain_per_cpu [12999.206714] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [12999.209136] pending: vmstat_update, lru_add_drain_per_cpu [12999.211193] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [12999.214023] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [12999.219166] workqueue xprtiod: flags=0xa [12999.221084] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [12999.226813] pending: xprt_autoclose, xs_error_handle [12999.231043] workqueue ipv6_addrconf: flags=0x40008 [12999.233431] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [12999.241517] pending: addrconf_verify_work [12999.245131] workqueue knbd0-recv: flags=0x1a [12999.249021] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [12999.259600] in-flight: 651:recv_work [12999.263111] workqueue ptlrpc_pinger: flags=0x2 [12999.267544] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [12999.278490] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [12999.288671] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10362s workers=4 idle: 15979 31824 8 [12999.298669] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=12989s workers=2 idle: 654 [13008.057011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [13008.059147] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [13008.060262] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13008.060514] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13008.060523] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13008.060526] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13008.060546] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13008.060551] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13008.060556] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13008.060559] RDX: 00000000000028be RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13008.060561] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13008.060564] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13008.060567] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13008.060571] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13008.060574] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13008.060578] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13008.060586] Call Trace: [13008.060593] [13008.060600] ? show_regs.cold.9+0x22/0x2f [13008.060612] ? watchdog_timer_fn.cold.6+0x62/0x10a [13008.060618] ? __hrtimer_run_queues+0x1c2/0x4d0 [13008.060624] ? watchdog+0x40/0x40 [13008.060629] ? hrtimer_interrupt+0x128/0x2c0 [13008.060634] ? smp_apic_timer_interrupt+0xa6/0x270 [13008.060644] ? apic_timer_interrupt+0xf/0x20 [13008.060649] [13008.060652] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13008.060657] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13008.060661] do_raw_spin_lock+0x177/0x190 [13008.060666] _raw_spin_lock+0x21/0x30 [13008.060674] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13008.062320] obd_mod_free+0x46/0x140 [ptlrpc] [13008.068339] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13008.068687] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13008.068694] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13008.068697] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13008.068776] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13008.068781] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13008.068786] RAX: 0000000000007587 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13008.068788] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13008.068790] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13008.068793] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13008.068795] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13008.068800] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13008.068803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13008.068805] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13008.068814] Call Trace: [13008.068822] [13008.068830] ? show_regs.cold.9+0x22/0x2f [13008.068843] ? watchdog_timer_fn.cold.6+0x62/0x10a [13008.068850] ? __hrtimer_run_queues+0x1c2/0x4d0 [13008.068855] ? watchdog+0x40/0x40 [13008.068859] ? hrtimer_interrupt+0x128/0x2c0 [13008.068863] ? smp_apic_timer_interrupt+0xa6/0x270 [13008.068874] ? apic_timer_interrupt+0xf/0x20 [13008.068878] [13008.068882] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13008.068889] do_raw_spin_lock+0x177/0x190 [13008.068894] _raw_spin_lock+0x21/0x30 [13008.068900] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13008.102802] mdc_commit_open+0x91/0xb0 [mdc] [13008.103020] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13008.105110] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13008.108112] after_reply+0xe2c/0x1780 [ptlrpc] [13008.111267] ? do_raw_spin_unlock+0x75/0x190 [13008.111303] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13008.111651] ? do_wait_intr+0xf0/0xf0 [13008.111663] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13008.112080] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13008.112301] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13008.112529] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13008.112659] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13008.128604] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13008.134390] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13008.149372] ? __next_timer_interrupt+0x160/0x160 [13008.151964] mdc_intent_lock+0x353/0x7e0 [mdc] [13008.152038] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13008.152228] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13008.152591] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13008.152626] lmv_intent_open+0x338/0x1150 [lmv] [13008.152660] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13008.152832] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13008.152914] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13008.152935] ll_intent_lock+0xc5/0xaa0 [lustre] [13008.153016] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13008.153116] ll_lookup_it+0xe0e/0x2270 [lustre] [13008.153211] ? do_raw_spin_unlock+0x75/0x190 [13008.153222] ? _raw_spin_unlock+0x12/0x30 [13008.153232] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13008.153270] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13008.153289] ll_atomic_open+0x36a/0xd70 [lustre] [13008.153374] lookup_open+0x356/0xb40 [13008.153385] do_last+0x313/0xfc0 [13008.153390] ? nd_jump_root+0xe5/0x160 [13008.153395] ? path_init+0x437/0x520 [13008.153400] path_openat+0xf7/0x500 [13008.153405] ? do_raw_spin_unlock+0x75/0x190 [13008.153410] do_filp_open+0x99/0x140 [13008.153418] ? getname_flags+0x6e/0x330 [13008.153425] ? __check_object_size+0xff/0x256 [13008.153435] ? do_raw_spin_unlock+0x75/0x190 [13008.153439] ? _raw_spin_unlock+0x12/0x30 [13008.153443] do_sys_openat2+0x2b4/0x410 [13008.153447] do_sys_open+0x73/0xa0 [13008.153451] __x64_sys_openat+0x24/0x30 [13008.153456] do_syscall_64+0xc1/0x440 [13008.153465] entry_SYSCALL_64_after_hwframe+0x49/0xae [13008.153471] RIP: 0033:0x7f53425a5332 [13008.153482] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13008.153486] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13008.153492] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13008.153495] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13008.153498] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13008.153500] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13008.153503] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13008.633632] ptlrpcd+0xa82/0xe40 [ptlrpc] [13008.639016] ? do_wait_intr+0xf0/0xf0 [13008.640424] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13008.643183] kthread+0x1d1/0x200 [13008.645273] ? set_kthread_struct+0x70/0x70 [13008.648972] ret_from_fork+0x1f/0x30 [13012.067186] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13012.088207] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13012.165163] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13012.187451] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13012.198619] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13012.217308] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13012.259177] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13012.275300] RAX: 0000000000007b87 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13012.289571] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13012.302910] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13012.319343] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13012.326394] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13012.337895] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13012.362913] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13012.392192] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13012.394704] Call Trace: [13012.416174] [13012.417389] ? show_regs.cold.9+0x22/0x2f [13012.425960] ? watchdog_timer_fn.cold.6+0x62/0x10a [13012.434472] ? __hrtimer_run_queues+0x1c2/0x4d0 [13012.435932] ? watchdog+0x40/0x40 [13012.455495] ? hrtimer_interrupt+0x128/0x2c0 [13012.463943] ? smp_apic_timer_interrupt+0xa6/0x270 [13012.474959] ? apic_timer_interrupt+0xf/0x20 [13012.481928] [13012.489804] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13012.511303] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13012.516221] do_raw_spin_lock+0x177/0x190 [13012.526494] _raw_spin_lock+0x21/0x30 [13012.533334] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13012.549272] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13012.565867] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13012.580441] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13012.600107] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13012.613425] ? woken_wake_function+0x30/0x30 [13012.618044] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13012.633064] kthread+0x1d1/0x200 [13012.641285] ? set_kthread_struct+0x70/0x70 [13012.642705] ret_from_fork+0x1f/0x30 [13024.053625] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [13024.062131] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13024.102866] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13024.115710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13024.124669] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13024.133771] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13024.141883] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13024.162171] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13024.170287] RAX: 000000000000130f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13024.177836] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13024.188255] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13024.193298] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13024.197092] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13024.203811] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13024.212609] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13024.217058] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13024.224102] Call Trace: [13024.226081] [13024.228356] ? show_regs.cold.9+0x22/0x2f [13024.232683] ? watchdog_timer_fn.cold.6+0x62/0x10a [13024.240569] ? __hrtimer_run_queues+0x1c2/0x4d0 [13024.243912] ? watchdog+0x40/0x40 [13024.248368] ? hrtimer_interrupt+0x128/0x2c0 [13024.252693] ? smp_apic_timer_interrupt+0xa6/0x270 [13024.256642] ? apic_timer_interrupt+0xf/0x20 [13024.260732] [13024.264247] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13024.269983] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13024.274668] ? newidle_balance.isra.93+0x207/0x580 [13024.277338] do_raw_spin_lock+0x177/0x190 [13024.283837] _raw_spin_lock+0x21/0x30 [13024.286412] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13024.292625] process_one_work+0x2c8/0x700 [13024.300059] worker_thread+0x296/0x6e0 [13024.301346] ? rescuer_thread+0x570/0x570 [13024.308047] kthread+0x1d1/0x200 [13024.311707] ? set_kthread_struct+0x70/0x70 [13024.315961] ret_from_fork+0x1f/0x30 [13029.351958] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10392s! [13029.359320] Showing busy workqueues and worker pools: [13029.364455] workqueue events: flags=0x0 [13029.366569] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13029.373409] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13029.388280] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13029.397650] pending: cpu_latency_work [ptlrpc] [13029.404586] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13029.411506] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13029.419630] workqueue events_long: flags=0x0 [13029.422679] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13029.428874] pending: lu_objects_flush [obdclass] [13029.434490] workqueue events_unbound: flags=0x2 [13029.437877] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13029.440705] pending: flush_memcg_stats_dwork, idle_cull_fn [13029.443745] workqueue events_power_efficient: flags=0x80 [13029.449024] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13029.455680] pending: check_lifetime, do_cache_clean [13029.460513] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13029.464258] pending: neigh_periodic_work, neigh_periodic_work [13029.468501] workqueue mm_percpu_wq: flags=0x8 [13029.474705] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13029.479340] pending: vmstat_update, lru_add_drain_per_cpu [13029.491146] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13029.506697] pending: vmstat_update, lru_add_drain_per_cpu [13029.513492] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13029.521798] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13029.532061] workqueue xprtiod: flags=0xa [13029.533495] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13029.540482] pending: xprt_autoclose, xs_error_handle [13029.546128] workqueue ipv6_addrconf: flags=0x40008 [13029.548646] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13029.552061] pending: addrconf_verify_work [13029.554397] workqueue knbd0-recv: flags=0x1a [13029.556579] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13029.559425] in-flight: 651:recv_work [13029.561414] workqueue ptlrpc_pinger: flags=0x2 [13029.563545] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13029.566493] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13029.569898] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10392s workers=4 idle: 15979 31824 8 [13029.573415] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13020s workers=2 idle: 654 [13036.059879] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [13036.061058] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [13036.064927] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13036.065099] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13036.065107] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13036.065110] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13036.065134] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13036.065138] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13036.065143] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13036.065146] RDX: 0000000000002189 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13036.065148] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13036.065150] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13036.065153] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13036.065156] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13036.065159] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13036.065161] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13036.065170] Call Trace: [13036.065185] [13036.065192] ? show_regs.cold.9+0x22/0x2f [13036.065201] ? watchdog_timer_fn.cold.6+0x62/0x10a [13036.065207] ? __hrtimer_run_queues+0x1c2/0x4d0 [13036.070034] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13036.070188] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13036.070195] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13036.070197] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13036.070270] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13036.070274] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13036.070278] RAX: 0000000000005421 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13036.070280] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13036.070283] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13036.070285] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13036.070287] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13036.070290] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13036.070294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13036.070296] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13036.070305] Call Trace: [13036.070310] [13036.070315] ? show_regs.cold.9+0x22/0x2f [13036.070325] ? watchdog_timer_fn.cold.6+0x62/0x10a [13036.070331] ? __hrtimer_run_queues+0x1c2/0x4d0 [13036.070335] ? watchdog+0x40/0x40 [13036.070340] ? hrtimer_interrupt+0x128/0x2c0 [13036.070344] ? smp_apic_timer_interrupt+0xa6/0x270 [13036.070352] ? apic_timer_interrupt+0xf/0x20 [13036.070357] [13036.070361] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13036.070369] do_raw_spin_lock+0x177/0x190 [13036.070373] _raw_spin_lock+0x21/0x30 [13036.070378] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13036.070963] ? do_raw_spin_unlock+0x75/0x190 [13036.070971] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13036.071171] ? do_wait_intr+0xf0/0xf0 [13036.071180] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13036.071398] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13036.072209] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13036.072464] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13036.072595] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13036.072849] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13036.072922] mdc_intent_lock+0x353/0x7e0 [mdc] [13036.072999] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13036.073084] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13036.073404] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13036.074014] lmv_intent_open+0x338/0x1150 [lmv] [13036.074047] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13036.074138] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13036.075431] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13036.075470] ll_intent_lock+0xc5/0xaa0 [lustre] [13036.076185] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13036.122536] ? watchdog+0x40/0x40 [13036.122671] ? hrtimer_interrupt+0x128/0x2c0 [13036.122677] ? smp_apic_timer_interrupt+0xa6/0x270 [13036.122832] ? apic_timer_interrupt+0xf/0x20 [13036.122838] [13036.122843] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13036.122850] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13036.122855] do_raw_spin_lock+0x177/0x190 [13036.122861] _raw_spin_lock+0x21/0x30 [13036.122870] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13036.123167] obd_mod_free+0x46/0x140 [ptlrpc] [13036.124917] mdc_commit_open+0x91/0xb0 [mdc] [13036.124959] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13036.125154] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13036.126571] after_reply+0xe2c/0x1780 [ptlrpc] [13036.126791] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13036.126985] ? __next_timer_interrupt+0x160/0x160 [13036.126996] ptlrpcd+0xa82/0xe40 [ptlrpc] [13036.127191] ? do_wait_intr+0xf0/0xf0 [13036.127200] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13036.129289] kthread+0x1d1/0x200 [13036.129297] ? set_kthread_struct+0x70/0x70 [13036.129301] ret_from_fork+0x1f/0x30 [13036.642735] ll_lookup_it+0xe0e/0x2270 [lustre] [13036.647972] ? do_raw_spin_unlock+0x75/0x190 [13036.652616] ? _raw_spin_unlock+0x12/0x30 [13036.656295] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13036.657908] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13036.663300] ll_atomic_open+0x36a/0xd70 [lustre] [13036.667039] lookup_open+0x356/0xb40 [13036.668923] do_last+0x313/0xfc0 [13036.672297] ? nd_jump_root+0xe5/0x160 [13036.675952] ? path_init+0x437/0x520 [13036.678987] path_openat+0xf7/0x500 [13036.680739] ? do_raw_spin_unlock+0x75/0x190 [13036.682516] do_filp_open+0x99/0x140 [13036.683770] ? getname_flags+0x6e/0x330 [13036.685903] ? __check_object_size+0xff/0x256 [13036.688778] ? do_raw_spin_unlock+0x75/0x190 [13036.691948] ? _raw_spin_unlock+0x12/0x30 [13036.693942] do_sys_openat2+0x2b4/0x410 [13036.695690] do_sys_open+0x73/0xa0 [13036.699005] __x64_sys_openat+0x24/0x30 [13036.702894] do_syscall_64+0xc1/0x440 [13036.704629] entry_SYSCALL_64_after_hwframe+0x49/0xae [13036.708638] RIP: 0033:0x7f53425a5332 [13036.710766] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13036.729306] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13036.736673] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13036.742233] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13036.749676] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13036.755172] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13036.759106] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13040.058070] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13040.061709] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13040.121991] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13040.138072] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13040.145100] RIP: 0010:native_safe_halt+0x2e/0x50 [13040.149749] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13040.165900] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13040.172012] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13040.175080] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13040.185127] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13040.191964] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13040.197254] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13040.203833] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13040.213592] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13040.219094] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13040.227370] Call Trace: [13040.228103] [13040.228643] ? show_regs.cold.9+0x22/0x2f [13040.229581] ? watchdog_timer_fn.cold.6+0x62/0x10a [13040.232954] ? __hrtimer_run_queues+0x1c2/0x4d0 [13040.237570] ? watchdog+0x40/0x40 [13040.241227] ? hrtimer_interrupt+0x128/0x2c0 [13040.242747] ? smp_apic_timer_interrupt+0xa6/0x270 [13040.245515] ? apic_timer_interrupt+0xf/0x20 [13040.248505] [13040.251413] ? native_safe_halt+0x2e/0x50 [13040.255292] kvm_wait+0xcd/0xe0 [13040.256441] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13040.259397] do_raw_spin_lock+0x177/0x190 [13040.262595] _raw_spin_lock+0x21/0x30 [13040.264812] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13040.267491] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13040.271710] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13040.275129] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13040.277723] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13040.280478] ? woken_wake_function+0x30/0x30 [13040.282505] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13040.286710] kthread+0x1d1/0x200 [13040.287940] ? set_kthread_struct+0x70/0x70 [13040.289425] ret_from_fork+0x1f/0x30 [13056.052059] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [13056.055508] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13056.084525] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13056.093665] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13056.101373] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13056.105100] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13056.108396] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13056.130237] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13056.141803] RAX: 00000000000072e3 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13056.144335] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13056.152028] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13056.155912] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13056.162511] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13056.168635] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13056.176173] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13056.181905] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13056.187725] Call Trace: [13056.188817] [13056.190026] ? show_regs.cold.9+0x22/0x2f [13056.191984] ? watchdog_timer_fn.cold.6+0x62/0x10a [13056.194731] ? __hrtimer_run_queues+0x1c2/0x4d0 [13056.197126] ? watchdog+0x40/0x40 [13056.199887] ? hrtimer_interrupt+0x128/0x2c0 [13056.207052] ? smp_apic_timer_interrupt+0xa6/0x270 [13056.210133] ? apic_timer_interrupt+0xf/0x20 [13056.214823] [13056.216469] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13056.219951] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13056.225248] ? newidle_balance.isra.93+0x207/0x580 [13056.228228] do_raw_spin_lock+0x177/0x190 [13056.232198] _raw_spin_lock+0x21/0x30 [13056.233850] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13056.236704] process_one_work+0x2c8/0x700 [13056.240041] worker_thread+0x296/0x6e0 [13056.242384] ? rescuer_thread+0x570/0x570 [13056.244155] kthread+0x1d1/0x200 [13056.245613] ? set_kthread_struct+0x70/0x70 [13056.247673] ret_from_fork+0x1f/0x30 [13060.064242] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10422s! [13060.071610] Showing busy workqueues and worker pools: [13060.080440] workqueue events: flags=0x0 [13060.089510] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13060.098464] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13060.112571] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13060.123517] pending: cpu_latency_work [ptlrpc] [13060.129752] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13060.135036] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13060.146177] workqueue events_long: flags=0x0 [13060.147581] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13060.153578] pending: lu_objects_flush [obdclass] [13060.156147] workqueue events_unbound: flags=0x2 [13060.157385] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13060.161805] pending: flush_memcg_stats_dwork, idle_cull_fn [13060.164008] workqueue events_power_efficient: flags=0x80 [13060.174502] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13060.187449] pending: check_lifetime, do_cache_clean [13060.191068] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13060.197192] pending: neigh_periodic_work, neigh_periodic_work [13060.203469] workqueue mm_percpu_wq: flags=0x8 [13060.206648] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13060.211382] pending: vmstat_update, lru_add_drain_per_cpu [13060.220585] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13060.227561] pending: vmstat_update, lru_add_drain_per_cpu [13060.230310] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13060.235541] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13060.240700] workqueue xprtiod: flags=0xa [13060.247523] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13060.252404] pending: xprt_autoclose, xs_error_handle [13060.256079] workqueue ipv6_addrconf: flags=0x40008 [13060.259711] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13060.268374] pending: addrconf_verify_work [13060.270102] workqueue knbd0-recv: flags=0x1a [13060.275736] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13060.281670] in-flight: 651:recv_work [13060.284886] workqueue ptlrpc_pinger: flags=0x2 [13060.286020] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13060.288088] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13060.291598] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10423s workers=4 idle: 15979 31824 8 [13060.297575] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13050s workers=2 idle: 654 [13064.055014] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [13064.063141] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [13064.068587] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13064.069169] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13064.069176] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13064.069179] RIP: 0010:native_safe_halt+0x2e/0x50 [13064.069192] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13064.069196] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13064.069200] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13064.069202] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13064.069204] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13064.069206] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13064.069208] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13064.069211] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13064.069214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13064.069216] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13064.069607] Call Trace: [13064.069615] [13064.069689] ? show_regs.cold.9+0x22/0x2f [13064.069700] ? watchdog_timer_fn.cold.6+0x62/0x10a [13064.069707] ? __hrtimer_run_queues+0x1c2/0x4d0 [13064.069712] ? watchdog+0x40/0x40 [13064.069716] ? hrtimer_interrupt+0x128/0x2c0 [13064.069736] ? smp_apic_timer_interrupt+0xa6/0x270 [13064.069744] ? apic_timer_interrupt+0xf/0x20 [13064.069748] [13064.069751] ? native_safe_halt+0x2e/0x50 [13064.069755] kvm_wait+0xcd/0xe0 [13064.069763] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13064.069773] do_raw_spin_lock+0x177/0x190 [13064.069777] _raw_spin_lock+0x21/0x30 [13064.069782] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13064.070344] ? do_raw_spin_unlock+0x75/0x190 [13064.070350] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13064.072769] ? do_wait_intr+0xf0/0xf0 [13064.072783] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13064.073192] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13064.085681] Modules linked in: [13064.134981] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13064.153229] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13064.153302] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13064.153309] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13064.153312] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13064.153983] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13064.153988] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13064.153992] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13064.153994] RDX: 0000000000001fe6 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13064.153997] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13064.153998] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13064.154000] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13064.154004] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13064.154006] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13064.154008] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13064.154016] Call Trace: [13064.154022] [13064.154029] ? show_regs.cold.9+0x22/0x2f [13064.154039] ? watchdog_timer_fn.cold.6+0x62/0x10a [13064.154046] ? __hrtimer_run_queues+0x1c2/0x4d0 [13064.154051] ? watchdog+0x40/0x40 [13064.154055] ? hrtimer_interrupt+0x128/0x2c0 [13064.154060] ? smp_apic_timer_interrupt+0xa6/0x270 [13064.154068] ? apic_timer_interrupt+0xf/0x20 [13064.154073] [13064.154076] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13064.154081] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13064.154086] do_raw_spin_lock+0x177/0x190 [13064.154091] _raw_spin_lock+0x21/0x30 [13064.154097] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13064.156984] obd_mod_free+0x46/0x140 [ptlrpc] [13064.168073] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13064.175807] mdc_commit_open+0x91/0xb0 [mdc] [13064.175857] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13064.176186] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13064.176819] after_reply+0xe2c/0x1780 [ptlrpc] [13064.177020] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13064.189947] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13064.189999] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13064.190028] mdc_intent_lock+0x353/0x7e0 [mdc] [13064.197012] ? __next_timer_interrupt+0x160/0x160 [13064.197037] ptlrpcd+0xa82/0xe40 [ptlrpc] [13064.206567] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13064.206846] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13064.207211] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13064.207420] lmv_intent_open+0x338/0x1150 [lmv] [13064.207451] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13064.207546] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13064.207631] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13064.207653] ll_intent_lock+0xc5/0xaa0 [lustre] [13064.207755] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13064.207836] ll_lookup_it+0xe0e/0x2270 [lustre] [13064.207918] ? do_raw_spin_unlock+0x75/0x190 [13064.211937] ? _raw_spin_unlock+0x12/0x30 [13064.211950] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13064.211981] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13064.212003] ll_atomic_open+0x36a/0xd70 [lustre] [13064.212379] lookup_open+0x356/0xb40 [13064.212391] do_last+0x313/0xfc0 [13064.212396] ? nd_jump_root+0xe5/0x160 [13064.212400] ? path_init+0x437/0x520 [13064.212405] path_openat+0xf7/0x500 [13064.212411] ? do_raw_spin_unlock+0x75/0x190 [13064.212417] do_filp_open+0x99/0x140 [13064.212423] ? getname_flags+0x6e/0x330 [13064.212427] ? __check_object_size+0xff/0x256 [13064.212435] ? do_raw_spin_unlock+0x75/0x190 [13064.212440] ? _raw_spin_unlock+0x12/0x30 [13064.212445] do_sys_openat2+0x2b4/0x410 [13064.212450] do_sys_open+0x73/0xa0 [13064.212454] __x64_sys_openat+0x24/0x30 [13064.212458] do_syscall_64+0xc1/0x440 [13064.212465] entry_SYSCALL_64_after_hwframe+0x49/0xae [13064.212471] RIP: 0033:0x7f53425a5332 [13064.212479] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13064.212484] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13064.212489] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13064.212491] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13064.212494] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13064.212497] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13064.212499] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13064.945163] ? do_wait_intr+0xf0/0xf0 [13064.948130] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13064.952338] kthread+0x1d1/0x200 [13064.958016] ? set_kthread_struct+0x70/0x70 [13064.961069] ret_from_fork+0x1f/0x30 [13068.058007] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13068.067077] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13068.104374] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13068.115483] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13068.120060] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13068.126283] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13068.140453] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13068.147505] RAX: 0000000000004b18 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13068.153957] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13068.164243] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13068.170363] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13068.176903] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13068.184219] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13068.189239] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13068.193770] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13068.200849] Call Trace: [13068.202242] [13068.203273] ? show_regs.cold.9+0x22/0x2f [13068.210033] ? watchdog_timer_fn.cold.6+0x62/0x10a [13068.212506] ? __hrtimer_run_queues+0x1c2/0x4d0 [13068.215147] ? watchdog+0x40/0x40 [13068.220436] ? hrtimer_interrupt+0x128/0x2c0 [13068.229350] ? smp_apic_timer_interrupt+0xa6/0x270 [13068.235337] ? apic_timer_interrupt+0xf/0x20 [13068.237771] [13068.238592] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13068.240949] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13068.243619] do_raw_spin_lock+0x177/0x190 [13068.246132] _raw_spin_lock+0x21/0x30 [13068.247322] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13068.250607] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13068.253954] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13068.257461] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13068.260461] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13068.263632] ? woken_wake_function+0x30/0x30 [13068.266404] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13068.270388] kthread+0x1d1/0x200 [13068.271624] ? set_kthread_struct+0x70/0x70 [13068.273053] ret_from_fork+0x1f/0x30 [13088.052211] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [13088.055029] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13088.088096] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13088.098776] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13088.110148] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13088.112112] RIP: 0010:native_safe_halt+0x2e/0x50 [13088.117056] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13088.135128] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13088.140688] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13088.145190] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [13088.149277] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13088.152673] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13088.155766] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13088.158259] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13088.161413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13088.164798] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13088.168854] Call Trace: [13088.170765] [13088.171545] ? show_regs.cold.9+0x22/0x2f [13088.173684] ? watchdog_timer_fn.cold.6+0x62/0x10a [13088.176890] ? __hrtimer_run_queues+0x1c2/0x4d0 [13088.179152] ? watchdog+0x40/0x40 [13088.180273] ? hrtimer_interrupt+0x128/0x2c0 [13088.182438] ? smp_apic_timer_interrupt+0xa6/0x270 [13088.192243] ? apic_timer_interrupt+0xf/0x20 [13088.195736] [13088.198322] ? native_safe_halt+0x2e/0x50 [13088.199947] kvm_wait+0xcd/0xe0 [13088.205414] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13088.207602] ? newidle_balance.isra.93+0x207/0x580 [13088.208831] do_raw_spin_lock+0x177/0x190 [13088.209972] _raw_spin_lock+0x21/0x30 [13088.211178] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13088.213100] process_one_work+0x2c8/0x700 [13088.214749] worker_thread+0x296/0x6e0 [13088.216504] ? rescuer_thread+0x570/0x570 [13088.218254] kthread+0x1d1/0x200 [13088.220079] ? set_kthread_struct+0x70/0x70 [13088.221883] ret_from_fork+0x1f/0x30 [13090.784555] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10453s! [13090.798079] Showing busy workqueues and worker pools: [13090.802017] workqueue events: flags=0x0 [13090.813028] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13090.821632] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13090.830366] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13090.839663] pending: cpu_latency_work [ptlrpc] [13090.843193] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13090.853170] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13090.864551] workqueue events_long: flags=0x0 [13090.867971] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13090.876881] pending: lu_objects_flush [obdclass] [13090.881223] workqueue events_unbound: flags=0x2 [13090.886089] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13090.894198] pending: flush_memcg_stats_dwork, idle_cull_fn [13090.900077] workqueue events_power_efficient: flags=0x80 [13090.905749] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13090.910773] pending: check_lifetime, do_cache_clean [13090.913537] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13090.917656] pending: neigh_periodic_work, neigh_periodic_work [13090.920158] workqueue mm_percpu_wq: flags=0x8 [13090.921413] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13090.924982] pending: vmstat_update, lru_add_drain_per_cpu [13090.929215] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13090.931302] pending: vmstat_update, lru_add_drain_per_cpu [13090.933994] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13090.940860] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13090.945265] workqueue xprtiod: flags=0xa [13090.946509] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13090.949579] pending: xprt_autoclose, xs_error_handle [13090.952210] workqueue ipv6_addrconf: flags=0x40008 [13090.954210] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13090.957073] pending: addrconf_verify_work [13090.959346] workqueue knbd0-recv: flags=0x1a [13090.961384] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13090.964429] in-flight: 651:recv_work [13090.966225] workqueue ptlrpc_pinger: flags=0x2 [13090.968194] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13090.971231] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13090.974925] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10453s workers=4 idle: 15979 31824 8 [13090.978806] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13081s workers=2 idle: 654 [13092.057040] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [13092.057111] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [13092.060039] Modules linked in: [13092.064242] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) [13092.065291] lustre(O) [13092.065294] ksocklnd(O) lnet(O) [13092.071850] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [13092.073349] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13092.073388] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13092.073394] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13092.073397] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13092.073409] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13092.073413] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13092.073417] RAX: 000000000000322e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13092.073419] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13092.073421] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13092.073423] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13092.073425] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13092.073428] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13092.073431] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13092.073433] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13092.073441] Call Trace: [13092.073446] [13092.073451] ? show_regs.cold.9+0x22/0x2f [13092.073461] ? watchdog_timer_fn.cold.6+0x62/0x10a [13092.073467] ? __hrtimer_run_queues+0x1c2/0x4d0 [13092.073472] ? watchdog+0x40/0x40 [13092.073476] ? hrtimer_interrupt+0x128/0x2c0 [13092.073481] ? smp_apic_timer_interrupt+0xa6/0x270 [13092.073488] ? apic_timer_interrupt+0xf/0x20 [13092.074743] obdecho(O) mgc(O) [13092.077404] [13092.087136] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13092.087198] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13092.087206] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13092.087209] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13092.087222] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13092.087285] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13092.087291] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13092.087294] RDX: 0000000000007c1c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13092.087296] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13092.087299] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13092.087301] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13092.087304] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13092.087307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13092.087309] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13092.087320] Call Trace: [13092.087325] [13092.087331] ? show_regs.cold.9+0x22/0x2f [13092.087339] ? watchdog_timer_fn.cold.6+0x62/0x10a [13092.087345] ? __hrtimer_run_queues+0x1c2/0x4d0 [13092.087349] ? watchdog+0x40/0x40 [13092.087353] ? hrtimer_interrupt+0x128/0x2c0 [13092.087358] ? smp_apic_timer_interrupt+0xa6/0x270 [13092.087366] ? apic_timer_interrupt+0xf/0x20 [13092.087370] [13092.087373] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13092.087378] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13092.087382] do_raw_spin_lock+0x177/0x190 [13092.087387] _raw_spin_lock+0x21/0x30 [13092.087393] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13092.087678] obd_mod_free+0x46/0x140 [ptlrpc] [13092.087876] mdc_commit_open+0x91/0xb0 [mdc] [13092.087917] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13092.088108] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13092.088429] after_reply+0xe2c/0x1780 [ptlrpc] [13092.088847] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13092.089040] ? __next_timer_interrupt+0x160/0x160 [13092.089049] ptlrpcd+0xa82/0xe40 [ptlrpc] [13092.099361] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13092.099385] do_raw_spin_lock+0x177/0x190 [13092.099392] _raw_spin_lock+0x21/0x30 [13092.099403] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13092.099719] ? do_raw_spin_unlock+0x75/0x190 [13092.099723] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13092.099890] ? do_wait_intr+0xf0/0xf0 [13092.099896] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13092.100103] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13092.100425] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13092.104506] ? do_wait_intr+0xf0/0xf0 [13092.104534] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13092.104800] kthread+0x1d1/0x200 [13092.104807] ? set_kthread_struct+0x70/0x70 [13092.104811] ret_from_fork+0x1f/0x30 [13092.250548] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13092.252499] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13092.254322] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13092.255817] mdc_intent_lock+0x353/0x7e0 [mdc] [13092.257418] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13092.259593] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13092.262034] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13092.263882] lmv_intent_open+0x338/0x1150 [lmv] [13092.265584] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13092.267683] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13092.269511] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13092.270932] ll_intent_lock+0xc5/0xaa0 [lustre] [13092.276679] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13092.280567] ll_lookup_it+0xe0e/0x2270 [lustre] [13092.286076] ? do_raw_spin_unlock+0x75/0x190 [13092.287687] ? _raw_spin_unlock+0x12/0x30 [13092.290906] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13092.295396] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13092.297855] ll_atomic_open+0x36a/0xd70 [lustre] [13092.301797] lookup_open+0x356/0xb40 [13092.303962] do_last+0x313/0xfc0 [13092.307590] ? nd_jump_root+0xe5/0x160 [13092.310114] ? path_init+0x437/0x520 [13092.315216] path_openat+0xf7/0x500 [13092.320674] ? do_raw_spin_unlock+0x75/0x190 [13092.323214] do_filp_open+0x99/0x140 [13092.327363] ? getname_flags+0x6e/0x330 [13092.329395] ? __check_object_size+0xff/0x256 [13092.332150] ? do_raw_spin_unlock+0x75/0x190 [13092.335326] ? _raw_spin_unlock+0x12/0x30 [13092.336448] do_sys_openat2+0x2b4/0x410 [13092.342373] do_sys_open+0x73/0xa0 [13092.345735] __x64_sys_openat+0x24/0x30 [13092.347172] do_syscall_64+0xc1/0x440 [13092.349094] entry_SYSCALL_64_after_hwframe+0x49/0xae [13092.351262] RIP: 0033:0x7f53425a5332 [13092.353494] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13092.368196] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13092.375124] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13092.381329] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13092.384631] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13092.387729] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13092.391380] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13096.058105] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13096.066678] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13096.121833] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13096.136405] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13096.142153] RIP: 0010:native_safe_halt+0x2e/0x50 [13096.146082] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13096.166083] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13096.170659] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13096.175633] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13096.179088] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13096.182913] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13096.185218] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13096.191039] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13096.194552] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13096.198141] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13096.201429] Call Trace: [13096.202589] [13096.203184] ? show_regs.cold.9+0x22/0x2f [13096.204270] ? watchdog_timer_fn.cold.6+0x62/0x10a [13096.206153] ? __hrtimer_run_queues+0x1c2/0x4d0 [13096.207605] ? watchdog+0x40/0x40 [13096.208704] ? hrtimer_interrupt+0x128/0x2c0 [13096.209985] ? smp_apic_timer_interrupt+0xa6/0x270 [13096.211318] ? apic_timer_interrupt+0xf/0x20 [13096.215568] [13096.216361] ? native_safe_halt+0x2e/0x50 [13096.217618] kvm_wait+0xcd/0xe0 [13096.218597] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13096.221751] do_raw_spin_lock+0x177/0x190 [13096.223730] _raw_spin_lock+0x21/0x30 [13096.225057] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13096.231655] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13096.236942] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13096.241489] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13096.246386] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13096.250715] ? woken_wake_function+0x30/0x30 [13096.257036] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13096.260219] kthread+0x1d1/0x200 [13096.261001] ? set_kthread_struct+0x70/0x70 [13096.263596] ret_from_fork+0x1f/0x30 [13116.052022] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [13116.058034] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13116.090662] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13116.099455] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13116.105002] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13116.109354] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13116.113533] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13116.126007] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13116.133625] RAX: 0000000000000f0a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13116.138662] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13116.144912] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13116.150478] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13116.153939] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13116.158518] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13116.163181] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13116.167630] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13116.171268] Call Trace: [13116.172172] [13116.173305] ? show_regs.cold.9+0x22/0x2f [13116.176547] ? watchdog_timer_fn.cold.6+0x62/0x10a [13116.178920] ? __hrtimer_run_queues+0x1c2/0x4d0 [13116.180680] ? watchdog+0x40/0x40 [13116.182191] ? hrtimer_interrupt+0x128/0x2c0 [13116.184281] ? smp_apic_timer_interrupt+0xa6/0x270 [13116.186186] ? apic_timer_interrupt+0xf/0x20 [13116.187824] [13116.188672] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13116.190654] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13116.192734] ? newidle_balance.isra.93+0x207/0x580 [13116.194817] do_raw_spin_lock+0x177/0x190 [13116.196373] _raw_spin_lock+0x21/0x30 [13116.197633] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13116.200477] process_one_work+0x2c8/0x700 [13116.201956] worker_thread+0x296/0x6e0 [13116.203465] ? rescuer_thread+0x570/0x570 [13116.204713] kthread+0x1d1/0x200 [13116.206061] ? set_kthread_struct+0x70/0x70 [13116.208248] ret_from_fork+0x1f/0x30 [13120.057341] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [13120.058454] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [13120.059624] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13120.059693] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13120.059699] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13120.059702] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13120.059715] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13120.059718] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13120.059723] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13120.059725] RDX: 0000000000002235 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13120.059728] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13120.059731] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13120.059733] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13120.059737] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13120.059751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13120.059753] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13120.059761] Call Trace: [13120.059767] [13120.059772] ? show_regs.cold.9+0x22/0x2f [13120.059781] ? watchdog_timer_fn.cold.6+0x62/0x10a [13120.059789] ? __hrtimer_run_queues+0x1c2/0x4d0 [13120.059793] ? watchdog+0x40/0x40 [13120.059797] ? hrtimer_interrupt+0x128/0x2c0 [13120.059802] ? smp_apic_timer_interrupt+0xa6/0x270 [13120.059809] ? apic_timer_interrupt+0xf/0x20 [13120.059814] [13120.059817] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13120.059822] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13120.059827] do_raw_spin_lock+0x177/0x190 [13120.059832] _raw_spin_lock+0x21/0x30 [13120.059838] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13120.060099] obd_mod_free+0x46/0x140 [ptlrpc] [13120.060305] mdc_commit_open+0x91/0xb0 [mdc] [13120.060341] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13120.060537] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13120.060733] after_reply+0xe2c/0x1780 [ptlrpc] [13120.062214] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13120.062414] ? __next_timer_interrupt+0x160/0x160 [13120.062424] ptlrpcd+0xa82/0xe40 [ptlrpc] [13120.067318] Modules linked in: lustre(O) [13120.096246] ? do_wait_intr+0xf0/0xf0 [13120.096268] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13120.097418] kthread+0x1d1/0x200 [13120.097426] ? set_kthread_struct+0x70/0x70 [13120.097429] ret_from_fork+0x1f/0x30 [13120.360514] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13120.403000] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13120.414977] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13120.420375] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13120.423314] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13120.442465] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13120.448314] RAX: 0000000000007e2c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13120.461970] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13120.476156] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13120.488116] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13120.500561] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13120.513123] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13120.520129] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13120.523757] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13120.528097] Call Trace: [13120.535035] [13120.541523] ? show_regs.cold.9+0x22/0x2f [13120.543193] ? watchdog_timer_fn.cold.6+0x62/0x10a [13120.548249] ? __hrtimer_run_queues+0x1c2/0x4d0 [13120.550756] ? watchdog+0x40/0x40 [13120.551707] ? hrtimer_interrupt+0x128/0x2c0 [13120.555306] ? smp_apic_timer_interrupt+0xa6/0x270 [13120.558442] ? apic_timer_interrupt+0xf/0x20 [13120.562375] [13120.565154] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13120.570306] do_raw_spin_lock+0x177/0x190 [13120.575156] _raw_spin_lock+0x21/0x30 [13120.577017] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13120.582396] ? do_raw_spin_unlock+0x75/0x190 [13120.589899] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13120.591853] ? do_wait_intr+0xf0/0xf0 [13120.596627] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13120.599992] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13120.608175] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13120.613410] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13120.624475] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13120.632807] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13120.644177] mdc_intent_lock+0x353/0x7e0 [mdc] [13120.649506] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13120.658052] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13120.665796] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13120.669167] lmv_intent_open+0x338/0x1150 [lmv] [13120.676581] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13120.681204] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13120.689160] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13120.690819] ll_intent_lock+0xc5/0xaa0 [lustre] [13120.692545] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13120.706429] ll_lookup_it+0xe0e/0x2270 [lustre] [13120.709287] ? do_raw_spin_unlock+0x75/0x190 [13120.711499] ? _raw_spin_unlock+0x12/0x30 [13120.714405] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13120.717635] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13120.720343] ll_atomic_open+0x36a/0xd70 [lustre] [13120.727373] lookup_open+0x356/0xb40 [13120.732396] do_last+0x313/0xfc0 [13120.736454] ? nd_jump_root+0xe5/0x160 [13120.739312] ? path_init+0x437/0x520 [13120.742856] path_openat+0xf7/0x500 [13120.744649] ? do_raw_spin_unlock+0x75/0x190 [13120.746182] do_filp_open+0x99/0x140 [13120.748447] ? getname_flags+0x6e/0x330 [13120.752375] ? __check_object_size+0xff/0x256 [13120.755492] ? do_raw_spin_unlock+0x75/0x190 [13120.759717] ? _raw_spin_unlock+0x12/0x30 [13120.761421] do_sys_openat2+0x2b4/0x410 [13120.765403] do_sys_open+0x73/0xa0 [13120.766629] __x64_sys_openat+0x24/0x30 [13120.768011] do_syscall_64+0xc1/0x440 [13120.772083] entry_SYSCALL_64_after_hwframe+0x49/0xae [13120.780180] RIP: 0033:0x7f53425a5332 [13120.787560] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13120.820923] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13120.832050] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13120.839867] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13120.848524] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13120.872633] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13120.887281] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13121.001111] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10483s! [13121.011125] Showing busy workqueues and worker pools: [13121.014926] workqueue events: flags=0x0 [13121.022634] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13121.034373] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13121.052369] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13121.066495] pending: cpu_latency_work [ptlrpc] [13121.072908] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13121.083913] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13121.094904] workqueue events_long: flags=0x0 [13121.100431] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13121.105984] pending: lu_objects_flush [obdclass] [13121.115134] workqueue events_unbound: flags=0x2 [13121.119934] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13121.126107] pending: flush_memcg_stats_dwork, idle_cull_fn [13121.134447] workqueue events_power_efficient: flags=0x80 [13121.141526] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13121.146229] pending: check_lifetime, do_cache_clean [13121.154435] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13121.163287] pending: neigh_periodic_work, neigh_periodic_work [13121.179427] workqueue mm_percpu_wq: flags=0x8 [13121.184237] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13121.187996] pending: vmstat_update, lru_add_drain_per_cpu [13121.194981] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13121.200135] pending: vmstat_update, lru_add_drain_per_cpu [13121.213469] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13121.222421] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13121.229177] workqueue xprtiod: flags=0xa [13121.231357] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13121.241955] pending: xprt_autoclose, xs_error_handle [13121.246067] workqueue ipv6_addrconf: flags=0x40008 [13121.247703] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13121.254331] pending: addrconf_verify_work [13121.270829] workqueue knbd0-recv: flags=0x1a [13121.272129] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13121.285421] in-flight: 651:recv_work [13121.292992] workqueue ptlrpc_pinger: flags=0x2 [13121.295438] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13121.300813] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13121.308400] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10484s workers=4 idle: 15979 31824 8 [13121.325630] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13111s workers=2 idle: 654 [13124.063192] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13124.080287] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13124.148323] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13124.157363] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13124.160966] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13124.163550] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13124.170563] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13124.173583] RAX: 0000000000003aa4 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13124.188422] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13124.196604] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13124.204490] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13124.210423] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13124.215923] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13124.219957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13124.225856] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13124.233463] Call Trace: [13124.234585] [13124.240187] ? show_regs.cold.9+0x22/0x2f [13124.241440] ? watchdog_timer_fn.cold.6+0x62/0x10a [13124.242957] ? __hrtimer_run_queues+0x1c2/0x4d0 [13124.248952] ? watchdog+0x40/0x40 [13124.254586] ? hrtimer_interrupt+0x128/0x2c0 [13124.258838] ? smp_apic_timer_interrupt+0xa6/0x270 [13124.260557] ? apic_timer_interrupt+0xf/0x20 [13124.269883] [13124.271297] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13124.282764] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13124.289158] do_raw_spin_lock+0x177/0x190 [13124.295959] _raw_spin_lock+0x21/0x30 [13124.300934] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13124.311156] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13124.321031] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13124.331451] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13124.340866] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13124.344901] ? woken_wake_function+0x30/0x30 [13124.350694] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13124.359428] kthread+0x1d1/0x200 [13124.363372] ? set_kthread_struct+0x70/0x70 [13124.371019] ret_from_fork+0x1f/0x30 [13134.696704] rcu: INFO: rcu_sched self-detected stall on CPU [13134.702427] rcu: 2-....: (6130755 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [13134.715153] (t=11040270 jiffies g=274185 q=66736) [13134.717970] rcu: rcu_sched kthread starved for 10498636 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [13134.723297] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [13134.727429] rcu: RCU grace-period kthread stack dump: [13134.729828] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [13134.734400] Call Trace: [13134.735591] __schedule+0x351/0xcb0 [13134.737156] schedule+0xc0/0x180 [13134.738756] schedule_timeout+0xb4/0x190 [13134.740965] ? do_raw_spin_unlock+0x75/0x190 [13134.742976] ? __next_timer_interrupt+0x160/0x160 [13134.745318] ? prepare_to_swait_event+0xf3/0x140 [13134.747434] rcu_gp_kthread+0x8dd/0x15b0 [13134.748857] ? force_qs_rnp+0x2c0/0x2c0 [13134.750392] kthread+0x1d1/0x200 [13134.751379] ? set_kthread_struct+0x70/0x70 [13134.753926] ret_from_fork+0x1f/0x30 [13134.755391] rcu: Stack dump where RCU GP kthread last ran: [13134.758049] Sending NMI from CPU 2 to CPUs 3: [13134.767327] NMI backtrace for cpu 3 [13134.767335] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13134.767337] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13134.767339] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13134.767341] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13134.767344] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 [13134.767346] RAX: 0000000000007f63 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13134.767348] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13134.767349] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13134.767351] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13134.767352] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13134.767353] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13134.767355] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13134.767371] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13134.767372] Call Trace: [13134.767373] [13134.767374] ? show_regs.cold.9+0x22/0x2f [13134.767375] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [13134.767376] ? nmi_cpu_backtrace_handler+0x15/0x40 [13134.767377] ? nmi_handle+0xbd/0x210 [13134.767378] ? default_do_nmi+0x4b/0x1b0 [13134.767379] ? do_nmi+0x280/0x380 [13134.767379] ? end_repeat_nmi+0x16/0x1f [13134.767381] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13134.767382] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13134.767383] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13134.767384] [13134.767385] do_raw_spin_lock+0x177/0x190 [13134.767386] _raw_spin_lock+0x21/0x30 [13134.767515] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13134.767517] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13134.767518] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13134.767520] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13134.767521] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13134.767522] ? woken_wake_function+0x30/0x30 [13134.767523] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13134.767524] kthread+0x1d1/0x200 [13134.767525] ? set_kthread_struct+0x70/0x70 [13134.767526] ret_from_fork+0x1f/0x30 [13134.769443] Sending NMI from CPU 2 to CPUs 1: [13135.000904] NMI backtrace for cpu 1 [13135.000913] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13135.000915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13135.000916] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13135.000918] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13135.000920] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [13135.000923] RAX: 0000000000006693 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13135.000924] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13135.000926] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13135.000927] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13135.000929] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13135.000931] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13135.000932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13135.000934] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13135.000935] Call Trace: [13135.000936] [13135.000937] ? show_regs.cold.9+0x22/0x2f [13135.000938] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [13135.000939] ? nmi_cpu_backtrace_handler+0x15/0x40 [13135.000940] ? nmi_handle+0xbd/0x210 [13135.000941] ? default_do_nmi+0x4b/0x1b0 [13135.000942] ? do_nmi+0x280/0x380 [13135.000943] ? end_repeat_nmi+0x16/0x1f [13135.000945] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13135.000946] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13135.000947] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13135.000948] [13135.000950] do_raw_spin_lock+0x177/0x190 [13135.000951] _raw_spin_lock+0x21/0x30 [13135.000952] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13135.000954] ? do_raw_spin_unlock+0x75/0x190 [13135.000955] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13135.000956] ? do_wait_intr+0xf0/0xf0 [13135.000957] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13135.000958] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13135.000959] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13135.000960] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13135.000962] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13135.000963] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13135.000964] mdc_intent_lock+0x353/0x7e0 [mdc] [13135.000965] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13135.000966] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13135.000967] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13135.000969] lmv_intent_open+0x338/0x1150 [lmv] [13135.000970] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13135.000971] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13135.000972] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13135.000974] ll_intent_lock+0xc5/0xaa0 [lustre] [13135.000975] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13135.000976] ll_lookup_it+0xe0e/0x2270 [lustre] [13135.000978] ? do_raw_spin_unlock+0x75/0x190 [13135.000979] ? _raw_spin_unlock+0x12/0x30 [13135.000980] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13135.000981] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13135.000982] ll_atomic_open+0x36a/0xd70 [lustre] [13135.000984] lookup_open+0x356/0xb40 [13135.000984] do_last+0x313/0xfc0 [13135.000986] ? nd_jump_root+0xe5/0x160 [13135.000986] ? path_init+0x437/0x520 [13135.000988] path_openat+0xf7/0x500 [13135.000989] ? do_raw_spin_unlock+0x75/0x190 [13135.000990] do_filp_open+0x99/0x140 [13135.000991] ? getname_flags+0x6e/0x330 [13135.000992] ? __check_object_size+0xff/0x256 [13135.000993] ? do_raw_spin_unlock+0x75/0x190 [13135.000994] ? _raw_spin_unlock+0x12/0x30 [13135.000995] do_sys_openat2+0x2b4/0x410 [13135.000996] do_sys_open+0x73/0xa0 [13135.000997] __x64_sys_openat+0x24/0x30 [13135.000998] do_syscall_64+0xc1/0x440 [13135.000999] entry_SYSCALL_64_after_hwframe+0x49/0xae [13135.001001] RIP: 0033:0x7f53425a5332 [13135.001002] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13135.001004] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13135.001006] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13135.001008] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13135.001009] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13135.001010] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13135.001011] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13135.003101] NMI backtrace for cpu 2 [13135.304034] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13135.318326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13135.334307] Call Trace: [13135.339370] [13135.340238] ? dump_stack+0xbb/0x10e [13135.342877] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13135.354319] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13135.358188] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13135.364985] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13135.376572] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13135.380583] ? rcu_dump_cpu_stacks+0x11b/0x171 [13135.381805] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [13135.391240] ? update_process_times+0x6d/0xc0 [13135.398716] ? tick_sched_handle.isra.18+0x27/0x90 [13135.403647] ? tick_sched_timer+0x53/0xb0 [13135.407406] ? __hrtimer_run_queues+0x1c2/0x4d0 [13135.415912] ? tick_sched_handle.isra.18+0x90/0x90 [13135.420961] ? hrtimer_interrupt+0x128/0x2c0 [13135.432400] ? smp_apic_timer_interrupt+0xa6/0x270 [13135.441835] ? apic_timer_interrupt+0xf/0x20 [13135.445043] [13135.446158] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13135.451523] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13135.456161] ? do_raw_spin_lock+0x177/0x190 [13135.459893] ? _raw_spin_lock+0x21/0x30 [13135.460921] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13135.464404] ? obd_mod_free+0x46/0x140 [ptlrpc] [13135.467698] ? mdc_commit_open+0x91/0xb0 [mdc] [13135.475336] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13135.480738] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13135.488801] ? after_reply+0xe2c/0x1780 [ptlrpc] [13135.493381] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13135.498986] ? __next_timer_interrupt+0x160/0x160 [13135.500425] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13135.509891] ? do_wait_intr+0xf0/0xf0 [13135.511286] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13135.518387] ? kthread+0x1d1/0x200 [13135.521325] ? set_kthread_struct+0x70/0x70 [13135.525935] ? ret_from_fork+0x1f/0x30 [13148.053817] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [13148.057802] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13148.098034] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13148.109926] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13148.121196] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13148.128692] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13148.131487] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13148.142314] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13148.147470] RAX: 0000000000007d2e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13148.151865] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13148.153700] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13148.155722] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13148.157731] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13148.159810] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13148.162582] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13148.164060] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13148.165930] Call Trace: [13148.166669] [13148.167216] ? show_regs.cold.9+0x22/0x2f [13148.168730] ? watchdog_timer_fn.cold.6+0x62/0x10a [13148.170118] ? __hrtimer_run_queues+0x1c2/0x4d0 [13148.171863] ? watchdog+0x40/0x40 [13148.172806] ? hrtimer_interrupt+0x128/0x2c0 [13148.174277] ? smp_apic_timer_interrupt+0xa6/0x270 [13148.177094] ? apic_timer_interrupt+0xf/0x20 [13148.178932] [13148.179740] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13148.182120] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13148.184834] ? newidle_balance.isra.93+0x207/0x580 [13148.187172] do_raw_spin_lock+0x177/0x190 [13148.188787] _raw_spin_lock+0x21/0x30 [13148.190475] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13148.192918] process_one_work+0x2c8/0x700 [13148.194412] worker_thread+0x296/0x6e0 [13148.196044] ? rescuer_thread+0x570/0x570 [13148.197515] kthread+0x1d1/0x200 [13148.198876] ? set_kthread_struct+0x70/0x70 [13148.200660] ret_from_fork+0x1f/0x30 [13151.711209] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10514s! [13151.715416] Showing busy workqueues and worker pools: [13151.717281] workqueue events: flags=0x0 [13151.718725] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13151.721371] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13151.725804] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13151.728609] pending: cpu_latency_work [ptlrpc] [13151.731026] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13151.733172] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13151.737606] workqueue events_long: flags=0x0 [13151.739328] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13151.742002] pending: lu_objects_flush [obdclass] [13151.744482] workqueue events_unbound: flags=0x2 [13151.746081] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13151.750051] pending: flush_memcg_stats_dwork, idle_cull_fn [13151.753405] workqueue events_power_efficient: flags=0x80 [13151.760457] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13151.767837] pending: check_lifetime, do_cache_clean [13151.771473] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13151.776671] pending: neigh_periodic_work, neigh_periodic_work [13151.778913] workqueue mm_percpu_wq: flags=0x8 [13151.780481] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13151.784736] pending: vmstat_update, lru_add_drain_per_cpu [13151.787432] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13151.792242] pending: vmstat_update, lru_add_drain_per_cpu [13151.795562] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13151.798594] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13151.801524] workqueue xprtiod: flags=0xa [13151.803034] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13151.805224] pending: xprt_autoclose, xs_error_handle [13151.807170] workqueue ipv6_addrconf: flags=0x40008 [13151.809489] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13151.812299] pending: addrconf_verify_work [13151.814367] workqueue knbd0-recv: flags=0x1a [13151.815848] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13151.821908] in-flight: 651:recv_work [13151.824348] workqueue ptlrpc_pinger: flags=0x2 [13151.830580] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13151.836297] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13151.843111] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10514s workers=4 idle: 15979 31824 8 [13151.847100] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13142s workers=2 idle: 654 [13160.057013] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [13160.057795] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [13160.059861] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [13160.060543] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13160.060629] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13160.060635] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13160.060644] Modules linked in: [13160.060638] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13160.060646] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) [13160.060652] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13160.060654] mgs(O) [13160.060655] lquota(O) lfsck(O) [13160.060656] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 [13160.060658] obdecho(O) [13160.060659] ORIG_RAX: ffffffffffffff13 [13160.060660] mgc(O) mdc(O) [13160.060661] RAX: 0000000000006984 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13160.060662] lov(O) osc(O) [13160.060664] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13160.060665] lmv(O) ec(O) [13160.060666] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13160.060667] fid(O) fld(O) [13160.060669] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13160.060670] ptlrpc_gss(O) ptlrpc(O) [13160.060671] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13160.060673] obdclass(O) ksocklnd(O) [13160.060674] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13160.060675] lnet(O) [13160.060677] libcfs(O) [13160.060677] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13160.060678] rpcsec_gss_krb5 auth_rpcgss [13160.060680] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13160.060681] nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 [13160.060687] Call Trace: [13160.060688] pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel [13160.060693] [13160.060693] ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13160.060698] ? show_regs.cold.9+0x22/0x2f [13160.060704] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13160.060708] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13160.060706] ? watchdog_timer_fn.cold.6+0x62/0x10a [13160.060711] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13160.060713] ? __hrtimer_run_queues+0x1c2/0x4d0 [13160.060716] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13160.060717] ? watchdog+0x40/0x40 [13160.060720] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13160.060722] ? hrtimer_interrupt+0x128/0x2c0 [13160.060724] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13160.060726] RDX: 0000000000001074 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13160.060728] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13160.060730] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13160.060726] ? smp_apic_timer_interrupt+0xa6/0x270 [13160.060732] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13160.060735] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13160.060734] ? apic_timer_interrupt+0xf/0x20 [13160.060738] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13160.060739] [13160.060740] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13160.060742] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13160.060746] Call Trace: [13160.060748] [13160.060750] ? show_regs.cold.9+0x22/0x2f [13160.060746] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13160.060755] ? watchdog_timer_fn.cold.6+0x62/0x10a [13160.060758] do_raw_spin_lock+0x177/0x190 [13160.060760] ? __hrtimer_run_queues+0x1c2/0x4d0 [13160.060763] ? watchdog+0x40/0x40 [13160.060763] _raw_spin_lock+0x21/0x30 [13160.060768] ? hrtimer_interrupt+0x128/0x2c0 [13160.060771] ? smp_apic_timer_interrupt+0xa6/0x270 [13160.060775] ? apic_timer_interrupt+0xf/0x20 [13160.060779] [13160.060782] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13160.060786] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13160.060790] do_raw_spin_lock+0x177/0x190 [13160.060794] _raw_spin_lock+0x21/0x30 [13160.060768] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13160.060797] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13160.061193] obd_mod_free+0x46/0x140 [ptlrpc] [13160.061400] mdc_commit_open+0x91/0xb0 [mdc] [13160.061446] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13160.061840] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13160.061855] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13160.062840] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13160.063589] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13160.063809] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13160.064262] ? woken_wake_function+0x30/0x30 [13160.064272] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13160.064490] kthread+0x1d1/0x200 [13160.064495] ? set_kthread_struct+0x70/0x70 [13160.064499] ret_from_fork+0x1f/0x30 [13160.074382] Modules linked in: [13160.081445] after_reply+0xe2c/0x1780 [ptlrpc] [13160.081721] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13160.081912] ? __next_timer_interrupt+0x160/0x160 [13160.081923] ptlrpcd+0xa82/0xe40 [ptlrpc] [13160.082133] ? do_wait_intr+0xf0/0xf0 [13160.082143] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13160.082340] kthread+0x1d1/0x200 [13160.082346] ? set_kthread_struct+0x70/0x70 [13160.082350] ret_from_fork+0x1f/0x30 [13160.605143] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13160.666219] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13160.679201] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13160.687185] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13160.692428] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13160.706286] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13160.714199] RAX: 0000000000007c84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13160.724687] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13160.731808] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13160.736516] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13160.744009] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13160.750930] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13160.757013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13160.760318] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13160.770156] Call Trace: [13160.771056] [13160.773503] ? show_regs.cold.9+0x22/0x2f [13160.777397] ? watchdog_timer_fn.cold.6+0x62/0x10a [13160.781461] ? __hrtimer_run_queues+0x1c2/0x4d0 [13160.783126] ? watchdog+0x40/0x40 [13160.784326] ? hrtimer_interrupt+0x128/0x2c0 [13160.787184] ? smp_apic_timer_interrupt+0xa6/0x270 [13160.789416] ? apic_timer_interrupt+0xf/0x20 [13160.791459] [13160.792367] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13160.795052] do_raw_spin_lock+0x177/0x190 [13160.797206] _raw_spin_lock+0x21/0x30 [13160.798701] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13160.801410] ? do_raw_spin_unlock+0x75/0x190 [13160.803076] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13160.805465] ? do_wait_intr+0xf0/0xf0 [13160.807097] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13160.809171] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13160.811364] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13160.813716] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13160.816254] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13160.818513] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13160.820526] mdc_intent_lock+0x353/0x7e0 [mdc] [13160.822745] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13160.825182] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13160.833574] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13160.841795] lmv_intent_open+0x338/0x1150 [lmv] [13160.848240] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13160.853176] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13160.858516] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13160.861414] ll_intent_lock+0xc5/0xaa0 [lustre] [13160.862805] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13160.869890] ll_lookup_it+0xe0e/0x2270 [lustre] [13160.878616] ? do_raw_spin_unlock+0x75/0x190 [13160.882430] ? _raw_spin_unlock+0x12/0x30 [13160.888426] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13160.895956] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13160.902662] ll_atomic_open+0x36a/0xd70 [lustre] [13160.906302] lookup_open+0x356/0xb40 [13160.907627] do_last+0x313/0xfc0 [13160.910959] ? nd_jump_root+0xe5/0x160 [13160.913469] ? path_init+0x437/0x520 [13160.917378] path_openat+0xf7/0x500 [13160.918648] ? do_raw_spin_unlock+0x75/0x190 [13160.925798] do_filp_open+0x99/0x140 [13160.927018] ? getname_flags+0x6e/0x330 [13160.928017] ? __check_object_size+0xff/0x256 [13160.934213] ? do_raw_spin_unlock+0x75/0x190 [13160.938891] ? _raw_spin_unlock+0x12/0x30 [13160.942043] do_sys_openat2+0x2b4/0x410 [13160.947748] do_sys_open+0x73/0xa0 [13160.950107] __x64_sys_openat+0x24/0x30 [13160.951276] do_syscall_64+0xc1/0x440 [13160.954778] entry_SYSCALL_64_after_hwframe+0x49/0xae [13160.957777] RIP: 0033:0x7f53425a5332 [13160.964651] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13160.985316] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13160.992009] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13160.996582] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13161.003413] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13161.006909] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13161.010774] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13176.055349] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [13176.064999] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13176.131885] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13176.140127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13176.149097] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13176.154691] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13176.160352] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13176.186968] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13176.193001] RAX: 000000000000633e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13176.199497] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13176.203427] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13176.205748] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13176.209671] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13176.219584] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13176.225684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13176.230602] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13176.241075] Call Trace: [13176.243811] [13176.246912] ? show_regs.cold.9+0x22/0x2f [13176.252120] ? watchdog_timer_fn.cold.6+0x62/0x10a [13176.258750] ? __hrtimer_run_queues+0x1c2/0x4d0 [13176.265879] ? watchdog+0x40/0x40 [13176.269073] ? hrtimer_interrupt+0x128/0x2c0 [13176.276644] ? smp_apic_timer_interrupt+0xa6/0x270 [13176.283027] ? apic_timer_interrupt+0xf/0x20 [13176.289617] [13176.292982] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13176.294510] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13176.301375] ? newidle_balance.isra.93+0x207/0x580 [13176.305076] do_raw_spin_lock+0x177/0x190 [13176.306708] _raw_spin_lock+0x21/0x30 [13176.314900] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13176.319655] process_one_work+0x2c8/0x700 [13176.324992] worker_thread+0x296/0x6e0 [13176.329962] ? rescuer_thread+0x570/0x570 [13176.337114] kthread+0x1d1/0x200 [13176.342920] ? set_kthread_struct+0x70/0x70 [13176.349184] ret_from_fork+0x1f/0x30 [13181.927015] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10544s! [13181.930021] Showing busy workqueues and worker pools: [13181.934014] workqueue events: flags=0x0 [13181.935598] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13181.944358] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13181.947235] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13181.950362] pending: cpu_latency_work [ptlrpc] [13181.953475] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13181.956861] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13181.964186] workqueue events_long: flags=0x0 [13181.968127] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13181.970434] pending: lu_objects_flush [obdclass] [13181.973948] workqueue events_unbound: flags=0x2 [13181.976136] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13181.979625] pending: flush_memcg_stats_dwork, idle_cull_fn [13181.981841] workqueue events_power_efficient: flags=0x80 [13181.984042] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13181.987226] pending: check_lifetime, do_cache_clean [13181.989826] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13181.992817] pending: neigh_periodic_work, neigh_periodic_work [13181.995027] workqueue mm_percpu_wq: flags=0x8 [13181.996644] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13181.999555] pending: vmstat_update, lru_add_drain_per_cpu [13182.001416] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13182.003544] pending: vmstat_update, lru_add_drain_per_cpu [13182.005789] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13182.008239] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13182.014236] workqueue xprtiod: flags=0xa [13182.018857] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13182.024740] pending: xprt_autoclose, xs_error_handle [13182.028069] workqueue ipv6_addrconf: flags=0x40008 [13182.030095] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13182.033632] pending: addrconf_verify_work [13182.036623] workqueue knbd0-recv: flags=0x1a [13182.039551] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13182.044518] in-flight: 651:recv_work [13182.046809] workqueue ptlrpc_pinger: flags=0x2 [13182.049149] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13182.052224] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13182.060042] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10544s workers=4 idle: 15979 31824 8 [13182.066038] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13172s workers=2 idle: 654 [13188.057025] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [13188.058237] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [13188.060065] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13188.060147] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13188.060154] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13188.060157] RIP: 0010:native_safe_halt+0x2e/0x50 [13188.060227] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13188.060232] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13188.060237] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13188.060240] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13188.060242] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13188.060245] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13188.060247] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13188.060250] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13188.060253] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13188.060256] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13188.060264] Call Trace: [13188.060381] [13188.063100] Modules linked in: [13188.069887] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [13188.069902] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13188.069968] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13188.069973] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13188.069976] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13188.069989] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13188.069994] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13188.069999] RAX: 0000000000006590 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13188.070001] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13188.070003] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13188.070005] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13188.070008] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13188.070011] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13188.070014] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13188.070016] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13188.070025] Call Trace: [13188.070031] [13188.070036] ? show_regs.cold.9+0x22/0x2f [13188.070045] ? watchdog_timer_fn.cold.6+0x62/0x10a [13188.070051] ? __hrtimer_run_queues+0x1c2/0x4d0 [13188.070055] ? watchdog+0x40/0x40 [13188.070060] ? hrtimer_interrupt+0x128/0x2c0 [13188.070063] ? smp_apic_timer_interrupt+0xa6/0x270 [13188.070071] ? apic_timer_interrupt+0xf/0x20 [13188.070076] [13188.070080] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13188.070087] do_raw_spin_lock+0x177/0x190 [13188.070092] _raw_spin_lock+0x21/0x30 [13188.070098] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13188.073925] ? do_raw_spin_unlock+0x75/0x190 [13188.073935] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13188.074153] ? do_wait_intr+0xf0/0xf0 [13188.074162] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13188.074470] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13188.074699] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13188.074929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13188.075059] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13188.075105] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13188.075138] mdc_intent_lock+0x353/0x7e0 [mdc] [13188.075241] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13188.075335] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13188.075583] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13188.075617] lmv_intent_open+0x338/0x1150 [lmv] [13188.075652] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13188.075747] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13188.075840] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13188.075865] ll_intent_lock+0xc5/0xaa0 [lustre] [13188.075958] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13188.076047] ll_lookup_it+0xe0e/0x2270 [lustre] [13188.076137] ? do_raw_spin_unlock+0x75/0x190 [13188.076144] ? _raw_spin_unlock+0x12/0x30 [13188.076152] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13188.077362] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13188.077388] ll_atomic_open+0x36a/0xd70 [lustre] [13188.077874] lookup_open+0x356/0xb40 [13188.077885] do_last+0x313/0xfc0 [13188.077890] ? nd_jump_root+0xe5/0x160 [13188.077895] ? path_init+0x437/0x520 [13188.077900] path_openat+0xf7/0x500 [13188.077905] ? do_raw_spin_unlock+0x75/0x190 [13188.077911] do_filp_open+0x99/0x140 [13188.077918] ? getname_flags+0x6e/0x330 [13188.077922] ? __check_object_size+0xff/0x256 [13188.077929] ? do_raw_spin_unlock+0x75/0x190 [13188.077933] ? _raw_spin_unlock+0x12/0x30 [13188.077939] do_sys_openat2+0x2b4/0x410 [13188.077944] do_sys_open+0x73/0xa0 [13188.077948] __x64_sys_openat+0x24/0x30 [13188.077952] do_syscall_64+0xc1/0x440 [13188.077958] entry_SYSCALL_64_after_hwframe+0x49/0xae [13188.077964] RIP: 0033:0x7f53425a5332 [13188.077972] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13188.077976] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13188.077982] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13188.077984] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13188.077986] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13188.077988] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13188.077991] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13188.082720] ? show_regs.cold.9+0x22/0x2f [13188.082744] ? watchdog_timer_fn.cold.6+0x62/0x10a [13188.082750] ? __hrtimer_run_queues+0x1c2/0x4d0 [13188.082755] ? watchdog+0x40/0x40 [13188.082759] ? hrtimer_interrupt+0x128/0x2c0 [13188.082763] ? smp_apic_timer_interrupt+0xa6/0x270 [13188.082771] ? apic_timer_interrupt+0xf/0x20 [13188.082775] [13188.082780] ? native_safe_halt+0x2e/0x50 [13188.082784] kvm_wait+0xcd/0xe0 [13188.082792] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13188.082799] do_raw_spin_lock+0x177/0x190 [13188.082804] _raw_spin_lock+0x21/0x30 [13188.082811] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13188.108339] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13188.108409] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13188.108416] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13188.108419] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13188.108433] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13188.108436] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13188.108440] RAX: 0000000000007407 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13188.108443] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13188.108445] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13188.108447] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13188.108449] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13188.108452] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13188.108455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13188.108456] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13188.108466] Call Trace: [13188.115186] obd_mod_free+0x46/0x140 [ptlrpc] [13188.120451] [13188.129212] mdc_commit_open+0x91/0xb0 [mdc] [13188.133024] ? show_regs.cold.9+0x22/0x2f [13188.135861] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13188.138734] ? watchdog_timer_fn.cold.6+0x62/0x10a [13188.141355] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13188.144563] ? __hrtimer_run_queues+0x1c2/0x4d0 [13188.147439] after_reply+0xe2c/0x1780 [ptlrpc] [13188.151141] ? watchdog+0x40/0x40 [13188.153703] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13188.153982] ? __next_timer_interrupt+0x160/0x160 [13188.153994] ptlrpcd+0xa82/0xe40 [ptlrpc] [13188.156293] ? hrtimer_interrupt+0x128/0x2c0 [13188.156307] ? smp_apic_timer_interrupt+0xa6/0x270 [13188.156316] ? apic_timer_interrupt+0xf/0x20 [13188.156320] [13188.156324] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13188.156331] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13188.156338] do_raw_spin_lock+0x177/0x190 [13188.157579] ? do_wait_intr+0xf0/0xf0 [13188.157594] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13188.157856] kthread+0x1d1/0x200 [13188.157863] ? set_kthread_struct+0x70/0x70 [13188.157866] ret_from_fork+0x1f/0x30 [13188.884230] _raw_spin_lock+0x21/0x30 [13188.889443] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13188.895289] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13188.900885] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13188.905007] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13188.908485] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13188.911757] ? woken_wake_function+0x30/0x30 [13188.914971] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13188.919613] kthread+0x1d1/0x200 [13188.924370] ? set_kthread_struct+0x70/0x70 [13188.927435] ret_from_fork+0x1f/0x30 [13208.053001] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [13208.058751] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13208.089834] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13208.098860] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13208.103465] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13208.107095] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13208.110114] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13208.121584] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13208.126177] RAX: 0000000000007e1a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13208.133531] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13208.142634] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13208.147465] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13208.156779] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13208.162986] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13208.170948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13208.176918] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13208.191049] Call Trace: [13208.194021] [13208.194704] ? show_regs.cold.9+0x22/0x2f [13208.199064] ? watchdog_timer_fn.cold.6+0x62/0x10a [13208.204570] ? __hrtimer_run_queues+0x1c2/0x4d0 [13208.208973] ? watchdog+0x40/0x40 [13208.213253] ? hrtimer_interrupt+0x128/0x2c0 [13208.221135] ? smp_apic_timer_interrupt+0xa6/0x270 [13208.223715] ? apic_timer_interrupt+0xf/0x20 [13208.225505] [13208.229107] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13208.230718] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13208.234714] ? newidle_balance.isra.93+0x207/0x580 [13208.239707] do_raw_spin_lock+0x177/0x190 [13208.241518] _raw_spin_lock+0x21/0x30 [13208.242984] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13208.249932] process_one_work+0x2c8/0x700 [13208.255165] worker_thread+0x296/0x6e0 [13208.256135] ? rescuer_thread+0x570/0x570 [13208.257157] kthread+0x1d1/0x200 [13208.264570] ? set_kthread_struct+0x70/0x70 [13208.267559] ret_from_fork+0x1f/0x30 [13212.129072] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10575s! [13212.134274] Showing busy workqueues and worker pools: [13212.141257] workqueue events: flags=0x0 [13212.143033] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13212.150962] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13212.161803] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13212.165028] pending: cpu_latency_work [ptlrpc] [13212.167514] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13212.170597] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13212.174574] workqueue events_long: flags=0x0 [13212.176633] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13212.179884] pending: lu_objects_flush [obdclass] [13212.182982] workqueue events_unbound: flags=0x2 [13212.185374] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13212.188630] pending: flush_memcg_stats_dwork, idle_cull_fn [13212.191689] workqueue events_power_efficient: flags=0x80 [13212.194445] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13212.197769] pending: check_lifetime, do_cache_clean [13212.199963] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13212.203300] pending: neigh_periodic_work, neigh_periodic_work [13212.206917] workqueue mm_percpu_wq: flags=0x8 [13212.208812] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13212.212034] pending: vmstat_update, lru_add_drain_per_cpu [13212.214808] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13212.217781] pending: vmstat_update, lru_add_drain_per_cpu [13212.220388] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13212.223365] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13212.226480] workqueue xprtiod: flags=0xa [13212.229275] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13212.233228] pending: xprt_autoclose, xs_error_handle [13212.236175] workqueue ipv6_addrconf: flags=0x40008 [13212.238662] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13212.241514] pending: addrconf_verify_work [13212.243824] workqueue knbd0-recv: flags=0x1a [13212.246519] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13212.249107] in-flight: 651:recv_work [13212.251460] workqueue ptlrpc_pinger: flags=0x2 [13212.254757] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13212.258660] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13212.261828] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10575s workers=4 idle: 15979 31824 8 [13212.265239] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13202s workers=2 idle: 654 [13216.057689] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [13216.058567] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [13216.059042] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13216.059050] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13216.059179] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13216.059186] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13216.059189] RIP: 0010:native_safe_halt+0x2e/0x50 [13216.059202] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13216.059206] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13216.059211] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13216.059213] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13216.059216] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13216.059218] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13216.059220] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13216.059277] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13216.059280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13216.059282] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13216.059290] Call Trace: [13216.059296] [13216.059302] ? show_regs.cold.9+0x22/0x2f [13216.059312] ? watchdog_timer_fn.cold.6+0x62/0x10a [13216.059318] ? __hrtimer_run_queues+0x1c2/0x4d0 [13216.059322] ? watchdog+0x40/0x40 [13216.059326] ? hrtimer_interrupt+0x128/0x2c0 [13216.059330] ? smp_apic_timer_interrupt+0xa6/0x270 [13216.059336] ? apic_timer_interrupt+0xf/0x20 [13216.059341] [13216.059354] ? native_safe_halt+0x2e/0x50 [13216.059359] kvm_wait+0xcd/0xe0 [13216.059366] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13216.059375] do_raw_spin_lock+0x177/0x190 [13216.059379] _raw_spin_lock+0x21/0x30 [13216.059628] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13216.059883] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13216.060155] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13216.060379] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13216.060598] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13216.060816] ? woken_wake_function+0x30/0x30 [13216.060825] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13216.061033] kthread+0x1d1/0x200 [13216.061039] ? set_kthread_struct+0x70/0x70 [13216.061043] ret_from_fork+0x1f/0x30 [13216.061162] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13216.061229] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13216.061236] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13216.061239] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13216.061249] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13216.061253] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13216.061257] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13216.061259] RDX: 0000000000007f81 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13216.061262] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13216.061264] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13216.061266] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13216.061269] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13216.061272] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13216.061274] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13216.061282] Call Trace: [13216.061286] [13216.061290] ? show_regs.cold.9+0x22/0x2f [13216.061300] ? watchdog_timer_fn.cold.6+0x62/0x10a [13216.061306] ? __hrtimer_run_queues+0x1c2/0x4d0 [13216.061310] ? watchdog+0x40/0x40 [13216.061314] ? hrtimer_interrupt+0x128/0x2c0 [13216.061317] ? smp_apic_timer_interrupt+0xa6/0x270 [13216.061322] ? apic_timer_interrupt+0xf/0x20 [13216.061326] [13216.061329] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13216.061334] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13216.061338] do_raw_spin_lock+0x177/0x190 [13216.061342] _raw_spin_lock+0x21/0x30 [13216.061348] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13216.061549] obd_mod_free+0x46/0x140 [ptlrpc] [13216.073322] Modules linked in: [13216.078345] mdc_commit_open+0x91/0xb0 [mdc] [13216.078397] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13216.078678] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13216.078879] after_reply+0xe2c/0x1780 [ptlrpc] [13216.080517] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13216.080720] ? __next_timer_interrupt+0x160/0x160 [13216.080731] ptlrpcd+0xa82/0xe40 [ptlrpc] [13216.107204] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13216.107382] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13216.107389] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13216.107392] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13216.107405] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13216.107408] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13216.107412] RAX: 0000000000007f0a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13216.107415] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13216.107417] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13216.107419] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13216.107421] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13216.107424] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13216.107427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13216.107429] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13216.107437] Call Trace: [13216.107443] [13216.107448] ? show_regs.cold.9+0x22/0x2f [13216.107457] ? watchdog_timer_fn.cold.6+0x62/0x10a [13216.107463] ? __hrtimer_run_queues+0x1c2/0x4d0 [13216.107467] ? watchdog+0x40/0x40 [13216.107471] ? hrtimer_interrupt+0x128/0x2c0 [13216.107475] ? smp_apic_timer_interrupt+0xa6/0x270 [13216.107482] ? apic_timer_interrupt+0xf/0x20 [13216.107487] [13216.107490] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13216.107496] do_raw_spin_lock+0x177/0x190 [13216.107501] _raw_spin_lock+0x21/0x30 [13216.107505] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13216.107792] ? do_raw_spin_unlock+0x75/0x190 [13216.107797] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13216.108316] ? do_wait_intr+0xf0/0xf0 [13216.108326] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13216.108534] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13216.108812] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13216.114470] ? do_wait_intr+0xf0/0xf0 [13216.114493] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13216.114826] kthread+0x1d1/0x200 [13216.114833] ? set_kthread_struct+0x70/0x70 [13216.114837] ret_from_fork+0x1f/0x30 [13216.556828] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13216.559220] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13216.561485] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13216.563020] mdc_intent_lock+0x353/0x7e0 [mdc] [13216.566270] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13216.568331] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13216.571664] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13216.574693] lmv_intent_open+0x338/0x1150 [lmv] [13216.576764] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13216.584601] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13216.586532] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13216.589026] ll_intent_lock+0xc5/0xaa0 [lustre] [13216.591276] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13216.593628] ll_lookup_it+0xe0e/0x2270 [lustre] [13216.595772] ? do_raw_spin_unlock+0x75/0x190 [13216.597711] ? _raw_spin_unlock+0x12/0x30 [13216.599539] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13216.601796] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13216.604344] ll_atomic_open+0x36a/0xd70 [lustre] [13216.606487] lookup_open+0x356/0xb40 [13216.607989] do_last+0x313/0xfc0 [13216.609670] ? nd_jump_root+0xe5/0x160 [13216.611360] ? path_init+0x437/0x520 [13216.612810] path_openat+0xf7/0x500 [13216.614291] ? do_raw_spin_unlock+0x75/0x190 [13216.616330] do_filp_open+0x99/0x140 [13216.617813] ? getname_flags+0x6e/0x330 [13216.619606] ? __check_object_size+0xff/0x256 [13216.621550] ? do_raw_spin_unlock+0x75/0x190 [13216.623457] ? _raw_spin_unlock+0x12/0x30 [13216.625250] do_sys_openat2+0x2b4/0x410 [13216.626598] do_sys_open+0x73/0xa0 [13216.628246] __x64_sys_openat+0x24/0x30 [13216.629805] do_syscall_64+0xc1/0x440 [13216.631517] entry_SYSCALL_64_after_hwframe+0x49/0xae [13216.633764] RIP: 0033:0x7f53425a5332 [13216.635488] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13216.641546] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13216.644521] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13216.647629] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13216.650601] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13216.652266] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13216.654255] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13240.054229] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [13240.058538] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13240.103775] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13240.114153] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13240.127191] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13240.135366] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13240.137427] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13240.157462] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13240.162611] RAX: 0000000000006b0d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13240.166742] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13240.171649] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13240.176543] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13240.183893] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13240.188900] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13240.193321] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13240.197270] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13240.201281] Call Trace: [13240.202925] [13240.204945] ? show_regs.cold.9+0x22/0x2f [13240.208051] ? watchdog_timer_fn.cold.6+0x62/0x10a [13240.216240] ? __hrtimer_run_queues+0x1c2/0x4d0 [13240.219601] ? watchdog+0x40/0x40 [13240.223489] ? hrtimer_interrupt+0x128/0x2c0 [13240.228511] ? smp_apic_timer_interrupt+0xa6/0x270 [13240.232461] ? apic_timer_interrupt+0xf/0x20 [13240.238450] [13240.240389] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13240.243329] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13240.246877] ? newidle_balance.isra.93+0x207/0x580 [13240.249249] do_raw_spin_lock+0x177/0x190 [13240.252210] _raw_spin_lock+0x21/0x30 [13240.255519] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13240.259401] process_one_work+0x2c8/0x700 [13240.261946] worker_thread+0x296/0x6e0 [13240.264648] ? rescuer_thread+0x570/0x570 [13240.267365] kthread+0x1d1/0x200 [13240.270433] ? set_kthread_struct+0x70/0x70 [13240.275400] ret_from_fork+0x1f/0x30 [13242.336023] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10605s! [13242.339280] Showing busy workqueues and worker pools: [13242.342867] workqueue events: flags=0x0 [13242.347833] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13242.352070] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13242.364097] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13242.377106] pending: cpu_latency_work [ptlrpc] [13242.379348] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13242.386884] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13242.390237] workqueue events_long: flags=0x0 [13242.402330] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13242.411427] pending: lu_objects_flush [obdclass] [13242.415803] workqueue events_unbound: flags=0x2 [13242.418448] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13242.422322] pending: flush_memcg_stats_dwork, idle_cull_fn [13242.428102] workqueue events_power_efficient: flags=0x80 [13242.432447] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13242.437439] pending: check_lifetime, do_cache_clean [13242.439915] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13242.443099] pending: neigh_periodic_work, neigh_periodic_work [13242.445916] workqueue mm_percpu_wq: flags=0x8 [13242.448721] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13242.452164] pending: vmstat_update, lru_add_drain_per_cpu [13242.455250] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13242.461069] pending: vmstat_update, lru_add_drain_per_cpu [13242.463454] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13242.470004] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13242.476688] workqueue xprtiod: flags=0xa [13242.479257] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13242.486551] pending: xprt_autoclose, xs_error_handle [13242.491707] workqueue ipv6_addrconf: flags=0x40008 [13242.498396] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13242.505250] pending: addrconf_verify_work [13242.509316] workqueue knbd0-recv: flags=0x1a [13242.517432] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13242.524493] in-flight: 651:recv_work [13242.526651] workqueue ptlrpc_pinger: flags=0x2 [13242.529677] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13242.536676] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13242.544049] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10605s workers=4 idle: 15979 31824 8 [13242.554432] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13233s workers=2 idle: 654 [13244.057747] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [13244.058772] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13244.061407] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [13244.061425] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13244.061496] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13244.061502] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13244.061504] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13244.061519] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13244.061523] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13244.061527] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13244.061529] RDX: 000000000000740e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13244.061532] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13244.061534] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13244.061536] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13244.061539] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13244.061542] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13244.061544] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13244.061556] Call Trace: [13244.061573] [13244.061580] ? show_regs.cold.9+0x22/0x2f [13244.061592] ? watchdog_timer_fn.cold.6+0x62/0x10a [13244.061600] ? __hrtimer_run_queues+0x1c2/0x4d0 [13244.061604] ? watchdog+0x40/0x40 [13244.061609] ? hrtimer_interrupt+0x128/0x2c0 [13244.061613] ? smp_apic_timer_interrupt+0xa6/0x270 [13244.061624] ? apic_timer_interrupt+0xf/0x20 [13244.061628] [13244.061631] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13244.061636] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13244.061641] do_raw_spin_lock+0x177/0x190 [13244.061646] _raw_spin_lock+0x21/0x30 [13244.061653] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13244.064179] obd_mod_free+0x46/0x140 [ptlrpc] [13244.065010] mdc_commit_open+0x91/0xb0 [mdc] [13244.065055] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13244.065261] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13244.065477] after_reply+0xe2c/0x1780 [ptlrpc] [13244.065979] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13244.066178] ? __next_timer_interrupt+0x160/0x160 [13244.066192] ptlrpcd+0xa82/0xe40 [ptlrpc] [13244.066409] ? do_wait_intr+0xf0/0xf0 [13244.066423] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13244.067052] kthread+0x1d1/0x200 [13244.067058] ? set_kthread_struct+0x70/0x70 [13244.067062] ret_from_fork+0x1f/0x30 [13244.070210] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw [13244.073717] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13244.073799] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13244.073807] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13244.073810] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13244.073825] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13244.073830] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13244.073835] RAX: 0000000000003285 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13244.073838] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13244.073841] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13244.073843] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13244.073846] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13244.073849] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13244.073852] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13244.073854] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13244.073862] Call Trace: [13244.073868] [13244.073873] ? show_regs.cold.9+0x22/0x2f [13244.073882] ? watchdog_timer_fn.cold.6+0x62/0x10a [13244.073888] ? __hrtimer_run_queues+0x1c2/0x4d0 [13244.073893] ? watchdog+0x40/0x40 [13244.073897] ? hrtimer_interrupt+0x128/0x2c0 [13244.073901] ? smp_apic_timer_interrupt+0xa6/0x270 [13244.073910] ? apic_timer_interrupt+0xf/0x20 [13244.073915] [13244.073918] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13244.073923] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13244.073930] do_raw_spin_lock+0x177/0x190 [13244.073934] _raw_spin_lock+0x21/0x30 [13244.073940] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13244.074204] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13244.074395] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13244.074917] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13244.075132] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13244.075343] ? woken_wake_function+0x30/0x30 [13244.075352] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13244.075978] kthread+0x1d1/0x200 [13244.075987] ? set_kthread_struct+0x70/0x70 [13244.075992] ret_from_fork+0x1f/0x30 [13244.592707] libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13244.595716] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13244.600167] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13244.603360] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13244.605520] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13244.619591] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13244.629123] RAX: 000000000000380b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13244.632511] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13244.638524] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13244.645029] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13244.648700] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13244.652273] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13244.658454] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13244.660105] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13244.663284] Call Trace: [13244.664119] [13244.668688] ? show_regs.cold.9+0x22/0x2f [13244.672404] ? watchdog_timer_fn.cold.6+0x62/0x10a [13244.674745] ? __hrtimer_run_queues+0x1c2/0x4d0 [13244.678528] ? watchdog+0x40/0x40 [13244.680735] ? hrtimer_interrupt+0x128/0x2c0 [13244.683938] ? smp_apic_timer_interrupt+0xa6/0x270 [13244.686058] ? apic_timer_interrupt+0xf/0x20 [13244.692704] [13244.693476] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13244.695635] do_raw_spin_lock+0x177/0x190 [13244.697261] _raw_spin_lock+0x21/0x30 [13244.707285] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13244.710458] ? do_raw_spin_unlock+0x75/0x190 [13244.714821] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13244.717912] ? do_wait_intr+0xf0/0xf0 [13244.719665] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13244.722527] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13244.724955] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13244.727079] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13244.738711] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13244.741690] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13244.743943] mdc_intent_lock+0x353/0x7e0 [mdc] [13244.747959] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13244.749749] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13244.757047] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13244.761686] lmv_intent_open+0x338/0x1150 [lmv] [13244.765191] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13244.770828] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13244.775949] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13244.778628] ll_intent_lock+0xc5/0xaa0 [lustre] [13244.780828] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13244.785753] ll_lookup_it+0xe0e/0x2270 [lustre] [13244.788818] ? do_raw_spin_unlock+0x75/0x190 [13244.791114] ? _raw_spin_unlock+0x12/0x30 [13244.793455] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13244.796393] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13244.798779] ll_atomic_open+0x36a/0xd70 [lustre] [13244.800407] lookup_open+0x356/0xb40 [13244.805734] do_last+0x313/0xfc0 [13244.807048] ? nd_jump_root+0xe5/0x160 [13244.808922] ? path_init+0x437/0x520 [13244.810519] path_openat+0xf7/0x500 [13244.812339] ? do_raw_spin_unlock+0x75/0x190 [13244.816089] do_filp_open+0x99/0x140 [13244.817292] ? getname_flags+0x6e/0x330 [13244.820039] ? __check_object_size+0xff/0x256 [13244.822030] ? do_raw_spin_unlock+0x75/0x190 [13244.823953] ? _raw_spin_unlock+0x12/0x30 [13244.826030] do_sys_openat2+0x2b4/0x410 [13244.827813] do_sys_open+0x73/0xa0 [13244.829421] __x64_sys_openat+0x24/0x30 [13244.831062] do_syscall_64+0xc1/0x440 [13244.832766] entry_SYSCALL_64_after_hwframe+0x49/0xae [13244.836084] RIP: 0033:0x7f53425a5332 [13244.837981] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13244.847093] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13244.850238] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13244.854027] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13244.857087] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13244.860095] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13244.862736] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13268.052065] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [13268.056584] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13268.075443] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13268.080208] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13268.084057] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13268.086841] RIP: 0010:native_safe_halt+0x2e/0x50 [13268.088537] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13268.100952] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13268.105033] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13268.112848] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13268.116044] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13268.123399] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13268.127105] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13268.130720] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13268.137681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13268.140718] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13268.143803] Call Trace: [13268.144659] [13268.146789] ? show_regs.cold.9+0x22/0x2f [13268.147895] ? watchdog_timer_fn.cold.6+0x62/0x10a [13268.153123] ? __hrtimer_run_queues+0x1c2/0x4d0 [13268.154911] ? watchdog+0x40/0x40 [13268.156598] ? hrtimer_interrupt+0x128/0x2c0 [13268.162251] ? smp_apic_timer_interrupt+0xa6/0x270 [13268.163915] ? apic_timer_interrupt+0xf/0x20 [13268.169100] [13268.169858] ? native_safe_halt+0x2e/0x50 [13268.171531] kvm_wait+0xcd/0xe0 [13268.173417] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13268.175495] ? newidle_balance.isra.93+0x207/0x580 [13268.180127] do_raw_spin_lock+0x177/0x190 [13268.183697] _raw_spin_lock+0x21/0x30 [13268.185343] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13268.188761] process_one_work+0x2c8/0x700 [13268.190373] worker_thread+0x296/0x6e0 [13268.193085] ? rescuer_thread+0x570/0x570 [13268.197695] kthread+0x1d1/0x200 [13268.200079] ? set_kthread_struct+0x70/0x70 [13268.205132] ret_from_fork+0x1f/0x30 [13272.057015] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [13272.058342] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13272.058804] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [13272.058811] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13272.058885] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13272.058890] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13272.058893] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13272.058912] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13272.058917] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13272.058922] RAX: 0000000000007f86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13272.058924] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13272.058927] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13272.058929] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13272.058931] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13272.058934] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13272.058938] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13272.058940] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13272.058950] Call Trace: [13272.058956] [13272.058962] ? show_regs.cold.9+0x22/0x2f [13272.058974] ? watchdog_timer_fn.cold.6+0x62/0x10a [13272.058980] ? __hrtimer_run_queues+0x1c2/0x4d0 [13272.058984] ? watchdog+0x40/0x40 [13272.058988] ? hrtimer_interrupt+0x128/0x2c0 [13272.058992] ? smp_apic_timer_interrupt+0xa6/0x270 [13272.059001] ? apic_timer_interrupt+0xf/0x20 [13272.059005] [13272.059022] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13272.059029] do_raw_spin_lock+0x177/0x190 [13272.059034] _raw_spin_lock+0x21/0x30 [13272.059038] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13272.059940] ? do_raw_spin_unlock+0x75/0x190 [13272.060058] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [13272.059947] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13272.060088] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13272.060139] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13272.060145] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13272.060147] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13272.060156] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13272.060160] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13272.060164] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13272.060167] RDX: 0000000000004873 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13272.060169] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13272.060171] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13272.060173] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13272.060176] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13272.060179] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13272.060182] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13272.060191] Call Trace: [13272.060195] [13272.060199] ? show_regs.cold.9+0x22/0x2f [13272.060207] ? watchdog_timer_fn.cold.6+0x62/0x10a [13272.060212] ? __hrtimer_run_queues+0x1c2/0x4d0 [13272.060216] ? watchdog+0x40/0x40 [13272.060219] ? hrtimer_interrupt+0x128/0x2c0 [13272.060223] ? smp_apic_timer_interrupt+0xa6/0x270 [13272.060230] ? apic_timer_interrupt+0xf/0x20 [13272.060234] [13272.060237] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13272.060241] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13272.060245] do_raw_spin_lock+0x177/0x190 [13272.060249] _raw_spin_lock+0x21/0x30 [13272.060253] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13272.060447] obd_mod_free+0x46/0x140 [ptlrpc] [13272.060641] mdc_commit_open+0x91/0xb0 [mdc] [13272.060693] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13272.060887] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13272.063504] after_reply+0xe2c/0x1780 [ptlrpc] [13272.063735] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13272.063927] ? __next_timer_interrupt+0x160/0x160 [13272.063951] ptlrpcd+0xa82/0xe40 [ptlrpc] [13272.064166] ? do_wait_intr+0xf0/0xf0 [13272.064176] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13272.064767] kthread+0x1d1/0x200 [13272.064773] ? set_kthread_struct+0x70/0x70 [13272.064777] ret_from_fork+0x1f/0x30 [13272.068985] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13272.069064] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13272.069085] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13272.069088] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13272.069103] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13272.069108] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13272.069113] RAX: 0000000000002d94 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13272.069115] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13272.069117] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13272.069119] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13272.069122] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13272.069125] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13272.069128] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13272.069130] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13272.069140] Call Trace: [13272.069151] [13272.069158] ? show_regs.cold.9+0x22/0x2f [13272.069169] ? watchdog_timer_fn.cold.6+0x62/0x10a [13272.069176] ? __hrtimer_run_queues+0x1c2/0x4d0 [13272.069180] ? watchdog+0x40/0x40 [13272.069185] ? hrtimer_interrupt+0x128/0x2c0 [13272.069189] ? smp_apic_timer_interrupt+0xa6/0x270 [13272.069199] ? apic_timer_interrupt+0xf/0x20 [13272.069203] [13272.069206] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13272.069211] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13272.069217] do_raw_spin_lock+0x177/0x190 [13272.069221] _raw_spin_lock+0x21/0x30 [13272.069227] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13272.069526] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13272.069726] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13272.069902] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13272.070103] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13272.070313] ? woken_wake_function+0x30/0x30 [13272.070322] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13272.070535] kthread+0x1d1/0x200 [13272.070543] ? set_kthread_struct+0x70/0x70 [13272.070546] ret_from_fork+0x1f/0x30 [13272.806786] ? do_wait_intr+0xf0/0xf0 [13272.809494] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13272.812758] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13272.815338] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13272.818738] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13272.828696] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13272.830680] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13272.836401] mdc_intent_lock+0x353/0x7e0 [mdc] [13272.839997] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13272.843860] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13272.848568] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13272.851936] lmv_intent_open+0x338/0x1150 [lmv] [13272.855099] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13272.863133] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13272.866871] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13272.869627] ll_intent_lock+0xc5/0xaa0 [lustre] [13272.875860] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13272.878679] ll_lookup_it+0xe0e/0x2270 [lustre] [13272.880576] ? do_raw_spin_unlock+0x75/0x190 [13272.881946] ? _raw_spin_unlock+0x12/0x30 [13272.894812] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13272.898890] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13272.901359] ll_atomic_open+0x36a/0xd70 [lustre] [13272.904858] lookup_open+0x356/0xb40 [13272.906047] do_last+0x313/0xfc0 [13272.910327] ? nd_jump_root+0xe5/0x160 [13272.914615] ? path_init+0x437/0x520 [13272.916983] path_openat+0xf7/0x500 [13272.919183] ? do_raw_spin_unlock+0x75/0x190 [13272.921741] do_filp_open+0x99/0x140 [13272.925086] ? getname_flags+0x6e/0x330 [13272.933643] ? __check_object_size+0xff/0x256 [13272.937758] ? do_raw_spin_unlock+0x75/0x190 [13272.941428] ? _raw_spin_unlock+0x12/0x30 [13272.944035] do_sys_openat2+0x2b4/0x410 [13272.946436] do_sys_open+0x73/0xa0 [13272.949610] __x64_sys_openat+0x24/0x30 [13272.951609] do_syscall_64+0xc1/0x440 [13272.953912] entry_SYSCALL_64_after_hwframe+0x49/0xae [13272.960874] RIP: 0033:0x7f53425a5332 [13272.963624] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13272.972823] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13272.981989] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13272.986620] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13272.991948] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13272.996969] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13273.000787] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13273.060262] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10635s! [13273.066763] Showing busy workqueues and worker pools: [13273.073705] workqueue events: flags=0x0 [13273.085305] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13273.097112] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13273.103696] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13273.106792] pending: cpu_latency_work [ptlrpc] [13273.113317] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13273.118806] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13273.125802] workqueue events_long: flags=0x0 [13273.127830] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13273.133907] pending: lu_objects_flush [obdclass] [13273.143514] workqueue events_unbound: flags=0x2 [13273.149485] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13273.152240] pending: flush_memcg_stats_dwork, idle_cull_fn [13273.155524] workqueue events_power_efficient: flags=0x80 [13273.159881] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13273.165873] pending: check_lifetime, do_cache_clean [13273.169027] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13273.173607] pending: neigh_periodic_work, neigh_periodic_work [13273.175928] workqueue mm_percpu_wq: flags=0x8 [13273.181420] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13273.184288] pending: vmstat_update, lru_add_drain_per_cpu [13273.187401] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13273.192533] pending: vmstat_update, lru_add_drain_per_cpu [13273.196582] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13273.198793] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13273.205872] workqueue xprtiod: flags=0xa [13273.212375] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13273.215897] pending: xprt_autoclose, xs_error_handle [13273.218636] workqueue ipv6_addrconf: flags=0x40008 [13273.222234] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13273.224411] pending: addrconf_verify_work [13273.228305] workqueue knbd0-recv: flags=0x1a [13273.230010] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13273.233549] in-flight: 651:recv_work [13273.235537] workqueue ptlrpc_pinger: flags=0x2 [13273.237583] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13273.240554] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13273.243418] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10636s workers=4 idle: 15979 31824 8 [13273.249458] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13263s workers=2 idle: 654 [13300.052468] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [13300.055033] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [13300.058037] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13300.058053] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13300.058140] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13300.058146] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13300.058149] RIP: 0010:native_safe_halt+0x2e/0x50 [13300.058165] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13300.058170] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13300.058174] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13300.058177] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13300.058179] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13300.058181] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13300.058184] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13300.058187] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13300.058191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13300.058193] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13300.058204] Call Trace: [13300.058210] [13300.058216] ? show_regs.cold.9+0x22/0x2f [13300.058226] ? watchdog_timer_fn.cold.6+0x62/0x10a [13300.058232] ? __hrtimer_run_queues+0x1c2/0x4d0 [13300.058237] ? watchdog+0x40/0x40 [13300.058242] ? hrtimer_interrupt+0x128/0x2c0 [13300.058247] ? smp_apic_timer_interrupt+0xa6/0x270 [13300.058254] ? apic_timer_interrupt+0xf/0x20 [13300.058259] [13300.058263] ? native_safe_halt+0x2e/0x50 [13300.058267] kvm_wait+0xcd/0xe0 [13300.058276] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13300.058285] do_raw_spin_lock+0x177/0x190 [13300.058290] _raw_spin_lock+0x21/0x30 [13300.058295] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13300.058618] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13300.058840] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13300.059065] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13300.059311] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13300.059577] ? woken_wake_function+0x30/0x30 [13300.059597] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13300.059809] kthread+0x1d1/0x200 [13300.059816] ? set_kthread_struct+0x70/0x70 [13300.059820] ret_from_fork+0x1f/0x30 [13300.060882] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13300.060956] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13300.060963] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13300.060970] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13300.061226] RIP: 0010:native_safe_halt+0x2e/0x50 [13300.061235] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13300.061239] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13300.061245] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13300.061248] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [13300.061252] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13300.061255] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13300.061257] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13300.061262] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13300.061265] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13300.061268] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13300.061278] Call Trace: [13300.061283] [13300.061288] ? show_regs.cold.9+0x22/0x2f [13300.061299] ? watchdog_timer_fn.cold.6+0x62/0x10a [13300.061306] ? __hrtimer_run_queues+0x1c2/0x4d0 [13300.061310] ? watchdog+0x40/0x40 [13300.061315] ? hrtimer_interrupt+0x128/0x2c0 [13300.061319] ? smp_apic_timer_interrupt+0xa6/0x270 [13300.061326] ? apic_timer_interrupt+0xf/0x20 [13300.061331] [13300.061334] ? native_safe_halt+0x2e/0x50 [13300.061338] kvm_wait+0xcd/0xe0 [13300.061344] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13300.061351] ? newidle_balance.isra.93+0x207/0x580 [13300.061357] do_raw_spin_lock+0x177/0x190 [13300.061361] _raw_spin_lock+0x21/0x30 [13300.061366] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13300.062000] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [13300.062007] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13300.062068] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13300.062074] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13300.062076] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13300.062085] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13300.062089] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13300.062093] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13300.062096] RDX: 0000000000007f84 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13300.062098] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13300.062100] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13300.062103] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13300.062106] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13300.062109] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13300.062111] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13300.062119] Call Trace: [13300.062122] [13300.062126] ? show_regs.cold.9+0x22/0x2f [13300.062133] ? watchdog_timer_fn.cold.6+0x62/0x10a [13300.062138] ? __hrtimer_run_queues+0x1c2/0x4d0 [13300.062142] ? watchdog+0x40/0x40 [13300.062146] ? hrtimer_interrupt+0x128/0x2c0 [13300.062150] ? smp_apic_timer_interrupt+0xa6/0x270 [13300.062157] ? apic_timer_interrupt+0xf/0x20 [13300.062161] [13300.062164] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13300.062170] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13300.062175] do_raw_spin_lock+0x177/0x190 [13300.062179] _raw_spin_lock+0x21/0x30 [13300.062184] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13300.063178] obd_mod_free+0x46/0x140 [ptlrpc] [13300.063320] process_one_work+0x2c8/0x700 [13300.063334] worker_thread+0x296/0x6e0 [13300.063339] ? rescuer_thread+0x570/0x570 [13300.063343] kthread+0x1d1/0x200 [13300.063348] ? set_kthread_struct+0x70/0x70 [13300.063352] ret_from_fork+0x1f/0x30 [13300.064046] mdc_commit_open+0x91/0xb0 [mdc] [13300.064094] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13300.065434] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13300.065768] after_reply+0xe2c/0x1780 [ptlrpc] [13300.066705] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13300.067751] ? __next_timer_interrupt+0x160/0x160 [13300.067765] ptlrpcd+0xa82/0xe40 [ptlrpc] [13300.068430] ? do_wait_intr+0xf0/0xf0 [13300.068440] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13300.069339] kthread+0x1d1/0x200 [13300.069346] ? set_kthread_struct+0x70/0x70 [13300.069350] ret_from_fork+0x1f/0x30 [13300.463046] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13300.491037] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13300.501858] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13300.505782] RIP: 0010:native_safe_halt+0x2e/0x50 [13300.507522] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13300.514793] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13300.517734] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13300.524521] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13300.530982] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13300.534090] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13300.536716] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13300.542346] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13300.548175] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13300.550184] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13300.553095] Call Trace: [13300.555041] [13300.555936] ? show_regs.cold.9+0x22/0x2f [13300.561170] ? watchdog_timer_fn.cold.6+0x62/0x10a [13300.564717] ? __hrtimer_run_queues+0x1c2/0x4d0 [13300.566265] ? watchdog+0x40/0x40 [13300.567173] ? hrtimer_interrupt+0x128/0x2c0 [13300.568288] ? smp_apic_timer_interrupt+0xa6/0x270 [13300.570016] ? apic_timer_interrupt+0xf/0x20 [13300.571319] [13300.571999] ? native_safe_halt+0x2e/0x50 [13300.574735] kvm_wait+0xcd/0xe0 [13300.579685] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13300.583232] do_raw_spin_lock+0x177/0x190 [13300.586462] _raw_spin_lock+0x21/0x30 [13300.592227] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13300.599895] ? do_raw_spin_unlock+0x75/0x190 [13300.601329] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13300.603339] ? do_wait_intr+0xf0/0xf0 [13300.605787] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13300.607624] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13300.611529] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13300.614156] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13300.616769] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13300.621314] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13300.624987] mdc_intent_lock+0x353/0x7e0 [mdc] [13300.628017] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13300.631218] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13300.635376] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13300.641267] lmv_intent_open+0x338/0x1150 [lmv] [13300.645512] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13300.648844] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13300.650942] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13300.653181] ll_intent_lock+0xc5/0xaa0 [lustre] [13300.655817] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13300.659100] ll_lookup_it+0xe0e/0x2270 [lustre] [13300.663477] ? do_raw_spin_unlock+0x75/0x190 [13300.665329] ? _raw_spin_unlock+0x12/0x30 [13300.666563] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13300.669006] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13300.672913] ll_atomic_open+0x36a/0xd70 [lustre] [13300.679035] lookup_open+0x356/0xb40 [13300.681486] do_last+0x313/0xfc0 [13300.688099] ? nd_jump_root+0xe5/0x160 [13300.689314] ? path_init+0x437/0x520 [13300.690512] path_openat+0xf7/0x500 [13300.691532] ? do_raw_spin_unlock+0x75/0x190 [13300.700502] do_filp_open+0x99/0x140 [13300.701774] ? getname_flags+0x6e/0x330 [13300.704158] ? __check_object_size+0xff/0x256 [13300.707755] ? do_raw_spin_unlock+0x75/0x190 [13300.711064] ? _raw_spin_unlock+0x12/0x30 [13300.714695] do_sys_openat2+0x2b4/0x410 [13300.718466] do_sys_open+0x73/0xa0 [13300.719754] __x64_sys_openat+0x24/0x30 [13300.722447] do_syscall_64+0xc1/0x440 [13300.723647] entry_SYSCALL_64_after_hwframe+0x49/0xae [13300.727514] RIP: 0033:0x7f53425a5332 [13300.729419] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13300.744067] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13300.754480] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13300.760464] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13300.763417] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13300.767886] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13300.770862] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13303.264018] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10666s! [13303.275237] Showing busy workqueues and worker pools: [13303.281293] workqueue events: flags=0x0 [13303.283184] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13303.289885] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13303.300303] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13303.303812] pending: cpu_latency_work [ptlrpc] [13303.306182] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13303.310696] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13303.315163] workqueue events_long: flags=0x0 [13303.316801] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13303.319595] pending: lu_objects_flush [obdclass] [13303.321763] workqueue events_unbound: flags=0x2 [13303.323235] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13303.325508] pending: flush_memcg_stats_dwork, idle_cull_fn [13303.327717] workqueue events_power_efficient: flags=0x80 [13303.329069] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13303.331491] pending: check_lifetime, do_cache_clean [13303.333600] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13303.342325] pending: neigh_periodic_work, neigh_periodic_work [13303.345166] workqueue mm_percpu_wq: flags=0x8 [13303.348772] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13303.351026] pending: vmstat_update, lru_add_drain_per_cpu [13303.356839] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13303.359975] pending: vmstat_update, lru_add_drain_per_cpu [13303.363204] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13303.373648] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13303.383881] workqueue xprtiod: flags=0xa [13303.385301] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13303.396474] pending: xprt_autoclose, xs_error_handle [13303.400146] workqueue ipv6_addrconf: flags=0x40008 [13303.406172] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13303.411499] pending: addrconf_verify_work [13303.413799] workqueue knbd0-recv: flags=0x1a [13303.415353] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13303.421089] in-flight: 651:recv_work [13303.425203] workqueue ptlrpc_pinger: flags=0x2 [13303.427236] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13303.433352] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13303.439890] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10666s workers=4 idle: 15979 31824 8 [13303.448670] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13293s workers=2 idle: 654 [13314.699764] rcu: INFO: rcu_sched self-detected stall on CPU [13314.702076] rcu: 2-....: (6233357 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [13314.706070] (t=11220273 jiffies g=274185 q=66773) [13314.708801] rcu: rcu_sched kthread starved for 10678639 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [13314.717525] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [13314.721980] rcu: RCU grace-period kthread stack dump: [13314.724264] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [13314.731856] Call Trace: [13314.732880] __schedule+0x351/0xcb0 [13314.737199] schedule+0xc0/0x180 [13314.743093] schedule_timeout+0xb4/0x190 [13314.745168] ? do_raw_spin_unlock+0x75/0x190 [13314.746968] ? __next_timer_interrupt+0x160/0x160 [13314.749281] ? prepare_to_swait_event+0xf3/0x140 [13314.751177] rcu_gp_kthread+0x8dd/0x15b0 [13314.753065] ? force_qs_rnp+0x2c0/0x2c0 [13314.758950] kthread+0x1d1/0x200 [13314.760625] ? set_kthread_struct+0x70/0x70 [13314.764910] ret_from_fork+0x1f/0x30 [13314.769971] rcu: Stack dump where RCU GP kthread last ran: [13314.771783] Sending NMI from CPU 2 to CPUs 1: [13314.778521] NMI backtrace for cpu 1 [13314.778529] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13314.778532] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13314.778534] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13314.778536] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13314.778538] RSP: 0018:ffffbb5381503500 EFLAGS: 00000212 [13314.778541] RAX: 0000000000005913 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13314.778543] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13314.778544] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13314.778546] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13314.778548] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13314.778549] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13314.778551] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13314.778553] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13314.778554] Call Trace: [13314.778556] [13314.778557] ? show_regs.cold.9+0x22/0x2f [13314.778558] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [13314.778559] ? nmi_cpu_backtrace_handler+0x15/0x40 [13314.778560] ? nmi_handle+0xbd/0x210 [13314.778561] ? default_do_nmi+0x4b/0x1b0 [13314.778562] ? do_nmi+0x280/0x380 [13314.778563] ? end_repeat_nmi+0x16/0x1f [13314.778564] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13314.778565] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13314.778566] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13314.778567] [13314.778568] do_raw_spin_lock+0x177/0x190 [13314.778569] _raw_spin_lock+0x21/0x30 [13314.778570] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13314.778571] ? do_raw_spin_unlock+0x75/0x190 [13314.778572] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13314.778574] ? do_wait_intr+0xf0/0xf0 [13314.778575] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13314.778576] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13314.778577] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13314.778578] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13314.778579] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13314.778581] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13314.778582] mdc_intent_lock+0x353/0x7e0 [mdc] [13314.778583] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13314.778584] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13314.778585] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13314.778593] lmv_intent_open+0x338/0x1150 [lmv] [13314.778595] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13314.778596] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13314.778597] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13314.778599] ll_intent_lock+0xc5/0xaa0 [lustre] [13314.778600] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13314.778601] ll_lookup_it+0xe0e/0x2270 [lustre] [13314.778603] ? do_raw_spin_unlock+0x75/0x190 [13314.778604] ? _raw_spin_unlock+0x12/0x30 [13314.778605] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13314.778607] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13314.778608] ll_atomic_open+0x36a/0xd70 [lustre] [13314.778609] lookup_open+0x356/0xb40 [13314.778610] do_last+0x313/0xfc0 [13314.778616] ? nd_jump_root+0xe5/0x160 [13314.778618] ? path_init+0x437/0x520 [13314.778619] path_openat+0xf7/0x500 [13314.778620] ? do_raw_spin_unlock+0x75/0x190 [13314.778621] do_filp_open+0x99/0x140 [13314.778623] ? getname_flags+0x6e/0x330 [13314.778624] ? __check_object_size+0xff/0x256 [13314.778625] ? do_raw_spin_unlock+0x75/0x190 [13314.778626] ? _raw_spin_unlock+0x12/0x30 [13314.778627] do_sys_openat2+0x2b4/0x410 [13314.778628] do_sys_open+0x73/0xa0 [13314.778629] __x64_sys_openat+0x24/0x30 [13314.778630] do_syscall_64+0xc1/0x440 [13314.778632] entry_SYSCALL_64_after_hwframe+0x49/0xae [13314.778634] RIP: 0033:0x7f53425a5332 [13314.778635] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13314.778638] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13314.778641] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13314.778642] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13314.778644] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13314.778645] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13314.778647] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13314.779260] Sending NMI from CPU 2 to CPUs 1: [13315.109260] NMI backtrace for cpu 1 [13315.109334] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13315.109337] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13315.109339] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13315.109340] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13315.109343] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [13315.109345] RAX: 0000000000007e20 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13315.109346] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13315.109348] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13315.109349] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13315.109351] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13315.109352] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13315.109353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13315.109354] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13315.109355] Call Trace: [13315.109356] [13315.109357] ? show_regs.cold.9+0x22/0x2f [13315.109358] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [13315.109359] ? nmi_cpu_backtrace_handler+0x15/0x40 [13315.109360] ? nmi_handle+0xbd/0x210 [13315.109361] ? default_do_nmi+0x4b/0x1b0 [13315.109362] ? do_nmi+0x280/0x380 [13315.109363] ? end_repeat_nmi+0x16/0x1f [13315.109364] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13315.109365] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13315.109367] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13315.109371] [13315.109372] do_raw_spin_lock+0x177/0x190 [13315.109373] _raw_spin_lock+0x21/0x30 [13315.109374] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13315.109375] ? do_raw_spin_unlock+0x75/0x190 [13315.109376] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13315.109377] ? do_wait_intr+0xf0/0xf0 [13315.109378] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13315.109379] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13315.109381] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13315.109382] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13315.109383] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13315.109384] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13315.109385] mdc_intent_lock+0x353/0x7e0 [mdc] [13315.109386] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13315.109387] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13315.109388] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13315.109389] lmv_intent_open+0x338/0x1150 [lmv] [13315.109390] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13315.109391] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13315.109392] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13315.109393] ll_intent_lock+0xc5/0xaa0 [lustre] [13315.109394] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13315.109396] ll_lookup_it+0xe0e/0x2270 [lustre] [13315.109397] ? do_raw_spin_unlock+0x75/0x190 [13315.109487] ? _raw_spin_unlock+0x12/0x30 [13315.109489] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13315.109490] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13315.109491] ll_atomic_open+0x36a/0xd70 [lustre] [13315.109492] lookup_open+0x356/0xb40 [13315.109493] do_last+0x313/0xfc0 [13315.109494] ? nd_jump_root+0xe5/0x160 [13315.109495] ? path_init+0x437/0x520 [13315.109496] path_openat+0xf7/0x500 [13315.109497] ? do_raw_spin_unlock+0x75/0x190 [13315.109498] do_filp_open+0x99/0x140 [13315.109499] ? getname_flags+0x6e/0x330 [13315.109500] ? __check_object_size+0xff/0x256 [13315.109500] ? do_raw_spin_unlock+0x75/0x190 [13315.109502] ? _raw_spin_unlock+0x12/0x30 [13315.109503] do_sys_openat2+0x2b4/0x410 [13315.109503] do_sys_open+0x73/0xa0 [13315.109504] __x64_sys_openat+0x24/0x30 [13315.109505] do_syscall_64+0xc1/0x440 [13315.109506] entry_SYSCALL_64_after_hwframe+0x49/0xae [13315.109507] RIP: 0033:0x7f53425a5332 [13315.109509] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13315.109511] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13315.109513] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13315.109515] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13315.109516] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13315.109518] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13315.109520] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13315.109600] NMI backtrace for cpu 2 [13315.344214] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13315.353121] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13315.356911] Call Trace: [13315.358706] [13315.359833] ? dump_stack+0xbb/0x10e [13315.361227] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13315.365913] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13315.371142] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13315.372798] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13315.380562] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13315.383020] ? rcu_dump_cpu_stacks+0x11b/0x171 [13315.385131] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [13315.389000] ? update_process_times+0x6d/0xc0 [13315.391690] ? tick_sched_handle.isra.18+0x27/0x90 [13315.394873] ? tick_sched_timer+0x53/0xb0 [13315.396865] ? __hrtimer_run_queues+0x1c2/0x4d0 [13315.405089] ? tick_sched_handle.isra.18+0x90/0x90 [13315.408869] ? hrtimer_interrupt+0x128/0x2c0 [13315.411293] ? smp_apic_timer_interrupt+0xa6/0x270 [13315.413856] ? apic_timer_interrupt+0xf/0x20 [13315.416074] [13315.417054] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13315.420143] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13315.425180] ? do_raw_spin_lock+0x177/0x190 [13315.427240] ? _raw_spin_lock+0x21/0x30 [13315.431383] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13315.434929] ? obd_mod_free+0x46/0x140 [ptlrpc] [13315.437582] ? mdc_commit_open+0x91/0xb0 [mdc] [13315.439795] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13315.443815] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13315.446928] ? after_reply+0xe2c/0x1780 [ptlrpc] [13315.450222] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13315.453954] ? __next_timer_interrupt+0x160/0x160 [13315.455606] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13315.460094] ? do_wait_intr+0xf0/0xf0 [13315.463102] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13315.466207] ? kthread+0x1d1/0x200 [13315.467647] ? set_kthread_struct+0x70/0x70 [13315.470628] ? ret_from_fork+0x1f/0x30 [13328.054024] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [13328.058156] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13328.061808] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13328.061944] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13328.061951] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13328.061957] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13328.063031] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13328.063061] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13328.063066] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13328.063071] RAX: 0000000000007f8d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13328.063073] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13328.063075] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13328.063078] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13328.063153] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13328.063158] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13328.063161] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13328.063163] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13328.063171] Call Trace: [13328.063176] [13328.063452] ? show_regs.cold.9+0x22/0x2f [13328.068804] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13328.070226] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13328.070234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13328.070236] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13328.070249] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13328.070254] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13328.070258] RAX: 0000000000002ecb RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13328.070261] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13328.070263] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13328.070265] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13328.070268] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13328.070271] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13328.070274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13328.070277] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13328.070285] Call Trace: [13328.070291] [13328.070296] ? show_regs.cold.9+0x22/0x2f [13328.070306] ? watchdog_timer_fn.cold.6+0x62/0x10a [13328.070312] ? __hrtimer_run_queues+0x1c2/0x4d0 [13328.070316] ? watchdog+0x40/0x40 [13328.070320] ? hrtimer_interrupt+0x128/0x2c0 [13328.070324] ? smp_apic_timer_interrupt+0xa6/0x270 [13328.070332] ? apic_timer_interrupt+0xf/0x20 [13328.070336] [13328.070338] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13328.070343] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13328.070348] do_raw_spin_lock+0x177/0x190 [13328.070353] _raw_spin_lock+0x21/0x30 [13328.070357] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13328.070617] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13328.071891] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13328.091551] ? watchdog_timer_fn.cold.6+0x62/0x10a [13328.091570] ? __hrtimer_run_queues+0x1c2/0x4d0 [13328.091575] ? watchdog+0x40/0x40 [13328.091580] ? hrtimer_interrupt+0x128/0x2c0 [13328.091585] ? smp_apic_timer_interrupt+0xa6/0x270 [13328.091594] ? apic_timer_interrupt+0xf/0x20 [13328.091599] [13328.091604] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13328.091611] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13328.091616] ? newidle_balance.isra.93+0x207/0x580 [13328.091624] do_raw_spin_lock+0x177/0x190 [13328.091630] _raw_spin_lock+0x21/0x30 [13328.091635] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13328.091999] process_one_work+0x2c8/0x700 [13328.092007] worker_thread+0x296/0x6e0 [13328.092012] ? rescuer_thread+0x570/0x570 [13328.092016] kthread+0x1d1/0x200 [13328.092021] ? set_kthread_struct+0x70/0x70 [13328.092025] ret_from_fork+0x1f/0x30 [13328.382031] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13328.384424] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13328.386683] ? woken_wake_function+0x30/0x30 [13328.388490] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13328.391340] kthread+0x1d1/0x200 [13328.393283] ? set_kthread_struct+0x70/0x70 [13328.394936] ret_from_fork+0x1f/0x30 [13333.471160] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10696s! [13333.481927] Showing busy workqueues and worker pools: [13333.485067] workqueue events: flags=0x0 [13333.486835] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13333.490977] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13333.496429] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13333.504799] pending: cpu_latency_work [ptlrpc] [13333.510065] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13333.517379] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13333.527351] workqueue events_long: flags=0x0 [13333.531457] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13333.537363] pending: lu_objects_flush [obdclass] [13333.544238] workqueue events_unbound: flags=0x2 [13333.548947] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13333.555341] pending: flush_memcg_stats_dwork, idle_cull_fn [13333.562913] workqueue events_power_efficient: flags=0x80 [13333.567741] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13333.575808] pending: check_lifetime, do_cache_clean [13333.581511] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13333.588334] pending: neigh_periodic_work, neigh_periodic_work [13333.594505] workqueue mm_percpu_wq: flags=0x8 [13333.597962] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13333.602912] pending: vmstat_update, lru_add_drain_per_cpu [13333.609695] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13333.613938] pending: vmstat_update, lru_add_drain_per_cpu [13333.620651] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13333.625863] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13333.632242] workqueue xprtiod: flags=0xa [13333.639844] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13333.647262] pending: xprt_autoclose, xs_error_handle [13333.653698] workqueue ipv6_addrconf: flags=0x40008 [13333.658509] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13333.665785] pending: addrconf_verify_work [13333.674583] workqueue knbd0-recv: flags=0x1a [13333.679238] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13333.685556] in-flight: 651:recv_work [13333.689787] workqueue ptlrpc_pinger: flags=0x2 [13333.692695] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13333.699358] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13333.706519] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10696s workers=4 idle: 15979 31824 8 [13333.714859] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13324s workers=2 idle: 654 [13340.055852] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [13340.058131] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13340.058592] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [13340.089558] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13340.089578] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13340.089581] RIP: 0010:native_safe_halt+0x2e/0x50 [13340.089600] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13340.089605] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13340.089612] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13340.089614] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13340.089616] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13340.089619] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13340.089626] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13340.089629] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13340.093364] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13340.093441] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13340.093449] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13340.093451] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13340.093465] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13340.093470] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13340.093475] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13340.093477] RDX: 0000000000003ab8 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13340.093480] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13340.093482] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13340.093485] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13340.093488] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13340.093491] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13340.093493] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13340.093565] Call Trace: [13340.093571] [13340.093577] ? show_regs.cold.9+0x22/0x2f [13340.093587] ? watchdog_timer_fn.cold.6+0x62/0x10a [13340.093594] ? __hrtimer_run_queues+0x1c2/0x4d0 [13340.093598] ? watchdog+0x40/0x40 [13340.093603] ? hrtimer_interrupt+0x128/0x2c0 [13340.093607] ? smp_apic_timer_interrupt+0xa6/0x270 [13340.093617] ? apic_timer_interrupt+0xf/0x20 [13340.093622] [13340.093626] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13340.093631] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13340.093635] do_raw_spin_lock+0x177/0x190 [13340.093640] _raw_spin_lock+0x21/0x30 [13340.093648] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13340.103638] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13340.114965] obd_mod_free+0x46/0x140 [ptlrpc] [13340.121502] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13340.132419] mdc_commit_open+0x91/0xb0 [mdc] [13340.136624] Call Trace: [13340.136636] [13340.136642] ? show_regs.cold.9+0x22/0x2f [13340.136656] ? watchdog_timer_fn.cold.6+0x62/0x10a [13340.136662] ? __hrtimer_run_queues+0x1c2/0x4d0 [13340.136666] ? watchdog+0x40/0x40 [13340.136670] ? hrtimer_interrupt+0x128/0x2c0 [13340.136674] ? smp_apic_timer_interrupt+0xa6/0x270 [13340.136682] ? apic_timer_interrupt+0xf/0x20 [13340.136686] [13340.136689] ? native_safe_halt+0x2e/0x50 [13340.136692] kvm_wait+0xcd/0xe0 [13340.136920] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13340.136930] do_raw_spin_lock+0x177/0x190 [13340.136934] _raw_spin_lock+0x21/0x30 [13340.136943] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13340.137622] ? do_raw_spin_unlock+0x75/0x190 [13340.137629] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13340.137939] ? do_wait_intr+0xf0/0xf0 [13340.137951] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13340.139723] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13340.140197] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13340.140424] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13340.140632] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13340.140683] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13340.140728] mdc_intent_lock+0x353/0x7e0 [mdc] [13340.140759] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13340.140863] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13340.141117] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13340.141149] lmv_intent_open+0x338/0x1150 [lmv] [13340.141181] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13340.141266] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13340.141477] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13340.141501] ll_intent_lock+0xc5/0xaa0 [lustre] [13340.141582] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13340.141664] ll_lookup_it+0xe0e/0x2270 [lustre] [13340.141744] ? do_raw_spin_unlock+0x75/0x190 [13340.141753] ? _raw_spin_unlock+0x12/0x30 [13340.141759] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13340.141808] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13340.141886] ll_atomic_open+0x36a/0xd70 [lustre] [13340.142044] lookup_open+0x356/0xb40 [13340.142054] do_last+0x313/0xfc0 [13340.142059] ? nd_jump_root+0xe5/0x160 [13340.142065] ? path_init+0x437/0x520 [13340.142069] path_openat+0xf7/0x500 [13340.142075] ? do_raw_spin_unlock+0x75/0x190 [13340.142080] do_filp_open+0x99/0x140 [13340.142087] ? getname_flags+0x6e/0x330 [13340.142093] ? __check_object_size+0xff/0x256 [13340.142101] ? do_raw_spin_unlock+0x75/0x190 [13340.142106] ? _raw_spin_unlock+0x12/0x30 [13340.142111] do_sys_openat2+0x2b4/0x410 [13340.142115] do_sys_open+0x73/0xa0 [13340.142120] __x64_sys_openat+0x24/0x30 [13340.142124] do_syscall_64+0xc1/0x440 [13340.142132] entry_SYSCALL_64_after_hwframe+0x49/0xae [13340.142139] RIP: 0033:0x7f53425a5332 [13340.142148] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13340.142153] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13340.142163] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13340.142166] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13340.142168] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13340.142170] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13340.142172] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13340.605489] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13340.607407] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13340.614100] after_reply+0xe2c/0x1780 [ptlrpc] [13340.618463] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13340.621860] ? __next_timer_interrupt+0x160/0x160 [13340.626675] ptlrpcd+0xa82/0xe40 [ptlrpc] [13340.631947] ? do_wait_intr+0xf0/0xf0 [13340.636125] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13340.638824] kthread+0x1d1/0x200 [13340.641288] ? set_kthread_struct+0x70/0x70 [13340.644224] ret_from_fork+0x1f/0x30 [13356.058009] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13356.061168] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13356.080556] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13356.085908] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13356.089578] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13356.092532] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13356.105280] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13356.110062] RAX: 0000000000001f6b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13356.113126] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13356.116489] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13356.119613] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13356.122549] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13356.125549] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13356.128439] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13356.130184] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13356.132261] Call Trace: [13356.133086] [13356.133845] ? show_regs.cold.9+0x22/0x2f [13356.135306] ? watchdog_timer_fn.cold.6+0x62/0x10a [13356.136869] ? __hrtimer_run_queues+0x1c2/0x4d0 [13356.138277] ? watchdog+0x40/0x40 [13356.139542] ? hrtimer_interrupt+0x128/0x2c0 [13356.140846] ? smp_apic_timer_interrupt+0xa6/0x270 [13356.142242] ? apic_timer_interrupt+0xf/0x20 [13356.144159] [13356.145148] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13356.147202] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13356.149317] do_raw_spin_lock+0x177/0x190 [13356.150396] _raw_spin_lock+0x21/0x30 [13356.151367] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13356.153513] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13356.155451] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13356.157429] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13356.159274] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13356.161483] ? woken_wake_function+0x30/0x30 [13356.162918] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13356.164950] kthread+0x1d1/0x200 [13356.165879] ? set_kthread_struct+0x70/0x70 [13356.167013] ret_from_fork+0x1f/0x30 [13360.052012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [13360.054941] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13360.073635] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13360.078224] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13360.081957] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13360.085114] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13360.087424] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13360.095389] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13360.098301] RAX: 0000000000001217 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13360.101094] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13360.103965] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13360.106988] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13360.109973] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13360.113123] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13360.116891] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13360.119182] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13360.121605] Call Trace: [13360.122369] [13360.123007] ? show_regs.cold.9+0x22/0x2f [13360.124312] ? watchdog_timer_fn.cold.6+0x62/0x10a [13360.125830] ? __hrtimer_run_queues+0x1c2/0x4d0 [13360.127584] ? watchdog+0x40/0x40 [13360.130158] ? hrtimer_interrupt+0x128/0x2c0 [13360.131826] ? smp_apic_timer_interrupt+0xa6/0x270 [13360.133425] ? apic_timer_interrupt+0xf/0x20 [13360.134817] [13360.135496] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13360.137020] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13360.138992] ? newidle_balance.isra.93+0x207/0x580 [13360.140721] do_raw_spin_lock+0x177/0x190 [13360.142291] _raw_spin_lock+0x21/0x30 [13360.143430] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13360.145241] process_one_work+0x2c8/0x700 [13360.146475] worker_thread+0x296/0x6e0 [13360.147739] ? rescuer_thread+0x570/0x570 [13360.151942] kthread+0x1d1/0x200 [13360.153989] ? set_kthread_struct+0x70/0x70 [13360.155925] ret_from_fork+0x1f/0x30 [13364.192032] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10727s! [13364.195344] Showing busy workqueues and worker pools: [13364.197288] workqueue events: flags=0x0 [13364.199011] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13364.202036] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13364.207000] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13364.209568] pending: cpu_latency_work [ptlrpc] [13364.213066] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13364.215558] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13364.219104] workqueue events_long: flags=0x0 [13364.220971] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13364.223919] pending: lu_objects_flush [obdclass] [13364.225913] workqueue events_unbound: flags=0x2 [13364.227493] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13364.229532] pending: flush_memcg_stats_dwork, idle_cull_fn [13364.231612] workqueue events_power_efficient: flags=0x80 [13364.238242] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13364.241602] pending: check_lifetime, do_cache_clean [13364.244310] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13364.249308] pending: neigh_periodic_work, neigh_periodic_work [13364.252931] workqueue mm_percpu_wq: flags=0x8 [13364.263156] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13364.270869] pending: vmstat_update, lru_add_drain_per_cpu [13364.273486] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13364.278848] pending: vmstat_update, lru_add_drain_per_cpu [13364.281741] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13364.287089] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13364.297978] workqueue xprtiod: flags=0xa [13364.300885] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13364.307568] pending: xprt_autoclose, xs_error_handle [13364.312836] workqueue ipv6_addrconf: flags=0x40008 [13364.316769] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13364.319616] pending: addrconf_verify_work [13364.321597] workqueue knbd0-recv: flags=0x1a [13364.323598] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13364.328895] in-flight: 651:recv_work [13364.330677] workqueue ptlrpc_pinger: flags=0x2 [13364.332690] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13364.341414] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13364.344874] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10727s workers=4 idle: 15979 31824 8 [13364.349574] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13354s workers=2 idle: 654 [13368.055081] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [13368.057928] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13368.058240] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [13368.082185] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13368.082202] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13368.082205] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13368.082222] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13368.082226] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13368.082232] RAX: 0000000000006f85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13368.082234] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13368.082236] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13368.082238] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13368.082240] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13368.082243] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13368.082246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13368.082248] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13368.082256] Call Trace: [13368.082261] [13368.082268] ? show_regs.cold.9+0x22/0x2f [13368.082278] ? watchdog_timer_fn.cold.6+0x62/0x10a [13368.082284] ? __hrtimer_run_queues+0x1c2/0x4d0 [13368.082289] ? watchdog+0x40/0x40 [13368.082293] ? hrtimer_interrupt+0x128/0x2c0 [13368.082297] ? smp_apic_timer_interrupt+0xa6/0x270 [13368.082307] ? apic_timer_interrupt+0xf/0x20 [13368.082311] [13368.082314] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13368.082321] do_raw_spin_lock+0x177/0x190 [13368.082325] _raw_spin_lock+0x21/0x30 [13368.082330] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13368.082678] ? do_raw_spin_unlock+0x75/0x190 [13368.082684] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13368.087999] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13368.088089] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13368.088096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13368.088099] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13368.088115] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13368.088120] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13368.088124] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13368.088126] RDX: 00000000000074e5 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13368.088129] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13368.088131] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13368.088133] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13368.088137] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13368.088140] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13368.088142] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13368.088150] Call Trace: [13368.088156] [13368.088161] ? show_regs.cold.9+0x22/0x2f [13368.088171] ? watchdog_timer_fn.cold.6+0x62/0x10a [13368.088178] ? __hrtimer_run_queues+0x1c2/0x4d0 [13368.088183] ? watchdog+0x40/0x40 [13368.088187] ? hrtimer_interrupt+0x128/0x2c0 [13368.088192] ? smp_apic_timer_interrupt+0xa6/0x270 [13368.088203] ? apic_timer_interrupt+0xf/0x20 [13368.088208] [13368.088210] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13368.088214] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13368.088219] do_raw_spin_lock+0x177/0x190 [13368.088224] _raw_spin_lock+0x21/0x30 [13368.088230] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13368.088602] obd_mod_free+0x46/0x140 [ptlrpc] [13368.088812] mdc_commit_open+0x91/0xb0 [mdc] [13368.088849] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13368.089098] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13368.089294] after_reply+0xe2c/0x1780 [ptlrpc] [13368.089562] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13368.089756] ? __next_timer_interrupt+0x160/0x160 [13368.089766] ptlrpcd+0xa82/0xe40 [ptlrpc] [13368.094627] ? do_wait_intr+0xf0/0xf0 [13368.094652] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13368.095660] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13368.095892] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13368.096194] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13368.096311] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13368.096349] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13368.096378] mdc_intent_lock+0x353/0x7e0 [mdc] [13368.096480] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13368.096567] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13368.097003] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13368.097183] lmv_intent_open+0x338/0x1150 [lmv] [13368.097228] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13368.097334] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13368.097581] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13368.097605] ll_intent_lock+0xc5/0xaa0 [lustre] [13368.099587] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13368.104638] ? do_wait_intr+0xf0/0xf0 [13368.104658] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13368.104917] kthread+0x1d1/0x200 [13368.104924] ? set_kthread_struct+0x70/0x70 [13368.104928] ret_from_fork+0x1f/0x30 [13368.393570] ll_lookup_it+0xe0e/0x2270 [lustre] [13368.396913] ? do_raw_spin_unlock+0x75/0x190 [13368.398977] ? _raw_spin_unlock+0x12/0x30 [13368.402623] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13368.404354] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13368.406270] ll_atomic_open+0x36a/0xd70 [lustre] [13368.407702] lookup_open+0x356/0xb40 [13368.409026] do_last+0x313/0xfc0 [13368.410239] ? nd_jump_root+0xe5/0x160 [13368.411762] ? path_init+0x437/0x520 [13368.413049] path_openat+0xf7/0x500 [13368.414517] ? do_raw_spin_unlock+0x75/0x190 [13368.417203] do_filp_open+0x99/0x140 [13368.422082] ? getname_flags+0x6e/0x330 [13368.425594] ? __check_object_size+0xff/0x256 [13368.428750] ? do_raw_spin_unlock+0x75/0x190 [13368.430007] ? _raw_spin_unlock+0x12/0x30 [13368.431257] do_sys_openat2+0x2b4/0x410 [13368.432139] do_sys_open+0x73/0xa0 [13368.433272] __x64_sys_openat+0x24/0x30 [13368.437301] do_syscall_64+0xc1/0x440 [13368.438857] entry_SYSCALL_64_after_hwframe+0x49/0xae [13368.441072] RIP: 0033:0x7f53425a5332 [13368.443657] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13368.453790] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13368.459595] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13368.463037] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13368.467085] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13368.469721] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13368.473277] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13384.058022] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13384.065384] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13384.107358] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13384.120833] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13384.130096] RIP: 0010:native_safe_halt+0x2e/0x50 [13384.133215] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13384.145300] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13384.151415] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13384.162587] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13384.171024] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13384.176814] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13384.186782] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13384.194346] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13384.198550] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13384.203464] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13384.209913] Call Trace: [13384.212322] [13384.214560] ? show_regs.cold.9+0x22/0x2f [13384.216046] ? watchdog_timer_fn.cold.6+0x62/0x10a [13384.221263] ? __hrtimer_run_queues+0x1c2/0x4d0 [13384.228290] ? watchdog+0x40/0x40 [13384.231653] ? hrtimer_interrupt+0x128/0x2c0 [13384.234484] ? smp_apic_timer_interrupt+0xa6/0x270 [13384.238238] ? apic_timer_interrupt+0xf/0x20 [13384.239669] [13384.240588] ? native_safe_halt+0x2e/0x50 [13384.241977] kvm_wait+0xcd/0xe0 [13384.243191] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13384.244927] do_raw_spin_lock+0x177/0x190 [13384.246553] _raw_spin_lock+0x21/0x30 [13384.249383] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13384.252623] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13384.256950] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13384.259678] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13384.262340] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13384.263912] ? woken_wake_function+0x30/0x30 [13384.266240] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13384.268039] kthread+0x1d1/0x200 [13384.268835] ? set_kthread_struct+0x70/0x70 [13384.271228] ret_from_fork+0x1f/0x30 [13392.053801] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [13392.060294] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13392.087636] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13392.097943] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13392.106670] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13392.111596] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13392.115728] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13392.128740] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13392.142511] RAX: 0000000000006784 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13392.165770] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13392.174488] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13392.179498] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13392.193849] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13392.200684] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13392.208410] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13392.213654] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13392.216655] Call Trace: [13392.217545] [13392.219634] ? show_regs.cold.9+0x22/0x2f [13392.225545] ? watchdog_timer_fn.cold.6+0x62/0x10a [13392.230888] ? __hrtimer_run_queues+0x1c2/0x4d0 [13392.233258] ? watchdog+0x40/0x40 [13392.235525] ? hrtimer_interrupt+0x128/0x2c0 [13392.238802] ? smp_apic_timer_interrupt+0xa6/0x270 [13392.241115] ? apic_timer_interrupt+0xf/0x20 [13392.242566] [13392.243279] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13392.252550] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13392.254638] ? newidle_balance.isra.93+0x207/0x580 [13392.257820] do_raw_spin_lock+0x177/0x190 [13392.262899] _raw_spin_lock+0x21/0x30 [13392.264141] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13392.271176] process_one_work+0x2c8/0x700 [13392.272557] worker_thread+0x296/0x6e0 [13392.283538] ? rescuer_thread+0x570/0x570 [13392.285050] kthread+0x1d1/0x200 [13392.287344] ? set_kthread_struct+0x70/0x70 [13392.291860] ret_from_fork+0x1f/0x30 [13394.403171] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10757s! [13394.407436] Showing busy workqueues and worker pools: [13394.409302] workqueue events: flags=0x0 [13394.410826] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13394.413164] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13394.418360] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13394.422389] pending: cpu_latency_work [ptlrpc] [13394.425049] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13394.428030] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13394.436375] workqueue events_long: flags=0x0 [13394.441031] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13394.450837] pending: lu_objects_flush [obdclass] [13394.454601] workqueue events_unbound: flags=0x2 [13394.457109] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13394.461904] pending: flush_memcg_stats_dwork, idle_cull_fn [13394.466681] workqueue events_power_efficient: flags=0x80 [13394.469749] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13394.474790] pending: check_lifetime, do_cache_clean [13394.476849] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13394.489873] pending: neigh_periodic_work, neigh_periodic_work [13394.496460] workqueue mm_percpu_wq: flags=0x8 [13394.501888] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13394.504329] pending: vmstat_update, lru_add_drain_per_cpu [13394.515457] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13394.519188] pending: vmstat_update, lru_add_drain_per_cpu [13394.521644] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13394.537222] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13394.542996] workqueue xprtiod: flags=0xa [13394.545365] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13394.548875] pending: xprt_autoclose, xs_error_handle [13394.551825] workqueue ipv6_addrconf: flags=0x40008 [13394.554729] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13394.561708] pending: addrconf_verify_work [13394.564588] workqueue knbd0-recv: flags=0x1a [13394.566607] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13394.570171] in-flight: 651:recv_work [13394.572405] workqueue ptlrpc_pinger: flags=0x2 [13394.575868] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13394.583711] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13394.591057] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10757s workers=4 idle: 15979 31824 8 [13394.598891] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13385s workers=2 idle: 654 [13396.056383] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [13396.058197] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [13396.060384] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13396.060461] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13396.060469] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13396.060471] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13396.060484] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13396.060488] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13396.060493] RAX: 000000000000283a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13396.060495] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13396.060497] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13396.060499] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13396.060502] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13396.060505] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13396.060507] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13396.060509] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13396.060519] Call Trace: [13396.060524] [13396.060530] ? show_regs.cold.9+0x22/0x2f [13396.060540] ? watchdog_timer_fn.cold.6+0x62/0x10a [13396.060545] ? __hrtimer_run_queues+0x1c2/0x4d0 [13396.060549] ? watchdog+0x40/0x40 [13396.060553] ? hrtimer_interrupt+0x128/0x2c0 [13396.060557] ? smp_apic_timer_interrupt+0xa6/0x270 [13396.060564] ? apic_timer_interrupt+0xf/0x20 [13396.060569] [13396.060572] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13396.060578] do_raw_spin_lock+0x177/0x190 [13396.060583] _raw_spin_lock+0x21/0x30 [13396.060589] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13396.061315] ? do_raw_spin_unlock+0x75/0x190 [13396.061324] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13396.061534] ? do_wait_intr+0xf0/0xf0 [13396.061542] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13396.064294] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13396.064574] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13396.069423] Modules linked in: lustre(O) [13396.088462] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13396.088583] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13396.088621] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13396.088667] mdc_intent_lock+0x353/0x7e0 [mdc] [13396.088695] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13396.088809] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13396.089310] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13396.089341] lmv_intent_open+0x338/0x1150 [lmv] [13396.089372] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13396.089466] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13396.089555] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13396.089578] ll_intent_lock+0xc5/0xaa0 [lustre] [13396.089689] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13396.095372] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) [13396.098690] ll_lookup_it+0xe0e/0x2270 [lustre] [13396.099260] ? do_raw_spin_unlock+0x75/0x190 [13396.099269] ? _raw_spin_unlock+0x12/0x30 [13396.099276] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13396.099304] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13396.099325] ll_atomic_open+0x36a/0xd70 [lustre] [13396.099413] lookup_open+0x356/0xb40 [13396.099423] do_last+0x313/0xfc0 [13396.099428] ? nd_jump_root+0xe5/0x160 [13396.099433] ? path_init+0x437/0x520 [13396.099438] path_openat+0xf7/0x500 [13396.099443] ? do_raw_spin_unlock+0x75/0x190 [13396.099448] do_filp_open+0x99/0x140 [13396.099455] ? getname_flags+0x6e/0x330 [13396.099460] ? __check_object_size+0xff/0x256 [13396.099466] ? do_raw_spin_unlock+0x75/0x190 [13396.099470] ? _raw_spin_unlock+0x12/0x30 [13396.099475] do_sys_openat2+0x2b4/0x410 [13396.099479] do_sys_open+0x73/0xa0 [13396.099484] __x64_sys_openat+0x24/0x30 [13396.099488] do_syscall_64+0xc1/0x440 [13396.099496] entry_SYSCALL_64_after_hwframe+0x49/0xae [13396.099502] RIP: 0033:0x7f53425a5332 [13396.099510] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13396.099513] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13396.099519] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13396.099521] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13396.099523] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13396.099525] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13396.099528] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13396.315367] lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13396.330815] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13396.335788] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13396.339295] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13396.343160] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13396.349630] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13396.351854] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13396.354328] RDX: 000000000000041b RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13396.356718] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13396.359129] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13396.361244] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13396.365066] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13396.367676] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13396.369667] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13396.372222] Call Trace: [13396.373011] [13396.373504] ? show_regs.cold.9+0x22/0x2f [13396.375077] ? watchdog_timer_fn.cold.6+0x62/0x10a [13396.376395] ? __hrtimer_run_queues+0x1c2/0x4d0 [13396.377743] ? watchdog+0x40/0x40 [13396.378774] ? hrtimer_interrupt+0x128/0x2c0 [13396.380458] ? smp_apic_timer_interrupt+0xa6/0x270 [13396.382003] ? apic_timer_interrupt+0xf/0x20 [13396.383330] [13396.383828] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13396.386158] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13396.387712] do_raw_spin_lock+0x177/0x190 [13396.389236] _raw_spin_lock+0x21/0x30 [13396.390392] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13396.392081] obd_mod_free+0x46/0x140 [ptlrpc] [13396.394294] mdc_commit_open+0x91/0xb0 [mdc] [13396.395534] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13396.397665] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13396.399850] after_reply+0xe2c/0x1780 [ptlrpc] [13396.401684] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13396.403388] ? __next_timer_interrupt+0x160/0x160 [13396.404736] ptlrpcd+0xa82/0xe40 [ptlrpc] [13396.406707] ? do_wait_intr+0xf0/0xf0 [13396.408043] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13396.410210] kthread+0x1d1/0x200 [13396.411371] ? set_kthread_struct+0x70/0x70 [13396.412775] ret_from_fork+0x1f/0x30 [13412.059983] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13412.068826] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13412.109297] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13412.116536] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13412.124563] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13412.130162] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13412.141919] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [13412.146417] RAX: 0000000000002689 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13412.152162] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13412.159867] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13412.164846] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13412.169559] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13412.177864] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13412.184168] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13412.187897] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13412.190444] Call Trace: [13412.192530] [13412.195188] ? show_regs.cold.9+0x22/0x2f [13412.205858] ? watchdog_timer_fn.cold.6+0x62/0x10a [13412.208039] ? __hrtimer_run_queues+0x1c2/0x4d0 [13412.211115] ? watchdog+0x40/0x40 [13412.215283] ? hrtimer_interrupt+0x128/0x2c0 [13412.219655] ? smp_apic_timer_interrupt+0xa6/0x270 [13412.223105] ? apic_timer_interrupt+0xf/0x20 [13412.225061] [13412.226144] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13412.228461] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13412.233811] do_raw_spin_lock+0x177/0x190 [13412.234969] _raw_spin_lock+0x21/0x30 [13412.241902] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13412.244036] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13412.251940] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13412.257977] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13412.260023] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13412.274835] ? woken_wake_function+0x30/0x30 [13412.279073] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13412.285040] kthread+0x1d1/0x200 [13412.286128] ? set_kthread_struct+0x70/0x70 [13412.292267] ret_from_fork+0x1f/0x30 [13420.052105] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [13420.056792] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13420.089107] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13420.095318] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13420.098994] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13420.102199] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13420.105334] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13420.113496] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13420.116666] RAX: 000000000000400a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13420.119615] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13420.122334] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13420.125630] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13420.129017] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13420.132016] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13420.136611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13420.141003] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13420.146660] Call Trace: [13420.147712] [13420.149926] ? show_regs.cold.9+0x22/0x2f [13420.151169] ? watchdog_timer_fn.cold.6+0x62/0x10a [13420.152352] ? __hrtimer_run_queues+0x1c2/0x4d0 [13420.158102] ? watchdog+0x40/0x40 [13420.164095] ? hrtimer_interrupt+0x128/0x2c0 [13420.166921] ? smp_apic_timer_interrupt+0xa6/0x270 [13420.171641] ? apic_timer_interrupt+0xf/0x20 [13420.176088] [13420.178028] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13420.180994] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13420.183237] ? newidle_balance.isra.93+0x207/0x580 [13420.190431] do_raw_spin_lock+0x177/0x190 [13420.192679] _raw_spin_lock+0x21/0x30 [13420.194630] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13420.198134] process_one_work+0x2c8/0x700 [13420.199388] worker_thread+0x296/0x6e0 [13420.201938] ? rescuer_thread+0x570/0x570 [13420.203935] kthread+0x1d1/0x200 [13420.205805] ? set_kthread_struct+0x70/0x70 [13420.211493] ret_from_fork+0x1f/0x30 [13424.055762] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [13424.057184] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [13424.063853] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) [13424.068731] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13424.068817] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13424.068825] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13424.068828] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13424.068843] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13424.068848] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13424.068852] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13424.068855] RDX: 000000000000329a RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13424.068857] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13424.068859] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13424.068861] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13424.068864] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13424.068867] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13424.068869] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13424.068879] Call Trace: [13424.068885] [13424.068891] ? show_regs.cold.9+0x22/0x2f [13424.068901] ? watchdog_timer_fn.cold.6+0x62/0x10a [13424.068907] ? __hrtimer_run_queues+0x1c2/0x4d0 [13424.068912] ? watchdog+0x40/0x40 [13424.068916] ? hrtimer_interrupt+0x128/0x2c0 [13424.068921] ? smp_apic_timer_interrupt+0xa6/0x270 [13424.068929] ? apic_timer_interrupt+0xf/0x20 [13424.068934] [13424.068937] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13424.068942] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13424.068948] do_raw_spin_lock+0x177/0x190 [13424.068952] _raw_spin_lock+0x21/0x30 [13424.068958] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13424.069776] obd_mod_free+0x46/0x140 [ptlrpc] [13424.069988] mdc_commit_open+0x91/0xb0 [mdc] [13424.070029] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13424.070608] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13424.070829] after_reply+0xe2c/0x1780 [ptlrpc] [13424.071035] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13424.073604] ? __next_timer_interrupt+0x160/0x160 [13424.073616] ptlrpcd+0xa82/0xe40 [ptlrpc] [13424.073869] ? do_wait_intr+0xf0/0xf0 [13424.073879] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13424.074097] kthread+0x1d1/0x200 [13424.074103] ? set_kthread_struct+0x70/0x70 [13424.074107] ret_from_fork+0x1f/0x30 [13424.358997] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13424.404262] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13424.415637] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13424.422379] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13424.425600] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13424.438815] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13424.448083] RAX: 0000000000005a18 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13424.452916] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13424.458829] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13424.468134] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13424.473079] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13424.483941] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13424.495917] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13424.502273] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13424.511185] Call Trace: [13424.513340] [13424.514126] ? show_regs.cold.9+0x22/0x2f [13424.516305] ? watchdog_timer_fn.cold.6+0x62/0x10a [13424.520981] ? __hrtimer_run_queues+0x1c2/0x4d0 [13424.526102] ? watchdog+0x40/0x40 [13424.529926] ? hrtimer_interrupt+0x128/0x2c0 [13424.534816] ? smp_apic_timer_interrupt+0xa6/0x270 [13424.539929] ? apic_timer_interrupt+0xf/0x20 [13424.546536] [13424.547230] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13424.549088] do_raw_spin_lock+0x177/0x190 [13424.552913] _raw_spin_lock+0x21/0x30 [13424.555287] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13424.561090] ? do_raw_spin_unlock+0x75/0x190 [13424.566526] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13424.570263] ? do_wait_intr+0xf0/0xf0 [13424.574557] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13424.578824] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13424.583944] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13424.585643] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13424.593015] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13424.598324] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13424.600377] mdc_intent_lock+0x353/0x7e0 [mdc] [13424.606141] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13424.607183] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10787s! [13424.610922] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13424.614552] Showing busy workqueues and worker pools: [13424.614560] workqueue events: flags=0x0 [13424.614566] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13424.614578] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13424.615486] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13424.615498] pending: cpu_latency_work [ptlrpc] [13424.615811] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13424.615819] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13424.616170] workqueue events_long: flags=0x0 [13424.616174] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13424.616181] pending: lu_objects_flush [obdclass] [13424.616385] workqueue events_unbound: flags=0x2 [13424.616388] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13424.616405] pending: flush_memcg_stats_dwork, idle_cull_fn [13424.616418] workqueue events_power_efficient: flags=0x80 [13424.616421] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13424.616428] pending: check_lifetime, do_cache_clean [13424.616441] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13424.616451] pending: neigh_periodic_work, neigh_periodic_work [13424.616464] workqueue mm_percpu_wq: flags=0x8 [13424.616467] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13424.616473] pending: vmstat_update, lru_add_drain_per_cpu [13424.616483] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13424.616490] pending: vmstat_update, lru_add_drain_per_cpu [13424.616497] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13424.616503] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13424.616533] workqueue xprtiod: flags=0xa [13424.621745] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13424.621799] lmv_intent_open+0x338/0x1150 [lmv] [13424.621830] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13424.621954] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13424.624354] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13424.624376] pending: xprt_autoclose, xs_error_handle [13424.624408] workqueue ipv6_addrconf: flags=0x40008 [13424.624413] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13424.624420] pending: addrconf_verify_work [13424.624438] workqueue knbd0-recv: flags=0x1a [13424.624441] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13424.624448] in-flight: 651:recv_work [13424.624459] workqueue ptlrpc_pinger: flags=0x2 [13424.624461] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13424.624467] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13424.624779] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10787s workers=4 idle: 15979 31824 8 [13424.624796] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13415s workers=2 idle: 654 [13424.825157] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13424.829166] ll_intent_lock+0xc5/0xaa0 [lustre] [13424.830852] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13424.844984] ll_lookup_it+0xe0e/0x2270 [lustre] [13424.848099] ? do_raw_spin_unlock+0x75/0x190 [13424.855811] ? _raw_spin_unlock+0x12/0x30 [13424.857222] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13424.863545] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13424.866380] ll_atomic_open+0x36a/0xd70 [lustre] [13424.872791] lookup_open+0x356/0xb40 [13424.877329] do_last+0x313/0xfc0 [13424.884687] ? nd_jump_root+0xe5/0x160 [13424.889783] ? path_init+0x437/0x520 [13424.890838] path_openat+0xf7/0x500 [13424.892097] ? do_raw_spin_unlock+0x75/0x190 [13424.906356] do_filp_open+0x99/0x140 [13424.907613] ? getname_flags+0x6e/0x330 [13424.909886] ? __check_object_size+0xff/0x256 [13424.918663] ? do_raw_spin_unlock+0x75/0x190 [13424.921210] ? _raw_spin_unlock+0x12/0x30 [13424.923028] do_sys_openat2+0x2b4/0x410 [13424.925147] do_sys_open+0x73/0xa0 [13424.926248] __x64_sys_openat+0x24/0x30 [13424.927749] do_syscall_64+0xc1/0x440 [13424.929114] entry_SYSCALL_64_after_hwframe+0x49/0xae [13424.930836] RIP: 0033:0x7f53425a5332 [13424.932051] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13424.939170] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13424.942162] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13424.945138] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13424.948220] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13424.952957] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13424.955799] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13440.058083] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [13440.062799] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13440.090498] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13440.096636] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13440.100762] RIP: 0010:native_safe_halt+0x2e/0x50 [13440.103249] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13440.113802] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13440.117719] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13440.120943] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13440.124571] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13440.127698] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13440.130879] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13440.134388] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13440.138070] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13440.140531] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13440.143768] Call Trace: [13440.144976] [13440.145883] ? show_regs.cold.9+0x22/0x2f [13440.147733] ? watchdog_timer_fn.cold.6+0x62/0x10a [13440.149965] ? __hrtimer_run_queues+0x1c2/0x4d0 [13440.152916] ? watchdog+0x40/0x40 [13440.154469] ? hrtimer_interrupt+0x128/0x2c0 [13440.156380] ? smp_apic_timer_interrupt+0xa6/0x270 [13440.158685] ? apic_timer_interrupt+0xf/0x20 [13440.160401] [13440.161413] ? native_safe_halt+0x2e/0x50 [13440.162934] kvm_wait+0xcd/0xe0 [13440.164554] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13440.166809] do_raw_spin_lock+0x177/0x190 [13440.168771] _raw_spin_lock+0x21/0x30 [13440.170505] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13440.173455] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13440.176525] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13440.179487] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13440.181873] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13440.183952] ? woken_wake_function+0x30/0x30 [13440.185774] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13440.188890] kthread+0x1d1/0x200 [13440.190197] ? set_kthread_struct+0x70/0x70 [13440.191647] ret_from_fork+0x1f/0x30 [13452.053720] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [13452.056196] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [13452.057385] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [13452.057394] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13452.057550] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13452.057556] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13452.057559] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13452.057572] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13452.057576] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13452.057581] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13452.057584] RDX: 0000000000005c55 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13452.057586] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13452.057589] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13452.057591] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13452.057594] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13452.057597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13452.057600] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13452.057608] Call Trace: [13452.057613] [13452.057619] ? show_regs.cold.9+0x22/0x2f [13452.057628] ? watchdog_timer_fn.cold.6+0x62/0x10a [13452.057634] ? __hrtimer_run_queues+0x1c2/0x4d0 [13452.057638] ? watchdog+0x40/0x40 [13452.057643] ? hrtimer_interrupt+0x128/0x2c0 [13452.057647] ? smp_apic_timer_interrupt+0xa6/0x270 [13452.057655] ? apic_timer_interrupt+0xf/0x20 [13452.057659] [13452.057662] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13452.057667] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13452.057672] do_raw_spin_lock+0x177/0x190 [13452.057676] _raw_spin_lock+0x21/0x30 [13452.057682] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13452.057993] obd_mod_free+0x46/0x140 [ptlrpc] [13452.058981] mdc_commit_open+0x91/0xb0 [mdc] [13452.059016] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13452.059270] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel [13452.059247] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13452.059421] ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13452.059436] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13452.059442] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13452.059547] after_reply+0xe2c/0x1780 [ptlrpc] [13452.060213] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13452.060428] ? __next_timer_interrupt+0x160/0x160 [13452.060437] ptlrpcd+0xa82/0xe40 [ptlrpc] [13452.060700] ? do_wait_intr+0xf0/0xf0 [13452.059448] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13452.061058] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13452.061068] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13452.061073] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13452.061078] RAX: 000000000000760e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13452.061080] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13452.061083] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13452.061085] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13452.061088] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13452.061091] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13452.061094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13452.061096] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13452.061104] Call Trace: [13452.061108] [13452.060709] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13452.062824] kthread+0x1d1/0x200 [13452.062833] ? set_kthread_struct+0x70/0x70 [13452.062837] ret_from_fork+0x1f/0x30 [13452.069405] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13452.069485] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13452.069492] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13452.069495] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13452.069509] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13452.069513] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13452.069517] RAX: 0000000000007586 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13452.069520] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13452.069522] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13452.069524] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13452.069527] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13452.069531] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13452.069533] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13452.069536] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13452.069545] Call Trace: [13452.069550] [13452.069556] ? show_regs.cold.9+0x22/0x2f [13452.069564] ? watchdog_timer_fn.cold.6+0x62/0x10a [13452.069570] ? __hrtimer_run_queues+0x1c2/0x4d0 [13452.069574] ? watchdog+0x40/0x40 [13452.069577] ? hrtimer_interrupt+0x128/0x2c0 [13452.069581] ? smp_apic_timer_interrupt+0xa6/0x270 [13452.069588] ? apic_timer_interrupt+0xf/0x20 [13452.069593] [13452.069596] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13452.069603] do_raw_spin_lock+0x177/0x190 [13452.069608] _raw_spin_lock+0x21/0x30 [13452.069613] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13452.069874] ? do_raw_spin_unlock+0x75/0x190 [13452.069879] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13452.082473] ? show_regs.cold.9+0x22/0x2f [13452.082495] ? watchdog_timer_fn.cold.6+0x62/0x10a [13452.082502] ? __hrtimer_run_queues+0x1c2/0x4d0 [13452.082507] ? watchdog+0x40/0x40 [13452.082511] ? hrtimer_interrupt+0x128/0x2c0 [13452.082515] ? smp_apic_timer_interrupt+0xa6/0x270 [13452.082523] ? apic_timer_interrupt+0xf/0x20 [13452.082527] [13452.082533] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13452.082539] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13452.082543] ? newidle_balance.isra.93+0x207/0x580 [13452.082550] do_raw_spin_lock+0x177/0x190 [13452.082556] _raw_spin_lock+0x21/0x30 [13452.082561] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13452.082834] process_one_work+0x2c8/0x700 [13452.082841] worker_thread+0x296/0x6e0 [13452.082846] ? rescuer_thread+0x570/0x570 [13452.082849] kthread+0x1d1/0x200 [13452.082854] ? set_kthread_struct+0x70/0x70 [13452.082859] ret_from_fork+0x1f/0x30 [13452.474248] ? do_wait_intr+0xf0/0xf0 [13452.475468] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13452.477292] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13452.478813] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13452.480855] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13452.482574] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13452.484182] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13452.485519] mdc_intent_lock+0x353/0x7e0 [mdc] [13452.486781] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13452.488575] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13452.491872] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13452.493486] lmv_intent_open+0x338/0x1150 [lmv] [13452.495767] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13452.499510] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13452.502346] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13452.506337] ll_intent_lock+0xc5/0xaa0 [lustre] [13452.509230] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13452.510607] ll_lookup_it+0xe0e/0x2270 [lustre] [13452.511861] ? do_raw_spin_unlock+0x75/0x190 [13452.518049] ? _raw_spin_unlock+0x12/0x30 [13452.520539] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13452.525178] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13452.526947] ll_atomic_open+0x36a/0xd70 [lustre] [13452.528692] lookup_open+0x356/0xb40 [13452.532230] do_last+0x313/0xfc0 [13452.533248] ? nd_jump_root+0xe5/0x160 [13452.535408] ? path_init+0x437/0x520 [13452.536429] path_openat+0xf7/0x500 [13452.541118] ? do_raw_spin_unlock+0x75/0x190 [13452.543504] do_filp_open+0x99/0x140 [13452.544528] ? getname_flags+0x6e/0x330 [13452.550170] ? __check_object_size+0xff/0x256 [13452.553421] ? do_raw_spin_unlock+0x75/0x190 [13452.558780] ? _raw_spin_unlock+0x12/0x30 [13452.562414] do_sys_openat2+0x2b4/0x410 [13452.564186] do_sys_open+0x73/0xa0 [13452.565381] __x64_sys_openat+0x24/0x30 [13452.566755] do_syscall_64+0xc1/0x440 [13452.568180] entry_SYSCALL_64_after_hwframe+0x49/0xae [13452.569526] RIP: 0033:0x7f53425a5332 [13452.570974] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13452.577397] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13452.579870] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13452.582166] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13452.584531] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13452.591856] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13452.595872] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13454.817944] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10817s! [13454.823098] Showing busy workqueues and worker pools: [13454.826779] workqueue events: flags=0x0 [13454.832342] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13454.834376] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13454.842245] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13454.850377] pending: cpu_latency_work [ptlrpc] [13454.853455] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13454.860574] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13454.872053] workqueue events_long: flags=0x0 [13454.873992] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13454.880697] pending: lu_objects_flush [obdclass] [13454.887586] workqueue events_unbound: flags=0x2 [13454.891435] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13454.897781] pending: flush_memcg_stats_dwork, idle_cull_fn [13454.903039] workqueue events_power_efficient: flags=0x80 [13454.908242] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13454.911844] pending: check_lifetime, do_cache_clean [13454.917009] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13454.923656] pending: neigh_periodic_work, neigh_periodic_work [13454.926791] workqueue mm_percpu_wq: flags=0x8 [13454.933633] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13454.941764] pending: vmstat_update, lru_add_drain_per_cpu [13454.948550] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13454.952789] pending: vmstat_update, lru_add_drain_per_cpu [13454.960584] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13454.965263] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13454.971184] workqueue xprtiod: flags=0xa [13454.976498] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13454.982442] pending: xprt_autoclose, xs_error_handle [13454.984011] workqueue ipv6_addrconf: flags=0x40008 [13454.987368] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13454.995134] pending: addrconf_verify_work [13454.997505] workqueue knbd0-recv: flags=0x1a [13455.002699] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13455.009097] in-flight: 651:recv_work [13455.012171] workqueue ptlrpc_pinger: flags=0x2 [13455.015827] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13455.019062] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13455.025514] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10817s workers=4 idle: 15979 31824 8 [13455.031304] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13445s workers=2 idle: 654 [13468.058788] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [13468.064411] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13468.111397] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13468.120346] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13468.126225] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13468.128820] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13468.145541] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13468.153325] RAX: 0000000000007e84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13468.161760] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13468.167383] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13468.174830] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13468.178816] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13468.184613] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13468.190082] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13468.193075] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13468.196373] Call Trace: [13468.197366] [13468.198072] ? show_regs.cold.9+0x22/0x2f [13468.199406] ? watchdog_timer_fn.cold.6+0x62/0x10a [13468.205568] ? __hrtimer_run_queues+0x1c2/0x4d0 [13468.208188] ? watchdog+0x40/0x40 [13468.209775] ? hrtimer_interrupt+0x128/0x2c0 [13468.212361] ? smp_apic_timer_interrupt+0xa6/0x270 [13468.214770] ? apic_timer_interrupt+0xf/0x20 [13468.217077] [13468.219466] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13468.223056] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13468.224981] do_raw_spin_lock+0x177/0x190 [13468.231261] _raw_spin_lock+0x21/0x30 [13468.235152] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13468.241747] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13468.244451] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13468.249552] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13468.252348] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13468.256439] ? woken_wake_function+0x30/0x30 [13468.260199] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13468.265227] kthread+0x1d1/0x200 [13468.266805] ? set_kthread_struct+0x70/0x70 [13468.271174] ret_from_fork+0x1f/0x30 [13480.052573] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [13480.056941] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13480.057077] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [13480.057083] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13480.057144] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13480.057149] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13480.057151] RIP: 0010:native_safe_halt+0x2e/0x50 [13480.057164] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13480.057169] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000212 [13480.057978] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [13480.057983] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13480.058027] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13480.058032] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13480.058035] RIP: 0010:native_safe_halt+0x2e/0x50 [13480.058050] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13480.058053] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13480.058060] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13480.058062] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13480.058065] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13480.058067] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13480.058070] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13480.058073] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13480.058076] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13480.058078] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13480.058086] Call Trace: [13480.058094] [13480.058099] ? show_regs.cold.9+0x22/0x2f [13480.058108] ? watchdog_timer_fn.cold.6+0x62/0x10a [13480.058114] ? __hrtimer_run_queues+0x1c2/0x4d0 [13480.058134] ? watchdog+0x40/0x40 [13480.058138] ? hrtimer_interrupt+0x128/0x2c0 [13480.058142] ? smp_apic_timer_interrupt+0xa6/0x270 [13480.058147] ? apic_timer_interrupt+0xf/0x20 [13480.058151] [13480.058154] ? native_safe_halt+0x2e/0x50 [13480.058157] kvm_wait+0xcd/0xe0 [13480.058163] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13480.058171] do_raw_spin_lock+0x177/0x190 [13480.058174] _raw_spin_lock+0x21/0x30 [13480.058178] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13480.059487] ? do_raw_spin_unlock+0x75/0x190 [13480.059496] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13480.059825] ? do_wait_intr+0xf0/0xf0 [13480.060610] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13480.060853] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13480.061261] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13480.061580] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13480.061703] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13480.061751] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13480.061780] mdc_intent_lock+0x353/0x7e0 [mdc] [13480.061797] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13480.061910] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13480.062361] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13480.062393] lmv_intent_open+0x338/0x1150 [lmv] [13480.062433] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13480.062779] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13480.063324] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13480.063348] ll_intent_lock+0xc5/0xaa0 [lustre] [13480.063454] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13480.063601] ll_lookup_it+0xe0e/0x2270 [lustre] [13480.063688] ? do_raw_spin_unlock+0x75/0x190 [13480.063696] ? _raw_spin_unlock+0x12/0x30 [13480.063702] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13480.063734] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13480.063755] ll_atomic_open+0x36a/0xd70 [lustre] [13480.063812] lookup_open+0x356/0xb40 [13480.063820] do_last+0x313/0xfc0 [13480.063824] ? nd_jump_root+0xe5/0x160 [13480.063949] ? path_init+0x437/0x520 [13480.063953] path_openat+0xf7/0x500 [13480.063957] ? do_raw_spin_unlock+0x75/0x190 [13480.063960] do_filp_open+0x99/0x140 [13480.063964] ? getname_flags+0x6e/0x330 [13480.063967] ? __check_object_size+0xff/0x256 [13480.063975] ? do_raw_spin_unlock+0x75/0x190 [13480.063977] ? _raw_spin_unlock+0x12/0x30 [13480.063980] do_sys_openat2+0x2b4/0x410 [13480.063983] do_sys_open+0x73/0xa0 [13480.063986] __x64_sys_openat+0x24/0x30 [13480.063988] do_syscall_64+0xc1/0x440 [13480.063996] entry_SYSCALL_64_after_hwframe+0x49/0xae [13480.064003] RIP: 0033:0x7f53425a5332 [13480.064009] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13480.064012] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13480.064016] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13480.064018] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13480.064491] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13480.064493] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13480.064496] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13480.077878] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13480.081479] ORIG_RAX: ffffffffffffff13 [13480.081490] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13480.081494] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13480.103875] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13480.103891] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13480.104276] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13480.104290] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13480.104293] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13480.104297] RAX: 00000000000032d2 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13480.104300] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13480.104302] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13480.104304] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13480.104306] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13480.104309] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13480.104312] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13480.104314] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13480.104323] Call Trace: [13480.104329] [13480.104334] ? show_regs.cold.9+0x22/0x2f [13480.104343] ? watchdog_timer_fn.cold.6+0x62/0x10a [13480.104350] ? __hrtimer_run_queues+0x1c2/0x4d0 [13480.104355] ? watchdog+0x40/0x40 [13480.104360] ? hrtimer_interrupt+0x128/0x2c0 [13480.104363] ? smp_apic_timer_interrupt+0xa6/0x270 [13480.104372] ? apic_timer_interrupt+0xf/0x20 [13480.104377] [13480.104379] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13480.104384] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13480.104388] ? newidle_balance.isra.93+0x207/0x580 [13480.104393] do_raw_spin_lock+0x177/0x190 [13480.104398] _raw_spin_lock+0x21/0x30 [13480.104403] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13480.104807] process_one_work+0x2c8/0x700 [13480.104814] worker_thread+0x296/0x6e0 [13480.104818] ? rescuer_thread+0x570/0x570 [13480.104822] kthread+0x1d1/0x200 [13480.104827] ? set_kthread_struct+0x70/0x70 [13480.104830] ret_from_fork+0x1f/0x30 [13480.486588] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13480.492884] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13480.501849] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13480.507145] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13480.514815] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13480.518704] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13480.521748] Call Trace: [13480.522641] [13480.524862] ? show_regs.cold.9+0x22/0x2f [13480.526869] ? watchdog_timer_fn.cold.6+0x62/0x10a [13480.530835] ? __hrtimer_run_queues+0x1c2/0x4d0 [13480.532756] ? watchdog+0x40/0x40 [13480.534056] ? hrtimer_interrupt+0x128/0x2c0 [13480.535895] ? smp_apic_timer_interrupt+0xa6/0x270 [13480.539068] ? apic_timer_interrupt+0xf/0x20 [13480.540683] [13480.542294] ? native_safe_halt+0x2e/0x50 [13480.545696] kvm_wait+0xcd/0xe0 [13480.548030] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13480.554940] do_raw_spin_lock+0x177/0x190 [13480.560069] _raw_spin_lock+0x21/0x30 [13480.563704] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13480.567658] obd_mod_free+0x46/0x140 [ptlrpc] [13480.573374] mdc_commit_open+0x91/0xb0 [mdc] [13480.576351] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13480.584804] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13480.590163] after_reply+0xe2c/0x1780 [ptlrpc] [13480.593363] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13480.597956] ? __next_timer_interrupt+0x160/0x160 [13480.603461] ptlrpcd+0xa82/0xe40 [ptlrpc] [13480.608264] ? do_wait_intr+0xf0/0xf0 [13480.612378] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13480.616343] kthread+0x1d1/0x200 [13480.617171] ? set_kthread_struct+0x70/0x70 [13480.623478] ret_from_fork+0x1f/0x30 [13485.535126] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10848s! [13485.544024] Showing busy workqueues and worker pools: [13485.547043] workqueue events: flags=0x0 [13485.550672] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13485.553643] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13485.565821] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13485.572755] pending: cpu_latency_work [ptlrpc] [13485.579298] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13485.584449] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13485.593812] workqueue events_long: flags=0x0 [13485.595868] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13485.598425] pending: lu_objects_flush [obdclass] [13485.599853] workqueue events_unbound: flags=0x2 [13485.605010] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13485.608402] pending: flush_memcg_stats_dwork, idle_cull_fn [13485.613547] workqueue events_power_efficient: flags=0x80 [13485.616714] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13485.623097] pending: check_lifetime, do_cache_clean [13485.625507] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13485.632777] pending: neigh_periodic_work, neigh_periodic_work [13485.638673] workqueue mm_percpu_wq: flags=0x8 [13485.642274] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13485.648265] pending: vmstat_update, lru_add_drain_per_cpu [13485.653837] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13485.660225] pending: vmstat_update, lru_add_drain_per_cpu [13485.663025] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13485.674227] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13485.678137] workqueue xprtiod: flags=0xa [13485.679728] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13485.682073] pending: xprt_autoclose, xs_error_handle [13485.684063] workqueue ipv6_addrconf: flags=0x40008 [13485.685935] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13485.688661] pending: addrconf_verify_work [13485.691446] workqueue knbd0-recv: flags=0x1a [13485.693654] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13485.696301] in-flight: 651:recv_work [13485.698244] workqueue ptlrpc_pinger: flags=0x2 [13485.700148] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13485.702709] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13485.705388] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10848s workers=4 idle: 15979 31824 8 [13485.708526] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13476s workers=2 idle: 654 [13494.703874] rcu: INFO: rcu_sched self-detected stall on CPU [13494.710933] rcu: 2-....: (6347833 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [13494.722545] (t=11400277 jiffies g=274185 q=66785) [13494.728575] rcu: rcu_sched kthread starved for 10858643 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [13494.739709] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [13494.750017] rcu: RCU grace-period kthread stack dump: [13494.753580] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [13494.763498] Call Trace: [13494.766710] __schedule+0x351/0xcb0 [13494.770447] schedule+0xc0/0x180 [13494.772524] schedule_timeout+0xb4/0x190 [13494.779337] ? do_raw_spin_unlock+0x75/0x190 [13494.789945] ? __next_timer_interrupt+0x160/0x160 [13494.797336] ? prepare_to_swait_event+0xf3/0x140 [13494.806668] rcu_gp_kthread+0x8dd/0x15b0 [13494.810227] ? force_qs_rnp+0x2c0/0x2c0 [13494.812187] kthread+0x1d1/0x200 [13494.815410] ? set_kthread_struct+0x70/0x70 [13494.817735] ret_from_fork+0x1f/0x30 [13494.818877] rcu: Stack dump where RCU GP kthread last ran: [13494.822707] Sending NMI from CPU 2 to CPUs 1: [13494.829307] NMI backtrace for cpu 1 [13494.829316] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13494.829318] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13494.829319] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13494.829321] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13494.829324] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [13494.829373] RAX: 00000000000050b4 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13494.829375] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13494.829376] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13494.829378] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13494.829379] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13494.829381] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13494.829383] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13494.829384] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13494.829385] Call Trace: [13494.829391] [13494.829391] ? show_regs.cold.9+0x22/0x2f [13494.829392] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [13494.829393] ? nmi_cpu_backtrace_handler+0x15/0x40 [13494.829395] ? nmi_handle+0xbd/0x210 [13494.829396] ? default_do_nmi+0x4b/0x1b0 [13494.829397] ? do_nmi+0x280/0x380 [13494.829398] ? end_repeat_nmi+0x16/0x1f [13494.829402] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13494.829403] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13494.829404] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13494.829406] [13494.829406] do_raw_spin_lock+0x177/0x190 [13494.829408] _raw_spin_lock+0x21/0x30 [13494.829409] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13494.829410] ? do_raw_spin_unlock+0x75/0x190 [13494.829411] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13494.829412] ? do_wait_intr+0xf0/0xf0 [13494.829413] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13494.829415] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13494.829416] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13494.829417] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13494.829419] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13494.829420] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13494.829636] mdc_intent_lock+0x353/0x7e0 [mdc] [13494.829640] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13494.829641] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13494.829642] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13494.829643] lmv_intent_open+0x338/0x1150 [lmv] [13494.829644] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13494.829645] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13494.829646] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13494.829647] ll_intent_lock+0xc5/0xaa0 [lustre] [13494.829648] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13494.829649] ll_lookup_it+0xe0e/0x2270 [lustre] [13494.829650] ? do_raw_spin_unlock+0x75/0x190 [13494.829651] ? _raw_spin_unlock+0x12/0x30 [13494.829652] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13494.829653] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13494.829654] ll_atomic_open+0x36a/0xd70 [lustre] [13494.829659] lookup_open+0x356/0xb40 [13494.829660] do_last+0x313/0xfc0 [13494.829661] ? nd_jump_root+0xe5/0x160 [13494.829662] ? path_init+0x437/0x520 [13494.829663] path_openat+0xf7/0x500 [13494.829664] ? do_raw_spin_unlock+0x75/0x190 [13494.829665] do_filp_open+0x99/0x140 [13494.829666] ? getname_flags+0x6e/0x330 [13494.829667] ? __check_object_size+0xff/0x256 [13494.829669] ? do_raw_spin_unlock+0x75/0x190 [13494.829670] ? _raw_spin_unlock+0x12/0x30 [13494.829671] do_sys_openat2+0x2b4/0x410 [13494.829672] do_sys_open+0x73/0xa0 [13494.829673] __x64_sys_openat+0x24/0x30 [13494.829674] do_syscall_64+0xc1/0x440 [13494.829676] entry_SYSCALL_64_after_hwframe+0x49/0xae [13494.829789] RIP: 0033:0x7f53425a5332 [13494.829790] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13494.829792] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13494.829795] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13494.829796] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13494.829798] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13494.829799] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13494.829804] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13494.831688] Sending NMI from CPU 2 to CPUs 1: [13495.186694] NMI backtrace for cpu 1 [13495.186704] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13495.186706] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13495.186708] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13495.186710] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13495.186711] RSP: 0018:ffffbb5381503500 EFLAGS: 00000216 [13495.186714] RAX: 0000000000002d6d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13495.186716] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13495.186718] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13495.186719] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13495.186720] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13495.186722] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13495.186723] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13495.186724] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13495.186725] Call Trace: [13495.186726] [13495.186727] ? show_regs.cold.9+0x22/0x2f [13495.186728] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [13495.186730] ? nmi_cpu_backtrace_handler+0x15/0x40 [13495.186731] ? nmi_handle+0xbd/0x210 [13495.186732] ? default_do_nmi+0x4b/0x1b0 [13495.186733] ? do_nmi+0x280/0x380 [13495.186734] ? end_repeat_nmi+0x16/0x1f [13495.186735] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13495.186736] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13495.186738] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13495.186739] [13495.186740] do_raw_spin_lock+0x177/0x190 [13495.186741] _raw_spin_lock+0x21/0x30 [13495.186742] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13495.186744] ? do_raw_spin_unlock+0x75/0x190 [13495.186745] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13495.186746] ? do_wait_intr+0xf0/0xf0 [13495.186747] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13495.186748] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13495.186749] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13495.186750] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13495.186752] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13495.186753] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13495.186754] mdc_intent_lock+0x353/0x7e0 [mdc] [13495.186755] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13495.186756] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13495.186757] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13495.186758] lmv_intent_open+0x338/0x1150 [lmv] [13495.186759] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13495.186760] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13495.186761] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13495.186762] ll_intent_lock+0xc5/0xaa0 [lustre] [13495.186763] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13495.186764] ll_lookup_it+0xe0e/0x2270 [lustre] [13495.186765] ? do_raw_spin_unlock+0x75/0x190 [13495.186766] ? _raw_spin_unlock+0x12/0x30 [13495.186767] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13495.186768] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13495.186769] ll_atomic_open+0x36a/0xd70 [lustre] [13495.186770] lookup_open+0x356/0xb40 [13495.186771] do_last+0x313/0xfc0 [13495.186772] ? nd_jump_root+0xe5/0x160 [13495.186772] ? path_init+0x437/0x520 [13495.186773] path_openat+0xf7/0x500 [13495.186774] ? do_raw_spin_unlock+0x75/0x190 [13495.186774] do_filp_open+0x99/0x140 [13495.186775] ? getname_flags+0x6e/0x330 [13495.186776] ? __check_object_size+0xff/0x256 [13495.186777] ? do_raw_spin_unlock+0x75/0x190 [13495.186778] ? _raw_spin_unlock+0x12/0x30 [13495.186779] do_sys_openat2+0x2b4/0x410 [13495.186780] do_sys_open+0x73/0xa0 [13495.186781] __x64_sys_openat+0x24/0x30 [13495.186783] do_syscall_64+0xc1/0x440 [13495.186784] entry_SYSCALL_64_after_hwframe+0x49/0xae [13495.186785] RIP: 0033:0x7f53425a5332 [13495.186786] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13495.186788] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13495.186791] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13495.186793] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13495.186794] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13495.186796] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13495.186797] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13495.187604] NMI backtrace for cpu 2 [13495.347885] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13495.352067] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13495.354885] Call Trace: [13495.355669] [13495.356297] ? dump_stack+0xbb/0x10e [13495.357285] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13495.358880] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13495.360421] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13495.361911] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13495.363610] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13495.365207] ? rcu_dump_cpu_stacks+0x11b/0x171 [13495.366760] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [13495.371421] ? update_process_times+0x6d/0xc0 [13495.372963] ? tick_sched_handle.isra.18+0x27/0x90 [13495.374730] ? tick_sched_timer+0x53/0xb0 [13495.376215] ? __hrtimer_run_queues+0x1c2/0x4d0 [13495.377981] ? tick_sched_handle.isra.18+0x90/0x90 [13495.379876] ? hrtimer_interrupt+0x128/0x2c0 [13495.381005] ? smp_apic_timer_interrupt+0xa6/0x270 [13495.382649] ? apic_timer_interrupt+0xf/0x20 [13495.384701] [13495.385503] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13495.386913] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13495.388773] ? do_raw_spin_lock+0x177/0x190 [13495.390297] ? _raw_spin_lock+0x21/0x30 [13495.391412] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13495.393292] ? obd_mod_free+0x46/0x140 [ptlrpc] [13495.394823] ? mdc_commit_open+0x91/0xb0 [mdc] [13495.396520] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13495.398140] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13495.399869] ? after_reply+0xe2c/0x1780 [ptlrpc] [13495.401492] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13495.403122] ? __next_timer_interrupt+0x160/0x160 [13495.404570] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13495.405967] ? do_wait_intr+0xf0/0xf0 [13495.407031] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13495.408717] ? kthread+0x1d1/0x200 [13495.409694] ? set_kthread_struct+0x70/0x70 [13495.410849] ? ret_from_fork+0x1f/0x30 [13496.058213] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [13496.062953] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13496.082428] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13496.085769] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13496.089131] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13496.091838] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13496.100016] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13496.103626] RAX: 0000000000007b38 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13496.106207] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13496.109496] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13496.112000] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13496.114810] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13496.117097] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13496.119876] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13496.122369] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13496.124992] Call Trace: [13496.126055] [13496.127442] ? show_regs.cold.9+0x22/0x2f [13496.128868] ? watchdog_timer_fn.cold.6+0x62/0x10a [13496.130820] ? __hrtimer_run_queues+0x1c2/0x4d0 [13496.132656] ? watchdog+0x40/0x40 [13496.133768] ? hrtimer_interrupt+0x128/0x2c0 [13496.135703] ? smp_apic_timer_interrupt+0xa6/0x270 [13496.137348] ? apic_timer_interrupt+0xf/0x20 [13496.139091] [13496.139837] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13496.141892] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13496.145382] do_raw_spin_lock+0x177/0x190 [13496.146720] _raw_spin_lock+0x21/0x30 [13496.151207] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13496.155090] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13496.157953] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13496.161911] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13496.165072] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13496.167894] ? woken_wake_function+0x30/0x30 [13496.170018] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13496.173566] kthread+0x1d1/0x200 [13496.174889] ? set_kthread_struct+0x70/0x70 [13496.177027] ret_from_fork+0x1f/0x30 [13512.054110] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [13512.060249] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13512.090504] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13512.096838] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13512.103425] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13512.110317] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13512.114481] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13512.129037] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13512.134933] RAX: 0000000000002e8c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13512.140334] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13512.154064] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13512.165198] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13512.169423] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13512.176614] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13512.184459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13512.187338] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13512.192822] Call Trace: [13512.194291] [13512.195184] ? show_regs.cold.9+0x22/0x2f [13512.196436] ? watchdog_timer_fn.cold.6+0x62/0x10a [13512.201106] ? __hrtimer_run_queues+0x1c2/0x4d0 [13512.202248] ? watchdog+0x40/0x40 [13512.202978] ? hrtimer_interrupt+0x128/0x2c0 [13512.204563] ? smp_apic_timer_interrupt+0xa6/0x270 [13512.206689] ? apic_timer_interrupt+0xf/0x20 [13512.208975] [13512.209812] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13512.212739] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13512.217996] ? newidle_balance.isra.93+0x207/0x580 [13512.219826] do_raw_spin_lock+0x177/0x190 [13512.221417] _raw_spin_lock+0x21/0x30 [13512.222773] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13512.225067] process_one_work+0x2c8/0x700 [13512.226583] worker_thread+0x296/0x6e0 [13512.228716] ? rescuer_thread+0x570/0x570 [13512.230613] kthread+0x1d1/0x200 [13512.233677] ? set_kthread_struct+0x70/0x70 [13512.236354] ret_from_fork+0x1f/0x30 [13515.748032] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10878s! [13515.750968] Showing busy workqueues and worker pools: [13515.755894] workqueue events: flags=0x0 [13515.757434] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13515.760263] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13515.764431] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13515.768340] pending: cpu_latency_work [ptlrpc] [13515.772327] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13515.774722] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13515.780515] workqueue events_long: flags=0x0 [13515.787668] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13515.796169] pending: lu_objects_flush [obdclass] [13515.801621] workqueue events_unbound: flags=0x2 [13515.807285] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13515.815635] pending: flush_memcg_stats_dwork, idle_cull_fn [13515.821346] workqueue events_power_efficient: flags=0x80 [13515.825333] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13515.829088] pending: check_lifetime, do_cache_clean [13515.834866] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13515.840780] pending: neigh_periodic_work, neigh_periodic_work [13515.848373] workqueue mm_percpu_wq: flags=0x8 [13515.849892] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13515.855079] pending: vmstat_update, lru_add_drain_per_cpu [13515.862132] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13515.868719] pending: vmstat_update, lru_add_drain_per_cpu [13515.872315] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13515.878271] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13515.880012] workqueue xprtiod: flags=0xa [13515.883642] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13515.892181] pending: xprt_autoclose, xs_error_handle [13515.893965] workqueue ipv6_addrconf: flags=0x40008 [13515.899355] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13515.906463] pending: addrconf_verify_work [13515.911348] workqueue knbd0-recv: flags=0x1a [13515.913406] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13515.919390] in-flight: 651:recv_work [13515.923898] workqueue ptlrpc_pinger: flags=0x2 [13515.926077] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13515.929227] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13515.932406] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10878s workers=4 idle: 15979 31824 8 [13515.936405] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13506s workers=2 idle: 654 [13520.055025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [13520.057057] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [13520.058521] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl [13520.062423] Modules linked in: [13520.062427] i2c_piix4 pcspkr [13520.082689] lustre(O) [13520.085903] squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13520.085934] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13520.085942] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13520.085945] RIP: 0010:native_safe_halt+0x2e/0x50 [13520.085959] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13520.085963] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13520.085968] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13520.085970] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13520.085972] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13520.085974] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13520.085977] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13520.085980] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13520.085983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13520.085985] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13520.085993] Call Trace: [13520.085999] [13520.086004] ? show_regs.cold.9+0x22/0x2f [13520.086013] ? watchdog_timer_fn.cold.6+0x62/0x10a [13520.086019] ? __hrtimer_run_queues+0x1c2/0x4d0 [13520.086023] ? watchdog+0x40/0x40 [13520.086027] ? hrtimer_interrupt+0x128/0x2c0 [13520.086031] ? smp_apic_timer_interrupt+0xa6/0x270 [13520.086037] ? apic_timer_interrupt+0xf/0x20 [13520.086041] [13520.086044] ? native_safe_halt+0x2e/0x50 [13520.086049] kvm_wait+0xcd/0xe0 [13520.086056] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13520.086065] do_raw_spin_lock+0x177/0x190 [13520.086069] _raw_spin_lock+0x21/0x30 [13520.086073] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13520.086328] ? do_raw_spin_unlock+0x75/0x190 [13520.086334] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13520.086527] ? do_wait_intr+0xf0/0xf0 [13520.086536] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13520.088250] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13520.088318] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13520.088325] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13520.088328] RIP: 0010:native_safe_halt+0x2e/0x50 [13520.090214] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13520.090506] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13520.091101] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13520.091211] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13520.091251] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13520.091280] mdc_intent_lock+0x353/0x7e0 [mdc] [13520.091308] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13520.091394] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13520.099542] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13520.099554] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13520.099559] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13520.099561] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13520.099562] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13520.099563] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13520.099565] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13520.099567] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13520.099570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13520.099571] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13520.099578] Call Trace: [13520.099582] [13520.099586] ? show_regs.cold.9+0x22/0x2f [13520.099595] ? watchdog_timer_fn.cold.6+0x62/0x10a [13520.099600] ? __hrtimer_run_queues+0x1c2/0x4d0 [13520.099602] ? watchdog+0x40/0x40 [13520.099605] ? hrtimer_interrupt+0x128/0x2c0 [13520.099608] ? smp_apic_timer_interrupt+0xa6/0x270 [13520.099613] ? apic_timer_interrupt+0xf/0x20 [13520.099617] [13520.099619] ? native_safe_halt+0x2e/0x50 [13520.099622] kvm_wait+0xcd/0xe0 [13520.099627] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13520.099632] do_raw_spin_lock+0x177/0x190 [13520.099634] _raw_spin_lock+0x21/0x30 [13520.099640] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13520.099895] obd_mod_free+0x46/0x140 [ptlrpc] [13520.100447] mdc_commit_open+0x91/0xb0 [mdc] [13520.100483] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13520.100678] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13520.100879] after_reply+0xe2c/0x1780 [ptlrpc] [13520.101338] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13520.101532] ? __next_timer_interrupt+0x160/0x160 [13520.101542] ptlrpcd+0xa82/0xe40 [ptlrpc] [13520.101750] ? do_wait_intr+0xf0/0xf0 [13520.101758] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13520.108682] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13520.113091] kthread+0x1d1/0x200 [13520.113111] ? set_kthread_struct+0x70/0x70 [13520.113116] ret_from_fork+0x1f/0x30 [13520.388135] lmv_intent_open+0x338/0x1150 [lmv] [13520.392162] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13520.394902] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13520.399204] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13520.403038] ll_intent_lock+0xc5/0xaa0 [lustre] [13520.406097] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13520.411731] ll_lookup_it+0xe0e/0x2270 [lustre] [13520.413408] ? do_raw_spin_unlock+0x75/0x190 [13520.414957] ? _raw_spin_unlock+0x12/0x30 [13520.416209] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13520.417678] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13520.419297] ll_atomic_open+0x36a/0xd70 [lustre] [13520.420674] lookup_open+0x356/0xb40 [13520.421775] do_last+0x313/0xfc0 [13520.422755] ? nd_jump_root+0xe5/0x160 [13520.423801] ? path_init+0x437/0x520 [13520.425278] path_openat+0xf7/0x500 [13520.426215] ? do_raw_spin_unlock+0x75/0x190 [13520.427450] do_filp_open+0x99/0x140 [13520.428587] ? getname_flags+0x6e/0x330 [13520.430310] ? __check_object_size+0xff/0x256 [13520.431622] ? do_raw_spin_unlock+0x75/0x190 [13520.432822] ? _raw_spin_unlock+0x12/0x30 [13520.434146] do_sys_openat2+0x2b4/0x410 [13520.435480] do_sys_open+0x73/0xa0 [13520.436613] __x64_sys_openat+0x24/0x30 [13520.437779] do_syscall_64+0xc1/0x440 [13520.439075] entry_SYSCALL_64_after_hwframe+0x49/0xae [13520.440651] RIP: 0033:0x7f53425a5332 [13520.441943] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13520.449194] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13520.452107] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13520.454613] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13520.457558] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13520.466280] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13520.472207] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13524.058350] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [13524.064022] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13524.104209] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13524.108591] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13524.113443] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13524.123578] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13524.133489] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13524.136018] RAX: 0000000000007f8f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13524.138993] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13524.142137] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13524.145593] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13524.148570] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13524.151251] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13524.153846] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13524.155581] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13524.157776] Call Trace: [13524.158533] [13524.159231] ? show_regs.cold.9+0x22/0x2f [13524.160545] ? watchdog_timer_fn.cold.6+0x62/0x10a [13524.162086] ? __hrtimer_run_queues+0x1c2/0x4d0 [13524.163994] ? watchdog+0x40/0x40 [13524.165201] ? hrtimer_interrupt+0x128/0x2c0 [13524.166457] ? smp_apic_timer_interrupt+0xa6/0x270 [13524.168192] ? apic_timer_interrupt+0xf/0x20 [13524.169770] [13524.170633] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13524.172843] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13524.176528] do_raw_spin_lock+0x177/0x190 [13524.177991] _raw_spin_lock+0x21/0x30 [13524.186751] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13524.196732] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13524.201447] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13524.207279] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13524.209260] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13524.212606] ? woken_wake_function+0x30/0x30 [13524.214256] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13524.217155] kthread+0x1d1/0x200 [13524.222953] ? set_kthread_struct+0x70/0x70 [13524.225377] ret_from_fork+0x1f/0x30 [13540.054308] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [13540.062191] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13540.098575] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13540.105251] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13540.114117] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13540.117845] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13540.124692] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13540.138760] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13540.143861] RAX: 0000000000007e91 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13540.153240] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13540.158880] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13540.164361] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13540.172289] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13540.180459] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13540.186753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13540.195069] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13540.200660] Call Trace: [13540.202573] [13540.207025] ? show_regs.cold.9+0x22/0x2f [13540.210325] ? watchdog_timer_fn.cold.6+0x62/0x10a [13540.213263] ? __hrtimer_run_queues+0x1c2/0x4d0 [13540.217004] ? watchdog+0x40/0x40 [13540.218650] ? hrtimer_interrupt+0x128/0x2c0 [13540.220823] ? smp_apic_timer_interrupt+0xa6/0x270 [13540.226761] ? apic_timer_interrupt+0xf/0x20 [13540.233135] [13540.238264] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13540.245712] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13540.249160] ? newidle_balance.isra.93+0x207/0x580 [13540.251288] do_raw_spin_lock+0x177/0x190 [13540.258114] _raw_spin_lock+0x21/0x30 [13540.259973] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13540.263836] process_one_work+0x2c8/0x700 [13540.265716] worker_thread+0x296/0x6e0 [13540.268016] ? rescuer_thread+0x570/0x570 [13540.269861] kthread+0x1d1/0x200 [13540.271580] ? set_kthread_struct+0x70/0x70 [13540.273333] ret_from_fork+0x1f/0x30 [13545.952437] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10908s! [13545.964679] Showing busy workqueues and worker pools: [13545.968202] workqueue events: flags=0x0 [13545.971819] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13545.977596] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13545.989065] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13545.994211] pending: cpu_latency_work [ptlrpc] [13545.997127] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13546.000826] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13546.007782] workqueue events_long: flags=0x0 [13546.010031] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13546.014887] pending: lu_objects_flush [obdclass] [13546.023063] workqueue events_unbound: flags=0x2 [13546.028766] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13546.031708] pending: flush_memcg_stats_dwork, idle_cull_fn [13546.034789] workqueue events_power_efficient: flags=0x80 [13546.041661] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13546.046847] pending: check_lifetime, do_cache_clean [13546.048539] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13546.057006] pending: neigh_periodic_work, neigh_periodic_work [13546.061708] workqueue mm_percpu_wq: flags=0x8 [13546.064728] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13546.067381] pending: vmstat_update, lru_add_drain_per_cpu [13546.070634] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13546.073874] pending: vmstat_update, lru_add_drain_per_cpu [13546.077709] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13546.080768] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13546.085030] workqueue xprtiod: flags=0xa [13546.086613] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13546.089554] pending: xprt_autoclose, xs_error_handle [13546.094134] workqueue ipv6_addrconf: flags=0x40008 [13546.098013] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13546.101734] pending: addrconf_verify_work [13546.104034] workqueue knbd0-recv: flags=0x1a [13546.106436] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13546.113832] in-flight: 651:recv_work [13546.121144] workqueue ptlrpc_pinger: flags=0x2 [13546.127226] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13546.132975] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13546.135433] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10909s workers=4 idle: 15979 31824 8 [13546.141642] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13536s workers=2 idle: 654 [13548.055043] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [13548.060029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [13548.060323] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13548.060394] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13548.060400] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13548.060403] RIP: 0010:native_safe_halt+0x2e/0x50 [13548.060416] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13548.060421] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13548.060425] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13548.060428] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13548.060430] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13548.060433] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13548.060436] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13548.060439] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13548.060442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13548.060444] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13548.060454] Call Trace: [13548.060459] [13548.060465] ? show_regs.cold.9+0x22/0x2f [13548.060474] ? watchdog_timer_fn.cold.6+0x62/0x10a [13548.060481] ? __hrtimer_run_queues+0x1c2/0x4d0 [13548.060485] ? watchdog+0x40/0x40 [13548.060490] ? hrtimer_interrupt+0x128/0x2c0 [13548.060495] ? smp_apic_timer_interrupt+0xa6/0x270 [13548.060501] ? apic_timer_interrupt+0xf/0x20 [13548.060506] [13548.060509] ? native_safe_halt+0x2e/0x50 [13548.060513] kvm_wait+0xcd/0xe0 [13548.060521] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13548.060529] do_raw_spin_lock+0x177/0x190 [13548.060533] _raw_spin_lock+0x21/0x30 [13548.060537] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13548.060823] ? do_raw_spin_unlock+0x75/0x190 [13548.060830] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13548.061740] ? do_wait_intr+0xf0/0xf0 [13548.061750] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13548.061975] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13548.062197] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13548.062605] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13548.062721] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13548.062760] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13548.062791] mdc_intent_lock+0x353/0x7e0 [mdc] [13548.062820] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13548.062964] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13548.063346] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13548.063377] lmv_intent_open+0x338/0x1150 [lmv] [13548.063405] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13548.063498] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13548.063582] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13548.063604] ll_intent_lock+0xc5/0xaa0 [lustre] [13548.063687] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13548.063771] ll_lookup_it+0xe0e/0x2270 [lustre] [13548.063853] ? do_raw_spin_unlock+0x75/0x190 [13548.063861] ? _raw_spin_unlock+0x12/0x30 [13548.063867] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13548.069401] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13548.069478] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13548.069486] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13548.069489] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13548.069504] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13548.069509] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13548.069513] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13548.069515] RDX: 0000000000004908 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13548.069518] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13548.069520] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13548.069522] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13548.069525] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13548.069528] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13548.069530] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13548.069539] Call Trace: [13548.069545] [13548.069550] ? show_regs.cold.9+0x22/0x2f [13548.069559] ? watchdog_timer_fn.cold.6+0x62/0x10a [13548.069564] ? __hrtimer_run_queues+0x1c2/0x4d0 [13548.069568] ? watchdog+0x40/0x40 [13548.069572] ? hrtimer_interrupt+0x128/0x2c0 [13548.069575] ? smp_apic_timer_interrupt+0xa6/0x270 [13548.069583] ? apic_timer_interrupt+0xf/0x20 [13548.069587] [13548.069590] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13548.069595] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13548.069599] do_raw_spin_lock+0x177/0x190 [13548.069603] _raw_spin_lock+0x21/0x30 [13548.069609] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13548.069884] obd_mod_free+0x46/0x140 [ptlrpc] [13548.070078] mdc_commit_open+0x91/0xb0 [mdc] [13548.070113] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13548.117625] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13548.117669] ll_atomic_open+0x36a/0xd70 [lustre] [13548.117787] lookup_open+0x356/0xb40 [13548.117798] do_last+0x313/0xfc0 [13548.117803] ? nd_jump_root+0xe5/0x160 [13548.117808] ? path_init+0x437/0x520 [13548.117813] path_openat+0xf7/0x500 [13548.117818] ? do_raw_spin_unlock+0x75/0x190 [13548.117826] do_filp_open+0x99/0x140 [13548.117832] ? getname_flags+0x6e/0x330 [13548.117837] ? __check_object_size+0xff/0x256 [13548.117844] ? do_raw_spin_unlock+0x75/0x190 [13548.117848] ? _raw_spin_unlock+0x12/0x30 [13548.117854] do_sys_openat2+0x2b4/0x410 [13548.117858] do_sys_open+0x73/0xa0 [13548.117862] __x64_sys_openat+0x24/0x30 [13548.117866] do_syscall_64+0xc1/0x440 [13548.117872] entry_SYSCALL_64_after_hwframe+0x49/0xae [13548.117878] RIP: 0033:0x7f53425a5332 [13548.117886] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13548.117890] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13548.117896] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13548.117899] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13548.117901] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13548.117903] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13548.117905] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13548.447758] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13548.449663] after_reply+0xe2c/0x1780 [ptlrpc] [13548.451737] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13548.457551] ? __next_timer_interrupt+0x160/0x160 [13548.459622] ptlrpcd+0xa82/0xe40 [ptlrpc] [13548.462697] ? do_wait_intr+0xf0/0xf0 [13548.464622] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13548.467267] kthread+0x1d1/0x200 [13548.468699] ? set_kthread_struct+0x70/0x70 [13548.470282] ret_from_fork+0x1f/0x30 [13552.058228] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [13552.062713] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13552.089208] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13552.095693] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13552.101418] RIP: 0010:native_safe_halt+0x2e/0x50 [13552.103385] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13552.112961] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13552.117509] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13552.121532] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13552.124943] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13552.128582] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13552.132582] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13552.136473] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13552.140013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13552.143426] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13552.148062] Call Trace: [13552.149232] [13552.150411] ? show_regs.cold.9+0x22/0x2f [13552.152136] ? watchdog_timer_fn.cold.6+0x62/0x10a [13552.154895] ? __hrtimer_run_queues+0x1c2/0x4d0 [13552.157379] ? watchdog+0x40/0x40 [13552.159225] ? hrtimer_interrupt+0x128/0x2c0 [13552.161457] ? smp_apic_timer_interrupt+0xa6/0x270 [13552.164442] ? apic_timer_interrupt+0xf/0x20 [13552.166415] [13552.168715] ? native_safe_halt+0x2e/0x50 [13552.170841] kvm_wait+0xcd/0xe0 [13552.173009] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13552.175934] do_raw_spin_lock+0x177/0x190 [13552.177967] _raw_spin_lock+0x21/0x30 [13552.179761] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13552.185914] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13552.189404] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13552.193849] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13552.197526] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13552.200566] ? woken_wake_function+0x30/0x30 [13552.203531] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13552.207449] kthread+0x1d1/0x200 [13552.208600] ? set_kthread_struct+0x70/0x70 [13552.212061] ret_from_fork+0x1f/0x30 [13572.052058] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [13572.056014] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13572.081854] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13572.088139] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13572.092339] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13572.095191] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13572.097776] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13572.105292] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13572.107465] RAX: 0000000000003d11 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13572.110626] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13572.113573] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13572.116633] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13572.119773] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13572.123334] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13572.126755] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13572.129313] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13572.131771] Call Trace: [13572.134140] [13572.135390] ? show_regs.cold.9+0x22/0x2f [13572.137146] ? watchdog_timer_fn.cold.6+0x62/0x10a [13572.139403] ? __hrtimer_run_queues+0x1c2/0x4d0 [13572.141330] ? watchdog+0x40/0x40 [13572.142910] ? hrtimer_interrupt+0x128/0x2c0 [13572.144955] ? smp_apic_timer_interrupt+0xa6/0x270 [13572.147324] ? apic_timer_interrupt+0xf/0x20 [13572.149459] [13572.150791] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13572.153112] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13572.155669] ? newidle_balance.isra.93+0x207/0x580 [13572.158163] do_raw_spin_lock+0x177/0x190 [13572.160416] _raw_spin_lock+0x21/0x30 [13572.162554] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13572.165356] process_one_work+0x2c8/0x700 [13572.167216] worker_thread+0x296/0x6e0 [13572.169091] ? rescuer_thread+0x570/0x570 [13572.171059] kthread+0x1d1/0x200 [13572.172580] ? set_kthread_struct+0x70/0x70 [13572.175068] ret_from_fork+0x1f/0x30 [13576.055017] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [13576.057069] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [13576.058128] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13576.058200] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13576.058207] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13576.058210] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13576.058223] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13576.058227] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13576.058232] RAX: 0000000000006dbf RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13576.058234] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13576.058237] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13576.058239] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13576.058241] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13576.058244] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13576.058247] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13576.058249] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13576.058257] Call Trace: [13576.058263] [13576.058268] ? show_regs.cold.9+0x22/0x2f [13576.058276] ? watchdog_timer_fn.cold.6+0x62/0x10a [13576.058282] ? __hrtimer_run_queues+0x1c2/0x4d0 [13576.058285] ? watchdog+0x40/0x40 [13576.058289] ? hrtimer_interrupt+0x128/0x2c0 [13576.058293] ? smp_apic_timer_interrupt+0xa6/0x270 [13576.058300] ? apic_timer_interrupt+0xf/0x20 [13576.058304] [13576.058307] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13576.058313] do_raw_spin_lock+0x177/0x190 [13576.058317] _raw_spin_lock+0x21/0x30 [13576.058321] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13576.058595] ? do_raw_spin_unlock+0x75/0x190 [13576.058602] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13576.058805] ? do_wait_intr+0xf0/0xf0 [13576.058814] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13576.059009] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13576.059220] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13576.059433] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13576.059555] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13576.059594] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13576.059620] mdc_intent_lock+0x353/0x7e0 [mdc] [13576.059646] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13576.059730] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13576.062232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13576.062301] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13576.062307] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13576.062310] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13576.062322] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13576.062326] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13576.062330] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13576.062332] RDX: 0000000000001382 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13576.062334] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13576.062336] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13576.062338] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13576.062341] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13576.062344] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13576.062346] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13576.062353] Call Trace: [13576.062357] [13576.062362] ? show_regs.cold.9+0x22/0x2f [13576.062371] ? watchdog_timer_fn.cold.6+0x62/0x10a [13576.062376] ? __hrtimer_run_queues+0x1c2/0x4d0 [13576.062379] ? watchdog+0x40/0x40 [13576.062383] ? hrtimer_interrupt+0x128/0x2c0 [13576.062387] ? smp_apic_timer_interrupt+0xa6/0x270 [13576.062394] ? apic_timer_interrupt+0xf/0x20 [13576.062398] [13576.062400] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13576.062404] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13576.062408] do_raw_spin_lock+0x177/0x190 [13576.062412] _raw_spin_lock+0x21/0x30 [13576.062417] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13576.102618] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13576.109496] obd_mod_free+0x46/0x140 [ptlrpc] [13576.111017] mdc_commit_open+0x91/0xb0 [mdc] [13576.111061] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13576.111285] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13576.113067] after_reply+0xe2c/0x1780 [ptlrpc] [13576.113267] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13576.126634] lmv_intent_open+0x338/0x1150 [lmv] [13576.126680] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13576.126822] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13576.126902] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13576.126923] ll_intent_lock+0xc5/0xaa0 [lustre] [13576.127007] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13576.127094] ll_lookup_it+0xe0e/0x2270 [lustre] [13576.127461] ? do_raw_spin_unlock+0x75/0x190 [13576.127471] ? _raw_spin_unlock+0x12/0x30 [13576.127479] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13576.129353] ? __next_timer_interrupt+0x160/0x160 [13576.129372] ptlrpcd+0xa82/0xe40 [ptlrpc] [13576.129644] ? do_wait_intr+0xf0/0xf0 [13576.129655] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13576.129877] kthread+0x1d1/0x200 [13576.129883] ? set_kthread_struct+0x70/0x70 [13576.129887] ret_from_fork+0x1f/0x30 [13576.160149] BUG: workqueue lockup - pool [13576.166179] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13576.169285] cpus=0-3 flags=0x4 nice=0 stuck for 10939s! [13576.169305] Showing busy workqueues and worker pools: [13576.169308] workqueue events: flags=0x0 [13576.169311] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13576.169319] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13576.169882] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13576.169892] pending: cpu_latency_work [ptlrpc] [13576.174089] ll_atomic_open+0x36a/0xd70 [lustre] [13576.177627] [13576.177664] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13576.177681] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13576.178108] workqueue events_long: flags=0x0 [13576.178113] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13576.178120] pending: lu_objects_flush [obdclass] [13576.178286] workqueue events_unbound: flags=0x2 [13576.178289] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13576.178297] pending: flush_memcg_stats_dwork, idle_cull_fn [13576.178310] workqueue events_power_efficient: flags=0x80 [13576.178313] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13576.178319] pending: check_lifetime, do_cache_clean [13576.178331] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13576.178337] pending: neigh_periodic_work, neigh_periodic_work [13576.180096] workqueue mm_percpu_wq: flags=0x8 [13576.180103] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13576.180112] pending: vmstat_update, lru_add_drain_per_cpu [13576.180125] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13576.180133] pending: vmstat_update, lru_add_drain_per_cpu [13576.180140] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13576.180147] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13576.180168] workqueue xprtiod: flags=0xa [13576.180170] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13576.180178] pending: xprt_autoclose, xs_error_handle [13576.180195] workqueue ipv6_addrconf: flags=0x40008 [13576.180197] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13576.180205] pending: addrconf_verify_work [13576.180216] workqueue knbd0-recv: flags=0x1a [13576.180219] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13576.180226] in-flight: 651:recv_work [13576.180237] workqueue ptlrpc_pinger: flags=0x2 [13576.180239] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13576.180244] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13576.180813] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10939s workers=4 idle: 15979 31824 8 [13576.180827] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13566s workers=2 idle: 654 [13576.634179] lookup_open+0x356/0xb40 [13576.637655] do_last+0x313/0xfc0 [13576.638773] ? nd_jump_root+0xe5/0x160 [13576.640090] ? path_init+0x437/0x520 [13576.642704] path_openat+0xf7/0x500 [13576.645427] ? do_raw_spin_unlock+0x75/0x190 [13576.648480] do_filp_open+0x99/0x140 [13576.650302] ? getname_flags+0x6e/0x330 [13576.652171] ? __check_object_size+0xff/0x256 [13576.653551] ? do_raw_spin_unlock+0x75/0x190 [13576.655536] ? _raw_spin_unlock+0x12/0x30 [13576.657292] do_sys_openat2+0x2b4/0x410 [13576.658404] do_sys_open+0x73/0xa0 [13576.659391] __x64_sys_openat+0x24/0x30 [13576.662914] do_syscall_64+0xc1/0x440 [13576.664314] entry_SYSCALL_64_after_hwframe+0x49/0xae [13576.665995] RIP: 0033:0x7f53425a5332 [13576.667653] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13576.675435] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13576.678955] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13576.691037] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13576.696120] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13576.699654] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13576.702343] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13580.058201] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13580.061540] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13580.089905] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13580.095575] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13580.100043] RIP: 0010:native_safe_halt+0x2e/0x50 [13580.102595] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13580.112972] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13580.117481] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13580.120905] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13580.124941] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13580.128644] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13580.133437] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13580.137469] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13580.142752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13580.146403] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13580.149882] Call Trace: [13580.151489] [13580.152612] ? show_regs.cold.9+0x22/0x2f [13580.154405] ? watchdog_timer_fn.cold.6+0x62/0x10a [13580.157757] ? __hrtimer_run_queues+0x1c2/0x4d0 [13580.160056] ? watchdog+0x40/0x40 [13580.162073] ? hrtimer_interrupt+0x128/0x2c0 [13580.164574] ? smp_apic_timer_interrupt+0xa6/0x270 [13580.166817] ? apic_timer_interrupt+0xf/0x20 [13580.168912] [13580.169836] ? native_safe_halt+0x2e/0x50 [13580.171576] kvm_wait+0xcd/0xe0 [13580.173622] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13580.176083] do_raw_spin_lock+0x177/0x190 [13580.178414] _raw_spin_lock+0x21/0x30 [13580.180821] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13580.184606] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13580.187518] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13580.190486] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13580.194553] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13580.197460] ? woken_wake_function+0x30/0x30 [13580.199626] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13580.201931] kthread+0x1d1/0x200 [13580.202935] ? set_kthread_struct+0x70/0x70 [13580.204875] ret_from_fork+0x1f/0x30 [13604.053104] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [13604.055299] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [13604.056357] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13604.056435] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13604.056441] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13604.056447] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13604.056743] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13604.056754] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13604.056758] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13604.056763] RAX: 0000000000007e95 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13604.056766] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13604.056768] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13604.056770] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13604.056772] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13604.056776] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13604.056779] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13604.056781] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13604.056791] Call Trace: [13604.056795] [13604.056802] ? show_regs.cold.9+0x22/0x2f [13604.060560] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [13604.060574] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13604.060642] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13604.060647] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13604.060650] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13604.060665] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13604.060669] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13604.060674] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13604.060676] RDX: 00000000000001ff RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13604.060678] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13604.060680] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13604.060683] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13604.060686] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13604.060689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13604.060691] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13604.060700] Call Trace: [13604.060705] [13604.060710] ? show_regs.cold.9+0x22/0x2f [13604.060719] ? watchdog_timer_fn.cold.6+0x62/0x10a [13604.060726] ? __hrtimer_run_queues+0x1c2/0x4d0 [13604.060730] ? watchdog+0x40/0x40 [13604.060734] ? hrtimer_interrupt+0x128/0x2c0 [13604.060738] ? smp_apic_timer_interrupt+0xa6/0x270 [13604.060745] ? apic_timer_interrupt+0xf/0x20 [13604.060749] [13604.060753] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13604.060757] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13604.060761] do_raw_spin_lock+0x177/0x190 [13604.060765] _raw_spin_lock+0x21/0x30 [13604.061453] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13604.061515] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13604.061521] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13604.061524] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13604.061535] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13604.061538] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13604.061543] RAX: 0000000000004ca1 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13604.061546] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13604.061548] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13604.061551] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13604.061553] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13604.061556] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13604.061559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13604.061561] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13604.061569] Call Trace: [13604.061573] [13604.061576] ? show_regs.cold.9+0x22/0x2f [13604.061584] ? watchdog_timer_fn.cold.6+0x62/0x10a [13604.061590] ? __hrtimer_run_queues+0x1c2/0x4d0 [13604.061594] ? watchdog+0x40/0x40 [13604.061598] ? hrtimer_interrupt+0x128/0x2c0 [13604.061603] ? smp_apic_timer_interrupt+0xa6/0x270 [13604.061609] ? apic_timer_interrupt+0xf/0x20 [13604.061615] [13604.061618] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13604.061625] do_raw_spin_lock+0x177/0x190 [13604.061630] _raw_spin_lock+0x21/0x30 [13604.060771] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13604.061635] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13604.061995] ? do_raw_spin_unlock+0x75/0x190 [13604.062001] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13604.063064] ? do_wait_intr+0xf0/0xf0 [13604.063074] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13604.063473] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13604.064594] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13604.064991] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13604.065482] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13604.065523] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13604.065554] mdc_intent_lock+0x353/0x7e0 [mdc] [13604.065791] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13604.065880] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13604.066879] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13604.066912] lmv_intent_open+0x338/0x1150 [lmv] [13604.066942] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13604.067036] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13604.067848] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13604.067876] ll_intent_lock+0xc5/0xaa0 [lustre] [13604.067962] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13604.068044] ll_lookup_it+0xe0e/0x2270 [lustre] [13604.068312] ? do_raw_spin_unlock+0x75/0x190 [13604.068321] ? _raw_spin_unlock+0x12/0x30 [13604.068328] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13604.068353] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13604.068373] ll_atomic_open+0x36a/0xd70 [lustre] [13604.068458] lookup_open+0x356/0xb40 [13604.068468] do_last+0x313/0xfc0 [13604.068473] ? nd_jump_root+0xe5/0x160 [13604.068478] ? path_init+0x437/0x520 [13604.068483] path_openat+0xf7/0x500 [13604.068487] ? do_raw_spin_unlock+0x75/0x190 [13604.068492] do_filp_open+0x99/0x140 [13604.068499] ? getname_flags+0x6e/0x330 [13604.068504] ? __check_object_size+0xff/0x256 [13604.068512] ? do_raw_spin_unlock+0x75/0x190 [13604.068516] ? _raw_spin_unlock+0x12/0x30 [13604.068522] do_sys_openat2+0x2b4/0x410 [13604.068526] do_sys_open+0x73/0xa0 [13604.068530] __x64_sys_openat+0x24/0x30 [13604.068534] do_syscall_64+0xc1/0x440 [13604.068540] entry_SYSCALL_64_after_hwframe+0x49/0xae [13604.068547] RIP: 0033:0x7f53425a5332 [13604.068555] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13604.068559] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13604.068565] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13604.068568] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13604.068569] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13604.068571] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13604.068573] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13604.090838] ? watchdog_timer_fn.cold.6+0x62/0x10a [13604.090861] ? __hrtimer_run_queues+0x1c2/0x4d0 [13604.090866] ? watchdog+0x40/0x40 [13604.090871] ? hrtimer_interrupt+0x128/0x2c0 [13604.090874] ? smp_apic_timer_interrupt+0xa6/0x270 [13604.090883] ? apic_timer_interrupt+0xf/0x20 [13604.090888] [13604.090893] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13604.090899] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13604.090903] ? newidle_balance.isra.93+0x207/0x580 [13604.090910] do_raw_spin_lock+0x177/0x190 [13604.090916] _raw_spin_lock+0x21/0x30 [13604.090920] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13604.091199] process_one_work+0x2c8/0x700 [13604.091207] worker_thread+0x296/0x6e0 [13604.091211] ? rescuer_thread+0x570/0x570 [13604.091214] kthread+0x1d1/0x200 [13604.091219] ? set_kthread_struct+0x70/0x70 [13604.091222] ret_from_fork+0x1f/0x30 [13605.021346] obd_mod_free+0x46/0x140 [ptlrpc] [13605.029055] mdc_commit_open+0x91/0xb0 [mdc] [13605.032981] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13605.043881] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13605.060305] after_reply+0xe2c/0x1780 [ptlrpc] [13605.063499] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13605.072097] ? __next_timer_interrupt+0x160/0x160 [13605.080523] ptlrpcd+0xa82/0xe40 [ptlrpc] [13605.082845] ? do_wait_intr+0xf0/0xf0 [13605.085503] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13605.095376] kthread+0x1d1/0x200 [13605.096819] ? set_kthread_struct+0x70/0x70 [13605.100581] ret_from_fork+0x1f/0x30 [13606.367597] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10969s! [13606.370131] Showing busy workqueues and worker pools: [13606.371389] workqueue events: flags=0x0 [13606.378789] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13606.382151] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13606.387453] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13606.390120] pending: cpu_latency_work [ptlrpc] [13606.393114] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13606.395616] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13606.407112] workqueue events_long: flags=0x0 [13606.411020] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13606.420644] pending: lu_objects_flush [obdclass] [13606.426657] workqueue events_unbound: flags=0x2 [13606.431242] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13606.434815] pending: flush_memcg_stats_dwork, idle_cull_fn [13606.439761] workqueue events_power_efficient: flags=0x80 [13606.442185] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13606.445858] pending: check_lifetime, do_cache_clean [13606.448185] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13606.451640] pending: neigh_periodic_work, neigh_periodic_work [13606.454788] workqueue mm_percpu_wq: flags=0x8 [13606.457087] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13606.460739] pending: vmstat_update, lru_add_drain_per_cpu [13606.463948] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13606.467586] pending: vmstat_update, lru_add_drain_per_cpu [13606.471753] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13606.475315] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13606.478788] workqueue xprtiod: flags=0xa [13606.482047] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13606.485569] pending: xprt_autoclose, xs_error_handle [13606.489477] workqueue ipv6_addrconf: flags=0x40008 [13606.492169] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13606.495633] pending: addrconf_verify_work [13606.497763] workqueue knbd0-recv: flags=0x1a [13606.499491] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13606.503219] in-flight: 651:recv_work [13606.506973] workqueue ptlrpc_pinger: flags=0x2 [13606.509751] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13606.513093] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13606.516597] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10969s workers=4 idle: 15979 31824 8 [13606.523062] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13596s workers=2 idle: 654 [13608.060430] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13608.070984] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13608.110437] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13608.120788] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13608.133938] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13608.135545] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13608.146944] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13608.166947] RAX: 000000000000198a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13608.170863] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13608.173905] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13608.176841] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13608.180643] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13608.182942] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13608.192606] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13608.196736] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13608.203131] Call Trace: [13608.208616] [13608.209939] ? show_regs.cold.9+0x22/0x2f [13608.216033] ? watchdog_timer_fn.cold.6+0x62/0x10a [13608.218129] ? __hrtimer_run_queues+0x1c2/0x4d0 [13608.235019] ? watchdog+0x40/0x40 [13608.237698] ? hrtimer_interrupt+0x128/0x2c0 [13608.242274] ? smp_apic_timer_interrupt+0xa6/0x270 [13608.244735] ? apic_timer_interrupt+0xf/0x20 [13608.247864] [13608.252271] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13608.254648] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13608.259222] do_raw_spin_lock+0x177/0x190 [13608.262003] _raw_spin_lock+0x21/0x30 [13608.265754] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13608.274227] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13608.278257] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13608.283076] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13608.288929] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13608.294369] ? woken_wake_function+0x30/0x30 [13608.295618] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13608.302137] kthread+0x1d1/0x200 [13608.303380] ? set_kthread_struct+0x70/0x70 [13608.307096] ret_from_fork+0x1f/0x30 [13632.052614] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [13632.055647] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13632.055736] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [13632.055744] Modules linked in: [13632.059614] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [13632.059629] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13632.059705] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13632.059711] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13632.059714] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13632.059729] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13632.059734] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13632.059739] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13632.059741] RDX: 0000000000007896 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13632.059743] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13632.059746] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13632.059749] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13632.059763] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13632.059766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13632.059769] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13632.059777] Call Trace: [13632.059783] [13632.059790] ? show_regs.cold.9+0x22/0x2f [13632.059801] ? watchdog_timer_fn.cold.6+0x62/0x10a [13632.059808] ? __hrtimer_run_queues+0x1c2/0x4d0 [13632.059813] ? watchdog+0x40/0x40 [13632.059817] ? hrtimer_interrupt+0x128/0x2c0 [13632.059822] ? smp_apic_timer_interrupt+0xa6/0x270 [13632.059831] ? apic_timer_interrupt+0xf/0x20 [13632.059846] [13632.059849] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13632.059855] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13632.059860] do_raw_spin_lock+0x177/0x190 [13632.059865] _raw_spin_lock+0x21/0x30 [13632.059871] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13632.060140] obd_mod_free+0x46/0x140 [ptlrpc] [13632.060342] mdc_commit_open+0x91/0xb0 [mdc] [13632.060380] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13632.060571] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13632.060768] after_reply+0xe2c/0x1780 [ptlrpc] [13632.060956] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13632.061162] ? __next_timer_interrupt+0x160/0x160 [13632.061172] ptlrpcd+0xa82/0xe40 [ptlrpc] [13632.061368] ? do_wait_intr+0xf0/0xf0 [13632.061377] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13632.061582] kthread+0x1d1/0x200 [13632.061588] ? set_kthread_struct+0x70/0x70 [13632.061592] ret_from_fork+0x1f/0x30 [13632.079733] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13632.079815] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13632.082741] lustre(O) [13632.084011] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13632.086896] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13632.086979] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13632.086986] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13632.086989] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13632.087001] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13632.087004] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13632.087009] RAX: 0000000000006d1d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13632.087011] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13632.087013] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13632.087015] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13632.087018] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13632.087021] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13632.087023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13632.087026] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13632.087034] Call Trace: [13632.087039] [13632.087044] ? show_regs.cold.9+0x22/0x2f [13632.087053] ? watchdog_timer_fn.cold.6+0x62/0x10a [13632.087059] ? __hrtimer_run_queues+0x1c2/0x4d0 [13632.087062] ? watchdog+0x40/0x40 [13632.087066] ? hrtimer_interrupt+0x128/0x2c0 [13632.087069] ? smp_apic_timer_interrupt+0xa6/0x270 [13632.087076] ? apic_timer_interrupt+0xf/0x20 [13632.087080] [13632.087083] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13632.087089] do_raw_spin_lock+0x177/0x190 [13632.087093] _raw_spin_lock+0x21/0x30 [13632.087097] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13632.087514] ? do_raw_spin_unlock+0x75/0x190 [13632.087520] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13632.116729] [13632.116740] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13632.120904] ? do_wait_intr+0xf0/0xf0 [13632.120925] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13632.121206] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13632.121510] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13632.121801] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13632.121915] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13632.121958] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13632.121988] mdc_intent_lock+0x353/0x7e0 [mdc] [13632.122015] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13632.122104] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13632.122437] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13632.122466] lmv_intent_open+0x338/0x1150 [lmv] [13632.122495] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13632.122579] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13632.122734] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13632.122815] ll_intent_lock+0xc5/0xaa0 [lustre] [13632.122900] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13632.122983] ll_lookup_it+0xe0e/0x2270 [lustre] [13632.123065] ? do_raw_spin_unlock+0x75/0x190 [13632.123071] ? _raw_spin_unlock+0x12/0x30 [13632.123078] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13632.123103] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13632.123121] ll_atomic_open+0x36a/0xd70 [lustre] [13632.123212] lookup_open+0x356/0xb40 [13632.123221] do_last+0x313/0xfc0 [13632.123225] ? nd_jump_root+0xe5/0x160 [13632.123229] ? path_init+0x437/0x520 [13632.123233] path_openat+0xf7/0x500 [13632.123237] ? do_raw_spin_unlock+0x75/0x190 [13632.123242] do_filp_open+0x99/0x140 [13632.123247] ? getname_flags+0x6e/0x330 [13632.123251] ? __check_object_size+0xff/0x256 [13632.123258] ? do_raw_spin_unlock+0x75/0x190 [13632.123261] ? _raw_spin_unlock+0x12/0x30 [13632.123265] do_sys_openat2+0x2b4/0x410 [13632.123269] do_sys_open+0x73/0xa0 [13632.123342] __x64_sys_openat+0x24/0x30 [13632.123346] do_syscall_64+0xc1/0x440 [13632.123353] entry_SYSCALL_64_after_hwframe+0x49/0xae [13632.123358] RIP: 0033:0x7f53425a5332 [13632.123365] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13632.123369] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13632.123374] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13632.123376] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13632.123378] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13632.123381] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13632.123383] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13632.367035] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13632.372799] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13632.375548] RAX: 000000000000663a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13632.377833] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13632.379861] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13632.382104] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13632.384197] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13632.386151] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13632.388561] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13632.390065] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13632.392385] Call Trace: [13632.393159] [13632.393845] ? show_regs.cold.9+0x22/0x2f [13632.394969] ? watchdog_timer_fn.cold.6+0x62/0x10a [13632.396227] ? __hrtimer_run_queues+0x1c2/0x4d0 [13632.397910] ? watchdog+0x40/0x40 [13632.399011] ? hrtimer_interrupt+0x128/0x2c0 [13632.400146] ? smp_apic_timer_interrupt+0xa6/0x270 [13632.401536] ? apic_timer_interrupt+0xf/0x20 [13632.403087] [13632.403684] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13632.405455] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13632.406955] ? newidle_balance.isra.93+0x207/0x580 [13632.408241] do_raw_spin_lock+0x177/0x190 [13632.409534] _raw_spin_lock+0x21/0x30 [13632.410624] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13632.412250] process_one_work+0x2c8/0x700 [13632.413852] worker_thread+0x296/0x6e0 [13632.415567] ? rescuer_thread+0x570/0x570 [13632.416955] kthread+0x1d1/0x200 [13632.417964] ? set_kthread_struct+0x70/0x70 [13632.419205] ret_from_fork+0x1f/0x30 [13636.058035] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13636.060421] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13636.081362] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13636.086064] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13636.089999] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13636.094056] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13636.102192] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [13636.104921] RAX: 0000000000004be2 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13636.107745] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13636.110593] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13636.113619] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13636.117436] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13636.120193] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13636.123949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13636.127087] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13636.131589] Call Trace: [13636.133502] [13636.134567] ? show_regs.cold.9+0x22/0x2f [13636.136751] ? watchdog_timer_fn.cold.6+0x62/0x10a [13636.140935] ? __hrtimer_run_queues+0x1c2/0x4d0 [13636.143000] ? watchdog+0x40/0x40 [13636.144561] ? hrtimer_interrupt+0x128/0x2c0 [13636.147876] ? smp_apic_timer_interrupt+0xa6/0x270 [13636.151356] ? apic_timer_interrupt+0xf/0x20 [13636.153865] [13636.155499] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13636.158620] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13636.161531] do_raw_spin_lock+0x177/0x190 [13636.165266] _raw_spin_lock+0x21/0x30 [13636.166547] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13636.171796] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13636.176476] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13636.181400] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13636.186318] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13636.190520] ? woken_wake_function+0x30/0x30 [13636.192062] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13636.194941] kthread+0x1d1/0x200 [13636.197058] ? set_kthread_struct+0x70/0x70 [13636.199267] ret_from_fork+0x1f/0x30 [13636.576098] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 10999s! [13636.579494] Showing busy workqueues and worker pools: [13636.581996] workqueue events: flags=0x0 [13636.583966] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13636.587568] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13636.593117] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13636.596430] pending: cpu_latency_work [ptlrpc] [13636.598970] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13636.602685] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13636.607900] workqueue events_long: flags=0x0 [13636.609838] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13636.611763] pending: lu_objects_flush [obdclass] [13636.613860] workqueue events_unbound: flags=0x2 [13636.616583] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13636.619811] pending: flush_memcg_stats_dwork, idle_cull_fn [13636.623390] workqueue events_power_efficient: flags=0x80 [13636.625745] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13636.630034] pending: check_lifetime, do_cache_clean [13636.631955] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13636.634881] pending: neigh_periodic_work, neigh_periodic_work [13636.638474] workqueue mm_percpu_wq: flags=0x8 [13636.640602] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13636.643621] pending: vmstat_update, lru_add_drain_per_cpu [13636.646338] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13636.649826] pending: vmstat_update, lru_add_drain_per_cpu [13636.655184] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13636.660134] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13636.663367] workqueue xprtiod: flags=0xa [13636.665453] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13636.669456] pending: xprt_autoclose, xs_error_handle [13636.672063] workqueue ipv6_addrconf: flags=0x40008 [13636.674099] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13636.676903] pending: addrconf_verify_work [13636.678940] workqueue knbd0-recv: flags=0x1a [13636.682602] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13636.685582] in-flight: 651:recv_work [13636.688366] workqueue ptlrpc_pinger: flags=0x2 [13636.690491] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13636.693261] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13636.696882] pool 8: cpus=0-3 flags=0x4 nice=0 hung=10999s workers=4 idle: 15979 31824 8 [13636.703918] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13627s workers=2 idle: 654 [13660.055047] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [13660.057506] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [13660.057645] Modules linked in: [13660.065077] Modules linked in: [13660.065079] lustre(O) osp(O) ofd(O) [13660.066101] lustre(O) [13660.067180] lod(O) [13660.069656] osp(O) [13660.071157] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr [13660.071997] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13660.072057] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13660.072064] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13660.072068] RIP: 0010:native_safe_halt+0x2e/0x50 [13660.072085] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13660.072090] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13660.072094] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13660.072097] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13660.072099] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13660.072102] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13660.072105] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13660.072108] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13660.072111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13660.072113] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13660.072122] Call Trace: [13660.072127] [13660.072132] ? show_regs.cold.9+0x22/0x2f [13660.072141] ? watchdog_timer_fn.cold.6+0x62/0x10a [13660.072147] ? __hrtimer_run_queues+0x1c2/0x4d0 [13660.072151] ? watchdog+0x40/0x40 [13660.072155] ? hrtimer_interrupt+0x128/0x2c0 [13660.072159] ? smp_apic_timer_interrupt+0xa6/0x270 [13660.072165] ? apic_timer_interrupt+0xf/0x20 [13660.072170] [13660.072173] ? native_safe_halt+0x2e/0x50 [13660.072177] kvm_wait+0xcd/0xe0 [13660.072184] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13660.072190] do_raw_spin_lock+0x177/0x190 [13660.072194] _raw_spin_lock+0x21/0x30 [13660.072199] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13660.072560] obd_mod_free+0x46/0x140 [ptlrpc] [13660.072752] mdc_commit_open+0x91/0xb0 [mdc] [13660.072845] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13660.073111] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13660.073388] after_reply+0xe2c/0x1780 [ptlrpc] [13660.073828] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13660.074503] ? __next_timer_interrupt+0x160/0x160 [13660.074514] ptlrpcd+0xa82/0xe40 [ptlrpc] [13660.074726] ? do_wait_intr+0xf0/0xf0 [13660.074736] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13660.076381] intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13660.076412] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13660.076418] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13660.076421] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13660.076432] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13660.076437] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13660.076441] RAX: 00000000000049b4 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13660.076444] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13660.076447] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13660.076449] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13660.076451] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13660.076454] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13660.076457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13660.076459] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13660.076468] Call Trace: [13660.076472] [13660.076476] ? show_regs.cold.9+0x22/0x2f [13660.076484] ? watchdog_timer_fn.cold.6+0x62/0x10a [13660.076490] ? __hrtimer_run_queues+0x1c2/0x4d0 [13660.076495] ? watchdog+0x40/0x40 [13660.076499] ? hrtimer_interrupt+0x128/0x2c0 [13660.076508] ? smp_apic_timer_interrupt+0xa6/0x270 [13660.076515] ? apic_timer_interrupt+0xf/0x20 [13660.076519] [13660.076521] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13660.076527] do_raw_spin_lock+0x177/0x190 [13660.076531] _raw_spin_lock+0x21/0x30 [13660.076535] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13660.076776] ? do_raw_spin_unlock+0x75/0x190 [13660.076781] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13660.077087] ? do_wait_intr+0xf0/0xf0 [13660.077097] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13660.077378] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13660.077634] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13660.078628] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13660.078794] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13660.078845] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13660.078896] mdc_intent_lock+0x353/0x7e0 [mdc] [13660.078931] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13660.079031] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13660.079619] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13660.079662] lmv_intent_open+0x338/0x1150 [lmv] [13660.079697] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13660.079820] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13660.079987] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13660.080012] ll_intent_lock+0xc5/0xaa0 [lustre] [13660.080344] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13660.080452] ll_lookup_it+0xe0e/0x2270 [lustre] [13660.081478] ? do_raw_spin_unlock+0x75/0x190 [13660.081487] ? _raw_spin_unlock+0x12/0x30 [13660.081495] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13660.081535] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13660.081555] ll_atomic_open+0x36a/0xd70 [lustre] [13660.100820] kthread+0x1d1/0x200 [13660.100842] ? set_kthread_struct+0x70/0x70 [13660.100847] ret_from_fork+0x1f/0x30 [13660.384488] lookup_open+0x356/0xb40 [13660.385599] do_last+0x313/0xfc0 [13660.386402] ? nd_jump_root+0xe5/0x160 [13660.387455] ? path_init+0x437/0x520 [13660.388389] path_openat+0xf7/0x500 [13660.389135] ? do_raw_spin_unlock+0x75/0x190 [13660.390142] do_filp_open+0x99/0x140 [13660.390980] ? getname_flags+0x6e/0x330 [13660.391972] ? __check_object_size+0xff/0x256 [13660.393529] ? do_raw_spin_unlock+0x75/0x190 [13660.395157] ? _raw_spin_unlock+0x12/0x30 [13660.396865] do_sys_openat2+0x2b4/0x410 [13660.398467] do_sys_open+0x73/0xa0 [13660.399722] __x64_sys_openat+0x24/0x30 [13660.401133] do_syscall_64+0xc1/0x440 [13660.402668] entry_SYSCALL_64_after_hwframe+0x49/0xae [13660.404627] RIP: 0033:0x7f53425a5332 [13660.406057] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13660.413494] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13660.416515] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13660.419397] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13660.422009] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13660.424607] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13660.427803] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13664.052799] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [13664.058791] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13664.059636] Modules linked in: [13664.063909] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13664.063987] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13664.063995] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13664.063998] RIP: 0010:native_safe_halt+0x2e/0x50 [13664.064012] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13664.064017] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13664.064022] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13664.064024] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13664.064026] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13664.064028] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13664.064031] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13664.064034] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13664.064037] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13664.064040] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13664.064049] Call Trace: [13664.064054] [13664.064061] ? show_regs.cold.9+0x22/0x2f [13664.064071] ? watchdog_timer_fn.cold.6+0x62/0x10a [13664.064078] ? __hrtimer_run_queues+0x1c2/0x4d0 [13664.064082] ? watchdog+0x40/0x40 [13664.064087] ? hrtimer_interrupt+0x128/0x2c0 [13664.064092] ? smp_apic_timer_interrupt+0xa6/0x270 [13664.064099] ? apic_timer_interrupt+0xf/0x20 [13664.064103] [13664.064106] ? native_safe_halt+0x2e/0x50 [13664.064110] kvm_wait+0xcd/0xe0 [13664.064118] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13664.064128] do_raw_spin_lock+0x177/0x190 [13664.064134] _raw_spin_lock+0x21/0x30 [13664.064139] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13664.064519] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13664.064725] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13664.064951] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13664.065183] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13664.065496] ? woken_wake_function+0x30/0x30 [13664.065507] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13664.065732] kthread+0x1d1/0x200 [13664.065739] ? set_kthread_struct+0x70/0x70 [13664.065743] ret_from_fork+0x1f/0x30 [13664.186704] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13664.205360] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13664.210684] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13664.214554] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13664.217407] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13664.220027] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13664.228895] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13664.232080] RAX: 0000000000006c86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13664.237452] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13664.240060] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13664.248348] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13664.252580] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13664.256227] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13664.260366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13664.262908] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13664.267758] Call Trace: [13664.268678] [13664.274987] ? show_regs.cold.9+0x22/0x2f [13664.280567] ? watchdog_timer_fn.cold.6+0x62/0x10a [13664.283701] ? __hrtimer_run_queues+0x1c2/0x4d0 [13664.286040] ? watchdog+0x40/0x40 [13664.289928] ? hrtimer_interrupt+0x128/0x2c0 [13664.292575] ? smp_apic_timer_interrupt+0xa6/0x270 [13664.294685] ? apic_timer_interrupt+0xf/0x20 [13664.303525] [13664.305487] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13664.311938] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13664.315003] ? newidle_balance.isra.93+0x207/0x580 [13664.316664] do_raw_spin_lock+0x177/0x190 [13664.318960] _raw_spin_lock+0x21/0x30 [13664.321025] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13664.329694] process_one_work+0x2c8/0x700 [13664.333050] worker_thread+0x296/0x6e0 [13664.338611] ? rescuer_thread+0x570/0x570 [13664.341767] kthread+0x1d1/0x200 [13664.344151] ? set_kthread_struct+0x70/0x70 [13664.346668] ret_from_fork+0x1f/0x30 [13666.786040] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11029s! [13666.790273] Showing busy workqueues and worker pools: [13666.794769] workqueue events: flags=0x0 [13666.798866] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13666.804025] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13666.811877] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13666.816590] pending: cpu_latency_work [ptlrpc] [13666.819056] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13666.822796] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13666.832456] workqueue events_long: flags=0x0 [13666.834315] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13666.839335] pending: lu_objects_flush [obdclass] [13666.841735] workqueue events_unbound: flags=0x2 [13666.845202] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13666.847072] pending: flush_memcg_stats_dwork, idle_cull_fn [13666.850130] workqueue events_power_efficient: flags=0x80 [13666.852631] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13666.855814] pending: check_lifetime, do_cache_clean [13666.857919] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13666.860946] pending: neigh_periodic_work, neigh_periodic_work [13666.863631] workqueue mm_percpu_wq: flags=0x8 [13666.866409] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13666.870085] pending: vmstat_update, lru_add_drain_per_cpu [13666.872593] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13666.875440] pending: vmstat_update, lru_add_drain_per_cpu [13666.877770] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13666.880531] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13666.883281] workqueue xprtiod: flags=0xa [13666.885136] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13666.888513] pending: xprt_autoclose, xs_error_handle [13666.890586] workqueue ipv6_addrconf: flags=0x40008 [13666.893795] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13666.896476] pending: addrconf_verify_work [13666.898208] workqueue knbd0-recv: flags=0x1a [13666.900013] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13666.902643] in-flight: 651:recv_work [13666.904371] workqueue ptlrpc_pinger: flags=0x2 [13666.906359] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13666.908202] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13666.910042] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11029s workers=4 idle: 15979 31824 8 [13666.912670] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13657s workers=2 idle: 654 [13674.707407] rcu: INFO: rcu_sched self-detected stall on CPU [13674.710654] rcu: 2-....: (6458354 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [13674.722764] (t=11580281 jiffies g=274185 q=66838) [13674.727722] rcu: rcu_sched kthread starved for 11038647 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [13674.741735] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [13674.748763] rcu: RCU grace-period kthread stack dump: [13674.754695] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [13674.763607] Call Trace: [13674.764475] __schedule+0x351/0xcb0 [13674.771461] schedule+0xc0/0x180 [13674.775653] schedule_timeout+0xb4/0x190 [13674.778917] ? do_raw_spin_unlock+0x75/0x190 [13674.780433] ? __next_timer_interrupt+0x160/0x160 [13674.784237] ? prepare_to_swait_event+0xf3/0x140 [13674.786669] rcu_gp_kthread+0x8dd/0x15b0 [13674.789216] ? force_qs_rnp+0x2c0/0x2c0 [13674.791163] kthread+0x1d1/0x200 [13674.793581] ? set_kthread_struct+0x70/0x70 [13674.800392] ret_from_fork+0x1f/0x30 [13674.804149] rcu: Stack dump where RCU GP kthread last ran: [13674.811798] Sending NMI from CPU 2 to CPUs 3: [13674.815063] NMI backtrace for cpu 3 [13674.815073] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13674.815079] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13674.815080] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13674.815082] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13674.815084] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 [13674.815087] RAX: 00000000000071b4 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13674.815089] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13674.815091] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13674.815092] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13674.815094] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13674.815098] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13674.815099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13674.815101] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13674.815102] Call Trace: [13674.815103] [13674.815104] ? show_regs.cold.9+0x22/0x2f [13674.815105] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [13674.815107] ? nmi_cpu_backtrace_handler+0x15/0x40 [13674.815108] ? nmi_handle+0xbd/0x210 [13674.815109] ? default_do_nmi+0x4b/0x1b0 [13674.815111] ? do_nmi+0x280/0x380 [13674.815112] ? end_repeat_nmi+0x16/0x1f [13674.815113] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13674.815114] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13674.815116] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13674.815117] [13674.815118] do_raw_spin_lock+0x177/0x190 [13674.815119] _raw_spin_lock+0x21/0x30 [13674.815120] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13674.815122] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13674.815123] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13674.815124] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13674.815125] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13674.815126] ? woken_wake_function+0x30/0x30 [13674.815127] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13674.815128] kthread+0x1d1/0x200 [13674.815129] ? set_kthread_struct+0x70/0x70 [13674.815130] ret_from_fork+0x1f/0x30 [13674.815402] Sending NMI from CPU 2 to CPUs 1: [13674.947269] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [13674.948239] NMI backtrace for cpu 2 [13674.951648] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13674.955705] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13674.958368] Call Trace: [13674.959055] [13674.959718] ? dump_stack+0xbb/0x10e [13674.960687] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13674.962068] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13674.963613] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13674.965992] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13674.972716] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13674.975660] ? rcu_dump_cpu_stacks+0x11b/0x171 [13674.977733] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [13674.979983] ? update_process_times+0x6d/0xc0 [13674.981693] ? tick_sched_handle.isra.18+0x27/0x90 [13674.984568] ? tick_sched_timer+0x53/0xb0 [13674.986448] ? __hrtimer_run_queues+0x1c2/0x4d0 [13674.989336] ? tick_sched_handle.isra.18+0x90/0x90 [13674.991409] ? hrtimer_interrupt+0x128/0x2c0 [13674.993203] ? smp_apic_timer_interrupt+0xa6/0x270 [13674.995169] ? apic_timer_interrupt+0xf/0x20 [13674.996765] [13674.997679] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13675.000246] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13675.003678] ? do_raw_spin_lock+0x177/0x190 [13675.005615] ? _raw_spin_lock+0x21/0x30 [13675.007291] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13675.009691] ? obd_mod_free+0x46/0x140 [ptlrpc] [13675.012157] ? mdc_commit_open+0x91/0xb0 [mdc] [13675.014615] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13675.017413] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13675.020331] ? after_reply+0xe2c/0x1780 [ptlrpc] [13675.022671] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13675.025365] ? __next_timer_interrupt+0x160/0x160 [13675.027360] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13675.029688] ? do_wait_intr+0xf0/0xf0 [13675.031430] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13675.033780] ? kthread+0x1d1/0x200 [13675.035365] ? set_kthread_struct+0x70/0x70 [13675.036884] ? ret_from_fork+0x1f/0x30 [13692.054232] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [13692.065221] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13692.104693] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13692.118524] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13692.130226] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13692.138476] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13692.145471] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13692.159304] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13692.164388] RAX: 0000000000007e84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13692.168351] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13692.172223] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13692.178261] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13692.186300] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13692.195088] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13692.205064] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13692.209376] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13692.213212] Call Trace: [13692.214415] [13692.216359] ? show_regs.cold.9+0x22/0x2f [13692.218243] ? watchdog_timer_fn.cold.6+0x62/0x10a [13692.223567] ? __hrtimer_run_queues+0x1c2/0x4d0 [13692.227229] ? watchdog+0x40/0x40 [13692.229338] ? hrtimer_interrupt+0x128/0x2c0 [13692.231363] ? smp_apic_timer_interrupt+0xa6/0x270 [13692.234252] ? apic_timer_interrupt+0xf/0x20 [13692.235922] [13692.237006] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13692.240250] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13692.243022] ? newidle_balance.isra.93+0x207/0x580 [13692.245103] do_raw_spin_lock+0x177/0x190 [13692.247446] _raw_spin_lock+0x21/0x30 [13692.249470] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13692.252207] process_one_work+0x2c8/0x700 [13692.254193] worker_thread+0x296/0x6e0 [13692.255962] ? rescuer_thread+0x570/0x570 [13692.257928] kthread+0x1d1/0x200 [13692.259457] ? set_kthread_struct+0x70/0x70 [13692.261427] ret_from_fork+0x1f/0x30 [13696.991135] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11059s! [13696.998286] Showing busy workqueues and worker pools: [13697.000904] workqueue events: flags=0x0 [13697.002759] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13697.005168] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13697.010814] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13697.013110] pending: cpu_latency_work [ptlrpc] [13697.015174] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13697.017521] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13697.021441] workqueue events_long: flags=0x0 [13697.023138] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13697.025779] pending: lu_objects_flush [obdclass] [13697.027916] workqueue events_unbound: flags=0x2 [13697.029579] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13697.035546] pending: flush_memcg_stats_dwork, idle_cull_fn [13697.043951] workqueue events_power_efficient: flags=0x80 [13697.047708] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13697.051150] pending: check_lifetime, do_cache_clean [13697.055920] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13697.059137] pending: neigh_periodic_work, neigh_periodic_work [13697.061464] workqueue mm_percpu_wq: flags=0x8 [13697.063940] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13697.068685] pending: vmstat_update, lru_add_drain_per_cpu [13697.072977] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13697.076669] pending: vmstat_update, lru_add_drain_per_cpu [13697.079044] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13697.081424] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13697.084713] workqueue xprtiod: flags=0xa [13697.086676] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13697.089730] pending: xprt_autoclose, xs_error_handle [13697.092046] workqueue ipv6_addrconf: flags=0x40008 [13697.094191] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13697.097188] pending: addrconf_verify_work [13697.099520] workqueue knbd0-recv: flags=0x1a [13697.101507] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13697.105099] in-flight: 651:recv_work [13697.107073] workqueue ptlrpc_pinger: flags=0x2 [13697.108992] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13697.112115] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13697.116553] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11060s workers=4 idle: 15979 31824 8 [13697.119915] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13687s workers=2 idle: 654 [13700.056532] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [13700.058580] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [13700.058615] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [13700.058620] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13700.058763] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13700.058769] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13700.058772] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13700.058786] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13700.058790] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13700.058794] RAX: 0000000000006944 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13700.058796] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13700.058798] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13700.058800] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13700.058802] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13700.058806] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13700.058808] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13700.058810] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13700.058820] Call Trace: [13700.058826] [13700.058832] ? show_regs.cold.9+0x22/0x2f [13700.058841] ? watchdog_timer_fn.cold.6+0x62/0x10a [13700.058848] ? __hrtimer_run_queues+0x1c2/0x4d0 [13700.058852] ? watchdog+0x40/0x40 [13700.058857] ? hrtimer_interrupt+0x128/0x2c0 [13700.058861] ? smp_apic_timer_interrupt+0xa6/0x270 [13700.058870] ? apic_timer_interrupt+0xf/0x20 [13700.058875] [13700.058878] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13700.058884] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13700.058890] do_raw_spin_lock+0x177/0x190 [13700.058895] _raw_spin_lock+0x21/0x30 [13700.058899] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13700.061720] Modules linked in: [13700.061395] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13700.061725] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13700.061792] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13700.061798] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13700.061801] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13700.061813] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13700.061817] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13700.061822] RAX: 0000000000003a5a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13700.061824] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13700.061828] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13700.061832] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13700.061836] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13700.061839] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13700.061842] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13700.061844] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13700.061854] Call Trace: [13700.061858] [13700.061861] ? show_regs.cold.9+0x22/0x2f [13700.061870] ? watchdog_timer_fn.cold.6+0x62/0x10a [13700.061876] ? __hrtimer_run_queues+0x1c2/0x4d0 [13700.061881] ? watchdog+0x40/0x40 [13700.061885] ? hrtimer_interrupt+0x128/0x2c0 [13700.061889] ? smp_apic_timer_interrupt+0xa6/0x270 [13700.061897] ? apic_timer_interrupt+0xf/0x20 [13700.061902] [13700.061905] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13700.061910] do_raw_spin_lock+0x177/0x190 [13700.061915] _raw_spin_lock+0x21/0x30 [13700.061833] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13700.061919] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13700.062384] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13700.062617] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13700.062829] ? woken_wake_function+0x30/0x30 [13700.062837] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13700.063037] kthread+0x1d1/0x200 [13700.063043] ? set_kthread_struct+0x70/0x70 [13700.063047] ret_from_fork+0x1f/0x30 [13700.065155] ? do_raw_spin_unlock+0x75/0x190 [13700.065167] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13700.065404] ? do_wait_intr+0xf0/0xf0 [13700.065412] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13700.065623] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13700.065835] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13700.072578] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13700.072656] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13700.072665] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13700.072668] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13700.072682] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13700.072687] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13700.072692] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13700.072694] RDX: 0000000000007052 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13700.072696] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13700.072699] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13700.072702] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13700.072705] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13700.072708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13700.072710] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13700.072718] Call Trace: [13700.072724] [13700.072730] ? show_regs.cold.9+0x22/0x2f [13700.072739] ? watchdog_timer_fn.cold.6+0x62/0x10a [13700.072744] ? __hrtimer_run_queues+0x1c2/0x4d0 [13700.072748] ? watchdog+0x40/0x40 [13700.072752] ? hrtimer_interrupt+0x128/0x2c0 [13700.072756] ? smp_apic_timer_interrupt+0xa6/0x270 [13700.072766] ? apic_timer_interrupt+0xf/0x20 [13700.072771] [13700.072774] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13700.072779] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13700.072784] do_raw_spin_lock+0x177/0x190 [13700.082546] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13700.082798] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13700.082841] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13700.082871] mdc_intent_lock+0x353/0x7e0 [mdc] [13700.082899] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13700.082984] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13700.083269] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13700.083298] lmv_intent_open+0x338/0x1150 [lmv] [13700.083329] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13700.083972] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13700.084058] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13700.084084] ll_intent_lock+0xc5/0xaa0 [lustre] [13700.084168] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13700.084247] ll_lookup_it+0xe0e/0x2270 [lustre] [13700.084328] ? do_raw_spin_unlock+0x75/0x190 [13700.084337] ? _raw_spin_unlock+0x12/0x30 [13700.084345] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13700.084383] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13700.084418] ll_atomic_open+0x36a/0xd70 [lustre] [13700.084501] lookup_open+0x356/0xb40 [13700.084512] do_last+0x313/0xfc0 [13700.084516] ? nd_jump_root+0xe5/0x160 [13700.084521] ? path_init+0x437/0x520 [13700.084525] path_openat+0xf7/0x500 [13700.084531] ? do_raw_spin_unlock+0x75/0x190 [13700.084536] do_filp_open+0x99/0x140 [13700.084542] ? getname_flags+0x6e/0x330 [13700.084546] ? __check_object_size+0xff/0x256 [13700.084554] ? do_raw_spin_unlock+0x75/0x190 [13700.084558] ? _raw_spin_unlock+0x12/0x30 [13700.084573] do_sys_openat2+0x2b4/0x410 [13700.084578] do_sys_open+0x73/0xa0 [13700.084581] __x64_sys_openat+0x24/0x30 [13700.084585] do_syscall_64+0xc1/0x440 [13700.084592] entry_SYSCALL_64_after_hwframe+0x49/0xae [13700.084599] RIP: 0033:0x7f53425a5332 [13700.084654] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13700.084659] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13700.084664] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13700.084667] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13700.084669] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13700.084672] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13700.084674] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13700.972871] _raw_spin_lock+0x21/0x30 [13700.975495] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13700.984677] obd_mod_free+0x46/0x140 [ptlrpc] [13700.988265] mdc_commit_open+0x91/0xb0 [mdc] [13700.991094] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13700.998860] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13701.001655] after_reply+0xe2c/0x1780 [ptlrpc] [13701.004255] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13701.006767] ? __next_timer_interrupt+0x160/0x160 [13701.009979] ptlrpcd+0xa82/0xe40 [ptlrpc] [13701.016403] ? do_wait_intr+0xf0/0xf0 [13701.023826] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13701.026862] kthread+0x1d1/0x200 [13701.030516] ? set_kthread_struct+0x70/0x70 [13701.034078] ret_from_fork+0x1f/0x30 [13724.054704] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [13724.062727] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13724.092657] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13724.100843] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13724.105553] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13724.111469] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13724.117643] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13724.130010] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13724.133038] RAX: 000000000000210d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13724.136633] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13724.139121] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13724.141598] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13724.145997] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13724.155064] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13724.163750] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13724.166837] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13724.175549] Call Trace: [13724.176881] [13724.177922] ? show_regs.cold.9+0x22/0x2f [13724.180948] ? watchdog_timer_fn.cold.6+0x62/0x10a [13724.183917] ? __hrtimer_run_queues+0x1c2/0x4d0 [13724.186054] ? watchdog+0x40/0x40 [13724.188963] ? hrtimer_interrupt+0x128/0x2c0 [13724.191962] ? smp_apic_timer_interrupt+0xa6/0x270 [13724.193430] ? apic_timer_interrupt+0xf/0x20 [13724.194597] [13724.196590] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13724.199045] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13724.207053] ? newidle_balance.isra.93+0x207/0x580 [13724.209582] do_raw_spin_lock+0x177/0x190 [13724.214943] _raw_spin_lock+0x21/0x30 [13724.217646] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13724.221073] process_one_work+0x2c8/0x700 [13724.230941] worker_thread+0x296/0x6e0 [13724.232546] ? rescuer_thread+0x570/0x570 [13724.235037] kthread+0x1d1/0x200 [13724.240394] ? set_kthread_struct+0x70/0x70 [13724.244297] ret_from_fork+0x1f/0x30 [13727.207210] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11090s! [13727.217734] Showing busy workqueues and worker pools: [13727.220886] workqueue events: flags=0x0 [13727.226479] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13727.230714] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13727.236396] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13727.247155] pending: cpu_latency_work [ptlrpc] [13727.249079] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13727.259809] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13727.270498] workqueue events_long: flags=0x0 [13727.279423] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13727.284207] pending: lu_objects_flush [obdclass] [13727.287693] workqueue events_unbound: flags=0x2 [13727.290069] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13727.293588] pending: flush_memcg_stats_dwork, idle_cull_fn [13727.297745] workqueue events_power_efficient: flags=0x80 [13727.303686] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13727.307531] pending: check_lifetime, do_cache_clean [13727.312770] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13727.315249] pending: neigh_periodic_work, neigh_periodic_work [13727.319434] workqueue mm_percpu_wq: flags=0x8 [13727.321872] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13727.325015] pending: vmstat_update, lru_add_drain_per_cpu [13727.328543] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13727.334550] pending: vmstat_update, lru_add_drain_per_cpu [13727.344027] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13727.348194] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13727.353568] workqueue xprtiod: flags=0xa [13727.354876] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13727.359388] pending: xprt_autoclose, xs_error_handle [13727.362700] workqueue ipv6_addrconf: flags=0x40008 [13727.365797] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13727.370658] pending: addrconf_verify_work [13727.374427] workqueue knbd0-recv: flags=0x1a [13727.376673] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13727.378845] in-flight: 651:recv_work [13727.380209] workqueue ptlrpc_pinger: flags=0x2 [13727.381671] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13727.384672] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13727.390824] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11090s workers=4 idle: 15979 31824 8 [13727.396617] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13717s workers=2 idle: 654 [13728.055235] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [13728.057709] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [13728.063206] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13728.063287] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13728.063296] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13728.063300] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13728.063315] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13728.063319] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13728.063324] RAX: 0000000000005994 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13728.063326] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13728.063329] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13728.063331] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13728.063333] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13728.063336] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13728.063339] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13728.063342] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13728.063350] Call Trace: [13728.063356] [13728.063362] ? show_regs.cold.9+0x22/0x2f [13728.063372] ? watchdog_timer_fn.cold.6+0x62/0x10a [13728.063378] ? __hrtimer_run_queues+0x1c2/0x4d0 [13728.063382] ? watchdog+0x40/0x40 [13728.063387] ? hrtimer_interrupt+0x128/0x2c0 [13728.063390] ? smp_apic_timer_interrupt+0xa6/0x270 [13728.063398] ? apic_timer_interrupt+0xf/0x20 [13728.063403] [13728.063405] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13728.063412] do_raw_spin_lock+0x177/0x190 [13728.063416] _raw_spin_lock+0x21/0x30 [13728.063420] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13728.063727] ? do_raw_spin_unlock+0x75/0x190 [13728.063733] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13728.063868] ? do_wait_intr+0xf0/0xf0 [13728.063875] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13728.065258] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [13728.065264] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13728.066422] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13728.066428] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13728.066431] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13728.066446] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13728.066450] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13728.066455] RAX: 0000000000000c10 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13728.066457] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13728.066459] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13728.066461] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13728.066463] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13728.066467] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13728.066469] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13728.066471] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13728.066481] Call Trace: [13728.066486] [13728.066492] ? show_regs.cold.9+0x22/0x2f [13728.066502] ? watchdog_timer_fn.cold.6+0x62/0x10a [13728.066509] ? __hrtimer_run_queues+0x1c2/0x4d0 [13728.066513] ? watchdog+0x40/0x40 [13728.066517] ? hrtimer_interrupt+0x128/0x2c0 [13728.066521] ? smp_apic_timer_interrupt+0xa6/0x270 [13728.066530] ? apic_timer_interrupt+0xf/0x20 [13728.066535] [13728.066537] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13728.066542] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13728.066549] do_raw_spin_lock+0x177/0x190 [13728.066553] _raw_spin_lock+0x21/0x30 [13728.066558] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13728.066852] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13728.067103] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13728.070491] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13728.070773] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13728.070986] ? woken_wake_function+0x30/0x30 [13728.070996] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13728.071227] kthread+0x1d1/0x200 [13728.071234] ? set_kthread_struct+0x70/0x70 [13728.071238] ret_from_fork+0x1f/0x30 [13728.071712] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13728.071784] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13728.071790] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13728.071793] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13728.071805] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13728.071810] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13728.071815] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13728.071817] RDX: 00000000000073b8 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13728.071819] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13728.071821] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13728.071823] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13728.071827] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13728.071829] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13728.071831] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13728.071839] Call Trace: [13728.071843] [13728.071847] ? show_regs.cold.9+0x22/0x2f [13728.071856] ? watchdog_timer_fn.cold.6+0x62/0x10a [13728.071862] ? __hrtimer_run_queues+0x1c2/0x4d0 [13728.071866] ? watchdog+0x40/0x40 [13728.071870] ? hrtimer_interrupt+0x128/0x2c0 [13728.071874] ? smp_apic_timer_interrupt+0xa6/0x270 [13728.071880] ? apic_timer_interrupt+0xf/0x20 [13728.071885] [13728.071888] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13728.071893] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13728.071899] do_raw_spin_lock+0x177/0x190 [13728.071903] _raw_spin_lock+0x21/0x30 [13728.071908] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13728.090439] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13728.090810] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13728.091188] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13728.091304] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13728.091343] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13728.091371] mdc_intent_lock+0x353/0x7e0 [mdc] [13728.091396] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13728.091477] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13728.091690] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13728.091718] lmv_intent_open+0x338/0x1150 [lmv] [13728.091750] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13728.091844] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13728.091928] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13728.092726] ll_intent_lock+0xc5/0xaa0 [lustre] [13728.093763] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13728.093875] ll_lookup_it+0xe0e/0x2270 [lustre] [13728.093967] ? do_raw_spin_unlock+0x75/0x190 [13728.093975] ? _raw_spin_unlock+0x12/0x30 [13728.093983] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13728.094011] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13728.094040] ll_atomic_open+0x36a/0xd70 [lustre] [13728.094229] lookup_open+0x356/0xb40 [13728.094242] do_last+0x313/0xfc0 [13728.094247] ? nd_jump_root+0xe5/0x160 [13728.094251] ? path_init+0x437/0x520 [13728.094256] path_openat+0xf7/0x500 [13728.094261] ? do_raw_spin_unlock+0x75/0x190 [13728.094265] do_filp_open+0x99/0x140 [13728.094272] ? getname_flags+0x6e/0x330 [13728.094277] ? __check_object_size+0xff/0x256 [13728.094285] ? do_raw_spin_unlock+0x75/0x190 [13728.094290] ? _raw_spin_unlock+0x12/0x30 [13728.094294] do_sys_openat2+0x2b4/0x410 [13728.094299] do_sys_open+0x73/0xa0 [13728.094304] __x64_sys_openat+0x24/0x30 [13728.094307] do_syscall_64+0xc1/0x440 [13728.094314] entry_SYSCALL_64_after_hwframe+0x49/0xae [13728.094322] RIP: 0033:0x7f53425a5332 [13728.094330] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13728.094334] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13728.094339] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13728.094341] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13728.094344] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13728.094346] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13728.094348] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13728.644427] obd_mod_free+0x46/0x140 [ptlrpc] [13728.649874] mdc_commit_open+0x91/0xb0 [mdc] [13728.654628] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13728.661885] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13728.672636] after_reply+0xe2c/0x1780 [ptlrpc] [13728.677704] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13728.684079] ? __next_timer_interrupt+0x160/0x160 [13728.685640] ptlrpcd+0xa82/0xe40 [ptlrpc] [13728.691701] ? do_wait_intr+0xf0/0xf0 [13728.697657] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13728.704876] kthread+0x1d1/0x200 [13728.708126] ? set_kthread_struct+0x70/0x70 [13728.714573] ret_from_fork+0x1f/0x30 [13752.052089] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [13752.054884] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13752.080572] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13752.086215] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13752.090899] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13752.094663] RIP: 0010:native_safe_halt+0x2e/0x50 [13752.097544] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13752.106447] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13752.110393] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13752.114225] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [13752.118235] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13752.121083] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13752.124206] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13752.127138] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13752.130373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13752.133069] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13752.135856] Call Trace: [13752.136709] [13752.137382] ? show_regs.cold.9+0x22/0x2f [13752.139159] ? watchdog_timer_fn.cold.6+0x62/0x10a [13752.141228] ? __hrtimer_run_queues+0x1c2/0x4d0 [13752.142933] ? watchdog+0x40/0x40 [13752.144498] ? hrtimer_interrupt+0x128/0x2c0 [13752.146279] ? smp_apic_timer_interrupt+0xa6/0x270 [13752.148484] ? apic_timer_interrupt+0xf/0x20 [13752.150141] [13752.150889] ? native_safe_halt+0x2e/0x50 [13752.152274] kvm_wait+0xcd/0xe0 [13752.153419] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13752.155272] ? newidle_balance.isra.93+0x207/0x580 [13752.157018] do_raw_spin_lock+0x177/0x190 [13752.158770] _raw_spin_lock+0x21/0x30 [13752.160206] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13752.162543] process_one_work+0x2c8/0x700 [13752.164305] worker_thread+0x296/0x6e0 [13752.165678] ? rescuer_thread+0x570/0x570 [13752.167006] kthread+0x1d1/0x200 [13752.168706] ? set_kthread_struct+0x70/0x70 [13752.170316] ret_from_fork+0x1f/0x30 [13756.057088] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [13756.058029] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [13756.058622] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [13756.058629] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13756.058697] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13756.058702] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13756.058705] RIP: 0010:native_safe_halt+0x2e/0x50 [13756.058718] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13756.058722] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13756.058726] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13756.058728] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13756.058730] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13756.058732] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13756.058734] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13756.058737] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13756.058740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13756.058742] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13756.058752] Call Trace: [13756.058758] [13756.058764] ? show_regs.cold.9+0x22/0x2f [13756.058774] ? watchdog_timer_fn.cold.6+0x62/0x10a [13756.058779] ? __hrtimer_run_queues+0x1c2/0x4d0 [13756.058784] ? watchdog+0x40/0x40 [13756.058789] ? hrtimer_interrupt+0x128/0x2c0 [13756.058792] ? smp_apic_timer_interrupt+0xa6/0x270 [13756.058799] ? apic_timer_interrupt+0xf/0x20 [13756.058803] [13756.058806] ? native_safe_halt+0x2e/0x50 [13756.058810] kvm_wait+0xcd/0xe0 [13756.058817] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13756.058826] do_raw_spin_lock+0x177/0x190 [13756.058831] _raw_spin_lock+0x21/0x30 [13756.058835] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13756.059106] ? do_raw_spin_unlock+0x75/0x190 [13756.059113] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13756.059312] ? do_wait_intr+0xf0/0xf0 [13756.059320] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13756.059524] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13756.059766] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13756.059981] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13756.060106] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13756.060150] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13756.060177] mdc_intent_lock+0x353/0x7e0 [mdc] [13756.060205] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13756.060292] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13756.060518] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13756.060548] lmv_intent_open+0x338/0x1150 [lmv] [13756.060595] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13756.060684] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13756.060770] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13756.060794] ll_intent_lock+0xc5/0xaa0 [lustre] [13756.060878] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13756.060961] ll_lookup_it+0xe0e/0x2270 [lustre] [13756.061047] ? do_raw_spin_unlock+0x75/0x190 [13756.061055] ? _raw_spin_unlock+0x12/0x30 [13756.061062] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13756.061091] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13756.061114] ll_atomic_open+0x36a/0xd70 [lustre] [13756.061205] lookup_open+0x356/0xb40 [13756.061214] do_last+0x313/0xfc0 [13756.061219] ? nd_jump_root+0xe5/0x160 [13756.061225] ? path_init+0x437/0x520 [13756.061230] path_openat+0xf7/0x500 [13756.061236] ? do_raw_spin_unlock+0x75/0x190 [13756.061241] do_filp_open+0x99/0x140 [13756.061248] ? getname_flags+0x6e/0x330 [13756.061253] ? __check_object_size+0xff/0x256 [13756.061262] ? do_raw_spin_unlock+0x75/0x190 [13756.061266] ? _raw_spin_unlock+0x12/0x30 [13756.061271] do_sys_openat2+0x2b4/0x410 [13756.061278] Modules linked in: [13756.061276] do_sys_open+0x73/0xa0 [13756.061283] lustre(O) [13756.061285] __x64_sys_openat+0x24/0x30 [13756.061290] osp(O) [13756.061294] do_syscall_64+0xc1/0x440 [13756.061302] entry_SYSCALL_64_after_hwframe+0x49/0xae [13756.061308] RIP: 0033:0x7f53425a5332 [13756.061317] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13756.061322] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13756.061327] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13756.061330] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13756.061332] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13756.061335] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13756.061337] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13756.061370] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13756.061438] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13756.061443] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13756.061446] RIP: 0010:native_safe_halt+0x2e/0x50 [13756.061453] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13756.061456] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13756.061461] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13756.061464] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13756.061466] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13756.061468] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13756.061471] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13756.061474] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13756.061477] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13756.061480] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13756.061489] Call Trace: [13756.061493] [13756.061496] ? show_regs.cold.9+0x22/0x2f [13756.061505] ? watchdog_timer_fn.cold.6+0x62/0x10a [13756.061512] ? __hrtimer_run_queues+0x1c2/0x4d0 [13756.061516] ? watchdog+0x40/0x40 [13756.061520] ? hrtimer_interrupt+0x128/0x2c0 [13756.061525] ? smp_apic_timer_interrupt+0xa6/0x270 [13756.061530] ? apic_timer_interrupt+0xf/0x20 [13756.061535] [13756.061538] ? native_safe_halt+0x2e/0x50 [13756.061542] kvm_wait+0xcd/0xe0 [13756.061550] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13756.061556] do_raw_spin_lock+0x177/0x190 [13756.061561] _raw_spin_lock+0x21/0x30 [13756.061566] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13756.062091] obd_mod_free+0x46/0x140 [ptlrpc] [13756.062399] mdc_commit_open+0x91/0xb0 [mdc] [13756.062440] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13756.062704] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13756.063083] after_reply+0xe2c/0x1780 [ptlrpc] [13756.063288] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13756.063561] ? __next_timer_interrupt+0x160/0x160 [13756.063573] ptlrpcd+0xa82/0xe40 [ptlrpc] [13756.068206] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13756.068280] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13756.068288] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13756.071068] RIP: 0010:native_safe_halt+0x2e/0x50 [13756.073073] ? do_wait_intr+0xf0/0xf0 [13756.073094] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13756.074111] kthread+0x1d1/0x200 [13756.074119] ? set_kthread_struct+0x70/0x70 [13756.074123] ret_from_fork+0x1f/0x30 [13756.487345] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13756.496544] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13756.499977] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13756.503207] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13756.506434] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13756.509519] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13756.513055] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13756.516265] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13756.519898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13756.522470] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13756.525903] Call Trace: [13756.527210] [13756.528095] ? show_regs.cold.9+0x22/0x2f [13756.530043] ? watchdog_timer_fn.cold.6+0x62/0x10a [13756.532458] ? __hrtimer_run_queues+0x1c2/0x4d0 [13756.534724] ? watchdog+0x40/0x40 [13756.535787] ? hrtimer_interrupt+0x128/0x2c0 [13756.537537] ? smp_apic_timer_interrupt+0xa6/0x270 [13756.540226] ? apic_timer_interrupt+0xf/0x20 [13756.542174] [13756.543162] ? native_safe_halt+0x2e/0x50 [13756.545081] kvm_wait+0xcd/0xe0 [13756.546325] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13756.548890] do_raw_spin_lock+0x177/0x190 [13756.550565] _raw_spin_lock+0x21/0x30 [13756.552344] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13756.555645] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13756.559899] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13756.562688] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13756.566615] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13756.569266] ? woken_wake_function+0x30/0x30 [13756.571139] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13756.575019] kthread+0x1d1/0x200 [13756.577061] ? set_kthread_struct+0x70/0x70 [13756.580193] ret_from_fork+0x1f/0x30 [13757.413635] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11120s! [13757.422030] Showing busy workqueues and worker pools: [13757.427862] workqueue events: flags=0x0 [13757.433932] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13757.443004] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13757.449967] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13757.452687] pending: cpu_latency_work [ptlrpc] [13757.456012] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13757.458939] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13757.469098] workqueue events_long: flags=0x0 [13757.471231] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13757.477143] pending: lu_objects_flush [obdclass] [13757.480207] workqueue events_unbound: flags=0x2 [13757.483267] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13757.489905] pending: flush_memcg_stats_dwork, idle_cull_fn [13757.492993] workqueue events_power_efficient: flags=0x80 [13757.494933] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13757.502204] pending: check_lifetime, do_cache_clean [13757.505377] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13757.508700] pending: neigh_periodic_work, neigh_periodic_work [13757.511114] workqueue mm_percpu_wq: flags=0x8 [13757.513331] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13757.517759] pending: vmstat_update, lru_add_drain_per_cpu [13757.520894] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13757.527214] pending: vmstat_update, lru_add_drain_per_cpu [13757.533572] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13757.536982] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13757.539731] workqueue xprtiod: flags=0xa [13757.545372] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13757.549151] pending: xprt_autoclose, xs_error_handle [13757.555884] workqueue ipv6_addrconf: flags=0x40008 [13757.559343] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13757.563688] pending: addrconf_verify_work [13757.566197] workqueue knbd0-recv: flags=0x1a [13757.571164] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13757.574886] in-flight: 651:recv_work [13757.579388] workqueue ptlrpc_pinger: flags=0x2 [13757.582019] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13757.585564] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13757.588339] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11120s workers=4 idle: 15979 31824 8 [13757.592459] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13748s workers=2 idle: 654 [13784.052225] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [13784.055141] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [13784.057155] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [13784.057175] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13784.057248] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13784.057254] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13784.057257] RIP: 0010:native_safe_halt+0x2e/0x50 [13784.057270] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13784.057274] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13784.057279] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13784.057281] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13784.057283] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13784.057286] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13784.057288] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13784.057291] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13784.057294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13784.057296] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13784.057305] Call Trace: [13784.057312] [13784.057319] ? show_regs.cold.9+0x22/0x2f [13784.057331] ? watchdog_timer_fn.cold.6+0x62/0x10a [13784.057338] ? __hrtimer_run_queues+0x1c2/0x4d0 [13784.057341] ? watchdog+0x40/0x40 [13784.057346] ? hrtimer_interrupt+0x128/0x2c0 [13784.057349] ? smp_apic_timer_interrupt+0xa6/0x270 [13784.057356] ? apic_timer_interrupt+0xf/0x20 [13784.057361] [13784.057363] ? native_safe_halt+0x2e/0x50 [13784.057367] kvm_wait+0xcd/0xe0 [13784.057374] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13784.057381] do_raw_spin_lock+0x177/0x190 [13784.057386] _raw_spin_lock+0x21/0x30 [13784.057391] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13784.057656] obd_mod_free+0x46/0x140 [ptlrpc] [13784.058024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13784.058030] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw [13784.058065] mdc_commit_open+0x91/0xb0 [mdc] [13784.058100] libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13784.058111] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13784.058119] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13784.058123] RIP: 0010:native_safe_halt+0x2e/0x50 [13784.058130] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13784.058134] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13784.058137] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13784.058140] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13784.058142] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13784.058144] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13784.058146] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13784.058149] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13784.058151] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13784.058154] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13784.058160] Call Trace: [13784.058163] [13784.058165] ? show_regs.cold.9+0x22/0x2f [13784.058173] ? watchdog_timer_fn.cold.6+0x62/0x10a [13784.058177] ? __hrtimer_run_queues+0x1c2/0x4d0 [13784.058183] ? watchdog+0x40/0x40 [13784.058259] ? hrtimer_interrupt+0x128/0x2c0 [13784.058263] ? smp_apic_timer_interrupt+0xa6/0x270 [13784.058269] ? apic_timer_interrupt+0xf/0x20 [13784.058273] [13784.058118] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13784.058276] ? native_safe_halt+0x2e/0x50 [13784.058284] kvm_wait+0xcd/0xe0 [13784.058289] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13784.058302] do_raw_spin_lock+0x177/0x190 [13784.058306] _raw_spin_lock+0x21/0x30 [13784.058310] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13784.058499] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13784.059270] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13784.059523] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13784.059847] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13784.059930] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13784.059937] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13784.059945] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13784.060840] RIP: 0010:native_safe_halt+0x2e/0x50 [13784.060852] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13784.060857] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13784.060861] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13784.060863] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [13784.060864] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13784.060866] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13784.060868] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13784.060870] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13784.060872] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13784.060874] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13784.060882] Call Trace: [13784.060886] [13784.060891] ? show_regs.cold.9+0x22/0x2f [13784.060900] ? watchdog_timer_fn.cold.6+0x62/0x10a [13784.060905] ? __hrtimer_run_queues+0x1c2/0x4d0 [13784.060908] ? watchdog+0x40/0x40 [13784.060911] ? hrtimer_interrupt+0x128/0x2c0 [13784.060914] ? smp_apic_timer_interrupt+0xa6/0x270 [13784.060919] ? apic_timer_interrupt+0xf/0x20 [13784.060922] [13784.060924] ? native_safe_halt+0x2e/0x50 [13784.060927] kvm_wait+0xcd/0xe0 [13784.060934] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13784.060940] ? newidle_balance.isra.93+0x207/0x580 [13784.060943] do_raw_spin_lock+0x177/0x190 [13784.060946] _raw_spin_lock+0x21/0x30 [13784.060949] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13784.059736] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13784.061601] process_one_work+0x2c8/0x700 [13784.061611] worker_thread+0x296/0x6e0 [13784.061616] ? rescuer_thread+0x570/0x570 [13784.061620] kthread+0x1d1/0x200 [13784.061626] ? set_kthread_struct+0x70/0x70 [13784.061628] ret_from_fork+0x1f/0x30 [13784.061729] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13784.062162] ? woken_wake_function+0x30/0x30 [13784.062173] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13784.062448] kthread+0x1d1/0x200 [13784.062454] ? set_kthread_struct+0x70/0x70 [13784.062458] ret_from_fork+0x1f/0x30 [13784.062539] after_reply+0xe2c/0x1780 [ptlrpc] [13784.072505] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13784.072588] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13784.072596] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13784.072599] RIP: 0010:native_safe_halt+0x2e/0x50 [13784.072614] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13784.072628] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13784.072634] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13784.072636] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13784.072638] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13784.072640] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13784.072643] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13784.072646] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13784.072649] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13784.072651] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13784.072660] Call Trace: [13784.072667] [13784.072672] ? show_regs.cold.9+0x22/0x2f [13784.072683] ? watchdog_timer_fn.cold.6+0x62/0x10a [13784.072690] ? __hrtimer_run_queues+0x1c2/0x4d0 [13784.072694] ? watchdog+0x40/0x40 [13784.072698] ? hrtimer_interrupt+0x128/0x2c0 [13784.072702] ? smp_apic_timer_interrupt+0xa6/0x270 [13784.072709] ? apic_timer_interrupt+0xf/0x20 [13784.072713] [13784.072716] ? native_safe_halt+0x2e/0x50 [13784.072720] kvm_wait+0xcd/0xe0 [13784.072727] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13784.072736] do_raw_spin_lock+0x177/0x190 [13784.072741] _raw_spin_lock+0x21/0x30 [13784.072745] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13784.073327] ? do_raw_spin_unlock+0x75/0x190 [13784.073334] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13784.079659] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13784.080679] ? __next_timer_interrupt+0x160/0x160 [13784.080777] ptlrpcd+0xa82/0xe40 [ptlrpc] [13784.109513] ? do_wait_intr+0xf0/0xf0 [13784.109537] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13784.109823] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13784.110048] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13784.110686] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13784.110799] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13784.110844] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13784.110874] mdc_intent_lock+0x353/0x7e0 [mdc] [13784.110901] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13784.110983] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13784.134945] ? do_wait_intr+0xf0/0xf0 [13784.134970] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13784.138967] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13784.139021] lmv_intent_open+0x338/0x1150 [lmv] [13784.139055] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13784.139187] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13784.139270] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13784.139293] ll_intent_lock+0xc5/0xaa0 [lustre] [13784.139378] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13784.139459] ll_lookup_it+0xe0e/0x2270 [lustre] [13784.141975] kthread+0x1d1/0x200 [13784.141993] ? set_kthread_struct+0x70/0x70 [13784.142068] ret_from_fork+0x1f/0x30 [13785.219864] ? do_raw_spin_unlock+0x75/0x190 [13785.222840] ? _raw_spin_unlock+0x12/0x30 [13785.225613] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13785.233178] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13785.236612] ll_atomic_open+0x36a/0xd70 [lustre] [13785.241652] lookup_open+0x356/0xb40 [13785.245854] do_last+0x313/0xfc0 [13785.248624] ? nd_jump_root+0xe5/0x160 [13785.252843] ? path_init+0x437/0x520 [13785.253975] path_openat+0xf7/0x500 [13785.257890] ? do_raw_spin_unlock+0x75/0x190 [13785.261287] do_filp_open+0x99/0x140 [13785.262939] ? getname_flags+0x6e/0x330 [13785.267903] ? __check_object_size+0xff/0x256 [13785.271395] ? do_raw_spin_unlock+0x75/0x190 [13785.272852] ? _raw_spin_unlock+0x12/0x30 [13785.274772] do_sys_openat2+0x2b4/0x410 [13785.277779] do_sys_open+0x73/0xa0 [13785.278998] __x64_sys_openat+0x24/0x30 [13785.280927] do_syscall_64+0xc1/0x440 [13785.282804] entry_SYSCALL_64_after_hwframe+0x49/0xae [13785.286326] RIP: 0033:0x7f53425a5332 [13785.287439] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13785.296741] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13785.303992] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13785.309260] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13785.311569] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13785.320844] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13785.326998] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13787.616069] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11150s! [13787.622498] Showing busy workqueues and worker pools: [13787.625679] workqueue events: flags=0x0 [13787.629671] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13787.633019] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13787.643013] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13787.645771] pending: cpu_latency_work [ptlrpc] [13787.648965] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13787.652110] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13787.656625] workqueue events_long: flags=0x0 [13787.658223] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13787.662606] pending: lu_objects_flush [obdclass] [13787.666964] workqueue events_unbound: flags=0x2 [13787.669376] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13787.671873] pending: flush_memcg_stats_dwork, idle_cull_fn [13787.675178] workqueue events_power_efficient: flags=0x80 [13787.677399] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13787.680127] pending: check_lifetime, do_cache_clean [13787.682016] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13787.685008] pending: neigh_periodic_work, neigh_periodic_work [13787.687213] workqueue mm_percpu_wq: flags=0x8 [13787.688888] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13787.692290] pending: vmstat_update, lru_add_drain_per_cpu [13787.694685] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13787.697365] pending: vmstat_update, lru_add_drain_per_cpu [13787.699988] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13787.702782] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13787.704898] workqueue xprtiod: flags=0xa [13787.706509] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13787.709581] pending: xprt_autoclose, xs_error_handle [13787.711565] workqueue ipv6_addrconf: flags=0x40008 [13787.713267] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13787.716152] pending: addrconf_verify_work [13787.717786] workqueue knbd0-recv: flags=0x1a [13787.719021] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13787.721204] in-flight: 651:recv_work [13787.722514] workqueue ptlrpc_pinger: flags=0x2 [13787.723930] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13787.726065] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13787.728372] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11150s workers=4 idle: 15979 31824 8 [13787.731464] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13778s workers=2 idle: 654 [13812.052079] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [13812.055707] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [13812.057139] Modules linked in: lustre(O) [13812.057677] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [13812.057683] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13812.057752] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13812.057758] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13812.057760] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13812.057773] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13812.057778] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13812.057783] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13812.057785] RDX: 0000000000003c9e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13812.057787] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13812.057790] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13812.057792] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13812.057795] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13812.057798] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13812.057800] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13812.057810] Call Trace: [13812.057817] [13812.057823] ? show_regs.cold.9+0x22/0x2f [13812.057834] ? watchdog_timer_fn.cold.6+0x62/0x10a [13812.057842] ? __hrtimer_run_queues+0x1c2/0x4d0 [13812.057846] ? watchdog+0x40/0x40 [13812.057851] ? hrtimer_interrupt+0x128/0x2c0 [13812.057855] ? smp_apic_timer_interrupt+0xa6/0x270 [13812.057864] ? apic_timer_interrupt+0xf/0x20 [13812.057869] [13812.057872] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13812.057876] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13812.057881] do_raw_spin_lock+0x177/0x190 [13812.057886] _raw_spin_lock+0x21/0x30 [13812.057893] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13812.058405] obd_mod_free+0x46/0x140 [ptlrpc] [13812.058626] mdc_commit_open+0x91/0xb0 [mdc] [13812.058670] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13812.058873] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13812.059071] after_reply+0xe2c/0x1780 [ptlrpc] [13812.059272] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13812.059549] ? __next_timer_interrupt+0x160/0x160 [13812.059560] ptlrpcd+0xa82/0xe40 [ptlrpc] [13812.059784] ? do_wait_intr+0xf0/0xf0 [13812.059794] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13812.059998] kthread+0x1d1/0x200 [13812.060004] ? set_kthread_struct+0x70/0x70 [13812.060008] ret_from_fork+0x1f/0x30 [13812.060110] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13812.060117] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13812.060186] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13812.060191] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13812.060194] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13812.060203] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13812.060206] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13812.060210] RAX: 0000000000001e93 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13812.060213] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13812.060215] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13812.060217] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13812.060220] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13812.060223] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13812.060226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13812.060228] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13812.060236] Call Trace: [13812.060239] [13812.060242] ? show_regs.cold.9+0x22/0x2f [13812.060251] ? watchdog_timer_fn.cold.6+0x62/0x10a [13812.060256] ? __hrtimer_run_queues+0x1c2/0x4d0 [13812.060260] ? watchdog+0x40/0x40 [13812.060265] ? hrtimer_interrupt+0x128/0x2c0 [13812.060270] ? smp_apic_timer_interrupt+0xa6/0x270 [13812.060275] ? apic_timer_interrupt+0xf/0x20 [13812.060280] [13812.060283] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13812.060287] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13812.060291] do_raw_spin_lock+0x177/0x190 [13812.060295] _raw_spin_lock+0x21/0x30 [13812.060299] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13812.060573] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13812.060783] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13812.060992] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13812.061204] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13812.061705] ? woken_wake_function+0x30/0x30 [13812.061714] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13812.061920] kthread+0x1d1/0x200 [13812.061926] ? set_kthread_struct+0x70/0x70 [13812.061930] ret_from_fork+0x1f/0x30 [13812.066768] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13812.066853] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13812.066862] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13812.066865] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13812.066880] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13812.066884] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13812.066889] RAX: 0000000000006ca5 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13812.066892] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13812.066894] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13812.066896] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13812.066899] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13812.066902] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13812.066906] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13812.066908] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13812.066918] Call Trace: [13812.066924] [13812.066929] ? show_regs.cold.9+0x22/0x2f [13812.066938] ? watchdog_timer_fn.cold.6+0x62/0x10a [13812.066945] ? __hrtimer_run_queues+0x1c2/0x4d0 [13812.066949] ? watchdog+0x40/0x40 [13812.066954] ? hrtimer_interrupt+0x128/0x2c0 [13812.066958] ? smp_apic_timer_interrupt+0xa6/0x270 [13812.066967] ? apic_timer_interrupt+0xf/0x20 [13812.066972] [13812.066975] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13812.066982] do_raw_spin_lock+0x177/0x190 [13812.066988] _raw_spin_lock+0x21/0x30 [13812.066993] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13812.072180] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13812.072273] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13812.072282] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13812.072289] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13812.073571] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13812.073607] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13812.073612] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13812.073617] RAX: 0000000000001f89 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13812.073619] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13812.073621] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13812.073624] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13812.073626] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13812.073630] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13812.073633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13812.073635] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13812.073642] Call Trace: [13812.073647] [13812.073653] ? show_regs.cold.9+0x22/0x2f [13812.073663] ? watchdog_timer_fn.cold.6+0x62/0x10a [13812.073670] ? __hrtimer_run_queues+0x1c2/0x4d0 [13812.073674] ? watchdog+0x40/0x40 [13812.073678] ? hrtimer_interrupt+0x128/0x2c0 [13812.073941] ? smp_apic_timer_interrupt+0xa6/0x270 [13812.073951] ? apic_timer_interrupt+0xf/0x20 [13812.073955] [13812.073958] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13812.073963] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13812.073967] ? newidle_balance.isra.93+0x207/0x580 [13812.073973] do_raw_spin_lock+0x177/0x190 [13812.073977] _raw_spin_lock+0x21/0x30 [13812.073982] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13812.078684] ? do_raw_spin_unlock+0x75/0x190 [13812.078709] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13812.079004] ? do_wait_intr+0xf0/0xf0 [13812.079015] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13812.079641] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13812.079870] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13812.145350] process_one_work+0x2c8/0x700 [13812.145373] worker_thread+0x296/0x6e0 [13812.145378] ? rescuer_thread+0x570/0x570 [13812.147303] kthread+0x1d1/0x200 [13812.147316] ? set_kthread_struct+0x70/0x70 [13812.147320] ret_from_fork+0x1f/0x30 [13812.837564] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13812.839441] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13812.841673] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13812.843558] mdc_intent_lock+0x353/0x7e0 [mdc] [13812.845454] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13812.847946] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13812.850405] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13812.853932] lmv_intent_open+0x338/0x1150 [lmv] [13812.856541] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13812.860714] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13812.863947] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13812.871097] ll_intent_lock+0xc5/0xaa0 [lustre] [13812.874246] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13812.876088] ll_lookup_it+0xe0e/0x2270 [lustre] [13812.877539] ? do_raw_spin_unlock+0x75/0x190 [13812.882491] ? _raw_spin_unlock+0x12/0x30 [13812.884515] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13812.888758] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13812.891039] ll_atomic_open+0x36a/0xd70 [lustre] [13812.892994] lookup_open+0x356/0xb40 [13812.894155] do_last+0x313/0xfc0 [13812.900978] ? nd_jump_root+0xe5/0x160 [13812.902462] ? path_init+0x437/0x520 [13812.907089] path_openat+0xf7/0x500 [13812.915574] ? do_raw_spin_unlock+0x75/0x190 [13812.928934] do_filp_open+0x99/0x140 [13812.930418] ? getname_flags+0x6e/0x330 [13812.935969] ? __check_object_size+0xff/0x256 [13812.937936] ? do_raw_spin_unlock+0x75/0x190 [13812.940531] ? _raw_spin_unlock+0x12/0x30 [13812.942091] do_sys_openat2+0x2b4/0x410 [13812.943843] do_sys_open+0x73/0xa0 [13812.945051] __x64_sys_openat+0x24/0x30 [13812.946449] do_syscall_64+0xc1/0x440 [13812.952652] entry_SYSCALL_64_after_hwframe+0x49/0xae [13812.955648] RIP: 0033:0x7f53425a5332 [13812.957140] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13812.966670] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13812.969899] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13812.973006] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13812.975572] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13812.984812] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13812.986947] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13817.832275] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11180s! [13817.837867] Showing busy workqueues and worker pools: [13817.841025] workqueue events: flags=0x0 [13817.848190] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13817.850280] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13817.859975] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13817.862756] pending: cpu_latency_work [ptlrpc] [13817.866433] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13817.869447] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13817.877938] workqueue events_long: flags=0x0 [13817.881645] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13817.883927] pending: lu_objects_flush [obdclass] [13817.887443] workqueue events_unbound: flags=0x2 [13817.890041] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13817.894535] pending: flush_memcg_stats_dwork, idle_cull_fn [13817.896712] workqueue events_power_efficient: flags=0x80 [13817.908452] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13817.919425] pending: check_lifetime, do_cache_clean [13817.925843] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13817.935191] pending: neigh_periodic_work, neigh_periodic_work [13817.946060] workqueue mm_percpu_wq: flags=0x8 [13817.948938] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13817.952236] pending: vmstat_update, lru_add_drain_per_cpu [13817.959851] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13817.962745] pending: vmstat_update, lru_add_drain_per_cpu [13817.966640] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13817.972371] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13817.983462] workqueue xprtiod: flags=0xa [13817.986440] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13817.990856] pending: xprt_autoclose, xs_error_handle [13817.994218] workqueue ipv6_addrconf: flags=0x40008 [13817.998444] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13818.007326] pending: addrconf_verify_work [13818.009940] workqueue knbd0-recv: flags=0x1a [13818.014284] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13818.019210] in-flight: 651:recv_work [13818.022042] workqueue ptlrpc_pinger: flags=0x2 [13818.024405] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13818.032022] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13818.035130] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11180s workers=4 idle: 15979 31824 8 [13818.054249] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13808s workers=2 idle: 654 [13840.055761] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [13840.057122] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [13840.061509] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13840.061526] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13840.062161] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13840.062167] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13840.062170] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13840.062275] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13840.062281] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13840.062285] RAX: 0000000000000f06 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13840.062287] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13840.062491] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13840.062494] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13840.062496] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13840.062499] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13840.062503] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13840.062505] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13840.062514] Call Trace: [13840.062533] [13840.062540] ? show_regs.cold.9+0x22/0x2f [13840.062550] ? watchdog_timer_fn.cold.6+0x62/0x10a [13840.062556] ? __hrtimer_run_queues+0x1c2/0x4d0 [13840.062560] ? watchdog+0x40/0x40 [13840.062565] ? hrtimer_interrupt+0x128/0x2c0 [13840.062568] ? smp_apic_timer_interrupt+0xa6/0x270 [13840.062576] ? apic_timer_interrupt+0xf/0x20 [13840.062580] [13840.062583] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13840.062587] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13840.062594] do_raw_spin_lock+0x177/0x190 [13840.062599] _raw_spin_lock+0x21/0x30 [13840.062604] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13840.065119] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13840.067690] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13840.070604] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13840.070820] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13840.071037] ? woken_wake_function+0x30/0x30 [13840.071046] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13840.071257] kthread+0x1d1/0x200 [13840.071264] ? set_kthread_struct+0x70/0x70 [13840.071268] ret_from_fork+0x1f/0x30 [13840.071502] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13840.071573] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13840.071580] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13840.071584] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13840.071594] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13840.071598] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13840.071603] RAX: 0000000000000c14 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13840.071605] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13840.071608] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13840.071610] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13840.071613] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13840.071617] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13840.071620] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13840.071622] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13840.071630] Call Trace: [13840.071635] [13840.071638] ? show_regs.cold.9+0x22/0x2f [13840.071648] ? watchdog_timer_fn.cold.6+0x62/0x10a [13840.071655] ? __hrtimer_run_queues+0x1c2/0x4d0 [13840.071660] ? watchdog+0x40/0x40 [13840.071664] ? hrtimer_interrupt+0x128/0x2c0 [13840.071668] ? smp_apic_timer_interrupt+0xa6/0x270 [13840.071674] ? apic_timer_interrupt+0xf/0x20 [13840.071679] [13840.071682] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13840.071688] do_raw_spin_lock+0x177/0x190 [13840.071693] _raw_spin_lock+0x21/0x30 [13840.071698] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13840.071920] ? do_raw_spin_unlock+0x75/0x190 [13840.071925] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13840.072122] ? do_wait_intr+0xf0/0xf0 [13840.072130] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13840.084756] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13840.099060] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13840.148784] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13840.148799] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13840.148802] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13840.148821] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13840.148825] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13840.148830] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13840.148833] RDX: 0000000000004771 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13840.148984] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13840.148986] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13840.148989] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13840.148992] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13840.148994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13840.148997] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13840.149005] Call Trace: [13840.149010] [13840.149016] ? show_regs.cold.9+0x22/0x2f [13840.149028] ? watchdog_timer_fn.cold.6+0x62/0x10a [13840.149035] ? __hrtimer_run_queues+0x1c2/0x4d0 [13840.149040] ? watchdog+0x40/0x40 [13840.149044] ? hrtimer_interrupt+0x128/0x2c0 [13840.149049] ? smp_apic_timer_interrupt+0xa6/0x270 [13840.149057] ? apic_timer_interrupt+0xf/0x20 [13840.149062] [13840.149064] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13840.149069] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13840.149073] do_raw_spin_lock+0x177/0x190 [13840.149077] _raw_spin_lock+0x21/0x30 [13840.149085] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13840.149712] obd_mod_free+0x46/0x140 [ptlrpc] [13840.149990] mdc_commit_open+0x91/0xb0 [mdc] [13840.150193] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13840.150659] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13840.151169] after_reply+0xe2c/0x1780 [ptlrpc] [13840.151532] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13840.151930] ? __next_timer_interrupt+0x160/0x160 [13840.151941] ptlrpcd+0xa82/0xe40 [ptlrpc] [13840.152577] ? do_wait_intr+0xf0/0xf0 [13840.152587] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13840.152800] kthread+0x1d1/0x200 [13840.152807] ? set_kthread_struct+0x70/0x70 [13840.152811] ret_from_fork+0x1f/0x30 [13841.235506] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13841.239870] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13841.247055] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13841.252064] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13841.254530] mdc_intent_lock+0x353/0x7e0 [mdc] [13841.258893] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13841.260857] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13841.274346] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13841.277979] lmv_intent_open+0x338/0x1150 [lmv] [13841.281079] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13841.285911] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13841.289394] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13841.293804] ll_intent_lock+0xc5/0xaa0 [lustre] [13841.301579] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13841.306582] ll_lookup_it+0xe0e/0x2270 [lustre] [13841.312493] ? do_raw_spin_unlock+0x75/0x190 [13841.314700] ? _raw_spin_unlock+0x12/0x30 [13841.320801] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13841.325336] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13841.328030] ll_atomic_open+0x36a/0xd70 [lustre] [13841.332377] lookup_open+0x356/0xb40 [13841.334615] do_last+0x313/0xfc0 [13841.340516] ? nd_jump_root+0xe5/0x160 [13841.343754] ? path_init+0x437/0x520 [13841.345943] path_openat+0xf7/0x500 [13841.348458] ? do_raw_spin_unlock+0x75/0x190 [13841.352531] do_filp_open+0x99/0x140 [13841.357838] ? getname_flags+0x6e/0x330 [13841.358958] ? __check_object_size+0xff/0x256 [13841.365697] ? do_raw_spin_unlock+0x75/0x190 [13841.369384] ? _raw_spin_unlock+0x12/0x30 [13841.373408] do_sys_openat2+0x2b4/0x410 [13841.374770] do_sys_open+0x73/0xa0 [13841.376016] __x64_sys_openat+0x24/0x30 [13841.386636] do_syscall_64+0xc1/0x440 [13841.391142] entry_SYSCALL_64_after_hwframe+0x49/0xae [13841.395568] RIP: 0033:0x7f53425a5332 [13841.400803] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13841.421742] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13841.431773] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13841.442342] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13841.450506] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13841.461584] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13841.469551] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13844.060967] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [13844.067805] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13844.124727] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13844.156279] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13844.168298] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13844.183248] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13844.190522] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13844.231913] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13844.252559] RAX: 0000000000004e0e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13844.260381] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13844.270224] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13844.280497] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13844.292988] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13844.305909] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13844.319631] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13844.321665] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13844.333525] Call Trace: [13844.341456] [13844.342244] ? show_regs.cold.9+0x22/0x2f [13844.345330] ? watchdog_timer_fn.cold.6+0x62/0x10a [13844.358385] ? __hrtimer_run_queues+0x1c2/0x4d0 [13844.368132] ? watchdog+0x40/0x40 [13844.369407] ? hrtimer_interrupt+0x128/0x2c0 [13844.370875] ? smp_apic_timer_interrupt+0xa6/0x270 [13844.383383] ? apic_timer_interrupt+0xf/0x20 [13844.394158] [13844.395017] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13844.397833] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13844.406831] ? newidle_balance.isra.93+0x207/0x580 [13844.414723] do_raw_spin_lock+0x177/0x190 [13844.420413] _raw_spin_lock+0x21/0x30 [13844.427642] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13844.432181] process_one_work+0x2c8/0x700 [13844.439969] worker_thread+0x296/0x6e0 [13844.441782] ? rescuer_thread+0x570/0x570 [13844.448762] kthread+0x1d1/0x200 [13844.449964] ? set_kthread_struct+0x70/0x70 [13844.460263] ret_from_fork+0x1f/0x30 [13848.547464] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11211s! [13848.558719] Showing busy workqueues and worker pools: [13848.571588] workqueue events: flags=0x0 [13848.574529] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13848.588595] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13848.601233] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13848.611690] pending: cpu_latency_work [ptlrpc] [13848.615895] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13848.620829] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13848.641460] workqueue events_long: flags=0x0 [13848.645658] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13848.653419] pending: lu_objects_flush [obdclass] [13848.661115] workqueue events_unbound: flags=0x2 [13848.665331] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13848.668288] pending: flush_memcg_stats_dwork, idle_cull_fn [13848.671342] workqueue events_power_efficient: flags=0x80 [13848.674396] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13848.676657] pending: check_lifetime, do_cache_clean [13848.679069] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13848.681709] pending: neigh_periodic_work, neigh_periodic_work [13848.691364] workqueue mm_percpu_wq: flags=0x8 [13848.696996] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13848.703550] pending: vmstat_update, lru_add_drain_per_cpu [13848.719683] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13848.729621] pending: vmstat_update, lru_add_drain_per_cpu [13848.740507] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13848.744524] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13848.754720] workqueue xprtiod: flags=0xa [13848.764648] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13848.770673] pending: xprt_autoclose, xs_error_handle [13848.774836] workqueue ipv6_addrconf: flags=0x40008 [13848.788606] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13848.792352] pending: addrconf_verify_work [13848.796585] workqueue knbd0-recv: flags=0x1a [13848.801469] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13848.810300] in-flight: 651:recv_work [13848.816395] workqueue ptlrpc_pinger: flags=0x2 [13848.822024] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13848.830933] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13848.842730] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11211s workers=4 idle: 15979 31824 8 [13848.853630] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13839s workers=2 idle: 654 [13854.710152] rcu: INFO: rcu_sched self-detected stall on CPU [13854.717741] rcu: 2-....: (6550399 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [13854.731433] (t=11760284 jiffies g=274185 q=66899) [13854.736941] rcu: rcu_sched kthread starved for 11218650 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [13854.745772] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [13854.759683] rcu: RCU grace-period kthread stack dump: [13854.766282] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [13854.773435] Call Trace: [13854.780267] __schedule+0x351/0xcb0 [13854.782976] schedule+0xc0/0x180 [13854.786778] schedule_timeout+0xb4/0x190 [13854.794128] ? do_raw_spin_unlock+0x75/0x190 [13854.799865] ? __next_timer_interrupt+0x160/0x160 [13854.811692] ? prepare_to_swait_event+0xf3/0x140 [13854.816994] rcu_gp_kthread+0x8dd/0x15b0 [13854.825844] ? force_qs_rnp+0x2c0/0x2c0 [13854.833209] kthread+0x1d1/0x200 [13854.839145] ? set_kthread_struct+0x70/0x70 [13854.846586] ret_from_fork+0x1f/0x30 [13854.850410] rcu: Stack dump where RCU GP kthread last ran: [13854.864337] Sending NMI from CPU 2 to CPUs 0: [13854.875166] NMI backtrace for cpu 0 [13854.875173] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13854.875175] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13854.875177] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13854.875179] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13854.875180] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13854.875182] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 [13854.875184] RAX: 0000000000007983 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13854.875186] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13854.875187] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13854.875188] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13854.875190] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13854.875191] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13854.875192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13854.875193] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13854.875195] Call Trace: [13854.875195] [13854.875196] ? show_regs.cold.9+0x22/0x2f [13854.875197] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [13854.875198] ? nmi_cpu_backtrace_handler+0x15/0x40 [13854.875199] ? nmi_handle+0xbd/0x210 [13854.875200] ? default_do_nmi+0x4b/0x1b0 [13854.875201] ? do_nmi+0x280/0x380 [13854.875202] ? end_repeat_nmi+0x16/0x1f [13854.875203] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13854.875204] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13854.875205] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13854.875206] [13854.875208] ? newidle_balance.isra.93+0x207/0x580 [13854.875209] do_raw_spin_lock+0x177/0x190 [13854.875209] _raw_spin_lock+0x21/0x30 [13854.875210] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13854.875212] process_one_work+0x2c8/0x700 [13854.875213] worker_thread+0x296/0x6e0 [13854.875214] ? rescuer_thread+0x570/0x570 [13854.875215] kthread+0x1d1/0x200 [13854.875215] ? set_kthread_struct+0x70/0x70 [13854.875216] ret_from_fork+0x1f/0x30 [13854.876174] Sending NMI from CPU 2 to CPUs 1: [13855.164117] NMI backtrace for cpu 1 [13855.164125] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13855.164127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13855.164129] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13855.164131] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13855.164134] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [13855.164136] RAX: 0000000000005581 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13855.164138] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13855.164139] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13855.164141] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13855.164142] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13855.164143] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13855.164145] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13855.164146] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13855.164147] Call Trace: [13855.164148] [13855.164149] ? show_regs.cold.9+0x22/0x2f [13855.164149] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [13855.164150] ? nmi_cpu_backtrace_handler+0x15/0x40 [13855.164151] ? nmi_handle+0xbd/0x210 [13855.164153] ? default_do_nmi+0x4b/0x1b0 [13855.164153] ? do_nmi+0x280/0x380 [13855.164154] ? end_repeat_nmi+0x16/0x1f [13855.164155] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13855.164157] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13855.164158] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13855.164159] [13855.164160] do_raw_spin_lock+0x177/0x190 [13855.164161] _raw_spin_lock+0x21/0x30 [13855.164162] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13855.164163] ? do_raw_spin_unlock+0x75/0x190 [13855.164164] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13855.164165] ? do_wait_intr+0xf0/0xf0 [13855.164166] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13855.164167] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13855.164168] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13855.164169] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13855.164170] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13855.164171] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13855.164172] mdc_intent_lock+0x353/0x7e0 [mdc] [13855.164173] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13855.164174] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13855.164175] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13855.164176] lmv_intent_open+0x338/0x1150 [lmv] [13855.164177] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13855.164179] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13855.164180] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13855.164180] ll_intent_lock+0xc5/0xaa0 [lustre] [13855.164181] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13855.164182] ll_lookup_it+0xe0e/0x2270 [lustre] [13855.164183] ? do_raw_spin_unlock+0x75/0x190 [13855.164184] ? _raw_spin_unlock+0x12/0x30 [13855.164186] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13855.164186] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13855.164187] ll_atomic_open+0x36a/0xd70 [lustre] [13855.164188] lookup_open+0x356/0xb40 [13855.164189] do_last+0x313/0xfc0 [13855.164190] ? nd_jump_root+0xe5/0x160 [13855.164192] ? path_init+0x437/0x520 [13855.164192] path_openat+0xf7/0x500 [13855.164193] ? do_raw_spin_unlock+0x75/0x190 [13855.164194] do_filp_open+0x99/0x140 [13855.164195] ? getname_flags+0x6e/0x330 [13855.164196] ? __check_object_size+0xff/0x256 [13855.164198] ? do_raw_spin_unlock+0x75/0x190 [13855.164199] ? _raw_spin_unlock+0x12/0x30 [13855.164200] do_sys_openat2+0x2b4/0x410 [13855.164200] do_sys_open+0x73/0xa0 [13855.164202] __x64_sys_openat+0x24/0x30 [13855.164202] do_syscall_64+0xc1/0x440 [13855.164204] entry_SYSCALL_64_after_hwframe+0x49/0xae [13855.164204] RIP: 0033:0x7f53425a5332 [13855.164206] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13855.164208] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13855.164210] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13855.164211] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13855.164213] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13855.164214] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13855.164216] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13855.164872] NMI backtrace for cpu 2 [13855.662685] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13855.679555] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13855.699224] Call Trace: [13855.702419] [13855.705547] ? dump_stack+0xbb/0x10e [13855.707065] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [13855.708464] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [13855.715930] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [13855.717631] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [13855.731081] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [13855.738828] ? rcu_dump_cpu_stacks+0x11b/0x171 [13855.751789] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [13855.758446] ? update_process_times+0x6d/0xc0 [13855.761253] ? tick_sched_handle.isra.18+0x27/0x90 [13855.767728] ? tick_sched_timer+0x53/0xb0 [13855.776106] ? __hrtimer_run_queues+0x1c2/0x4d0 [13855.779396] ? tick_sched_handle.isra.18+0x90/0x90 [13855.780807] ? hrtimer_interrupt+0x128/0x2c0 [13855.789237] ? smp_apic_timer_interrupt+0xa6/0x270 [13855.793090] ? apic_timer_interrupt+0xf/0x20 [13855.796708] [13855.802201] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13855.813430] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13855.822460] ? do_raw_spin_lock+0x177/0x190 [13855.826715] ? _raw_spin_lock+0x21/0x30 [13855.829953] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13855.832752] ? obd_mod_free+0x46/0x140 [ptlrpc] [13855.837022] ? mdc_commit_open+0x91/0xb0 [mdc] [13855.843800] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13855.845552] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13855.851952] ? after_reply+0xe2c/0x1780 [ptlrpc] [13855.857082] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13855.867653] ? __next_timer_interrupt+0x160/0x160 [13855.870671] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [13855.879581] ? do_wait_intr+0xf0/0xf0 [13855.881803] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13855.891607] ? kthread+0x1d1/0x200 [13855.895653] ? set_kthread_struct+0x70/0x70 [13855.902777] ? ret_from_fork+0x1f/0x30 [13868.062991] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13868.073346] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13868.126075] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13868.136925] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13868.144034] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13868.147243] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13868.170218] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13868.174543] RAX: 00000000000077a6 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13868.180378] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13868.187244] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13868.193039] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13868.199905] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13868.204813] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13868.209018] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13868.212165] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13868.217378] Call Trace: [13868.220393] [13868.221519] ? show_regs.cold.9+0x22/0x2f [13868.225365] ? watchdog_timer_fn.cold.6+0x62/0x10a [13868.231135] ? __hrtimer_run_queues+0x1c2/0x4d0 [13868.232577] ? watchdog+0x40/0x40 [13868.239526] ? hrtimer_interrupt+0x128/0x2c0 [13868.253285] ? smp_apic_timer_interrupt+0xa6/0x270 [13868.257987] ? apic_timer_interrupt+0xf/0x20 [13868.267779] [13868.269959] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13868.282114] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13868.284484] do_raw_spin_lock+0x177/0x190 [13868.287042] _raw_spin_lock+0x21/0x30 [13868.288558] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13868.298182] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13868.302333] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13868.311282] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13868.315763] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13868.320354] ? woken_wake_function+0x30/0x30 [13868.325736] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13868.331420] kthread+0x1d1/0x200 [13868.336040] ? set_kthread_struct+0x70/0x70 [13868.339993] ret_from_fork+0x1f/0x30 [13879.264166] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11242s! [13879.277376] Showing busy workqueues and worker pools: [13879.285114] workqueue events: flags=0x0 [13879.289505] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13879.294554] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13879.307160] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13879.319486] pending: cpu_latency_work [ptlrpc] [13879.326975] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13879.332756] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13879.341692] workqueue events_long: flags=0x0 [13879.343223] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13879.354005] pending: lu_objects_flush [obdclass] [13879.356574] workqueue events_unbound: flags=0x2 [13879.358852] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13879.364323] pending: flush_memcg_stats_dwork, idle_cull_fn [13879.367581] workqueue events_power_efficient: flags=0x80 [13879.372324] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13879.377733] pending: check_lifetime, do_cache_clean [13879.382973] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13879.388122] pending: neigh_periodic_work, neigh_periodic_work [13879.391840] workqueue mm_percpu_wq: flags=0x8 [13879.393368] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13879.406814] pending: vmstat_update, lru_add_drain_per_cpu [13879.410184] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13879.414002] pending: vmstat_update, lru_add_drain_per_cpu [13879.419178] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13879.424030] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13879.431806] workqueue xprtiod: flags=0xa [13879.433151] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13879.437755] pending: xprt_autoclose, xs_error_handle [13879.439578] workqueue ipv6_addrconf: flags=0x40008 [13879.446069] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13879.448318] pending: addrconf_verify_work [13879.458881] workqueue knbd0-recv: flags=0x1a [13879.465851] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13879.473395] in-flight: 651:recv_work [13879.475932] workqueue ptlrpc_pinger: flags=0x2 [13879.479079] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13879.481937] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13879.485972] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11242s workers=4 idle: 15979 31824 8 [13879.493126] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13869s workers=2 idle: 654 [13880.056460] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [13880.058091] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [13880.063386] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13880.063471] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13880.063479] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13880.063482] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13880.063496] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13880.063501] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13880.063506] RAX: 0000000000001d11 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13880.063509] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13880.063511] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13880.063513] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13880.063516] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13880.063519] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13880.063523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13880.063525] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13880.063535] Call Trace: [13880.063540] [13880.063546] ? show_regs.cold.9+0x22/0x2f [13880.063555] ? watchdog_timer_fn.cold.6+0x62/0x10a [13880.063562] ? __hrtimer_run_queues+0x1c2/0x4d0 [13880.063567] ? watchdog+0x40/0x40 [13880.063571] ? hrtimer_interrupt+0x128/0x2c0 [13880.063575] ? smp_apic_timer_interrupt+0xa6/0x270 [13880.063584] ? apic_timer_interrupt+0xf/0x20 [13880.063589] [13880.063592] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13880.063599] do_raw_spin_lock+0x177/0x190 [13880.063605] _raw_spin_lock+0x21/0x30 [13880.063610] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13880.068104] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13880.068237] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13880.068245] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13880.068248] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13880.068395] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13880.068400] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13880.068406] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13880.068416] RDX: 000000000000759b RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13880.068421] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13880.068424] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13880.068426] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13880.068430] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13880.068433] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13880.068436] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13880.068450] Call Trace: [13880.068457] [13880.068463] ? show_regs.cold.9+0x22/0x2f [13880.068476] ? watchdog_timer_fn.cold.6+0x62/0x10a [13880.068482] ? __hrtimer_run_queues+0x1c2/0x4d0 [13880.068486] ? watchdog+0x40/0x40 [13880.068491] ? hrtimer_interrupt+0x128/0x2c0 [13880.068495] ? smp_apic_timer_interrupt+0xa6/0x270 [13880.068503] ? apic_timer_interrupt+0xf/0x20 [13880.068508] [13880.068511] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13880.068515] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13880.068521] do_raw_spin_lock+0x177/0x190 [13880.068526] _raw_spin_lock+0x21/0x30 [13880.068538] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13880.069509] obd_mod_free+0x46/0x140 [ptlrpc] [13880.069762] mdc_commit_open+0x91/0xb0 [mdc] [13880.069798] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13880.069996] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13880.070207] after_reply+0xe2c/0x1780 [ptlrpc] [13880.070402] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13880.070680] ? __next_timer_interrupt+0x160/0x160 [13880.070691] ptlrpcd+0xa82/0xe40 [ptlrpc] [13880.129915] ? do_raw_spin_unlock+0x75/0x190 [13880.129936] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13880.130211] ? do_wait_intr+0xf0/0xf0 [13880.130221] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13880.136557] ? do_wait_intr+0xf0/0xf0 [13880.136581] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13880.137057] kthread+0x1d1/0x200 [13880.137064] ? set_kthread_struct+0x70/0x70 [13880.137068] ret_from_fork+0x1f/0x30 [13880.447232] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13880.452927] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13880.466205] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13880.467610] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13880.469663] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13880.472238] mdc_intent_lock+0x353/0x7e0 [mdc] [13880.476764] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13880.483970] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13880.491848] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13880.498138] lmv_intent_open+0x338/0x1150 [lmv] [13880.503956] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13880.508663] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13880.511999] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13880.520329] ll_intent_lock+0xc5/0xaa0 [lustre] [13880.522590] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13880.524984] ll_lookup_it+0xe0e/0x2270 [lustre] [13880.527246] ? do_raw_spin_unlock+0x75/0x190 [13880.530935] ? _raw_spin_unlock+0x12/0x30 [13880.532856] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13880.535200] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13880.541351] ll_atomic_open+0x36a/0xd70 [lustre] [13880.543685] lookup_open+0x356/0xb40 [13880.546587] do_last+0x313/0xfc0 [13880.548146] ? nd_jump_root+0xe5/0x160 [13880.549867] ? path_init+0x437/0x520 [13880.551733] path_openat+0xf7/0x500 [13880.553615] ? do_raw_spin_unlock+0x75/0x190 [13880.556631] do_filp_open+0x99/0x140 [13880.559843] ? getname_flags+0x6e/0x330 [13880.563787] ? __check_object_size+0xff/0x256 [13880.567953] ? do_raw_spin_unlock+0x75/0x190 [13880.573588] ? _raw_spin_unlock+0x12/0x30 [13880.581226] do_sys_openat2+0x2b4/0x410 [13880.582564] do_sys_open+0x73/0xa0 [13880.589350] __x64_sys_openat+0x24/0x30 [13880.590594] do_syscall_64+0xc1/0x440 [13880.591649] entry_SYSCALL_64_after_hwframe+0x49/0xae [13880.600996] RIP: 0033:0x7f53425a5332 [13880.602241] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13880.627316] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13880.633659] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13880.643181] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13880.649602] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13880.656964] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13880.665092] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13896.062885] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13896.075347] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13896.115601] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13896.128622] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13896.142886] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13896.147081] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13896.164306] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13896.180419] RAX: 0000000000007f04 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13896.188217] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13896.193818] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13896.203514] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13896.208412] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13896.213529] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13896.216973] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13896.219602] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13896.222472] Call Trace: [13896.223551] [13896.224420] ? show_regs.cold.9+0x22/0x2f [13896.226044] ? watchdog_timer_fn.cold.6+0x62/0x10a [13896.228049] ? __hrtimer_run_queues+0x1c2/0x4d0 [13896.230021] ? watchdog+0x40/0x40 [13896.231540] ? hrtimer_interrupt+0x128/0x2c0 [13896.233677] ? smp_apic_timer_interrupt+0xa6/0x270 [13896.235645] ? apic_timer_interrupt+0xf/0x20 [13896.237428] [13896.238343] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13896.240567] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13896.246613] do_raw_spin_lock+0x177/0x190 [13896.248257] _raw_spin_lock+0x21/0x30 [13896.250896] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13896.255798] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13896.258530] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13896.261087] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13896.266069] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13896.269309] ? woken_wake_function+0x30/0x30 [13896.271568] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13896.274678] kthread+0x1d1/0x200 [13896.276044] ? set_kthread_struct+0x70/0x70 [13896.278056] ret_from_fork+0x1f/0x30 [13904.052015] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [13904.057868] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13904.112423] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13904.121507] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13904.127414] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13904.132180] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13904.138653] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13904.163261] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13904.168675] RAX: 0000000000003e7c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13904.175625] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13904.178634] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13904.181517] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13904.183821] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13904.186808] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13904.189888] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13904.191597] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13904.194232] Call Trace: [13904.195310] [13904.198757] ? show_regs.cold.9+0x22/0x2f [13904.202426] ? watchdog_timer_fn.cold.6+0x62/0x10a [13904.205499] ? __hrtimer_run_queues+0x1c2/0x4d0 [13904.207607] ? watchdog+0x40/0x40 [13904.209254] ? hrtimer_interrupt+0x128/0x2c0 [13904.211229] ? smp_apic_timer_interrupt+0xa6/0x270 [13904.213796] ? apic_timer_interrupt+0xf/0x20 [13904.215735] [13904.216706] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13904.219262] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13904.221750] ? newidle_balance.isra.93+0x207/0x580 [13904.223843] do_raw_spin_lock+0x177/0x190 [13904.225650] _raw_spin_lock+0x21/0x30 [13904.227340] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13904.229789] process_one_work+0x2c8/0x700 [13904.231642] worker_thread+0x296/0x6e0 [13904.233286] ? rescuer_thread+0x570/0x570 [13904.234975] kthread+0x1d1/0x200 [13904.236629] ? set_kthread_struct+0x70/0x70 [13904.238649] ret_from_fork+0x1f/0x30 [13908.057138] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [13908.059411] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [13908.061279] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13908.061371] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13908.061378] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13908.061381] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13908.061402] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13908.061406] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13908.061410] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13908.061413] RDX: 000000000000173d RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13908.061415] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13908.061418] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13908.061420] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13908.061426] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13908.061430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13908.061435] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13908.061442] Call Trace: [13908.061456] [13908.061462] ? show_regs.cold.9+0x22/0x2f [13908.061471] ? watchdog_timer_fn.cold.6+0x62/0x10a [13908.061477] ? __hrtimer_run_queues+0x1c2/0x4d0 [13908.061482] ? watchdog+0x40/0x40 [13908.061486] ? hrtimer_interrupt+0x128/0x2c0 [13908.061490] ? smp_apic_timer_interrupt+0xa6/0x270 [13908.061498] ? apic_timer_interrupt+0xf/0x20 [13908.061503] [13908.061506] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13908.061511] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13908.061516] do_raw_spin_lock+0x177/0x190 [13908.061520] _raw_spin_lock+0x21/0x30 [13908.061526] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13908.061801] obd_mod_free+0x46/0x140 [ptlrpc] [13908.073703] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13908.073775] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13908.073782] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13908.073785] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13908.073799] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13908.073804] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13908.073808] RAX: 0000000000006b11 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13908.073810] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13908.073812] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13908.073814] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13908.073817] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13908.073820] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13908.073823] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13908.073825] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13908.073835] Call Trace: [13908.073841] [13908.073846] ? show_regs.cold.9+0x22/0x2f [13908.073864] ? watchdog_timer_fn.cold.6+0x62/0x10a [13908.073877] ? __hrtimer_run_queues+0x1c2/0x4d0 [13908.073882] ? watchdog+0x40/0x40 [13908.073886] ? hrtimer_interrupt+0x128/0x2c0 [13908.073890] ? smp_apic_timer_interrupt+0xa6/0x270 [13908.073898] ? apic_timer_interrupt+0xf/0x20 [13908.073903] [13908.073906] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13908.073913] do_raw_spin_lock+0x177/0x190 [13908.073930] _raw_spin_lock+0x21/0x30 [13908.073936] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13908.095458] mdc_commit_open+0x91/0xb0 [mdc] [13908.095516] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13908.095803] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13908.096147] after_reply+0xe2c/0x1780 [ptlrpc] [13908.096412] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13908.096602] ? __next_timer_interrupt+0x160/0x160 [13908.096613] ptlrpcd+0xa82/0xe40 [ptlrpc] [13908.096810] ? do_wait_intr+0xf0/0xf0 [13908.096820] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13908.097294] kthread+0x1d1/0x200 [13908.097301] ? set_kthread_struct+0x70/0x70 [13908.097305] ret_from_fork+0x1f/0x30 [13908.457775] ? do_raw_spin_unlock+0x75/0x190 [13908.459446] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13908.464750] ? do_wait_intr+0xf0/0xf0 [13908.468658] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13908.472672] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13908.474364] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13908.477772] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13908.479580] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13908.483401] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13908.484682] mdc_intent_lock+0x353/0x7e0 [mdc] [13908.486880] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13908.490931] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13908.492827] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13908.498749] lmv_intent_open+0x338/0x1150 [lmv] [13908.502693] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13908.505128] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13908.509549] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13908.513063] ll_intent_lock+0xc5/0xaa0 [lustre] [13908.516115] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13908.517864] ll_lookup_it+0xe0e/0x2270 [lustre] [13908.523637] ? do_raw_spin_unlock+0x75/0x190 [13908.525261] ? _raw_spin_unlock+0x12/0x30 [13908.537188] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13908.538901] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13908.546121] ll_atomic_open+0x36a/0xd70 [lustre] [13908.551082] lookup_open+0x356/0xb40 [13908.553311] do_last+0x313/0xfc0 [13908.556533] ? nd_jump_root+0xe5/0x160 [13908.565077] ? path_init+0x437/0x520 [13908.568899] path_openat+0xf7/0x500 [13908.572778] ? do_raw_spin_unlock+0x75/0x190 [13908.574452] do_filp_open+0x99/0x140 [13908.576493] ? getname_flags+0x6e/0x330 [13908.582102] ? __check_object_size+0xff/0x256 [13908.583717] ? do_raw_spin_unlock+0x75/0x190 [13908.585290] ? _raw_spin_unlock+0x12/0x30 [13908.588550] do_sys_openat2+0x2b4/0x410 [13908.591444] do_sys_open+0x73/0xa0 [13908.592507] __x64_sys_openat+0x24/0x30 [13908.595819] do_syscall_64+0xc1/0x440 [13908.598756] entry_SYSCALL_64_after_hwframe+0x49/0xae [13908.603873] RIP: 0033:0x7f53425a5332 [13908.606101] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13908.619266] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13908.623001] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13908.625098] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13908.633537] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13908.638823] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13908.642025] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13909.986038] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11272s! [13909.989274] Showing busy workqueues and worker pools: [13909.995046] workqueue events: flags=0x0 [13909.998132] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13910.006816] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13910.019230] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13910.022640] pending: cpu_latency_work [ptlrpc] [13910.033323] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13910.037918] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13910.049272] workqueue events_long: flags=0x0 [13910.050711] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13910.058121] pending: lu_objects_flush [obdclass] [13910.063244] workqueue events_unbound: flags=0x2 [13910.067269] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13910.075940] pending: flush_memcg_stats_dwork, idle_cull_fn [13910.083102] workqueue events_power_efficient: flags=0x80 [13910.087262] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13910.092768] pending: check_lifetime, do_cache_clean [13910.096215] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13910.103028] pending: neigh_periodic_work, neigh_periodic_work [13910.110180] workqueue mm_percpu_wq: flags=0x8 [13910.113436] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13910.120939] pending: vmstat_update, lru_add_drain_per_cpu [13910.122623] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13910.128669] pending: vmstat_update, lru_add_drain_per_cpu [13910.132977] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13910.138400] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13910.140193] workqueue xprtiod: flags=0xa [13910.147011] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13910.157118] pending: xprt_autoclose, xs_error_handle [13910.160415] workqueue ipv6_addrconf: flags=0x40008 [13910.165799] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13910.170979] pending: addrconf_verify_work [13910.174206] workqueue knbd0-recv: flags=0x1a [13910.176373] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13910.180689] in-flight: 651:recv_work [13910.183594] workqueue ptlrpc_pinger: flags=0x2 [13910.185472] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13910.188476] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13910.204230] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11273s workers=4 idle: 15979 31824 8 [13910.210417] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13900s workers=2 idle: 654 [13924.062017] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13924.071423] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13924.102584] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13924.109349] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13924.115064] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13924.117013] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13924.127170] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13924.130171] RAX: 0000000000002788 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13924.133095] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13924.138175] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13924.141018] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13924.144115] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13924.147113] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13924.153874] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13924.161188] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13924.167613] Call Trace: [13924.170167] [13924.173318] ? show_regs.cold.9+0x22/0x2f [13924.175293] ? watchdog_timer_fn.cold.6+0x62/0x10a [13924.178190] ? __hrtimer_run_queues+0x1c2/0x4d0 [13924.180558] ? watchdog+0x40/0x40 [13924.181752] ? hrtimer_interrupt+0x128/0x2c0 [13924.188144] ? smp_apic_timer_interrupt+0xa6/0x270 [13924.190933] ? apic_timer_interrupt+0xf/0x20 [13924.193087] [13924.195176] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13924.200698] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13924.203079] do_raw_spin_lock+0x177/0x190 [13924.206071] _raw_spin_lock+0x21/0x30 [13924.208109] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13924.212479] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13924.217575] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13924.221441] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13924.226118] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13924.228236] ? woken_wake_function+0x30/0x30 [13924.230547] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13924.241502] kthread+0x1d1/0x200 [13924.244554] ? set_kthread_struct+0x70/0x70 [13924.253300] ret_from_fork+0x1f/0x30 [13936.052090] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [13936.056610] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [13936.059034] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13936.059116] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13936.059125] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13936.059132] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13936.059578] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13936.059590] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13936.059594] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13936.059599] RAX: 000000000000448c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13936.059602] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13936.059605] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13936.059607] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13936.059609] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13936.059613] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13936.059616] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13936.059619] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13936.059628] Call Trace: [13936.059633] [13936.059638] ? show_regs.cold.9+0x22/0x2f [13936.059648] ? watchdog_timer_fn.cold.6+0x62/0x10a [13936.059668] ? __hrtimer_run_queues+0x1c2/0x4d0 [13936.059674] ? watchdog+0x40/0x40 [13936.059679] ? hrtimer_interrupt+0x128/0x2c0 [13936.059683] ? smp_apic_timer_interrupt+0xa6/0x270 [13936.059692] ? apic_timer_interrupt+0xf/0x20 [13936.059697] [13936.059700] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13936.059705] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13936.059709] ? newidle_balance.isra.93+0x207/0x580 [13936.059715] do_raw_spin_lock+0x177/0x190 [13936.059720] _raw_spin_lock+0x21/0x30 [13936.059725] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13936.060116] process_one_work+0x2c8/0x700 [13936.060123] worker_thread+0x296/0x6e0 [13936.060128] ? rescuer_thread+0x570/0x570 [13936.060132] kthread+0x1d1/0x200 [13936.060136] ? set_kthread_struct+0x70/0x70 [13936.060141] ret_from_fork+0x1f/0x30 [13936.060860] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [13936.060868] Modules linked in: lustre(O) osp(O) [13936.066218] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13936.066299] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13936.066307] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13936.066310] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13936.066323] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13936.066327] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13936.066332] RAX: 0000000000007495 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13936.066335] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13936.066337] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13936.066340] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13936.066342] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13936.066345] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13936.066348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13936.066350] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13936.066359] Call Trace: [13936.066365] [13936.066372] ? show_regs.cold.9+0x22/0x2f [13936.066381] ? watchdog_timer_fn.cold.6+0x62/0x10a [13936.066388] ? __hrtimer_run_queues+0x1c2/0x4d0 [13936.066392] ? watchdog+0x40/0x40 [13936.066396] ? hrtimer_interrupt+0x128/0x2c0 [13936.066400] ? smp_apic_timer_interrupt+0xa6/0x270 [13936.066409] ? apic_timer_interrupt+0xf/0x20 [13936.066414] [13936.066417] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13936.102284] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13936.102370] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13936.102379] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13936.102382] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13936.102396] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13936.102401] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13936.102405] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13936.102408] RDX: 0000000000006ba1 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13936.102410] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13936.102412] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13936.102415] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13936.102418] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13936.102421] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13936.102688] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13936.102701] Call Trace: [13936.102708] [13936.102714] ? show_regs.cold.9+0x22/0x2f [13936.114685] do_raw_spin_lock+0x177/0x190 [13936.114710] _raw_spin_lock+0x21/0x30 [13936.114719] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13936.115004] ? do_raw_spin_unlock+0x75/0x190 [13936.115011] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13936.115225] ? do_wait_intr+0xf0/0xf0 [13936.115234] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13936.115466] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13936.115714] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13936.115961] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13936.116082] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13936.116122] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13936.116164] mdc_intent_lock+0x353/0x7e0 [mdc] [13936.116197] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13936.116291] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13936.116543] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13936.116578] lmv_intent_open+0x338/0x1150 [lmv] [13936.116616] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13936.116717] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13936.116812] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13936.116835] ll_intent_lock+0xc5/0xaa0 [lustre] [13936.117497] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13936.125603] ? watchdog_timer_fn.cold.6+0x62/0x10a [13936.125627] ? __hrtimer_run_queues+0x1c2/0x4d0 [13936.125631] ? watchdog+0x40/0x40 [13936.125635] ? hrtimer_interrupt+0x128/0x2c0 [13936.125639] ? smp_apic_timer_interrupt+0xa6/0x270 [13936.125652] ? apic_timer_interrupt+0xf/0x20 [13936.125657] [13936.125662] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13936.125669] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13936.125674] do_raw_spin_lock+0x177/0x190 [13936.125680] _raw_spin_lock+0x21/0x30 [13936.125688] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13936.126006] obd_mod_free+0x46/0x140 [ptlrpc] [13936.126305] mdc_commit_open+0x91/0xb0 [mdc] [13936.126351] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13936.126654] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13936.126880] after_reply+0xe2c/0x1780 [ptlrpc] [13936.129081] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13936.130164] ? __next_timer_interrupt+0x160/0x160 [13936.130174] ptlrpcd+0xa82/0xe40 [ptlrpc] [13936.130749] ? do_wait_intr+0xf0/0xf0 [13936.130841] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13936.137400] ll_lookup_it+0xe0e/0x2270 [lustre] [13936.137578] ? do_raw_spin_unlock+0x75/0x190 [13936.137587] ? _raw_spin_unlock+0x12/0x30 [13936.137595] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13936.137633] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13936.137653] ll_atomic_open+0x36a/0xd70 [lustre] [13936.137778] lookup_open+0x356/0xb40 [13936.137790] do_last+0x313/0xfc0 [13936.137799] ? nd_jump_root+0xe5/0x160 [13936.137805] ? path_init+0x437/0x520 [13936.137810] path_openat+0xf7/0x500 [13936.137816] ? do_raw_spin_unlock+0x75/0x190 [13936.137821] do_filp_open+0x99/0x140 [13936.137827] ? getname_flags+0x6e/0x330 [13936.137831] ? __check_object_size+0xff/0x256 [13936.137838] ? do_raw_spin_unlock+0x75/0x190 [13936.137847] ? _raw_spin_unlock+0x12/0x30 [13936.137852] do_sys_openat2+0x2b4/0x410 [13936.137857] do_sys_open+0x73/0xa0 [13936.137865] __x64_sys_openat+0x24/0x30 [13936.137869] do_syscall_64+0xc1/0x440 [13936.137877] entry_SYSCALL_64_after_hwframe+0x49/0xae [13936.137884] RIP: 0033:0x7f53425a5332 [13936.137897] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13936.137901] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13936.137905] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13936.137908] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13936.137910] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13936.137913] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13936.137915] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13936.848839] kthread+0x1d1/0x200 [13936.850153] ? set_kthread_struct+0x70/0x70 [13936.852818] ret_from_fork+0x1f/0x30 [13940.703168] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11303s! [13940.707112] Showing busy workqueues and worker pools: [13940.709278] workqueue events: flags=0x0 [13940.721333] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13940.725325] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13940.737082] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13940.741274] pending: cpu_latency_work [ptlrpc] [13940.744319] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13940.750961] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13940.757434] workqueue events_long: flags=0x0 [13940.759734] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13940.765180] pending: lu_objects_flush [obdclass] [13940.767968] workqueue events_unbound: flags=0x2 [13940.771596] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13940.774271] pending: flush_memcg_stats_dwork, idle_cull_fn [13940.777237] workqueue events_power_efficient: flags=0x80 [13940.779591] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13940.783265] pending: check_lifetime, do_cache_clean [13940.789701] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13940.793990] pending: neigh_periodic_work, neigh_periodic_work [13940.797181] workqueue mm_percpu_wq: flags=0x8 [13940.805094] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13940.811452] pending: vmstat_update, lru_add_drain_per_cpu [13940.815549] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13940.820869] pending: vmstat_update, lru_add_drain_per_cpu [13940.823853] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13940.830194] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13940.837327] workqueue xprtiod: flags=0xa [13940.841719] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13940.849666] pending: xprt_autoclose, xs_error_handle [13940.856284] workqueue ipv6_addrconf: flags=0x40008 [13940.860539] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13940.866579] pending: addrconf_verify_work [13940.870623] workqueue knbd0-recv: flags=0x1a [13940.875454] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13940.879159] in-flight: 651:recv_work [13940.883130] workqueue ptlrpc_pinger: flags=0x2 [13940.888245] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13940.898408] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13940.900714] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11303s workers=4 idle: 15979 31824 8 [13940.914023] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13931s workers=2 idle: 654 [13952.058586] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13952.062211] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13952.124834] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13952.132519] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13952.145642] RIP: 0010:native_safe_halt+0x2e/0x50 [13952.149533] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13952.169821] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13952.175597] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13952.180919] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13952.184273] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13952.186284] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13952.188780] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13952.191554] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13952.195433] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13952.198552] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13952.201644] Call Trace: [13952.202890] [13952.203741] ? show_regs.cold.9+0x22/0x2f [13952.205422] ? watchdog_timer_fn.cold.6+0x62/0x10a [13952.207211] ? __hrtimer_run_queues+0x1c2/0x4d0 [13952.209839] ? watchdog+0x40/0x40 [13952.211235] ? hrtimer_interrupt+0x128/0x2c0 [13952.213026] ? smp_apic_timer_interrupt+0xa6/0x270 [13952.214897] ? apic_timer_interrupt+0xf/0x20 [13952.216648] [13952.217550] ? native_safe_halt+0x2e/0x50 [13952.221260] kvm_wait+0xcd/0xe0 [13952.222736] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13952.225440] do_raw_spin_lock+0x177/0x190 [13952.229808] _raw_spin_lock+0x21/0x30 [13952.233638] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13952.237768] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13952.244121] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13952.256736] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13952.264131] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13952.271097] ? woken_wake_function+0x30/0x30 [13952.273922] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13952.278159] kthread+0x1d1/0x200 [13952.283058] ? set_kthread_struct+0x70/0x70 [13952.288626] ret_from_fork+0x1f/0x30 [13964.056241] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [13964.062086] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [13964.064814] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13964.064891] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13964.064898] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13964.064900] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13964.064915] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13964.064920] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13964.064924] RAX: 0000000000007f17 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13964.064927] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13964.064929] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13964.064931] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13964.066241] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13964.066247] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13964.066250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13964.066252] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13964.066260] Call Trace: [13964.066265] [13964.066494] ? show_regs.cold.9+0x22/0x2f [13964.066508] ? watchdog_timer_fn.cold.6+0x62/0x10a [13964.066516] ? __hrtimer_run_queues+0x1c2/0x4d0 [13964.066520] ? watchdog+0x40/0x40 [13964.066525] ? hrtimer_interrupt+0x128/0x2c0 [13964.066528] ? smp_apic_timer_interrupt+0xa6/0x270 [13964.066535] ? apic_timer_interrupt+0xf/0x20 [13964.066539] [13964.066543] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13964.066551] do_raw_spin_lock+0x177/0x190 [13964.066555] _raw_spin_lock+0x21/0x30 [13964.066560] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13964.067554] ? do_raw_spin_unlock+0x75/0x190 [13964.067561] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13964.067783] ? do_wait_intr+0xf0/0xf0 [13964.067792] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13964.069258] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13964.069841] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13964.071354] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13964.079806] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13964.079881] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13964.079888] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13964.079891] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13964.079905] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13964.079910] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13964.079915] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13964.079917] RDX: 0000000000007da5 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13964.079919] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13964.079921] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13964.079924] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13964.079927] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13964.079929] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13964.079932] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13964.079951] Call Trace: [13964.079957] [13964.079962] ? show_regs.cold.9+0x22/0x2f [13964.079972] ? watchdog_timer_fn.cold.6+0x62/0x10a [13964.079979] ? __hrtimer_run_queues+0x1c2/0x4d0 [13964.079983] ? watchdog+0x40/0x40 [13964.079987] ? hrtimer_interrupt+0x128/0x2c0 [13964.079991] ? smp_apic_timer_interrupt+0xa6/0x270 [13964.079999] ? apic_timer_interrupt+0xf/0x20 [13964.080004] [13964.080009] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13964.080014] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13964.080018] do_raw_spin_lock+0x177/0x190 [13964.080023] _raw_spin_lock+0x21/0x30 [13964.080030] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13964.080306] obd_mod_free+0x46/0x140 [ptlrpc] [13964.080497] mdc_commit_open+0x91/0xb0 [mdc] [13964.080533] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13964.081252] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13964.081451] after_reply+0xe2c/0x1780 [ptlrpc] [13964.118386] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13964.122706] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13964.127619] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13964.127672] mdc_intent_lock+0x353/0x7e0 [mdc] [13964.127698] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13964.129645] ? __next_timer_interrupt+0x160/0x160 [13964.129668] ptlrpcd+0xa82/0xe40 [ptlrpc] [13964.139919] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13964.143205] ? do_wait_intr+0xf0/0xf0 [13964.150811] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13964.161483] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13964.161812] kthread+0x1d1/0x200 [13964.161822] ? set_kthread_struct+0x70/0x70 [13964.161826] ret_from_fork+0x1f/0x30 [13964.527608] lmv_intent_open+0x338/0x1150 [lmv] [13964.530887] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13964.533704] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13964.540483] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13964.544338] ll_intent_lock+0xc5/0xaa0 [lustre] [13964.550757] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13964.555833] ll_lookup_it+0xe0e/0x2270 [lustre] [13964.563120] ? do_raw_spin_unlock+0x75/0x190 [13964.565380] ? _raw_spin_unlock+0x12/0x30 [13964.567043] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13964.572292] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13964.575418] ll_atomic_open+0x36a/0xd70 [lustre] [13964.580771] lookup_open+0x356/0xb40 [13964.583851] do_last+0x313/0xfc0 [13964.585502] ? nd_jump_root+0xe5/0x160 [13964.587917] ? path_init+0x437/0x520 [13964.592921] path_openat+0xf7/0x500 [13964.594756] ? do_raw_spin_unlock+0x75/0x190 [13964.598408] do_filp_open+0x99/0x140 [13964.600707] ? getname_flags+0x6e/0x330 [13964.602530] ? __check_object_size+0xff/0x256 [13964.604550] ? do_raw_spin_unlock+0x75/0x190 [13964.610123] ? _raw_spin_unlock+0x12/0x30 [13964.613067] do_sys_openat2+0x2b4/0x410 [13964.614644] do_sys_open+0x73/0xa0 [13964.618557] __x64_sys_openat+0x24/0x30 [13964.619695] do_syscall_64+0xc1/0x440 [13964.622618] entry_SYSCALL_64_after_hwframe+0x49/0xae [13964.624561] RIP: 0033:0x7f53425a5332 [13964.625834] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13964.635752] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13964.642653] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13964.646205] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13964.653742] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13964.657464] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13964.660844] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13968.052119] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [13968.057771] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13968.085183] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13968.093067] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13968.100349] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13968.105036] RIP: 0010:native_safe_halt+0x2e/0x50 [13968.108213] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [13968.119954] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [13968.125622] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13968.136634] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [13968.149713] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13968.152499] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13968.155914] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13968.158921] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13968.161938] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13968.164414] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13968.171305] Call Trace: [13968.172628] [13968.173537] ? show_regs.cold.9+0x22/0x2f [13968.178187] ? watchdog_timer_fn.cold.6+0x62/0x10a [13968.180843] ? __hrtimer_run_queues+0x1c2/0x4d0 [13968.184805] ? watchdog+0x40/0x40 [13968.186197] ? hrtimer_interrupt+0x128/0x2c0 [13968.189000] ? smp_apic_timer_interrupt+0xa6/0x270 [13968.190549] ? apic_timer_interrupt+0xf/0x20 [13968.192543] [13968.193576] ? native_safe_halt+0x2e/0x50 [13968.195276] kvm_wait+0xcd/0xe0 [13968.196568] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13968.198726] ? newidle_balance.isra.93+0x207/0x580 [13968.201172] do_raw_spin_lock+0x177/0x190 [13968.202806] _raw_spin_lock+0x21/0x30 [13968.204477] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13968.208410] process_one_work+0x2c8/0x700 [13968.210584] worker_thread+0x296/0x6e0 [13968.212369] ? rescuer_thread+0x570/0x570 [13968.215250] kthread+0x1d1/0x200 [13968.216570] ? set_kthread_struct+0x70/0x70 [13968.218342] ret_from_fork+0x1f/0x30 [13971.423030] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11334s! [13971.428173] Showing busy workqueues and worker pools: [13971.431675] workqueue events: flags=0x0 [13971.433740] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [13971.438205] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [13971.445728] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13971.449977] pending: cpu_latency_work [ptlrpc] [13971.453488] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [13971.456795] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [13971.462714] workqueue events_long: flags=0x0 [13971.466176] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [13971.471781] pending: lu_objects_flush [obdclass] [13971.475353] workqueue events_unbound: flags=0x2 [13971.478715] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [13971.483292] pending: flush_memcg_stats_dwork, idle_cull_fn [13971.487547] workqueue events_power_efficient: flags=0x80 [13971.491717] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13971.496527] pending: check_lifetime, do_cache_clean [13971.499772] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13971.503507] pending: neigh_periodic_work, neigh_periodic_work [13971.507014] workqueue mm_percpu_wq: flags=0x8 [13971.509611] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13971.513398] pending: vmstat_update, lru_add_drain_per_cpu [13971.516933] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [13971.521157] pending: vmstat_update, lru_add_drain_per_cpu [13971.524593] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [13971.528435] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [13971.532189] workqueue xprtiod: flags=0xa [13971.534240] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [13971.538196] pending: xprt_autoclose, xs_error_handle [13971.541451] workqueue ipv6_addrconf: flags=0x40008 [13971.544344] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [13971.547590] pending: addrconf_verify_work [13971.549803] workqueue knbd0-recv: flags=0x1a [13971.552122] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [13971.555567] in-flight: 651:recv_work [13971.557619] workqueue ptlrpc_pinger: flags=0x2 [13971.560161] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [13971.563719] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [13971.567467] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11334s workers=4 idle: 15979 31824 8 [13971.570568] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13962s workers=2 idle: 654 [13980.064721] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [13980.070093] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13980.110037] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13980.126035] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13980.136781] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13980.142807] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13980.160181] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13980.167858] RAX: 0000000000007ea1 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13980.171804] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [13980.175587] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [13980.182186] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [13980.189885] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [13980.196556] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [13980.202946] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13980.205011] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [13980.210773] Call Trace: [13980.211910] [13980.215513] ? show_regs.cold.9+0x22/0x2f [13980.216991] ? watchdog_timer_fn.cold.6+0x62/0x10a [13980.220010] ? __hrtimer_run_queues+0x1c2/0x4d0 [13980.224888] ? watchdog+0x40/0x40 [13980.227081] ? hrtimer_interrupt+0x128/0x2c0 [13980.232720] ? smp_apic_timer_interrupt+0xa6/0x270 [13980.237315] ? apic_timer_interrupt+0xf/0x20 [13980.239355] [13980.242050] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13980.246731] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13980.253252] do_raw_spin_lock+0x177/0x190 [13980.255635] _raw_spin_lock+0x21/0x30 [13980.257036] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [13980.262493] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [13980.268083] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [13980.273729] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [13980.276033] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [13980.282065] ? woken_wake_function+0x30/0x30 [13980.286586] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [13980.290280] kthread+0x1d1/0x200 [13980.294187] ? set_kthread_struct+0x70/0x70 [13980.297585] ret_from_fork+0x1f/0x30 [13992.057690] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [13992.062463] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [13992.066171] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13992.066228] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13992.066234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13992.066236] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [13992.066247] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [13992.066251] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [13992.066254] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [13992.066255] RDX: 00000000000043a3 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [13992.066257] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [13992.066259] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [13992.066260] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [13992.066263] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [13992.066265] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13992.066266] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [13992.066273] Call Trace: [13992.066278] [13992.066282] ? show_regs.cold.9+0x22/0x2f [13992.066289] ? watchdog_timer_fn.cold.6+0x62/0x10a [13992.066294] ? __hrtimer_run_queues+0x1c2/0x4d0 [13992.066296] ? watchdog+0x40/0x40 [13992.066299] ? hrtimer_interrupt+0x128/0x2c0 [13992.066301] ? smp_apic_timer_interrupt+0xa6/0x270 [13992.066307] ? apic_timer_interrupt+0xf/0x20 [13992.066311] [13992.066313] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [13992.066316] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [13992.066319] do_raw_spin_lock+0x177/0x190 [13992.066324] _raw_spin_lock+0x21/0x30 [13992.066329] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [13992.066605] obd_mod_free+0x46/0x140 [ptlrpc] [13992.067204] mdc_commit_open+0x91/0xb0 [mdc] [13992.067243] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [13992.067461] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [13992.067738] after_reply+0xe2c/0x1780 [ptlrpc] [13992.068013] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [13992.068209] ? __next_timer_interrupt+0x160/0x160 [13992.068219] ptlrpcd+0xa82/0xe40 [ptlrpc] [13992.068418] ? do_wait_intr+0xf0/0xf0 [13992.068426] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [13992.068630] kthread+0x1d1/0x200 [13992.068636] ? set_kthread_struct+0x70/0x70 [13992.068639] ret_from_fork+0x1f/0x30 [13992.345787] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13992.387920] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13992.404523] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13992.410863] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13992.413879] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13992.428575] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13992.432084] RAX: 00000000000073d6 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13992.436565] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [13992.440137] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [13992.445177] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [13992.448615] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [13992.452179] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [13992.456063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13992.458379] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [13992.461215] Call Trace: [13992.462293] [13992.463215] ? show_regs.cold.9+0x22/0x2f [13992.465183] ? watchdog_timer_fn.cold.6+0x62/0x10a [13992.467019] ? __hrtimer_run_queues+0x1c2/0x4d0 [13992.468979] ? watchdog+0x40/0x40 [13992.470347] ? hrtimer_interrupt+0x128/0x2c0 [13992.472306] ? smp_apic_timer_interrupt+0xa6/0x270 [13992.474848] ? apic_timer_interrupt+0xf/0x20 [13992.476865] [13992.480997] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13992.488093] do_raw_spin_lock+0x177/0x190 [13992.496265] _raw_spin_lock+0x21/0x30 [13992.502296] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [13992.515155] ? do_raw_spin_unlock+0x75/0x190 [13992.517413] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [13992.519261] ? do_wait_intr+0xf0/0xf0 [13992.522359] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [13992.529271] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [13992.532196] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13992.536098] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13992.541081] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13992.544324] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [13992.547336] mdc_intent_lock+0x353/0x7e0 [mdc] [13992.548927] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13992.559844] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [13992.562278] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [13992.566655] lmv_intent_open+0x338/0x1150 [lmv] [13992.568515] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13992.571567] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13992.574086] lmv_intent_lock+0x1dd/0x4c0 [lmv] [13992.576070] ll_intent_lock+0xc5/0xaa0 [lustre] [13992.578213] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [13992.580559] ll_lookup_it+0xe0e/0x2270 [lustre] [13992.582919] ? do_raw_spin_unlock+0x75/0x190 [13992.588253] ? _raw_spin_unlock+0x12/0x30 [13992.594270] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [13992.599617] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [13992.602404] ll_atomic_open+0x36a/0xd70 [lustre] [13992.609240] lookup_open+0x356/0xb40 [13992.615580] do_last+0x313/0xfc0 [13992.622234] ? nd_jump_root+0xe5/0x160 [13992.624935] ? path_init+0x437/0x520 [13992.631468] path_openat+0xf7/0x500 [13992.638226] ? do_raw_spin_unlock+0x75/0x190 [13992.644474] do_filp_open+0x99/0x140 [13992.648857] ? getname_flags+0x6e/0x330 [13992.651408] ? __check_object_size+0xff/0x256 [13992.658889] ? do_raw_spin_unlock+0x75/0x190 [13992.662373] ? _raw_spin_unlock+0x12/0x30 [13992.663643] do_sys_openat2+0x2b4/0x410 [13992.667736] do_sys_open+0x73/0xa0 [13992.669765] __x64_sys_openat+0x24/0x30 [13992.674748] do_syscall_64+0xc1/0x440 [13992.677139] entry_SYSCALL_64_after_hwframe+0x49/0xae [13992.679370] RIP: 0033:0x7f53425a5332 [13992.681151] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [13992.689228] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [13992.696124] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [13992.702325] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [13992.705426] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [13992.709523] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [13992.713122] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [13996.052389] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [13996.062854] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [13996.134835] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [13996.148668] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [13996.160856] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [13996.163056] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13996.177302] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [13996.194095] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [13996.209968] RAX: 0000000000001ddf RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [13996.221892] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [13996.224332] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [13996.238868] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [13996.252252] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [13996.266957] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [13996.274221] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13996.277969] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [13996.289013] Call Trace: [13996.291900] [13996.296084] ? show_regs.cold.9+0x22/0x2f [13996.306076] ? watchdog_timer_fn.cold.6+0x62/0x10a [13996.313226] ? __hrtimer_run_queues+0x1c2/0x4d0 [13996.323394] ? watchdog+0x40/0x40 [13996.324626] ? hrtimer_interrupt+0x128/0x2c0 [13996.325994] ? smp_apic_timer_interrupt+0xa6/0x270 [13996.340269] ? apic_timer_interrupt+0xf/0x20 [13996.346559] [13996.347316] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [13996.349033] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [13996.368580] ? newidle_balance.isra.93+0x207/0x580 [13996.378200] do_raw_spin_lock+0x177/0x190 [13996.389357] _raw_spin_lock+0x21/0x30 [13996.392568] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [13996.395852] process_one_work+0x2c8/0x700 [13996.400203] worker_thread+0x296/0x6e0 [13996.403536] ? rescuer_thread+0x570/0x570 [13996.408054] kthread+0x1d1/0x200 [13996.409219] ? set_kthread_struct+0x70/0x70 [13996.410246] ret_from_fork+0x1f/0x30 [14001.635088] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11364s! [14001.645085] Showing busy workqueues and worker pools: [14001.648958] workqueue events: flags=0x0 [14001.655404] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14001.657835] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14001.674654] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14001.682429] pending: cpu_latency_work [ptlrpc] [14001.684420] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14001.691305] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14001.701569] workqueue events_long: flags=0x0 [14001.703055] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14001.706919] pending: lu_objects_flush [obdclass] [14001.709334] workqueue events_unbound: flags=0x2 [14001.712994] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14001.722586] pending: flush_memcg_stats_dwork, idle_cull_fn [14001.727925] workqueue events_power_efficient: flags=0x80 [14001.732881] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14001.736983] pending: check_lifetime, do_cache_clean [14001.742370] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14001.744874] pending: neigh_periodic_work, neigh_periodic_work [14001.748609] workqueue mm_percpu_wq: flags=0x8 [14001.750715] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14001.756151] pending: vmstat_update, lru_add_drain_per_cpu [14001.761406] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14001.766500] pending: vmstat_update, lru_add_drain_per_cpu [14001.769981] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14001.776093] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14001.780578] workqueue xprtiod: flags=0xa [14001.783443] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14001.790513] pending: xprt_autoclose, xs_error_handle [14001.798675] workqueue ipv6_addrconf: flags=0x40008 [14001.802095] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14001.804799] pending: addrconf_verify_work [14001.807015] workqueue knbd0-recv: flags=0x1a [14001.809168] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14001.813993] in-flight: 651:recv_work [14001.816035] workqueue ptlrpc_pinger: flags=0x2 [14001.820493] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14001.826519] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14001.831784] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11364s workers=4 idle: 15979 31824 8 [14001.838349] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=13992s workers=2 idle: 654 [14008.058973] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [14008.065901] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14008.129048] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14008.145684] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14008.159647] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14008.166936] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14008.185003] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14008.193261] RAX: 0000000000007927 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14008.200130] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14008.205806] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14008.212922] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14008.216362] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14008.223492] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14008.230679] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14008.235237] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14008.241056] Call Trace: [14008.243835] [14008.246380] ? show_regs.cold.9+0x22/0x2f [14008.250897] ? watchdog_timer_fn.cold.6+0x62/0x10a [14008.256232] ? __hrtimer_run_queues+0x1c2/0x4d0 [14008.258076] ? watchdog+0x40/0x40 [14008.259976] ? hrtimer_interrupt+0x128/0x2c0 [14008.262835] ? smp_apic_timer_interrupt+0xa6/0x270 [14008.270923] ? apic_timer_interrupt+0xf/0x20 [14008.275966] [14008.277264] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14008.279588] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14008.287256] do_raw_spin_lock+0x177/0x190 [14008.291841] _raw_spin_lock+0x21/0x30 [14008.294148] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14008.302642] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14008.306255] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14008.310862] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14008.319006] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14008.321876] ? woken_wake_function+0x30/0x30 [14008.323115] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14008.329236] kthread+0x1d1/0x200 [14008.343389] ? set_kthread_struct+0x70/0x70 [14008.344846] ret_from_fork+0x1f/0x30 [14020.056008] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [14020.058905] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) [14020.059008] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [14020.058943] ptlrpc_gss(O) [14020.081915] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14020.082069] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14020.082076] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14020.082079] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14020.082095] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14020.082099] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14020.082104] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14020.082106] RDX: 0000000000001eb5 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14020.082108] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14020.082110] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14020.082113] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14020.082115] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14020.082119] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14020.082121] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14020.082130] Call Trace: [14020.082138] [14020.082145] ? show_regs.cold.9+0x22/0x2f [14020.082155] ? watchdog_timer_fn.cold.6+0x62/0x10a [14020.082162] ? __hrtimer_run_queues+0x1c2/0x4d0 [14020.082167] ? watchdog+0x40/0x40 [14020.082172] ? hrtimer_interrupt+0x128/0x2c0 [14020.082176] ? smp_apic_timer_interrupt+0xa6/0x270 [14020.082186] ? apic_timer_interrupt+0xf/0x20 [14020.082191] [14020.082194] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14020.082199] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14020.082204] do_raw_spin_lock+0x177/0x190 [14020.082208] _raw_spin_lock+0x21/0x30 [14020.082213] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14020.082668] obd_mod_free+0x46/0x140 [ptlrpc] [14020.083196] mdc_commit_open+0x91/0xb0 [mdc] [14020.083243] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14020.083607] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14020.086495] after_reply+0xe2c/0x1780 [ptlrpc] [14020.087317] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14020.093003] ptlrpc(O) [14020.096471] ? __next_timer_interrupt+0x160/0x160 [14020.096490] ptlrpcd+0xa82/0xe40 [ptlrpc] [14020.098274] ? do_wait_intr+0xf0/0xf0 [14020.098287] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14020.144403] obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14020.144457] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14020.144466] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14020.144470] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14020.144485] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14020.144490] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14020.144495] RAX: 0000000000007a11 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14020.144498] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14020.144500] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14020.144502] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14020.144505] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14020.144508] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14020.144512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14020.144514] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14020.144524] Call Trace: [14020.144529] [14020.144535] ? show_regs.cold.9+0x22/0x2f [14020.144545] ? watchdog_timer_fn.cold.6+0x62/0x10a [14020.144551] ? __hrtimer_run_queues+0x1c2/0x4d0 [14020.144556] ? watchdog+0x40/0x40 [14020.144560] ? hrtimer_interrupt+0x128/0x2c0 [14020.144564] ? smp_apic_timer_interrupt+0xa6/0x270 [14020.144573] ? apic_timer_interrupt+0xf/0x20 [14020.144577] [14020.144580] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14020.144587] do_raw_spin_lock+0x177/0x190 [14020.144592] _raw_spin_lock+0x21/0x30 [14020.144597] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14020.145030] ? do_raw_spin_unlock+0x75/0x190 [14020.145036] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14020.145233] ? do_wait_intr+0xf0/0xf0 [14020.145242] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14020.145446] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14020.145664] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14020.156203] kthread+0x1d1/0x200 [14020.156227] ? set_kthread_struct+0x70/0x70 [14020.156232] ret_from_fork+0x1f/0x30 [14020.483978] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14020.486212] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14020.490555] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14020.493680] mdc_intent_lock+0x353/0x7e0 [mdc] [14020.495178] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14020.497089] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14020.499044] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14020.500609] lmv_intent_open+0x338/0x1150 [lmv] [14020.504795] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14020.509211] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14020.511653] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14020.512939] ll_intent_lock+0xc5/0xaa0 [lustre] [14020.517351] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14020.519770] ll_lookup_it+0xe0e/0x2270 [lustre] [14020.521441] ? do_raw_spin_unlock+0x75/0x190 [14020.527913] ? _raw_spin_unlock+0x12/0x30 [14020.530387] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14020.532836] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14020.535656] ll_atomic_open+0x36a/0xd70 [lustre] [14020.537418] lookup_open+0x356/0xb40 [14020.541185] do_last+0x313/0xfc0 [14020.542467] ? nd_jump_root+0xe5/0x160 [14020.545582] ? path_init+0x437/0x520 [14020.547840] path_openat+0xf7/0x500 [14020.550494] ? do_raw_spin_unlock+0x75/0x190 [14020.553591] do_filp_open+0x99/0x140 [14020.555744] ? getname_flags+0x6e/0x330 [14020.557851] ? __check_object_size+0xff/0x256 [14020.559892] ? do_raw_spin_unlock+0x75/0x190 [14020.561926] ? _raw_spin_unlock+0x12/0x30 [14020.563082] do_sys_openat2+0x2b4/0x410 [14020.567870] do_sys_open+0x73/0xa0 [14020.572385] __x64_sys_openat+0x24/0x30 [14020.573621] do_syscall_64+0xc1/0x440 [14020.579191] entry_SYSCALL_64_after_hwframe+0x49/0xae [14020.581239] RIP: 0033:0x7f53425a5332 [14020.587447] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14020.599918] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14020.605426] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14020.607848] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14020.612495] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14020.619089] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14020.621476] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14028.052140] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [14028.056129] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14028.085943] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14028.095855] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14028.103432] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14028.109601] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14028.117720] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14028.127179] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14028.130393] RAX: 0000000000001204 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14028.132877] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14028.138729] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14028.142662] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14028.147340] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14028.149563] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14028.153010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14028.155573] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14028.160366] Call Trace: [14028.161198] [14028.164390] ? show_regs.cold.9+0x22/0x2f [14028.166095] ? watchdog_timer_fn.cold.6+0x62/0x10a [14028.167924] ? __hrtimer_run_queues+0x1c2/0x4d0 [14028.170410] ? watchdog+0x40/0x40 [14028.172288] ? hrtimer_interrupt+0x128/0x2c0 [14028.175030] ? smp_apic_timer_interrupt+0xa6/0x270 [14028.177037] ? apic_timer_interrupt+0xf/0x20 [14028.178605] [14028.179664] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14028.181985] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14028.184173] ? newidle_balance.isra.93+0x207/0x580 [14028.186107] do_raw_spin_lock+0x177/0x190 [14028.187613] _raw_spin_lock+0x21/0x30 [14028.189155] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14028.191110] process_one_work+0x2c8/0x700 [14028.192680] worker_thread+0x296/0x6e0 [14028.193958] ? rescuer_thread+0x570/0x570 [14028.195994] kthread+0x1d1/0x200 [14028.197199] ? set_kthread_struct+0x70/0x70 [14028.198555] ret_from_fork+0x1f/0x30 [14031.840053] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11394s! [14031.843326] Showing busy workqueues and worker pools: [14031.846901] workqueue events: flags=0x0 [14031.848839] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14031.851736] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14031.857829] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14031.860949] pending: cpu_latency_work [ptlrpc] [14031.867465] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14031.870499] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14031.876501] workqueue events_long: flags=0x0 [14031.878397] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14031.882901] pending: lu_objects_flush [obdclass] [14031.886426] workqueue events_unbound: flags=0x2 [14031.892732] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14031.896015] pending: flush_memcg_stats_dwork, idle_cull_fn [14031.899402] workqueue events_power_efficient: flags=0x80 [14031.906478] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14031.915357] pending: check_lifetime, do_cache_clean [14031.920871] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14031.925624] pending: neigh_periodic_work, neigh_periodic_work [14031.930813] workqueue mm_percpu_wq: flags=0x8 [14031.936889] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14031.941587] pending: vmstat_update, lru_add_drain_per_cpu [14031.944955] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14031.951374] pending: vmstat_update, lru_add_drain_per_cpu [14031.954737] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14031.961396] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14031.972612] workqueue xprtiod: flags=0xa [14031.978625] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14031.984831] pending: xprt_autoclose, xs_error_handle [14031.990625] workqueue ipv6_addrconf: flags=0x40008 [14031.993836] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14031.997471] pending: addrconf_verify_work [14032.000661] workqueue knbd0-recv: flags=0x1a [14032.003218] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14032.009868] in-flight: 651:recv_work [14032.013642] workqueue ptlrpc_pinger: flags=0x2 [14032.017282] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14032.020450] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14032.025462] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11394s workers=4 idle: 15979 31824 8 [14032.033249] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14022s workers=2 idle: 654 [14034.720388] rcu: INFO: rcu_sched self-detected stall on CPU [14034.723622] rcu: 2-....: (6645240 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [14034.736676] (t=11940294 jiffies g=274185 q=66906) [14034.738918] rcu: rcu_sched kthread starved for 11398660 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [14034.748717] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14034.755943] rcu: RCU grace-period kthread stack dump: [14034.758857] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14034.770316] Call Trace: [14034.772126] __schedule+0x351/0xcb0 [14034.773391] schedule+0xc0/0x180 [14034.774512] schedule_timeout+0xb4/0x190 [14034.779089] ? do_raw_spin_unlock+0x75/0x190 [14034.782994] ? __next_timer_interrupt+0x160/0x160 [14034.790202] ? prepare_to_swait_event+0xf3/0x140 [14034.796903] rcu_gp_kthread+0x8dd/0x15b0 [14034.799448] ? force_qs_rnp+0x2c0/0x2c0 [14034.801041] kthread+0x1d1/0x200 [14034.802366] ? set_kthread_struct+0x70/0x70 [14034.807339] ret_from_fork+0x1f/0x30 [14034.809103] rcu: Stack dump where RCU GP kthread last ran: [14034.812967] Sending NMI from CPU 2 to CPUs 0: [14034.819917] NMI backtrace for cpu 0 [14034.819924] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14034.819926] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14034.819928] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14034.819930] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14034.819932] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14034.819935] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000216 [14034.819937] RAX: 00000000000067a4 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14034.819938] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14034.819940] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14034.819941] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14034.819942] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14034.819943] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14034.819945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14034.819946] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14034.819947] Call Trace: [14034.819948] [14034.819949] ? show_regs.cold.9+0x22/0x2f [14034.819950] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [14034.819951] ? nmi_cpu_backtrace_handler+0x15/0x40 [14034.819952] ? nmi_handle+0xbd/0x210 [14034.819953] ? default_do_nmi+0x4b/0x1b0 [14034.819954] ? do_nmi+0x280/0x380 [14034.819955] ? end_repeat_nmi+0x16/0x1f [14034.819957] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14034.819958] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14034.819959] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14034.819961] [14034.819962] ? newidle_balance.isra.93+0x207/0x580 [14034.819963] do_raw_spin_lock+0x177/0x190 [14034.819964] _raw_spin_lock+0x21/0x30 [14034.819965] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14034.819966] process_one_work+0x2c8/0x700 [14034.819967] worker_thread+0x296/0x6e0 [14034.819968] ? rescuer_thread+0x570/0x570 [14034.819969] kthread+0x1d1/0x200 [14034.819970] ? set_kthread_struct+0x70/0x70 [14034.819971] ret_from_fork+0x1f/0x30 [14034.820473] Sending NMI from CPU 2 to CPUs 1: [14034.972279] NMI backtrace for cpu 1 [14034.972286] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14034.972288] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14034.972290] RIP: 0010:__pv_queued_spin_lock_slowpath+0x320/0x6d0 [14034.972292] Code: 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 <48> 83 05 18 d4 dc 02 01 eb e0 48 83 05 06 d4 dc 02 01 48 83 05 be [14034.972294] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [14034.972296] RAX: 0000000000003f78 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14034.972298] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14034.972299] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14034.972301] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14034.972303] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14034.972304] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14034.972306] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14034.972307] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14034.972308] Call Trace: [14034.972309] [14034.972310] ? show_regs.cold.9+0x22/0x2f [14034.972311] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [14034.972312] ? nmi_cpu_backtrace_handler+0x15/0x40 [14034.972313] ? nmi_handle+0xbd/0x210 [14034.972314] ? default_do_nmi+0x4b/0x1b0 [14034.972316] ? do_nmi+0x280/0x380 [14034.972317] ? end_repeat_nmi+0x16/0x1f [14034.972318] ? __pv_queued_spin_lock_slowpath+0x320/0x6d0 [14034.972319] ? __pv_queued_spin_lock_slowpath+0x320/0x6d0 [14034.972320] ? __pv_queued_spin_lock_slowpath+0x320/0x6d0 [14034.972322] [14034.972323] do_raw_spin_lock+0x177/0x190 [14034.972324] _raw_spin_lock+0x21/0x30 [14034.972325] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14034.972326] ? do_raw_spin_unlock+0x75/0x190 [14034.972327] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14034.972329] ? do_wait_intr+0xf0/0xf0 [14034.972330] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14034.972331] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14034.972332] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14034.972333] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14034.972334] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14034.972335] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14034.972336] mdc_intent_lock+0x353/0x7e0 [mdc] [14034.972337] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14034.972339] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14034.972340] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14034.972340] lmv_intent_open+0x338/0x1150 [lmv] [14034.972342] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14034.972343] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14034.972344] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14034.972345] ll_intent_lock+0xc5/0xaa0 [lustre] [14034.972346] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14034.972347] ll_lookup_it+0xe0e/0x2270 [lustre] [14034.972348] ? do_raw_spin_unlock+0x75/0x190 [14034.972349] ? _raw_spin_unlock+0x12/0x30 [14034.972351] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14034.972352] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14034.972353] ll_atomic_open+0x36a/0xd70 [lustre] [14034.972354] lookup_open+0x356/0xb40 [14034.972355] do_last+0x313/0xfc0 [14034.972356] ? nd_jump_root+0xe5/0x160 [14034.972358] ? path_init+0x437/0x520 [14034.972359] path_openat+0xf7/0x500 [14034.972360] ? do_raw_spin_unlock+0x75/0x190 [14034.972361] do_filp_open+0x99/0x140 [14034.972362] ? getname_flags+0x6e/0x330 [14034.972363] ? __check_object_size+0xff/0x256 [14034.972364] ? do_raw_spin_unlock+0x75/0x190 [14034.972366] ? _raw_spin_unlock+0x12/0x30 [14034.972367] do_sys_openat2+0x2b4/0x410 [14034.972368] do_sys_open+0x73/0xa0 [14034.972369] __x64_sys_openat+0x24/0x30 [14034.972370] do_syscall_64+0xc1/0x440 [14034.972371] entry_SYSCALL_64_after_hwframe+0x49/0xae [14034.972372] RIP: 0033:0x7f53425a5332 [14034.972373] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14034.972375] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14034.972378] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14034.972379] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14034.972381] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14034.972382] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14034.972383] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14034.973108] NMI backtrace for cpu 2 [14035.268423] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14035.274156] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14035.279388] Call Trace: [14035.280313] [14035.281195] ? dump_stack+0xbb/0x10e [14035.282424] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14035.285084] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14035.288136] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14035.293680] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14035.296081] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14035.299481] ? rcu_dump_cpu_stacks+0x11b/0x171 [14035.302048] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14035.304406] ? update_process_times+0x6d/0xc0 [14035.306640] ? tick_sched_handle.isra.18+0x27/0x90 [14035.308547] ? tick_sched_timer+0x53/0xb0 [14035.310274] ? __hrtimer_run_queues+0x1c2/0x4d0 [14035.313184] ? tick_sched_handle.isra.18+0x90/0x90 [14035.315979] ? hrtimer_interrupt+0x128/0x2c0 [14035.317604] ? smp_apic_timer_interrupt+0xa6/0x270 [14035.321513] ? apic_timer_interrupt+0xf/0x20 [14035.323126] [14035.324328] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14035.326850] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14035.328982] ? do_raw_spin_lock+0x177/0x190 [14035.332844] ? _raw_spin_lock+0x21/0x30 [14035.334632] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14035.339912] ? obd_mod_free+0x46/0x140 [ptlrpc] [14035.343514] ? mdc_commit_open+0x91/0xb0 [mdc] [14035.346129] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14035.349141] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14035.352412] ? after_reply+0xe2c/0x1780 [ptlrpc] [14035.355619] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14035.360357] ? __next_timer_interrupt+0x160/0x160 [14035.362678] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14035.365433] ? do_wait_intr+0xf0/0xf0 [14035.367584] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14035.371905] ? kthread+0x1d1/0x200 [14035.373677] ? set_kthread_struct+0x70/0x70 [14035.377415] ? ret_from_fork+0x1f/0x30 [14036.058093] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [14036.060814] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14036.092603] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14036.102422] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14036.109832] RIP: 0010:native_safe_halt+0x2e/0x50 [14036.111484] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14036.119784] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14036.122225] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14036.124613] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14036.127424] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14036.129406] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14036.131453] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14036.134018] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14036.137518] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14036.139490] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14036.142352] Call Trace: [14036.143444] [14036.144361] ? show_regs.cold.9+0x22/0x2f [14036.146253] ? watchdog_timer_fn.cold.6+0x62/0x10a [14036.148442] ? __hrtimer_run_queues+0x1c2/0x4d0 [14036.149568] ? watchdog+0x40/0x40 [14036.150598] ? hrtimer_interrupt+0x128/0x2c0 [14036.152015] ? smp_apic_timer_interrupt+0xa6/0x270 [14036.153336] ? apic_timer_interrupt+0xf/0x20 [14036.154568] [14036.155469] ? native_safe_halt+0x2e/0x50 [14036.156988] kvm_wait+0xcd/0xe0 [14036.158207] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14036.160200] do_raw_spin_lock+0x177/0x190 [14036.161878] _raw_spin_lock+0x21/0x30 [14036.163141] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14036.168719] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14036.170418] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14036.174245] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14036.177630] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14036.182338] ? woken_wake_function+0x30/0x30 [14036.184270] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14036.188166] kthread+0x1d1/0x200 [14036.189035] ? set_kthread_struct+0x70/0x70 [14036.190865] ret_from_fork+0x1f/0x30 [14060.053749] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [14060.055403] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [14060.057464] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [14060.057476] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14060.057541] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14060.057546] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14060.057548] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14060.057563] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14060.057568] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14060.057571] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14060.057574] RDX: 00000000000028a6 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14060.057576] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14060.057578] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14060.057580] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14060.057583] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14060.057586] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14060.057588] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14060.057597] Call Trace: [14060.057602] [14060.057608] ? show_regs.cold.9+0x22/0x2f [14060.057617] ? watchdog_timer_fn.cold.6+0x62/0x10a [14060.057637] ? __hrtimer_run_queues+0x1c2/0x4d0 [14060.057641] ? watchdog+0x40/0x40 [14060.057646] ? hrtimer_interrupt+0x128/0x2c0 [14060.057650] ? smp_apic_timer_interrupt+0xa6/0x270 [14060.057659] ? apic_timer_interrupt+0xf/0x20 [14060.057663] [14060.057666] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14060.057671] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14060.057676] do_raw_spin_lock+0x177/0x190 [14060.057681] _raw_spin_lock+0x21/0x30 [14060.057687] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14060.058023] obd_mod_free+0x46/0x140 [ptlrpc] [14060.058431] mdc_commit_open+0x91/0xb0 [mdc] [14060.058469] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14060.058667] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14060.059608] after_reply+0xe2c/0x1780 [ptlrpc] [14060.059817] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14060.059963] ? __next_timer_interrupt+0x160/0x160 [14060.059970] ptlrpcd+0xa82/0xe40 [ptlrpc] [14060.060589] ? do_wait_intr+0xf0/0xf0 [14060.060599] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14060.061442] kthread+0x1d1/0x200 [14060.061449] ? set_kthread_struct+0x70/0x70 [14060.061454] ret_from_fork+0x1f/0x30 [14060.061706] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14060.061778] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14060.061784] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14060.061792] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14060.062241] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14060.062307] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14060.062312] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14060.062316] RAX: 0000000000002e90 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14060.062319] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14060.062321] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14060.062323] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14060.062326] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14060.062329] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14060.062332] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14060.062334] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14060.062343] Call Trace: [14060.062347] [14060.062350] ? show_regs.cold.9+0x22/0x2f [14060.062359] ? watchdog_timer_fn.cold.6+0x62/0x10a [14060.062365] ? __hrtimer_run_queues+0x1c2/0x4d0 [14060.062369] ? watchdog+0x40/0x40 [14060.062373] ? hrtimer_interrupt+0x128/0x2c0 [14060.062377] ? smp_apic_timer_interrupt+0xa6/0x270 [14060.062384] ? apic_timer_interrupt+0xf/0x20 [14060.062389] [14060.062391] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14060.062396] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14060.062400] ? newidle_balance.isra.93+0x207/0x580 [14060.062405] do_raw_spin_lock+0x177/0x190 [14060.062409] _raw_spin_lock+0x21/0x30 [14060.062414] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14060.062619] process_one_work+0x2c8/0x700 [14060.062626] worker_thread+0x296/0x6e0 [14060.062630] ? rescuer_thread+0x570/0x570 [14060.062633] kthread+0x1d1/0x200 [14060.062637] ? set_kthread_struct+0x70/0x70 [14060.062640] ret_from_fork+0x1f/0x30 [14060.424089] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14060.451180] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14060.457000] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14060.460930] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14060.463456] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14060.471224] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14060.474477] RAX: 0000000000001d62 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14060.477316] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14060.480111] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14060.483005] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14060.485348] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14060.487146] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14060.489735] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14060.491889] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14060.495185] Call Trace: [14060.496140] [14060.497324] ? show_regs.cold.9+0x22/0x2f [14060.499240] ? watchdog_timer_fn.cold.6+0x62/0x10a [14060.501524] ? __hrtimer_run_queues+0x1c2/0x4d0 [14060.503439] ? watchdog+0x40/0x40 [14060.504685] ? hrtimer_interrupt+0x128/0x2c0 [14060.506228] ? smp_apic_timer_interrupt+0xa6/0x270 [14060.508122] ? apic_timer_interrupt+0xf/0x20 [14060.509857] [14060.510604] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14060.512696] do_raw_spin_lock+0x177/0x190 [14060.514165] _raw_spin_lock+0x21/0x30 [14060.515631] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14060.517800] ? do_raw_spin_unlock+0x75/0x190 [14060.519445] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14060.521493] ? do_wait_intr+0xf0/0xf0 [14060.522914] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14060.525040] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14060.526981] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14060.529495] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14060.531675] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14060.533783] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14060.535619] mdc_intent_lock+0x353/0x7e0 [mdc] [14060.537130] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14060.539209] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14060.541722] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14060.544928] lmv_intent_open+0x338/0x1150 [lmv] [14060.547566] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14060.549840] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14060.552994] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14060.554855] ll_intent_lock+0xc5/0xaa0 [lustre] [14060.556662] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14060.558857] ll_lookup_it+0xe0e/0x2270 [lustre] [14060.560774] ? do_raw_spin_unlock+0x75/0x190 [14060.562327] ? _raw_spin_unlock+0x12/0x30 [14060.563915] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14060.565829] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14060.567939] ll_atomic_open+0x36a/0xd70 [lustre] [14060.569904] lookup_open+0x356/0xb40 [14060.571608] do_last+0x313/0xfc0 [14060.572940] ? nd_jump_root+0xe5/0x160 [14060.574626] ? path_init+0x437/0x520 [14060.576148] path_openat+0xf7/0x500 [14060.577526] ? do_raw_spin_unlock+0x75/0x190 [14060.579705] do_filp_open+0x99/0x140 [14060.581320] ? getname_flags+0x6e/0x330 [14060.582942] ? __check_object_size+0xff/0x256 [14060.584762] ? do_raw_spin_unlock+0x75/0x190 [14060.586619] ? _raw_spin_unlock+0x12/0x30 [14060.588404] do_sys_openat2+0x2b4/0x410 [14060.589978] do_sys_open+0x73/0xa0 [14060.591452] __x64_sys_openat+0x24/0x30 [14060.593070] do_syscall_64+0xc1/0x440 [14060.595392] entry_SYSCALL_64_after_hwframe+0x49/0xae [14060.597380] RIP: 0033:0x7f53425a5332 [14060.598858] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14060.606460] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14060.609593] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14060.612485] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14060.615417] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14060.618429] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14060.621175] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14062.049236] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11424s! [14062.052773] Showing busy workqueues and worker pools: [14062.055852] workqueue events: flags=0x0 [14062.057807] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14062.061098] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14062.067426] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14062.073099] pending: cpu_latency_work [ptlrpc] [14062.079128] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14062.083186] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14062.088073] workqueue events_long: flags=0x0 [14062.093077] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14062.095813] pending: lu_objects_flush [obdclass] [14062.101187] workqueue events_unbound: flags=0x2 [14062.104656] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14062.106374] pending: flush_memcg_stats_dwork, idle_cull_fn [14062.109084] workqueue events_power_efficient: flags=0x80 [14062.110649] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14062.112187] pending: check_lifetime, do_cache_clean [14062.116132] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14062.119322] pending: neigh_periodic_work, neigh_periodic_work [14062.122674] workqueue mm_percpu_wq: flags=0x8 [14062.126368] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14062.132713] pending: vmstat_update, lru_add_drain_per_cpu [14062.136165] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14062.139234] pending: vmstat_update, lru_add_drain_per_cpu [14062.141441] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14062.144117] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14062.149898] workqueue xprtiod: flags=0xa [14062.151091] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14062.153854] pending: xprt_autoclose, xs_error_handle [14062.158889] workqueue ipv6_addrconf: flags=0x40008 [14062.163939] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14062.166620] pending: addrconf_verify_work [14062.168776] workqueue knbd0-recv: flags=0x1a [14062.178976] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14062.180644] in-flight: 651:recv_work [14062.181608] workqueue ptlrpc_pinger: flags=0x2 [14062.187807] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14062.192473] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14062.195844] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11425s workers=4 idle: 15979 31824 8 [14062.203906] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14052s workers=2 idle: 654 [14064.066244] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [14064.069837] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14064.116022] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14064.122366] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14064.132670] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14064.139825] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14064.158997] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14064.167012] RAX: 0000000000007811 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14064.180582] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14064.187624] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14064.190050] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14064.203360] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14064.209060] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14064.215347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14064.218962] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14064.227884] Call Trace: [14064.228600] [14064.229199] ? show_regs.cold.9+0x22/0x2f [14064.232259] ? watchdog_timer_fn.cold.6+0x62/0x10a [14064.235599] ? __hrtimer_run_queues+0x1c2/0x4d0 [14064.238073] ? watchdog+0x40/0x40 [14064.239884] ? hrtimer_interrupt+0x128/0x2c0 [14064.245929] ? smp_apic_timer_interrupt+0xa6/0x270 [14064.249116] ? apic_timer_interrupt+0xf/0x20 [14064.256256] [14064.257288] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14064.259408] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14064.263285] do_raw_spin_lock+0x177/0x190 [14064.265544] _raw_spin_lock+0x21/0x30 [14064.269006] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14064.272224] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14064.278380] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14064.280514] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14064.285955] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14064.289526] ? woken_wake_function+0x30/0x30 [14064.293692] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14064.298519] kthread+0x1d1/0x200 [14064.300627] ? set_kthread_struct+0x70/0x70 [14064.303028] ret_from_fork+0x1f/0x30 [14088.055846] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [14088.060167] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14088.064719] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [14088.064800] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [14088.064806] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14088.064879] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14088.064885] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14088.064888] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14088.064902] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14088.064907] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14088.064912] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14088.064914] RDX: 0000000000002103 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14088.064917] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14088.064919] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14088.064922] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14088.064925] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14088.064928] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14088.064930] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14088.064938] Call Trace: [14088.064944] [14088.064950] ? show_regs.cold.9+0x22/0x2f [14088.064960] ? watchdog_timer_fn.cold.6+0x62/0x10a [14088.064967] ? __hrtimer_run_queues+0x1c2/0x4d0 [14088.064972] ? watchdog+0x40/0x40 [14088.064976] ? hrtimer_interrupt+0x128/0x2c0 [14088.064981] ? smp_apic_timer_interrupt+0xa6/0x270 [14088.064990] ? apic_timer_interrupt+0xf/0x20 [14088.064995] [14088.064998] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14088.065004] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14088.065009] do_raw_spin_lock+0x177/0x190 [14088.065014] _raw_spin_lock+0x21/0x30 [14088.065020] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14088.065282] obd_mod_free+0x46/0x140 [ptlrpc] [14088.065477] mdc_commit_open+0x91/0xb0 [mdc] [14088.065513] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14088.065709] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14088.065920] after_reply+0xe2c/0x1780 [ptlrpc] [14088.066117] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14088.066313] ? __next_timer_interrupt+0x160/0x160 [14088.066324] ptlrpcd+0xa82/0xe40 [ptlrpc] [14088.066521] ? do_wait_intr+0xf0/0xf0 [14088.066531] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14088.066729] kthread+0x1d1/0x200 [14088.066736] ? set_kthread_struct+0x70/0x70 [14088.066739] ret_from_fork+0x1f/0x30 [14088.080978] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14088.080993] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14088.080996] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14088.081011] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14088.081016] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14088.081021] RAX: 00000000000041a7 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14088.081023] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14088.081026] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14088.081028] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14088.081030] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14088.081033] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14088.081037] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14088.081039] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14088.081046] Call Trace: [14088.081052] [14088.081056] ? show_regs.cold.9+0x22/0x2f [14088.081066] ? watchdog_timer_fn.cold.6+0x62/0x10a [14088.081073] ? __hrtimer_run_queues+0x1c2/0x4d0 [14088.081076] ? watchdog+0x40/0x40 [14088.081080] ? hrtimer_interrupt+0x128/0x2c0 [14088.081084] ? smp_apic_timer_interrupt+0xa6/0x270 [14088.081091] ? apic_timer_interrupt+0xf/0x20 [14088.081096] [14088.081099] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14088.081107] do_raw_spin_lock+0x177/0x190 [14088.081111] _raw_spin_lock+0x21/0x30 [14088.081116] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14088.081656] ? do_raw_spin_unlock+0x75/0x190 [14088.081664] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14088.081930] ? do_wait_intr+0xf0/0xf0 [14088.081939] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14088.082140] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14088.082517] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14088.082745] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14088.082976] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14088.083015] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14088.083047] mdc_intent_lock+0x353/0x7e0 [mdc] [14088.083072] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14088.083155] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14088.083435] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14088.083468] lmv_intent_open+0x338/0x1150 [lmv] [14088.083551] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14088.083637] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14088.083717] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14088.083738] ll_intent_lock+0xc5/0xaa0 [lustre] [14088.083908] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14088.083989] ll_lookup_it+0xe0e/0x2270 [lustre] [14088.084069] ? do_raw_spin_unlock+0x75/0x190 [14088.084075] ? _raw_spin_unlock+0x12/0x30 [14088.084082] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14088.084108] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14088.084128] ll_atomic_open+0x36a/0xd70 [lustre] [14088.084219] lookup_open+0x356/0xb40 [14088.084227] do_last+0x313/0xfc0 [14088.084231] ? nd_jump_root+0xe5/0x160 [14088.084235] ? path_init+0x437/0x520 [14088.084239] path_openat+0xf7/0x500 [14088.084244] ? do_raw_spin_unlock+0x75/0x190 [14088.084248] do_filp_open+0x99/0x140 [14088.084253] ? getname_flags+0x6e/0x330 [14088.084257] ? __check_object_size+0xff/0x256 [14088.084263] ? do_raw_spin_unlock+0x75/0x190 [14088.084267] ? _raw_spin_unlock+0x12/0x30 [14088.084270] do_sys_openat2+0x2b4/0x410 [14088.084274] do_sys_open+0x73/0xa0 [14088.084278] __x64_sys_openat+0x24/0x30 [14088.084281] do_syscall_64+0xc1/0x440 [14088.084287] entry_SYSCALL_64_after_hwframe+0x49/0xae [14088.084293] RIP: 0033:0x7f53425a5332 [14088.084301] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14088.084305] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14088.084435] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14088.084437] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14088.084439] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14088.084441] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14088.084443] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14089.075662] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14089.139532] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14089.164452] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14089.171710] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14089.185792] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14089.193216] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14089.235171] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14089.250206] RAX: 0000000000007e82 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14089.262875] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14089.272693] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14089.285796] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14089.307712] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14089.317558] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14089.326331] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14089.336295] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14089.366406] Call Trace: [14089.375647] [14089.383673] ? show_regs.cold.9+0x22/0x2f [14089.393509] ? watchdog_timer_fn.cold.6+0x62/0x10a [14089.395289] ? __hrtimer_run_queues+0x1c2/0x4d0 [14089.422548] ? watchdog+0x40/0x40 [14089.434662] ? hrtimer_interrupt+0x128/0x2c0 [14089.441687] ? smp_apic_timer_interrupt+0xa6/0x270 [14089.446037] ? apic_timer_interrupt+0xf/0x20 [14089.447562] [14089.448254] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14089.471613] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14089.489765] ? newidle_balance.isra.93+0x207/0x580 [14089.491437] do_raw_spin_lock+0x177/0x190 [14089.504815] _raw_spin_lock+0x21/0x30 [14089.509404] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14089.519426] process_one_work+0x2c8/0x700 [14089.529238] worker_thread+0x296/0x6e0 [14089.533184] ? rescuer_thread+0x570/0x570 [14089.534165] kthread+0x1d1/0x200 [14089.537638] ? set_kthread_struct+0x70/0x70 [14089.548293] ret_from_fork+0x1f/0x30 [14092.058439] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [14092.066041] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14092.111628] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14092.119277] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14092.135254] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14092.144096] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14092.168225] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14092.185256] RAX: 0000000000007791 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14092.193384] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14092.202308] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14092.210051] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14092.223280] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14092.226126] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14092.232509] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14092.239197] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14092.242497] Call Trace: [14092.245193] [14092.247014] ? show_regs.cold.9+0x22/0x2f [14092.248316] ? watchdog_timer_fn.cold.6+0x62/0x10a [14092.250222] ? __hrtimer_run_queues+0x1c2/0x4d0 [14092.251783] ? watchdog+0x40/0x40 [14092.252764] ? hrtimer_interrupt+0x128/0x2c0 [14092.254534] ? smp_apic_timer_interrupt+0xa6/0x270 [14092.255965] ? apic_timer_interrupt+0xf/0x20 [14092.257355] [14092.258234] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14092.259978] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14092.261565] do_raw_spin_lock+0x177/0x190 [14092.263083] _raw_spin_lock+0x21/0x30 [14092.264255] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14092.266958] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14092.267808] BUG: workqueue lockup - pool [14092.268961] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14092.269282] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14092.269492] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14092.269819] ? woken_wake_function+0x30/0x30 [14092.269828] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14092.270104] kthread+0x1d1/0x200 [14092.270110] ? set_kthread_struct+0x70/0x70 [14092.270114] ret_from_fork+0x1f/0x30 [14092.342316] cpus=0-3 flags=0x4 nice=0 stuck for 11455s! [14092.356377] Showing busy workqueues and worker pools: [14092.364029] workqueue events: flags=0x0 [14092.375672] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14092.394323] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14092.412606] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14092.419805] pending: cpu_latency_work [ptlrpc] [14092.443747] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14092.454919] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14092.474105] workqueue events_long: flags=0x0 [14092.478176] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14092.490066] pending: lu_objects_flush [obdclass] [14092.496790] workqueue events_unbound: flags=0x2 [14092.516113] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14092.529053] pending: flush_memcg_stats_dwork, idle_cull_fn [14092.538872] workqueue events_power_efficient: flags=0x80 [14092.544237] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14092.550722] pending: check_lifetime, do_cache_clean [14092.553440] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14092.564547] pending: neigh_periodic_work, neigh_periodic_work [14092.568658] workqueue mm_percpu_wq: flags=0x8 [14092.576438] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14092.582554] pending: vmstat_update, lru_add_drain_per_cpu [14092.592911] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14092.600889] pending: vmstat_update, lru_add_drain_per_cpu [14092.604636] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14092.610567] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14092.618310] workqueue xprtiod: flags=0xa [14092.624502] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14092.633483] pending: xprt_autoclose, xs_error_handle [14092.644350] workqueue ipv6_addrconf: flags=0x40008 [14092.648985] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14092.655504] pending: addrconf_verify_work [14092.659935] workqueue knbd0-recv: flags=0x1a [14092.665743] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14092.675334] in-flight: 651:recv_work [14092.678626] workqueue ptlrpc_pinger: flags=0x2 [14092.694538] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14092.696492] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14092.716292] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11455s workers=4 idle: 15979 31824 8 [14092.719769] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14083s workers=2 idle: 654 [14116.055672] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [14116.060764] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14116.061328] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [14116.092106] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14116.092123] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14116.092126] RIP: 0010:native_safe_halt+0x2e/0x50 [14116.092150] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14116.092154] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14116.092160] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14116.092166] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14116.092172] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14116.092174] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14116.092177] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14116.092181] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14116.092184] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14116.092187] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14116.092195] Call Trace: [14116.092207] [14116.092214] ? show_regs.cold.9+0x22/0x2f [14116.092231] ? watchdog_timer_fn.cold.6+0x62/0x10a [14116.092238] ? __hrtimer_run_queues+0x1c2/0x4d0 [14116.092248] ? watchdog+0x40/0x40 [14116.092251] ? hrtimer_interrupt+0x128/0x2c0 [14116.092255] ? smp_apic_timer_interrupt+0xa6/0x270 [14116.092261] ? apic_timer_interrupt+0xf/0x20 [14116.092265] [14116.092268] ? native_safe_halt+0x2e/0x50 [14116.095879] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14116.095962] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14116.095969] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14116.095973] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14116.095987] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14116.095992] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14116.095996] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14116.095998] RDX: 0000000000007ce2 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14116.096001] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14116.096003] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14116.096005] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14116.096009] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14116.096012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14116.096014] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14116.096022] Call Trace: [14116.096028] [14116.096033] ? show_regs.cold.9+0x22/0x2f [14116.096043] ? watchdog_timer_fn.cold.6+0x62/0x10a [14116.096049] ? __hrtimer_run_queues+0x1c2/0x4d0 [14116.096054] ? watchdog+0x40/0x40 [14116.096059] ? hrtimer_interrupt+0x128/0x2c0 [14116.096063] ? smp_apic_timer_interrupt+0xa6/0x270 [14116.101224] kvm_wait+0xcd/0xe0 [14116.101246] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14116.101257] do_raw_spin_lock+0x177/0x190 [14116.101262] _raw_spin_lock+0x21/0x30 [14116.101329] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14116.101680] ? do_raw_spin_unlock+0x75/0x190 [14116.101686] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14116.102119] ? do_wait_intr+0xf0/0xf0 [14116.102129] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14116.102347] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14116.102723] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14116.102939] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14116.103066] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14116.103172] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14116.103203] mdc_intent_lock+0x353/0x7e0 [mdc] [14116.103228] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14116.103330] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14116.103692] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14116.103721] lmv_intent_open+0x338/0x1150 [lmv] [14116.103751] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14116.103837] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14116.108202] ? apic_timer_interrupt+0xf/0x20 [14116.108221] [14116.108226] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14116.108233] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14116.108238] do_raw_spin_lock+0x177/0x190 [14116.108244] _raw_spin_lock+0x21/0x30 [14116.108251] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14116.108792] obd_mod_free+0x46/0x140 [ptlrpc] [14116.108992] mdc_commit_open+0x91/0xb0 [mdc] [14116.109048] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14116.109250] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14116.109591] after_reply+0xe2c/0x1780 [ptlrpc] [14116.109809] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14116.110004] ? __next_timer_interrupt+0x160/0x160 [14116.110016] ptlrpcd+0xa82/0xe40 [ptlrpc] [14116.110226] ? do_wait_intr+0xf0/0xf0 [14116.110241] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14116.110607] kthread+0x1d1/0x200 [14116.110613] ? set_kthread_struct+0x70/0x70 [14116.110617] ret_from_fork+0x1f/0x30 [14116.285850] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14116.287168] ll_intent_lock+0xc5/0xaa0 [lustre] [14116.288711] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14116.290732] ll_lookup_it+0xe0e/0x2270 [lustre] [14116.292168] ? do_raw_spin_unlock+0x75/0x190 [14116.293662] ? _raw_spin_unlock+0x12/0x30 [14116.294883] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14116.296455] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14116.297979] ll_atomic_open+0x36a/0xd70 [lustre] [14116.299483] lookup_open+0x356/0xb40 [14116.300581] do_last+0x313/0xfc0 [14116.301661] ? nd_jump_root+0xe5/0x160 [14116.302849] ? path_init+0x437/0x520 [14116.303993] path_openat+0xf7/0x500 [14116.305018] ? do_raw_spin_unlock+0x75/0x190 [14116.306307] do_filp_open+0x99/0x140 [14116.307448] ? getname_flags+0x6e/0x330 [14116.308634] ? __check_object_size+0xff/0x256 [14116.309950] ? do_raw_spin_unlock+0x75/0x190 [14116.311151] ? _raw_spin_unlock+0x12/0x30 [14116.312389] do_sys_openat2+0x2b4/0x410 [14116.313602] do_sys_open+0x73/0xa0 [14116.314626] __x64_sys_openat+0x24/0x30 [14116.315910] do_syscall_64+0xc1/0x440 [14116.316896] entry_SYSCALL_64_after_hwframe+0x49/0xae [14116.318215] RIP: 0033:0x7f53425a5332 [14116.319387] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14116.325050] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14116.327226] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14116.329861] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14116.332041] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14116.334348] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14116.336607] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14120.052632] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [14120.056329] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14120.058423] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [14120.080111] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14120.080125] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14120.080131] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14120.085046] Modules linked in: [14120.094677] [14120.094687] RIP: 0010:native_safe_halt+0x2e/0x50 [14120.094704] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14120.094709] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14120.094714] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14120.094716] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [14120.094718] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14120.094721] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14120.094722] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14120.094725] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14120.094729] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14120.094731] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14120.094740] Call Trace: [14120.094746] [14120.094752] ? show_regs.cold.9+0x22/0x2f [14120.094764] ? watchdog_timer_fn.cold.6+0x62/0x10a [14120.094771] ? __hrtimer_run_queues+0x1c2/0x4d0 [14120.094774] ? watchdog+0x40/0x40 [14120.094778] ? hrtimer_interrupt+0x128/0x2c0 [14120.094782] ? smp_apic_timer_interrupt+0xa6/0x270 [14120.094789] ? apic_timer_interrupt+0xf/0x20 [14120.094793] [14120.094797] ? native_safe_halt+0x2e/0x50 [14120.094801] kvm_wait+0xcd/0xe0 [14120.094808] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14120.094816] ? newidle_balance.isra.93+0x207/0x580 [14120.094823] do_raw_spin_lock+0x177/0x190 [14120.094827] _raw_spin_lock+0x21/0x30 [14120.094832] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14120.095105] process_one_work+0x2c8/0x700 [14120.095113] worker_thread+0x296/0x6e0 [14120.095117] ? rescuer_thread+0x570/0x570 [14120.095120] kthread+0x1d1/0x200 [14120.095124] ? set_kthread_struct+0x70/0x70 [14120.095128] ret_from_fork+0x1f/0x30 [14120.201670] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14120.241945] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14120.259973] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14120.274672] RIP: 0010:native_safe_halt+0x2e/0x50 [14120.281098] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14120.289838] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14120.293165] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14120.296464] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14120.300953] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14120.304136] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14120.307753] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14120.311786] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14120.316715] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14120.319148] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14120.322388] Call Trace: [14120.323472] [14120.324457] ? show_regs.cold.9+0x22/0x2f [14120.326163] ? watchdog_timer_fn.cold.6+0x62/0x10a [14120.328371] ? __hrtimer_run_queues+0x1c2/0x4d0 [14120.330645] ? watchdog+0x40/0x40 [14120.334733] ? hrtimer_interrupt+0x128/0x2c0 [14120.336720] ? smp_apic_timer_interrupt+0xa6/0x270 [14120.338702] ? apic_timer_interrupt+0xf/0x20 [14120.341845] [14120.344874] ? native_safe_halt+0x2e/0x50 [14120.347578] kvm_wait+0xcd/0xe0 [14120.353065] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14120.355387] do_raw_spin_lock+0x177/0x190 [14120.358509] _raw_spin_lock+0x21/0x30 [14120.362606] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14120.367314] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14120.370076] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14120.372668] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14120.375349] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14120.377571] ? woken_wake_function+0x30/0x30 [14120.379164] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14120.381610] kthread+0x1d1/0x200 [14120.383357] ? set_kthread_struct+0x70/0x70 [14120.384840] ret_from_fork+0x1f/0x30 [14122.980420] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11485s! [14122.988327] Showing busy workqueues and worker pools: [14122.992415] workqueue events: flags=0x0 [14122.996800] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14123.000823] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14123.008785] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14123.011458] pending: cpu_latency_work [ptlrpc] [14123.016610] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14123.023427] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14123.033358] workqueue events_long: flags=0x0 [14123.035224] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14123.040959] pending: lu_objects_flush [obdclass] [14123.044021] workqueue events_unbound: flags=0x2 [14123.046549] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14123.053028] pending: flush_memcg_stats_dwork, idle_cull_fn [14123.054878] workqueue events_power_efficient: flags=0x80 [14123.058143] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14123.061096] pending: check_lifetime, do_cache_clean [14123.064340] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14123.068487] pending: neigh_periodic_work, neigh_periodic_work [14123.072813] workqueue mm_percpu_wq: flags=0x8 [14123.074460] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14123.077529] pending: vmstat_update, lru_add_drain_per_cpu [14123.079905] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14123.084267] pending: vmstat_update, lru_add_drain_per_cpu [14123.086936] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14123.091446] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14123.095136] workqueue xprtiod: flags=0xa [14123.096448] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14123.104325] pending: xprt_autoclose, xs_error_handle [14123.110494] workqueue ipv6_addrconf: flags=0x40008 [14123.114722] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14123.122095] pending: addrconf_verify_work [14123.123393] workqueue knbd0-recv: flags=0x1a [14123.125859] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14123.127660] in-flight: 651:recv_work [14123.129681] workqueue ptlrpc_pinger: flags=0x2 [14123.132093] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14123.138470] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14123.142659] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11486s workers=4 idle: 15979 31824 8 [14123.147311] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14113s workers=2 idle: 654 [14144.055138] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [14144.057195] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [14144.060592] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14144.060674] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14144.060683] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14144.060686] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14144.060744] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14144.060750] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14144.060754] RAX: 0000000000000892 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14144.060757] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14144.060759] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14144.060761] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14144.060764] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14144.060767] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14144.069696] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14144.069848] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14144.069856] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14144.069859] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14144.069874] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14144.069878] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14144.069882] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14144.069885] RDX: 0000000000006042 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14144.069887] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14144.069889] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14144.069892] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14144.069895] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14144.069898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14144.069900] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14144.069910] Call Trace: [14144.069959] [14144.069965] ? show_regs.cold.9+0x22/0x2f [14144.069976] ? watchdog_timer_fn.cold.6+0x62/0x10a [14144.069983] ? __hrtimer_run_queues+0x1c2/0x4d0 [14144.069988] ? watchdog+0x40/0x40 [14144.069992] ? hrtimer_interrupt+0x128/0x2c0 [14144.069996] ? smp_apic_timer_interrupt+0xa6/0x270 [14144.070004] ? apic_timer_interrupt+0xf/0x20 [14144.070009] [14144.070012] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14144.070018] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14144.070022] do_raw_spin_lock+0x177/0x190 [14144.070027] _raw_spin_lock+0x21/0x30 [14144.070105] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14144.070587] obd_mod_free+0x46/0x140 [ptlrpc] [14144.071215] mdc_commit_open+0x91/0xb0 [mdc] [14144.071260] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14144.071459] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14144.071661] after_reply+0xe2c/0x1780 [ptlrpc] [14144.073431] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14144.135767] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14144.135918] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14144.135929] Call Trace: [14144.135936] [14144.135943] ? show_regs.cold.9+0x22/0x2f [14144.135958] ? watchdog_timer_fn.cold.6+0x62/0x10a [14144.135965] ? __hrtimer_run_queues+0x1c2/0x4d0 [14144.135971] ? watchdog+0x40/0x40 [14144.135976] ? hrtimer_interrupt+0x128/0x2c0 [14144.135980] ? smp_apic_timer_interrupt+0xa6/0x270 [14144.135989] ? apic_timer_interrupt+0xf/0x20 [14144.135994] [14144.135997] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14144.136008] do_raw_spin_lock+0x177/0x190 [14144.136013] _raw_spin_lock+0x21/0x30 [14144.136019] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14144.136677] ? do_raw_spin_unlock+0x75/0x190 [14144.136683] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14144.137169] ? do_wait_intr+0xf0/0xf0 [14144.137179] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14144.144672] ? __next_timer_interrupt+0x160/0x160 [14144.144697] ptlrpcd+0xa82/0xe40 [ptlrpc] [14144.145232] ? do_wait_intr+0xf0/0xf0 [14144.145299] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14144.145500] kthread+0x1d1/0x200 [14144.145506] ? set_kthread_struct+0x70/0x70 [14144.145510] ret_from_fork+0x1f/0x30 [14144.552140] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14144.556974] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14144.562679] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14144.568653] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14144.571289] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14144.575013] mdc_intent_lock+0x353/0x7e0 [mdc] [14144.577881] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14144.580858] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14144.588910] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14144.593921] lmv_intent_open+0x338/0x1150 [lmv] [14144.596458] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14144.601189] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14144.603635] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14144.606309] ll_intent_lock+0xc5/0xaa0 [lustre] [14144.610159] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14144.612897] ll_lookup_it+0xe0e/0x2270 [lustre] [14144.617601] ? do_raw_spin_unlock+0x75/0x190 [14144.619909] ? _raw_spin_unlock+0x12/0x30 [14144.621593] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14144.624265] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14144.627174] ll_atomic_open+0x36a/0xd70 [lustre] [14144.629580] lookup_open+0x356/0xb40 [14144.631286] do_last+0x313/0xfc0 [14144.632717] ? nd_jump_root+0xe5/0x160 [14144.634330] ? path_init+0x437/0x520 [14144.636031] path_openat+0xf7/0x500 [14144.637658] ? do_raw_spin_unlock+0x75/0x190 [14144.640019] do_filp_open+0x99/0x140 [14144.641651] ? getname_flags+0x6e/0x330 [14144.643486] ? __check_object_size+0xff/0x256 [14144.645659] ? do_raw_spin_unlock+0x75/0x190 [14144.647634] ? _raw_spin_unlock+0x12/0x30 [14144.649248] do_sys_openat2+0x2b4/0x410 [14144.651014] do_sys_open+0x73/0xa0 [14144.653713] __x64_sys_openat+0x24/0x30 [14144.656626] do_syscall_64+0xc1/0x440 [14144.659020] entry_SYSCALL_64_after_hwframe+0x49/0xae [14144.662940] RIP: 0033:0x7f53425a5332 [14144.663809] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14144.670274] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14144.673425] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14144.676380] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14144.679435] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14144.684548] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14144.688085] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14148.059977] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [14148.061830] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [14148.063927] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) [14148.075157] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14148.075284] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14148.075292] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14148.075294] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14148.075307] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14148.075312] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14148.075316] RAX: 0000000000001f5c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14148.075319] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14148.075321] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14148.075324] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14148.075326] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14148.075330] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14148.075333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14148.077468] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14148.077532] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14148.077540] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14148.077546] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14148.077877] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14148.077890] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14148.077893] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14148.077898] RAX: 0000000000002685 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14148.077903] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14148.077905] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14148.077907] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14148.077909] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14148.077912] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14148.077914] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14148.077916] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14148.077922] Call Trace: [14148.077931] [14148.077935] ? show_regs.cold.9+0x22/0x2f [14148.077943] ? watchdog_timer_fn.cold.6+0x62/0x10a [14148.077951] ? __hrtimer_run_queues+0x1c2/0x4d0 [14148.077955] ? watchdog+0x40/0x40 [14148.077958] ? hrtimer_interrupt+0x128/0x2c0 [14148.077962] ? smp_apic_timer_interrupt+0xa6/0x270 [14148.077971] ? apic_timer_interrupt+0xf/0x20 [14148.077976] [14148.077979] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14148.077984] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14148.077988] ? newidle_balance.isra.93+0x207/0x580 [14148.077993] do_raw_spin_lock+0x177/0x190 [14148.077997] _raw_spin_lock+0x21/0x30 [14148.078002] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14148.078272] process_one_work+0x2c8/0x700 [14148.078280] worker_thread+0x296/0x6e0 [14148.078284] ? rescuer_thread+0x570/0x570 [14148.078288] kthread+0x1d1/0x200 [14148.078293] ? set_kthread_struct+0x70/0x70 [14148.078296] ret_from_fork+0x1f/0x30 [14148.461888] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14148.465445] Call Trace: [14148.467388] [14148.471130] ? show_regs.cold.9+0x22/0x2f [14148.473910] ? watchdog_timer_fn.cold.6+0x62/0x10a [14148.477830] ? __hrtimer_run_queues+0x1c2/0x4d0 [14148.481962] ? watchdog+0x40/0x40 [14148.484003] ? hrtimer_interrupt+0x128/0x2c0 [14148.488722] ? smp_apic_timer_interrupt+0xa6/0x270 [14148.491256] ? apic_timer_interrupt+0xf/0x20 [14148.497806] [14148.498288] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14148.505035] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14148.507238] do_raw_spin_lock+0x177/0x190 [14148.509338] _raw_spin_lock+0x21/0x30 [14148.511078] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14148.515458] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14148.522869] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14148.526302] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14148.532907] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14148.540523] ? woken_wake_function+0x30/0x30 [14148.545171] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14148.552370] kthread+0x1d1/0x200 [14148.554222] ? set_kthread_struct+0x70/0x70 [14148.555527] ret_from_fork+0x1f/0x30 [14153.184048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11516s! [14153.187686] Showing busy workqueues and worker pools: [14153.189988] workqueue events: flags=0x0 [14153.191746] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14153.194733] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14153.199081] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14153.202532] pending: cpu_latency_work [ptlrpc] [14153.207242] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14153.210235] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14153.214392] workqueue events_long: flags=0x0 [14153.216844] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14153.219521] pending: lu_objects_flush [obdclass] [14153.222433] workqueue events_unbound: flags=0x2 [14153.223982] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14153.226242] pending: flush_memcg_stats_dwork, idle_cull_fn [14153.230073] workqueue events_power_efficient: flags=0x80 [14153.232374] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14153.236069] pending: check_lifetime, do_cache_clean [14153.237958] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14153.240695] pending: neigh_periodic_work, neigh_periodic_work [14153.243119] workqueue mm_percpu_wq: flags=0x8 [14153.247968] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14153.253714] pending: vmstat_update, lru_add_drain_per_cpu [14153.256735] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14153.259391] pending: vmstat_update, lru_add_drain_per_cpu [14153.267195] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14153.270185] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14153.272221] workqueue xprtiod: flags=0xa [14153.273633] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14153.275217] pending: xprt_autoclose, xs_error_handle [14153.281052] workqueue ipv6_addrconf: flags=0x40008 [14153.283854] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14153.288490] pending: addrconf_verify_work [14153.290478] workqueue knbd0-recv: flags=0x1a [14153.296064] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14153.299954] in-flight: 651:recv_work [14153.303935] workqueue ptlrpc_pinger: flags=0x2 [14153.306541] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14153.311033] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14153.317721] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11516s workers=4 idle: 15979 31824 8 [14153.324853] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14143s workers=2 idle: 654 [14172.055099] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [14172.062160] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14172.059955] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [14172.122428] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14172.122442] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14172.122445] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14172.122470] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14172.122474] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14172.122480] RAX: 0000000000002993 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14172.122482] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14172.122485] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14172.122488] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14172.122490] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14172.122494] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14172.122497] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14172.122500] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14172.122636] Call Trace: [14172.122644] [14172.122651] ? show_regs.cold.9+0x22/0x2f [14172.122662] ? watchdog_timer_fn.cold.6+0x62/0x10a [14172.122669] ? __hrtimer_run_queues+0x1c2/0x4d0 [14172.122674] ? watchdog+0x40/0x40 [14172.122678] ? hrtimer_interrupt+0x128/0x2c0 [14172.122682] ? smp_apic_timer_interrupt+0xa6/0x270 [14172.122689] ? apic_timer_interrupt+0xf/0x20 [14172.122693] [14172.122696] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14172.122704] do_raw_spin_lock+0x177/0x190 [14172.122710] _raw_spin_lock+0x21/0x30 [14172.122715] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14172.122988] ? do_raw_spin_unlock+0x75/0x190 [14172.122994] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14172.123830] ? do_wait_intr+0xf0/0xf0 [14172.124227] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14172.138028] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14172.138109] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14172.138117] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14172.138120] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14172.138139] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14172.138144] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14172.138149] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14172.138151] RDX: 00000000000058f3 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14172.138153] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14172.138155] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14172.138158] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14172.138161] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14172.138164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14172.138166] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14172.138175] Call Trace: [14172.138182] [14172.138188] ? show_regs.cold.9+0x22/0x2f [14172.138198] ? watchdog_timer_fn.cold.6+0x62/0x10a [14172.138205] ? __hrtimer_run_queues+0x1c2/0x4d0 [14172.138209] ? watchdog+0x40/0x40 [14172.138214] ? hrtimer_interrupt+0x128/0x2c0 [14172.138219] ? smp_apic_timer_interrupt+0xa6/0x270 [14172.138227] ? apic_timer_interrupt+0xf/0x20 [14172.138232] [14172.138235] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14172.138240] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14172.138254] do_raw_spin_lock+0x177/0x190 [14172.138259] _raw_spin_lock+0x21/0x30 [14172.138266] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14172.147225] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14172.147842] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14172.148127] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14172.148245] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14172.148361] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14172.158718] obd_mod_free+0x46/0x140 [ptlrpc] [14172.163860] mdc_intent_lock+0x353/0x7e0 [mdc] [14172.163911] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14172.164098] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14172.184999] mdc_commit_open+0x91/0xb0 [mdc] [14172.189104] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14172.191776] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14172.198680] lmv_intent_open+0x338/0x1150 [lmv] [14172.205596] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14172.205884] after_reply+0xe2c/0x1780 [ptlrpc] [14172.210159] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14172.213337] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14172.213614] ? __next_timer_interrupt+0x160/0x160 [14172.213624] ptlrpcd+0xa82/0xe40 [ptlrpc] [14172.213822] ? do_wait_intr+0xf0/0xf0 [14172.213832] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14172.214030] kthread+0x1d1/0x200 [14172.214036] ? set_kthread_struct+0x70/0x70 [14172.214039] ret_from_fork+0x1f/0x30 [14172.501219] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14172.505760] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14172.508073] ll_intent_lock+0xc5/0xaa0 [lustre] [14172.515215] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14172.521920] ll_lookup_it+0xe0e/0x2270 [lustre] [14172.524693] ? do_raw_spin_unlock+0x75/0x190 [14172.527363] ? _raw_spin_unlock+0x12/0x30 [14172.528828] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14172.536351] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14172.543002] ll_atomic_open+0x36a/0xd70 [lustre] [14172.546933] lookup_open+0x356/0xb40 [14172.548685] do_last+0x313/0xfc0 [14172.550648] ? nd_jump_root+0xe5/0x160 [14172.553832] ? path_init+0x437/0x520 [14172.555758] path_openat+0xf7/0x500 [14172.558548] ? do_raw_spin_unlock+0x75/0x190 [14172.561645] do_filp_open+0x99/0x140 [14172.563746] ? getname_flags+0x6e/0x330 [14172.571605] ? __check_object_size+0xff/0x256 [14172.575979] ? do_raw_spin_unlock+0x75/0x190 [14172.579136] ? _raw_spin_unlock+0x12/0x30 [14172.581429] do_sys_openat2+0x2b4/0x410 [14172.583130] do_sys_open+0x73/0xa0 [14172.588491] __x64_sys_openat+0x24/0x30 [14172.590983] do_syscall_64+0xc1/0x440 [14172.593457] entry_SYSCALL_64_after_hwframe+0x49/0xae [14172.595258] RIP: 0033:0x7f53425a5332 [14172.596689] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14172.605635] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14172.610377] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14172.615050] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14172.620961] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14172.632733] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14172.640907] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14176.058510] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [14176.064033] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14176.133293] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14176.141009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14176.146211] RIP: 0010:native_safe_halt+0x2e/0x50 [14176.148536] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14176.157519] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14176.162061] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14176.171033] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14176.174106] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14176.179368] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14176.183007] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14176.189606] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14176.198457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14176.203639] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14176.211639] Call Trace: [14176.213012] [14176.215528] ? show_regs.cold.9+0x22/0x2f [14176.216948] ? watchdog_timer_fn.cold.6+0x62/0x10a [14176.221705] ? __hrtimer_run_queues+0x1c2/0x4d0 [14176.225483] ? watchdog+0x40/0x40 [14176.229827] ? hrtimer_interrupt+0x128/0x2c0 [14176.232208] ? smp_apic_timer_interrupt+0xa6/0x270 [14176.235397] ? apic_timer_interrupt+0xf/0x20 [14176.238042] [14176.239148] ? native_safe_halt+0x2e/0x50 [14176.242520] kvm_wait+0xcd/0xe0 [14176.244720] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14176.251688] do_raw_spin_lock+0x177/0x190 [14176.253523] _raw_spin_lock+0x21/0x30 [14176.257163] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14176.260286] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14176.263765] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14176.276517] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14176.280416] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14176.284480] ? woken_wake_function+0x30/0x30 [14176.285743] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14176.290703] kthread+0x1d1/0x200 [14176.293586] ? set_kthread_struct+0x70/0x70 [14176.296775] ret_from_fork+0x1f/0x30 [14180.052122] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [14180.054628] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14180.073706] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14180.078705] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14180.082047] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14180.088005] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14180.089856] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14180.104514] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14180.110243] RAX: 00000000000047a5 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14180.114059] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14180.117556] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14180.120694] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14180.123089] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14180.126554] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14180.131919] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14180.135781] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14180.140255] Call Trace: [14180.142508] [14180.144041] ? show_regs.cold.9+0x22/0x2f [14180.148425] ? watchdog_timer_fn.cold.6+0x62/0x10a [14180.149884] ? __hrtimer_run_queues+0x1c2/0x4d0 [14180.151893] ? watchdog+0x40/0x40 [14180.152966] ? hrtimer_interrupt+0x128/0x2c0 [14180.154137] ? smp_apic_timer_interrupt+0xa6/0x270 [14180.156030] ? apic_timer_interrupt+0xf/0x20 [14180.157964] [14180.158836] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14180.161974] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14180.164885] ? newidle_balance.isra.93+0x207/0x580 [14180.167820] do_raw_spin_lock+0x177/0x190 [14180.169316] _raw_spin_lock+0x21/0x30 [14180.170542] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14180.175256] process_one_work+0x2c8/0x700 [14180.178810] worker_thread+0x296/0x6e0 [14180.181692] ? rescuer_thread+0x570/0x570 [14180.183355] kthread+0x1d1/0x200 [14180.184600] ? set_kthread_struct+0x70/0x70 [14180.186289] ret_from_fork+0x1f/0x30 [14183.393194] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11546s! [14183.397080] Showing busy workqueues and worker pools: [14183.398790] workqueue events: flags=0x0 [14183.401543] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14183.404513] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14183.414567] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14183.421582] pending: cpu_latency_work [ptlrpc] [14183.425458] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14183.431524] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14183.438982] workqueue events_long: flags=0x0 [14183.442465] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14183.445622] pending: lu_objects_flush [obdclass] [14183.448308] workqueue events_unbound: flags=0x2 [14183.450549] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14183.453437] pending: flush_memcg_stats_dwork, idle_cull_fn [14183.456597] workqueue events_power_efficient: flags=0x80 [14183.458724] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14183.462196] pending: check_lifetime, do_cache_clean [14183.463984] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14183.465647] pending: neigh_periodic_work, neigh_periodic_work [14183.467294] workqueue mm_percpu_wq: flags=0x8 [14183.468402] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14183.470262] pending: vmstat_update, lru_add_drain_per_cpu [14183.471606] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14183.473524] pending: vmstat_update, lru_add_drain_per_cpu [14183.475032] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14183.476716] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14183.478418] workqueue xprtiod: flags=0xa [14183.479936] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14183.482472] pending: xprt_autoclose, xs_error_handle [14183.484084] workqueue ipv6_addrconf: flags=0x40008 [14183.485495] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14183.488625] pending: addrconf_verify_work [14183.490613] workqueue knbd0-recv: flags=0x1a [14183.492628] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14183.495466] in-flight: 651:recv_work [14183.497051] workqueue ptlrpc_pinger: flags=0x2 [14183.499075] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14183.502617] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14183.506446] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11546s workers=4 idle: 15979 31824 8 [14183.509817] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14173s workers=2 idle: 654 [14200.055344] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [14200.057537] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [14200.065934] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14200.066010] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14200.066028] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14200.066030] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14200.066044] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14200.066048] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14200.066053] RAX: 0000000000007489 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14200.066055] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14200.066058] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14200.066060] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14200.066063] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14200.066066] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14200.066068] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14200.066070] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14200.066080] Call Trace: [14200.066085] [14200.066090] ? show_regs.cold.9+0x22/0x2f [14200.066099] ? watchdog_timer_fn.cold.6+0x62/0x10a [14200.066106] ? __hrtimer_run_queues+0x1c2/0x4d0 [14200.066110] ? watchdog+0x40/0x40 [14200.066113] ? hrtimer_interrupt+0x128/0x2c0 [14200.066117] ? smp_apic_timer_interrupt+0xa6/0x270 [14200.066125] ? apic_timer_interrupt+0xf/0x20 [14200.066129] [14200.066132] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14200.066138] do_raw_spin_lock+0x177/0x190 [14200.066143] _raw_spin_lock+0x21/0x30 [14200.066148] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14200.073492] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14200.073578] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14200.073586] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14200.073589] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14200.073603] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14200.073608] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14200.073613] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14200.073616] RDX: 0000000000003a1c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14200.073619] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14200.073621] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14200.073623] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14200.073626] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14200.073630] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14200.073632] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14200.073641] Call Trace: [14200.073647] [14200.073652] ? show_regs.cold.9+0x22/0x2f [14200.073662] ? watchdog_timer_fn.cold.6+0x62/0x10a [14200.073668] ? __hrtimer_run_queues+0x1c2/0x4d0 [14200.073672] ? watchdog+0x40/0x40 [14200.073677] ? hrtimer_interrupt+0x128/0x2c0 [14200.073681] ? smp_apic_timer_interrupt+0xa6/0x270 [14200.073688] ? apic_timer_interrupt+0xf/0x20 [14200.073692] [14200.073695] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14200.073700] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14200.073705] do_raw_spin_lock+0x177/0x190 [14200.073710] _raw_spin_lock+0x21/0x30 [14200.073716] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14200.075354] obd_mod_free+0x46/0x140 [ptlrpc] [14200.075749] mdc_commit_open+0x91/0xb0 [mdc] [14200.075789] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14200.076008] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14200.076209] after_reply+0xe2c/0x1780 [ptlrpc] [14200.076423] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14200.076634] ? __next_timer_interrupt+0x160/0x160 [14200.076645] ptlrpcd+0xa82/0xe40 [ptlrpc] [14200.076876] ? do_wait_intr+0xf0/0xf0 [14200.076886] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14200.077095] kthread+0x1d1/0x200 [14200.077102] ? set_kthread_struct+0x70/0x70 [14200.077106] ret_from_fork+0x1f/0x30 [14200.588906] ? do_raw_spin_unlock+0x75/0x190 [14200.591301] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14200.595417] ? do_wait_intr+0xf0/0xf0 [14200.599886] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14200.605995] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14200.610741] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14200.619233] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14200.626448] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14200.632779] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14200.638459] mdc_intent_lock+0x353/0x7e0 [mdc] [14200.640121] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14200.649315] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14200.657510] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14200.664709] lmv_intent_open+0x338/0x1150 [lmv] [14200.665969] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14200.669957] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14200.678533] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14200.683555] ll_intent_lock+0xc5/0xaa0 [lustre] [14200.693550] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14200.703186] ll_lookup_it+0xe0e/0x2270 [lustre] [14200.705713] ? do_raw_spin_unlock+0x75/0x190 [14200.712766] ? _raw_spin_unlock+0x12/0x30 [14200.724700] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14200.726437] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14200.735439] ll_atomic_open+0x36a/0xd70 [lustre] [14200.741435] lookup_open+0x356/0xb40 [14200.743317] do_last+0x313/0xfc0 [14200.748961] ? nd_jump_root+0xe5/0x160 [14200.752548] ? path_init+0x437/0x520 [14200.760716] path_openat+0xf7/0x500 [14200.763998] ? do_raw_spin_unlock+0x75/0x190 [14200.770225] do_filp_open+0x99/0x140 [14200.771576] ? getname_flags+0x6e/0x330 [14200.775456] ? __check_object_size+0xff/0x256 [14200.778110] ? do_raw_spin_unlock+0x75/0x190 [14200.781733] ? _raw_spin_unlock+0x12/0x30 [14200.783498] do_sys_openat2+0x2b4/0x410 [14200.785166] do_sys_open+0x73/0xa0 [14200.786364] __x64_sys_openat+0x24/0x30 [14200.792489] do_syscall_64+0xc1/0x440 [14200.795205] entry_SYSCALL_64_after_hwframe+0x49/0xae [14200.797974] RIP: 0033:0x7f53425a5332 [14200.799535] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14200.816125] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14200.820277] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14200.824077] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14200.827385] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14200.833453] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14200.838474] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14204.067078] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [14204.084298] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14204.150384] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14204.170321] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14204.183002] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14204.192592] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14204.222864] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14204.230376] RAX: 0000000000004e8f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14204.244097] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14204.254827] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14204.266226] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14204.276333] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14204.278839] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14204.299877] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14204.303955] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14204.312594] Call Trace: [14204.315010] [14204.324935] ? show_regs.cold.9+0x22/0x2f [14204.326869] ? watchdog_timer_fn.cold.6+0x62/0x10a [14204.335719] ? __hrtimer_run_queues+0x1c2/0x4d0 [14204.339194] ? watchdog+0x40/0x40 [14204.342385] ? hrtimer_interrupt+0x128/0x2c0 [14204.345256] ? smp_apic_timer_interrupt+0xa6/0x270 [14204.350367] ? apic_timer_interrupt+0xf/0x20 [14204.357312] [14204.360095] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14204.364121] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14204.372240] do_raw_spin_lock+0x177/0x190 [14204.376948] _raw_spin_lock+0x21/0x30 [14204.382917] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14204.388933] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14204.395230] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14204.398119] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14204.408495] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14204.424803] ? woken_wake_function+0x30/0x30 [14204.435154] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14204.443256] kthread+0x1d1/0x200 [14204.454279] ? set_kthread_struct+0x70/0x70 [14204.455583] ret_from_fork+0x1f/0x30 [14208.062622] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [14208.089927] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14208.149091] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14208.163029] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14208.172709] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14208.177234] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14208.182039] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14208.192546] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14208.195974] RAX: 0000000000006022 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14208.202966] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14208.206937] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14208.209737] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14208.212119] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14208.218097] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14208.224091] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14208.229175] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14208.244240] Call Trace: [14208.249269] [14208.252337] ? show_regs.cold.9+0x22/0x2f [14208.254267] ? watchdog_timer_fn.cold.6+0x62/0x10a [14208.257758] ? __hrtimer_run_queues+0x1c2/0x4d0 [14208.259051] ? watchdog+0x40/0x40 [14208.261131] ? hrtimer_interrupt+0x128/0x2c0 [14208.269832] ? smp_apic_timer_interrupt+0xa6/0x270 [14208.276874] ? apic_timer_interrupt+0xf/0x20 [14208.278422] [14208.280715] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14208.282532] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14208.285949] ? newidle_balance.isra.93+0x207/0x580 [14208.291917] do_raw_spin_lock+0x177/0x190 [14208.297317] _raw_spin_lock+0x21/0x30 [14208.302354] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14208.305600] process_one_work+0x2c8/0x700 [14208.307006] worker_thread+0x296/0x6e0 [14208.311265] ? rescuer_thread+0x570/0x570 [14208.314001] kthread+0x1d1/0x200 [14208.315821] ? set_kthread_struct+0x70/0x70 [14208.318274] ret_from_fork+0x1f/0x30 [14213.605493] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11576s! [14213.620752] Showing busy workqueues and worker pools: [14213.626502] workqueue events: flags=0x0 [14213.629698] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14213.635823] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14213.649208] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14213.657156] pending: cpu_latency_work [ptlrpc] [14213.665157] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14213.671934] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14213.683219] workqueue events_long: flags=0x0 [14213.684851] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14213.688255] pending: lu_objects_flush [obdclass] [14213.691363] workqueue events_unbound: flags=0x2 [14213.697066] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14213.701056] pending: flush_memcg_stats_dwork, idle_cull_fn [14213.707287] workqueue events_power_efficient: flags=0x80 [14213.715071] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14213.719510] pending: check_lifetime, do_cache_clean [14213.724910] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14213.730546] pending: neigh_periodic_work, neigh_periodic_work [14213.734597] workqueue mm_percpu_wq: flags=0x8 [14213.736220] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14213.741453] pending: vmstat_update, lru_add_drain_per_cpu [14213.743496] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14213.755885] pending: vmstat_update, lru_add_drain_per_cpu [14213.766655] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14213.774915] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14213.784916] workqueue xprtiod: flags=0xa [14213.794861] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14213.803174] pending: xprt_autoclose, xs_error_handle [14213.808405] workqueue ipv6_addrconf: flags=0x40008 [14213.814400] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14213.820883] pending: addrconf_verify_work [14213.823129] workqueue knbd0-recv: flags=0x1a [14213.828767] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14213.840035] in-flight: 651:recv_work [14213.843178] workqueue ptlrpc_pinger: flags=0x2 [14213.847198] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14213.853326] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14213.858095] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11576s workers=4 idle: 15979 31824 8 [14213.869675] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14204s workers=2 idle: 654 [14214.724099] rcu: INFO: rcu_sched self-detected stall on CPU [14214.732339] rcu: 2-....: (6738770 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [14214.740309] (t=12120298 jiffies g=274185 q=66939) [14214.742515] rcu: rcu_sched kthread starved for 11578664 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [14214.748950] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14214.752332] rcu: RCU grace-period kthread stack dump: [14214.755511] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14214.761250] Call Trace: [14214.763652] __schedule+0x351/0xcb0 [14214.765185] schedule+0xc0/0x180 [14214.766884] schedule_timeout+0xb4/0x190 [14214.769590] ? do_raw_spin_unlock+0x75/0x190 [14214.771084] ? __next_timer_interrupt+0x160/0x160 [14214.772898] ? prepare_to_swait_event+0xf3/0x140 [14214.775720] rcu_gp_kthread+0x8dd/0x15b0 [14214.777287] ? force_qs_rnp+0x2c0/0x2c0 [14214.779607] kthread+0x1d1/0x200 [14214.781137] ? set_kthread_struct+0x70/0x70 [14214.783377] ret_from_fork+0x1f/0x30 [14214.785120] rcu: Stack dump where RCU GP kthread last ran: [14214.788806] Sending NMI from CPU 2 to CPUs 1: [14214.801689] NMI backtrace for cpu 1 [14214.801697] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14214.801699] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14214.801701] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14214.801702] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14214.801705] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [14214.801707] RAX: 0000000000005986 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14214.801709] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14214.801710] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14214.801712] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14214.801714] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14214.801715] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14214.801717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14214.801718] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14214.801719] Call Trace: [14214.801720] [14214.801721] ? show_regs.cold.9+0x22/0x2f [14214.801722] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [14214.801723] ? nmi_cpu_backtrace_handler+0x15/0x40 [14214.801724] ? nmi_handle+0xbd/0x210 [14214.801725] ? default_do_nmi+0x4b/0x1b0 [14214.801726] ? do_nmi+0x280/0x380 [14214.801727] ? end_repeat_nmi+0x16/0x1f [14214.801728] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14214.801730] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14214.801731] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14214.801732] [14214.801733] do_raw_spin_lock+0x177/0x190 [14214.801734] _raw_spin_lock+0x21/0x30 [14214.801735] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14214.801736] ? do_raw_spin_unlock+0x75/0x190 [14214.801738] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14214.801739] ? do_wait_intr+0xf0/0xf0 [14214.801740] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14214.801741] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14214.801742] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14214.801744] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14214.801745] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14214.801746] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14214.801747] mdc_intent_lock+0x353/0x7e0 [mdc] [14214.801748] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14214.801749] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14214.801750] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14214.801751] lmv_intent_open+0x338/0x1150 [lmv] [14214.801752] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14214.801753] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14214.801754] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14214.801755] ll_intent_lock+0xc5/0xaa0 [lustre] [14214.801757] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14214.801758] ll_lookup_it+0xe0e/0x2270 [lustre] [14214.801759] ? do_raw_spin_unlock+0x75/0x190 [14214.801760] ? _raw_spin_unlock+0x12/0x30 [14214.801761] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14214.801763] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14214.801764] ll_atomic_open+0x36a/0xd70 [lustre] [14214.801765] lookup_open+0x356/0xb40 [14214.801766] do_last+0x313/0xfc0 [14214.801767] ? nd_jump_root+0xe5/0x160 [14214.801768] ? path_init+0x437/0x520 [14214.801769] path_openat+0xf7/0x500 [14214.801770] ? do_raw_spin_unlock+0x75/0x190 [14214.801771] do_filp_open+0x99/0x140 [14214.801772] ? getname_flags+0x6e/0x330 [14214.801773] ? __check_object_size+0xff/0x256 [14214.801774] ? do_raw_spin_unlock+0x75/0x190 [14214.801775] ? _raw_spin_unlock+0x12/0x30 [14214.801776] do_sys_openat2+0x2b4/0x410 [14214.801777] do_sys_open+0x73/0xa0 [14214.801779] __x64_sys_openat+0x24/0x30 [14214.801780] do_syscall_64+0xc1/0x440 [14214.801781] entry_SYSCALL_64_after_hwframe+0x49/0xae [14214.801782] RIP: 0033:0x7f53425a5332 [14214.801784] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14214.801786] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14214.801789] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14214.801790] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14214.801792] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14214.801793] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14214.801794] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14214.802964] Sending NMI from CPU 2 to CPUs 1: [14215.225064] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [14215.228470] NMI backtrace for cpu 2 [14215.232252] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14215.246030] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14215.259857] Call Trace: [14215.263860] [14215.266326] ? dump_stack+0xbb/0x10e [14215.269803] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14215.276887] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14215.283144] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14215.288893] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14215.293858] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14215.302784] ? rcu_dump_cpu_stacks+0x11b/0x171 [14215.305334] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14215.315335] ? update_process_times+0x6d/0xc0 [14215.317751] ? tick_sched_handle.isra.18+0x27/0x90 [14215.319825] ? tick_sched_timer+0x53/0xb0 [14215.322772] ? __hrtimer_run_queues+0x1c2/0x4d0 [14215.325797] ? tick_sched_handle.isra.18+0x90/0x90 [14215.335481] ? hrtimer_interrupt+0x128/0x2c0 [14215.339773] ? smp_apic_timer_interrupt+0xa6/0x270 [14215.346203] ? apic_timer_interrupt+0xf/0x20 [14215.351837] [14215.352353] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14215.355154] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14215.360273] ? do_raw_spin_lock+0x177/0x190 [14215.365591] ? _raw_spin_lock+0x21/0x30 [14215.370490] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14215.377740] ? obd_mod_free+0x46/0x140 [ptlrpc] [14215.384226] ? mdc_commit_open+0x91/0xb0 [mdc] [14215.387298] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14215.396484] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14215.402641] ? after_reply+0xe2c/0x1780 [ptlrpc] [14215.410906] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14215.416169] ? __next_timer_interrupt+0x160/0x160 [14215.421466] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14215.425569] ? do_wait_intr+0xf0/0xf0 [14215.431568] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14215.438279] ? kthread+0x1d1/0x200 [14215.441575] ? set_kthread_struct+0x70/0x70 [14215.443045] ? ret_from_fork+0x1f/0x30 [14232.058764] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [14232.061802] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14232.084627] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14232.090652] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14232.094348] RIP: 0010:native_safe_halt+0x2e/0x50 [14232.096208] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14232.104500] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14232.107895] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14232.110887] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14232.113893] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14232.116961] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14232.120047] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14232.123293] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14232.126565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14232.128770] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14232.131836] Call Trace: [14232.133399] [14232.134254] ? show_regs.cold.9+0x22/0x2f [14232.136206] ? watchdog_timer_fn.cold.6+0x62/0x10a [14232.139324] ? __hrtimer_run_queues+0x1c2/0x4d0 [14232.142561] ? watchdog+0x40/0x40 [14232.144405] ? hrtimer_interrupt+0x128/0x2c0 [14232.146920] ? smp_apic_timer_interrupt+0xa6/0x270 [14232.148850] ? apic_timer_interrupt+0xf/0x20 [14232.150747] [14232.151647] ? native_safe_halt+0x2e/0x50 [14232.153410] kvm_wait+0xcd/0xe0 [14232.154826] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14232.157302] do_raw_spin_lock+0x177/0x190 [14232.158796] _raw_spin_lock+0x21/0x30 [14232.160520] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14232.163456] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14232.165804] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14232.169040] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14232.171731] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14232.174431] ? woken_wake_function+0x30/0x30 [14232.176048] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14232.178915] kthread+0x1d1/0x200 [14232.180606] ? set_kthread_struct+0x70/0x70 [14232.182214] ret_from_fork+0x1f/0x30 [14240.053201] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [14240.055226] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [14240.057048] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [14240.057060] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14240.057199] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14240.057204] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14240.057206] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14240.057232] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14240.057237] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14240.057241] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14240.057243] RDX: 0000000000000333 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14240.057245] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14240.057247] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14240.057249] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14240.057252] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14240.057255] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14240.057256] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14240.057265] Call Trace: [14240.057270] [14240.057275] ? show_regs.cold.9+0x22/0x2f [14240.057285] ? watchdog_timer_fn.cold.6+0x62/0x10a [14240.057291] ? __hrtimer_run_queues+0x1c2/0x4d0 [14240.057294] ? watchdog+0x40/0x40 [14240.057298] ? hrtimer_interrupt+0x128/0x2c0 [14240.057302] ? smp_apic_timer_interrupt+0xa6/0x270 [14240.057310] ? apic_timer_interrupt+0xf/0x20 [14240.057315] [14240.057317] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14240.057322] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14240.057326] do_raw_spin_lock+0x177/0x190 [14240.057330] _raw_spin_lock+0x21/0x30 [14240.057336] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14240.057630] obd_mod_free+0x46/0x140 [ptlrpc] [14240.058112] mdc_commit_open+0x91/0xb0 [mdc] [14240.058156] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14240.058358] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14240.058571] after_reply+0xe2c/0x1780 [ptlrpc] [14240.059282] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14240.059471] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14240.059534] ? __next_timer_interrupt+0x160/0x160 [14240.059542] [14240.059547] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14240.059556] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14240.059562] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14240.059923] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14240.059932] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14240.059936] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14240.059941] RAX: 0000000000004188 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14240.059943] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14240.059945] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14240.059947] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14240.059949] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14240.059952] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14240.059954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14240.059956] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14240.059966] Call Trace: [14240.059969] [14240.059972] ? show_regs.cold.9+0x22/0x2f [14240.059981] ? watchdog_timer_fn.cold.6+0x62/0x10a [14240.059987] ? __hrtimer_run_queues+0x1c2/0x4d0 [14240.059991] ? watchdog+0x40/0x40 [14240.059995] ? hrtimer_interrupt+0x128/0x2c0 [14240.059999] ? smp_apic_timer_interrupt+0xa6/0x270 [14240.060006] ? apic_timer_interrupt+0xf/0x20 [14240.060011] [14240.060013] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14240.060018] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14240.060022] ? newidle_balance.isra.93+0x207/0x580 [14240.060027] do_raw_spin_lock+0x177/0x190 [14240.060032] _raw_spin_lock+0x21/0x30 [14240.060037] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14240.060409] process_one_work+0x2c8/0x700 [14240.060417] worker_thread+0x296/0x6e0 [14240.060421] ? rescuer_thread+0x570/0x570 [14240.060425] kthread+0x1d1/0x200 [14240.060429] ? set_kthread_struct+0x70/0x70 [14240.060433] ret_from_fork+0x1f/0x30 [14240.059549] ptlrpcd+0xa82/0xe40 [ptlrpc] [14240.060922] ? do_wait_intr+0xf0/0xf0 [14240.060930] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14240.061223] kthread+0x1d1/0x200 [14240.061228] ? set_kthread_struct+0x70/0x70 [14240.061232] ret_from_fork+0x1f/0x30 [14240.499674] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14240.537947] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14240.542286] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14240.549182] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14240.553892] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14240.561742] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14240.564642] RAX: 0000000000007d8f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14240.567206] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14240.569469] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14240.572541] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14240.577581] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14240.579798] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14240.588272] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14240.593666] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14240.595815] Call Trace: [14240.599751] [14240.600520] ? show_regs.cold.9+0x22/0x2f [14240.602965] ? watchdog_timer_fn.cold.6+0x62/0x10a [14240.608580] ? __hrtimer_run_queues+0x1c2/0x4d0 [14240.610064] ? watchdog+0x40/0x40 [14240.612594] ? hrtimer_interrupt+0x128/0x2c0 [14240.614760] ? smp_apic_timer_interrupt+0xa6/0x270 [14240.616726] ? apic_timer_interrupt+0xf/0x20 [14240.618023] [14240.618702] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14240.620851] do_raw_spin_lock+0x177/0x190 [14240.622129] _raw_spin_lock+0x21/0x30 [14240.624014] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14240.626533] ? do_raw_spin_unlock+0x75/0x190 [14240.628139] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14240.629894] ? do_wait_intr+0xf0/0xf0 [14240.635065] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14240.638759] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14240.641283] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14240.648244] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14240.651811] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14240.653986] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14240.657696] mdc_intent_lock+0x353/0x7e0 [mdc] [14240.659164] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14240.665047] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14240.667673] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14240.669938] lmv_intent_open+0x338/0x1150 [lmv] [14240.672302] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14240.676753] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14240.683050] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14240.685302] ll_intent_lock+0xc5/0xaa0 [lustre] [14240.687431] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14240.690024] ll_lookup_it+0xe0e/0x2270 [lustre] [14240.693665] ? do_raw_spin_unlock+0x75/0x190 [14240.695765] ? _raw_spin_unlock+0x12/0x30 [14240.697225] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14240.699685] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14240.701917] ll_atomic_open+0x36a/0xd70 [lustre] [14240.703641] lookup_open+0x356/0xb40 [14240.705579] do_last+0x313/0xfc0 [14240.709546] ? nd_jump_root+0xe5/0x160 [14240.710782] ? path_init+0x437/0x520 [14240.711868] path_openat+0xf7/0x500 [14240.715285] ? do_raw_spin_unlock+0x75/0x190 [14240.717609] do_filp_open+0x99/0x140 [14240.719417] ? getname_flags+0x6e/0x330 [14240.725529] ? __check_object_size+0xff/0x256 [14240.727156] ? do_raw_spin_unlock+0x75/0x190 [14240.728556] ? _raw_spin_unlock+0x12/0x30 [14240.730293] do_sys_openat2+0x2b4/0x410 [14240.731634] do_sys_open+0x73/0xa0 [14240.732866] __x64_sys_openat+0x24/0x30 [14240.734508] do_syscall_64+0xc1/0x440 [14240.735701] entry_SYSCALL_64_after_hwframe+0x49/0xae [14240.737717] RIP: 0033:0x7f53425a5332 [14240.740543] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14240.748528] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14240.752825] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14240.755757] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14240.758301] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14240.764696] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14240.769351] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14244.319561] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11607s! [14244.324871] Showing busy workqueues and worker pools: [14244.328062] workqueue events: flags=0x0 [14244.330469] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14244.334522] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14244.341375] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14244.344912] pending: cpu_latency_work [ptlrpc] [14244.348198] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14244.351774] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14244.356837] workqueue events_long: flags=0x0 [14244.359442] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14244.363574] pending: lu_objects_flush [obdclass] [14244.366427] workqueue events_unbound: flags=0x2 [14244.369716] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14244.373428] pending: flush_memcg_stats_dwork, idle_cull_fn [14244.376774] workqueue events_power_efficient: flags=0x80 [14244.380956] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14244.385760] pending: check_lifetime, do_cache_clean [14244.389277] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14244.393567] pending: neigh_periodic_work, neigh_periodic_work [14244.396815] workqueue mm_percpu_wq: flags=0x8 [14244.398852] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14244.402543] pending: vmstat_update, lru_add_drain_per_cpu [14244.405855] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14244.409471] pending: vmstat_update, lru_add_drain_per_cpu [14244.418476] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14244.426499] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14244.429965] workqueue xprtiod: flags=0xa [14244.432413] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14244.436473] pending: xprt_autoclose, xs_error_handle [14244.438967] workqueue ipv6_addrconf: flags=0x40008 [14244.441706] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14244.445189] pending: addrconf_verify_work [14244.447497] workqueue knbd0-recv: flags=0x1a [14244.450092] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14244.453607] in-flight: 651:recv_work [14244.455688] workqueue ptlrpc_pinger: flags=0x2 [14244.457919] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14244.461114] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14244.464662] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11607s workers=4 idle: 15979 31824 8 [14244.468975] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14234s workers=2 idle: 654 [14260.058025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [14260.060449] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14260.098257] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14260.104064] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14260.109358] RIP: 0010:native_safe_halt+0x2e/0x50 [14260.115247] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14260.129289] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14260.136272] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14260.150646] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14260.156702] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14260.163216] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14260.167500] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14260.172420] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14260.177389] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14260.183894] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14260.186391] Call Trace: [14260.191371] [14260.192459] ? show_regs.cold.9+0x22/0x2f [14260.196436] ? watchdog_timer_fn.cold.6+0x62/0x10a [14260.198718] ? __hrtimer_run_queues+0x1c2/0x4d0 [14260.202785] ? watchdog+0x40/0x40 [14260.203968] ? hrtimer_interrupt+0x128/0x2c0 [14260.213944] ? smp_apic_timer_interrupt+0xa6/0x270 [14260.222465] ? apic_timer_interrupt+0xf/0x20 [14260.227726] [14260.232328] ? native_safe_halt+0x2e/0x50 [14260.237615] kvm_wait+0xcd/0xe0 [14260.247456] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14260.257587] do_raw_spin_lock+0x177/0x190 [14260.261396] _raw_spin_lock+0x21/0x30 [14260.263406] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14260.265776] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14260.268364] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14260.271239] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14260.273923] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14260.276463] ? woken_wake_function+0x30/0x30 [14260.279064] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14260.283509] kthread+0x1d1/0x200 [14260.286112] ? set_kthread_struct+0x70/0x70 [14260.288005] ret_from_fork+0x1f/0x30 [14268.055805] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [14268.062182] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14268.063776] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [14268.093138] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14268.093156] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14268.093160] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14268.093176] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14268.093180] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14268.093186] RAX: 0000000000007f82 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14268.093188] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14268.093191] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14268.093193] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14268.093196] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14268.093199] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14268.093202] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14268.093205] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14268.093213] Call Trace: [14268.093217] [14268.093223] ? show_regs.cold.9+0x22/0x2f [14268.093233] ? watchdog_timer_fn.cold.6+0x62/0x10a [14268.093239] ? __hrtimer_run_queues+0x1c2/0x4d0 [14268.093243] ? watchdog+0x40/0x40 [14268.093248] ? hrtimer_interrupt+0x128/0x2c0 [14268.093252] ? smp_apic_timer_interrupt+0xa6/0x270 [14268.093260] ? apic_timer_interrupt+0xf/0x20 [14268.093265] [14268.093267] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14268.093275] do_raw_spin_lock+0x177/0x190 [14268.093280] _raw_spin_lock+0x21/0x30 [14268.093284] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14268.096086] ? do_raw_spin_unlock+0x75/0x190 [14268.096099] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14268.103786] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14268.103866] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14268.103874] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14268.103876] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14268.103890] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14268.103894] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14268.103898] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14268.103900] RDX: 0000000000004b7e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14268.103902] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14268.103904] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14268.103906] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14268.103909] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14268.103912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14268.103915] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14268.103924] Call Trace: [14268.103930] [14268.103936] ? show_regs.cold.9+0x22/0x2f [14268.103946] ? watchdog_timer_fn.cold.6+0x62/0x10a [14268.103953] ? __hrtimer_run_queues+0x1c2/0x4d0 [14268.103957] ? watchdog+0x40/0x40 [14268.103960] ? hrtimer_interrupt+0x128/0x2c0 [14268.103964] ? smp_apic_timer_interrupt+0xa6/0x270 [14268.103973] ? apic_timer_interrupt+0xf/0x20 [14268.103977] [14268.103980] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14268.103985] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14268.103990] do_raw_spin_lock+0x177/0x190 [14268.103995] _raw_spin_lock+0x21/0x30 [14268.104001] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14268.114454] ? do_wait_intr+0xf0/0xf0 [14268.114476] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14268.118948] obd_mod_free+0x46/0x140 [ptlrpc] [14268.119330] mdc_commit_open+0x91/0xb0 [mdc] [14268.119368] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14268.119560] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14268.119848] after_reply+0xe2c/0x1780 [ptlrpc] [14268.120040] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14268.120434] ? __next_timer_interrupt+0x160/0x160 [14268.120445] ptlrpcd+0xa82/0xe40 [ptlrpc] [14268.120658] ? do_wait_intr+0xf0/0xf0 [14268.120667] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14268.120984] kthread+0x1d1/0x200 [14268.120990] ? set_kthread_struct+0x70/0x70 [14268.120994] ret_from_fork+0x1f/0x30 [14268.456402] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14268.461001] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14268.465231] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14268.470809] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14268.478229] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14268.484093] mdc_intent_lock+0x353/0x7e0 [mdc] [14268.486423] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14268.489234] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14268.492985] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14268.495412] lmv_intent_open+0x338/0x1150 [lmv] [14268.497367] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14268.501229] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14268.503248] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14268.505454] ll_intent_lock+0xc5/0xaa0 [lustre] [14268.510322] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14268.514362] ll_lookup_it+0xe0e/0x2270 [lustre] [14268.520994] ? do_raw_spin_unlock+0x75/0x190 [14268.523097] ? _raw_spin_unlock+0x12/0x30 [14268.524514] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14268.536269] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14268.537973] ll_atomic_open+0x36a/0xd70 [lustre] [14268.549034] lookup_open+0x356/0xb40 [14268.550311] do_last+0x313/0xfc0 [14268.554060] ? nd_jump_root+0xe5/0x160 [14268.559434] ? path_init+0x437/0x520 [14268.565360] path_openat+0xf7/0x500 [14268.573065] ? do_raw_spin_unlock+0x75/0x190 [14268.576086] do_filp_open+0x99/0x140 [14268.579164] ? getname_flags+0x6e/0x330 [14268.582040] ? __check_object_size+0xff/0x256 [14268.586823] ? do_raw_spin_unlock+0x75/0x190 [14268.588771] ? _raw_spin_unlock+0x12/0x30 [14268.592973] do_sys_openat2+0x2b4/0x410 [14268.596084] do_sys_open+0x73/0xa0 [14268.597421] __x64_sys_openat+0x24/0x30 [14268.599221] do_syscall_64+0xc1/0x440 [14268.601084] entry_SYSCALL_64_after_hwframe+0x49/0xae [14268.604412] RIP: 0033:0x7f53425a5332 [14268.608857] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14268.619257] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14268.624551] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14268.627364] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14268.630420] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14268.634107] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14268.637318] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14272.054056] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [14272.063861] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14272.123133] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14272.130442] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14272.143433] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14272.150414] RIP: 0010:native_safe_halt+0x2e/0x50 [14272.154084] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14272.165669] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14272.169182] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14272.179618] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14272.181601] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14272.191760] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14272.203131] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14272.210125] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14272.215344] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14272.218824] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14272.222618] Call Trace: [14272.224509] [14272.225973] ? show_regs.cold.9+0x22/0x2f [14272.228449] ? watchdog_timer_fn.cold.6+0x62/0x10a [14272.231865] ? __hrtimer_run_queues+0x1c2/0x4d0 [14272.234421] ? watchdog+0x40/0x40 [14272.236390] ? hrtimer_interrupt+0x128/0x2c0 [14272.238863] ? smp_apic_timer_interrupt+0xa6/0x270 [14272.241804] ? apic_timer_interrupt+0xf/0x20 [14272.244444] [14272.245745] ? native_safe_halt+0x2e/0x50 [14272.248055] kvm_wait+0xcd/0xe0 [14272.249829] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14272.252757] ? newidle_balance.isra.93+0x207/0x580 [14272.255889] do_raw_spin_lock+0x177/0x190 [14272.258291] _raw_spin_lock+0x21/0x30 [14272.260515] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14272.263880] process_one_work+0x2c8/0x700 [14272.266143] worker_thread+0x296/0x6e0 [14272.268151] ? rescuer_thread+0x570/0x570 [14272.270087] kthread+0x1d1/0x200 [14272.271685] ? set_kthread_struct+0x70/0x70 [14272.273806] ret_from_fork+0x1f/0x30 [14274.530097] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11637s! [14274.543766] Showing busy workqueues and worker pools: [14274.549265] workqueue events: flags=0x0 [14274.555831] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14274.561472] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14274.575439] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14274.583654] pending: cpu_latency_work [ptlrpc] [14274.587331] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14274.592478] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14274.603271] workqueue events_long: flags=0x0 [14274.607821] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14274.615047] pending: lu_objects_flush [obdclass] [14274.621466] workqueue events_unbound: flags=0x2 [14274.629136] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14274.634273] pending: flush_memcg_stats_dwork, idle_cull_fn [14274.638821] workqueue events_power_efficient: flags=0x80 [14274.643389] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14274.647389] pending: check_lifetime, do_cache_clean [14274.650033] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14274.653428] pending: neigh_periodic_work, neigh_periodic_work [14274.657331] workqueue mm_percpu_wq: flags=0x8 [14274.660059] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14274.663787] pending: vmstat_update, lru_add_drain_per_cpu [14274.666764] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14274.670975] pending: vmstat_update, lru_add_drain_per_cpu [14274.674268] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14274.678037] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14274.681210] workqueue xprtiod: flags=0xa [14274.683129] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14274.690780] pending: xprt_autoclose, xs_error_handle [14274.697451] workqueue ipv6_addrconf: flags=0x40008 [14274.704380] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14274.709705] pending: addrconf_verify_work [14274.711693] workqueue knbd0-recv: flags=0x1a [14274.713864] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14274.718406] in-flight: 651:recv_work [14274.720442] workqueue ptlrpc_pinger: flags=0x2 [14274.723289] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14274.726641] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14274.731751] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11637s workers=4 idle: 15979 31824 8 [14274.744632] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14265s workers=2 idle: 654 [14288.058103] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [14288.062072] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14288.087733] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14288.092149] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14288.095894] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14288.098733] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14288.105056] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14288.107720] RAX: 0000000000005f03 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14288.110481] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14288.113089] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14288.115696] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14288.118510] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14288.120659] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14288.123670] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14288.125653] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14288.127679] Call Trace: [14288.128788] [14288.129661] ? show_regs.cold.9+0x22/0x2f [14288.131142] ? watchdog_timer_fn.cold.6+0x62/0x10a [14288.133180] ? __hrtimer_run_queues+0x1c2/0x4d0 [14288.134970] ? watchdog+0x40/0x40 [14288.136420] ? hrtimer_interrupt+0x128/0x2c0 [14288.137736] ? smp_apic_timer_interrupt+0xa6/0x270 [14288.139737] ? apic_timer_interrupt+0xf/0x20 [14288.141177] [14288.141967] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14288.144182] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14288.146190] do_raw_spin_lock+0x177/0x190 [14288.147572] _raw_spin_lock+0x21/0x30 [14288.148938] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14288.151469] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14288.153657] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14288.155715] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14288.157926] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14288.159956] ? woken_wake_function+0x30/0x30 [14288.161557] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14288.164184] kthread+0x1d1/0x200 [14288.165782] ? set_kthread_struct+0x70/0x70 [14288.167238] ret_from_fork+0x1f/0x30 [14296.055340] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [14296.060945] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [14296.061857] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14296.061937] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14296.061945] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14296.061948] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14296.061963] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14296.061967] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14296.061972] RAX: 000000000000572d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14296.061974] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14296.061977] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14296.061979] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14296.061982] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14296.061985] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14296.061988] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14296.061990] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14296.061998] Call Trace: [14296.062004] [14296.062010] ? show_regs.cold.9+0x22/0x2f [14296.062020] ? watchdog_timer_fn.cold.6+0x62/0x10a [14296.062028] ? __hrtimer_run_queues+0x1c2/0x4d0 [14296.062033] ? watchdog+0x40/0x40 [14296.062038] ? hrtimer_interrupt+0x128/0x2c0 [14296.062043] ? smp_apic_timer_interrupt+0xa6/0x270 [14296.062051] ? apic_timer_interrupt+0xf/0x20 [14296.062056] [14296.062059] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14296.062067] do_raw_spin_lock+0x177/0x190 [14296.062072] _raw_spin_lock+0x21/0x30 [14296.063227] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14296.063499] ? do_raw_spin_unlock+0x75/0x190 [14296.063506] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14296.063704] ? do_wait_intr+0xf0/0xf0 [14296.063714] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14296.063909] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14296.066278] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14296.066530] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14296.066643] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14296.066679] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14296.066707] mdc_intent_lock+0x353/0x7e0 [mdc] [14296.066732] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14296.066811] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14296.076667] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14296.076750] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14296.076758] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14296.076761] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14296.076775] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14296.076780] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14296.077492] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14296.077495] RDX: 000000000000150c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14296.077498] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14296.077500] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14296.077502] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14296.077506] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14296.077509] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14296.077511] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14296.077530] Call Trace: [14296.077536] [14296.077542] ? show_regs.cold.9+0x22/0x2f [14296.077553] ? watchdog_timer_fn.cold.6+0x62/0x10a [14296.077559] ? __hrtimer_run_queues+0x1c2/0x4d0 [14296.077563] ? watchdog+0x40/0x40 [14296.077567] ? hrtimer_interrupt+0x128/0x2c0 [14296.077571] ? smp_apic_timer_interrupt+0xa6/0x270 [14296.077578] ? apic_timer_interrupt+0xf/0x20 [14296.077583] [14296.077586] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14296.077592] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14296.077597] do_raw_spin_lock+0x177/0x190 [14296.077601] _raw_spin_lock+0x21/0x30 [14296.077609] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14296.077902] obd_mod_free+0x46/0x140 [ptlrpc] [14296.140915] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14296.145334] mdc_commit_open+0x91/0xb0 [mdc] [14296.145465] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14296.145879] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14296.158511] lmv_intent_open+0x338/0x1150 [lmv] [14296.169683] after_reply+0xe2c/0x1780 [ptlrpc] [14296.171057] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14296.171787] ? __next_timer_interrupt+0x160/0x160 [14296.171799] ptlrpcd+0xa82/0xe40 [ptlrpc] [14296.172093] ? do_wait_intr+0xf0/0xf0 [14296.172109] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14296.172398] kthread+0x1d1/0x200 [14296.172405] ? set_kthread_struct+0x70/0x70 [14296.172409] ret_from_fork+0x1f/0x30 [14296.737856] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14296.754989] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14296.760062] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14296.770887] ll_intent_lock+0xc5/0xaa0 [lustre] [14296.777785] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14296.779777] ll_lookup_it+0xe0e/0x2270 [lustre] [14296.781460] ? do_raw_spin_unlock+0x75/0x190 [14296.782968] ? _raw_spin_unlock+0x12/0x30 [14296.784590] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14296.786669] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14296.788616] ll_atomic_open+0x36a/0xd70 [lustre] [14296.790578] lookup_open+0x356/0xb40 [14296.791669] do_last+0x313/0xfc0 [14296.792814] ? nd_jump_root+0xe5/0x160 [14296.794016] ? path_init+0x437/0x520 [14296.797797] path_openat+0xf7/0x500 [14296.799622] ? do_raw_spin_unlock+0x75/0x190 [14296.801048] do_filp_open+0x99/0x140 [14296.802341] ? getname_flags+0x6e/0x330 [14296.803635] ? __check_object_size+0xff/0x256 [14296.806782] ? do_raw_spin_unlock+0x75/0x190 [14296.808743] ? _raw_spin_unlock+0x12/0x30 [14296.810582] do_sys_openat2+0x2b4/0x410 [14296.812597] do_sys_open+0x73/0xa0 [14296.815780] __x64_sys_openat+0x24/0x30 [14296.817428] do_syscall_64+0xc1/0x440 [14296.819595] entry_SYSCALL_64_after_hwframe+0x49/0xae [14296.822675] RIP: 0033:0x7f53425a5332 [14296.825325] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14296.832604] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14296.835193] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14296.838350] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14296.840763] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14296.844054] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14296.846754] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14300.052209] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [14300.058077] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14300.083859] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14300.091962] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14300.098785] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14300.107449] RIP: 0010:native_safe_halt+0x2e/0x50 [14300.117702] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14300.125575] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14300.134363] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14300.142753] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [14300.148876] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14300.151346] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14300.154577] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14300.157835] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14300.160949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14300.163765] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14300.167295] Call Trace: [14300.168904] [14300.170162] ? show_regs.cold.9+0x22/0x2f [14300.171885] ? watchdog_timer_fn.cold.6+0x62/0x10a [14300.174353] ? __hrtimer_run_queues+0x1c2/0x4d0 [14300.175931] ? watchdog+0x40/0x40 [14300.177512] ? hrtimer_interrupt+0x128/0x2c0 [14300.179843] ? smp_apic_timer_interrupt+0xa6/0x270 [14300.181850] ? apic_timer_interrupt+0xf/0x20 [14300.183823] [14300.184873] ? native_safe_halt+0x2e/0x50 [14300.186673] kvm_wait+0xcd/0xe0 [14300.187902] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14300.189977] ? newidle_balance.isra.93+0x207/0x580 [14300.192268] do_raw_spin_lock+0x177/0x190 [14300.193696] _raw_spin_lock+0x21/0x30 [14300.194974] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14300.197442] process_one_work+0x2c8/0x700 [14300.199269] worker_thread+0x296/0x6e0 [14300.200740] ? rescuer_thread+0x570/0x570 [14300.202427] kthread+0x1d1/0x200 [14300.203772] ? set_kthread_struct+0x70/0x70 [14300.205607] ret_from_fork+0x1f/0x30 [14305.252033] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11668s! [14305.264272] Showing busy workqueues and worker pools: [14305.268799] workqueue events: flags=0x0 [14305.272393] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14305.279612] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14305.288342] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14305.294847] pending: cpu_latency_work [ptlrpc] [14305.299776] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14305.310410] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14305.325543] workqueue events_long: flags=0x0 [14305.329352] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14305.335864] pending: lu_objects_flush [obdclass] [14305.343226] workqueue events_unbound: flags=0x2 [14305.348659] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14305.350445] pending: flush_memcg_stats_dwork, idle_cull_fn [14305.354781] workqueue events_power_efficient: flags=0x80 [14305.357679] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14305.361875] pending: check_lifetime, do_cache_clean [14305.364778] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14305.368564] pending: neigh_periodic_work, neigh_periodic_work [14305.372944] workqueue mm_percpu_wq: flags=0x8 [14305.376294] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14305.379221] pending: vmstat_update, lru_add_drain_per_cpu [14305.382246] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14305.386983] pending: vmstat_update, lru_add_drain_per_cpu [14305.390936] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14305.394167] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14305.396673] workqueue xprtiod: flags=0xa [14305.397881] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14305.400048] pending: xprt_autoclose, xs_error_handle [14305.401603] workqueue ipv6_addrconf: flags=0x40008 [14305.403782] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14305.406726] pending: addrconf_verify_work [14305.408839] workqueue knbd0-recv: flags=0x1a [14305.411103] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14305.413903] in-flight: 651:recv_work [14305.415586] workqueue ptlrpc_pinger: flags=0x2 [14305.417722] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14305.420245] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14305.423223] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11668s workers=4 idle: 15979 31824 8 [14305.426594] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14295s workers=2 idle: 654 [14316.061147] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [14316.063665] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14316.093264] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14316.098260] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14316.101752] RIP: 0010:native_safe_halt+0x2e/0x50 [14316.103872] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14316.111321] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14316.114754] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14316.118876] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14316.122850] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14316.127684] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14316.131570] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14316.142867] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14316.147225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14316.150302] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14316.154725] Call Trace: [14316.155962] [14316.158431] ? show_regs.cold.9+0x22/0x2f [14316.161168] ? watchdog_timer_fn.cold.6+0x62/0x10a [14316.163240] ? __hrtimer_run_queues+0x1c2/0x4d0 [14316.166711] ? watchdog+0x40/0x40 [14316.168260] ? hrtimer_interrupt+0x128/0x2c0 [14316.170142] ? smp_apic_timer_interrupt+0xa6/0x270 [14316.171674] ? apic_timer_interrupt+0xf/0x20 [14316.173417] [14316.174229] ? native_safe_halt+0x2e/0x50 [14316.175943] kvm_wait+0xcd/0xe0 [14316.177028] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14316.178877] do_raw_spin_lock+0x177/0x190 [14316.180335] _raw_spin_lock+0x21/0x30 [14316.181492] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14316.183729] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14316.185625] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14316.187691] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14316.189678] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14316.192044] ? woken_wake_function+0x30/0x30 [14316.193583] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14316.195938] kthread+0x1d1/0x200 [14316.197112] ? set_kthread_struct+0x70/0x70 [14316.198509] ret_from_fork+0x1f/0x30 [14324.056001] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [14324.059795] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [14324.074079] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14324.074153] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14324.074161] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14324.074163] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14324.074177] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14324.074182] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14324.074187] RAX: 0000000000004ba4 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14324.074189] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14324.074192] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14324.074194] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14324.074197] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14324.075162] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14324.075166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14324.075168] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14324.075177] Call Trace: [14324.075183] [14324.075189] ? show_regs.cold.9+0x22/0x2f [14324.077333] ? watchdog_timer_fn.cold.6+0x62/0x10a [14324.077342] ? __hrtimer_run_queues+0x1c2/0x4d0 [14324.077346] ? watchdog+0x40/0x40 [14324.077351] ? hrtimer_interrupt+0x128/0x2c0 [14324.077355] ? smp_apic_timer_interrupt+0xa6/0x270 [14324.077363] ? apic_timer_interrupt+0xf/0x20 [14324.077367] [14324.077372] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14324.077380] do_raw_spin_lock+0x177/0x190 [14324.077386] _raw_spin_lock+0x21/0x30 [14324.077391] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14324.077673] ? do_raw_spin_unlock+0x75/0x190 [14324.077679] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14324.080131] ? do_wait_intr+0xf0/0xf0 [14324.080142] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14324.081525] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14324.083916] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14324.103011] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14324.103105] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14324.103113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14324.103116] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14324.103133] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14324.103138] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14324.103143] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14324.103145] RDX: 000000000000623d RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14324.103148] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14324.103150] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14324.103153] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14324.103157] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14324.103159] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14324.103161] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14324.103170] Call Trace: [14324.103176] [14324.103181] ? show_regs.cold.9+0x22/0x2f [14324.103191] ? watchdog_timer_fn.cold.6+0x62/0x10a [14324.103197] ? __hrtimer_run_queues+0x1c2/0x4d0 [14324.103202] ? watchdog+0x40/0x40 [14324.103206] ? hrtimer_interrupt+0x128/0x2c0 [14324.103210] ? smp_apic_timer_interrupt+0xa6/0x270 [14324.103218] ? apic_timer_interrupt+0xf/0x20 [14324.103222] [14324.103225] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14324.103230] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14324.103236] do_raw_spin_lock+0x177/0x190 [14324.103240] _raw_spin_lock+0x21/0x30 [14324.103247] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14324.103510] obd_mod_free+0x46/0x140 [ptlrpc] [14324.165021] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14324.165569] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14324.165612] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14324.165765] mdc_intent_lock+0x353/0x7e0 [mdc] [14324.165796] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14324.165886] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14324.170734] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14324.170772] lmv_intent_open+0x338/0x1150 [lmv] [14324.170805] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14324.170920] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14324.171003] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14324.171026] ll_intent_lock+0xc5/0xaa0 [lustre] [14324.171401] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14324.171486] ll_lookup_it+0xe0e/0x2270 [lustre] [14324.171564] ? do_raw_spin_unlock+0x75/0x190 [14324.171572] ? _raw_spin_unlock+0x12/0x30 [14324.171579] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14324.171605] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14324.171643] ll_atomic_open+0x36a/0xd70 [lustre] [14324.171738] lookup_open+0x356/0xb40 [14324.171747] do_last+0x313/0xfc0 [14324.171752] ? nd_jump_root+0xe5/0x160 [14324.171757] ? path_init+0x437/0x520 [14324.171762] path_openat+0xf7/0x500 [14324.171767] ? do_raw_spin_unlock+0x75/0x190 [14324.171772] do_filp_open+0x99/0x140 [14324.171778] ? getname_flags+0x6e/0x330 [14324.171782] ? __check_object_size+0xff/0x256 [14324.171789] ? do_raw_spin_unlock+0x75/0x190 [14324.171793] ? _raw_spin_unlock+0x12/0x30 [14324.171798] do_sys_openat2+0x2b4/0x410 [14324.171802] do_sys_open+0x73/0xa0 [14324.171807] __x64_sys_openat+0x24/0x30 [14324.171810] do_syscall_64+0xc1/0x440 [14324.171818] entry_SYSCALL_64_after_hwframe+0x49/0xae [14324.171825] RIP: 0033:0x7f53425a5332 [14324.171832] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14324.171836] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14324.171842] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14324.171844] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14324.171847] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14324.171848] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14324.171850] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14325.049482] mdc_commit_open+0x91/0xb0 [mdc] [14325.059392] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14325.074595] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14325.086942] after_reply+0xe2c/0x1780 [ptlrpc] [14325.088935] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14325.107487] ? __next_timer_interrupt+0x160/0x160 [14325.117182] ptlrpcd+0xa82/0xe40 [ptlrpc] [14325.122481] ? do_wait_intr+0xf0/0xf0 [14325.133505] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14325.141858] kthread+0x1d1/0x200 [14325.157258] ? set_kthread_struct+0x70/0x70 [14325.161284] ret_from_fork+0x1f/0x30 [14332.053505] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [14332.060238] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14332.127736] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14332.137367] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14332.143875] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14332.149894] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14332.153698] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14332.174144] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14332.184633] RAX: 0000000000007e28 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14332.192407] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14332.196434] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14332.203773] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14332.211388] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14332.218137] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14332.224002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14332.232091] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14332.244946] Call Trace: [14332.247757] [14332.250454] ? show_regs.cold.9+0x22/0x2f [14332.252028] ? watchdog_timer_fn.cold.6+0x62/0x10a [14332.256903] ? __hrtimer_run_queues+0x1c2/0x4d0 [14332.263780] ? watchdog+0x40/0x40 [14332.267988] ? hrtimer_interrupt+0x128/0x2c0 [14332.272788] ? smp_apic_timer_interrupt+0xa6/0x270 [14332.274560] ? apic_timer_interrupt+0xf/0x20 [14332.278866] [14332.280724] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14332.291040] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14332.292875] ? newidle_balance.isra.93+0x207/0x580 [14332.308087] do_raw_spin_lock+0x177/0x190 [14332.309374] _raw_spin_lock+0x21/0x30 [14332.321781] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14332.326956] process_one_work+0x2c8/0x700 [14332.332978] worker_thread+0x296/0x6e0 [14332.336801] ? rescuer_thread+0x570/0x570 [14332.339013] kthread+0x1d1/0x200 [14332.346369] ? set_kthread_struct+0x70/0x70 [14332.349640] ret_from_fork+0x1f/0x30 [14335.457084] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11698s! [14335.460870] Showing busy workqueues and worker pools: [14335.462323] workqueue events: flags=0x0 [14335.464239] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14335.478751] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14335.498288] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14335.509215] pending: cpu_latency_work [ptlrpc] [14335.514884] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14335.518865] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14335.527105] workqueue events_long: flags=0x0 [14335.533401] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14335.538859] pending: lu_objects_flush [obdclass] [14335.543719] workqueue events_unbound: flags=0x2 [14335.548072] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14335.550795] pending: flush_memcg_stats_dwork, idle_cull_fn [14335.557510] workqueue events_power_efficient: flags=0x80 [14335.564674] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14335.570050] pending: check_lifetime, do_cache_clean [14335.575880] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14335.578950] pending: neigh_periodic_work, neigh_periodic_work [14335.585932] workqueue mm_percpu_wq: flags=0x8 [14335.593396] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14335.601474] pending: vmstat_update, lru_add_drain_per_cpu [14335.605945] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14335.618350] pending: vmstat_update, lru_add_drain_per_cpu [14335.621395] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14335.624868] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14335.630969] workqueue xprtiod: flags=0xa [14335.634680] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14335.642713] pending: xprt_autoclose, xs_error_handle [14335.646077] workqueue ipv6_addrconf: flags=0x40008 [14335.649387] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14335.657587] pending: addrconf_verify_work [14335.661561] workqueue knbd0-recv: flags=0x1a [14335.666691] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14335.671990] in-flight: 651:recv_work [14335.673795] workqueue ptlrpc_pinger: flags=0x2 [14335.676078] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14335.678998] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14335.686151] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11698s workers=4 idle: 15979 31824 8 [14335.692624] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14326s workers=2 idle: 654 [14344.059588] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [14344.063998] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14344.088932] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14344.098783] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14344.106993] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14344.111248] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14344.126590] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14344.129709] RAX: 0000000000007d86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14344.142942] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14344.154722] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14344.157860] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14344.163077] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14344.167153] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14344.173752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14344.179974] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14344.183679] Call Trace: [14344.184947] [14344.186346] ? show_regs.cold.9+0x22/0x2f [14344.187994] ? watchdog_timer_fn.cold.6+0x62/0x10a [14344.192372] ? __hrtimer_run_queues+0x1c2/0x4d0 [14344.195133] ? watchdog+0x40/0x40 [14344.197537] ? hrtimer_interrupt+0x128/0x2c0 [14344.199036] ? smp_apic_timer_interrupt+0xa6/0x270 [14344.201895] ? apic_timer_interrupt+0xf/0x20 [14344.206966] [14344.208068] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14344.211624] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14344.215939] do_raw_spin_lock+0x177/0x190 [14344.219993] _raw_spin_lock+0x21/0x30 [14344.221733] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14344.228057] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14344.233193] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14344.236805] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14344.240977] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14344.246662] ? woken_wake_function+0x30/0x30 [14344.249121] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14344.252338] kthread+0x1d1/0x200 [14344.253854] ? set_kthread_struct+0x70/0x70 [14344.258688] ret_from_fork+0x1f/0x30 [14352.058255] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [14352.058916] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [14352.069874] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14352.069961] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14352.069969] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14352.069972] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14352.069987] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14352.069991] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14352.069995] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14352.069998] RDX: 0000000000000501 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14352.070001] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14352.070002] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14352.070005] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14352.070008] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14352.070011] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14352.070014] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14352.070023] Call Trace: [14352.070028] [14352.070034] ? show_regs.cold.9+0x22/0x2f [14352.070044] ? watchdog_timer_fn.cold.6+0x62/0x10a [14352.070050] ? __hrtimer_run_queues+0x1c2/0x4d0 [14352.070054] ? watchdog+0x40/0x40 [14352.070058] ? hrtimer_interrupt+0x128/0x2c0 [14352.070061] ? smp_apic_timer_interrupt+0xa6/0x270 [14352.070078] ? apic_timer_interrupt+0xf/0x20 [14352.070082] [14352.070085] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14352.070090] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14352.070095] do_raw_spin_lock+0x177/0x190 [14352.070099] _raw_spin_lock+0x21/0x30 [14352.070105] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14352.070580] obd_mod_free+0x46/0x140 [ptlrpc] [14352.070772] mdc_commit_open+0x91/0xb0 [mdc] [14352.070808] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14352.071068] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14352.075710] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14352.075787] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14352.075795] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14352.075798] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14352.075812] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14352.075817] RSP: 0018:ffffbb5381503500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [14352.075822] RAX: 0000000000001f16 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14352.075824] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14352.075826] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14352.075828] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14352.075830] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14352.075833] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14352.075836] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14352.075838] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14352.075847] Call Trace: [14352.076001] [14352.076010] ? show_regs.cold.9+0x22/0x2f [14352.076023] ? watchdog_timer_fn.cold.6+0x62/0x10a [14352.076034] ? __hrtimer_run_queues+0x1c2/0x4d0 [14352.076040] ? watchdog+0x40/0x40 [14352.076046] ? hrtimer_interrupt+0x128/0x2c0 [14352.076051] ? smp_apic_timer_interrupt+0xa6/0x270 [14352.076063] ? apic_timer_interrupt+0xf/0x20 [14352.107778] after_reply+0xe2c/0x1780 [ptlrpc] [14352.108268] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14352.108485] ? __next_timer_interrupt+0x160/0x160 [14352.108496] ptlrpcd+0xa82/0xe40 [ptlrpc] [14352.108710] ? do_wait_intr+0xf0/0xf0 [14352.108720] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14352.110174] kthread+0x1d1/0x200 [14352.110183] ? set_kthread_struct+0x70/0x70 [14352.110187] ret_from_fork+0x1f/0x30 [14352.484623] [14352.488439] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14352.493519] do_raw_spin_lock+0x177/0x190 [14352.496208] _raw_spin_lock+0x21/0x30 [14352.499906] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14352.504481] ? do_raw_spin_unlock+0x75/0x190 [14352.506276] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14352.509875] ? do_wait_intr+0xf0/0xf0 [14352.511049] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14352.515518] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14352.520477] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14352.523115] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14352.526508] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14352.529728] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14352.537035] mdc_intent_lock+0x353/0x7e0 [mdc] [14352.538630] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14352.551708] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14352.553878] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14352.564455] lmv_intent_open+0x338/0x1150 [lmv] [14352.568926] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14352.573835] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14352.578511] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14352.580014] ll_intent_lock+0xc5/0xaa0 [lustre] [14352.599877] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14352.606105] ll_lookup_it+0xe0e/0x2270 [lustre] [14352.607765] ? do_raw_spin_unlock+0x75/0x190 [14352.616309] ? _raw_spin_unlock+0x12/0x30 [14352.619784] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14352.636302] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14352.641084] ll_atomic_open+0x36a/0xd70 [lustre] [14352.645300] lookup_open+0x356/0xb40 [14352.650498] do_last+0x313/0xfc0 [14352.654929] ? nd_jump_root+0xe5/0x160 [14352.658309] ? path_init+0x437/0x520 [14352.663846] path_openat+0xf7/0x500 [14352.665025] ? do_raw_spin_unlock+0x75/0x190 [14352.670462] do_filp_open+0x99/0x140 [14352.675087] ? getname_flags+0x6e/0x330 [14352.680370] ? __check_object_size+0xff/0x256 [14352.681918] ? do_raw_spin_unlock+0x75/0x190 [14352.688832] ? _raw_spin_unlock+0x12/0x30 [14352.692322] do_sys_openat2+0x2b4/0x410 [14352.695949] do_sys_open+0x73/0xa0 [14352.698557] __x64_sys_openat+0x24/0x30 [14352.709921] do_syscall_64+0xc1/0x440 [14352.712635] entry_SYSCALL_64_after_hwframe+0x49/0xae [14352.719249] RIP: 0033:0x7f53425a5332 [14352.726095] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14352.740391] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14352.763530] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14352.765994] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14352.788830] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14352.794772] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14352.802699] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14360.056210] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [14360.067851] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14360.096411] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14360.101707] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14360.105772] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14360.109308] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14360.111942] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14360.121589] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14360.125185] RAX: 0000000000007d92 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14360.127900] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14360.130888] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14360.134414] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14360.137509] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14360.140350] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14360.148176] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14360.153988] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14360.160448] Call Trace: [14360.161895] [14360.163130] ? show_regs.cold.9+0x22/0x2f [14360.164931] ? watchdog_timer_fn.cold.6+0x62/0x10a [14360.167732] ? __hrtimer_run_queues+0x1c2/0x4d0 [14360.170486] ? watchdog+0x40/0x40 [14360.172370] ? hrtimer_interrupt+0x128/0x2c0 [14360.174480] ? smp_apic_timer_interrupt+0xa6/0x270 [14360.177027] ? apic_timer_interrupt+0xf/0x20 [14360.179353] [14360.181881] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14360.187851] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14360.192957] ? newidle_balance.isra.93+0x207/0x580 [14360.196928] do_raw_spin_lock+0x177/0x190 [14360.200765] _raw_spin_lock+0x21/0x30 [14360.202150] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14360.207439] process_one_work+0x2c8/0x700 [14360.212478] worker_thread+0x296/0x6e0 [14360.214886] ? rescuer_thread+0x570/0x570 [14360.216908] kthread+0x1d1/0x200 [14360.220515] ? set_kthread_struct+0x70/0x70 [14360.224916] ret_from_fork+0x1f/0x30 [14366.175839] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11729s! [14366.184715] Showing busy workqueues and worker pools: [14366.193849] workqueue events: flags=0x0 [14366.198405] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14366.202207] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14366.211435] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14366.214691] pending: cpu_latency_work [ptlrpc] [14366.219240] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14366.228538] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14366.236038] workqueue events_long: flags=0x0 [14366.239158] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14366.247878] pending: lu_objects_flush [obdclass] [14366.253200] workqueue events_unbound: flags=0x2 [14366.256178] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14366.260190] pending: flush_memcg_stats_dwork, idle_cull_fn [14366.264813] workqueue events_power_efficient: flags=0x80 [14366.268290] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14366.272315] pending: check_lifetime, do_cache_clean [14366.275700] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14366.282740] pending: neigh_periodic_work, neigh_periodic_work [14366.290797] workqueue mm_percpu_wq: flags=0x8 [14366.294879] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14366.304404] pending: vmstat_update, lru_add_drain_per_cpu [14366.309813] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14366.312278] pending: vmstat_update, lru_add_drain_per_cpu [14366.314931] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14366.318742] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14366.324883] workqueue xprtiod: flags=0xa [14366.328367] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14366.331814] pending: xprt_autoclose, xs_error_handle [14366.334190] workqueue ipv6_addrconf: flags=0x40008 [14366.336049] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14366.338135] pending: addrconf_verify_work [14366.340389] workqueue knbd0-recv: flags=0x1a [14366.342423] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14366.345425] in-flight: 651:recv_work [14366.350277] workqueue ptlrpc_pinger: flags=0x2 [14366.353435] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14366.358433] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14366.364281] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11729s workers=4 idle: 15979 31824 8 [14366.369163] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14356s workers=2 idle: 654 [14372.058017] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [14372.061082] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14372.084873] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14372.090960] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14372.099435] RIP: 0010:native_safe_halt+0x2e/0x50 [14372.102980] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14372.116775] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14372.120655] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14372.127993] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14372.133417] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14372.137384] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14372.143638] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14372.149064] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14372.152861] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14372.155586] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14372.160118] Call Trace: [14372.161989] [14372.163018] ? show_regs.cold.9+0x22/0x2f [14372.165007] ? watchdog_timer_fn.cold.6+0x62/0x10a [14372.166952] ? __hrtimer_run_queues+0x1c2/0x4d0 [14372.169267] ? watchdog+0x40/0x40 [14372.170750] ? hrtimer_interrupt+0x128/0x2c0 [14372.172434] ? smp_apic_timer_interrupt+0xa6/0x270 [14372.174842] ? apic_timer_interrupt+0xf/0x20 [14372.176631] [14372.177552] ? native_safe_halt+0x2e/0x50 [14372.179570] kvm_wait+0xcd/0xe0 [14372.181335] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14372.183275] do_raw_spin_lock+0x177/0x190 [14372.184779] _raw_spin_lock+0x21/0x30 [14372.186479] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14372.190725] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14372.193327] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14372.196213] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14372.202066] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14372.207961] ? woken_wake_function+0x30/0x30 [14372.212105] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14372.217693] kthread+0x1d1/0x200 [14372.223102] ? set_kthread_struct+0x70/0x70 [14372.225109] ret_from_fork+0x1f/0x30 [14380.056370] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [14380.059179] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [14380.061501] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14380.061577] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14380.061585] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14380.061588] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14380.061601] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14380.061604] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14380.061609] RAX: 0000000000007089 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14380.061611] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14380.061613] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14380.061615] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14380.061617] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14380.061620] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14380.061623] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14380.061625] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14380.061634] Call Trace: [14380.061639] [14380.061645] ? show_regs.cold.9+0x22/0x2f [14380.061664] ? watchdog_timer_fn.cold.6+0x62/0x10a [14380.061671] ? __hrtimer_run_queues+0x1c2/0x4d0 [14380.061675] ? watchdog+0x40/0x40 [14380.061680] ? hrtimer_interrupt+0x128/0x2c0 [14380.061683] ? smp_apic_timer_interrupt+0xa6/0x270 [14380.061691] ? apic_timer_interrupt+0xf/0x20 [14380.061695] [14380.061698] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14380.061704] do_raw_spin_lock+0x177/0x190 [14380.070005] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14380.070251] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14380.070258] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14380.070261] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14380.070274] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14380.070279] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14380.070283] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14380.070286] RDX: 0000000000002cce RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14380.070288] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14380.070291] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14380.070293] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14380.070296] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14380.070299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14380.070302] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14380.070309] Call Trace: [14380.070314] [14380.070319] ? show_regs.cold.9+0x22/0x2f [14380.070328] ? watchdog_timer_fn.cold.6+0x62/0x10a [14380.070333] ? __hrtimer_run_queues+0x1c2/0x4d0 [14380.070337] ? watchdog+0x40/0x40 [14380.070342] ? hrtimer_interrupt+0x128/0x2c0 [14380.070346] ? smp_apic_timer_interrupt+0xa6/0x270 [14380.070354] ? apic_timer_interrupt+0xf/0x20 [14380.070359] [14380.070361] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14380.070366] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14380.070495] do_raw_spin_lock+0x177/0x190 [14380.070501] _raw_spin_lock+0x21/0x30 [14380.070507] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14380.070946] obd_mod_free+0x46/0x140 [ptlrpc] [14380.071298] mdc_commit_open+0x91/0xb0 [mdc] [14380.071346] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14380.071834] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14380.072076] after_reply+0xe2c/0x1780 [ptlrpc] [14380.072297] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14380.072777] ? __next_timer_interrupt+0x160/0x160 [14380.072788] ptlrpcd+0xa82/0xe40 [ptlrpc] [14380.073023] ? do_wait_intr+0xf0/0xf0 [14380.073032] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14380.073248] kthread+0x1d1/0x200 [14380.073254] ? set_kthread_struct+0x70/0x70 [14380.073258] ret_from_fork+0x1f/0x30 [14380.533712] _raw_spin_lock+0x21/0x30 [14380.535016] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14380.543314] ? do_raw_spin_unlock+0x75/0x190 [14380.551071] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14380.555862] ? do_wait_intr+0xf0/0xf0 [14380.562399] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14380.566095] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14380.569514] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14380.577815] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14380.580061] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14380.589579] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14380.602055] mdc_intent_lock+0x353/0x7e0 [mdc] [14380.610311] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14380.622513] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14380.638466] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14380.642218] lmv_intent_open+0x338/0x1150 [lmv] [14380.649321] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14380.652449] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14380.658039] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14380.659519] ll_intent_lock+0xc5/0xaa0 [lustre] [14380.671215] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14380.676228] ll_lookup_it+0xe0e/0x2270 [lustre] [14380.680291] ? do_raw_spin_unlock+0x75/0x190 [14380.691034] ? _raw_spin_unlock+0x12/0x30 [14380.695156] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14380.706845] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14380.708572] ll_atomic_open+0x36a/0xd70 [lustre] [14380.715529] lookup_open+0x356/0xb40 [14380.720136] do_last+0x313/0xfc0 [14380.724391] ? nd_jump_root+0xe5/0x160 [14380.727069] ? path_init+0x437/0x520 [14380.728294] path_openat+0xf7/0x500 [14380.729469] ? do_raw_spin_unlock+0x75/0x190 [14380.731951] do_filp_open+0x99/0x140 [14380.733186] ? getname_flags+0x6e/0x330 [14380.735717] ? __check_object_size+0xff/0x256 [14380.738188] ? do_raw_spin_unlock+0x75/0x190 [14380.740449] ? _raw_spin_unlock+0x12/0x30 [14380.742857] do_sys_openat2+0x2b4/0x410 [14380.747186] do_sys_open+0x73/0xa0 [14380.749361] __x64_sys_openat+0x24/0x30 [14380.759897] do_syscall_64+0xc1/0x440 [14380.771375] entry_SYSCALL_64_after_hwframe+0x49/0xae [14380.778468] RIP: 0033:0x7f53425a5332 [14380.785310] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14380.803856] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14380.818258] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14380.831357] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14380.837056] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14380.848473] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14380.850957] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14392.052025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [14392.055167] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14392.109976] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14392.125264] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14392.134301] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14392.147780] RIP: 0010:native_safe_halt+0x2e/0x50 [14392.151983] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14392.171211] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14392.182480] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14392.188047] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [14392.193300] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14392.197305] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14392.200683] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14392.204192] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14392.208528] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14392.212569] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14392.216145] Call Trace: [14392.217644] [14392.218982] ? show_regs.cold.9+0x22/0x2f [14392.220801] ? watchdog_timer_fn.cold.6+0x62/0x10a [14392.224722] ? __hrtimer_run_queues+0x1c2/0x4d0 [14392.228226] ? watchdog+0x40/0x40 [14392.230824] ? hrtimer_interrupt+0x128/0x2c0 [14392.235170] ? smp_apic_timer_interrupt+0xa6/0x270 [14392.238696] ? apic_timer_interrupt+0xf/0x20 [14392.242824] [14392.243944] ? native_safe_halt+0x2e/0x50 [14392.247278] kvm_wait+0xcd/0xe0 [14392.249197] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14392.251790] ? newidle_balance.isra.93+0x207/0x580 [14392.254379] do_raw_spin_lock+0x177/0x190 [14392.257167] _raw_spin_lock+0x21/0x30 [14392.259343] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14392.264827] process_one_work+0x2c8/0x700 [14392.267214] worker_thread+0x296/0x6e0 [14392.269576] ? rescuer_thread+0x570/0x570 [14392.271868] kthread+0x1d1/0x200 [14392.275368] ? set_kthread_struct+0x70/0x70 [14392.282127] ret_from_fork+0x1f/0x30 [14394.727063] rcu: INFO: rcu_sched self-detected stall on CPU [14394.731863] rcu: 2-....: (6826789 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [14394.734515] (t=12300301 jiffies g=274185 q=67023) [14394.735664] rcu: rcu_sched kthread starved for 11758667 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [14394.739143] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14394.741619] rcu: RCU grace-period kthread stack dump: [14394.743076] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14394.745710] Call Trace: [14394.746306] __schedule+0x351/0xcb0 [14394.747290] schedule+0xc0/0x180 [14394.748364] schedule_timeout+0xb4/0x190 [14394.749541] ? do_raw_spin_unlock+0x75/0x190 [14394.750685] ? __next_timer_interrupt+0x160/0x160 [14394.752049] ? prepare_to_swait_event+0xf3/0x140 [14394.753311] rcu_gp_kthread+0x8dd/0x15b0 [14394.754340] ? force_qs_rnp+0x2c0/0x2c0 [14394.755447] kthread+0x1d1/0x200 [14394.756347] ? set_kthread_struct+0x70/0x70 [14394.757513] ret_from_fork+0x1f/0x30 [14394.758699] rcu: Stack dump where RCU GP kthread last ran: [14394.760216] NMI backtrace for cpu 2 [14394.761511] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14394.765087] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14394.767525] Call Trace: [14394.768195] [14394.768695] ? dump_stack+0xbb/0x10e [14394.769654] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14394.771040] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14394.772181] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14394.784148] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14394.785357] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14394.787011] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [14394.790009] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [14394.792356] ? update_process_times+0x6d/0xc0 [14394.794198] ? tick_sched_handle.isra.18+0x27/0x90 [14394.795756] ? tick_sched_timer+0x53/0xb0 [14394.796750] ? __hrtimer_run_queues+0x1c2/0x4d0 [14394.798023] ? tick_sched_handle.isra.18+0x90/0x90 [14394.799298] ? hrtimer_interrupt+0x128/0x2c0 [14394.800517] ? smp_apic_timer_interrupt+0xa6/0x270 [14394.801864] ? apic_timer_interrupt+0xf/0x20 [14394.803010] [14394.803530] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14394.806192] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14394.807824] ? do_raw_spin_lock+0x177/0x190 [14394.809183] ? _raw_spin_lock+0x21/0x30 [14394.810275] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14394.812143] ? obd_mod_free+0x46/0x140 [ptlrpc] [14394.813746] ? mdc_commit_open+0x91/0xb0 [mdc] [14394.815078] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14394.816832] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14394.818674] ? after_reply+0xe2c/0x1780 [ptlrpc] [14394.820262] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14394.822260] ? __next_timer_interrupt+0x160/0x160 [14394.823672] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14394.825125] ? do_wait_intr+0xf0/0xf0 [14394.826498] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14394.828223] ? kthread+0x1d1/0x200 [14394.829246] ? set_kthread_struct+0x70/0x70 [14394.830361] ? ret_from_fork+0x1f/0x30 [14394.831514] Sending NMI from CPU 2 to CPUs 1: [14394.832752] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [14394.834966] NMI backtrace for cpu 2 [14394.835843] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14394.843178] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14394.847048] Call Trace: [14394.847711] [14394.849300] ? dump_stack+0xbb/0x10e [14394.850556] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14394.852548] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14394.855296] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14394.856701] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14394.857789] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14394.859707] ? rcu_dump_cpu_stacks+0x11b/0x171 [14394.861852] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14394.863240] ? update_process_times+0x6d/0xc0 [14394.864355] ? tick_sched_handle.isra.18+0x27/0x90 [14394.865471] ? tick_sched_timer+0x53/0xb0 [14394.866259] ? __hrtimer_run_queues+0x1c2/0x4d0 [14394.868321] ? tick_sched_handle.isra.18+0x90/0x90 [14394.869701] ? hrtimer_interrupt+0x128/0x2c0 [14394.870746] ? smp_apic_timer_interrupt+0xa6/0x270 [14394.872231] ? apic_timer_interrupt+0xf/0x20 [14394.873454] [14394.873965] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14394.875218] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14394.876278] ? do_raw_spin_lock+0x177/0x190 [14394.877246] ? _raw_spin_lock+0x21/0x30 [14394.878303] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14394.880016] ? obd_mod_free+0x46/0x140 [ptlrpc] [14394.882068] ? mdc_commit_open+0x91/0xb0 [mdc] [14394.883027] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14394.884545] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14394.886305] ? after_reply+0xe2c/0x1780 [ptlrpc] [14394.887954] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14394.889750] ? __next_timer_interrupt+0x160/0x160 [14394.890887] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14394.892848] ? do_wait_intr+0xf0/0xf0 [14394.893745] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14394.895460] ? kthread+0x1d1/0x200 [14394.896293] ? set_kthread_struct+0x70/0x70 [14394.897269] ? ret_from_fork+0x1f/0x30 [14396.394649] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11759s! [14396.398645] Showing busy workqueues and worker pools: [14396.404462] workqueue events: flags=0x0 [14396.410471] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14396.416477] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14396.425154] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14396.433307] pending: cpu_latency_work [ptlrpc] [14396.441751] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14396.449107] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14396.465621] workqueue events_long: flags=0x0 [14396.470296] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14396.472925] pending: lu_objects_flush [obdclass] [14396.475640] workqueue events_unbound: flags=0x2 [14396.477247] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14396.479556] pending: flush_memcg_stats_dwork, idle_cull_fn [14396.482607] workqueue events_power_efficient: flags=0x80 [14396.484719] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14396.487629] pending: check_lifetime, do_cache_clean [14396.490023] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14396.494037] pending: neigh_periodic_work, neigh_periodic_work [14396.496055] workqueue mm_percpu_wq: flags=0x8 [14396.498215] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14396.502764] pending: vmstat_update, lru_add_drain_per_cpu [14396.508255] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14396.511121] pending: vmstat_update, lru_add_drain_per_cpu [14396.513663] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14396.516564] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14396.519640] workqueue xprtiod: flags=0xa [14396.521299] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14396.524601] pending: xprt_autoclose, xs_error_handle [14396.527097] workqueue ipv6_addrconf: flags=0x40008 [14396.529351] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14396.532225] pending: addrconf_verify_work [14396.534221] workqueue knbd0-recv: flags=0x1a [14396.536312] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14396.539206] in-flight: 651:recv_work [14396.541185] workqueue ptlrpc_pinger: flags=0x2 [14396.543309] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14396.546746] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14396.549662] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11759s workers=4 idle: 15979 31824 8 [14396.553282] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14387s workers=2 idle: 654 [14400.058898] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [14400.078724] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14400.144244] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14400.158114] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14400.168060] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14400.172103] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14400.188700] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14400.198859] RAX: 0000000000005ff1 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14400.204885] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14400.220896] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14400.230999] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14400.240777] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14400.247852] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14400.264565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14400.266616] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14400.289985] Call Trace: [14400.290930] [14400.299699] ? show_regs.cold.9+0x22/0x2f [14400.303935] ? watchdog_timer_fn.cold.6+0x62/0x10a [14400.307928] ? __hrtimer_run_queues+0x1c2/0x4d0 [14400.311036] ? watchdog+0x40/0x40 [14400.314669] ? hrtimer_interrupt+0x128/0x2c0 [14400.318665] ? smp_apic_timer_interrupt+0xa6/0x270 [14400.323146] ? apic_timer_interrupt+0xf/0x20 [14400.325770] [14400.327587] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14400.331506] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14400.336512] do_raw_spin_lock+0x177/0x190 [14400.340933] _raw_spin_lock+0x21/0x30 [14400.344840] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14400.352187] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14400.356130] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14400.362921] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14400.374078] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14400.382594] ? woken_wake_function+0x30/0x30 [14400.386689] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14400.390580] kthread+0x1d1/0x200 [14400.393165] ? set_kthread_struct+0x70/0x70 [14400.399614] ret_from_fork+0x1f/0x30 [14420.055917] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [14420.062091] Modules linked in: lustre(O) [14420.062101] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [14420.062112] Modules linked in: [14420.064914] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14420.064998] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14420.065005] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14420.065008] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14420.065024] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14420.065028] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14420.065033] RAX: 0000000000006386 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14420.065035] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14420.065038] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14420.065040] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14420.065042] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14420.065045] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14420.065048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14420.065050] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14420.065059] Call Trace: [14420.065065] [14420.065071] ? show_regs.cold.9+0x22/0x2f [14420.065081] ? watchdog_timer_fn.cold.6+0x62/0x10a [14420.065086] ? __hrtimer_run_queues+0x1c2/0x4d0 [14420.065091] ? watchdog+0x40/0x40 [14420.065095] ? hrtimer_interrupt+0x128/0x2c0 [14420.065099] ? smp_apic_timer_interrupt+0xa6/0x270 [14420.065106] ? apic_timer_interrupt+0xf/0x20 [14420.065111] [14420.065114] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14420.065121] do_raw_spin_lock+0x177/0x190 [14420.065126] _raw_spin_lock+0x21/0x30 [14420.065131] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14420.065411] ? do_raw_spin_unlock+0x75/0x190 [14420.065417] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14420.065600] ? do_wait_intr+0xf0/0xf0 [14420.065608] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14420.065803] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14420.066030] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14420.066237] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14420.066360] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14420.066400] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14420.066430] mdc_intent_lock+0x353/0x7e0 [mdc] [14420.066456] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14420.073404] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14420.073486] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14420.073495] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14420.073498] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14420.073513] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14420.073518] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14420.073522] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14420.073525] RDX: 0000000000005009 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14420.073528] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14420.073530] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14420.073533] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14420.073536] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14420.073539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14420.073541] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14420.073550] Call Trace: [14420.073556] [14420.073562] ? show_regs.cold.9+0x22/0x2f [14420.073571] ? watchdog_timer_fn.cold.6+0x62/0x10a [14420.073577] ? __hrtimer_run_queues+0x1c2/0x4d0 [14420.073581] ? watchdog+0x40/0x40 [14420.073585] ? hrtimer_interrupt+0x128/0x2c0 [14420.073589] ? smp_apic_timer_interrupt+0xa6/0x270 [14420.073598] ? apic_timer_interrupt+0xf/0x20 [14420.073603] [14420.073606] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14420.073611] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14420.073615] do_raw_spin_lock+0x177/0x190 [14420.073619] _raw_spin_lock+0x21/0x30 [14420.073625] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14420.076543] obd_mod_free+0x46/0x140 [ptlrpc] [14420.076746] mdc_commit_open+0x91/0xb0 [mdc] [14420.076785] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14420.077278] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14420.077496] after_reply+0xe2c/0x1780 [ptlrpc] [14420.077699] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14420.079273] ? __next_timer_interrupt+0x160/0x160 [14420.079288] ptlrpcd+0xa82/0xe40 [ptlrpc] [14420.079542] ? do_wait_intr+0xf0/0xf0 [14420.079552] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14420.079755] kthread+0x1d1/0x200 [14420.079762] ? set_kthread_struct+0x70/0x70 [14420.079766] ret_from_fork+0x1f/0x30 [14420.620392] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14420.622335] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14420.635666] lmv_intent_open+0x338/0x1150 [lmv] [14420.642564] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14420.644676] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14420.653455] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14420.662637] ll_intent_lock+0xc5/0xaa0 [lustre] [14420.669626] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14420.681520] ll_lookup_it+0xe0e/0x2270 [lustre] [14420.689291] ? do_raw_spin_unlock+0x75/0x190 [14420.694980] ? _raw_spin_unlock+0x12/0x30 [14420.706539] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14420.708375] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14420.710235] ll_atomic_open+0x36a/0xd70 [lustre] [14420.712066] lookup_open+0x356/0xb40 [14420.713390] do_last+0x313/0xfc0 [14420.714594] ? nd_jump_root+0xe5/0x160 [14420.719924] ? path_init+0x437/0x520 [14420.721549] path_openat+0xf7/0x500 [14420.722778] ? do_raw_spin_unlock+0x75/0x190 [14420.724182] do_filp_open+0x99/0x140 [14420.725408] ? getname_flags+0x6e/0x330 [14420.726891] ? __check_object_size+0xff/0x256 [14420.728582] ? do_raw_spin_unlock+0x75/0x190 [14420.730231] ? _raw_spin_unlock+0x12/0x30 [14420.731600] do_sys_openat2+0x2b4/0x410 [14420.732994] do_sys_open+0x73/0xa0 [14420.734138] __x64_sys_openat+0x24/0x30 [14420.735475] do_syscall_64+0xc1/0x440 [14420.736862] entry_SYSCALL_64_after_hwframe+0x49/0xae [14420.738737] RIP: 0033:0x7f53425a5332 [14420.740360] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14420.758303] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14420.762806] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14420.765129] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14420.768535] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14420.773075] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14420.775849] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14424.054857] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [14424.061939] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14424.127666] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14424.135179] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14424.139752] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14424.152335] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14424.156190] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14424.183759] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14424.187215] RAX: 0000000000001e13 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14424.193784] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14424.195777] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14424.201132] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14424.217431] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14424.224539] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14424.233463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14424.235613] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14424.239634] Call Trace: [14424.243363] [14424.252286] ? show_regs.cold.9+0x22/0x2f [14424.258256] ? watchdog_timer_fn.cold.6+0x62/0x10a [14424.264279] ? __hrtimer_run_queues+0x1c2/0x4d0 [14424.270604] ? watchdog+0x40/0x40 [14424.271803] ? hrtimer_interrupt+0x128/0x2c0 [14424.276934] ? smp_apic_timer_interrupt+0xa6/0x270 [14424.284009] ? apic_timer_interrupt+0xf/0x20 [14424.294352] [14424.299659] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14424.301451] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14424.307385] ? newidle_balance.isra.93+0x207/0x580 [14424.310709] do_raw_spin_lock+0x177/0x190 [14424.312523] _raw_spin_lock+0x21/0x30 [14424.314540] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14424.318973] process_one_work+0x2c8/0x700 [14424.328087] worker_thread+0x296/0x6e0 [14424.331223] ? rescuer_thread+0x570/0x570 [14424.334044] kthread+0x1d1/0x200 [14424.336238] ? set_kthread_struct+0x70/0x70 [14424.338180] ret_from_fork+0x1f/0x30 [14426.592824] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11789s! [14426.598391] Showing busy workqueues and worker pools: [14426.600996] workqueue events: flags=0x0 [14426.602633] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14426.605894] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14426.611659] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14426.615595] pending: cpu_latency_work [ptlrpc] [14426.618428] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14426.621599] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14426.627151] workqueue events_long: flags=0x0 [14426.629161] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14426.632414] pending: lu_objects_flush [obdclass] [14426.635458] workqueue events_unbound: flags=0x2 [14426.638094] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14426.641591] pending: flush_memcg_stats_dwork, idle_cull_fn [14426.644863] workqueue events_power_efficient: flags=0x80 [14426.647495] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14426.651139] pending: check_lifetime, do_cache_clean [14426.653671] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14426.657556] pending: neigh_periodic_work, neigh_periodic_work [14426.660638] workqueue mm_percpu_wq: flags=0x8 [14426.663036] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14426.666330] pending: vmstat_update, lru_add_drain_per_cpu [14426.669706] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14426.673532] pending: vmstat_update, lru_add_drain_per_cpu [14426.677103] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14426.680950] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14426.684444] workqueue xprtiod: flags=0xa [14426.686430] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14426.690595] pending: xprt_autoclose, xs_error_handle [14426.693383] workqueue ipv6_addrconf: flags=0x40008 [14426.695595] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14426.698953] pending: addrconf_verify_work [14426.701516] workqueue knbd0-recv: flags=0x1a [14426.704095] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14426.707369] in-flight: 651:recv_work [14426.709470] workqueue ptlrpc_pinger: flags=0x2 [14426.711420] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14426.714483] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14426.717728] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11789s workers=4 idle: 15979 31824 8 [14426.721478] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14417s workers=2 idle: 654 [14428.058208] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [14428.062777] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14428.119510] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14428.129013] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14428.142270] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14428.153002] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14428.170898] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14428.180334] RAX: 0000000000001908 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14428.182638] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14428.193688] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14428.199704] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14428.203271] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14428.214903] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14428.217876] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14428.219943] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14428.222166] Call Trace: [14428.223339] [14428.223986] ? show_regs.cold.9+0x22/0x2f [14428.225342] ? watchdog_timer_fn.cold.6+0x62/0x10a [14428.230832] ? __hrtimer_run_queues+0x1c2/0x4d0 [14428.232479] ? watchdog+0x40/0x40 [14428.239763] ? hrtimer_interrupt+0x128/0x2c0 [14428.242919] ? smp_apic_timer_interrupt+0xa6/0x270 [14428.245784] ? apic_timer_interrupt+0xf/0x20 [14428.249363] [14428.256789] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14428.264004] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14428.270671] do_raw_spin_lock+0x177/0x190 [14428.272654] _raw_spin_lock+0x21/0x30 [14428.274735] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14428.278036] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14428.286023] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14428.290219] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14428.298858] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14428.305344] ? woken_wake_function+0x30/0x30 [14428.309406] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14428.312869] kthread+0x1d1/0x200 [14428.322969] ? set_kthread_struct+0x70/0x70 [14428.327245] ret_from_fork+0x1f/0x30 [14448.056837] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [14448.057777] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [14448.074592] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14448.074672] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14448.074678] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14448.074683] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14448.074698] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14448.074702] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14448.074706] RAX: 0000000000005b0e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14448.074708] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14448.074711] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14448.074712] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14448.074715] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14448.074718] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14448.074721] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14448.074723] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14448.074731] Call Trace: [14448.074737] [14448.074743] ? show_regs.cold.9+0x22/0x2f [14448.074753] ? watchdog_timer_fn.cold.6+0x62/0x10a [14448.074759] ? __hrtimer_run_queues+0x1c2/0x4d0 [14448.074763] ? watchdog+0x40/0x40 [14448.074766] ? hrtimer_interrupt+0x128/0x2c0 [14448.074770] ? smp_apic_timer_interrupt+0xa6/0x270 [14448.074778] ? apic_timer_interrupt+0xf/0x20 [14448.074783] [14448.074786] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14448.074792] do_raw_spin_lock+0x177/0x190 [14448.074797] _raw_spin_lock+0x21/0x30 [14448.074802] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14448.076632] ? do_raw_spin_unlock+0x75/0x190 [14448.076644] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14448.089275] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14448.089356] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14448.089363] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14448.089366] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14448.089380] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14448.089385] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14448.089389] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14448.089392] RDX: 00000000000048f0 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14448.089394] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14448.089396] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14448.089398] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14448.089402] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14448.089405] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14448.089407] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14448.089415] Call Trace: [14448.089420] [14448.089427] ? show_regs.cold.9+0x22/0x2f [14448.089436] ? watchdog_timer_fn.cold.6+0x62/0x10a [14448.089443] ? __hrtimer_run_queues+0x1c2/0x4d0 [14448.089447] ? watchdog+0x40/0x40 [14448.089452] ? hrtimer_interrupt+0x128/0x2c0 [14448.089469] ? smp_apic_timer_interrupt+0xa6/0x270 [14448.089478] ? apic_timer_interrupt+0xf/0x20 [14448.089483] [14448.089486] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14448.089491] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14448.089496] do_raw_spin_lock+0x177/0x190 [14448.089501] _raw_spin_lock+0x21/0x30 [14448.089507] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14448.089768] obd_mod_free+0x46/0x140 [ptlrpc] [14448.089973] mdc_commit_open+0x91/0xb0 [mdc] [14448.090010] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14448.090226] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14448.090428] after_reply+0xe2c/0x1780 [ptlrpc] [14448.148526] ? do_wait_intr+0xf0/0xf0 [14448.148552] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14448.148837] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14448.149051] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14448.149287] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14448.152131] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14448.154641] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14448.154676] mdc_intent_lock+0x353/0x7e0 [mdc] [14448.154703] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14448.155424] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14448.175702] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14448.176716] ? __next_timer_interrupt+0x160/0x160 [14448.176730] ptlrpcd+0xa82/0xe40 [ptlrpc] [14448.176961] ? do_wait_intr+0xf0/0xf0 [14448.176985] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14448.182671] kthread+0x1d1/0x200 [14448.182681] ? set_kthread_struct+0x70/0x70 [14448.182686] ret_from_fork+0x1f/0x30 [14448.747144] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14448.752909] lmv_intent_open+0x338/0x1150 [lmv] [14448.764359] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14448.771436] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14448.777929] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14448.783925] ll_intent_lock+0xc5/0xaa0 [lustre] [14448.793231] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14448.799252] ll_lookup_it+0xe0e/0x2270 [lustre] [14448.806483] ? do_raw_spin_unlock+0x75/0x190 [14448.810903] ? _raw_spin_unlock+0x12/0x30 [14448.815918] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14448.823399] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14448.830391] ll_atomic_open+0x36a/0xd70 [lustre] [14448.840436] lookup_open+0x356/0xb40 [14448.842714] do_last+0x313/0xfc0 [14448.848868] ? nd_jump_root+0xe5/0x160 [14448.854906] ? path_init+0x437/0x520 [14448.862252] path_openat+0xf7/0x500 [14448.866153] ? do_raw_spin_unlock+0x75/0x190 [14448.878197] do_filp_open+0x99/0x140 [14448.881253] ? getname_flags+0x6e/0x330 [14448.887282] ? __check_object_size+0xff/0x256 [14448.888867] ? do_raw_spin_unlock+0x75/0x190 [14448.898756] ? _raw_spin_unlock+0x12/0x30 [14448.903198] do_sys_openat2+0x2b4/0x410 [14448.908001] do_sys_open+0x73/0xa0 [14448.909338] __x64_sys_openat+0x24/0x30 [14448.919374] do_syscall_64+0xc1/0x440 [14448.925211] entry_SYSCALL_64_after_hwframe+0x49/0xae [14448.930511] RIP: 0033:0x7f53425a5332 [14448.941894] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14448.974898] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14448.982979] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14448.992795] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14448.997627] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14449.007864] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14449.014713] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14452.052576] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [14452.062019] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14452.124152] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14452.129856] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14452.141066] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14452.151394] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14452.157066] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14452.183288] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14452.185957] RAX: 0000000000007d97 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14452.197998] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14452.207064] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14452.218344] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14452.229997] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14452.237789] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14452.245804] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14452.252017] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14452.255603] Call Trace: [14452.257165] [14452.258279] ? show_regs.cold.9+0x22/0x2f [14452.260802] ? watchdog_timer_fn.cold.6+0x62/0x10a [14452.263130] ? __hrtimer_run_queues+0x1c2/0x4d0 [14452.265672] ? watchdog+0x40/0x40 [14452.267937] ? hrtimer_interrupt+0x128/0x2c0 [14452.271813] ? smp_apic_timer_interrupt+0xa6/0x270 [14452.275868] ? apic_timer_interrupt+0xf/0x20 [14452.279318] [14452.280289] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14452.282687] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14452.285143] ? newidle_balance.isra.93+0x207/0x580 [14452.287671] do_raw_spin_lock+0x177/0x190 [14452.290102] _raw_spin_lock+0x21/0x30 [14452.291780] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14452.294798] process_one_work+0x2c8/0x700 [14452.296967] worker_thread+0x296/0x6e0 [14452.301723] ? rescuer_thread+0x570/0x570 [14452.308267] kthread+0x1d1/0x200 [14452.310926] ? set_kthread_struct+0x70/0x70 [14452.316342] ret_from_fork+0x1f/0x30 [14456.058036] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [14456.063103] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14456.092576] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14456.097864] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14456.102297] RIP: 0010:native_safe_halt+0x2e/0x50 [14456.104382] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14456.113377] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14456.116903] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14456.126797] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14456.133925] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14456.139060] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14456.142424] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14456.145767] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14456.149765] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14456.152659] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14456.156684] Call Trace: [14456.157415] [14456.158066] ? show_regs.cold.9+0x22/0x2f [14456.160193] ? watchdog_timer_fn.cold.6+0x62/0x10a [14456.163512] ? __hrtimer_run_queues+0x1c2/0x4d0 [14456.165199] ? watchdog+0x40/0x40 [14456.166705] ? hrtimer_interrupt+0x128/0x2c0 [14456.168807] ? smp_apic_timer_interrupt+0xa6/0x270 [14456.170717] ? apic_timer_interrupt+0xf/0x20 [14456.172207] [14456.172939] ? native_safe_halt+0x2e/0x50 [14456.177364] kvm_wait+0xcd/0xe0 [14456.178784] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14456.186167] do_raw_spin_lock+0x177/0x190 [14456.189724] _raw_spin_lock+0x21/0x30 [14456.194411] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14456.200625] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14456.202568] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14456.206765] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14456.211002] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14456.215691] ? woken_wake_function+0x30/0x30 [14456.220295] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14456.229862] kthread+0x1d1/0x200 [14456.234036] ? set_kthread_struct+0x70/0x70 [14456.235927] ret_from_fork+0x1f/0x30 [14456.799120] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11819s! [14456.802823] Showing busy workqueues and worker pools: [14456.805746] workqueue events: flags=0x0 [14456.808195] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14456.811900] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14456.816418] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14456.819317] pending: cpu_latency_work [ptlrpc] [14456.821647] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14456.824830] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14456.829101] workqueue events_long: flags=0x0 [14456.830557] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14456.833506] pending: lu_objects_flush [obdclass] [14456.837218] workqueue events_unbound: flags=0x2 [14456.839433] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14456.842290] pending: flush_memcg_stats_dwork, idle_cull_fn [14456.845244] workqueue events_power_efficient: flags=0x80 [14456.847766] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14456.850874] pending: check_lifetime, do_cache_clean [14456.853464] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14456.856281] pending: neigh_periodic_work, neigh_periodic_work [14456.860176] workqueue mm_percpu_wq: flags=0x8 [14456.865867] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14456.870812] pending: vmstat_update, lru_add_drain_per_cpu [14456.874863] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14456.879455] pending: vmstat_update, lru_add_drain_per_cpu [14456.886914] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14456.894296] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14456.899111] workqueue xprtiod: flags=0xa [14456.901645] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14456.905855] pending: xprt_autoclose, xs_error_handle [14456.907177] workqueue ipv6_addrconf: flags=0x40008 [14456.908801] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14456.912089] pending: addrconf_verify_work [14456.913371] workqueue knbd0-recv: flags=0x1a [14456.915456] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14456.920753] in-flight: 651:recv_work [14456.924878] workqueue ptlrpc_pinger: flags=0x2 [14456.928155] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14456.932423] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14456.938215] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11819s workers=4 idle: 15979 31824 8 [14456.944244] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14447s workers=2 idle: 654 [14476.056909] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [14476.060294] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [14476.062057] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14476.062132] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14476.062141] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14476.062143] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14476.062157] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14476.062161] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14476.062166] RAX: 0000000000003457 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14476.062168] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14476.062171] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14476.062173] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14476.062175] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14476.062178] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14476.062181] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14476.062184] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14476.065044] Call Trace: [14476.065059] [14476.065066] ? show_regs.cold.9+0x22/0x2f [14476.065081] ? watchdog_timer_fn.cold.6+0x62/0x10a [14476.065087] ? __hrtimer_run_queues+0x1c2/0x4d0 [14476.065092] ? watchdog+0x40/0x40 [14476.065097] ? hrtimer_interrupt+0x128/0x2c0 [14476.065100] ? smp_apic_timer_interrupt+0xa6/0x270 [14476.065108] ? apic_timer_interrupt+0xf/0x20 [14476.065112] [14476.065115] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14476.065126] do_raw_spin_lock+0x177/0x190 [14476.065131] _raw_spin_lock+0x21/0x30 [14476.065136] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14476.066125] ? do_raw_spin_unlock+0x75/0x190 [14476.066138] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14476.066403] ? do_wait_intr+0xf0/0xf0 [14476.066545] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14476.066850] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14476.067182] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14476.068358] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14476.083352] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14476.083427] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14476.083434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14476.083436] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14476.083452] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14476.083456] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14476.083460] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14476.083462] RDX: 0000000000003970 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14476.083464] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14476.083466] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14476.083468] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14476.083471] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14476.083476] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14476.105111] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14476.105163] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14476.105194] mdc_intent_lock+0x353/0x7e0 [mdc] [14476.105224] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14476.105560] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14476.106482] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14476.106519] lmv_intent_open+0x338/0x1150 [lmv] [14476.106549] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14476.106696] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14476.106781] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14476.106804] ll_intent_lock+0xc5/0xaa0 [lustre] [14476.106942] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14476.107043] ll_lookup_it+0xe0e/0x2270 [lustre] [14476.107135] ? do_raw_spin_unlock+0x75/0x190 [14476.107147] ? _raw_spin_unlock+0x12/0x30 [14476.107154] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14476.107179] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14476.107204] ll_atomic_open+0x36a/0xd70 [lustre] [14476.107304] lookup_open+0x356/0xb40 [14476.107314] do_last+0x313/0xfc0 [14476.107319] ? nd_jump_root+0xe5/0x160 [14476.107324] ? path_init+0x437/0x520 [14476.107329] path_openat+0xf7/0x500 [14476.107432] ? do_raw_spin_unlock+0x75/0x190 [14476.107441] do_filp_open+0x99/0x140 [14476.107448] ? getname_flags+0x6e/0x330 [14476.107452] ? __check_object_size+0xff/0x256 [14476.107458] ? do_raw_spin_unlock+0x75/0x190 [14476.107462] ? _raw_spin_unlock+0x12/0x30 [14476.107467] do_sys_openat2+0x2b4/0x410 [14476.107471] do_sys_open+0x73/0xa0 [14476.107475] __x64_sys_openat+0x24/0x30 [14476.107479] do_syscall_64+0xc1/0x440 [14476.107485] entry_SYSCALL_64_after_hwframe+0x49/0xae [14476.107491] RIP: 0033:0x7f53425a5332 [14476.107502] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14476.107505] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14476.107511] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14476.107514] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14476.107516] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14476.107518] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14476.107521] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14476.676813] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14476.685407] Call Trace: [14476.687052] [14476.688256] ? show_regs.cold.9+0x22/0x2f [14476.690659] ? watchdog_timer_fn.cold.6+0x62/0x10a [14476.696195] ? __hrtimer_run_queues+0x1c2/0x4d0 [14476.698220] ? watchdog+0x40/0x40 [14476.699265] ? hrtimer_interrupt+0x128/0x2c0 [14476.700697] ? smp_apic_timer_interrupt+0xa6/0x270 [14476.715704] ? apic_timer_interrupt+0xf/0x20 [14476.719796] [14476.720778] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14476.723770] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14476.727218] do_raw_spin_lock+0x177/0x190 [14476.729392] _raw_spin_lock+0x21/0x30 [14476.734199] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14476.742738] obd_mod_free+0x46/0x140 [ptlrpc] [14476.752452] mdc_commit_open+0x91/0xb0 [mdc] [14476.757835] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14476.759744] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14476.777801] after_reply+0xe2c/0x1780 [ptlrpc] [14476.782152] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14476.793135] ? __next_timer_interrupt+0x160/0x160 [14476.798859] ptlrpcd+0xa82/0xe40 [ptlrpc] [14476.800519] ? do_wait_intr+0xf0/0xf0 [14476.812871] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14476.816180] kthread+0x1d1/0x200 [14476.818097] ? set_kthread_struct+0x70/0x70 [14476.822126] ret_from_fork+0x1f/0x30 [14484.052049] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [14484.057197] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14484.059367] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [14484.103009] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14484.103025] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14484.103032] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14484.103322] RIP: 0010:native_safe_halt+0x2e/0x50 [14484.103331] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14484.103335] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14484.103339] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14484.103341] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14484.103343] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14484.103346] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14484.103348] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14484.103351] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14484.103354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14484.103356] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14484.103366] Call Trace: [14484.103371] [14484.103377] ? show_regs.cold.9+0x22/0x2f [14484.103386] ? watchdog_timer_fn.cold.6+0x62/0x10a [14484.103392] ? __hrtimer_run_queues+0x1c2/0x4d0 [14484.103396] ? watchdog+0x40/0x40 [14484.103400] ? hrtimer_interrupt+0x128/0x2c0 [14484.103404] ? smp_apic_timer_interrupt+0xa6/0x270 [14484.103411] ? apic_timer_interrupt+0xf/0x20 [14484.103415] [14484.103417] ? native_safe_halt+0x2e/0x50 [14484.103421] kvm_wait+0xcd/0xe0 [14484.103429] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14484.103435] ? newidle_balance.isra.93+0x207/0x580 [14484.103440] do_raw_spin_lock+0x177/0x190 [14484.103445] _raw_spin_lock+0x21/0x30 [14484.103449] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14484.106170] process_one_work+0x2c8/0x700 [14484.106179] worker_thread+0x296/0x6e0 [14484.106184] ? rescuer_thread+0x570/0x570 [14484.106188] kthread+0x1d1/0x200 [14484.106193] ? set_kthread_struct+0x70/0x70 [14484.106196] ret_from_fork+0x1f/0x30 [14484.331730] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14484.374993] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14484.388958] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14484.399697] RIP: 0010:native_safe_halt+0x2e/0x50 [14484.408104] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14484.440288] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14484.461779] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14484.474385] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14484.487190] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14484.489611] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14484.501029] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14484.511195] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14484.520192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14484.523533] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14484.530066] Call Trace: [14484.533120] [14484.540763] ? show_regs.cold.9+0x22/0x2f [14484.544571] ? watchdog_timer_fn.cold.6+0x62/0x10a [14484.549960] ? __hrtimer_run_queues+0x1c2/0x4d0 [14484.554237] ? watchdog+0x40/0x40 [14484.555331] ? hrtimer_interrupt+0x128/0x2c0 [14484.571972] ? smp_apic_timer_interrupt+0xa6/0x270 [14484.582249] ? apic_timer_interrupt+0xf/0x20 [14484.586694] [14484.587578] ? native_safe_halt+0x2e/0x50 [14484.589677] kvm_wait+0xcd/0xe0 [14484.592743] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14484.597038] do_raw_spin_lock+0x177/0x190 [14484.602582] _raw_spin_lock+0x21/0x30 [14484.610196] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14484.616059] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14484.622434] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14484.627919] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14484.635244] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14484.638212] ? woken_wake_function+0x30/0x30 [14484.643284] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14484.646653] kthread+0x1d1/0x200 [14484.648682] ? set_kthread_struct+0x70/0x70 [14484.650764] ret_from_fork+0x1f/0x30 [14487.013047] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11849s! [14487.019592] Showing busy workqueues and worker pools: [14487.025466] workqueue events: flags=0x0 [14487.031558] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14487.035495] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14487.041564] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14487.044627] pending: cpu_latency_work [ptlrpc] [14487.047212] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14487.049594] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14487.053955] workqueue events_long: flags=0x0 [14487.055840] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14487.059164] pending: lu_objects_flush [obdclass] [14487.061641] workqueue events_unbound: flags=0x2 [14487.063700] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14487.066612] pending: flush_memcg_stats_dwork, idle_cull_fn [14487.069837] workqueue events_power_efficient: flags=0x80 [14487.075026] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14487.082148] pending: check_lifetime, do_cache_clean [14487.087489] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14487.093405] pending: neigh_periodic_work, neigh_periodic_work [14487.097752] workqueue mm_percpu_wq: flags=0x8 [14487.101413] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14487.108461] pending: vmstat_update, lru_add_drain_per_cpu [14487.114482] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14487.123486] pending: vmstat_update, lru_add_drain_per_cpu [14487.126653] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14487.129416] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14487.132019] workqueue xprtiod: flags=0xa [14487.133537] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14487.136205] pending: xprt_autoclose, xs_error_handle [14487.138403] workqueue ipv6_addrconf: flags=0x40008 [14487.140388] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14487.143574] pending: addrconf_verify_work [14487.145429] workqueue knbd0-recv: flags=0x1a [14487.148030] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14487.150592] in-flight: 651:recv_work [14487.152235] workqueue ptlrpc_pinger: flags=0x2 [14487.154052] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14487.156233] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14487.158937] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11850s workers=4 idle: 15979 31824 8 [14487.162015] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14477s workers=2 idle: 654 [14504.055019] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [14504.057150] Modules linked in: [14504.057178] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [14504.057276] lustre(O) [14504.061715] Modules linked in: [14504.064665] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14504.064735] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14504.064742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14504.064745] RIP: 0010:native_safe_halt+0x2e/0x50 [14504.064759] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14504.064763] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14504.064768] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14504.064770] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14504.064772] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14504.064774] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14504.064777] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14504.064780] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14504.064784] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14504.064786] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14504.064795] Call Trace: [14504.064801] [14504.064808] ? show_regs.cold.9+0x22/0x2f [14504.064817] ? watchdog_timer_fn.cold.6+0x62/0x10a [14504.064824] ? __hrtimer_run_queues+0x1c2/0x4d0 [14504.064828] ? watchdog+0x40/0x40 [14504.064833] ? hrtimer_interrupt+0x128/0x2c0 [14504.064837] ? smp_apic_timer_interrupt+0xa6/0x270 [14504.064844] ? apic_timer_interrupt+0xf/0x20 [14504.064848] [14504.064851] ? native_safe_halt+0x2e/0x50 [14504.064855] kvm_wait+0xcd/0xe0 [14504.064863] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14504.064872] do_raw_spin_lock+0x177/0x190 [14504.064877] _raw_spin_lock+0x21/0x30 [14504.064881] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14504.065166] ? do_raw_spin_unlock+0x75/0x190 [14504.070889] lustre(O) [14504.072371] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14504.072748] ? do_wait_intr+0xf0/0xf0 [14504.072760] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14504.073121] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14504.073350] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14504.073649] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14504.073762] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14504.073798] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14504.073826] mdc_intent_lock+0x353/0x7e0 [mdc] [14504.073853] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14504.073939] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14504.074158] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14504.074188] lmv_intent_open+0x338/0x1150 [lmv] [14504.074219] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14504.074304] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14504.141079] osp(O) [14504.155147] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14504.155196] ll_intent_lock+0xc5/0xaa0 [lustre] [14504.155308] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14504.155391] ll_lookup_it+0xe0e/0x2270 [lustre] [14504.155469] ? do_raw_spin_unlock+0x75/0x190 [14504.156009] ? _raw_spin_unlock+0x12/0x30 [14504.156017] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14504.156044] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14504.156063] ll_atomic_open+0x36a/0xd70 [lustre] [14504.156153] lookup_open+0x356/0xb40 [14504.156162] do_last+0x313/0xfc0 [14504.156168] ? nd_jump_root+0xe5/0x160 [14504.156173] ? path_init+0x437/0x520 [14504.156177] path_openat+0xf7/0x500 [14504.156182] ? do_raw_spin_unlock+0x75/0x190 [14504.156188] do_filp_open+0x99/0x140 [14504.156195] ? getname_flags+0x6e/0x330 [14504.156200] ? __check_object_size+0xff/0x256 [14504.156207] ? do_raw_spin_unlock+0x75/0x190 [14504.156211] ? _raw_spin_unlock+0x12/0x30 [14504.156216] do_sys_openat2+0x2b4/0x410 [14504.156221] do_sys_open+0x73/0xa0 [14504.156225] __x64_sys_openat+0x24/0x30 [14504.156229] do_syscall_64+0xc1/0x440 [14504.156235] entry_SYSCALL_64_after_hwframe+0x49/0xae [14504.156242] RIP: 0033:0x7f53425a5332 [14504.156250] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14504.156254] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14504.156260] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14504.156263] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14504.156265] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14504.156267] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14504.156270] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14504.345591] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14504.381465] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14504.386949] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14504.391291] RIP: 0010:native_safe_halt+0x2e/0x50 [14504.393104] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14504.398620] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14504.401328] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14504.403341] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14504.406078] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14504.408748] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14504.411617] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14504.414316] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14504.417629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14504.422923] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14504.431383] Call Trace: [14504.432360] [14504.433102] ? show_regs.cold.9+0x22/0x2f [14504.436141] ? watchdog_timer_fn.cold.6+0x62/0x10a [14504.438722] ? __hrtimer_run_queues+0x1c2/0x4d0 [14504.441145] ? watchdog+0x40/0x40 [14504.442464] ? hrtimer_interrupt+0x128/0x2c0 [14504.444397] ? smp_apic_timer_interrupt+0xa6/0x270 [14504.446809] ? apic_timer_interrupt+0xf/0x20 [14504.448715] [14504.450225] ? native_safe_halt+0x2e/0x50 [14504.452757] kvm_wait+0xcd/0xe0 [14504.454337] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14504.457405] do_raw_spin_lock+0x177/0x190 [14504.460017] _raw_spin_lock+0x21/0x30 [14504.461948] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14504.465056] obd_mod_free+0x46/0x140 [ptlrpc] [14504.468183] mdc_commit_open+0x91/0xb0 [mdc] [14504.470915] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14504.473628] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14504.477213] after_reply+0xe2c/0x1780 [ptlrpc] [14504.480925] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14504.484208] ? __next_timer_interrupt+0x160/0x160 [14504.487221] ptlrpcd+0xa82/0xe40 [ptlrpc] [14504.492207] ? do_wait_intr+0xf0/0xf0 [14504.493100] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14504.499226] kthread+0x1d1/0x200 [14504.500777] ? set_kthread_struct+0x70/0x70 [14504.504949] ret_from_fork+0x1f/0x30 [14512.054085] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [14512.060147] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14512.060556] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [14512.128195] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14512.128208] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14512.128216] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14512.132141] Modules linked in: [14512.161950] [14512.161960] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14512.161977] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14512.161982] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14512.161987] RAX: 000000000000783c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14512.161989] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14512.161992] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14512.161994] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14512.161996] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14512.161999] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14512.162002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14512.162004] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14512.162012] Call Trace: [14512.162018] [14512.162024] ? show_regs.cold.9+0x22/0x2f [14512.162034] ? watchdog_timer_fn.cold.6+0x62/0x10a [14512.162041] ? __hrtimer_run_queues+0x1c2/0x4d0 [14512.162045] ? watchdog+0x40/0x40 [14512.162049] ? hrtimer_interrupt+0x128/0x2c0 [14512.162054] ? smp_apic_timer_interrupt+0xa6/0x270 [14512.162062] ? apic_timer_interrupt+0xf/0x20 [14512.168094] lustre(O) [14512.177983] [14512.177995] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14512.178008] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14512.178259] ? newidle_balance.isra.93+0x207/0x580 [14512.178267] do_raw_spin_lock+0x177/0x190 [14512.178274] _raw_spin_lock+0x21/0x30 [14512.178624] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14512.178905] process_one_work+0x2c8/0x700 [14512.178912] worker_thread+0x296/0x6e0 [14512.178917] ? rescuer_thread+0x570/0x570 [14512.178921] kthread+0x1d1/0x200 [14512.178926] ? set_kthread_struct+0x70/0x70 [14512.178929] ret_from_fork+0x1f/0x30 [14512.381072] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14512.419949] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14512.431083] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14512.440855] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14512.448841] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14512.463714] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14512.473136] RAX: 000000000000220e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14512.478595] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14512.482239] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14512.487008] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14512.494973] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14512.501903] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14512.507559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14512.513639] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14512.524155] Call Trace: [14512.528691] [14512.529470] ? show_regs.cold.9+0x22/0x2f [14512.534877] ? watchdog_timer_fn.cold.6+0x62/0x10a [14512.541009] ? __hrtimer_run_queues+0x1c2/0x4d0 [14512.544702] ? watchdog+0x40/0x40 [14512.548269] ? hrtimer_interrupt+0x128/0x2c0 [14512.550981] ? smp_apic_timer_interrupt+0xa6/0x270 [14512.559013] ? apic_timer_interrupt+0xf/0x20 [14512.564989] [14512.566384] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14512.572385] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14512.580139] do_raw_spin_lock+0x177/0x190 [14512.582805] _raw_spin_lock+0x21/0x30 [14512.585084] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14512.597787] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14512.603840] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14512.609018] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14512.613931] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14512.621909] ? woken_wake_function+0x30/0x30 [14512.626017] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14512.629725] kthread+0x1d1/0x200 [14512.631949] ? set_kthread_struct+0x70/0x70 [14512.637050] ret_from_fork+0x1f/0x30 [14517.217160] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11880s! [14517.224724] Showing busy workqueues and worker pools: [14517.229709] workqueue events: flags=0x0 [14517.230857] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14517.236037] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14517.242447] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14517.256656] pending: cpu_latency_work [ptlrpc] [14517.261747] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14517.272227] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14517.281816] workqueue events_long: flags=0x0 [14517.283779] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14517.286857] pending: lu_objects_flush [obdclass] [14517.290450] workqueue events_unbound: flags=0x2 [14517.291708] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14517.294134] pending: flush_memcg_stats_dwork, idle_cull_fn [14517.296830] workqueue events_power_efficient: flags=0x80 [14517.299812] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14517.305548] pending: check_lifetime, do_cache_clean [14517.309616] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14517.318871] pending: neigh_periodic_work, neigh_periodic_work [14517.324382] workqueue mm_percpu_wq: flags=0x8 [14517.329468] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14517.335081] pending: vmstat_update, lru_add_drain_per_cpu [14517.339538] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14517.343184] pending: vmstat_update, lru_add_drain_per_cpu [14517.350859] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14517.356184] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14517.362220] workqueue xprtiod: flags=0xa [14517.364730] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14517.368409] pending: xprt_autoclose, xs_error_handle [14517.373094] workqueue ipv6_addrconf: flags=0x40008 [14517.376496] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14517.378972] pending: addrconf_verify_work [14517.387612] workqueue knbd0-recv: flags=0x1a [14517.390380] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14517.397534] in-flight: 651:recv_work [14517.407711] workqueue ptlrpc_pinger: flags=0x2 [14517.409305] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14517.426481] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14517.434444] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11880s workers=4 idle: 15979 31824 8 [14517.441618] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14507s workers=2 idle: 654 [14532.055462] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [14532.057027] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [14532.058853] Modules linked in: lustre(O) [14532.063217] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14532.063294] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14532.063302] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14532.063304] RIP: 0010:native_safe_halt+0x2e/0x50 [14532.063317] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14532.063322] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14532.063327] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14532.063329] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14532.063331] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14532.063333] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14532.063336] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14532.063339] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14532.063342] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14532.063344] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14532.063352] Call Trace: [14532.063358] [14532.063364] ? show_regs.cold.9+0x22/0x2f [14532.063375] ? watchdog_timer_fn.cold.6+0x62/0x10a [14532.063381] ? __hrtimer_run_queues+0x1c2/0x4d0 [14532.063386] ? watchdog+0x40/0x40 [14532.063390] ? hrtimer_interrupt+0x128/0x2c0 [14532.063394] ? smp_apic_timer_interrupt+0xa6/0x270 [14532.063401] ? apic_timer_interrupt+0xf/0x20 [14532.063406] [14532.063409] ? native_safe_halt+0x2e/0x50 [14532.063413] kvm_wait+0xcd/0xe0 [14532.063420] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14532.063426] do_raw_spin_lock+0x177/0x190 [14532.063431] _raw_spin_lock+0x21/0x30 [14532.063437] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14532.063865] obd_mod_free+0x46/0x140 [ptlrpc] [14532.064378] mdc_commit_open+0x91/0xb0 [mdc] [14532.064417] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14532.064687] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14532.065094] after_reply+0xe2c/0x1780 [ptlrpc] [14532.065461] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14532.065809] ? __next_timer_interrupt+0x160/0x160 [14532.065821] ptlrpcd+0xa82/0xe40 [ptlrpc] [14532.066176] ? do_wait_intr+0xf0/0xf0 [14532.066200] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14532.071195] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [14532.093530] kthread+0x1d1/0x200 [14532.093618] ? set_kthread_struct+0x70/0x70 [14532.093623] ret_from_fork+0x1f/0x30 [14532.247910] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14532.266492] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14532.277544] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14532.284042] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14532.287686] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14532.297728] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14532.301427] RAX: 0000000000002f43 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14532.304442] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14532.307459] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14532.310126] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14532.312136] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14532.315143] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14532.317503] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14532.319086] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14532.322356] Call Trace: [14532.323177] [14532.323731] ? show_regs.cold.9+0x22/0x2f [14532.327342] ? watchdog_timer_fn.cold.6+0x62/0x10a [14532.329561] ? __hrtimer_run_queues+0x1c2/0x4d0 [14532.332093] ? watchdog+0x40/0x40 [14532.333354] ? hrtimer_interrupt+0x128/0x2c0 [14532.335834] ? smp_apic_timer_interrupt+0xa6/0x270 [14532.337684] ? apic_timer_interrupt+0xf/0x20 [14532.338920] [14532.341409] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14532.345758] do_raw_spin_lock+0x177/0x190 [14532.350165] _raw_spin_lock+0x21/0x30 [14532.351731] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14532.354706] ? do_raw_spin_unlock+0x75/0x190 [14532.357262] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14532.360384] ? do_wait_intr+0xf0/0xf0 [14532.361818] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14532.365899] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14532.368770] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14532.371658] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14532.378596] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14532.382972] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14532.384821] mdc_intent_lock+0x353/0x7e0 [mdc] [14532.388240] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14532.389794] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14532.393302] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14532.395971] lmv_intent_open+0x338/0x1150 [lmv] [14532.398961] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14532.404091] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14532.405690] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14532.408752] ll_intent_lock+0xc5/0xaa0 [lustre] [14532.412135] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14532.427108] ll_lookup_it+0xe0e/0x2270 [lustre] [14532.430141] ? do_raw_spin_unlock+0x75/0x190 [14532.432332] ? _raw_spin_unlock+0x12/0x30 [14532.436591] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14532.441584] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14532.444614] ll_atomic_open+0x36a/0xd70 [lustre] [14532.453657] lookup_open+0x356/0xb40 [14532.454696] do_last+0x313/0xfc0 [14532.462554] ? nd_jump_root+0xe5/0x160 [14532.465570] ? path_init+0x437/0x520 [14532.468057] path_openat+0xf7/0x500 [14532.472381] ? do_raw_spin_unlock+0x75/0x190 [14532.478258] do_filp_open+0x99/0x140 [14532.479707] ? getname_flags+0x6e/0x330 [14532.483403] ? __check_object_size+0xff/0x256 [14532.487563] ? do_raw_spin_unlock+0x75/0x190 [14532.491631] ? _raw_spin_unlock+0x12/0x30 [14532.492663] do_sys_openat2+0x2b4/0x410 [14532.493723] do_sys_open+0x73/0xa0 [14532.496081] __x64_sys_openat+0x24/0x30 [14532.499082] do_syscall_64+0xc1/0x440 [14532.504387] entry_SYSCALL_64_after_hwframe+0x49/0xae [14532.511471] RIP: 0033:0x7f53425a5332 [14532.513392] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14532.530208] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14532.537587] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14532.542622] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14532.545032] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14532.553040] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14532.559243] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14540.058029] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [14540.061639] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14540.103955] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14540.109023] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14540.114047] RIP: 0010:native_safe_halt+0x2e/0x50 [14540.118383] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14540.135000] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14540.140667] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14540.145063] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14540.150834] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14540.156999] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14540.165530] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14540.175321] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14540.184655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14540.189299] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14540.195531] Call Trace: [14540.197117] [14540.197853] ? show_regs.cold.9+0x22/0x2f [14540.199986] ? watchdog_timer_fn.cold.6+0x62/0x10a [14540.203955] ? __hrtimer_run_queues+0x1c2/0x4d0 [14540.205895] ? watchdog+0x40/0x40 [14540.207791] ? hrtimer_interrupt+0x128/0x2c0 [14540.213104] ? smp_apic_timer_interrupt+0xa6/0x270 [14540.215978] ? apic_timer_interrupt+0xf/0x20 [14540.220871] [14540.221967] ? native_safe_halt+0x2e/0x50 [14540.224058] kvm_wait+0xcd/0xe0 [14540.225211] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14540.230142] do_raw_spin_lock+0x177/0x190 [14540.232204] _raw_spin_lock+0x21/0x30 [14540.235161] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14540.241128] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14540.243628] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14540.249619] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14540.252969] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14540.262811] ? woken_wake_function+0x30/0x30 [14540.264729] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14540.270689] kthread+0x1d1/0x200 [14540.274025] ? set_kthread_struct+0x70/0x70 [14540.276162] ret_from_fork+0x1f/0x30 [14544.052060] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [14544.057121] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14544.098458] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14544.106405] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14544.115105] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14544.120797] RIP: 0010:native_safe_halt+0x2e/0x50 [14544.123110] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14544.136537] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14544.141543] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14544.147657] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [14544.153329] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14544.158754] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14544.166351] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14544.169553] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14544.178149] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14544.191615] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14544.198278] Call Trace: [14544.202464] [14544.205441] ? show_regs.cold.9+0x22/0x2f [14544.210982] ? watchdog_timer_fn.cold.6+0x62/0x10a [14544.218542] ? __hrtimer_run_queues+0x1c2/0x4d0 [14544.224400] ? watchdog+0x40/0x40 [14544.227401] ? hrtimer_interrupt+0x128/0x2c0 [14544.229568] ? smp_apic_timer_interrupt+0xa6/0x270 [14544.237045] ? apic_timer_interrupt+0xf/0x20 [14544.239921] [14544.241678] ? native_safe_halt+0x2e/0x50 [14544.244182] kvm_wait+0xcd/0xe0 [14544.245464] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14544.248458] ? newidle_balance.isra.93+0x207/0x580 [14544.261733] do_raw_spin_lock+0x177/0x190 [14544.263834] _raw_spin_lock+0x21/0x30 [14544.266391] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14544.277132] process_one_work+0x2c8/0x700 [14544.280298] worker_thread+0x296/0x6e0 [14544.281594] ? rescuer_thread+0x570/0x570 [14544.284533] kthread+0x1d1/0x200 [14544.288062] ? set_kthread_struct+0x70/0x70 [14544.294280] ret_from_fork+0x1f/0x30 [14547.945356] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11910s! [14547.949228] Showing busy workqueues and worker pools: [14547.952693] workqueue events: flags=0x0 [14547.955233] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14547.959180] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14547.967958] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14547.977361] pending: cpu_latency_work [ptlrpc] [14547.984178] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14547.988019] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14547.992649] workqueue events_long: flags=0x0 [14547.994514] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14547.997333] pending: lu_objects_flush [obdclass] [14548.000150] workqueue events_unbound: flags=0x2 [14548.002197] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14548.005131] pending: flush_memcg_stats_dwork, idle_cull_fn [14548.010044] workqueue events_power_efficient: flags=0x80 [14548.012921] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14548.016700] pending: check_lifetime, do_cache_clean [14548.019406] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14548.023299] pending: neigh_periodic_work, neigh_periodic_work [14548.029576] workqueue mm_percpu_wq: flags=0x8 [14548.033014] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14548.037623] pending: vmstat_update, lru_add_drain_per_cpu [14548.040873] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14548.044760] pending: vmstat_update, lru_add_drain_per_cpu [14548.048079] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14548.052120] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14548.055617] workqueue xprtiod: flags=0xa [14548.058144] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14548.063193] pending: xprt_autoclose, xs_error_handle [14548.066995] workqueue ipv6_addrconf: flags=0x40008 [14548.070022] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14548.074593] pending: addrconf_verify_work [14548.076677] workqueue knbd0-recv: flags=0x1a [14548.078942] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14548.082312] in-flight: 651:recv_work [14548.084517] workqueue ptlrpc_pinger: flags=0x2 [14548.086598] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14548.089377] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14548.093628] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11911s workers=4 idle: 15979 31824 8 [14548.098979] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14538s workers=2 idle: 654 [14560.057020] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [14560.059674] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [14560.059771] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14560.059845] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14560.059851] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14560.059855] RIP: 0010:native_safe_halt+0x2e/0x50 [14560.059868] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14560.059873] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14560.059877] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14560.059880] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14560.059882] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14560.059884] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14560.059887] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14560.059890] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14560.059893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14560.059895] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14560.059905] Call Trace: [14560.059911] [14560.059916] ? show_regs.cold.9+0x22/0x2f [14560.059925] ? watchdog_timer_fn.cold.6+0x62/0x10a [14560.059932] ? __hrtimer_run_queues+0x1c2/0x4d0 [14560.059937] ? watchdog+0x40/0x40 [14560.059941] ? hrtimer_interrupt+0x128/0x2c0 [14560.059946] ? smp_apic_timer_interrupt+0xa6/0x270 [14560.059953] ? apic_timer_interrupt+0xf/0x20 [14560.059957] [14560.059960] ? native_safe_halt+0x2e/0x50 [14560.059964] kvm_wait+0xcd/0xe0 [14560.059973] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14560.059981] do_raw_spin_lock+0x177/0x190 [14560.059986] _raw_spin_lock+0x21/0x30 [14560.059991] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14560.070510] Modules linked in: lustre(O) [14560.087021] obd_mod_free+0x46/0x140 [ptlrpc] [14560.087284] mdc_commit_open+0x91/0xb0 [mdc] [14560.087330] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14560.100478] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic [14560.104635] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14560.104910] after_reply+0xe2c/0x1780 [ptlrpc] [14560.105105] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14560.105301] ? __next_timer_interrupt+0x160/0x160 [14560.105310] ptlrpcd+0xa82/0xe40 [ptlrpc] [14560.105450] ? do_wait_intr+0xf0/0xf0 [14560.105460] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14560.108822] crct10dif_pclmul [14560.117406] kthread+0x1d1/0x200 [14560.117423] ? set_kthread_struct+0x70/0x70 [14560.117427] ret_from_fork+0x1f/0x30 [14560.306797] crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14560.319046] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14560.325045] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14560.330075] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14560.337904] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14560.356669] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14560.360719] RAX: 0000000000007da4 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14560.371886] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14560.376482] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14560.380349] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14560.388122] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14560.393408] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14560.398312] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14560.402808] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14560.409053] Call Trace: [14560.410074] [14560.412459] ? show_regs.cold.9+0x22/0x2f [14560.414926] ? watchdog_timer_fn.cold.6+0x62/0x10a [14560.421609] ? __hrtimer_run_queues+0x1c2/0x4d0 [14560.424969] ? watchdog+0x40/0x40 [14560.427197] ? hrtimer_interrupt+0x128/0x2c0 [14560.429436] ? smp_apic_timer_interrupt+0xa6/0x270 [14560.431122] ? apic_timer_interrupt+0xf/0x20 [14560.447048] [14560.451376] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14560.457952] do_raw_spin_lock+0x177/0x190 [14560.461177] _raw_spin_lock+0x21/0x30 [14560.464575] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14560.472236] ? do_raw_spin_unlock+0x75/0x190 [14560.477889] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14560.480669] ? do_wait_intr+0xf0/0xf0 [14560.482021] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14560.508726] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14560.519213] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14560.525202] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14560.536053] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14560.542274] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14560.546447] mdc_intent_lock+0x353/0x7e0 [mdc] [14560.553681] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14560.559860] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14560.562155] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14560.567595] lmv_intent_open+0x338/0x1150 [lmv] [14560.570477] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14560.573864] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14560.578729] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14560.581916] ll_intent_lock+0xc5/0xaa0 [lustre] [14560.590588] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14560.597649] ll_lookup_it+0xe0e/0x2270 [lustre] [14560.602638] ? do_raw_spin_unlock+0x75/0x190 [14560.606054] ? _raw_spin_unlock+0x12/0x30 [14560.609555] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14560.613567] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14560.627189] ll_atomic_open+0x36a/0xd70 [lustre] [14560.634720] lookup_open+0x356/0xb40 [14560.638289] do_last+0x313/0xfc0 [14560.639477] ? nd_jump_root+0xe5/0x160 [14560.651778] ? path_init+0x437/0x520 [14560.658054] path_openat+0xf7/0x500 [14560.660104] ? do_raw_spin_unlock+0x75/0x190 [14560.666151] do_filp_open+0x99/0x140 [14560.667930] ? getname_flags+0x6e/0x330 [14560.671630] ? __check_object_size+0xff/0x256 [14560.674568] ? do_raw_spin_unlock+0x75/0x190 [14560.676107] ? _raw_spin_unlock+0x12/0x30 [14560.682666] do_sys_openat2+0x2b4/0x410 [14560.685357] do_sys_open+0x73/0xa0 [14560.686912] __x64_sys_openat+0x24/0x30 [14560.689655] do_syscall_64+0xc1/0x440 [14560.695468] entry_SYSCALL_64_after_hwframe+0x49/0xae [14560.701086] RIP: 0033:0x7f53425a5332 [14560.706348] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14560.725002] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14560.730043] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14560.742849] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14560.757313] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14560.767726] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14560.781894] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14568.058289] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [14568.062001] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14568.085468] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14568.093059] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14568.101207] RIP: 0010:native_safe_halt+0x2e/0x50 [14568.104757] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14568.114011] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14568.119725] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14568.123922] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14568.126594] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14568.130050] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14568.136041] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14568.139770] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14568.147414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14568.152630] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14568.157804] Call Trace: [14568.160481] [14568.161487] ? show_regs.cold.9+0x22/0x2f [14568.166041] ? watchdog_timer_fn.cold.6+0x62/0x10a [14568.169160] ? __hrtimer_run_queues+0x1c2/0x4d0 [14568.177565] ? watchdog+0x40/0x40 [14568.178728] ? hrtimer_interrupt+0x128/0x2c0 [14568.180121] ? smp_apic_timer_interrupt+0xa6/0x270 [14568.183957] ? apic_timer_interrupt+0xf/0x20 [14568.186066] [14568.187123] ? native_safe_halt+0x2e/0x50 [14568.189192] kvm_wait+0xcd/0xe0 [14568.191659] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14568.193970] do_raw_spin_lock+0x177/0x190 [14568.196948] _raw_spin_lock+0x21/0x30 [14568.199689] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14568.202369] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14568.204336] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14568.207389] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14568.210625] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14568.214195] ? woken_wake_function+0x30/0x30 [14568.215715] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14568.218908] kthread+0x1d1/0x200 [14568.220770] ? set_kthread_struct+0x70/0x70 [14568.222687] ret_from_fork+0x1f/0x30 [14574.730047] rcu: INFO: rcu_sched self-detected stall on CPU [14574.732853] rcu: 2-....: (6918603 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [14574.743627] (t=12480304 jiffies g=274185 q=67101) [14574.747514] rcu: rcu_sched kthread starved for 11938670 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [14574.756165] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14574.766522] rcu: RCU grace-period kthread stack dump: [14574.768127] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14574.778502] Call Trace: [14574.780359] __schedule+0x351/0xcb0 [14574.782178] schedule+0xc0/0x180 [14574.788818] schedule_timeout+0xb4/0x190 [14574.792707] ? do_raw_spin_unlock+0x75/0x190 [14574.794677] ? __next_timer_interrupt+0x160/0x160 [14574.797061] ? prepare_to_swait_event+0xf3/0x140 [14574.799972] rcu_gp_kthread+0x8dd/0x15b0 [14574.804478] ? force_qs_rnp+0x2c0/0x2c0 [14574.806311] kthread+0x1d1/0x200 [14574.807964] ? set_kthread_struct+0x70/0x70 [14574.810336] ret_from_fork+0x1f/0x30 [14574.812092] rcu: Stack dump where RCU GP kthread last ran: [14574.819182] Sending NMI from CPU 2 to CPUs 0: [14574.821162] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [14574.824799] Sending NMI from CPU 2 to CPUs 1: [14574.828413] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [14574.831660] NMI backtrace for cpu 2 [14574.838724] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14574.845654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14574.854844] Call Trace: [14574.857399] [14574.863586] ? dump_stack+0xbb/0x10e [14574.868522] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14574.872816] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14574.876194] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14574.879215] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14574.882483] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14574.886084] ? rcu_dump_cpu_stacks+0x11b/0x171 [14574.889974] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14574.896382] ? update_process_times+0x6d/0xc0 [14574.900890] ? tick_sched_handle.isra.18+0x27/0x90 [14574.907436] ? tick_sched_timer+0x53/0xb0 [14574.914602] ? __hrtimer_run_queues+0x1c2/0x4d0 [14574.925392] ? tick_sched_handle.isra.18+0x90/0x90 [14574.929946] ? hrtimer_interrupt+0x128/0x2c0 [14574.933186] ? smp_apic_timer_interrupt+0xa6/0x270 [14574.935410] ? apic_timer_interrupt+0xf/0x20 [14574.937762] [14574.939729] ? native_safe_halt+0x2e/0x50 [14574.941657] ? kvm_wait+0xcd/0xe0 [14574.942913] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14574.947779] ? do_raw_spin_lock+0x177/0x190 [14574.952481] ? _raw_spin_lock+0x21/0x30 [14574.959687] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14574.968696] ? obd_mod_free+0x46/0x140 [ptlrpc] [14574.974469] ? mdc_commit_open+0x91/0xb0 [mdc] [14574.978199] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14574.982018] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14574.987724] ? after_reply+0xe2c/0x1780 [ptlrpc] [14574.995347] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14574.998199] ? __next_timer_interrupt+0x160/0x160 [14575.000938] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14575.003145] ? do_wait_intr+0xf0/0xf0 [14575.008967] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14575.014375] ? kthread+0x1d1/0x200 [14575.017603] ? set_kthread_struct+0x70/0x70 [14575.019209] ? ret_from_fork+0x1f/0x30 [14578.149375] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11941s! [14578.153945] Showing busy workqueues and worker pools: [14578.156328] workqueue events: flags=0x0 [14578.157744] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14578.160323] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14578.165016] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14578.167897] pending: cpu_latency_work [ptlrpc] [14578.169997] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14578.172576] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14578.176139] workqueue events_long: flags=0x0 [14578.177696] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14578.180237] pending: lu_objects_flush [obdclass] [14578.182288] workqueue events_unbound: flags=0x2 [14578.184333] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14578.186457] pending: flush_memcg_stats_dwork, idle_cull_fn [14578.189186] workqueue events_power_efficient: flags=0x80 [14578.191490] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14578.194233] pending: check_lifetime, do_cache_clean [14578.196461] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14578.198974] pending: neigh_periodic_work, neigh_periodic_work [14578.201888] workqueue mm_percpu_wq: flags=0x8 [14578.203684] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14578.206113] pending: vmstat_update, lru_add_drain_per_cpu [14578.208056] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14578.210339] pending: vmstat_update, lru_add_drain_per_cpu [14578.212434] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14578.214778] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14578.217170] workqueue xprtiod: flags=0xa [14578.218789] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14578.221199] pending: xprt_autoclose, xs_error_handle [14578.223083] workqueue ipv6_addrconf: flags=0x40008 [14578.224786] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14578.226903] pending: addrconf_verify_work [14578.228770] workqueue knbd0-recv: flags=0x1a [14578.230356] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14578.232653] in-flight: 651:recv_work [14578.234060] workqueue ptlrpc_pinger: flags=0x2 [14578.235689] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14578.238153] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14578.240636] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11941s workers=4 idle: 15979 31824 8 [14578.244337] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14568s workers=2 idle: 654 [14596.058069] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [14596.061990] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14596.080504] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14596.085580] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14596.089281] RIP: 0010:native_safe_halt+0x2e/0x50 [14596.091422] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14596.099921] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14596.103032] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14596.105942] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14596.108681] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14596.111849] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14596.115162] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14596.117995] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14596.121323] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14596.123796] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14596.126631] Call Trace: [14596.127769] [14596.128630] ? show_regs.cold.9+0x22/0x2f [14596.130089] ? watchdog_timer_fn.cold.6+0x62/0x10a [14596.132367] ? __hrtimer_run_queues+0x1c2/0x4d0 [14596.134067] ? watchdog+0x40/0x40 [14596.135606] ? hrtimer_interrupt+0x128/0x2c0 [14596.137492] ? smp_apic_timer_interrupt+0xa6/0x270 [14596.139763] ? apic_timer_interrupt+0xf/0x20 [14596.141765] [14596.142785] ? native_safe_halt+0x2e/0x50 [14596.144601] kvm_wait+0xcd/0xe0 [14596.146016] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14596.148476] do_raw_spin_lock+0x177/0x190 [14596.150070] _raw_spin_lock+0x21/0x30 [14596.152250] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14596.155478] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14596.158659] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14596.161523] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14596.164702] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14596.167768] ? woken_wake_function+0x30/0x30 [14596.170024] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14596.173318] kthread+0x1d1/0x200 [14596.174809] ? set_kthread_struct+0x70/0x70 [14596.176777] ret_from_fork+0x1f/0x30 [14600.055023] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [14600.057016] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [14600.057849] Modules linked in: [14600.061542] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14600.061680] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14600.061687] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14600.061690] RIP: 0010:native_safe_halt+0x2e/0x50 [14600.061703] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14600.061707] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14600.061712] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14600.061714] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14600.061716] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14600.061719] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14600.061721] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14600.061725] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14600.061728] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14600.061729] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14600.061738] Call Trace: [14600.061743] [14600.061748] ? show_regs.cold.9+0x22/0x2f [14600.061757] ? watchdog_timer_fn.cold.6+0x62/0x10a [14600.061763] ? __hrtimer_run_queues+0x1c2/0x4d0 [14600.061767] ? watchdog+0x40/0x40 [14600.061770] ? hrtimer_interrupt+0x128/0x2c0 [14600.061774] ? smp_apic_timer_interrupt+0xa6/0x270 [14600.061781] ? apic_timer_interrupt+0xf/0x20 [14600.061784] [14600.061787] ? native_safe_halt+0x2e/0x50 [14600.061854] kvm_wait+0xcd/0xe0 [14600.061862] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14600.061872] do_raw_spin_lock+0x177/0x190 [14600.061959] _raw_spin_lock+0x21/0x30 [14600.061965] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14600.063525] lustre(O) [14600.084781] obd_mod_free+0x46/0x140 [ptlrpc] [14600.085611] mdc_commit_open+0x91/0xb0 [mdc] [14600.085646] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14600.085861] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14600.086177] after_reply+0xe2c/0x1780 [ptlrpc] [14600.086446] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14600.086633] ? __next_timer_interrupt+0x160/0x160 [14600.086643] ptlrpcd+0xa82/0xe40 [ptlrpc] [14600.087113] ? do_wait_intr+0xf0/0xf0 [14600.087122] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14600.087395] kthread+0x1d1/0x200 [14600.087402] ? set_kthread_struct+0x70/0x70 [14600.087406] ret_from_fork+0x1f/0x30 [14600.379669] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14600.414123] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14600.418087] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14600.421759] RIP: 0010:native_safe_halt+0x2e/0x50 [14600.423501] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14600.430694] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14600.433996] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14600.436990] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14600.441027] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14600.443480] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14600.464425] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14600.468571] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14600.475776] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14600.479762] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14600.484403] Call Trace: [14600.486353] [14600.487412] ? show_regs.cold.9+0x22/0x2f [14600.489185] ? watchdog_timer_fn.cold.6+0x62/0x10a [14600.491418] ? __hrtimer_run_queues+0x1c2/0x4d0 [14600.493547] ? watchdog+0x40/0x40 [14600.495126] ? hrtimer_interrupt+0x128/0x2c0 [14600.498595] ? smp_apic_timer_interrupt+0xa6/0x270 [14600.501090] ? apic_timer_interrupt+0xf/0x20 [14600.502758] [14600.503716] ? native_safe_halt+0x2e/0x50 [14600.510318] kvm_wait+0xcd/0xe0 [14600.511553] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14600.514032] do_raw_spin_lock+0x177/0x190 [14600.515976] _raw_spin_lock+0x21/0x30 [14600.519221] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14600.522024] ? do_raw_spin_unlock+0x75/0x190 [14600.526096] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14600.532517] ? do_wait_intr+0xf0/0xf0 [14600.537382] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14600.541306] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14600.545553] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14600.548382] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14600.550155] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14600.551997] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14600.553512] mdc_intent_lock+0x353/0x7e0 [mdc] [14600.555252] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14600.557262] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14600.559507] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14600.561567] lmv_intent_open+0x338/0x1150 [lmv] [14600.563584] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14600.565848] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14600.568264] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14600.570583] ll_intent_lock+0xc5/0xaa0 [lustre] [14600.572637] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14600.575203] ll_lookup_it+0xe0e/0x2270 [lustre] [14600.577116] ? do_raw_spin_unlock+0x75/0x190 [14600.578607] ? _raw_spin_unlock+0x12/0x30 [14600.580462] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14600.583046] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14600.585033] ll_atomic_open+0x36a/0xd70 [lustre] [14600.586959] lookup_open+0x356/0xb40 [14600.588328] do_last+0x313/0xfc0 [14600.589545] ? nd_jump_root+0xe5/0x160 [14600.590780] ? path_init+0x437/0x520 [14600.592193] path_openat+0xf7/0x500 [14600.593614] ? do_raw_spin_unlock+0x75/0x190 [14600.595329] do_filp_open+0x99/0x140 [14600.596673] ? getname_flags+0x6e/0x330 [14600.598292] ? __check_object_size+0xff/0x256 [14600.599840] ? do_raw_spin_unlock+0x75/0x190 [14600.601258] ? _raw_spin_unlock+0x12/0x30 [14600.603107] do_sys_openat2+0x2b4/0x410 [14600.604566] do_sys_open+0x73/0xa0 [14600.605850] __x64_sys_openat+0x24/0x30 [14600.607370] do_syscall_64+0xc1/0x440 [14600.608769] entry_SYSCALL_64_after_hwframe+0x49/0xae [14600.611364] RIP: 0033:0x7f53425a5332 [14600.612557] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14600.631339] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14600.639237] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14600.644503] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14600.652374] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14600.658325] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14600.664287] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14604.052249] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [14604.062632] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14604.134808] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14604.148392] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14604.163197] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14604.165597] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14604.174817] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14604.201041] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14604.205656] RAX: 0000000000005386 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14604.217532] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14604.226579] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14604.236089] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14604.239213] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14604.247582] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14604.265840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14604.272563] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14604.279941] Call Trace: [14604.283538] [14604.287457] ? show_regs.cold.9+0x22/0x2f [14604.294475] ? watchdog_timer_fn.cold.6+0x62/0x10a [14604.301293] ? __hrtimer_run_queues+0x1c2/0x4d0 [14604.314297] ? watchdog+0x40/0x40 [14604.321571] ? hrtimer_interrupt+0x128/0x2c0 [14604.334099] ? smp_apic_timer_interrupt+0xa6/0x270 [14604.338124] ? apic_timer_interrupt+0xf/0x20 [14604.351868] [14604.357803] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14604.376759] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14604.383418] ? newidle_balance.isra.93+0x207/0x580 [14604.394046] do_raw_spin_lock+0x177/0x190 [14604.395553] _raw_spin_lock+0x21/0x30 [14604.396932] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14604.420920] process_one_work+0x2c8/0x700 [14604.434783] worker_thread+0x296/0x6e0 [14604.440677] ? rescuer_thread+0x570/0x570 [14604.446874] kthread+0x1d1/0x200 [14604.455330] ? set_kthread_struct+0x70/0x70 [14604.469634] ret_from_fork+0x1f/0x30 [14608.354636] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 11971s! [14608.357489] Showing busy workqueues and worker pools: [14608.370360] workqueue events: flags=0x0 [14608.373575] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14608.383132] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14608.390454] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14608.395621] pending: cpu_latency_work [ptlrpc] [14608.401822] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14608.411599] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14608.427958] workqueue events_long: flags=0x0 [14608.434322] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14608.448463] pending: lu_objects_flush [obdclass] [14608.457869] workqueue events_unbound: flags=0x2 [14608.465828] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14608.468242] pending: flush_memcg_stats_dwork, idle_cull_fn [14608.477693] workqueue events_power_efficient: flags=0x80 [14608.483351] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14608.491144] pending: check_lifetime, do_cache_clean [14608.496348] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14608.500449] pending: neigh_periodic_work, neigh_periodic_work [14608.515175] workqueue mm_percpu_wq: flags=0x8 [14608.521579] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14608.537320] pending: vmstat_update, lru_add_drain_per_cpu [14608.548717] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14608.556324] pending: vmstat_update, lru_add_drain_per_cpu [14608.563381] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14608.571418] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14608.580203] workqueue xprtiod: flags=0xa [14608.586344] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14608.595606] pending: xprt_autoclose, xs_error_handle [14608.601299] workqueue ipv6_addrconf: flags=0x40008 [14608.607210] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14608.615243] pending: addrconf_verify_work [14608.629454] workqueue knbd0-recv: flags=0x1a [14608.632894] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14608.648110] in-flight: 651:recv_work [14608.662407] workqueue ptlrpc_pinger: flags=0x2 [14608.664054] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14608.677666] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14608.692806] pool 8: cpus=0-3 flags=0x4 nice=0 hung=11971s workers=4 idle: 15979 31824 8 [14608.707201] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14599s workers=2 idle: 654 [14624.058018] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [14624.063844] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14624.098105] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14624.104905] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14624.117101] RIP: 0010:native_safe_halt+0x2e/0x50 [14624.125670] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14624.145749] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14624.157727] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14624.165914] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14624.171406] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14624.174438] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14624.183092] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14624.189807] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14624.197646] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14624.202199] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14624.211039] Call Trace: [14624.211750] [14624.213251] ? show_regs.cold.9+0x22/0x2f [14624.214882] ? watchdog_timer_fn.cold.6+0x62/0x10a [14624.217090] ? __hrtimer_run_queues+0x1c2/0x4d0 [14624.220083] ? watchdog+0x40/0x40 [14624.222393] ? hrtimer_interrupt+0x128/0x2c0 [14624.224946] ? smp_apic_timer_interrupt+0xa6/0x270 [14624.229772] ? apic_timer_interrupt+0xf/0x20 [14624.231262] [14624.232174] ? native_safe_halt+0x2e/0x50 [14624.233897] kvm_wait+0xcd/0xe0 [14624.235110] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14624.237149] do_raw_spin_lock+0x177/0x190 [14624.238853] _raw_spin_lock+0x21/0x30 [14624.240250] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14624.243200] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14624.245881] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14624.249147] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14624.258703] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14624.260523] ? woken_wake_function+0x30/0x30 [14624.265045] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14624.269478] kthread+0x1d1/0x200 [14624.271280] ? set_kthread_struct+0x70/0x70 [14624.273324] ret_from_fork+0x1f/0x30 [14628.055600] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [14628.062021] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14628.062538] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [14628.094610] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14628.097194] Modules linked in: [14628.107980] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14628.107991] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14628.108006] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14628.108017] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14628.108025] RAX: 00000000000055f9 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14628.108029] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14628.108031] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14628.108034] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14628.108036] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14628.108040] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14628.108042] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14628.108045] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14628.108053] Call Trace: [14628.108059] [14628.108065] ? show_regs.cold.9+0x22/0x2f [14628.108074] ? watchdog_timer_fn.cold.6+0x62/0x10a [14628.108081] ? __hrtimer_run_queues+0x1c2/0x4d0 [14628.108086] ? watchdog+0x40/0x40 [14628.108090] ? hrtimer_interrupt+0x128/0x2c0 [14628.108094] ? smp_apic_timer_interrupt+0xa6/0x270 [14628.108103] ? apic_timer_interrupt+0xf/0x20 [14628.108107] [14628.108110] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14628.108118] do_raw_spin_lock+0x177/0x190 [14628.108123] _raw_spin_lock+0x21/0x30 [14628.108128] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14628.108413] ? do_raw_spin_unlock+0x75/0x190 [14628.108419] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14628.108621] ? do_wait_intr+0xf0/0xf0 [14628.108629] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14628.108839] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14628.109060] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14628.109294] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14628.112565] lustre(O) [14628.122005] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14628.124184] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14628.124254] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14628.124261] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14628.124264] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14628.124277] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14628.124282] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14628.124286] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14628.124289] RDX: 000000000000739c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14628.124291] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14628.124293] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14628.124295] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14628.124313] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14628.124316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14628.124319] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14628.124326] Call Trace: [14628.124331] [14628.124336] ? show_regs.cold.9+0x22/0x2f [14628.124345] ? watchdog_timer_fn.cold.6+0x62/0x10a [14628.124351] ? __hrtimer_run_queues+0x1c2/0x4d0 [14628.124354] ? watchdog+0x40/0x40 [14628.124358] ? hrtimer_interrupt+0x128/0x2c0 [14628.124362] ? smp_apic_timer_interrupt+0xa6/0x270 [14628.124370] ? apic_timer_interrupt+0xf/0x20 [14628.124375] [14628.124378] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14628.124391] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14628.124401] do_raw_spin_lock+0x177/0x190 [14628.124406] _raw_spin_lock+0x21/0x30 [14628.124412] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14628.124686] obd_mod_free+0x46/0x140 [ptlrpc] [14628.124894] mdc_commit_open+0x91/0xb0 [mdc] [14628.124932] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14628.125138] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14628.125894] after_reply+0xe2c/0x1780 [ptlrpc] [14628.126120] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14628.128241] ? __next_timer_interrupt+0x160/0x160 [14628.128253] ptlrpcd+0xa82/0xe40 [ptlrpc] [14628.129746] ? do_wait_intr+0xf0/0xf0 [14628.129760] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14628.131608] kthread+0x1d1/0x200 [14628.131615] ? set_kthread_struct+0x70/0x70 [14628.131620] ret_from_fork+0x1f/0x30 [14628.536400] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14628.539871] mdc_intent_lock+0x353/0x7e0 [mdc] [14628.545578] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14628.552518] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14628.559580] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14628.566005] lmv_intent_open+0x338/0x1150 [lmv] [14628.570286] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14628.572209] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14628.584531] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14628.592039] ll_intent_lock+0xc5/0xaa0 [lustre] [14628.602717] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14628.607688] ll_lookup_it+0xe0e/0x2270 [lustre] [14628.610685] ? do_raw_spin_unlock+0x75/0x190 [14628.612904] ? _raw_spin_unlock+0x12/0x30 [14628.614943] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14628.619233] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14628.622091] ll_atomic_open+0x36a/0xd70 [lustre] [14628.628541] lookup_open+0x356/0xb40 [14628.630652] do_last+0x313/0xfc0 [14628.634632] ? nd_jump_root+0xe5/0x160 [14628.637041] ? path_init+0x437/0x520 [14628.638374] path_openat+0xf7/0x500 [14628.644682] ? do_raw_spin_unlock+0x75/0x190 [14628.651561] do_filp_open+0x99/0x140 [14628.656558] ? getname_flags+0x6e/0x330 [14628.662528] ? __check_object_size+0xff/0x256 [14628.668193] ? do_raw_spin_unlock+0x75/0x190 [14628.670987] ? _raw_spin_unlock+0x12/0x30 [14628.676496] do_sys_openat2+0x2b4/0x410 [14628.679168] do_sys_open+0x73/0xa0 [14628.689347] __x64_sys_openat+0x24/0x30 [14628.692665] do_syscall_64+0xc1/0x440 [14628.695989] entry_SYSCALL_64_after_hwframe+0x49/0xae [14628.699053] RIP: 0033:0x7f53425a5332 [14628.703551] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14628.723848] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14628.728061] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14628.737568] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14628.743680] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14628.748362] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14628.754284] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14636.054457] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [14636.062608] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14636.096800] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14636.104458] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14636.108503] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14636.111702] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14636.113745] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14636.127700] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14636.130244] RAX: 0000000000007f84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14636.134402] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14636.138452] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14636.142833] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14636.151004] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14636.162051] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14636.174751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14636.181609] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14636.187178] Call Trace: [14636.188224] [14636.189351] ? show_regs.cold.9+0x22/0x2f [14636.190767] ? watchdog_timer_fn.cold.6+0x62/0x10a [14636.192500] ? __hrtimer_run_queues+0x1c2/0x4d0 [14636.194632] ? watchdog+0x40/0x40 [14636.198918] ? hrtimer_interrupt+0x128/0x2c0 [14636.203340] ? smp_apic_timer_interrupt+0xa6/0x270 [14636.207640] ? apic_timer_interrupt+0xf/0x20 [14636.213529] [14636.214393] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14636.221683] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14636.225288] ? newidle_balance.isra.93+0x207/0x580 [14636.230715] do_raw_spin_lock+0x177/0x190 [14636.239975] _raw_spin_lock+0x21/0x30 [14636.242071] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14636.244098] process_one_work+0x2c8/0x700 [14636.252295] worker_thread+0x296/0x6e0 [14636.260072] ? rescuer_thread+0x570/0x570 [14636.262491] kthread+0x1d1/0x200 [14636.263703] ? set_kthread_struct+0x70/0x70 [14636.270405] ret_from_fork+0x1f/0x30 [14639.076144] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12001s! [14639.093351] Showing busy workqueues and worker pools: [14639.103369] workqueue events: flags=0x0 [14639.104741] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14639.118384] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14639.131696] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14639.143780] pending: cpu_latency_work [ptlrpc] [14639.148909] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14639.154849] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14639.159989] workqueue events_long: flags=0x0 [14639.162643] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14639.166096] pending: lu_objects_flush [obdclass] [14639.168809] workqueue events_unbound: flags=0x2 [14639.172430] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14639.185815] pending: flush_memcg_stats_dwork, idle_cull_fn [14639.197909] workqueue events_power_efficient: flags=0x80 [14639.207309] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14639.220587] pending: check_lifetime, do_cache_clean [14639.238569] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14639.260370] pending: neigh_periodic_work, neigh_periodic_work [14639.275291] workqueue mm_percpu_wq: flags=0x8 [14639.280661] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14639.293535] pending: vmstat_update, lru_add_drain_per_cpu [14639.304382] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14639.314925] pending: vmstat_update, lru_add_drain_per_cpu [14639.322452] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14639.326963] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14639.346536] workqueue xprtiod: flags=0xa [14639.360019] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14639.372247] pending: xprt_autoclose, xs_error_handle [14639.379603] workqueue ipv6_addrconf: flags=0x40008 [14639.387468] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14639.403525] pending: addrconf_verify_work [14639.408917] workqueue knbd0-recv: flags=0x1a [14639.410576] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14639.415724] in-flight: 651:recv_work [14639.421984] workqueue ptlrpc_pinger: flags=0x2 [14639.429070] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14639.436054] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14639.448939] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12002s workers=4 idle: 15979 31824 8 [14639.463533] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14629s workers=2 idle: 654 [14652.058182] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [14652.062159] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14652.109692] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14652.121876] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14652.127586] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14652.133565] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14652.157208] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14652.165417] RAX: 000000000000408f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14652.168569] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14652.177075] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14652.182510] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14652.190138] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14652.193451] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14652.200559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14652.206485] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14652.213902] Call Trace: [14652.218021] [14652.219834] ? show_regs.cold.9+0x22/0x2f [14652.224047] ? watchdog_timer_fn.cold.6+0x62/0x10a [14652.228818] ? __hrtimer_run_queues+0x1c2/0x4d0 [14652.234183] ? watchdog+0x40/0x40 [14652.235491] ? hrtimer_interrupt+0x128/0x2c0 [14652.241305] ? smp_apic_timer_interrupt+0xa6/0x270 [14652.246059] ? apic_timer_interrupt+0xf/0x20 [14652.248391] [14652.251445] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14652.254943] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14652.258499] do_raw_spin_lock+0x177/0x190 [14652.260155] _raw_spin_lock+0x21/0x30 [14652.261610] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14652.264493] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14652.266869] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14652.269401] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14652.271902] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14652.274371] ? woken_wake_function+0x30/0x30 [14652.276858] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14652.282285] kthread+0x1d1/0x200 [14652.285905] ? set_kthread_struct+0x70/0x70 [14652.287545] ret_from_fork+0x1f/0x30 [14656.055021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [14656.059940] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [14656.061088] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14656.061169] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14656.061176] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14656.061180] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14656.061199] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14656.061203] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14656.061208] RAX: 0000000000001885 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14656.061210] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14656.061212] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14656.061214] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14656.061217] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14656.061220] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14656.061222] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14656.061225] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14656.061233] Call Trace: [14656.061239] [14656.061250] ? show_regs.cold.9+0x22/0x2f [14656.061339] ? watchdog_timer_fn.cold.6+0x62/0x10a [14656.061350] ? __hrtimer_run_queues+0x1c2/0x4d0 [14656.061355] ? watchdog+0x40/0x40 [14656.061359] ? hrtimer_interrupt+0x128/0x2c0 [14656.061363] ? smp_apic_timer_interrupt+0xa6/0x270 [14656.061372] ? apic_timer_interrupt+0xf/0x20 [14656.061376] [14656.061379] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14656.061386] do_raw_spin_lock+0x177/0x190 [14656.061391] _raw_spin_lock+0x21/0x30 [14656.061453] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14656.061917] ? do_raw_spin_unlock+0x75/0x190 [14656.062003] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14656.062217] ? do_wait_intr+0xf0/0xf0 [14656.062225] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14656.076204] Modules linked in: lustre(O) [14656.103173] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14656.116545] osp(O) [14656.121341] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14656.121637] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14656.122044] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14656.122085] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14656.122116] mdc_intent_lock+0x353/0x7e0 [mdc] [14656.122146] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14656.122230] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14656.123432] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14656.123470] lmv_intent_open+0x338/0x1150 [lmv] [14656.123504] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14656.123611] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14656.123694] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14656.123718] ll_intent_lock+0xc5/0xaa0 [lustre] [14656.124239] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14656.124332] ll_lookup_it+0xe0e/0x2270 [lustre] [14656.124415] ? do_raw_spin_unlock+0x75/0x190 [14656.124424] ? _raw_spin_unlock+0x12/0x30 [14656.124431] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14656.124458] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14656.124478] ll_atomic_open+0x36a/0xd70 [lustre] [14656.124562] lookup_open+0x356/0xb40 [14656.124571] do_last+0x313/0xfc0 [14656.124576] ? nd_jump_root+0xe5/0x160 [14656.124580] ? path_init+0x437/0x520 [14656.124585] path_openat+0xf7/0x500 [14656.124589] ? do_raw_spin_unlock+0x75/0x190 [14656.124594] do_filp_open+0x99/0x140 [14656.124600] ? getname_flags+0x6e/0x330 [14656.124605] ? __check_object_size+0xff/0x256 [14656.124614] ? do_raw_spin_unlock+0x75/0x190 [14656.124618] ? _raw_spin_unlock+0x12/0x30 [14656.124622] do_sys_openat2+0x2b4/0x410 [14656.124626] do_sys_open+0x73/0xa0 [14656.124631] __x64_sys_openat+0x24/0x30 [14656.124634] do_syscall_64+0xc1/0x440 [14656.124641] entry_SYSCALL_64_after_hwframe+0x49/0xae [14656.124648] RIP: 0033:0x7f53425a5332 [14656.124656] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14656.124660] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14656.124665] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14656.124667] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14656.124669] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14656.124671] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14656.124674] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14656.539871] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14656.560905] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14656.564875] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14656.572248] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14656.574343] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14656.582974] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14656.585779] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14656.588960] RDX: 0000000000002e18 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14656.592912] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14656.595729] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14656.598874] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14656.602002] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14656.605600] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14656.608198] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14656.611658] Call Trace: [14656.613012] [14656.613891] ? show_regs.cold.9+0x22/0x2f [14656.615546] ? watchdog_timer_fn.cold.6+0x62/0x10a [14656.617812] ? __hrtimer_run_queues+0x1c2/0x4d0 [14656.619727] ? watchdog+0x40/0x40 [14656.621491] ? hrtimer_interrupt+0x128/0x2c0 [14656.623406] ? smp_apic_timer_interrupt+0xa6/0x270 [14656.626896] ? apic_timer_interrupt+0xf/0x20 [14656.628757] [14656.629592] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14656.632352] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14656.634615] do_raw_spin_lock+0x177/0x190 [14656.637090] _raw_spin_lock+0x21/0x30 [14656.638705] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14656.641935] obd_mod_free+0x46/0x140 [ptlrpc] [14656.644891] mdc_commit_open+0x91/0xb0 [mdc] [14656.647638] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14656.650258] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14656.652753] after_reply+0xe2c/0x1780 [ptlrpc] [14656.656371] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14656.659796] ? __next_timer_interrupt+0x160/0x160 [14656.661789] ptlrpcd+0xa82/0xe40 [ptlrpc] [14656.664949] ? do_wait_intr+0xf0/0xf0 [14656.666905] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14656.669486] kthread+0x1d1/0x200 [14656.670880] ? set_kthread_struct+0x70/0x70 [14656.672893] ret_from_fork+0x1f/0x30 [14664.059155] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [14664.072834] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14664.127522] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14664.136983] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14664.143575] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14664.147595] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14664.150359] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14664.172130] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14664.183192] RAX: 0000000000004401 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14664.190984] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14664.200026] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14664.205454] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14664.210296] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14664.222544] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14664.229693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14664.236739] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14664.240602] Call Trace: [14664.253659] [14664.254415] ? show_regs.cold.9+0x22/0x2f [14664.264648] ? watchdog_timer_fn.cold.6+0x62/0x10a [14664.267778] ? __hrtimer_run_queues+0x1c2/0x4d0 [14664.271923] ? watchdog+0x40/0x40 [14664.273365] ? hrtimer_interrupt+0x128/0x2c0 [14664.275262] ? smp_apic_timer_interrupt+0xa6/0x270 [14664.278624] ? apic_timer_interrupt+0xf/0x20 [14664.281903] [14664.282889] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14664.286626] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14664.290212] ? newidle_balance.isra.93+0x207/0x580 [14664.294535] do_raw_spin_lock+0x177/0x190 [14664.298248] _raw_spin_lock+0x21/0x30 [14664.305361] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14664.308724] process_one_work+0x2c8/0x700 [14664.311184] worker_thread+0x296/0x6e0 [14664.314353] ? rescuer_thread+0x570/0x570 [14664.315842] kthread+0x1d1/0x200 [14664.317082] ? set_kthread_struct+0x70/0x70 [14664.323646] ret_from_fork+0x1f/0x30 [14669.792038] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12032s! [14669.795321] Showing busy workqueues and worker pools: [14669.797431] workqueue events: flags=0x0 [14669.799011] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14669.801663] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14669.806182] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14669.810165] pending: cpu_latency_work [ptlrpc] [14669.813311] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14669.820360] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14669.829135] workqueue events_long: flags=0x0 [14669.831983] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14669.835475] pending: lu_objects_flush [obdclass] [14669.840961] workqueue events_unbound: flags=0x2 [14669.845073] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14669.849297] pending: flush_memcg_stats_dwork, idle_cull_fn [14669.851811] workqueue events_power_efficient: flags=0x80 [14669.854015] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14669.856827] pending: check_lifetime, do_cache_clean [14669.859053] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14669.861870] pending: neigh_periodic_work, neigh_periodic_work [14669.864356] workqueue mm_percpu_wq: flags=0x8 [14669.866198] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14669.868918] pending: vmstat_update, lru_add_drain_per_cpu [14669.871155] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14669.873796] pending: vmstat_update, lru_add_drain_per_cpu [14669.876009] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14669.878492] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14669.880949] workqueue xprtiod: flags=0xa [14669.882329] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14669.884678] pending: xprt_autoclose, xs_error_handle [14669.886704] workqueue ipv6_addrconf: flags=0x40008 [14669.888491] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14669.890915] pending: addrconf_verify_work [14669.892530] workqueue knbd0-recv: flags=0x1a [14669.894232] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14669.896710] in-flight: 651:recv_work [14669.898198] workqueue ptlrpc_pinger: flags=0x2 [14669.899868] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14669.902196] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14669.904678] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12032s workers=4 idle: 15979 31824 8 [14669.907555] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14660s workers=2 idle: 654 [14680.060984] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [14680.073586] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14680.151978] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14680.178124] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14680.194997] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14680.206176] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14680.240775] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14680.250464] RAX: 0000000000007586 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14680.265904] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14680.276587] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14680.280165] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14680.289077] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14680.307806] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14680.314008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14680.325223] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14680.330519] Call Trace: [14680.335644] [14680.336346] ? show_regs.cold.9+0x22/0x2f [14680.345043] ? watchdog_timer_fn.cold.6+0x62/0x10a [14680.356235] ? __hrtimer_run_queues+0x1c2/0x4d0 [14680.368145] ? watchdog+0x40/0x40 [14680.371319] ? hrtimer_interrupt+0x128/0x2c0 [14680.372850] ? smp_apic_timer_interrupt+0xa6/0x270 [14680.380936] ? apic_timer_interrupt+0xf/0x20 [14680.386956] [14680.390201] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14680.394341] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14680.397142] do_raw_spin_lock+0x177/0x190 [14680.403081] _raw_spin_lock+0x21/0x30 [14680.410010] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14680.412885] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14680.417658] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14680.422809] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14680.432751] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14680.440144] ? woken_wake_function+0x30/0x30 [14680.452143] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14680.464488] kthread+0x1d1/0x200 [14680.468674] ? set_kthread_struct+0x70/0x70 [14680.470961] ret_from_fork+0x1f/0x30 [14684.057096] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [14684.057853] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [14684.072951] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14684.073034] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14684.073042] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14684.073045] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14684.073059] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14684.073063] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14684.073068] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14684.073070] RDX: 0000000000004e56 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14684.073073] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14684.073075] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14684.073077] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14684.073080] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14684.073083] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14684.073085] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14684.073094] Call Trace: [14684.073099] [14684.073105] ? show_regs.cold.9+0x22/0x2f [14684.073115] ? watchdog_timer_fn.cold.6+0x62/0x10a [14684.073121] ? __hrtimer_run_queues+0x1c2/0x4d0 [14684.073125] ? watchdog+0x40/0x40 [14684.073128] ? hrtimer_interrupt+0x128/0x2c0 [14684.073132] ? smp_apic_timer_interrupt+0xa6/0x270 [14684.073140] ? apic_timer_interrupt+0xf/0x20 [14684.073145] [14684.073148] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14684.073152] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14684.073157] do_raw_spin_lock+0x177/0x190 [14684.073161] _raw_spin_lock+0x21/0x30 [14684.073180] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14684.078028] obd_mod_free+0x46/0x140 [ptlrpc] [14684.081638] mdc_commit_open+0x91/0xb0 [mdc] [14684.081676] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14684.084272] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14684.086117] after_reply+0xe2c/0x1780 [ptlrpc] [14684.088748] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14684.090129] ? __next_timer_interrupt+0x160/0x160 [14684.090155] ptlrpcd+0xa82/0xe40 [ptlrpc] [14684.108988] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14684.109061] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14684.109069] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14684.109072] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14684.109086] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14684.109090] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14684.109094] RAX: 0000000000007e82 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14684.109097] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14684.109099] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14684.109113] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14684.109116] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14684.109120] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14684.109123] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14684.109125] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14684.109134] Call Trace: [14684.109140] [14684.109146] ? show_regs.cold.9+0x22/0x2f [14684.109156] ? watchdog_timer_fn.cold.6+0x62/0x10a [14684.109162] ? __hrtimer_run_queues+0x1c2/0x4d0 [14684.109167] ? watchdog+0x40/0x40 [14684.109173] ? hrtimer_interrupt+0x128/0x2c0 [14684.109177] ? smp_apic_timer_interrupt+0xa6/0x270 [14684.109187] ? apic_timer_interrupt+0xf/0x20 [14684.109191] [14684.109195] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14684.109202] do_raw_spin_lock+0x177/0x190 [14684.109208] _raw_spin_lock+0x21/0x30 [14684.109212] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14684.109501] ? do_raw_spin_unlock+0x75/0x190 [14684.109507] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14684.109703] ? do_wait_intr+0xf0/0xf0 [14684.109724] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14684.109953] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14684.110185] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14684.110403] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14684.110514] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14684.110556] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14684.210941] ? do_wait_intr+0xf0/0xf0 [14684.210963] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14684.240254] mdc_intent_lock+0x353/0x7e0 [mdc] [14684.240307] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14684.242091] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14684.264258] kthread+0x1d1/0x200 [14684.264278] ? set_kthread_struct+0x70/0x70 [14684.264282] ret_from_fork+0x1f/0x30 [14685.174473] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14685.180889] lmv_intent_open+0x338/0x1150 [lmv] [14685.194318] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14685.209767] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14685.215873] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14685.218519] ll_intent_lock+0xc5/0xaa0 [lustre] [14685.221945] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14685.225080] ll_lookup_it+0xe0e/0x2270 [lustre] [14685.228430] ? do_raw_spin_unlock+0x75/0x190 [14685.230974] ? _raw_spin_unlock+0x12/0x30 [14685.232968] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14685.236258] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14685.244938] ll_atomic_open+0x36a/0xd70 [lustre] [14685.256179] lookup_open+0x356/0xb40 [14685.259223] do_last+0x313/0xfc0 [14685.264908] ? nd_jump_root+0xe5/0x160 [14685.268105] ? path_init+0x437/0x520 [14685.274732] path_openat+0xf7/0x500 [14685.275966] ? do_raw_spin_unlock+0x75/0x190 [14685.286501] do_filp_open+0x99/0x140 [14685.289497] ? getname_flags+0x6e/0x330 [14685.296459] ? __check_object_size+0xff/0x256 [14685.302128] ? do_raw_spin_unlock+0x75/0x190 [14685.309372] ? _raw_spin_unlock+0x12/0x30 [14685.314050] do_sys_openat2+0x2b4/0x410 [14685.317667] do_sys_open+0x73/0xa0 [14685.323058] __x64_sys_openat+0x24/0x30 [14685.326821] do_syscall_64+0xc1/0x440 [14685.333640] entry_SYSCALL_64_after_hwframe+0x49/0xae [14685.337283] RIP: 0033:0x7f53425a5332 [14685.339678] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14685.354991] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14685.359966] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14685.366521] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14685.371389] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14685.383509] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14685.386043] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14696.053532] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [14696.060526] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14696.111882] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14696.147430] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14696.165825] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14696.180856] RIP: 0010:native_safe_halt+0x2e/0x50 [14696.189465] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14696.211555] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14696.226096] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14696.237925] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14696.246283] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14696.256371] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14696.269367] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14696.280850] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14696.293948] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14696.297980] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14696.302821] Call Trace: [14696.305956] [14696.309609] ? show_regs.cold.9+0x22/0x2f [14696.314902] ? watchdog_timer_fn.cold.6+0x62/0x10a [14696.323929] ? __hrtimer_run_queues+0x1c2/0x4d0 [14696.330052] ? watchdog+0x40/0x40 [14696.334064] ? hrtimer_interrupt+0x128/0x2c0 [14696.342280] ? smp_apic_timer_interrupt+0xa6/0x270 [14696.346991] ? apic_timer_interrupt+0xf/0x20 [14696.356919] [14696.359248] ? native_safe_halt+0x2e/0x50 [14696.365296] kvm_wait+0xcd/0xe0 [14696.369267] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14696.370992] ? newidle_balance.isra.93+0x207/0x580 [14696.378203] do_raw_spin_lock+0x177/0x190 [14696.380840] _raw_spin_lock+0x21/0x30 [14696.387871] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14696.397189] process_one_work+0x2c8/0x700 [14696.400520] worker_thread+0x296/0x6e0 [14696.403576] ? rescuer_thread+0x570/0x570 [14696.408477] kthread+0x1d1/0x200 [14696.410590] ? set_kthread_struct+0x70/0x70 [14696.416653] ret_from_fork+0x1f/0x30 [14700.001055] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12062s! [14700.017407] Showing busy workqueues and worker pools: [14700.023601] workqueue events: flags=0x0 [14700.027285] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14700.031543] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14700.041956] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14700.046159] pending: cpu_latency_work [ptlrpc] [14700.049515] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14700.052809] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14700.057271] workqueue events_long: flags=0x0 [14700.059211] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14700.062799] pending: lu_objects_flush [obdclass] [14700.069037] workqueue events_unbound: flags=0x2 [14700.072895] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14700.076460] pending: flush_memcg_stats_dwork, idle_cull_fn [14700.078252] workqueue events_power_efficient: flags=0x80 [14700.080117] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14700.082506] pending: check_lifetime, do_cache_clean [14700.084149] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14700.086289] pending: neigh_periodic_work, neigh_periodic_work [14700.088430] workqueue mm_percpu_wq: flags=0x8 [14700.089855] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14700.092138] pending: vmstat_update, lru_add_drain_per_cpu [14700.093955] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14700.095747] pending: vmstat_update, lru_add_drain_per_cpu [14700.099099] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14700.140636] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14700.154784] workqueue xprtiod: flags=0xa [14700.155986] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14700.157784] pending: xprt_autoclose, xs_error_handle [14700.159399] workqueue ipv6_addrconf: flags=0x40008 [14700.161277] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14700.163252] pending: addrconf_verify_work [14700.164567] workqueue knbd0-recv: flags=0x1a [14700.165716] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14700.167553] in-flight: 651:recv_work [14700.168532] workqueue ptlrpc_pinger: flags=0x2 [14700.169574] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14700.171129] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14700.173020] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12063s workers=4 idle: 15979 31824 8 [14700.175546] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14690s workers=2 idle: 654 [14708.058161] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [14708.060920] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14708.076636] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14708.080678] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14708.083658] RIP: 0010:native_safe_halt+0x2e/0x50 [14708.085107] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14708.090751] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14708.092774] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14708.094863] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14708.097059] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14708.099197] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14708.101258] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14708.103310] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14708.105607] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14708.107445] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14708.109529] Call Trace: [14708.110378] [14708.110957] ? show_regs.cold.9+0x22/0x2f [14708.112208] ? watchdog_timer_fn.cold.6+0x62/0x10a [14708.113625] ? __hrtimer_run_queues+0x1c2/0x4d0 [14708.115114] ? watchdog+0x40/0x40 [14708.116063] ? hrtimer_interrupt+0x128/0x2c0 [14708.117385] ? smp_apic_timer_interrupt+0xa6/0x270 [14708.118741] ? apic_timer_interrupt+0xf/0x20 [14708.120161] [14708.121152] ? native_safe_halt+0x2e/0x50 [14708.122398] kvm_wait+0xcd/0xe0 [14708.123398] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14708.125030] do_raw_spin_lock+0x177/0x190 [14708.126346] _raw_spin_lock+0x21/0x30 [14708.127480] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14708.129609] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14708.131398] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14708.133267] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14708.134870] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14708.136701] ? woken_wake_function+0x30/0x30 [14708.138173] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14708.140294] kthread+0x1d1/0x200 [14708.141376] ? set_kthread_struct+0x70/0x70 [14708.142676] ret_from_fork+0x1f/0x30 [14712.055032] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [14712.057022] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [14712.058049] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14712.058128] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14712.058134] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14712.058137] RIP: 0010:native_safe_halt+0x2e/0x50 [14712.058150] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14712.058153] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14712.058159] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14712.058161] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14712.058163] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14712.058166] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14712.058168] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14712.058171] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14712.058175] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14712.058177] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14712.058185] Call Trace: [14712.058190] [14712.058195] ? show_regs.cold.9+0x22/0x2f [14712.058204] ? watchdog_timer_fn.cold.6+0x62/0x10a [14712.058210] ? __hrtimer_run_queues+0x1c2/0x4d0 [14712.058214] ? watchdog+0x40/0x40 [14712.058219] ? hrtimer_interrupt+0x128/0x2c0 [14712.058223] ? smp_apic_timer_interrupt+0xa6/0x270 [14712.058228] ? apic_timer_interrupt+0xf/0x20 [14712.058232] [14712.058234] ? native_safe_halt+0x2e/0x50 [14712.058238] kvm_wait+0xcd/0xe0 [14712.058244] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14712.058253] do_raw_spin_lock+0x177/0x190 [14712.058256] _raw_spin_lock+0x21/0x30 [14712.058260] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14712.058532] ? do_raw_spin_unlock+0x75/0x190 [14712.058538] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14712.058737] ? do_wait_intr+0xf0/0xf0 [14712.058746] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14712.058993] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14712.059199] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14712.059413] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14712.059523] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14712.059560] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14712.059589] mdc_intent_lock+0x353/0x7e0 [mdc] [14712.059616] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14712.059700] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14712.059924] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14712.059954] lmv_intent_open+0x338/0x1150 [lmv] [14712.059984] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14712.060068] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14712.060150] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14712.060172] ll_intent_lock+0xc5/0xaa0 [lustre] [14712.060255] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14712.060336] ll_lookup_it+0xe0e/0x2270 [lustre] [14712.060419] ? do_raw_spin_unlock+0x75/0x190 [14712.060425] ? _raw_spin_unlock+0x12/0x30 [14712.060432] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14712.060460] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14712.060481] ll_atomic_open+0x36a/0xd70 [lustre] [14712.060565] lookup_open+0x356/0xb40 [14712.060577] do_last+0x313/0xfc0 [14712.060582] ? nd_jump_root+0xe5/0x160 [14712.060587] ? path_init+0x437/0x520 [14712.060592] path_openat+0xf7/0x500 [14712.060597] ? do_raw_spin_unlock+0x75/0x190 [14712.060602] do_filp_open+0x99/0x140 [14712.060607] ? getname_flags+0x6e/0x330 [14712.060612] ? __check_object_size+0xff/0x256 [14712.060619] ? do_raw_spin_unlock+0x75/0x190 [14712.060623] ? _raw_spin_unlock+0x12/0x30 [14712.060628] do_sys_openat2+0x2b4/0x410 [14712.060632] do_sys_open+0x73/0xa0 [14712.060636] __x64_sys_openat+0x24/0x30 [14712.060640] do_syscall_64+0xc1/0x440 [14712.060646] entry_SYSCALL_64_after_hwframe+0x49/0xae [14712.060653] RIP: 0033:0x7f53425a5332 [14712.060659] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14712.060663] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14712.060668] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14712.060671] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14712.060673] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14712.060675] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14712.060677] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14712.472939] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14712.508664] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14712.517607] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14712.532307] RIP: 0010:native_safe_halt+0x2e/0x50 [14712.538847] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14712.560962] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14712.565759] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14712.572555] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14712.578456] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14712.583675] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14712.587612] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14712.597966] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14712.605571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14712.613037] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14712.617890] Call Trace: [14712.619075] [14712.620960] ? show_regs.cold.9+0x22/0x2f [14712.629034] ? watchdog_timer_fn.cold.6+0x62/0x10a [14712.632766] ? __hrtimer_run_queues+0x1c2/0x4d0 [14712.638056] ? watchdog+0x40/0x40 [14712.643321] ? hrtimer_interrupt+0x128/0x2c0 [14712.645009] ? smp_apic_timer_interrupt+0xa6/0x270 [14712.646715] ? apic_timer_interrupt+0xf/0x20 [14712.648843] [14712.650031] ? native_safe_halt+0x2e/0x50 [14712.651980] kvm_wait+0xcd/0xe0 [14712.653688] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14712.661022] do_raw_spin_lock+0x177/0x190 [14712.662400] _raw_spin_lock+0x21/0x30 [14712.663576] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14712.675351] obd_mod_free+0x46/0x140 [ptlrpc] [14712.684952] mdc_commit_open+0x91/0xb0 [mdc] [14712.689577] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14712.692276] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14712.694336] after_reply+0xe2c/0x1780 [ptlrpc] [14712.697685] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14712.700958] ? __next_timer_interrupt+0x160/0x160 [14712.703369] ptlrpcd+0xa82/0xe40 [ptlrpc] [14712.706207] ? do_wait_intr+0xf0/0xf0 [14712.710094] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14712.712931] kthread+0x1d1/0x200 [14712.715287] ? set_kthread_struct+0x70/0x70 [14712.718242] ret_from_fork+0x1f/0x30 [14728.052123] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [14728.057822] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14728.083420] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14728.090890] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14728.096855] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14728.103932] RIP: 0010:native_safe_halt+0x2e/0x50 [14728.108220] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14728.119725] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [14728.123766] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14728.126988] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [14728.130268] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14728.133848] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14728.136556] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14728.139503] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14728.142487] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14728.144470] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14728.147805] Call Trace: [14728.149043] [14728.149956] ? show_regs.cold.9+0x22/0x2f [14728.151465] ? watchdog_timer_fn.cold.6+0x62/0x10a [14728.153957] ? __hrtimer_run_queues+0x1c2/0x4d0 [14728.155857] ? watchdog+0x40/0x40 [14728.157251] ? hrtimer_interrupt+0x128/0x2c0 [14728.158978] ? smp_apic_timer_interrupt+0xa6/0x270 [14728.160362] ? apic_timer_interrupt+0xf/0x20 [14728.161597] [14728.162161] ? native_safe_halt+0x2e/0x50 [14728.163966] kvm_wait+0xcd/0xe0 [14728.165674] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14728.167331] ? newidle_balance.isra.93+0x207/0x580 [14728.168713] do_raw_spin_lock+0x177/0x190 [14728.169693] _raw_spin_lock+0x21/0x30 [14728.170759] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14728.172419] process_one_work+0x2c8/0x700 [14728.173697] worker_thread+0x296/0x6e0 [14728.175057] ? rescuer_thread+0x570/0x570 [14728.176157] kthread+0x1d1/0x200 [14728.177027] ? set_kthread_struct+0x70/0x70 [14728.178213] ret_from_fork+0x1f/0x30 [14730.213428] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12093s! [14730.232088] Showing busy workqueues and worker pools: [14730.235986] workqueue events: flags=0x0 [14730.241229] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14730.257990] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14730.268381] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14730.283447] pending: cpu_latency_work [ptlrpc] [14730.293717] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14730.302459] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14730.311930] workqueue events_long: flags=0x0 [14730.314884] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14730.328808] pending: lu_objects_flush [obdclass] [14730.330727] workqueue events_unbound: flags=0x2 [14730.347796] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14730.355191] pending: flush_memcg_stats_dwork, idle_cull_fn [14730.363401] workqueue events_power_efficient: flags=0x80 [14730.372361] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14730.388091] pending: check_lifetime, do_cache_clean [14730.392733] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14730.408409] pending: neigh_periodic_work, neigh_periodic_work [14730.423278] workqueue mm_percpu_wq: flags=0x8 [14730.435250] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14730.447340] pending: vmstat_update, lru_add_drain_per_cpu [14730.460139] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14730.479107] pending: vmstat_update, lru_add_drain_per_cpu [14730.489117] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14730.496896] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14730.516626] workqueue xprtiod: flags=0xa [14730.523591] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14730.531490] pending: xprt_autoclose, xs_error_handle [14730.540673] workqueue ipv6_addrconf: flags=0x40008 [14730.549491] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14730.555140] pending: addrconf_verify_work [14730.560719] workqueue knbd0-recv: flags=0x1a [14730.563598] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14730.573660] in-flight: 651:recv_work [14730.579258] workqueue ptlrpc_pinger: flags=0x2 [14730.589766] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14730.594020] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14730.597522] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12093s workers=4 idle: 15979 31824 8 [14730.608081] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14721s workers=2 idle: 654 [14736.065817] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [14736.077433] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14736.107400] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14736.117203] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14736.121640] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14736.124892] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14736.145625] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14736.150182] RAX: 0000000000000202 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14736.154277] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14736.157965] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14736.163002] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14736.170310] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14736.178400] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14736.187719] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14736.194015] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14736.199921] Call Trace: [14736.200833] [14736.202307] ? show_regs.cold.9+0x22/0x2f [14736.204506] ? watchdog_timer_fn.cold.6+0x62/0x10a [14736.207470] ? __hrtimer_run_queues+0x1c2/0x4d0 [14736.209023] ? watchdog+0x40/0x40 [14736.210134] ? hrtimer_interrupt+0x128/0x2c0 [14736.213494] ? smp_apic_timer_interrupt+0xa6/0x270 [14736.221756] ? apic_timer_interrupt+0xf/0x20 [14736.224779] [14736.228807] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14736.232010] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14736.235656] do_raw_spin_lock+0x177/0x190 [14736.239890] _raw_spin_lock+0x21/0x30 [14736.241311] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14736.249280] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14736.253263] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14736.259210] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14736.265509] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14736.276031] ? woken_wake_function+0x30/0x30 [14736.277570] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14736.284775] kthread+0x1d1/0x200 [14736.286106] ? set_kthread_struct+0x70/0x70 [14736.289855] ret_from_fork+0x1f/0x30 [14740.055031] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [14740.057034] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [14740.059707] Modules linked in: [14740.063323] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) [14740.065626] lustre(O) osp(O) [14740.070012] mgs(O) lquota(O) lfsck(O) [14740.071930] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14740.071999] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14740.072006] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14740.072009] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14740.072023] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14740.072027] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14740.072032] RAX: 00000000000041b3 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14740.072035] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14740.072037] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14740.072039] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14740.072042] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14740.072045] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14740.072048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14740.072051] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14740.072059] Call Trace: [14740.072065] [14740.072071] ? show_regs.cold.9+0x22/0x2f [14740.072080] ? watchdog_timer_fn.cold.6+0x62/0x10a [14740.072086] ? __hrtimer_run_queues+0x1c2/0x4d0 [14740.072090] ? watchdog+0x40/0x40 [14740.072094] ? hrtimer_interrupt+0x128/0x2c0 [14740.072098] ? smp_apic_timer_interrupt+0xa6/0x270 [14740.072105] ? apic_timer_interrupt+0xf/0x20 [14740.072110] [14740.072112] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14740.072118] do_raw_spin_lock+0x177/0x190 [14740.072123] _raw_spin_lock+0x21/0x30 [14740.072127] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14740.072461] ? do_raw_spin_unlock+0x75/0x190 [14740.072467] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14740.076165] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14740.076237] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14740.076246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14740.076249] RIP: 0010:native_safe_halt+0x2e/0x50 [14740.076262] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14740.076266] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14740.076271] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14740.076274] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14740.076276] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14740.076279] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14740.076281] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14740.076285] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14740.076288] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14740.076290] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14740.076298] Call Trace: [14740.076304] [14740.076310] ? show_regs.cold.9+0x22/0x2f [14740.076320] ? watchdog_timer_fn.cold.6+0x62/0x10a [14740.076326] ? __hrtimer_run_queues+0x1c2/0x4d0 [14740.076331] ? watchdog+0x40/0x40 [14740.076335] ? hrtimer_interrupt+0x128/0x2c0 [14740.076350] ? smp_apic_timer_interrupt+0xa6/0x270 [14740.076357] ? apic_timer_interrupt+0xf/0x20 [14740.076362] [14740.076365] ? native_safe_halt+0x2e/0x50 [14740.076369] kvm_wait+0xcd/0xe0 [14740.076378] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14740.076385] do_raw_spin_lock+0x177/0x190 [14740.076391] _raw_spin_lock+0x21/0x30 [14740.076396] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14740.076667] obd_mod_free+0x46/0x140 [ptlrpc] [14740.076862] mdc_commit_open+0x91/0xb0 [mdc] [14740.076909] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14740.077103] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14740.077297] after_reply+0xe2c/0x1780 [ptlrpc] [14740.077505] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14740.077700] ? __next_timer_interrupt+0x160/0x160 [14740.077709] ptlrpcd+0xa82/0xe40 [ptlrpc] [14740.077913] ? do_wait_intr+0xf0/0xf0 [14740.077923] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14740.078125] kthread+0x1d1/0x200 [14740.078132] ? set_kthread_struct+0x70/0x70 [14740.078136] ret_from_fork+0x1f/0x30 [14740.420404] ? do_wait_intr+0xf0/0xf0 [14740.421745] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14740.425550] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14740.429429] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14740.434805] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14740.446636] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14740.449657] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14740.451330] mdc_intent_lock+0x353/0x7e0 [mdc] [14740.458906] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14740.461862] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14740.468419] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14740.472004] lmv_intent_open+0x338/0x1150 [lmv] [14740.475199] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14740.479705] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14740.483152] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14740.488298] ll_intent_lock+0xc5/0xaa0 [lustre] [14740.492090] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14740.498214] ll_lookup_it+0xe0e/0x2270 [lustre] [14740.505031] ? do_raw_spin_unlock+0x75/0x190 [14740.507727] ? _raw_spin_unlock+0x12/0x30 [14740.509736] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14740.513125] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14740.516231] ll_atomic_open+0x36a/0xd70 [lustre] [14740.519239] lookup_open+0x356/0xb40 [14740.521522] do_last+0x313/0xfc0 [14740.524829] ? nd_jump_root+0xe5/0x160 [14740.526302] ? path_init+0x437/0x520 [14740.528804] path_openat+0xf7/0x500 [14740.533807] ? do_raw_spin_unlock+0x75/0x190 [14740.536475] do_filp_open+0x99/0x140 [14740.538088] ? getname_flags+0x6e/0x330 [14740.540226] ? __check_object_size+0xff/0x256 [14740.546077] ? do_raw_spin_unlock+0x75/0x190 [14740.547901] ? _raw_spin_unlock+0x12/0x30 [14740.550609] do_sys_openat2+0x2b4/0x410 [14740.552859] do_sys_open+0x73/0xa0 [14740.555913] __x64_sys_openat+0x24/0x30 [14740.560633] do_syscall_64+0xc1/0x440 [14740.564565] entry_SYSCALL_64_after_hwframe+0x49/0xae [14740.571926] RIP: 0033:0x7f53425a5332 [14740.573129] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14740.605027] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14740.618657] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14740.633191] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14740.640540] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14740.651237] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14740.662744] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14754.733029] rcu: INFO: rcu_sched self-detected stall on CPU [14754.736242] rcu: 2-....: (7011857 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [14754.740933] (t=12660307 jiffies g=274185 q=67204) [14754.743680] rcu: rcu_sched kthread starved for 12118673 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [14754.748944] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14754.752843] rcu: RCU grace-period kthread stack dump: [14754.754992] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14754.759785] Call Trace: [14754.760974] __schedule+0x351/0xcb0 [14754.762543] schedule+0xc0/0x180 [14754.763888] schedule_timeout+0xb4/0x190 [14754.768990] ? do_raw_spin_unlock+0x75/0x190 [14754.773861] ? __next_timer_interrupt+0x160/0x160 [14754.777941] ? prepare_to_swait_event+0xf3/0x140 [14754.782829] rcu_gp_kthread+0x8dd/0x15b0 [14754.787252] ? force_qs_rnp+0x2c0/0x2c0 [14754.791573] kthread+0x1d1/0x200 [14754.794055] ? set_kthread_struct+0x70/0x70 [14754.799652] ret_from_fork+0x1f/0x30 [14754.803245] rcu: Stack dump where RCU GP kthread last ran: [14754.809668] Sending NMI from CPU 2 to CPUs 3: [14754.811901] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [14754.814903] Sending NMI from CPU 2 to CPUs 1: [14754.818989] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [14754.820159] NMI backtrace for cpu 2 [14754.825353] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14754.836920] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14754.841887] Call Trace: [14754.849043] [14754.849787] ? dump_stack+0xbb/0x10e [14754.851040] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14754.859801] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14754.868465] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14754.874693] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14754.878295] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14754.884937] ? rcu_dump_cpu_stacks+0x11b/0x171 [14754.901760] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14754.905211] ? update_process_times+0x6d/0xc0 [14754.906970] ? tick_sched_handle.isra.18+0x27/0x90 [14754.914983] ? tick_sched_timer+0x53/0xb0 [14754.917547] ? __hrtimer_run_queues+0x1c2/0x4d0 [14754.926231] ? tick_sched_handle.isra.18+0x90/0x90 [14754.931379] ? hrtimer_interrupt+0x128/0x2c0 [14754.935729] ? smp_apic_timer_interrupt+0xa6/0x270 [14754.944570] ? apic_timer_interrupt+0xf/0x20 [14754.956202] [14754.959709] ? native_safe_halt+0x2e/0x50 [14754.961056] ? kvm_wait+0xcd/0xe0 [14754.966764] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14754.979570] ? do_raw_spin_lock+0x177/0x190 [14754.981111] ? _raw_spin_lock+0x21/0x30 [14754.988646] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14755.004226] ? obd_mod_free+0x46/0x140 [ptlrpc] [14755.006759] ? mdc_commit_open+0x91/0xb0 [mdc] [14755.012988] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14755.016915] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14755.024222] ? after_reply+0xe2c/0x1780 [ptlrpc] [14755.033315] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14755.037274] ? __next_timer_interrupt+0x160/0x160 [14755.040843] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14755.046246] ? do_wait_intr+0xf0/0xf0 [14755.051223] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14755.060304] ? kthread+0x1d1/0x200 [14755.063967] ? set_kthread_struct+0x70/0x70 [14755.070666] ? ret_from_fork+0x1f/0x30 [14756.054809] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [14756.064394] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14756.100524] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14756.112329] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14756.117798] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14756.124746] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14756.130630] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14756.143702] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14756.152815] RAX: 0000000000006d92 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14756.170236] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14756.178695] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14756.193087] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14756.199582] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14756.201913] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14756.207212] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14756.209762] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14756.217330] Call Trace: [14756.229084] [14756.229851] ? show_regs.cold.9+0x22/0x2f [14756.231188] ? watchdog_timer_fn.cold.6+0x62/0x10a [14756.235359] ? __hrtimer_run_queues+0x1c2/0x4d0 [14756.237819] ? watchdog+0x40/0x40 [14756.242509] ? hrtimer_interrupt+0x128/0x2c0 [14756.248564] ? smp_apic_timer_interrupt+0xa6/0x270 [14756.255865] ? apic_timer_interrupt+0xf/0x20 [14756.259835] [14756.263682] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14756.269952] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14756.274872] ? newidle_balance.isra.93+0x207/0x580 [14756.276515] do_raw_spin_lock+0x177/0x190 [14756.282549] _raw_spin_lock+0x21/0x30 [14756.288836] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14756.299593] process_one_work+0x2c8/0x700 [14756.306393] worker_thread+0x296/0x6e0 [14756.307493] ? rescuer_thread+0x570/0x570 [14756.308557] kthread+0x1d1/0x200 [14756.310331] ? set_kthread_struct+0x70/0x70 [14756.313106] ret_from_fork+0x1f/0x30 [14760.929070] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12123s! [14760.938065] Showing busy workqueues and worker pools: [14760.940688] workqueue events: flags=0x0 [14760.942273] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14760.953445] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14760.963324] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14760.970963] pending: cpu_latency_work [ptlrpc] [14760.979202] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14760.984929] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14760.998257] workqueue events_long: flags=0x0 [14761.000078] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14761.008189] pending: lu_objects_flush [obdclass] [14761.010893] workqueue events_unbound: flags=0x2 [14761.012537] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14761.020598] pending: flush_memcg_stats_dwork, idle_cull_fn [14761.027490] workqueue events_power_efficient: flags=0x80 [14761.029561] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14761.035457] pending: check_lifetime, do_cache_clean [14761.041081] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14761.043637] pending: neigh_periodic_work, neigh_periodic_work [14761.050995] workqueue mm_percpu_wq: flags=0x8 [14761.054155] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14761.062604] pending: vmstat_update, lru_add_drain_per_cpu [14761.066107] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14761.069464] pending: vmstat_update, lru_add_drain_per_cpu [14761.072424] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14761.078033] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14761.088488] workqueue xprtiod: flags=0xa [14761.093618] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14761.099909] pending: xprt_autoclose, xs_error_handle [14761.105772] workqueue ipv6_addrconf: flags=0x40008 [14761.106990] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14761.109568] pending: addrconf_verify_work [14761.110871] workqueue knbd0-recv: flags=0x1a [14761.117053] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14761.118756] in-flight: 651:recv_work [14761.119829] workqueue ptlrpc_pinger: flags=0x2 [14761.123289] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14761.125929] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14761.130553] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12124s workers=4 idle: 15979 31824 8 [14761.140458] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14751s workers=2 idle: 654 [14780.055017] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [14780.058028] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [14780.058372] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash [14780.061586] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14780.061676] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14780.061684] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14780.061687] RIP: 0010:native_safe_halt+0x2e/0x50 [14780.061703] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14780.061708] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14780.061713] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14780.061715] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14780.061718] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14780.061720] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14780.061723] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14780.061727] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14780.061730] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14780.061732] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14780.061742] Call Trace: [14780.061748] [14780.061754] ? show_regs.cold.9+0x22/0x2f [14780.061765] ? watchdog_timer_fn.cold.6+0x62/0x10a [14780.061772] ? __hrtimer_run_queues+0x1c2/0x4d0 [14780.061777] ? watchdog+0x40/0x40 [14780.061781] ? hrtimer_interrupt+0x128/0x2c0 [14780.061785] ? smp_apic_timer_interrupt+0xa6/0x270 [14780.061793] ? apic_timer_interrupt+0xf/0x20 [14780.061797] [14780.061800] ? native_safe_halt+0x2e/0x50 [14780.061805] kvm_wait+0xcd/0xe0 [14780.061815] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14780.063115] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [14780.063323] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14780.063395] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14780.063399] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14780.063402] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14780.063415] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14780.063418] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14780.063423] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14780.063425] RDX: 000000000000572c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14780.063427] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14780.063430] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14780.063432] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14780.063435] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14780.063438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14780.063441] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14780.063449] Call Trace: [14780.063452] [14780.063456] ? show_regs.cold.9+0x22/0x2f [14780.063466] ? watchdog_timer_fn.cold.6+0x62/0x10a [14780.063473] ? __hrtimer_run_queues+0x1c2/0x4d0 [14780.063477] ? watchdog+0x40/0x40 [14780.063481] ? hrtimer_interrupt+0x128/0x2c0 [14780.063485] ? smp_apic_timer_interrupt+0xa6/0x270 [14780.063494] ? apic_timer_interrupt+0xf/0x20 [14780.063499] [14780.063502] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14780.063506] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14780.063511] do_raw_spin_lock+0x177/0x190 [14780.063516] _raw_spin_lock+0x21/0x30 [14780.063522] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14780.063812] obd_mod_free+0x46/0x140 [ptlrpc] [14780.064011] mdc_commit_open+0x91/0xb0 [mdc] [14780.064057] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14780.066658] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14780.066864] after_reply+0xe2c/0x1780 [ptlrpc] [14780.067076] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14780.068527] ? __next_timer_interrupt+0x160/0x160 [14780.068538] ptlrpcd+0xa82/0xe40 [ptlrpc] [14780.068755] ? do_wait_intr+0xf0/0xf0 [14780.068764] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14780.068973] kthread+0x1d1/0x200 [14780.068979] ? set_kthread_struct+0x70/0x70 [14780.068982] ret_from_fork+0x1f/0x30 [14780.098126] dm_log dm_mod sha512_ssse3 sha512_generic [14780.098145] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14780.098154] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14780.115991] do_raw_spin_lock+0x177/0x190 [14780.116010] _raw_spin_lock+0x21/0x30 [14780.116016] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14780.116319] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14780.116526] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14780.116904] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14780.117221] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14780.117442] ? woken_wake_function+0x30/0x30 [14780.117452] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14780.117782] kthread+0x1d1/0x200 [14780.117789] ? set_kthread_struct+0x70/0x70 [14780.117793] ret_from_fork+0x1f/0x30 [14780.428205] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14780.430141] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14780.435747] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14780.438123] RAX: 0000000000002d9d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14780.440110] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14780.442224] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14780.444204] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14780.446510] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14780.448532] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14780.450919] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14780.452725] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14780.454552] Call Trace: [14780.455193] [14780.455863] ? show_regs.cold.9+0x22/0x2f [14780.457067] ? watchdog_timer_fn.cold.6+0x62/0x10a [14780.458495] ? __hrtimer_run_queues+0x1c2/0x4d0 [14780.459825] ? watchdog+0x40/0x40 [14780.460833] ? hrtimer_interrupt+0x128/0x2c0 [14780.462180] ? smp_apic_timer_interrupt+0xa6/0x270 [14780.463963] ? apic_timer_interrupt+0xf/0x20 [14780.465476] [14780.466288] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14780.468351] do_raw_spin_lock+0x177/0x190 [14780.469595] _raw_spin_lock+0x21/0x30 [14780.470839] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14780.472732] ? do_raw_spin_unlock+0x75/0x190 [14780.473908] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14780.475498] ? do_wait_intr+0xf0/0xf0 [14780.476349] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14780.477978] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14780.479746] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14780.481852] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14780.483526] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14780.485169] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14780.486800] mdc_intent_lock+0x353/0x7e0 [mdc] [14780.488157] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14780.489984] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14780.492156] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14780.494074] lmv_intent_open+0x338/0x1150 [lmv] [14780.496076] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14780.498492] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14780.500731] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14780.503258] ll_intent_lock+0xc5/0xaa0 [lustre] [14780.505148] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14780.507544] ll_lookup_it+0xe0e/0x2270 [lustre] [14780.509533] ? do_raw_spin_unlock+0x75/0x190 [14780.511497] ? _raw_spin_unlock+0x12/0x30 [14780.513357] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14780.515879] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14780.518013] ll_atomic_open+0x36a/0xd70 [lustre] [14780.520973] lookup_open+0x356/0xb40 [14780.522602] do_last+0x313/0xfc0 [14780.524185] ? nd_jump_root+0xe5/0x160 [14780.525773] ? path_init+0x437/0x520 [14780.527550] path_openat+0xf7/0x500 [14780.528715] ? do_raw_spin_unlock+0x75/0x190 [14780.530203] do_filp_open+0x99/0x140 [14780.531297] ? getname_flags+0x6e/0x330 [14780.532597] ? __check_object_size+0xff/0x256 [14780.534010] ? do_raw_spin_unlock+0x75/0x190 [14780.535350] ? _raw_spin_unlock+0x12/0x30 [14780.537093] do_sys_openat2+0x2b4/0x410 [14780.539198] do_sys_open+0x73/0xa0 [14780.540194] __x64_sys_openat+0x24/0x30 [14780.541368] do_syscall_64+0xc1/0x440 [14780.543287] entry_SYSCALL_64_after_hwframe+0x49/0xae [14780.544954] RIP: 0033:0x7f53425a5332 [14780.546055] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14780.552323] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14780.554796] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14780.557428] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14780.559750] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14780.562062] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14780.564395] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14788.052030] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [14788.055769] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14788.115379] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14788.130601] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14788.140953] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14788.146099] RIP: 0010:native_safe_halt+0x2e/0x50 [14788.157527] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14788.181774] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14788.183553] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14788.204755] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [14788.217341] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14788.226301] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14788.232967] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14788.240095] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14788.248983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14788.251034] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14788.267982] Call Trace: [14788.268865] [14788.280650] ? show_regs.cold.9+0x22/0x2f [14788.283422] ? watchdog_timer_fn.cold.6+0x62/0x10a [14788.285952] ? __hrtimer_run_queues+0x1c2/0x4d0 [14788.287557] ? watchdog+0x40/0x40 [14788.290404] ? hrtimer_interrupt+0x128/0x2c0 [14788.292024] ? smp_apic_timer_interrupt+0xa6/0x270 [14788.294125] ? apic_timer_interrupt+0xf/0x20 [14788.299468] [14788.300366] ? native_safe_halt+0x2e/0x50 [14788.305819] kvm_wait+0xcd/0xe0 [14788.307029] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14788.324227] ? newidle_balance.isra.93+0x207/0x580 [14788.325885] do_raw_spin_lock+0x177/0x190 [14788.327186] _raw_spin_lock+0x21/0x30 [14788.334268] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14788.339243] process_one_work+0x2c8/0x700 [14788.340979] worker_thread+0x296/0x6e0 [14788.345414] ? rescuer_thread+0x570/0x570 [14788.346778] kthread+0x1d1/0x200 [14788.347892] ? set_kthread_struct+0x70/0x70 [14788.352061] ret_from_fork+0x1f/0x30 [14791.650700] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12154s! [14791.657789] Showing busy workqueues and worker pools: [14791.659766] workqueue events: flags=0x0 [14791.662034] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14791.664962] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14791.673088] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14791.676263] pending: cpu_latency_work [ptlrpc] [14791.679179] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14791.683135] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14791.688108] workqueue events_long: flags=0x0 [14791.690600] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14791.694717] pending: lu_objects_flush [obdclass] [14791.697784] workqueue events_unbound: flags=0x2 [14791.699595] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14791.705121] pending: flush_memcg_stats_dwork, idle_cull_fn [14791.707925] workqueue events_power_efficient: flags=0x80 [14791.710727] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14791.714078] pending: check_lifetime, do_cache_clean [14791.717060] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14791.720685] pending: neigh_periodic_work, neigh_periodic_work [14791.724120] workqueue mm_percpu_wq: flags=0x8 [14791.726560] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14791.730361] pending: vmstat_update, lru_add_drain_per_cpu [14791.733601] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14791.737389] pending: vmstat_update, lru_add_drain_per_cpu [14791.740807] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14791.744521] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14791.748069] workqueue xprtiod: flags=0xa [14791.751016] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14791.754752] pending: xprt_autoclose, xs_error_handle [14791.757373] workqueue ipv6_addrconf: flags=0x40008 [14791.759224] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14791.762935] pending: addrconf_verify_work [14791.765688] workqueue knbd0-recv: flags=0x1a [14791.768116] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14791.771312] in-flight: 651:recv_work [14791.773661] workqueue ptlrpc_pinger: flags=0x2 [14791.775814] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14791.778592] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14791.783353] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12154s workers=4 idle: 15979 31824 8 [14791.789096] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14782s workers=2 idle: 654 [14808.055248] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [14808.057553] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [14808.058011] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [14808.058019] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14808.058101] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14808.058107] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14808.058110] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14808.058127] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14808.058212] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14808.058218] RAX: 0000000000004981 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14808.058221] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14808.058223] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14808.058226] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14808.058229] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14808.058232] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14808.058239] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14808.058242] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14808.058249] Call Trace: [14808.058254] [14808.058260] ? show_regs.cold.9+0x22/0x2f [14808.058269] ? watchdog_timer_fn.cold.6+0x62/0x10a [14808.058280] ? __hrtimer_run_queues+0x1c2/0x4d0 [14808.058285] ? watchdog+0x40/0x40 [14808.058290] ? hrtimer_interrupt+0x128/0x2c0 [14808.058295] ? smp_apic_timer_interrupt+0xa6/0x270 [14808.058304] ? apic_timer_interrupt+0xf/0x20 [14808.058308] [14808.058312] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14808.058317] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14808.058323] do_raw_spin_lock+0x177/0x190 [14808.058328] _raw_spin_lock+0x21/0x30 [14808.058332] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14808.059563] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14808.059847] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14808.060082] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14808.060418] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14808.060679] ? woken_wake_function+0x30/0x30 [14808.060688] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14808.060946] kthread+0x1d1/0x200 [14808.060957] ? set_kthread_struct+0x70/0x70 [14808.060961] ret_from_fork+0x1f/0x30 [14808.062488] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14808.062565] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14808.068654] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14808.068832] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14808.068839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14808.068845] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14808.068858] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14808.068862] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14808.068867] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14808.068869] RDX: 000000000000087c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14808.068872] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14808.068944] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14808.068947] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14808.068950] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14808.068952] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14808.068955] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14808.068962] Call Trace: [14808.068971] [14808.068977] ? show_regs.cold.9+0x22/0x2f [14808.068986] ? watchdog_timer_fn.cold.6+0x62/0x10a [14808.068992] ? __hrtimer_run_queues+0x1c2/0x4d0 [14808.068997] ? watchdog+0x40/0x40 [14808.069001] ? hrtimer_interrupt+0x128/0x2c0 [14808.069006] ? smp_apic_timer_interrupt+0xa6/0x270 [14808.069015] ? apic_timer_interrupt+0xf/0x20 [14808.069020] [14808.069022] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14808.069027] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14808.069032] do_raw_spin_lock+0x177/0x190 [14808.069087] _raw_spin_lock+0x21/0x30 [14808.069097] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14808.069548] obd_mod_free+0x46/0x140 [ptlrpc] [14808.069742] mdc_commit_open+0x91/0xb0 [mdc] [14808.069780] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14808.071424] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14808.071627] after_reply+0xe2c/0x1780 [ptlrpc] [14808.071820] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14808.072016] ? __next_timer_interrupt+0x160/0x160 [14808.072132] ptlrpcd+0xa82/0xe40 [ptlrpc] [14808.077598] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14808.077610] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14808.077624] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14808.077632] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14808.077638] RAX: 00000000000021e9 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14808.077641] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14808.077643] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14808.077645] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14808.077647] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14808.077651] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14808.077657] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14808.077660] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14808.077668] Call Trace: [14808.077674] [14808.077679] ? show_regs.cold.9+0x22/0x2f [14808.077693] ? watchdog_timer_fn.cold.6+0x62/0x10a [14808.077700] ? __hrtimer_run_queues+0x1c2/0x4d0 [14808.077704] ? watchdog+0x40/0x40 [14808.077708] ? hrtimer_interrupt+0x128/0x2c0 [14808.077716] ? smp_apic_timer_interrupt+0xa6/0x270 [14808.077725] ? apic_timer_interrupt+0xf/0x20 [14808.077729] [14808.077732] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14808.077740] do_raw_spin_lock+0x177/0x190 [14808.077745] _raw_spin_lock+0x21/0x30 [14808.077750] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14808.078049] ? do_raw_spin_unlock+0x75/0x190 [14808.078055] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14808.082419] ? do_wait_intr+0xf0/0xf0 [14808.082430] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14808.082644] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14808.082857] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14808.135524] ? do_wait_intr+0xf0/0xf0 [14808.135548] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14808.142660] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14808.142805] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14808.142846] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14808.143948] mdc_intent_lock+0x353/0x7e0 [mdc] [14808.143996] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14808.147385] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14808.156503] kthread+0x1d1/0x200 [14808.156523] ? set_kthread_struct+0x70/0x70 [14808.156527] ret_from_fork+0x1f/0x30 [14808.773373] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14808.774891] lmv_intent_open+0x338/0x1150 [lmv] [14808.776573] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14808.778713] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14808.782556] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14808.784388] ll_intent_lock+0xc5/0xaa0 [lustre] [14808.785972] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14808.787982] ll_lookup_it+0xe0e/0x2270 [lustre] [14808.789721] ? do_raw_spin_unlock+0x75/0x190 [14808.791329] ? _raw_spin_unlock+0x12/0x30 [14808.792777] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14808.794691] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14808.796681] ll_atomic_open+0x36a/0xd70 [lustre] [14808.798558] lookup_open+0x356/0xb40 [14808.799934] do_last+0x313/0xfc0 [14808.801127] ? nd_jump_root+0xe5/0x160 [14808.802481] ? path_init+0x437/0x520 [14808.803690] path_openat+0xf7/0x500 [14808.807397] ? do_raw_spin_unlock+0x75/0x190 [14808.809054] do_filp_open+0x99/0x140 [14808.810612] ? getname_flags+0x6e/0x330 [14808.812071] ? __check_object_size+0xff/0x256 [14808.813676] ? do_raw_spin_unlock+0x75/0x190 [14808.815080] ? _raw_spin_unlock+0x12/0x30 [14808.816340] do_sys_openat2+0x2b4/0x410 [14808.817683] do_sys_open+0x73/0xa0 [14808.818713] __x64_sys_openat+0x24/0x30 [14808.819975] do_syscall_64+0xc1/0x440 [14808.821474] entry_SYSCALL_64_after_hwframe+0x49/0xae [14808.822933] RIP: 0033:0x7f53425a5332 [14808.824115] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14808.829721] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14808.832309] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14808.835508] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14808.838083] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14808.841717] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14808.844474] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14816.052058] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [14816.056637] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14816.102408] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14816.116293] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14816.121470] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14816.128321] RIP: 0010:native_safe_halt+0x2e/0x50 [14816.130503] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14816.146114] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14816.152358] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14816.161213] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [14816.163198] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14816.168975] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14816.173395] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14816.183303] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14816.185398] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14816.192548] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14816.198523] Call Trace: [14816.199772] [14816.200512] ? show_regs.cold.9+0x22/0x2f [14816.201928] ? watchdog_timer_fn.cold.6+0x62/0x10a [14816.203863] ? __hrtimer_run_queues+0x1c2/0x4d0 [14816.205592] ? watchdog+0x40/0x40 [14816.208269] ? hrtimer_interrupt+0x128/0x2c0 [14816.209507] ? smp_apic_timer_interrupt+0xa6/0x270 [14816.213648] ? apic_timer_interrupt+0xf/0x20 [14816.214932] [14816.215729] ? native_safe_halt+0x2e/0x50 [14816.217296] kvm_wait+0xcd/0xe0 [14816.218392] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14816.222213] ? newidle_balance.isra.93+0x207/0x580 [14816.224871] do_raw_spin_lock+0x177/0x190 [14816.226885] _raw_spin_lock+0x21/0x30 [14816.229323] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14816.231886] process_one_work+0x2c8/0x700 [14816.234259] worker_thread+0x296/0x6e0 [14816.237811] ? rescuer_thread+0x570/0x570 [14816.240544] kthread+0x1d1/0x200 [14816.245948] ? set_kthread_struct+0x70/0x70 [14816.250914] ret_from_fork+0x1f/0x30 [14821.857033] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12184s! [14821.867124] Showing busy workqueues and worker pools: [14821.872447] workqueue events: flags=0x0 [14821.877584] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14821.886854] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14821.903055] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14821.911438] pending: cpu_latency_work [ptlrpc] [14821.927390] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14821.940798] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14821.955283] workqueue events_long: flags=0x0 [14821.961165] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14821.974315] pending: lu_objects_flush [obdclass] [14821.990063] workqueue events_unbound: flags=0x2 [14821.999094] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14822.011283] pending: flush_memcg_stats_dwork, idle_cull_fn [14822.016031] workqueue events_power_efficient: flags=0x80 [14822.024537] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14822.037034] pending: check_lifetime, do_cache_clean [14822.042359] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14822.056872] pending: neigh_periodic_work, neigh_periodic_work [14822.067369] workqueue mm_percpu_wq: flags=0x8 [14822.073987] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14822.082174] pending: vmstat_update, lru_add_drain_per_cpu [14822.088180] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14822.091297] pending: vmstat_update, lru_add_drain_per_cpu [14822.101402] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14822.111071] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14822.119217] workqueue xprtiod: flags=0xa [14822.122072] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14822.125877] pending: xprt_autoclose, xs_error_handle [14822.132417] workqueue ipv6_addrconf: flags=0x40008 [14822.136861] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14822.146004] pending: addrconf_verify_work [14822.156405] workqueue knbd0-recv: flags=0x1a [14822.161867] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14822.168154] in-flight: 651:recv_work [14822.172582] workqueue ptlrpc_pinger: flags=0x2 [14822.179229] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14822.189587] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14822.195119] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12185s workers=4 idle: 15979 31824 8 [14822.200389] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14812s workers=2 idle: 654 [14836.056412] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [14836.057167] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [14836.058136] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [14836.058145] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14836.058220] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14836.058225] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14836.058227] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14836.058240] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14836.058245] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14836.058250] RAX: 0000000000004c94 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14836.058252] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14836.058254] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14836.058257] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14836.058259] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14836.058262] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14836.058265] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14836.058267] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14836.058276] Call Trace: [14836.058280] [14836.058286] ? show_regs.cold.9+0x22/0x2f [14836.058296] ? watchdog_timer_fn.cold.6+0x62/0x10a [14836.058303] ? __hrtimer_run_queues+0x1c2/0x4d0 [14836.058307] ? watchdog+0x40/0x40 [14836.058312] ? hrtimer_interrupt+0x128/0x2c0 [14836.058316] ? smp_apic_timer_interrupt+0xa6/0x270 [14836.058326] ? apic_timer_interrupt+0xf/0x20 [14836.058330] [14836.058333] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14836.058339] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14836.058345] do_raw_spin_lock+0x177/0x190 [14836.058351] _raw_spin_lock+0x21/0x30 [14836.058356] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14836.058783] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14836.059001] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14836.059333] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14836.060264] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14836.060560] ? woken_wake_function+0x30/0x30 [14836.060572] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14836.061152] kthread+0x1d1/0x200 [14836.061159] ? set_kthread_struct+0x70/0x70 [14836.061164] ret_from_fork+0x1f/0x30 [14836.064246] Modules linked in: [14836.067583] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14836.067850] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14836.067863] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14836.067866] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14836.067881] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14836.067886] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14836.067895] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14836.067898] RDX: 0000000000005e3d RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14836.067903] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14836.067906] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14836.067908] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14836.067911] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14836.067914] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14836.067917] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14836.067925] Call Trace: [14836.067931] [14836.067942] ? show_regs.cold.9+0x22/0x2f [14836.067952] ? watchdog_timer_fn.cold.6+0x62/0x10a [14836.067958] ? __hrtimer_run_queues+0x1c2/0x4d0 [14836.067962] ? watchdog+0x40/0x40 [14836.067966] ? hrtimer_interrupt+0x128/0x2c0 [14836.067975] ? smp_apic_timer_interrupt+0xa6/0x270 [14836.067985] ? apic_timer_interrupt+0xf/0x20 [14836.067990] [14836.067993] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14836.067998] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14836.068002] do_raw_spin_lock+0x177/0x190 [14836.068007] _raw_spin_lock+0x21/0x30 [14836.068016] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14836.071168] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic [14836.092677] obd_mod_free+0x46/0x140 [ptlrpc] [14836.092966] mdc_commit_open+0x91/0xb0 [mdc] [14836.093006] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14836.093202] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14836.093406] after_reply+0xe2c/0x1780 [ptlrpc] [14836.099077] crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14836.099108] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14836.099116] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14836.099119] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14836.099133] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14836.099137] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14836.099142] RAX: 0000000000007ec1 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14836.099144] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14836.099146] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14836.099149] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14836.099151] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14836.099154] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14836.099157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14836.099160] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14836.099169] Call Trace: [14836.099175] [14836.099181] ? show_regs.cold.9+0x22/0x2f [14836.099522] ? watchdog_timer_fn.cold.6+0x62/0x10a [14836.099528] ? __hrtimer_run_queues+0x1c2/0x4d0 [14836.099532] ? watchdog+0x40/0x40 [14836.099537] ? hrtimer_interrupt+0x128/0x2c0 [14836.099545] ? smp_apic_timer_interrupt+0xa6/0x270 [14836.099553] ? apic_timer_interrupt+0xf/0x20 [14836.099557] [14836.099561] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14836.099568] do_raw_spin_lock+0x177/0x190 [14836.099573] _raw_spin_lock+0x21/0x30 [14836.099578] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14836.099910] ? do_raw_spin_unlock+0x75/0x190 [14836.099917] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14836.100148] ? do_wait_intr+0xf0/0xf0 [14836.100157] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14836.100402] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14836.100627] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14836.100882] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14836.101005] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14836.101047] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14836.101079] mdc_intent_lock+0x353/0x7e0 [mdc] [14836.101109] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14836.101206] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14836.101452] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14836.101481] lmv_intent_open+0x338/0x1150 [lmv] [14836.101514] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14836.101597] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14836.101959] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14836.101984] ll_intent_lock+0xc5/0xaa0 [lustre] [14836.102082] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14836.102163] ll_lookup_it+0xe0e/0x2270 [lustre] [14836.102247] ? do_raw_spin_unlock+0x75/0x190 [14836.102255] ? _raw_spin_unlock+0x12/0x30 [14836.102262] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14836.102292] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14836.102314] ll_atomic_open+0x36a/0xd70 [lustre] [14836.107083] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14836.107410] ? __next_timer_interrupt+0x160/0x160 [14836.107420] ptlrpcd+0xa82/0xe40 [ptlrpc] [14836.107635] ? do_wait_intr+0xf0/0xf0 [14836.107644] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14836.107881] kthread+0x1d1/0x200 [14836.107886] ? set_kthread_struct+0x70/0x70 [14836.107891] ret_from_fork+0x1f/0x30 [14836.393713] lookup_open+0x356/0xb40 [14836.394993] do_last+0x313/0xfc0 [14836.396148] ? nd_jump_root+0xe5/0x160 [14836.397270] ? path_init+0x437/0x520 [14836.398643] path_openat+0xf7/0x500 [14836.400066] ? do_raw_spin_unlock+0x75/0x190 [14836.402096] do_filp_open+0x99/0x140 [14836.403345] ? getname_flags+0x6e/0x330 [14836.404474] ? __check_object_size+0xff/0x256 [14836.406237] ? do_raw_spin_unlock+0x75/0x190 [14836.407844] ? _raw_spin_unlock+0x12/0x30 [14836.408996] do_sys_openat2+0x2b4/0x410 [14836.410280] do_sys_open+0x73/0xa0 [14836.411252] __x64_sys_openat+0x24/0x30 [14836.412351] do_syscall_64+0xc1/0x440 [14836.413542] entry_SYSCALL_64_after_hwframe+0x49/0xae [14836.415470] RIP: 0033:0x7f53425a5332 [14836.416529] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14836.422510] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14836.425286] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14836.427536] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14836.429973] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14836.432429] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14836.434921] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14848.052025] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [14848.054419] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14848.069049] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14848.072674] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14848.075710] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14848.078060] RIP: 0010:native_safe_halt+0x2e/0x50 [14848.079686] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14848.085643] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [14848.088149] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14848.089940] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14848.091607] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14848.093832] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14848.095687] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14848.097413] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14848.099583] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14848.100828] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14848.102573] Call Trace: [14848.103281] [14848.103867] ? show_regs.cold.9+0x22/0x2f [14848.105011] ? watchdog_timer_fn.cold.6+0x62/0x10a [14848.106694] ? __hrtimer_run_queues+0x1c2/0x4d0 [14848.108382] ? watchdog+0x40/0x40 [14848.120753] ? hrtimer_interrupt+0x128/0x2c0 [14848.136148] ? smp_apic_timer_interrupt+0xa6/0x270 [14848.152462] ? apic_timer_interrupt+0xf/0x20 [14848.159187] [14848.163654] ? native_safe_halt+0x2e/0x50 [14848.165385] kvm_wait+0xcd/0xe0 [14848.166842] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14848.170252] ? newidle_balance.isra.93+0x207/0x580 [14848.172267] do_raw_spin_lock+0x177/0x190 [14848.175805] _raw_spin_lock+0x21/0x30 [14848.176922] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14848.178867] process_one_work+0x2c8/0x700 [14848.179925] worker_thread+0x296/0x6e0 [14848.181138] ? rescuer_thread+0x570/0x570 [14848.182024] kthread+0x1d1/0x200 [14848.183217] ? set_kthread_struct+0x70/0x70 [14848.186853] ret_from_fork+0x1f/0x30 [14852.581200] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12215s! [14852.593082] Showing busy workqueues and worker pools: [14852.605617] workqueue events: flags=0x0 [14852.606808] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14852.614161] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14852.629477] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14852.635614] pending: cpu_latency_work [ptlrpc] [14852.646610] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14852.654075] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14852.676101] workqueue events_long: flags=0x0 [14852.686317] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14852.693649] pending: lu_objects_flush [obdclass] [14852.701973] workqueue events_unbound: flags=0x2 [14852.706055] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14852.710848] pending: flush_memcg_stats_dwork, idle_cull_fn [14852.714829] workqueue events_power_efficient: flags=0x80 [14852.718657] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14852.721854] pending: check_lifetime, do_cache_clean [14852.728515] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14852.730600] pending: neigh_periodic_work, neigh_periodic_work [14852.743947] workqueue mm_percpu_wq: flags=0x8 [14852.748494] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14852.754838] pending: vmstat_update, lru_add_drain_per_cpu [14852.759060] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14852.761895] pending: vmstat_update, lru_add_drain_per_cpu [14852.763894] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14852.772810] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14852.778492] workqueue xprtiod: flags=0xa [14852.781423] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14852.783846] pending: xprt_autoclose, xs_error_handle [14852.785929] workqueue ipv6_addrconf: flags=0x40008 [14852.787929] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14852.790920] pending: addrconf_verify_work [14852.792488] workqueue knbd0-recv: flags=0x1a [14852.794523] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14852.797754] in-flight: 651:recv_work [14852.799084] workqueue ptlrpc_pinger: flags=0x2 [14852.800688] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14852.803373] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14852.806235] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12215s workers=4 idle: 15979 31824 8 [14852.809942] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14843s workers=2 idle: 654 [14864.058246] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [14864.059018] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [14864.063071] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14864.063148] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14864.063156] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14864.063159] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14864.063186] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14864.063190] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14864.063194] RAX: 0000000000001f9a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14864.063196] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14864.063198] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14864.063201] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14864.063203] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14864.063206] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14864.063209] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14864.063212] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14864.063220] Call Trace: [14864.063225] [14864.063231] ? show_regs.cold.9+0x22/0x2f [14864.063240] ? watchdog_timer_fn.cold.6+0x62/0x10a [14864.063246] ? __hrtimer_run_queues+0x1c2/0x4d0 [14864.063250] ? watchdog+0x40/0x40 [14864.063254] ? hrtimer_interrupt+0x128/0x2c0 [14864.063258] ? smp_apic_timer_interrupt+0xa6/0x270 [14864.063266] ? apic_timer_interrupt+0xf/0x20 [14864.063272] [14864.063275] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14864.063282] do_raw_spin_lock+0x177/0x190 [14864.063287] _raw_spin_lock+0x21/0x30 [14864.063292] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14864.063580] ? do_raw_spin_unlock+0x75/0x190 [14864.063586] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14864.064608] ? do_wait_intr+0xf0/0xf0 [14864.064618] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14864.065065] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [14864.065072] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) [14864.064992] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14864.065097] fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14864.065142] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14864.065148] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14864.065150] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14864.065159] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14864.065163] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14864.065178] RAX: 0000000000001588 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14864.065180] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14864.065183] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14864.065185] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14864.065187] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14864.065189] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14864.065192] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14864.065194] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14864.065200] Call Trace: [14864.065203] [14864.065207] ? show_regs.cold.9+0x22/0x2f [14864.065214] ? watchdog_timer_fn.cold.6+0x62/0x10a [14864.065219] ? __hrtimer_run_queues+0x1c2/0x4d0 [14864.065223] ? watchdog+0x40/0x40 [14864.065227] ? hrtimer_interrupt+0x128/0x2c0 [14864.065231] ? smp_apic_timer_interrupt+0xa6/0x270 [14864.065238] ? apic_timer_interrupt+0xf/0x20 [14864.065243] [14864.065245] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14864.065259] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14864.065266] do_raw_spin_lock+0x177/0x190 [14864.065271] _raw_spin_lock+0x21/0x30 [14864.065276] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14864.065758] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14864.065961] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14864.066294] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14864.066791] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14864.067003] ? woken_wake_function+0x30/0x30 [14864.067012] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14864.065537] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14864.068075] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14864.073788] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14864.073940] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14864.073948] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14864.073951] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14864.073965] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14864.073970] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14864.073974] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14864.073977] RDX: 00000000000054d5 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14864.073979] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14864.073981] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14864.073983] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14864.073986] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14864.073989] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14864.073991] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14864.073999] Call Trace: [14864.074004] [14864.074009] ? show_regs.cold.9+0x22/0x2f [14864.074017] ? watchdog_timer_fn.cold.6+0x62/0x10a [14864.074023] ? __hrtimer_run_queues+0x1c2/0x4d0 [14864.074027] ? watchdog+0x40/0x40 [14864.074032] ? hrtimer_interrupt+0x128/0x2c0 [14864.074036] ? smp_apic_timer_interrupt+0xa6/0x270 [14864.074044] ? apic_timer_interrupt+0xf/0x20 [14864.074049] [14864.074052] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14864.074057] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14864.074061] do_raw_spin_lock+0x177/0x190 [14864.074065] _raw_spin_lock+0x21/0x30 [14864.074236] kthread+0x1d1/0x200 [14864.074072] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14864.074243] ? set_kthread_struct+0x70/0x70 [14864.074479] obd_mod_free+0x46/0x140 [ptlrpc] [14864.074250] ret_from_fork+0x1f/0x30 [14864.076002] mdc_commit_open+0x91/0xb0 [mdc] [14864.076050] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14864.080731] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14864.080956] after_reply+0xe2c/0x1780 [ptlrpc] [14864.081155] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14864.081615] ? __next_timer_interrupt+0x160/0x160 [14864.081626] ptlrpcd+0xa82/0xe40 [ptlrpc] [14864.082650] ? do_wait_intr+0xf0/0xf0 [14864.082660] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14864.084511] kthread+0x1d1/0x200 [14864.084520] ? set_kthread_struct+0x70/0x70 [14864.084525] ret_from_fork+0x1f/0x30 [14864.939955] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14864.947984] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14864.954083] mdc_intent_lock+0x353/0x7e0 [mdc] [14864.958541] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14864.962978] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14864.971704] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14864.974770] lmv_intent_open+0x338/0x1150 [lmv] [14864.977416] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14864.980679] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14864.983328] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14864.990304] ll_intent_lock+0xc5/0xaa0 [lustre] [14864.995289] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14864.998142] ll_lookup_it+0xe0e/0x2270 [lustre] [14865.005668] ? do_raw_spin_unlock+0x75/0x190 [14865.018120] ? _raw_spin_unlock+0x12/0x30 [14865.021139] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14865.023939] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14865.027547] ll_atomic_open+0x36a/0xd70 [lustre] [14865.029684] lookup_open+0x356/0xb40 [14865.031142] do_last+0x313/0xfc0 [14865.033593] ? nd_jump_root+0xe5/0x160 [14865.034968] ? path_init+0x437/0x520 [14865.038911] path_openat+0xf7/0x500 [14865.040711] ? do_raw_spin_unlock+0x75/0x190 [14865.042976] do_filp_open+0x99/0x140 [14865.047274] ? getname_flags+0x6e/0x330 [14865.049821] ? __check_object_size+0xff/0x256 [14865.053864] ? do_raw_spin_unlock+0x75/0x190 [14865.059464] ? _raw_spin_unlock+0x12/0x30 [14865.060860] do_sys_openat2+0x2b4/0x410 [14865.067110] do_sys_open+0x73/0xa0 [14865.069274] __x64_sys_openat+0x24/0x30 [14865.077375] do_syscall_64+0xc1/0x440 [14865.081880] entry_SYSCALL_64_after_hwframe+0x49/0xae [14865.087571] RIP: 0033:0x7f53425a5332 [14865.091897] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14865.103947] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14865.110447] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14865.115403] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14865.120922] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14865.127575] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14865.129325] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14880.064103] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [14880.071716] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14880.164096] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14880.196040] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14880.217174] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14880.235010] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14880.250185] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14880.295489] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14880.310602] RAX: 000000000000300b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14880.317188] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14880.326066] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14880.334002] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14880.340608] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14880.348860] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14880.359510] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14880.362484] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14880.369641] Call Trace: [14880.370970] [14880.373781] ? show_regs.cold.9+0x22/0x2f [14880.376982] ? watchdog_timer_fn.cold.6+0x62/0x10a [14880.383658] ? __hrtimer_run_queues+0x1c2/0x4d0 [14880.389347] ? watchdog+0x40/0x40 [14880.391244] ? hrtimer_interrupt+0x128/0x2c0 [14880.397102] ? smp_apic_timer_interrupt+0xa6/0x270 [14880.399497] ? apic_timer_interrupt+0xf/0x20 [14880.401640] [14880.403038] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14880.405577] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14880.408360] ? newidle_balance.isra.93+0x207/0x580 [14880.413623] do_raw_spin_lock+0x177/0x190 [14880.416005] _raw_spin_lock+0x21/0x30 [14880.418855] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14880.421692] process_one_work+0x2c8/0x700 [14880.423763] worker_thread+0x296/0x6e0 [14880.425770] ? rescuer_thread+0x570/0x570 [14880.428173] kthread+0x1d1/0x200 [14880.430060] ? set_kthread_struct+0x70/0x70 [14880.433056] ret_from_fork+0x1f/0x30 [14883.296039] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12246s! [14883.299814] Showing busy workqueues and worker pools: [14883.302780] workqueue events: flags=0x0 [14883.304399] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14883.308817] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14883.315251] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14883.317931] pending: cpu_latency_work [ptlrpc] [14883.320611] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14883.323756] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14883.327872] workqueue events_long: flags=0x0 [14883.329733] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14883.331739] pending: lu_objects_flush [obdclass] [14883.334273] workqueue events_unbound: flags=0x2 [14883.336361] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14883.339187] pending: flush_memcg_stats_dwork, idle_cull_fn [14883.341969] workqueue events_power_efficient: flags=0x80 [14883.344101] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14883.346778] pending: check_lifetime, do_cache_clean [14883.348776] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14883.351760] pending: neigh_periodic_work, neigh_periodic_work [14883.354401] workqueue mm_percpu_wq: flags=0x8 [14883.356346] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14883.359160] pending: vmstat_update, lru_add_drain_per_cpu [14883.361202] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14883.363968] pending: vmstat_update, lru_add_drain_per_cpu [14883.366537] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14883.368921] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14883.371955] workqueue xprtiod: flags=0xa [14883.375931] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14883.379730] pending: xprt_autoclose, xs_error_handle [14883.381917] workqueue ipv6_addrconf: flags=0x40008 [14883.383776] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14883.386565] pending: addrconf_verify_work [14883.388388] workqueue knbd0-recv: flags=0x1a [14883.390131] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14883.392871] in-flight: 651:recv_work [14883.394579] workqueue ptlrpc_pinger: flags=0x2 [14883.396529] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14883.399354] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14883.401943] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12246s workers=4 idle: 15979 31824 8 [14883.405538] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14873s workers=2 idle: 654 [14892.057499] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [14892.058008] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [14892.058958] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [14892.058965] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14892.059035] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14892.059041] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14892.059043] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14892.059059] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14892.059063] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14892.059068] RAX: 0000000000005210 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14892.059070] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14892.059072] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14892.059075] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14892.059077] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14892.059080] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14892.059083] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14892.059085] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14892.059094] Call Trace: [14892.059100] [14892.059105] ? show_regs.cold.9+0x22/0x2f [14892.059115] ? watchdog_timer_fn.cold.6+0x62/0x10a [14892.059121] ? __hrtimer_run_queues+0x1c2/0x4d0 [14892.059125] ? watchdog+0x40/0x40 [14892.059129] ? hrtimer_interrupt+0x128/0x2c0 [14892.059133] ? smp_apic_timer_interrupt+0xa6/0x270 [14892.059141] ? apic_timer_interrupt+0xf/0x20 [14892.059145] [14892.059147] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14892.059153] do_raw_spin_lock+0x177/0x190 [14892.059157] _raw_spin_lock+0x21/0x30 [14892.059161] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14892.061057] ? do_raw_spin_unlock+0x75/0x190 [14892.061070] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14892.063410] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14892.063484] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14892.063492] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14892.063495] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14892.063510] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14892.063514] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14892.063519] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14892.063521] RDX: 000000000000051d RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14892.063523] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14892.063525] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14892.063528] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14892.063531] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14892.063534] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14892.063536] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14892.063545] Call Trace: [14892.063550] [14892.063556] ? show_regs.cold.9+0x22/0x2f [14892.063566] ? watchdog_timer_fn.cold.6+0x62/0x10a [14892.063572] ? __hrtimer_run_queues+0x1c2/0x4d0 [14892.063576] ? watchdog+0x40/0x40 [14892.063580] ? hrtimer_interrupt+0x128/0x2c0 [14892.063584] ? smp_apic_timer_interrupt+0xa6/0x270 [14892.063593] ? apic_timer_interrupt+0xf/0x20 [14892.063598] [14892.063601] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14892.063606] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14892.063610] do_raw_spin_lock+0x177/0x190 [14892.063614] _raw_spin_lock+0x21/0x30 [14892.063864] ? do_wait_intr+0xf0/0xf0 [14892.063874] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14892.064083] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14892.064381] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14892.064598] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14892.065417] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14892.065459] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14892.065492] mdc_intent_lock+0x353/0x7e0 [mdc] [14892.065533] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14892.065630] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14892.065909] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14892.065940] lmv_intent_open+0x338/0x1150 [lmv] [14892.065972] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14892.063619] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14892.066113] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14892.066584] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14892.066609] ll_intent_lock+0xc5/0xaa0 [lustre] [14892.066734] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14892.068858] ll_lookup_it+0xe0e/0x2270 [lustre] [14892.068954] ? do_raw_spin_unlock+0x75/0x190 [14892.068963] ? _raw_spin_unlock+0x12/0x30 [14892.068971] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14892.069001] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14892.069023] ll_atomic_open+0x36a/0xd70 [lustre] [14892.069116] lookup_open+0x356/0xb40 [14892.069126] do_last+0x313/0xfc0 [14892.069132] ? nd_jump_root+0xe5/0x160 [14892.069138] ? path_init+0x437/0x520 [14892.069143] path_openat+0xf7/0x500 [14892.069148] ? do_raw_spin_unlock+0x75/0x190 [14892.069153] do_filp_open+0x99/0x140 [14892.069160] ? getname_flags+0x6e/0x330 [14892.069165] ? __check_object_size+0xff/0x256 [14892.069172] ? do_raw_spin_unlock+0x75/0x190 [14892.069176] ? _raw_spin_unlock+0x12/0x30 [14892.069771] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14892.069846] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14892.069853] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14892.069856] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14892.069866] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14892.069871] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14892.069876] RAX: 000000000000243c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14892.069879] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14892.069881] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14892.069883] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14892.069886] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14892.069889] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14892.069892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14892.069895] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14892.069904] Call Trace: [14892.069910] [14892.069913] ? show_regs.cold.9+0x22/0x2f [14892.069925] ? watchdog_timer_fn.cold.6+0x62/0x10a [14892.069932] ? __hrtimer_run_queues+0x1c2/0x4d0 [14892.069936] ? watchdog+0x40/0x40 [14892.069941] ? hrtimer_interrupt+0x128/0x2c0 [14892.069945] ? smp_apic_timer_interrupt+0xa6/0x270 [14892.069951] ? apic_timer_interrupt+0xf/0x20 [14892.069956] [14892.069959] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14892.069964] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14892.069970] do_raw_spin_lock+0x177/0x190 [14892.069975] _raw_spin_lock+0x21/0x30 [14892.069980] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14892.069181] do_sys_openat2+0x2b4/0x410 [14892.070219] do_sys_open+0x73/0xa0 [14892.070225] __x64_sys_openat+0x24/0x30 [14892.070229] do_syscall_64+0xc1/0x440 [14892.070235] entry_SYSCALL_64_after_hwframe+0x49/0xae [14892.070240] RIP: 0033:0x7f53425a5332 [14892.070247] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14892.070251] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14892.070255] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14892.070258] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14892.070266] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14892.070269] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14892.070272] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14892.070269] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14892.074733] obd_mod_free+0x46/0x140 [ptlrpc] [14892.075260] mdc_commit_open+0x91/0xb0 [mdc] [14892.075303] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14892.075649] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14892.075871] after_reply+0xe2c/0x1780 [ptlrpc] [14892.076077] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14892.076280] ? __next_timer_interrupt+0x160/0x160 [14892.076291] ptlrpcd+0xa82/0xe40 [ptlrpc] [14892.076649] ? do_wait_intr+0xf0/0xf0 [14892.076659] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14892.076897] kthread+0x1d1/0x200 [14892.076903] ? set_kthread_struct+0x70/0x70 [14892.076907] ret_from_fork+0x1f/0x30 [14892.919879] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14892.922633] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14892.925362] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14892.928362] ? woken_wake_function+0x30/0x30 [14892.931075] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14892.934159] kthread+0x1d1/0x200 [14892.935374] ? set_kthread_struct+0x70/0x70 [14892.936800] ret_from_fork+0x1f/0x30 [14908.056132] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [14908.061103] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14908.080076] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14908.088814] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14908.094521] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14908.099152] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14908.106011] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14908.117860] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14908.124306] RAX: 0000000000002986 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14908.138082] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14908.152532] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14908.157623] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14908.163979] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14908.167120] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14908.171367] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14908.177358] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14908.184146] Call Trace: [14908.185116] [14908.194153] ? show_regs.cold.9+0x22/0x2f [14908.196556] ? watchdog_timer_fn.cold.6+0x62/0x10a [14908.202118] ? __hrtimer_run_queues+0x1c2/0x4d0 [14908.203664] ? watchdog+0x40/0x40 [14908.212177] ? hrtimer_interrupt+0x128/0x2c0 [14908.213746] ? smp_apic_timer_interrupt+0xa6/0x270 [14908.221326] ? apic_timer_interrupt+0xf/0x20 [14908.225119] [14908.228889] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14908.234974] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14908.242782] ? newidle_balance.isra.93+0x207/0x580 [14908.249160] do_raw_spin_lock+0x177/0x190 [14908.253740] _raw_spin_lock+0x21/0x30 [14908.254919] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14908.257555] process_one_work+0x2c8/0x700 [14908.258977] worker_thread+0x296/0x6e0 [14908.269211] ? rescuer_thread+0x570/0x570 [14908.271331] kthread+0x1d1/0x200 [14908.279554] ? set_kthread_struct+0x70/0x70 [14908.280980] ret_from_fork+0x1f/0x30 [14913.504214] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12276s! [14913.508118] Showing busy workqueues and worker pools: [14913.510069] workqueue events: flags=0x0 [14913.511737] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14913.517129] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14913.530127] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14913.542594] pending: cpu_latency_work [ptlrpc] [14913.550947] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14913.562694] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14913.581886] workqueue events_long: flags=0x0 [14913.586918] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14913.599024] pending: lu_objects_flush [obdclass] [14913.607049] workqueue events_unbound: flags=0x2 [14913.611719] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14913.626362] pending: flush_memcg_stats_dwork, idle_cull_fn [14913.640983] workqueue events_power_efficient: flags=0x80 [14913.647794] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14913.657206] pending: check_lifetime, do_cache_clean [14913.663768] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14913.666048] pending: neigh_periodic_work, neigh_periodic_work [14913.677049] workqueue mm_percpu_wq: flags=0x8 [14913.687743] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14913.700370] pending: vmstat_update, lru_add_drain_per_cpu [14913.705148] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14913.712169] pending: vmstat_update, lru_add_drain_per_cpu [14913.718759] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14913.725079] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14913.730614] workqueue xprtiod: flags=0xa [14913.733469] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14913.740445] pending: xprt_autoclose, xs_error_handle [14913.742049] workqueue ipv6_addrconf: flags=0x40008 [14913.747879] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14913.751840] pending: addrconf_verify_work [14913.754712] workqueue knbd0-recv: flags=0x1a [14913.758617] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14913.765368] in-flight: 651:recv_work [14913.771205] workqueue ptlrpc_pinger: flags=0x2 [14913.773487] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14913.776998] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14913.785492] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12276s workers=4 idle: 15979 31824 8 [14913.790178] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14904s workers=2 idle: 654 [14920.056720] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [14920.059173] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [14920.060363] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [14920.060373] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14920.060451] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14920.060457] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14920.060460] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14920.060473] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14920.060478] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14920.060482] RAX: 00000000000061b3 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14920.060485] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14920.060487] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14920.060489] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14920.060492] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14920.060496] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14920.060498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14920.060500] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14920.060509] Call Trace: [14920.060516] [14920.060521] ? show_regs.cold.9+0x22/0x2f [14920.060531] ? watchdog_timer_fn.cold.6+0x62/0x10a [14920.060538] ? __hrtimer_run_queues+0x1c2/0x4d0 [14920.060543] ? watchdog+0x40/0x40 [14920.060547] ? hrtimer_interrupt+0x128/0x2c0 [14920.060551] ? smp_apic_timer_interrupt+0xa6/0x270 [14920.060558] ? apic_timer_interrupt+0xf/0x20 [14920.060563] [14920.060566] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14920.060571] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14920.060577] do_raw_spin_lock+0x177/0x190 [14920.060581] _raw_spin_lock+0x21/0x30 [14920.060586] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14920.060844] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14920.063361] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14920.063622] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14920.063828] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14920.064063] ? woken_wake_function+0x30/0x30 [14920.064074] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14920.064309] kthread+0x1d1/0x200 [14920.064315] ? set_kthread_struct+0x70/0x70 [14920.064319] ret_from_fork+0x1f/0x30 [14920.066064] Modules linked in: [14920.069240] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14920.069317] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14920.069325] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14920.069328] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14920.069342] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14920.069346] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14920.069350] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14920.069352] RDX: 000000000000245a RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14920.069355] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14920.069357] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14920.069360] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14920.069363] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14920.069366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14920.069368] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14920.069378] Call Trace: [14920.069384] [14920.069390] ? show_regs.cold.9+0x22/0x2f [14920.069400] ? watchdog_timer_fn.cold.6+0x62/0x10a [14920.069406] ? __hrtimer_run_queues+0x1c2/0x4d0 [14920.069411] ? watchdog+0x40/0x40 [14920.069415] ? hrtimer_interrupt+0x128/0x2c0 [14920.069420] ? smp_apic_timer_interrupt+0xa6/0x270 [14920.069427] ? apic_timer_interrupt+0xf/0x20 [14920.069431] [14920.069434] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14920.069440] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14920.069444] do_raw_spin_lock+0x177/0x190 [14920.069505] _raw_spin_lock+0x21/0x30 [14920.069513] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14920.069779] obd_mod_free+0x46/0x140 [ptlrpc] [14920.084272] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14920.084350] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14920.084357] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14920.084360] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14920.084373] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14920.084377] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14920.084382] RAX: 0000000000004b82 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14920.084384] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14920.084387] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14920.084390] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14920.084393] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14920.084397] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14920.084400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14920.084403] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14920.084413] Call Trace: [14920.084418] [14920.084424] ? show_regs.cold.9+0x22/0x2f [14920.084434] ? watchdog_timer_fn.cold.6+0x62/0x10a [14920.084440] ? __hrtimer_run_queues+0x1c2/0x4d0 [14920.084445] ? watchdog+0x40/0x40 [14920.084449] ? hrtimer_interrupt+0x128/0x2c0 [14920.084453] ? smp_apic_timer_interrupt+0xa6/0x270 [14920.084462] ? apic_timer_interrupt+0xf/0x20 [14920.084466] [14920.084469] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14920.084475] do_raw_spin_lock+0x177/0x190 [14920.084480] _raw_spin_lock+0x21/0x30 [14920.084484] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14920.148565] mdc_commit_open+0x91/0xb0 [mdc] [14920.151616] ? do_raw_spin_unlock+0x75/0x190 [14920.151632] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14920.151862] ? do_wait_intr+0xf0/0xf0 [14920.151872] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14920.152005] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14920.152150] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14920.152323] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14920.152409] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14920.152437] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14920.152461] mdc_intent_lock+0x353/0x7e0 [mdc] [14920.152486] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14920.152551] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14920.152701] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14920.152720] lmv_intent_open+0x338/0x1150 [lmv] [14920.152739] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14920.152800] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14920.152859] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14920.152877] ll_intent_lock+0xc5/0xaa0 [lustre] [14920.153016] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14920.153077] ll_lookup_it+0xe0e/0x2270 [lustre] [14920.153132] ? do_raw_spin_unlock+0x75/0x190 [14920.153136] ? _raw_spin_unlock+0x12/0x30 [14920.153141] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14920.153241] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14920.153255] ll_atomic_open+0x36a/0xd70 [lustre] [14920.153312] lookup_open+0x356/0xb40 [14920.161564] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14920.163613] do_last+0x313/0xfc0 [14920.172286] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14920.172621] after_reply+0xe2c/0x1780 [ptlrpc] [14920.172823] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14920.173026] ? __next_timer_interrupt+0x160/0x160 [14920.173036] ptlrpcd+0xa82/0xe40 [ptlrpc] [14920.176597] ? nd_jump_root+0xe5/0x160 [14920.176614] ? path_init+0x437/0x520 [14920.176620] path_openat+0xf7/0x500 [14920.176626] ? do_raw_spin_unlock+0x75/0x190 [14920.176632] do_filp_open+0x99/0x140 [14920.176639] ? getname_flags+0x6e/0x330 [14920.176644] ? __check_object_size+0xff/0x256 [14920.176651] ? do_raw_spin_unlock+0x75/0x190 [14920.176655] ? _raw_spin_unlock+0x12/0x30 [14920.176662] do_sys_openat2+0x2b4/0x410 [14920.176667] do_sys_open+0x73/0xa0 [14920.176671] __x64_sys_openat+0x24/0x30 [14920.176675] do_syscall_64+0xc1/0x440 [14920.176681] entry_SYSCALL_64_after_hwframe+0x49/0xae [14920.176688] RIP: 0033:0x7f53425a5332 [14920.176696] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14920.176700] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14920.176706] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14920.176708] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14920.176711] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14920.176713] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14920.176716] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14920.775964] ? do_wait_intr+0xf0/0xf0 [14920.777857] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14920.781587] kthread+0x1d1/0x200 [14920.793626] ? set_kthread_struct+0x70/0x70 [14920.800857] ret_from_fork+0x1f/0x30 [14934.736209] rcu: INFO: rcu_sched self-detected stall on CPU [14934.743713] rcu: 2-....: (7112090 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [14934.754635] (t=12840310 jiffies g=274185 q=67267) [14934.760063] rcu: rcu_sched kthread starved for 12298676 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [14934.767747] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [14934.777201] rcu: RCU grace-period kthread stack dump: [14934.783910] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [14934.792656] Call Trace: [14934.797300] __schedule+0x351/0xcb0 [14934.799886] schedule+0xc0/0x180 [14934.801610] schedule_timeout+0xb4/0x190 [14934.808609] ? do_raw_spin_unlock+0x75/0x190 [14934.810913] ? __next_timer_interrupt+0x160/0x160 [14934.814507] ? prepare_to_swait_event+0xf3/0x140 [14934.819167] rcu_gp_kthread+0x8dd/0x15b0 [14934.821281] ? force_qs_rnp+0x2c0/0x2c0 [14934.825846] kthread+0x1d1/0x200 [14934.828463] ? set_kthread_struct+0x70/0x70 [14934.835281] ret_from_fork+0x1f/0x30 [14934.836737] rcu: Stack dump where RCU GP kthread last ran: [14934.840272] Sending NMI from CPU 2 to CPUs 1: [14934.845025] NMI backtrace for cpu 1 [14934.845031] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14934.845033] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14934.845035] RIP: 0010:__pv_queued_spin_lock_slowpath+0x2fa/0x6d0 [14934.845037] Code: 5b d4 dc 02 01 49 c1 e5 05 48 98 49 81 c5 40 41 03 00 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 <48> 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f [14934.845039] RSP: 0018:ffffbb5381503500 EFLAGS: 00000246 [14934.845041] RAX: 0000000000006700 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14934.845043] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14934.845044] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14934.845045] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14934.845047] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14934.845048] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14934.845049] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14934.845050] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14934.845052] Call Trace: [14934.845053] [14934.845054] ? show_regs.cold.9+0x22/0x2f [14934.845055] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [14934.845056] ? nmi_cpu_backtrace_handler+0x15/0x40 [14934.845057] ? nmi_handle+0xbd/0x210 [14934.845058] ? default_do_nmi+0x4b/0x1b0 [14934.845059] ? do_nmi+0x280/0x380 [14934.845060] ? end_repeat_nmi+0x16/0x1f [14934.845061] ? __pv_queued_spin_lock_slowpath+0x2fa/0x6d0 [14934.845062] ? __pv_queued_spin_lock_slowpath+0x2fa/0x6d0 [14934.845063] ? __pv_queued_spin_lock_slowpath+0x2fa/0x6d0 [14934.845064] [14934.845065] do_raw_spin_lock+0x177/0x190 [14934.845066] _raw_spin_lock+0x21/0x30 [14934.845067] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14934.845068] ? do_raw_spin_unlock+0x75/0x190 [14934.845070] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14934.845071] ? do_wait_intr+0xf0/0xf0 [14934.845072] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14934.845073] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14934.845074] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14934.845075] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14934.845076] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14934.845078] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14934.845079] mdc_intent_lock+0x353/0x7e0 [mdc] [14934.845080] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14934.845081] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14934.845082] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14934.845083] lmv_intent_open+0x338/0x1150 [lmv] [14934.845084] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14934.845085] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14934.845086] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14934.845087] ll_intent_lock+0xc5/0xaa0 [lustre] [14934.845088] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14934.845089] ll_lookup_it+0xe0e/0x2270 [lustre] [14934.845090] ? do_raw_spin_unlock+0x75/0x190 [14934.845091] ? _raw_spin_unlock+0x12/0x30 [14934.845092] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14934.845093] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14934.845094] ll_atomic_open+0x36a/0xd70 [lustre] [14934.845095] lookup_open+0x356/0xb40 [14934.845096] do_last+0x313/0xfc0 [14934.845097] ? nd_jump_root+0xe5/0x160 [14934.845098] ? path_init+0x437/0x520 [14934.845099] path_openat+0xf7/0x500 [14934.845100] ? do_raw_spin_unlock+0x75/0x190 [14934.845101] do_filp_open+0x99/0x140 [14934.845102] ? getname_flags+0x6e/0x330 [14934.845103] ? __check_object_size+0xff/0x256 [14934.845104] ? do_raw_spin_unlock+0x75/0x190 [14934.845105] ? _raw_spin_unlock+0x12/0x30 [14934.845106] do_sys_openat2+0x2b4/0x410 [14934.845107] do_sys_open+0x73/0xa0 [14934.845108] __x64_sys_openat+0x24/0x30 [14934.845109] do_syscall_64+0xc1/0x440 [14934.845110] entry_SYSCALL_64_after_hwframe+0x49/0xae [14934.845111] RIP: 0033:0x7f53425a5332 [14934.845112] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14934.845114] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14934.845116] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14934.845117] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14934.845119] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14934.845181] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14934.845182] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14934.845452] Sending NMI from CPU 2 to CPUs 1: [14935.269577] NMI backtrace for cpu 1 [14935.269584] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14935.269587] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14935.269588] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14935.269590] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14935.269593] RSP: 0018:ffffbb5381503500 EFLAGS: 00000212 [14935.269596] RAX: 0000000000007386 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14935.269597] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14935.269599] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14935.269600] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14935.269601] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14935.269603] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14935.269604] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14935.269605] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14935.269607] Call Trace: [14935.269608] [14935.269608] ? show_regs.cold.9+0x22/0x2f [14935.269610] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [14935.269611] ? nmi_cpu_backtrace_handler+0x15/0x40 [14935.269612] ? nmi_handle+0xbd/0x210 [14935.269613] ? default_do_nmi+0x4b/0x1b0 [14935.269614] ? do_nmi+0x280/0x380 [14935.269615] ? end_repeat_nmi+0x16/0x1f [14935.269617] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14935.269618] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14935.269619] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14935.269620] [14935.269621] do_raw_spin_lock+0x177/0x190 [14935.269622] _raw_spin_lock+0x21/0x30 [14935.269623] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14935.269624] ? do_raw_spin_unlock+0x75/0x190 [14935.269625] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14935.269627] ? do_wait_intr+0xf0/0xf0 [14935.269628] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14935.269629] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14935.269630] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14935.269632] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14935.269633] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14935.269634] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14935.269635] mdc_intent_lock+0x353/0x7e0 [mdc] [14935.269636] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14935.269637] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14935.269638] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14935.269640] lmv_intent_open+0x338/0x1150 [lmv] [14935.269641] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14935.269642] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14935.269643] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14935.269644] ll_intent_lock+0xc5/0xaa0 [lustre] [14935.269645] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14935.269647] ll_lookup_it+0xe0e/0x2270 [lustre] [14935.269648] ? do_raw_spin_unlock+0x75/0x190 [14935.269649] ? _raw_spin_unlock+0x12/0x30 [14935.269650] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14935.269652] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14935.269653] ll_atomic_open+0x36a/0xd70 [lustre] [14935.269654] lookup_open+0x356/0xb40 [14935.269655] do_last+0x313/0xfc0 [14935.269656] ? nd_jump_root+0xe5/0x160 [14935.269658] ? path_init+0x437/0x520 [14935.269658] path_openat+0xf7/0x500 [14935.269659] ? do_raw_spin_unlock+0x75/0x190 [14935.269660] do_filp_open+0x99/0x140 [14935.269661] ? getname_flags+0x6e/0x330 [14935.269662] ? __check_object_size+0xff/0x256 [14935.269663] ? do_raw_spin_unlock+0x75/0x190 [14935.269664] ? _raw_spin_unlock+0x12/0x30 [14935.269664] do_sys_openat2+0x2b4/0x410 [14935.269666] do_sys_open+0x73/0xa0 [14935.269667] __x64_sys_openat+0x24/0x30 [14935.269668] do_syscall_64+0xc1/0x440 [14935.269669] entry_SYSCALL_64_after_hwframe+0x49/0xae [14935.269670] RIP: 0033:0x7f53425a5332 [14935.269671] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14935.269673] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14935.269676] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14935.269677] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14935.269679] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14935.269680] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14935.269682] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14935.270010] NMI backtrace for cpu 2 [14935.537208] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14935.549222] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14935.554430] Call Trace: [14935.555714] [14935.561286] ? dump_stack+0xbb/0x10e [14935.563185] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [14935.570895] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [14935.575009] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [14935.579437] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [14935.582724] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [14935.584174] ? rcu_dump_cpu_stacks+0x11b/0x171 [14935.585532] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [14935.594188] ? update_process_times+0x6d/0xc0 [14935.598055] ? tick_sched_handle.isra.18+0x27/0x90 [14935.600945] ? tick_sched_timer+0x53/0xb0 [14935.603008] ? __hrtimer_run_queues+0x1c2/0x4d0 [14935.606239] ? tick_sched_handle.isra.18+0x90/0x90 [14935.608524] ? hrtimer_interrupt+0x128/0x2c0 [14935.610765] ? smp_apic_timer_interrupt+0xa6/0x270 [14935.613016] ? apic_timer_interrupt+0xf/0x20 [14935.615080] [14935.615714] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14935.617742] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14935.620037] ? do_raw_spin_lock+0x177/0x190 [14935.621672] ? _raw_spin_lock+0x21/0x30 [14935.623437] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14935.626199] ? obd_mod_free+0x46/0x140 [ptlrpc] [14935.628225] ? mdc_commit_open+0x91/0xb0 [mdc] [14935.630422] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14935.633063] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14935.635571] ? after_reply+0xe2c/0x1780 [ptlrpc] [14935.637473] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14935.640042] ? __next_timer_interrupt+0x160/0x160 [14935.642247] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [14935.644017] ? do_wait_intr+0xf0/0xf0 [14935.645200] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14935.647389] ? kthread+0x1d1/0x200 [14935.649011] ? set_kthread_struct+0x70/0x70 [14935.650332] ? ret_from_fork+0x1f/0x30 [14940.058054] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [14940.065467] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14940.129188] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14940.158030] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14940.165007] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14940.173118] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14940.182254] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14940.208623] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14940.218489] RAX: 000000000000778f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14940.224172] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [14940.238027] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14940.240429] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14940.251838] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14940.266061] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14940.274225] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14940.286050] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14940.288674] Call Trace: [14940.296288] [14940.300214] ? show_regs.cold.9+0x22/0x2f [14940.306812] ? watchdog_timer_fn.cold.6+0x62/0x10a [14940.310786] ? __hrtimer_run_queues+0x1c2/0x4d0 [14940.319450] ? watchdog+0x40/0x40 [14940.326626] ? hrtimer_interrupt+0x128/0x2c0 [14940.331426] ? smp_apic_timer_interrupt+0xa6/0x270 [14940.339621] ? apic_timer_interrupt+0xf/0x20 [14940.341025] [14940.341743] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14940.359752] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14940.370359] ? newidle_balance.isra.93+0x207/0x580 [14940.372932] do_raw_spin_lock+0x177/0x190 [14940.376236] _raw_spin_lock+0x21/0x30 [14940.379523] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14940.387036] process_one_work+0x2c8/0x700 [14940.390558] worker_thread+0x296/0x6e0 [14940.395745] ? rescuer_thread+0x570/0x570 [14940.400224] kthread+0x1d1/0x200 [14940.405113] ? set_kthread_struct+0x70/0x70 [14940.418693] ret_from_fork+0x1f/0x30 [14944.225327] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12307s! [14944.237458] Showing busy workqueues and worker pools: [14944.241228] workqueue events: flags=0x0 [14944.242978] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14944.246861] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14944.257563] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14944.269615] pending: cpu_latency_work [ptlrpc] [14944.277041] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14944.280879] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14944.287138] workqueue events_long: flags=0x0 [14944.292516] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14944.296111] pending: lu_objects_flush [obdclass] [14944.301646] workqueue events_unbound: flags=0x2 [14944.312361] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14944.321305] pending: flush_memcg_stats_dwork, idle_cull_fn [14944.332230] workqueue events_power_efficient: flags=0x80 [14944.336728] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14944.345822] pending: check_lifetime, do_cache_clean [14944.347542] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14944.360634] pending: neigh_periodic_work, neigh_periodic_work [14944.366568] workqueue mm_percpu_wq: flags=0x8 [14944.370390] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14944.378998] pending: vmstat_update, lru_add_drain_per_cpu [14944.385047] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14944.392748] pending: vmstat_update, lru_add_drain_per_cpu [14944.396222] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14944.403382] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14944.405594] workqueue xprtiod: flags=0xa [14944.416058] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14944.423327] pending: xprt_autoclose, xs_error_handle [14944.428194] workqueue ipv6_addrconf: flags=0x40008 [14944.434163] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14944.443711] pending: addrconf_verify_work [14944.455832] workqueue knbd0-recv: flags=0x1a [14944.457338] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14944.459992] in-flight: 651:recv_work [14944.461578] workqueue ptlrpc_pinger: flags=0x2 [14944.471281] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14944.474563] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14944.479025] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12307s workers=4 idle: 15979 31824 8 [14944.492131] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14934s workers=2 idle: 654 [14948.058192] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [14948.062266] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14948.101203] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14948.109066] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14948.114061] RIP: 0010:native_safe_halt+0x2e/0x50 [14948.119114] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14948.140543] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14948.147608] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14948.153865] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14948.157673] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14948.163980] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14948.170625] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14948.177961] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14948.188226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14948.193249] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14948.200494] Call Trace: [14948.202149] [14948.203292] ? show_regs.cold.9+0x22/0x2f [14948.207257] ? watchdog_timer_fn.cold.6+0x62/0x10a [14948.210155] ? __hrtimer_run_queues+0x1c2/0x4d0 [14948.212232] ? watchdog+0x40/0x40 [14948.214027] ? hrtimer_interrupt+0x128/0x2c0 [14948.215736] ? smp_apic_timer_interrupt+0xa6/0x270 [14948.217554] ? apic_timer_interrupt+0xf/0x20 [14948.219083] [14948.219702] ? native_safe_halt+0x2e/0x50 [14948.221641] kvm_wait+0xcd/0xe0 [14948.226470] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14948.230010] do_raw_spin_lock+0x177/0x190 [14948.231584] _raw_spin_lock+0x21/0x30 [14948.233285] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14948.236391] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14948.244469] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14948.251863] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14948.261493] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14948.274271] ? woken_wake_function+0x30/0x30 [14948.282369] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14948.285559] kthread+0x1d1/0x200 [14948.290469] ? set_kthread_struct+0x70/0x70 [14948.294585] ret_from_fork+0x1f/0x30 [14960.057026] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [14960.057279] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [14960.060313] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14960.060388] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14960.060395] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14960.060399] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14960.060414] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14960.060417] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14960.060422] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14960.060425] RDX: 00000000000054f0 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14960.060427] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14960.060429] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14960.060431] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14960.060435] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14960.060438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14960.060440] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14960.060449] Call Trace: [14960.060457] [14960.060464] ? show_regs.cold.9+0x22/0x2f [14960.060475] ? watchdog_timer_fn.cold.6+0x62/0x10a [14960.060574] ? __hrtimer_run_queues+0x1c2/0x4d0 [14960.060581] ? watchdog+0x40/0x40 [14960.060586] ? hrtimer_interrupt+0x128/0x2c0 [14960.060590] ? smp_apic_timer_interrupt+0xa6/0x270 [14960.060598] ? apic_timer_interrupt+0xf/0x20 [14960.060603] [14960.060606] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14960.060611] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14960.060616] do_raw_spin_lock+0x177/0x190 [14960.060622] _raw_spin_lock+0x21/0x30 [14960.060629] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14960.060902] obd_mod_free+0x46/0x140 [ptlrpc] [14960.068129] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14960.068203] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14960.068211] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14960.068214] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14960.080922] mdc_commit_open+0x91/0xb0 [mdc] [14960.080976] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14960.081305] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14960.085594] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14960.085606] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14960.085612] RAX: 0000000000001413 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14960.085614] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14960.085616] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14960.085618] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14960.085620] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14960.085623] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14960.085626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14960.085628] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14960.085635] Call Trace: [14960.085639] [14960.085644] ? show_regs.cold.9+0x22/0x2f [14960.085655] ? watchdog_timer_fn.cold.6+0x62/0x10a [14960.085662] ? __hrtimer_run_queues+0x1c2/0x4d0 [14960.085666] ? watchdog+0x40/0x40 [14960.085670] ? hrtimer_interrupt+0x128/0x2c0 [14960.085674] ? smp_apic_timer_interrupt+0xa6/0x270 [14960.085683] ? apic_timer_interrupt+0xf/0x20 [14960.085688] [14960.085691] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14960.085701] do_raw_spin_lock+0x177/0x190 [14960.085706] _raw_spin_lock+0x21/0x30 [14960.085711] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14960.086037] ? do_raw_spin_unlock+0x75/0x190 [14960.086043] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14960.086314] ? do_wait_intr+0xf0/0xf0 [14960.086323] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14960.089698] after_reply+0xe2c/0x1780 [ptlrpc] [14960.090034] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14960.090298] ? __next_timer_interrupt+0x160/0x160 [14960.090310] ptlrpcd+0xa82/0xe40 [ptlrpc] [14960.090467] ? do_wait_intr+0xf0/0xf0 [14960.090475] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14960.090642] kthread+0x1d1/0x200 [14960.090650] ? set_kthread_struct+0x70/0x70 [14960.090652] ret_from_fork+0x1f/0x30 [14960.327914] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14960.330778] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14960.333580] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14960.339730] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14960.348878] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14960.350603] mdc_intent_lock+0x353/0x7e0 [mdc] [14960.352532] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14960.354444] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14960.357295] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14960.359845] lmv_intent_open+0x338/0x1150 [lmv] [14960.369254] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14960.371360] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14960.373915] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14960.381689] ll_intent_lock+0xc5/0xaa0 [lustre] [14960.387136] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14960.393495] ll_lookup_it+0xe0e/0x2270 [lustre] [14960.395764] ? do_raw_spin_unlock+0x75/0x190 [14960.400804] ? _raw_spin_unlock+0x12/0x30 [14960.403911] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14960.406102] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14960.407770] ll_atomic_open+0x36a/0xd70 [lustre] [14960.409641] lookup_open+0x356/0xb40 [14960.410943] do_last+0x313/0xfc0 [14960.411955] ? nd_jump_root+0xe5/0x160 [14960.416288] ? path_init+0x437/0x520 [14960.417345] path_openat+0xf7/0x500 [14960.419275] ? do_raw_spin_unlock+0x75/0x190 [14960.420887] do_filp_open+0x99/0x140 [14960.422110] ? getname_flags+0x6e/0x330 [14960.424754] ? __check_object_size+0xff/0x256 [14960.426135] ? do_raw_spin_unlock+0x75/0x190 [14960.427719] ? _raw_spin_unlock+0x12/0x30 [14960.429140] do_sys_openat2+0x2b4/0x410 [14960.430618] do_sys_open+0x73/0xa0 [14960.431887] __x64_sys_openat+0x24/0x30 [14960.433319] do_syscall_64+0xc1/0x440 [14960.434529] entry_SYSCALL_64_after_hwframe+0x49/0xae [14960.436805] RIP: 0033:0x7f53425a5332 [14960.437957] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14960.448248] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14960.450252] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14960.452329] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14960.454730] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14960.457098] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14960.459242] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [14972.054206] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [14972.057188] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14972.115639] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14972.131305] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14972.134188] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [14972.149262] RIP: 0010:native_safe_halt+0x2e/0x50 [14972.157623] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14972.181934] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14972.189888] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14972.192720] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [14972.197524] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [14972.204617] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [14972.208543] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [14972.214659] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [14972.223427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14972.234535] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [14972.243317] Call Trace: [14972.249505] [14972.254699] ? show_regs.cold.9+0x22/0x2f [14972.261009] ? watchdog_timer_fn.cold.6+0x62/0x10a [14972.264854] ? __hrtimer_run_queues+0x1c2/0x4d0 [14972.271624] ? watchdog+0x40/0x40 [14972.273474] ? hrtimer_interrupt+0x128/0x2c0 [14972.278807] ? smp_apic_timer_interrupt+0xa6/0x270 [14972.288245] ? apic_timer_interrupt+0xf/0x20 [14972.292984] [14972.298408] ? native_safe_halt+0x2e/0x50 [14972.304972] kvm_wait+0xcd/0xe0 [14972.311231] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14972.318884] ? newidle_balance.isra.93+0x207/0x580 [14972.325769] do_raw_spin_lock+0x177/0x190 [14972.331013] _raw_spin_lock+0x21/0x30 [14972.336752] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [14972.344615] process_one_work+0x2c8/0x700 [14972.351400] worker_thread+0x296/0x6e0 [14972.353413] ? rescuer_thread+0x570/0x570 [14972.358219] kthread+0x1d1/0x200 [14972.363745] ? set_kthread_struct+0x70/0x70 [14972.367697] ret_from_fork+0x1f/0x30 [14974.948025] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12337s! [14974.952442] Showing busy workqueues and worker pools: [14974.955036] workqueue events: flags=0x0 [14974.957659] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [14974.960737] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [14974.965810] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14974.968661] pending: cpu_latency_work [ptlrpc] [14974.971064] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [14974.974128] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [14974.978143] workqueue events_long: flags=0x0 [14974.980275] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [14974.983098] pending: lu_objects_flush [obdclass] [14974.984750] workqueue events_unbound: flags=0x2 [14974.985932] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [14974.988122] pending: flush_memcg_stats_dwork, idle_cull_fn [14974.990732] workqueue events_power_efficient: flags=0x80 [14974.993268] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14974.996370] pending: check_lifetime, do_cache_clean [14974.998021] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14975.000024] pending: neigh_periodic_work, neigh_periodic_work [14975.001922] workqueue mm_percpu_wq: flags=0x8 [14975.003115] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14975.006144] pending: vmstat_update, lru_add_drain_per_cpu [14975.009286] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [14975.012935] pending: vmstat_update, lru_add_drain_per_cpu [14975.016920] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [14975.020271] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [14975.023454] workqueue xprtiod: flags=0xa [14975.025231] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [14975.028558] pending: xprt_autoclose, xs_error_handle [14975.030773] workqueue ipv6_addrconf: flags=0x40008 [14975.032835] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [14975.035985] pending: addrconf_verify_work [14975.038517] workqueue knbd0-recv: flags=0x1a [14975.040382] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [14975.043713] in-flight: 651:recv_work [14975.045619] workqueue ptlrpc_pinger: flags=0x2 [14975.047526] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [14975.050640] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [14975.053785] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12337s workers=4 idle: 15979 31824 8 [14975.057288] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14965s workers=2 idle: 654 [14976.059389] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [14976.061933] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14976.080244] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14976.084323] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14976.087537] RIP: 0010:native_safe_halt+0x2e/0x50 [14976.089591] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [14976.096557] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [14976.099290] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14976.101858] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [14976.105054] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [14976.109091] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [14976.113006] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [14976.116781] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [14976.121196] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14976.128011] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [14976.131658] Call Trace: [14976.132761] [14976.134669] ? show_regs.cold.9+0x22/0x2f [14976.136432] ? watchdog_timer_fn.cold.6+0x62/0x10a [14976.138638] ? __hrtimer_run_queues+0x1c2/0x4d0 [14976.142097] ? watchdog+0x40/0x40 [14976.144121] ? hrtimer_interrupt+0x128/0x2c0 [14976.148900] ? smp_apic_timer_interrupt+0xa6/0x270 [14976.152146] ? apic_timer_interrupt+0xf/0x20 [14976.155612] [14976.156636] ? native_safe_halt+0x2e/0x50 [14976.159356] kvm_wait+0xcd/0xe0 [14976.161111] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [14976.164041] do_raw_spin_lock+0x177/0x190 [14976.166542] _raw_spin_lock+0x21/0x30 [14976.168453] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [14976.171045] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [14976.172629] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [14976.175134] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [14976.177453] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [14976.179713] ? woken_wake_function+0x30/0x30 [14976.181637] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [14976.185628] kthread+0x1d1/0x200 [14976.187073] ? set_kthread_struct+0x70/0x70 [14976.188926] ret_from_fork+0x1f/0x30 [14988.055438] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [14988.057011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [14988.058931] Modules linked in: [14988.062851] Modules linked in: [14988.062853] lustre(O) osp(O) ofd(O) [14988.064772] lustre(O) [14988.066223] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) [14988.068125] osp(O) [14988.069448] fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14988.069497] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14988.069504] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14988.069507] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14988.069521] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [14988.069525] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14988.069530] RAX: 0000000000007506 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [14988.069532] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [14988.069535] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [14988.069537] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [14988.069539] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [14988.069543] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [14988.069546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14988.069548] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [14988.069558] Call Trace: [14988.069563] [14988.069568] ? show_regs.cold.9+0x22/0x2f [14988.069643] ? watchdog_timer_fn.cold.6+0x62/0x10a [14988.069650] ? __hrtimer_run_queues+0x1c2/0x4d0 [14988.069654] ? watchdog+0x40/0x40 [14988.069659] ? hrtimer_interrupt+0x128/0x2c0 [14988.069663] ? smp_apic_timer_interrupt+0xa6/0x270 [14988.069672] ? apic_timer_interrupt+0xf/0x20 [14988.069677] [14988.069680] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [14988.069686] do_raw_spin_lock+0x177/0x190 [14988.069692] _raw_spin_lock+0x21/0x30 [14988.069698] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [14988.076512] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [14988.076831] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [14988.076838] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [14988.076842] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [14988.076857] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [14988.076862] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [14988.076866] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [14988.076869] RDX: 0000000000002714 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [14988.076872] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [14988.076879] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [14988.076885] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [14988.076888] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [14988.076896] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [14988.076899] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [14988.076910] Call Trace: [14988.076921] [14988.076927] ? show_regs.cold.9+0x22/0x2f [14988.076937] ? watchdog_timer_fn.cold.6+0x62/0x10a [14988.076949] ? __hrtimer_run_queues+0x1c2/0x4d0 [14988.076957] ? watchdog+0x40/0x40 [14988.076962] ? hrtimer_interrupt+0x128/0x2c0 [14988.076966] ? smp_apic_timer_interrupt+0xa6/0x270 [14988.076976] ? apic_timer_interrupt+0xf/0x20 [14988.076981] [14988.076984] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [14988.078057] ? do_raw_spin_unlock+0x75/0x190 [14988.078070] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [14988.095954] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [14988.095978] do_raw_spin_lock+0x177/0x190 [14988.096097] _raw_spin_lock+0x21/0x30 [14988.096110] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [14988.096403] obd_mod_free+0x46/0x140 [ptlrpc] [14988.108110] ? do_wait_intr+0xf0/0xf0 [14988.112815] mdc_commit_open+0x91/0xb0 [mdc] [14988.112868] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [14988.113395] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [14988.113622] after_reply+0xe2c/0x1780 [ptlrpc] [14988.113838] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [14988.114309] ? __next_timer_interrupt+0x160/0x160 [14988.114320] ptlrpcd+0xa82/0xe40 [ptlrpc] [14988.114666] ? do_wait_intr+0xf0/0xf0 [14988.114676] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [14988.114896] kthread+0x1d1/0x200 [14988.114903] ? set_kthread_struct+0x70/0x70 [14988.114907] ret_from_fork+0x1f/0x30 [14988.506968] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [14988.511367] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [14988.514711] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14988.518252] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14988.521927] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14988.528210] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [14988.530086] mdc_intent_lock+0x353/0x7e0 [mdc] [14988.532077] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14988.534260] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [14988.537242] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [14988.541933] lmv_intent_open+0x338/0x1150 [lmv] [14988.545105] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14988.547526] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14988.549802] lmv_intent_lock+0x1dd/0x4c0 [lmv] [14988.552884] ll_intent_lock+0xc5/0xaa0 [lustre] [14988.556081] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [14988.561488] ll_lookup_it+0xe0e/0x2270 [lustre] [14988.565325] ? do_raw_spin_unlock+0x75/0x190 [14988.569228] ? _raw_spin_unlock+0x12/0x30 [14988.571906] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [14988.575891] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [14988.579261] ll_atomic_open+0x36a/0xd70 [lustre] [14988.583301] lookup_open+0x356/0xb40 [14988.585731] do_last+0x313/0xfc0 [14988.587906] ? nd_jump_root+0xe5/0x160 [14988.592246] ? path_init+0x437/0x520 [14988.594100] path_openat+0xf7/0x500 [14988.597149] ? do_raw_spin_unlock+0x75/0x190 [14988.602483] do_filp_open+0x99/0x140 [14988.604567] ? getname_flags+0x6e/0x330 [14988.608290] ? __check_object_size+0xff/0x256 [14988.616106] ? do_raw_spin_unlock+0x75/0x190 [14988.617788] ? _raw_spin_unlock+0x12/0x30 [14988.622689] do_sys_openat2+0x2b4/0x410 [14988.624731] do_sys_open+0x73/0xa0 [14988.626935] __x64_sys_openat+0x24/0x30 [14988.629666] do_syscall_64+0xc1/0x440 [14988.635542] entry_SYSCALL_64_after_hwframe+0x49/0xae [14988.637854] RIP: 0033:0x7f53425a5332 [14988.642418] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [14988.661784] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [14988.666557] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [14988.674452] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [14988.679710] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [14988.693924] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [14988.698655] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15000.052302] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [15000.060328] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15000.120517] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15000.135223] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15000.141422] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15000.156283] RIP: 0010:native_safe_halt+0x2e/0x50 [15000.162228] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15000.197286] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15000.211452] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15000.217684] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [15000.221417] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15000.230422] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15000.242111] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15000.260587] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15000.265254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15000.267890] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15000.277579] Call Trace: [15000.286628] [15000.288658] ? show_regs.cold.9+0x22/0x2f [15000.299567] ? watchdog_timer_fn.cold.6+0x62/0x10a [15000.310351] ? __hrtimer_run_queues+0x1c2/0x4d0 [15000.317497] ? watchdog+0x40/0x40 [15000.321737] ? hrtimer_interrupt+0x128/0x2c0 [15000.326599] ? smp_apic_timer_interrupt+0xa6/0x270 [15000.331301] ? apic_timer_interrupt+0xf/0x20 [15000.336049] [15000.337429] ? native_safe_halt+0x2e/0x50 [15000.339428] kvm_wait+0xcd/0xe0 [15000.341079] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15000.343345] ? newidle_balance.isra.93+0x207/0x580 [15000.345017] do_raw_spin_lock+0x177/0x190 [15000.347916] _raw_spin_lock+0x21/0x30 [15000.349924] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15000.358221] process_one_work+0x2c8/0x700 [15000.359518] worker_thread+0x296/0x6e0 [15000.363605] ? rescuer_thread+0x570/0x570 [15000.373558] kthread+0x1d1/0x200 [15000.379306] ? set_kthread_struct+0x70/0x70 [15000.384474] ret_from_fork+0x1f/0x30 [15004.059824] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15004.071051] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15004.118660] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15004.131294] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15004.133984] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15004.136304] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15004.149937] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15004.156617] RAX: 0000000000004508 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15004.160285] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15004.163647] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15004.166884] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15004.177074] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15004.184057] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15004.190282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15004.194501] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15004.199867] Call Trace: [15004.203460] [15004.206945] ? show_regs.cold.9+0x22/0x2f [15004.208463] ? watchdog_timer_fn.cold.6+0x62/0x10a [15004.214408] ? __hrtimer_run_queues+0x1c2/0x4d0 [15004.221937] ? watchdog+0x40/0x40 [15004.224096] ? hrtimer_interrupt+0x128/0x2c0 [15004.231123] ? smp_apic_timer_interrupt+0xa6/0x270 [15004.237122] ? apic_timer_interrupt+0xf/0x20 [15004.239493] [15004.242273] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15004.250402] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15004.261087] do_raw_spin_lock+0x177/0x190 [15004.271099] _raw_spin_lock+0x21/0x30 [15004.272344] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15004.286897] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15004.295630] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15004.305240] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15004.317239] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15004.322150] ? woken_wake_function+0x30/0x30 [15004.323673] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15004.351836] kthread+0x1d1/0x200 [15004.360826] ? set_kthread_struct+0x70/0x70 [15004.368224] ret_from_fork+0x1f/0x30 [15005.151074] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12368s! [15005.154440] Showing busy workqueues and worker pools: [15005.162290] workqueue events: flags=0x0 [15005.165142] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15005.172294] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15005.180289] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15005.185164] pending: cpu_latency_work [ptlrpc] [15005.188725] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15005.191115] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15005.196088] workqueue events_long: flags=0x0 [15005.199140] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15005.202677] pending: lu_objects_flush [obdclass] [15005.205941] workqueue events_unbound: flags=0x2 [15005.208217] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15005.211732] pending: flush_memcg_stats_dwork, idle_cull_fn [15005.217418] workqueue events_power_efficient: flags=0x80 [15005.219363] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15005.231823] pending: check_lifetime, do_cache_clean [15005.233174] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15005.244518] pending: neigh_periodic_work, neigh_periodic_work [15005.247857] workqueue mm_percpu_wq: flags=0x8 [15005.250150] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15005.254424] pending: vmstat_update, lru_add_drain_per_cpu [15005.257019] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15005.260109] pending: vmstat_update, lru_add_drain_per_cpu [15005.263077] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15005.266817] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15005.269685] workqueue xprtiod: flags=0xa [15005.271534] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15005.273448] pending: xprt_autoclose, xs_error_handle [15005.277875] workqueue ipv6_addrconf: flags=0x40008 [15005.279481] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15005.282595] pending: addrconf_verify_work [15005.286568] workqueue knbd0-recv: flags=0x1a [15005.291484] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15005.297781] in-flight: 651:recv_work [15005.301607] workqueue ptlrpc_pinger: flags=0x2 [15005.306944] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15005.314910] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15005.322113] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12368s workers=4 idle: 15979 31824 8 [15005.327767] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=14995s workers=2 idle: 654 [15016.055111] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [15016.059153] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15016.059910] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [15016.080891] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15016.083223] Modules linked in: lustre(O) osp(O) [15016.088793] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15016.088802] ofd(O) lod(O) [15016.090106] RIP: 0010:native_safe_halt+0x2e/0x50 [15016.094146] mdt(O) [15016.094992] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15016.101748] mdd(O) [15016.103219] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15016.103231] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15016.103233] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15016.103235] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15016.103237] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15016.103239] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15016.103242] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15016.103245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15016.103247] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15016.103255] Call Trace: [15016.103263] [15016.103270] ? show_regs.cold.9+0x22/0x2f [15016.103286] ? watchdog_timer_fn.cold.6+0x62/0x10a [15016.103292] ? __hrtimer_run_queues+0x1c2/0x4d0 [15016.103296] ? watchdog+0x40/0x40 [15016.103300] ? hrtimer_interrupt+0x128/0x2c0 [15016.103304] ? smp_apic_timer_interrupt+0xa6/0x270 [15016.103312] ? apic_timer_interrupt+0xf/0x20 [15016.103316] [15016.103319] ? native_safe_halt+0x2e/0x50 [15016.103324] kvm_wait+0xcd/0xe0 [15016.103334] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15016.103345] do_raw_spin_lock+0x177/0x190 [15016.103349] _raw_spin_lock+0x21/0x30 [15016.103354] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15016.103746] ? do_raw_spin_unlock+0x75/0x190 [15016.103753] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15016.103960] ? do_wait_intr+0xf0/0xf0 [15016.103970] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15016.120151] mgs(O) [15016.120723] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15016.121033] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15016.121251] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15016.121378] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15016.121422] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15016.121454] mdc_intent_lock+0x353/0x7e0 [mdc] [15016.121486] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15016.121585] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15016.123838] lquota(O) [15016.125745] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15016.125807] lmv_intent_open+0x338/0x1150 [lmv] [15016.125841] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15016.128260] lfsck(O) [15016.130230] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15016.132119] obdecho(O) [15016.134534] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15016.136967] mgc(O) [15016.138699] ll_intent_lock+0xc5/0xaa0 [lustre] [15016.140830] mdc(O) [15016.141816] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15016.142316] lov(O) [15016.143470] ll_lookup_it+0xe0e/0x2270 [lustre] [15016.145030] osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) [15016.146295] ? do_raw_spin_unlock+0x75/0x190 [15016.147284] ptlrpc(O) [15016.148330] ? _raw_spin_unlock+0x12/0x30 [15016.148345] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15016.148373] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15016.148396] ll_atomic_open+0x36a/0xd70 [lustre] [15016.149949] obdclass(O) [15016.151473] lookup_open+0x356/0xb40 [15016.152121] ksocklnd(O) [15016.153211] do_last+0x313/0xfc0 [15016.154211] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 [15016.155588] ? nd_jump_root+0xe5/0x160 [15016.155602] ? path_init+0x437/0x520 [15016.155607] path_openat+0xf7/0x500 [15016.155612] ? do_raw_spin_unlock+0x75/0x190 [15016.155620] do_filp_open+0x99/0x140 [15016.155627] ? getname_flags+0x6e/0x330 [15016.155632] ? __check_object_size+0xff/0x256 [15016.155640] ? do_raw_spin_unlock+0x75/0x190 [15016.155644] ? _raw_spin_unlock+0x12/0x30 [15016.155652] do_sys_openat2+0x2b4/0x410 [15016.155657] do_sys_open+0x73/0xa0 [15016.155662] __x64_sys_openat+0x24/0x30 [15016.155666] do_syscall_64+0xc1/0x440 [15016.155673] entry_SYSCALL_64_after_hwframe+0x49/0xae [15016.156765] sha512_generic [15016.157898] RIP: 0033:0x7f53425a5332 [15016.159212] [15016.160521] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15016.160530] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15016.160536] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15016.160539] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15016.160541] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15016.160544] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15016.160546] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15016.267383] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15016.270838] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15016.273547] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15016.275417] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15016.281003] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15016.283324] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15016.285440] RDX: 0000000000004cd7 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15016.287657] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15016.290118] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15016.292181] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15016.294350] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15016.297006] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15016.298692] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15016.300895] Call Trace: [15016.301515] [15016.302233] ? show_regs.cold.9+0x22/0x2f [15016.303393] ? watchdog_timer_fn.cold.6+0x62/0x10a [15016.304951] ? __hrtimer_run_queues+0x1c2/0x4d0 [15016.306305] ? watchdog+0x40/0x40 [15016.307254] ? hrtimer_interrupt+0x128/0x2c0 [15016.308291] ? smp_apic_timer_interrupt+0xa6/0x270 [15016.309895] ? apic_timer_interrupt+0xf/0x20 [15016.311161] [15016.311798] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15016.313350] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15016.314581] do_raw_spin_lock+0x177/0x190 [15016.315803] _raw_spin_lock+0x21/0x30 [15016.316916] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15016.318907] obd_mod_free+0x46/0x140 [ptlrpc] [15016.321473] mdc_commit_open+0x91/0xb0 [mdc] [15016.322968] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15016.325398] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15016.327430] after_reply+0xe2c/0x1780 [ptlrpc] [15016.329340] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15016.331308] ? __next_timer_interrupt+0x160/0x160 [15016.332898] ptlrpcd+0xa82/0xe40 [ptlrpc] [15016.334355] ? do_wait_intr+0xf0/0xf0 [15016.335334] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15016.337323] kthread+0x1d1/0x200 [15016.338287] ? set_kthread_struct+0x70/0x70 [15016.339591] ret_from_fork+0x1f/0x30 [15032.053640] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [15032.061528] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15032.064776] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15032.064928] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15032.064936] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15032.064942] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15032.065381] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15032.065392] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15032.066723] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15032.066729] RAX: 0000000000006912 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15032.066732] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15032.066734] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15032.066736] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15032.066739] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15032.066743] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15032.066745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15032.066747] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15032.066756] Call Trace: [15032.066762] [15032.066768] ? show_regs.cold.9+0x22/0x2f [15032.066780] ? watchdog_timer_fn.cold.6+0x62/0x10a [15032.066787] ? __hrtimer_run_queues+0x1c2/0x4d0 [15032.066792] ? watchdog+0x40/0x40 [15032.066796] ? hrtimer_interrupt+0x128/0x2c0 [15032.066873] ? smp_apic_timer_interrupt+0xa6/0x270 [15032.066882] ? apic_timer_interrupt+0xf/0x20 [15032.066887] [15032.066890] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15032.066897] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15032.066900] ? newidle_balance.isra.93+0x207/0x580 [15032.066907] do_raw_spin_lock+0x177/0x190 [15032.066911] _raw_spin_lock+0x21/0x30 [15032.066916] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15032.067584] process_one_work+0x2c8/0x700 [15032.067592] worker_thread+0x296/0x6e0 [15032.067596] ? rescuer_thread+0x570/0x570 [15032.067599] kthread+0x1d1/0x200 [15032.067771] ? set_kthread_struct+0x70/0x70 [15032.067775] ret_from_fork+0x1f/0x30 [15032.290649] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15032.312650] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15032.318063] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15032.322074] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15032.324149] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15032.331372] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15032.334601] RAX: 0000000000006a83 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15032.339579] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15032.343403] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15032.348115] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15032.352927] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15032.358689] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15032.363059] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15032.366520] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15032.370586] Call Trace: [15032.372905] [15032.374924] ? show_regs.cold.9+0x22/0x2f [15032.378202] ? watchdog_timer_fn.cold.6+0x62/0x10a [15032.382639] ? __hrtimer_run_queues+0x1c2/0x4d0 [15032.387038] ? watchdog+0x40/0x40 [15032.389649] ? hrtimer_interrupt+0x128/0x2c0 [15032.393747] ? smp_apic_timer_interrupt+0xa6/0x270 [15032.401909] ? apic_timer_interrupt+0xf/0x20 [15032.405847] [15032.407692] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15032.409495] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15032.417081] do_raw_spin_lock+0x177/0x190 [15032.418914] _raw_spin_lock+0x21/0x30 [15032.420784] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15032.426170] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15032.435431] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15032.440096] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15032.443464] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15032.449171] ? woken_wake_function+0x30/0x30 [15032.450795] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15032.456704] kthread+0x1d1/0x200 [15032.458517] ? set_kthread_struct+0x70/0x70 [15032.459954] ret_from_fork+0x1f/0x30 [15035.359045] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12398s! [15035.362374] Showing busy workqueues and worker pools: [15035.365175] workqueue events: flags=0x0 [15035.367032] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15035.371118] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15035.377386] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15035.381211] pending: cpu_latency_work [ptlrpc] [15035.384627] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15035.387590] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15035.392528] workqueue events_long: flags=0x0 [15035.394381] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15035.398040] pending: lu_objects_flush [obdclass] [15035.400993] workqueue events_unbound: flags=0x2 [15035.403556] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15035.406391] pending: flush_memcg_stats_dwork, idle_cull_fn [15035.408438] workqueue events_power_efficient: flags=0x80 [15035.411378] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15035.414168] pending: check_lifetime, do_cache_clean [15035.416048] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15035.419468] pending: neigh_periodic_work, neigh_periodic_work [15035.422661] workqueue mm_percpu_wq: flags=0x8 [15035.425062] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15035.429133] pending: vmstat_update, lru_add_drain_per_cpu [15035.433039] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15035.436136] pending: vmstat_update, lru_add_drain_per_cpu [15035.438524] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15035.442156] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15035.446075] workqueue xprtiod: flags=0xa [15035.448231] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15035.451417] pending: xprt_autoclose, xs_error_handle [15035.453492] workqueue ipv6_addrconf: flags=0x40008 [15035.455647] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15035.458026] pending: addrconf_verify_work [15035.459593] workqueue knbd0-recv: flags=0x1a [15035.461219] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15035.464274] in-flight: 651:recv_work [15035.465607] workqueue ptlrpc_pinger: flags=0x2 [15035.467427] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15035.470113] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15035.472635] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12398s workers=4 idle: 15979 31824 8 [15035.475419] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15025s workers=2 idle: 654 [15044.057020] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [15044.061962] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15044.062929] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [15044.105937] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15044.105951] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15044.105954] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15044.105969] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15044.105973] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15044.105979] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15044.105982] RDX: 00000000000065ab RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15044.105985] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15044.105988] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15044.105990] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15044.105994] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15044.105996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15044.105998] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15044.106007] Call Trace: [15044.107618] [15044.107626] ? show_regs.cold.9+0x22/0x2f [15044.107639] ? watchdog_timer_fn.cold.6+0x62/0x10a [15044.107646] ? __hrtimer_run_queues+0x1c2/0x4d0 [15044.107650] ? watchdog+0x40/0x40 [15044.107654] ? hrtimer_interrupt+0x128/0x2c0 [15044.107658] ? smp_apic_timer_interrupt+0xa6/0x270 [15044.107667] ? apic_timer_interrupt+0xf/0x20 [15044.107671] [15044.107674] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15044.107680] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15044.107685] do_raw_spin_lock+0x177/0x190 [15044.107689] _raw_spin_lock+0x21/0x30 [15044.107698] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15044.107987] obd_mod_free+0x46/0x140 [ptlrpc] [15044.108206] mdc_commit_open+0x91/0xb0 [mdc] [15044.108247] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15044.108446] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15044.108989] after_reply+0xe2c/0x1780 [ptlrpc] [15044.109172] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15044.109297] ? __next_timer_interrupt+0x160/0x160 [15044.109305] ptlrpcd+0xa82/0xe40 [ptlrpc] [15044.109474] ? do_wait_intr+0xf0/0xf0 [15044.109482] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15044.116130] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15044.116214] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15044.116223] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15044.116226] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15044.116244] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15044.116250] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15044.116259] RAX: 0000000000007a37 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15044.116262] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15044.116264] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15044.116266] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15044.116271] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15044.116277] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15044.116281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15044.116283] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15044.116291] Call Trace: [15044.116297] [15044.116303] ? show_regs.cold.9+0x22/0x2f [15044.116313] ? watchdog_timer_fn.cold.6+0x62/0x10a [15044.116320] ? __hrtimer_run_queues+0x1c2/0x4d0 [15044.116325] ? watchdog+0x40/0x40 [15044.116334] ? hrtimer_interrupt+0x128/0x2c0 [15044.116338] ? smp_apic_timer_interrupt+0xa6/0x270 [15044.116347] ? apic_timer_interrupt+0xf/0x20 [15044.116351] [15044.116355] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15044.116424] do_raw_spin_lock+0x177/0x190 [15044.116428] _raw_spin_lock+0x21/0x30 [15044.116433] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15044.116853] ? do_raw_spin_unlock+0x75/0x190 [15044.116864] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15044.117426] ? do_wait_intr+0xf0/0xf0 [15044.117436] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15044.118223] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15044.118449] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15044.119444] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15044.119941] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15044.119983] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15044.120024] mdc_intent_lock+0x353/0x7e0 [mdc] [15044.120053] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15044.120826] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15044.121257] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15044.121904] lmv_intent_open+0x338/0x1150 [lmv] [15044.121935] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15044.133287] kthread+0x1d1/0x200 [15044.133308] ? set_kthread_struct+0x70/0x70 [15044.133313] ret_from_fork+0x1f/0x30 [15044.515268] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15044.521253] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15044.522814] ll_intent_lock+0xc5/0xaa0 [lustre] [15044.533290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15044.540369] ll_lookup_it+0xe0e/0x2270 [lustre] [15044.543518] ? do_raw_spin_unlock+0x75/0x190 [15044.547078] ? _raw_spin_unlock+0x12/0x30 [15044.549844] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15044.556679] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15044.558994] ll_atomic_open+0x36a/0xd70 [lustre] [15044.565182] lookup_open+0x356/0xb40 [15044.568096] do_last+0x313/0xfc0 [15044.576693] ? nd_jump_root+0xe5/0x160 [15044.585423] ? path_init+0x437/0x520 [15044.586801] path_openat+0xf7/0x500 [15044.596698] ? do_raw_spin_unlock+0x75/0x190 [15044.604321] do_filp_open+0x99/0x140 [15044.608115] ? getname_flags+0x6e/0x330 [15044.611740] ? __check_object_size+0xff/0x256 [15044.622441] ? do_raw_spin_unlock+0x75/0x190 [15044.623705] ? _raw_spin_unlock+0x12/0x30 [15044.626859] do_sys_openat2+0x2b4/0x410 [15044.637328] do_sys_open+0x73/0xa0 [15044.641613] __x64_sys_openat+0x24/0x30 [15044.646284] do_syscall_64+0xc1/0x440 [15044.650693] entry_SYSCALL_64_after_hwframe+0x49/0xae [15044.660706] RIP: 0033:0x7f53425a5332 [15044.661672] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15044.672337] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15044.679429] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15044.689794] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15044.693146] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15044.703304] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15044.707791] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15060.052053] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [15060.059706] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15060.061087] Modules linked in: lustre(O) [15060.063599] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15060.063673] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15060.063681] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15060.063684] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15060.063700] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15060.063705] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15060.063709] RAX: 0000000000006d68 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15060.063711] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15060.063713] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15060.063715] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15060.063717] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15060.063721] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15060.063723] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15060.063725] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15060.063747] Call Trace: [15060.063754] [15060.063761] ? show_regs.cold.9+0x22/0x2f [15060.063769] ? watchdog_timer_fn.cold.6+0x62/0x10a [15060.063774] ? __hrtimer_run_queues+0x1c2/0x4d0 [15060.063777] ? watchdog+0x40/0x40 [15060.063779] ? hrtimer_interrupt+0x128/0x2c0 [15060.063787] ? smp_apic_timer_interrupt+0xa6/0x270 [15060.063793] ? apic_timer_interrupt+0xf/0x20 [15060.063797] [15060.063799] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15060.063802] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15060.063806] do_raw_spin_lock+0x177/0x190 [15060.063809] _raw_spin_lock+0x21/0x30 [15060.063813] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15060.064034] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15060.064180] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15060.064444] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15060.065075] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15060.065254] ? woken_wake_function+0x30/0x30 [15060.065261] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15060.065622] kthread+0x1d1/0x200 [15060.065628] ? set_kthread_struct+0x70/0x70 [15060.065632] ret_from_fork+0x1f/0x30 [15060.269713] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15060.318423] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15060.330891] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15060.336421] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15060.340489] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [15060.347187] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [15060.366498] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15060.376089] RAX: 0000000000004342 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15060.385054] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15060.393110] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15060.402194] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15060.415414] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15060.429748] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15060.438401] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15060.446388] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15060.452463] Call Trace: [15060.454921] [15060.457072] ? show_regs.cold.9+0x22/0x2f [15060.463015] ? watchdog_timer_fn.cold.6+0x62/0x10a [15060.466224] ? __hrtimer_run_queues+0x1c2/0x4d0 [15060.468292] ? watchdog+0x40/0x40 [15060.470103] ? hrtimer_interrupt+0x128/0x2c0 [15060.472429] ? smp_apic_timer_interrupt+0xa6/0x270 [15060.475235] ? apic_timer_interrupt+0xf/0x20 [15060.479108] [15060.481893] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [15060.488571] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15060.492056] ? newidle_balance.isra.93+0x207/0x580 [15060.497226] do_raw_spin_lock+0x177/0x190 [15060.499222] _raw_spin_lock+0x21/0x30 [15060.505312] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15060.511679] process_one_work+0x2c8/0x700 [15060.516988] worker_thread+0x296/0x6e0 [15060.518127] ? rescuer_thread+0x570/0x570 [15060.524464] kthread+0x1d1/0x200 [15060.528375] ? set_kthread_struct+0x70/0x70 [15060.530118] ret_from_fork+0x1f/0x30 [15065.574646] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12428s! [15065.580045] Showing busy workqueues and worker pools: [15065.585054] workqueue events: flags=0x0 [15065.592574] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15065.598018] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15065.619578] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15065.622036] pending: cpu_latency_work [ptlrpc] [15065.635439] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15065.642454] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15065.651069] workqueue events_long: flags=0x0 [15065.654231] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15065.657193] pending: lu_objects_flush [obdclass] [15065.659190] workqueue events_unbound: flags=0x2 [15065.662431] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15065.668956] pending: flush_memcg_stats_dwork, idle_cull_fn [15065.672624] workqueue events_power_efficient: flags=0x80 [15065.675203] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15065.682126] pending: check_lifetime, do_cache_clean [15065.685100] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15065.690140] pending: neigh_periodic_work, neigh_periodic_work [15065.694677] workqueue mm_percpu_wq: flags=0x8 [15065.697038] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15065.703917] pending: vmstat_update, lru_add_drain_per_cpu [15065.706324] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15065.711473] pending: vmstat_update, lru_add_drain_per_cpu [15065.715723] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15065.718805] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15065.722315] workqueue xprtiod: flags=0xa [15065.725349] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15065.728204] pending: xprt_autoclose, xs_error_handle [15065.731254] workqueue ipv6_addrconf: flags=0x40008 [15065.734386] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15065.736521] pending: addrconf_verify_work [15065.739511] workqueue knbd0-recv: flags=0x1a [15065.742415] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15065.744970] in-flight: 651:recv_work [15065.746243] workqueue ptlrpc_pinger: flags=0x2 [15065.749426] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15065.753378] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15065.761585] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12428s workers=4 idle: 15979 31824 8 [15065.766011] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15056s workers=2 idle: 654 [15072.055009] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [15072.058604] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15072.059265] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [15072.094547] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15072.094561] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15072.094565] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15072.094580] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15072.094585] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15072.094591] RAX: 0000000000004309 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15072.094593] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15072.094595] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15072.094597] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15072.094599] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15072.094602] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15072.094605] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15072.094607] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15072.094616] Call Trace: [15072.094622] [15072.094627] ? show_regs.cold.9+0x22/0x2f [15072.094636] ? watchdog_timer_fn.cold.6+0x62/0x10a [15072.094643] ? __hrtimer_run_queues+0x1c2/0x4d0 [15072.094648] ? watchdog+0x40/0x40 [15072.094653] ? hrtimer_interrupt+0x128/0x2c0 [15072.094657] ? smp_apic_timer_interrupt+0xa6/0x270 [15072.094665] ? apic_timer_interrupt+0xf/0x20 [15072.094670] [15072.094673] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15072.094681] do_raw_spin_lock+0x177/0x190 [15072.094687] _raw_spin_lock+0x21/0x30 [15072.094694] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15072.098697] Modules linked in: [15072.111996] ? do_raw_spin_unlock+0x75/0x190 [15072.112013] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15072.112259] ? do_wait_intr+0xf0/0xf0 [15072.112267] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15072.112392] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15072.113983] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15072.114466] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15072.114670] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15072.114713] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15072.114899] mdc_intent_lock+0x353/0x7e0 [mdc] [15072.114925] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15072.115012] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15072.115342] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15072.115371] lmv_intent_open+0x338/0x1150 [lmv] [15072.115402] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15072.115484] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15072.129701] lustre(O) [15072.132248] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15072.132294] ll_intent_lock+0xc5/0xaa0 [lustre] [15072.132417] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15072.132513] ll_lookup_it+0xe0e/0x2270 [lustre] [15072.132610] ? do_raw_spin_unlock+0x75/0x190 [15072.132618] ? _raw_spin_unlock+0x12/0x30 [15072.132625] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15072.132650] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15072.132670] ll_atomic_open+0x36a/0xd70 [lustre] [15072.132767] lookup_open+0x356/0xb40 [15072.132777] do_last+0x313/0xfc0 [15072.132782] ? nd_jump_root+0xe5/0x160 [15072.132787] ? path_init+0x437/0x520 [15072.132792] path_openat+0xf7/0x500 [15072.132797] ? do_raw_spin_unlock+0x75/0x190 [15072.132802] do_filp_open+0x99/0x140 [15072.132808] ? getname_flags+0x6e/0x330 [15072.132813] ? __check_object_size+0xff/0x256 [15072.132820] ? do_raw_spin_unlock+0x75/0x190 [15072.132824] ? _raw_spin_unlock+0x12/0x30 [15072.132828] do_sys_openat2+0x2b4/0x410 [15072.132832] do_sys_open+0x73/0xa0 [15072.132836] __x64_sys_openat+0x24/0x30 [15072.132840] do_syscall_64+0xc1/0x440 [15072.132846] entry_SYSCALL_64_after_hwframe+0x49/0xae [15072.132852] RIP: 0033:0x7f53425a5332 [15072.132859] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15072.132863] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15072.132868] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15072.132871] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15072.132874] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15072.132876] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15072.132879] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15072.343231] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15072.363593] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15072.373394] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15072.382317] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15072.388808] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15072.405394] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15072.410743] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15072.413473] RDX: 00000000000054a5 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15072.421598] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15072.429699] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15072.439772] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15072.446711] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15072.451775] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15072.457183] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15072.461242] Call Trace: [15072.462470] [15072.463266] ? show_regs.cold.9+0x22/0x2f [15072.465316] ? watchdog_timer_fn.cold.6+0x62/0x10a [15072.467319] ? __hrtimer_run_queues+0x1c2/0x4d0 [15072.469559] ? watchdog+0x40/0x40 [15072.470861] ? hrtimer_interrupt+0x128/0x2c0 [15072.474824] ? smp_apic_timer_interrupt+0xa6/0x270 [15072.479662] ? apic_timer_interrupt+0xf/0x20 [15072.481144] [15072.483156] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15072.489738] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15072.495609] do_raw_spin_lock+0x177/0x190 [15072.498724] _raw_spin_lock+0x21/0x30 [15072.500440] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15072.503962] obd_mod_free+0x46/0x140 [ptlrpc] [15072.507142] mdc_commit_open+0x91/0xb0 [mdc] [15072.509742] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15072.513966] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15072.519293] after_reply+0xe2c/0x1780 [ptlrpc] [15072.526203] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15072.529278] ? __next_timer_interrupt+0x160/0x160 [15072.531860] ptlrpcd+0xa82/0xe40 [ptlrpc] [15072.534852] ? do_wait_intr+0xf0/0xf0 [15072.540098] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15072.546333] kthread+0x1d1/0x200 [15072.547442] ? set_kthread_struct+0x70/0x70 [15072.553153] ret_from_fork+0x1f/0x30 [15088.058082] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15088.062722] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15088.090071] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15088.095411] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15088.099188] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15088.101847] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15088.109666] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15088.112986] RAX: 0000000000006f11 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15088.115999] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15088.118800] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15088.122024] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15088.124495] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15088.128415] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15088.131972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15088.134755] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15088.138217] Call Trace: [15088.139212] [15088.140656] ? show_regs.cold.9+0x22/0x2f [15088.142804] ? watchdog_timer_fn.cold.6+0x62/0x10a [15088.144993] ? __hrtimer_run_queues+0x1c2/0x4d0 [15088.146821] ? watchdog+0x40/0x40 [15088.148528] ? hrtimer_interrupt+0x128/0x2c0 [15088.150656] ? smp_apic_timer_interrupt+0xa6/0x270 [15088.152623] ? apic_timer_interrupt+0xf/0x20 [15088.154481] [15088.155117] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15088.157717] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15088.159927] do_raw_spin_lock+0x177/0x190 [15088.161859] _raw_spin_lock+0x21/0x30 [15088.162977] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15088.165915] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15088.168404] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15088.170831] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15088.173683] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15088.176436] ? woken_wake_function+0x30/0x30 [15088.178420] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15088.181084] kthread+0x1d1/0x200 [15088.182681] ? set_kthread_struct+0x70/0x70 [15088.184662] ret_from_fork+0x1f/0x30 [15092.052031] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [15092.059386] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15092.089063] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15092.097905] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15092.107375] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15092.111404] RIP: 0010:native_safe_halt+0x2e/0x50 [15092.113571] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15092.131599] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15092.135830] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15092.141892] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [15092.149773] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15092.155061] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15092.163309] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15092.168382] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15092.173774] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15092.179300] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15092.183270] Call Trace: [15092.184988] [15092.186985] ? show_regs.cold.9+0x22/0x2f [15092.190232] ? watchdog_timer_fn.cold.6+0x62/0x10a [15092.193883] ? __hrtimer_run_queues+0x1c2/0x4d0 [15092.197841] ? watchdog+0x40/0x40 [15092.201061] ? hrtimer_interrupt+0x128/0x2c0 [15092.205239] ? smp_apic_timer_interrupt+0xa6/0x270 [15092.209893] ? apic_timer_interrupt+0xf/0x20 [15092.214162] [15092.216732] ? native_safe_halt+0x2e/0x50 [15092.217971] kvm_wait+0xcd/0xe0 [15092.223182] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15092.227087] ? newidle_balance.isra.93+0x207/0x580 [15092.229408] do_raw_spin_lock+0x177/0x190 [15092.234643] _raw_spin_lock+0x21/0x30 [15092.237395] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15092.244159] process_one_work+0x2c8/0x700 [15092.248443] worker_thread+0x296/0x6e0 [15092.253025] ? rescuer_thread+0x570/0x570 [15092.258636] kthread+0x1d1/0x200 [15092.261120] ? set_kthread_struct+0x70/0x70 [15092.267254] ret_from_fork+0x1f/0x30 [15095.782158] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12458s! [15095.787079] Showing busy workqueues and worker pools: [15095.796040] workqueue events: flags=0x0 [15095.799812] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15095.812034] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15095.825245] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15095.833088] pending: cpu_latency_work [ptlrpc] [15095.837641] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15095.844758] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15095.858373] workqueue events_long: flags=0x0 [15095.866589] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15095.879986] pending: lu_objects_flush [obdclass] [15095.892144] workqueue events_unbound: flags=0x2 [15095.893712] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15095.899197] pending: flush_memcg_stats_dwork, idle_cull_fn [15095.904986] workqueue events_power_efficient: flags=0x80 [15095.909972] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15095.916589] pending: check_lifetime, do_cache_clean [15095.920816] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15095.934205] pending: neigh_periodic_work, neigh_periodic_work [15095.944398] workqueue mm_percpu_wq: flags=0x8 [15095.946939] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15095.950571] pending: vmstat_update, lru_add_drain_per_cpu [15095.954113] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15095.958453] pending: vmstat_update, lru_add_drain_per_cpu [15095.968308] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15095.974994] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15095.977678] workqueue xprtiod: flags=0xa [15095.985090] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15095.991979] pending: xprt_autoclose, xs_error_handle [15095.995366] workqueue ipv6_addrconf: flags=0x40008 [15095.999087] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15096.003930] pending: addrconf_verify_work [15096.008599] workqueue knbd0-recv: flags=0x1a [15096.014177] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15096.023234] in-flight: 651:recv_work [15096.028742] workqueue ptlrpc_pinger: flags=0x2 [15096.031377] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15096.033274] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15096.042481] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12458s workers=4 idle: 15979 31824 8 [15096.053879] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15086s workers=2 idle: 654 [15100.055825] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [15100.059144] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15100.060726] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [15100.114445] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15100.114460] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15100.114463] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15100.115600] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15100.115608] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15100.115614] RAX: 0000000000007e8e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15100.115616] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15100.115618] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15100.115621] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15100.115623] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15100.115626] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15100.115629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15100.115631] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15100.115639] Call Trace: [15100.115667] [15100.115674] ? show_regs.cold.9+0x22/0x2f [15100.115684] ? watchdog_timer_fn.cold.6+0x62/0x10a [15100.115692] ? __hrtimer_run_queues+0x1c2/0x4d0 [15100.115696] ? watchdog+0x40/0x40 [15100.115700] ? hrtimer_interrupt+0x128/0x2c0 [15100.115705] ? smp_apic_timer_interrupt+0xa6/0x270 [15100.115714] ? apic_timer_interrupt+0xf/0x20 [15100.115719] [15100.115722] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15100.115731] do_raw_spin_lock+0x177/0x190 [15100.115736] _raw_spin_lock+0x21/0x30 [15100.115741] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15100.115950] ? do_raw_spin_unlock+0x75/0x190 [15100.115954] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15100.116123] ? do_wait_intr+0xf0/0xf0 [15100.116133] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15100.116317] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15100.116463] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15100.116618] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15100.116721] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15100.116751] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15100.116770] mdc_intent_lock+0x353/0x7e0 [mdc] [15100.116787] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15100.116843] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15100.116986] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15100.117005] lmv_intent_open+0x338/0x1150 [lmv] [15100.117025] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15100.117131] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15100.117214] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15100.117234] ll_intent_lock+0xc5/0xaa0 [lustre] [15100.117291] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15100.117346] ll_lookup_it+0xe0e/0x2270 [lustre] [15100.117400] ? do_raw_spin_unlock+0x75/0x190 [15100.117404] ? _raw_spin_unlock+0x12/0x30 [15100.117409] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15100.117428] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15100.117441] ll_atomic_open+0x36a/0xd70 [lustre] [15100.117497] lookup_open+0x356/0xb40 [15100.117503] do_last+0x313/0xfc0 [15100.117507] ? nd_jump_root+0xe5/0x160 [15100.117510] ? path_init+0x437/0x520 [15100.117513] path_openat+0xf7/0x500 [15100.117517] ? do_raw_spin_unlock+0x75/0x190 [15100.117520] do_filp_open+0x99/0x140 [15100.117525] ? getname_flags+0x6e/0x330 [15100.117528] ? __check_object_size+0xff/0x256 [15100.117533] ? do_raw_spin_unlock+0x75/0x190 [15100.117535] ? _raw_spin_unlock+0x12/0x30 [15100.117538] do_sys_openat2+0x2b4/0x410 [15100.117541] do_sys_open+0x73/0xa0 [15100.117543] __x64_sys_openat+0x24/0x30 [15100.117546] do_syscall_64+0xc1/0x440 [15100.117550] entry_SYSCALL_64_after_hwframe+0x49/0xae [15100.117554] RIP: 0033:0x7f53425a5332 [15100.117559] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15100.117562] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15100.117565] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15100.117567] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15100.117569] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15100.117570] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15100.117572] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15100.503399] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15100.567580] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15100.578546] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15100.588616] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15100.591196] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15100.598393] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15100.605095] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15100.606911] RDX: 0000000000003df8 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15100.609674] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15100.612425] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15100.614959] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15100.617826] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15100.628254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15100.632331] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15100.638305] Call Trace: [15100.639663] [15100.641249] ? show_regs.cold.9+0x22/0x2f [15100.644090] ? watchdog_timer_fn.cold.6+0x62/0x10a [15100.645691] ? __hrtimer_run_queues+0x1c2/0x4d0 [15100.648403] ? watchdog+0x40/0x40 [15100.651798] ? hrtimer_interrupt+0x128/0x2c0 [15100.657012] ? smp_apic_timer_interrupt+0xa6/0x270 [15100.663275] ? apic_timer_interrupt+0xf/0x20 [15100.669474] [15100.670635] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15100.674313] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15100.678805] do_raw_spin_lock+0x177/0x190 [15100.683490] _raw_spin_lock+0x21/0x30 [15100.687372] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15100.695565] obd_mod_free+0x46/0x140 [ptlrpc] [15100.705841] mdc_commit_open+0x91/0xb0 [mdc] [15100.717913] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15100.721244] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15100.725527] after_reply+0xe2c/0x1780 [ptlrpc] [15100.732542] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15100.741952] ? __next_timer_interrupt+0x160/0x160 [15100.748592] ptlrpcd+0xa82/0xe40 [ptlrpc] [15100.754183] ? do_wait_intr+0xf0/0xf0 [15100.759156] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15100.767614] kthread+0x1d1/0x200 [15100.768585] ? set_kthread_struct+0x70/0x70 [15100.769912] ret_from_fork+0x1f/0x30 [15114.739319] rcu: INFO: rcu_sched self-detected stall on CPU [15114.743924] rcu: 2-....: (7219610 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [15114.749422] (t=13020313 jiffies g=274185 q=67301) [15114.751607] rcu: rcu_sched kthread starved for 12478679 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [15114.758673] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [15114.768954] rcu: RCU grace-period kthread stack dump: [15114.771542] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [15114.778890] Call Trace: [15114.780974] __schedule+0x351/0xcb0 [15114.784310] schedule+0xc0/0x180 [15114.786117] schedule_timeout+0xb4/0x190 [15114.790708] ? do_raw_spin_unlock+0x75/0x190 [15114.794966] ? __next_timer_interrupt+0x160/0x160 [15114.796913] ? prepare_to_swait_event+0xf3/0x140 [15114.801398] rcu_gp_kthread+0x8dd/0x15b0 [15114.806829] ? force_qs_rnp+0x2c0/0x2c0 [15114.809117] kthread+0x1d1/0x200 [15114.811675] ? set_kthread_struct+0x70/0x70 [15114.813984] ret_from_fork+0x1f/0x30 [15114.816736] rcu: Stack dump where RCU GP kthread last ran: [15114.819181] Sending NMI from CPU 2 to CPUs 3: [15114.825385] NMI backtrace for cpu 3 [15114.825392] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15114.825394] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15114.825396] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15114.825398] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15114.825400] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 [15114.825403] RAX: 0000000000000713 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15114.825405] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15114.825406] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15114.825407] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15114.825408] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15114.825409] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15114.825411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15114.825412] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15114.825413] Call Trace: [15114.825414] [15114.825415] ? show_regs.cold.9+0x22/0x2f [15114.825416] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [15114.825417] ? nmi_cpu_backtrace_handler+0x15/0x40 [15114.825418] ? nmi_handle+0xbd/0x210 [15114.825419] ? default_do_nmi+0x4b/0x1b0 [15114.825420] ? do_nmi+0x280/0x380 [15114.825421] ? end_repeat_nmi+0x16/0x1f [15114.825422] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15114.825423] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15114.825425] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15114.825425] [15114.825426] do_raw_spin_lock+0x177/0x190 [15114.825427] _raw_spin_lock+0x21/0x30 [15114.825428] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15114.825429] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15114.825431] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15114.825432] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15114.825433] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15114.825434] ? woken_wake_function+0x30/0x30 [15114.825435] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15114.825436] kthread+0x1d1/0x200 [15114.825437] ? set_kthread_struct+0x70/0x70 [15114.825438] ret_from_fork+0x1f/0x30 [15114.825710] Sending NMI from CPU 2 to CPUs 1: [15115.092398] NMI backtrace for cpu 1 [15115.092405] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15115.092408] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15115.092410] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15115.092411] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15115.092414] RSP: 0018:ffffbb5381503500 EFLAGS: 00000216 [15115.092417] RAX: 0000000000003914 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15115.092419] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15115.092420] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15115.092422] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15115.092424] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15115.092425] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15115.092427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15115.092428] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15115.092429] Call Trace: [15115.092430] [15115.092431] ? show_regs.cold.9+0x22/0x2f [15115.092432] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [15115.092433] ? nmi_cpu_backtrace_handler+0x15/0x40 [15115.092435] ? nmi_handle+0xbd/0x210 [15115.092436] ? default_do_nmi+0x4b/0x1b0 [15115.092437] ? do_nmi+0x280/0x380 [15115.092438] ? end_repeat_nmi+0x16/0x1f [15115.092439] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15115.092441] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15115.092442] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15115.092443] [15115.092444] do_raw_spin_lock+0x177/0x190 [15115.092446] _raw_spin_lock+0x21/0x30 [15115.092447] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15115.092448] ? do_raw_spin_unlock+0x75/0x190 [15115.092449] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15115.092450] ? do_wait_intr+0xf0/0xf0 [15115.092451] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15115.092452] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15115.092454] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15115.092455] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15115.092456] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15115.092458] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15115.092459] mdc_intent_lock+0x353/0x7e0 [mdc] [15115.092460] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15115.092461] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15115.092463] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15115.092464] lmv_intent_open+0x338/0x1150 [lmv] [15115.092465] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15115.092466] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15115.092467] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15115.092468] ll_intent_lock+0xc5/0xaa0 [lustre] [15115.092470] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15115.092471] ll_lookup_it+0xe0e/0x2270 [lustre] [15115.092472] ? do_raw_spin_unlock+0x75/0x190 [15115.092473] ? _raw_spin_unlock+0x12/0x30 [15115.092474] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15115.092476] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15115.092477] ll_atomic_open+0x36a/0xd70 [lustre] [15115.093313] lookup_open+0x356/0xb40 [15115.093315] do_last+0x313/0xfc0 [15115.093316] ? nd_jump_root+0xe5/0x160 [15115.093317] ? path_init+0x437/0x520 [15115.093318] path_openat+0xf7/0x500 [15115.093319] ? do_raw_spin_unlock+0x75/0x190 [15115.093320] do_filp_open+0x99/0x140 [15115.093321] ? getname_flags+0x6e/0x330 [15115.093322] ? __check_object_size+0xff/0x256 [15115.093323] ? do_raw_spin_unlock+0x75/0x190 [15115.093324] ? _raw_spin_unlock+0x12/0x30 [15115.093326] do_sys_openat2+0x2b4/0x410 [15115.093327] do_sys_open+0x73/0xa0 [15115.093328] __x64_sys_openat+0x24/0x30 [15115.093329] do_syscall_64+0xc1/0x440 [15115.093330] entry_SYSCALL_64_after_hwframe+0x49/0xae [15115.093331] RIP: 0033:0x7f53425a5332 [15115.093332] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15115.093334] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15115.093337] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15115.093339] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15115.093340] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15115.093342] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15115.093344] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15115.099236] NMI backtrace for cpu 2 [15115.524002] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15115.532828] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15115.542040] Call Trace: [15115.545230] [15115.546466] ? dump_stack+0xbb/0x10e [15115.550832] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [15115.555977] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [15115.562869] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [15115.565439] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [15115.568945] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [15115.574597] ? rcu_dump_cpu_stacks+0x11b/0x171 [15115.580664] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [15115.588795] ? update_process_times+0x6d/0xc0 [15115.590812] ? tick_sched_handle.isra.18+0x27/0x90 [15115.595540] ? tick_sched_timer+0x53/0xb0 [15115.599977] ? __hrtimer_run_queues+0x1c2/0x4d0 [15115.609494] ? tick_sched_handle.isra.18+0x90/0x90 [15115.612741] ? hrtimer_interrupt+0x128/0x2c0 [15115.618050] ? smp_apic_timer_interrupt+0xa6/0x270 [15115.627784] ? apic_timer_interrupt+0xf/0x20 [15115.629899] [15115.632779] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15115.636225] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15115.639494] ? do_raw_spin_lock+0x177/0x190 [15115.644911] ? _raw_spin_lock+0x21/0x30 [15115.651917] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15115.656001] ? obd_mod_free+0x46/0x140 [ptlrpc] [15115.659437] ? mdc_commit_open+0x91/0xb0 [mdc] [15115.662850] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15115.666916] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15115.673639] ? after_reply+0xe2c/0x1780 [ptlrpc] [15115.679269] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15115.682288] ? __next_timer_interrupt+0x160/0x160 [15115.687405] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [15115.692820] ? do_wait_intr+0xf0/0xf0 [15115.694897] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15115.701183] ? kthread+0x1d1/0x200 [15115.702338] ? set_kthread_struct+0x70/0x70 [15115.707766] ? ret_from_fork+0x1f/0x30 [15124.052100] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [15124.057995] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15124.091693] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15124.098810] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15124.106728] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15124.111811] RIP: 0010:native_safe_halt+0x2e/0x50 [15124.114372] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15124.123410] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15124.125924] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15124.128736] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [15124.130974] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15124.133600] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15124.140254] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15124.143429] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15124.148466] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15124.153783] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15124.156986] Call Trace: [15124.157939] [15124.158510] ? show_regs.cold.9+0x22/0x2f [15124.160490] ? watchdog_timer_fn.cold.6+0x62/0x10a [15124.163922] ? __hrtimer_run_queues+0x1c2/0x4d0 [15124.170493] ? watchdog+0x40/0x40 [15124.172050] ? hrtimer_interrupt+0x128/0x2c0 [15124.175330] ? smp_apic_timer_interrupt+0xa6/0x270 [15124.179603] ? apic_timer_interrupt+0xf/0x20 [15124.181461] [15124.182230] ? native_safe_halt+0x2e/0x50 [15124.183849] kvm_wait+0xcd/0xe0 [15124.185857] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15124.187262] ? newidle_balance.isra.93+0x207/0x580 [15124.189558] do_raw_spin_lock+0x177/0x190 [15124.191252] _raw_spin_lock+0x21/0x30 [15124.194315] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15124.196495] process_one_work+0x2c8/0x700 [15124.198886] worker_thread+0x296/0x6e0 [15124.200523] ? rescuer_thread+0x570/0x570 [15124.202183] kthread+0x1d1/0x200 [15124.203211] ? set_kthread_struct+0x70/0x70 [15124.206099] ret_from_fork+0x1f/0x30 [15126.499030] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12489s! [15126.507013] Showing busy workqueues and worker pools: [15126.508723] workqueue events: flags=0x0 [15126.514836] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15126.518218] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15126.523591] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15126.527326] pending: cpu_latency_work [ptlrpc] [15126.530315] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15126.534392] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15126.548218] workqueue events_long: flags=0x0 [15126.556080] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15126.560320] pending: lu_objects_flush [obdclass] [15126.566449] workqueue events_unbound: flags=0x2 [15126.570905] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15126.573933] pending: flush_memcg_stats_dwork, idle_cull_fn [15126.577435] workqueue events_power_efficient: flags=0x80 [15126.582462] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15126.586778] pending: check_lifetime, do_cache_clean [15126.591541] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15126.598199] pending: neigh_periodic_work, neigh_periodic_work [15126.603316] workqueue mm_percpu_wq: flags=0x8 [15126.607535] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15126.611406] pending: vmstat_update, lru_add_drain_per_cpu [15126.614935] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15126.620622] pending: vmstat_update, lru_add_drain_per_cpu [15126.628660] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15126.633836] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15126.639457] workqueue xprtiod: flags=0xa [15126.641338] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15126.645490] pending: xprt_autoclose, xs_error_handle [15126.648100] workqueue ipv6_addrconf: flags=0x40008 [15126.650267] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15126.653283] pending: addrconf_verify_work [15126.655044] workqueue knbd0-recv: flags=0x1a [15126.656784] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15126.659638] in-flight: 651:recv_work [15126.661484] workqueue ptlrpc_pinger: flags=0x2 [15126.663327] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15126.666304] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15126.668925] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12489s workers=4 idle: 15979 31824 8 [15126.672745] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15117s workers=2 idle: 654 [15140.055461] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [15140.058025] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [15140.059187] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [15140.059196] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15140.059264] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15140.059271] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15140.059274] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15140.059706] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15140.059713] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15140.059718] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15140.059720] RDX: 0000000000000f54 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15140.059722] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15140.059724] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15140.059727] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15140.059730] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15140.059734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15140.059736] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15140.059745] Call Trace: [15140.059750] [15140.059757] ? show_regs.cold.9+0x22/0x2f [15140.059769] ? watchdog_timer_fn.cold.6+0x62/0x10a [15140.059776] ? __hrtimer_run_queues+0x1c2/0x4d0 [15140.059781] ? watchdog+0x40/0x40 [15140.059785] ? hrtimer_interrupt+0x128/0x2c0 [15140.059789] ? smp_apic_timer_interrupt+0xa6/0x270 [15140.059798] ? apic_timer_interrupt+0xf/0x20 [15140.059802] [15140.059805] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15140.059811] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15140.059815] do_raw_spin_lock+0x177/0x190 [15140.059820] _raw_spin_lock+0x21/0x30 [15140.059826] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15140.060123] obd_mod_free+0x46/0x140 [ptlrpc] [15140.062797] mdc_commit_open+0x91/0xb0 [mdc] [15140.062837] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15140.064157] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15140.065175] Modules linked in: lustre(O) [15140.065363] after_reply+0xe2c/0x1780 [ptlrpc] [15140.065691] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15140.065881] ? __next_timer_interrupt+0x160/0x160 [15140.065892] ptlrpcd+0xa82/0xe40 [ptlrpc] [15140.066093] ? do_wait_intr+0xf0/0xf0 [15140.066103] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15140.066311] kthread+0x1d1/0x200 [15140.066317] ? set_kthread_struct+0x70/0x70 [15140.066321] ret_from_fork+0x1f/0x30 [15140.074711] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15140.074787] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15140.074794] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15140.074798] RIP: 0010:native_safe_halt+0x2e/0x50 [15140.074811] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15140.074815] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15140.074820] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15140.074822] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15140.074825] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15140.074827] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15140.074829] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15140.074834] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15140.074836] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15140.074839] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15140.074847] Call Trace: [15140.074854] [15140.074860] ? show_regs.cold.9+0x22/0x2f [15140.074869] ? watchdog_timer_fn.cold.6+0x62/0x10a [15140.074876] ? __hrtimer_run_queues+0x1c2/0x4d0 [15140.074880] ? watchdog+0x40/0x40 [15140.074884] ? hrtimer_interrupt+0x128/0x2c0 [15140.074889] ? smp_apic_timer_interrupt+0xa6/0x270 [15140.074896] ? apic_timer_interrupt+0xf/0x20 [15140.074901] [15140.074904] ? native_safe_halt+0x2e/0x50 [15140.074908] kvm_wait+0xcd/0xe0 [15140.074917] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15140.074927] do_raw_spin_lock+0x177/0x190 [15140.074931] _raw_spin_lock+0x21/0x30 [15140.074936] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15140.075558] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15140.076250] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15140.081253] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15140.081516] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15140.081523] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15140.081526] RIP: 0010:native_safe_halt+0x2e/0x50 [15140.081538] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15140.081543] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15140.081547] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15140.081550] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15140.123500] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15140.123803] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15140.124007] ? woken_wake_function+0x30/0x30 [15140.124017] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15140.124223] kthread+0x1d1/0x200 [15140.124229] ? set_kthread_struct+0x70/0x70 [15140.124232] ret_from_fork+0x1f/0x30 [15140.649222] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15140.652460] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15140.655257] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15140.659721] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15140.662111] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15140.665217] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15140.668236] Call Trace: [15140.669538] [15140.670061] ? show_regs.cold.9+0x22/0x2f [15140.671553] ? watchdog_timer_fn.cold.6+0x62/0x10a [15140.673119] ? __hrtimer_run_queues+0x1c2/0x4d0 [15140.675691] ? watchdog+0x40/0x40 [15140.676625] ? hrtimer_interrupt+0x128/0x2c0 [15140.677648] ? smp_apic_timer_interrupt+0xa6/0x270 [15140.679019] ? apic_timer_interrupt+0xf/0x20 [15140.680153] [15140.680987] ? native_safe_halt+0x2e/0x50 [15140.682999] kvm_wait+0xcd/0xe0 [15140.684217] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15140.686501] do_raw_spin_lock+0x177/0x190 [15140.687608] _raw_spin_lock+0x21/0x30 [15140.688589] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15140.690837] ? do_raw_spin_unlock+0x75/0x190 [15140.693098] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15140.694777] ? do_wait_intr+0xf0/0xf0 [15140.696124] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15140.699525] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15140.701715] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15140.705227] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15140.707329] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15140.710653] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15140.712865] mdc_intent_lock+0x353/0x7e0 [mdc] [15140.714372] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15140.716243] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15140.718977] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15140.720977] lmv_intent_open+0x338/0x1150 [lmv] [15140.723916] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15140.726077] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15140.729265] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15140.731140] ll_intent_lock+0xc5/0xaa0 [lustre] [15140.733182] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15140.736114] ll_lookup_it+0xe0e/0x2270 [lustre] [15140.738832] ? do_raw_spin_unlock+0x75/0x190 [15140.740677] ? _raw_spin_unlock+0x12/0x30 [15140.743153] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15140.744818] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15140.746939] ll_atomic_open+0x36a/0xd70 [lustre] [15140.748602] lookup_open+0x356/0xb40 [15140.752591] do_last+0x313/0xfc0 [15140.754369] ? nd_jump_root+0xe5/0x160 [15140.757223] ? path_init+0x437/0x520 [15140.759511] path_openat+0xf7/0x500 [15140.761113] ? do_raw_spin_unlock+0x75/0x190 [15140.763196] do_filp_open+0x99/0x140 [15140.765055] ? getname_flags+0x6e/0x330 [15140.766639] ? __check_object_size+0xff/0x256 [15140.768867] ? do_raw_spin_unlock+0x75/0x190 [15140.772140] ? _raw_spin_unlock+0x12/0x30 [15140.774876] do_sys_openat2+0x2b4/0x410 [15140.776669] do_sys_open+0x73/0xa0 [15140.779866] __x64_sys_openat+0x24/0x30 [15140.782688] do_syscall_64+0xc1/0x440 [15140.784675] entry_SYSCALL_64_after_hwframe+0x49/0xae [15140.788552] RIP: 0033:0x7f53425a5332 [15140.790979] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15140.803553] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15140.811854] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15140.814173] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15140.821840] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15140.826067] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15140.830559] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15152.052427] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [15152.054973] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15152.080799] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15152.094017] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15152.104404] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15152.110395] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15152.118878] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15152.134263] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15152.136914] RAX: 0000000000007f86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15152.141975] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15152.144618] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15152.147331] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15152.149972] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15152.152675] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15152.156198] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15152.158320] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15152.161323] Call Trace: [15152.162285] [15152.163002] ? show_regs.cold.9+0x22/0x2f [15152.164568] ? watchdog_timer_fn.cold.6+0x62/0x10a [15152.166349] ? __hrtimer_run_queues+0x1c2/0x4d0 [15152.168022] ? watchdog+0x40/0x40 [15152.169370] ? hrtimer_interrupt+0x128/0x2c0 [15152.170923] ? smp_apic_timer_interrupt+0xa6/0x270 [15152.172769] ? apic_timer_interrupt+0xf/0x20 [15152.174428] [15152.175243] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15152.177347] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15152.179392] ? newidle_balance.isra.93+0x207/0x580 [15152.181226] do_raw_spin_lock+0x177/0x190 [15152.182686] _raw_spin_lock+0x21/0x30 [15152.184055] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15152.187226] process_one_work+0x2c8/0x700 [15152.188747] worker_thread+0x296/0x6e0 [15152.190250] ? rescuer_thread+0x570/0x570 [15152.191904] kthread+0x1d1/0x200 [15152.194043] ? set_kthread_struct+0x70/0x70 [15152.195861] ret_from_fork+0x1f/0x30 [15156.707054] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12519s! [15156.717930] Showing busy workqueues and worker pools: [15156.720731] workqueue events: flags=0x0 [15156.732564] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15156.744149] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15156.752716] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15156.762401] pending: cpu_latency_work [ptlrpc] [15156.766859] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15156.774424] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15156.782012] workqueue events_long: flags=0x0 [15156.788602] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15156.796621] pending: lu_objects_flush [obdclass] [15156.799780] workqueue events_unbound: flags=0x2 [15156.802300] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15156.805445] pending: flush_memcg_stats_dwork, idle_cull_fn [15156.810918] workqueue events_power_efficient: flags=0x80 [15156.815841] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15156.820009] pending: check_lifetime, do_cache_clean [15156.823553] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15156.828964] pending: neigh_periodic_work, neigh_periodic_work [15156.840203] workqueue mm_percpu_wq: flags=0x8 [15156.841845] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15156.853964] pending: vmstat_update, lru_add_drain_per_cpu [15156.856890] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15156.860834] pending: vmstat_update, lru_add_drain_per_cpu [15156.867826] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15156.877581] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15156.882955] workqueue xprtiod: flags=0xa [15156.891818] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15156.901389] pending: xprt_autoclose, xs_error_handle [15156.910420] workqueue ipv6_addrconf: flags=0x40008 [15156.913797] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15156.918559] pending: addrconf_verify_work [15156.925919] workqueue knbd0-recv: flags=0x1a [15156.929661] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15156.933875] in-flight: 651:recv_work [15156.937847] workqueue ptlrpc_pinger: flags=0x2 [15156.940764] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15156.943755] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15156.946021] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12519s workers=4 idle: 15979 31824 8 [15156.950083] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15147s workers=2 idle: 654 [15168.057020] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [15168.057864] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [15168.058007] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15168.058014] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15168.058089] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15168.058094] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15168.058097] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15168.058113] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15168.058117] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15168.058122] RAX: 0000000000001dab RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15168.058125] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15168.058127] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15168.058129] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15168.058132] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15168.058136] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15168.058138] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15168.058141] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15168.058149] Call Trace: [15168.058154] [15168.058159] ? show_regs.cold.9+0x22/0x2f [15168.058168] ? watchdog_timer_fn.cold.6+0x62/0x10a [15168.058174] ? __hrtimer_run_queues+0x1c2/0x4d0 [15168.058179] ? watchdog+0x40/0x40 [15168.058182] ? hrtimer_interrupt+0x128/0x2c0 [15168.058186] ? smp_apic_timer_interrupt+0xa6/0x270 [15168.058194] ? apic_timer_interrupt+0xf/0x20 [15168.058198] [15168.058201] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15168.058206] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15168.058212] do_raw_spin_lock+0x177/0x190 [15168.058218] _raw_spin_lock+0x21/0x30 [15168.058223] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15168.058626] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15168.059030] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15168.059237] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15168.059572] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15168.059912] ? woken_wake_function+0x30/0x30 [15168.059921] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15168.060186] kthread+0x1d1/0x200 [15168.060192] ? set_kthread_struct+0x70/0x70 [15168.060196] ret_from_fork+0x1f/0x30 [15168.060531] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15168.060596] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15168.060602] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15168.060605] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15168.060615] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15168.060619] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15168.060623] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15168.060626] RDX: 000000000000272e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15168.060628] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15168.060630] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15168.060632] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15168.060635] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15168.060638] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15168.060640] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15168.060648] Call Trace: [15168.060652] [15168.060655] ? show_regs.cold.9+0x22/0x2f [15168.060666] ? watchdog_timer_fn.cold.6+0x62/0x10a [15168.060672] ? __hrtimer_run_queues+0x1c2/0x4d0 [15168.060677] ? watchdog+0x40/0x40 [15168.060681] ? hrtimer_interrupt+0x128/0x2c0 [15168.060685] ? smp_apic_timer_interrupt+0xa6/0x270 [15168.060691] ? apic_timer_interrupt+0xf/0x20 [15168.060695] [15168.060698] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15168.060703] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15168.060707] do_raw_spin_lock+0x177/0x190 [15168.060712] _raw_spin_lock+0x21/0x30 [15168.060718] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15168.060921] obd_mod_free+0x46/0x140 [ptlrpc] [15168.061116] mdc_commit_open+0x91/0xb0 [mdc] [15168.061155] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15168.061354] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15168.061545] after_reply+0xe2c/0x1780 [ptlrpc] [15168.061736] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15168.073059] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 [15168.084088] ? __next_timer_interrupt+0x160/0x160 [15168.084111] ptlrpcd+0xa82/0xe40 [ptlrpc] [15168.084392] ? do_wait_intr+0xf0/0xf0 [15168.084403] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15168.131318] auth_rpcgss [15168.139127] kthread+0x1d1/0x200 [15168.139148] ? set_kthread_struct+0x70/0x70 [15168.139153] ret_from_fork+0x1f/0x30 [15168.555534] nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15168.579956] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15168.598474] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15168.616839] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15168.624601] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15168.647441] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15168.650707] RAX: 0000000000004d20 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15168.657622] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15168.669538] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15168.678452] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15168.685274] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15168.695763] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15168.702399] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15168.712309] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15168.718095] Call Trace: [15168.718906] [15168.719462] ? show_regs.cold.9+0x22/0x2f [15168.720578] ? watchdog_timer_fn.cold.6+0x62/0x10a [15168.732395] ? __hrtimer_run_queues+0x1c2/0x4d0 [15168.740284] ? watchdog+0x40/0x40 [15168.741565] ? hrtimer_interrupt+0x128/0x2c0 [15168.744470] ? smp_apic_timer_interrupt+0xa6/0x270 [15168.747435] ? apic_timer_interrupt+0xf/0x20 [15168.751961] [15168.753020] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15168.757494] do_raw_spin_lock+0x177/0x190 [15168.761947] _raw_spin_lock+0x21/0x30 [15168.765062] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15168.769732] ? do_raw_spin_unlock+0x75/0x190 [15168.773791] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15168.783982] ? do_wait_intr+0xf0/0xf0 [15168.787623] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15168.794345] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15168.796210] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15168.817353] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15168.828907] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15168.840025] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15168.846760] mdc_intent_lock+0x353/0x7e0 [mdc] [15168.854742] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15168.856371] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15168.869543] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15168.873779] lmv_intent_open+0x338/0x1150 [lmv] [15168.880471] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15168.889477] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15168.896519] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15168.899344] ll_intent_lock+0xc5/0xaa0 [lustre] [15168.902558] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15168.907697] ll_lookup_it+0xe0e/0x2270 [lustre] [15168.912594] ? do_raw_spin_unlock+0x75/0x190 [15168.917656] ? _raw_spin_unlock+0x12/0x30 [15168.920370] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15168.923484] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15168.926053] ll_atomic_open+0x36a/0xd70 [lustre] [15168.929442] lookup_open+0x356/0xb40 [15168.931909] do_last+0x313/0xfc0 [15168.937924] ? nd_jump_root+0xe5/0x160 [15168.939547] ? path_init+0x437/0x520 [15168.941494] path_openat+0xf7/0x500 [15168.947312] ? do_raw_spin_unlock+0x75/0x190 [15168.949634] do_filp_open+0x99/0x140 [15168.952249] ? getname_flags+0x6e/0x330 [15168.953760] ? __check_object_size+0xff/0x256 [15168.957550] ? do_raw_spin_unlock+0x75/0x190 [15168.959461] ? _raw_spin_unlock+0x12/0x30 [15168.960960] do_sys_openat2+0x2b4/0x410 [15168.962445] do_sys_open+0x73/0xa0 [15168.963735] __x64_sys_openat+0x24/0x30 [15168.965217] do_syscall_64+0xc1/0x440 [15168.966433] entry_SYSCALL_64_after_hwframe+0x49/0xae [15168.968445] RIP: 0033:0x7f53425a5332 [15168.969720] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15168.976311] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15168.979369] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15168.983339] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15168.986730] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15168.996310] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15169.004422] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15184.054404] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [15184.068984] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15184.108428] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15184.118853] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15184.127570] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15184.131261] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15184.139456] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15184.154087] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15184.162963] RAX: 0000000000007e85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15184.167430] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15184.174933] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15184.185956] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15184.191834] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15184.201982] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15184.206416] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15184.211126] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15184.218459] Call Trace: [15184.219829] [15184.221438] ? show_regs.cold.9+0x22/0x2f [15184.223892] ? watchdog_timer_fn.cold.6+0x62/0x10a [15184.226404] ? __hrtimer_run_queues+0x1c2/0x4d0 [15184.229235] ? watchdog+0x40/0x40 [15184.233110] ? hrtimer_interrupt+0x128/0x2c0 [15184.235910] ? smp_apic_timer_interrupt+0xa6/0x270 [15184.241800] ? apic_timer_interrupt+0xf/0x20 [15184.245171] [15184.247227] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15184.251244] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15184.255162] ? newidle_balance.isra.93+0x207/0x580 [15184.257360] do_raw_spin_lock+0x177/0x190 [15184.261158] _raw_spin_lock+0x21/0x30 [15184.262948] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15184.265468] process_one_work+0x2c8/0x700 [15184.270565] worker_thread+0x296/0x6e0 [15184.271918] ? rescuer_thread+0x570/0x570 [15184.273965] kthread+0x1d1/0x200 [15184.275872] ? set_kthread_struct+0x70/0x70 [15184.279352] ret_from_fork+0x1f/0x30 [15187.425056] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12550s! [15187.433869] Showing busy workqueues and worker pools: [15187.436321] workqueue events: flags=0x0 [15187.440977] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15187.444259] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15187.451187] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15187.459910] pending: cpu_latency_work [ptlrpc] [15187.463404] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15187.466473] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15187.473136] workqueue events_long: flags=0x0 [15187.475455] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15187.480338] pending: lu_objects_flush [obdclass] [15187.483390] workqueue events_unbound: flags=0x2 [15187.485779] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15187.489033] pending: flush_memcg_stats_dwork, idle_cull_fn [15187.492268] workqueue events_power_efficient: flags=0x80 [15187.497921] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15187.507434] pending: check_lifetime, do_cache_clean [15187.510488] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15187.519225] pending: neigh_periodic_work, neigh_periodic_work [15187.525456] workqueue mm_percpu_wq: flags=0x8 [15187.528177] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15187.535930] pending: vmstat_update, lru_add_drain_per_cpu [15187.539490] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15187.543350] pending: vmstat_update, lru_add_drain_per_cpu [15187.547462] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15187.552287] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15187.558468] workqueue xprtiod: flags=0xa [15187.566083] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15187.569380] pending: xprt_autoclose, xs_error_handle [15187.571435] workqueue ipv6_addrconf: flags=0x40008 [15187.574634] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15187.583305] pending: addrconf_verify_work [15187.586907] workqueue knbd0-recv: flags=0x1a [15187.589264] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15187.593393] in-flight: 651:recv_work [15187.597336] workqueue ptlrpc_pinger: flags=0x2 [15187.599941] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15187.607469] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15187.615251] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12550s workers=4 idle: 15979 31824 8 [15187.627235] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15178s workers=2 idle: 654 [15196.056715] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [15196.057006] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [15196.058025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15196.058032] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15196.058104] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15196.058190] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15196.058193] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15196.058207] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15196.058210] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15196.058215] RAX: 000000000000460c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15196.058218] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15196.058293] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15196.058295] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15196.058298] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15196.058301] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15196.058304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15196.058307] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15196.058316] Call Trace: [15196.058375] [15196.058383] ? show_regs.cold.9+0x22/0x2f [15196.058394] ? watchdog_timer_fn.cold.6+0x62/0x10a [15196.058400] ? __hrtimer_run_queues+0x1c2/0x4d0 [15196.058404] ? watchdog+0x40/0x40 [15196.058409] ? hrtimer_interrupt+0x128/0x2c0 [15196.058413] ? smp_apic_timer_interrupt+0xa6/0x270 [15196.058423] ? apic_timer_interrupt+0xf/0x20 [15196.058427] [15196.058431] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15196.058437] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15196.058443] do_raw_spin_lock+0x177/0x190 [15196.058448] _raw_spin_lock+0x21/0x30 [15196.058454] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15196.058854] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15196.059264] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15196.059618] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15196.059884] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15196.060372] ? woken_wake_function+0x30/0x30 [15196.060384] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15196.060724] kthread+0x1d1/0x200 [15196.060730] ? set_kthread_struct+0x70/0x70 [15196.060735] ret_from_fork+0x1f/0x30 [15196.065160] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15196.065241] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15196.065250] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15196.065253] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15196.065269] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15196.065274] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15196.065279] RAX: 00000000000042e6 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15196.065282] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15196.065284] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15196.065286] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15196.065289] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15196.065292] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15196.065295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15196.065297] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15196.065306] Call Trace: [15196.065311] [15196.065317] ? show_regs.cold.9+0x22/0x2f [15196.065328] ? watchdog_timer_fn.cold.6+0x62/0x10a [15196.065335] ? __hrtimer_run_queues+0x1c2/0x4d0 [15196.065339] ? watchdog+0x40/0x40 [15196.065343] ? hrtimer_interrupt+0x128/0x2c0 [15196.065479] ? smp_apic_timer_interrupt+0xa6/0x270 [15196.065491] ? apic_timer_interrupt+0xf/0x20 [15196.065497] [15196.065503] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15196.065527] do_raw_spin_lock+0x177/0x190 [15196.065532] _raw_spin_lock+0x21/0x30 [15196.065540] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15196.067155] ? do_raw_spin_unlock+0x75/0x190 [15196.067164] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15196.067379] ? do_wait_intr+0xf0/0xf0 [15196.067389] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15196.067656] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15196.068099] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15196.068315] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15196.069050] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15196.074152] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15196.074227] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15196.074234] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15196.074237] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15196.074252] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15196.074256] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15196.074260] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15196.074262] RDX: 00000000000022e6 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15196.074265] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15196.074266] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15196.074269] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15196.074272] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15196.074274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15196.074277] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15196.074285] Call Trace: [15196.074306] [15196.074313] ? show_regs.cold.9+0x22/0x2f [15196.074324] ? watchdog_timer_fn.cold.6+0x62/0x10a [15196.074330] ? __hrtimer_run_queues+0x1c2/0x4d0 [15196.074335] ? watchdog+0x40/0x40 [15196.074339] ? hrtimer_interrupt+0x128/0x2c0 [15196.074508] ? smp_apic_timer_interrupt+0xa6/0x270 [15196.074525] ? apic_timer_interrupt+0xf/0x20 [15196.074530] [15196.074534] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15196.074538] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15196.074542] do_raw_spin_lock+0x177/0x190 [15196.074547] _raw_spin_lock+0x21/0x30 [15196.074553] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15196.075062] obd_mod_free+0x46/0x140 [ptlrpc] [15196.075254] mdc_commit_open+0x91/0xb0 [mdc] [15196.075301] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15196.076687] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15196.086878] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15196.086947] mdc_intent_lock+0x353/0x7e0 [mdc] [15196.086979] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15196.087122] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15196.087813] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15196.087850] lmv_intent_open+0x338/0x1150 [lmv] [15196.087894] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15196.087997] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15196.088085] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15196.088109] ll_intent_lock+0xc5/0xaa0 [lustre] [15196.088310] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15196.088408] ll_lookup_it+0xe0e/0x2270 [lustre] [15196.088498] ? do_raw_spin_unlock+0x75/0x190 [15196.088507] ? _raw_spin_unlock+0x12/0x30 [15196.088514] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15196.088544] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15196.088562] ll_atomic_open+0x36a/0xd70 [lustre] [15196.088774] lookup_open+0x356/0xb40 [15196.088784] do_last+0x313/0xfc0 [15196.088788] ? nd_jump_root+0xe5/0x160 [15196.088793] ? path_init+0x437/0x520 [15196.088798] path_openat+0xf7/0x500 [15196.088803] ? do_raw_spin_unlock+0x75/0x190 [15196.088808] do_filp_open+0x99/0x140 [15196.088814] ? getname_flags+0x6e/0x330 [15196.088819] ? __check_object_size+0xff/0x256 [15196.088827] ? do_raw_spin_unlock+0x75/0x190 [15196.088831] ? _raw_spin_unlock+0x12/0x30 [15196.088836] do_sys_openat2+0x2b4/0x410 [15196.088840] do_sys_open+0x73/0xa0 [15196.088844] __x64_sys_openat+0x24/0x30 [15196.088848] do_syscall_64+0xc1/0x440 [15196.088923] entry_SYSCALL_64_after_hwframe+0x49/0xae [15196.088930] RIP: 0033:0x7f53425a5332 [15196.088943] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15196.088947] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15196.088952] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15196.088954] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15196.088957] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15196.088959] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15196.088961] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15196.808996] after_reply+0xe2c/0x1780 [ptlrpc] [15196.810703] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15196.813293] ? __next_timer_interrupt+0x160/0x160 [15196.819053] ptlrpcd+0xa82/0xe40 [ptlrpc] [15196.823085] ? do_wait_intr+0xf0/0xf0 [15196.827830] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15196.829799] kthread+0x1d1/0x200 [15196.835449] ? set_kthread_struct+0x70/0x70 [15196.836444] ret_from_fork+0x1f/0x30 [15212.052022] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [15212.056974] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15212.076368] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15212.079810] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15212.088667] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15212.091417] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15212.096696] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15212.105347] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15212.108396] RAX: 0000000000004b30 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15212.111218] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15212.114061] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15212.116463] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15212.119349] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15212.122690] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15212.125857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15212.128982] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15212.132418] Call Trace: [15212.134471] [15212.135404] ? show_regs.cold.9+0x22/0x2f [15212.137310] ? watchdog_timer_fn.cold.6+0x62/0x10a [15212.141868] ? __hrtimer_run_queues+0x1c2/0x4d0 [15212.144143] ? watchdog+0x40/0x40 [15212.145679] ? hrtimer_interrupt+0x128/0x2c0 [15212.147515] ? smp_apic_timer_interrupt+0xa6/0x270 [15212.150540] ? apic_timer_interrupt+0xf/0x20 [15212.152351] [15212.153307] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15212.155344] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15212.157623] ? newidle_balance.isra.93+0x207/0x580 [15212.160526] do_raw_spin_lock+0x177/0x190 [15212.163299] _raw_spin_lock+0x21/0x30 [15212.164937] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15212.168865] process_one_work+0x2c8/0x700 [15212.170320] worker_thread+0x296/0x6e0 [15212.173559] ? rescuer_thread+0x570/0x570 [15212.174909] kthread+0x1d1/0x200 [15212.176769] ? set_kthread_struct+0x70/0x70 [15212.178561] ret_from_fork+0x1f/0x30 [15217.652239] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12580s! [15217.663242] Showing busy workqueues and worker pools: [15217.669267] workqueue events: flags=0x0 [15217.675176] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15217.682217] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15217.694783] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15217.700231] pending: cpu_latency_work [ptlrpc] [15217.704187] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15217.711209] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15217.724271] workqueue events_long: flags=0x0 [15217.730421] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15217.738247] pending: lu_objects_flush [obdclass] [15217.740888] workqueue events_unbound: flags=0x2 [15217.751747] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15217.756236] pending: flush_memcg_stats_dwork, idle_cull_fn [15217.768153] workqueue events_power_efficient: flags=0x80 [15217.775197] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15217.784562] pending: check_lifetime, do_cache_clean [15217.790556] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15217.799234] pending: neigh_periodic_work, neigh_periodic_work [15217.807719] workqueue mm_percpu_wq: flags=0x8 [15217.827701] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15217.847725] pending: vmstat_update, lru_add_drain_per_cpu [15217.849769] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15217.884013] pending: vmstat_update, lru_add_drain_per_cpu [15217.904196] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15217.918887] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15217.930345] workqueue xprtiod: flags=0xa [15217.935907] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15217.949582] pending: xprt_autoclose, xs_error_handle [15217.960689] workqueue ipv6_addrconf: flags=0x40008 [15217.973729] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15217.986701] pending: addrconf_verify_work [15218.016117] workqueue knbd0-recv: flags=0x1a [15218.044977] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15218.062682] in-flight: 651:recv_work [15218.070677] workqueue ptlrpc_pinger: flags=0x2 [15218.078462] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15218.097247] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15218.113614] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12581s workers=4 idle: 15979 31824 8 [15218.137375] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15208s workers=2 idle: 654 [15224.055025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [15224.057012] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [15224.057598] Modules linked in: [15224.058022] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15224.058028] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15224.058100] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15224.058105] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15224.058108] RIP: 0010:native_safe_halt+0x2e/0x50 [15224.058121] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15224.058125] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [15224.058129] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15224.058132] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15224.058134] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15224.058136] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15224.058139] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15224.058142] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15224.058145] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15224.058148] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15224.058157] Call Trace: [15224.058163] [15224.058169] ? show_regs.cold.9+0x22/0x2f [15224.058179] ? watchdog_timer_fn.cold.6+0x62/0x10a [15224.058185] ? __hrtimer_run_queues+0x1c2/0x4d0 [15224.058188] ? watchdog+0x40/0x40 [15224.058193] ? hrtimer_interrupt+0x128/0x2c0 [15224.058197] ? smp_apic_timer_interrupt+0xa6/0x270 [15224.058203] ? apic_timer_interrupt+0xf/0x20 [15224.058207] [15224.058210] ? native_safe_halt+0x2e/0x50 [15224.058213] kvm_wait+0xcd/0xe0 [15224.058221] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15224.058237] do_raw_spin_lock+0x177/0x190 [15224.058243] _raw_spin_lock+0x21/0x30 [15224.058248] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15224.058512] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15224.058708] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15224.058960] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15224.059187] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15224.059415] ? woken_wake_function+0x30/0x30 [15224.059424] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15224.059653] kthread+0x1d1/0x200 [15224.059660] ? set_kthread_struct+0x70/0x70 [15224.059664] ret_from_fork+0x1f/0x30 [15224.064441] Modules linked in: [15224.069844] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15224.070360] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15224.070366] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15224.070368] RIP: 0010:native_safe_halt+0x2e/0x50 [15224.070381] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15224.070385] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15224.070390] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15224.070393] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15224.070396] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15224.070397] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15224.070400] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15224.070403] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15224.070406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15224.070409] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15224.070418] Call Trace: [15224.070425] [15224.070431] ? show_regs.cold.9+0x22/0x2f [15224.070441] ? watchdog_timer_fn.cold.6+0x62/0x10a [15224.070929] ? __hrtimer_run_queues+0x1c2/0x4d0 [15224.070938] ? watchdog+0x40/0x40 [15224.070943] ? hrtimer_interrupt+0x128/0x2c0 [15224.070948] ? smp_apic_timer_interrupt+0xa6/0x270 [15224.070955] ? apic_timer_interrupt+0xf/0x20 [15224.071088] [15224.071093] ? native_safe_halt+0x2e/0x50 [15224.071097] kvm_wait+0xcd/0xe0 [15224.071106] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15224.071119] do_raw_spin_lock+0x177/0x190 [15224.071124] _raw_spin_lock+0x21/0x30 [15224.071129] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15224.083367] lustre(O) [15224.127193] ? do_raw_spin_unlock+0x75/0x190 [15224.135136] osp(O) [15224.142363] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15224.142703] ? do_wait_intr+0xf0/0xf0 [15224.142720] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15224.146332] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15224.146407] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15224.146415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15224.146418] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15224.146433] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15224.146438] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15224.146443] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15224.146445] RDX: 000000000000384c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15224.146448] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15224.146450] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15224.146452] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15224.146456] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15224.146459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15224.146461] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15224.146470] Call Trace: [15224.146475] [15224.146480] ? show_regs.cold.9+0x22/0x2f [15224.146490] ? watchdog_timer_fn.cold.6+0x62/0x10a [15224.146496] ? __hrtimer_run_queues+0x1c2/0x4d0 [15224.146501] ? watchdog+0x40/0x40 [15224.146505] ? hrtimer_interrupt+0x128/0x2c0 [15224.146509] ? smp_apic_timer_interrupt+0xa6/0x270 [15224.146519] ? apic_timer_interrupt+0xf/0x20 [15224.146524] [15224.146527] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15224.146532] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15224.149436] do_raw_spin_lock+0x177/0x190 [15224.159176] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15224.159472] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15224.159681] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15224.159793] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15224.159831] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15224.165392] _raw_spin_lock+0x21/0x30 [15224.165416] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15224.172762] mdc_intent_lock+0x353/0x7e0 [mdc] [15224.172838] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15224.173006] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15224.173292] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15224.173323] lmv_intent_open+0x338/0x1150 [lmv] [15224.173368] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15224.173454] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15224.173537] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15224.173560] ll_intent_lock+0xc5/0xaa0 [lustre] [15224.173644] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15224.178866] obd_mod_free+0x46/0x140 [ptlrpc] [15224.179155] mdc_commit_open+0x91/0xb0 [mdc] [15224.179196] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15224.179381] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15224.179572] after_reply+0xe2c/0x1780 [ptlrpc] [15224.179758] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15224.189233] ll_lookup_it+0xe0e/0x2270 [lustre] [15224.189373] ? do_raw_spin_unlock+0x75/0x190 [15224.189383] ? _raw_spin_unlock+0x12/0x30 [15224.189390] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15224.189421] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15224.189443] ll_atomic_open+0x36a/0xd70 [lustre] [15224.189533] lookup_open+0x356/0xb40 [15224.189542] do_last+0x313/0xfc0 [15224.189546] ? nd_jump_root+0xe5/0x160 [15224.189551] ? path_init+0x437/0x520 [15224.189555] path_openat+0xf7/0x500 [15224.189560] ? do_raw_spin_unlock+0x75/0x190 [15224.189565] do_filp_open+0x99/0x140 [15224.189572] ? getname_flags+0x6e/0x330 [15224.189577] ? __check_object_size+0xff/0x256 [15224.189584] ? do_raw_spin_unlock+0x75/0x190 [15224.189588] ? _raw_spin_unlock+0x12/0x30 [15224.189592] do_sys_openat2+0x2b4/0x410 [15224.189597] do_sys_open+0x73/0xa0 [15224.189601] __x64_sys_openat+0x24/0x30 [15224.189605] do_syscall_64+0xc1/0x440 [15224.189611] entry_SYSCALL_64_after_hwframe+0x49/0xae [15224.189617] RIP: 0033:0x7f53425a5332 [15224.189626] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15224.189629] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15224.189635] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15224.189637] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15224.189640] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15224.189642] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15224.189644] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15224.642702] ? __next_timer_interrupt+0x160/0x160 [15224.644926] ptlrpcd+0xa82/0xe40 [ptlrpc] [15224.647135] ? do_wait_intr+0xf0/0xf0 [15224.648751] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15224.651638] kthread+0x1d1/0x200 [15224.653089] ? set_kthread_struct+0x70/0x70 [15224.655040] ret_from_fork+0x1f/0x30 [15244.052024] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [15244.054880] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15244.071916] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15244.076324] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15244.079377] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15244.081640] RIP: 0010:native_safe_halt+0x2e/0x50 [15244.082839] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15244.087993] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15244.090854] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15244.093135] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15244.095682] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15244.099042] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15244.101313] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15244.104005] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15244.106995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15244.110101] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15244.112760] Call Trace: [15244.113486] [15244.114388] ? show_regs.cold.9+0x22/0x2f [15244.115794] ? watchdog_timer_fn.cold.6+0x62/0x10a [15244.117676] ? __hrtimer_run_queues+0x1c2/0x4d0 [15244.119316] ? watchdog+0x40/0x40 [15244.120658] ? hrtimer_interrupt+0x128/0x2c0 [15244.123006] ? smp_apic_timer_interrupt+0xa6/0x270 [15244.125011] ? apic_timer_interrupt+0xf/0x20 [15244.127082] [15244.127839] ? native_safe_halt+0x2e/0x50 [15244.129365] kvm_wait+0xcd/0xe0 [15244.130706] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15244.132501] ? newidle_balance.isra.93+0x207/0x580 [15244.134010] do_raw_spin_lock+0x177/0x190 [15244.135588] _raw_spin_lock+0x21/0x30 [15244.137027] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15244.139259] process_one_work+0x2c8/0x700 [15244.140889] worker_thread+0x296/0x6e0 [15244.142139] ? rescuer_thread+0x570/0x570 [15244.143494] kthread+0x1d1/0x200 [15244.144670] ? set_kthread_struct+0x70/0x70 [15244.146303] ret_from_fork+0x1f/0x30 [15248.355036] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12611s! [15248.357509] Showing busy workqueues and worker pools: [15248.358980] workqueue events: flags=0x0 [15248.360035] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15248.361928] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15248.365291] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15248.369660] pending: cpu_latency_work [ptlrpc] [15248.371713] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15248.373593] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15248.376290] workqueue events_long: flags=0x0 [15248.377618] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15248.379544] pending: lu_objects_flush [obdclass] [15248.380987] workqueue events_unbound: flags=0x2 [15248.382611] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15248.384300] pending: flush_memcg_stats_dwork, idle_cull_fn [15248.385888] workqueue events_power_efficient: flags=0x80 [15248.387118] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15248.388699] pending: check_lifetime, do_cache_clean [15248.390026] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15248.391743] pending: neigh_periodic_work, neigh_periodic_work [15248.393518] workqueue mm_percpu_wq: flags=0x8 [15248.394646] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15248.396398] pending: vmstat_update, lru_add_drain_per_cpu [15248.398254] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15248.400126] pending: vmstat_update, lru_add_drain_per_cpu [15248.401591] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15248.403565] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15248.405177] workqueue xprtiod: flags=0xa [15248.406201] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15248.407790] pending: xprt_autoclose, xs_error_handle [15248.409163] workqueue ipv6_addrconf: flags=0x40008 [15248.410483] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15248.412174] pending: addrconf_verify_work [15248.413295] workqueue knbd0-recv: flags=0x1a [15248.414838] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15248.416697] in-flight: 651:recv_work [15248.417767] workqueue ptlrpc_pinger: flags=0x2 [15248.419124] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15248.420785] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15248.422631] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12611s workers=4 idle: 15979 31824 8 [15248.424888] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15238s workers=2 idle: 654 [15252.057891] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [15252.059282] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15252.061025] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15252.061098] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15252.061105] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15252.061107] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15252.061122] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15252.065684] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul [15252.071338] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [15252.071352] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15252.071437] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15252.071444] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15252.071447] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15252.071462] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15252.071466] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15252.071471] RAX: 0000000000002405 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15252.071473] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15252.071476] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15252.071478] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15252.071480] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15252.071483] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15252.071486] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15252.071489] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15252.071499] Call Trace: [15252.071505] [15252.071511] ? show_regs.cold.9+0x22/0x2f [15252.071522] ? watchdog_timer_fn.cold.6+0x62/0x10a [15252.071529] ? __hrtimer_run_queues+0x1c2/0x4d0 [15252.071534] ? watchdog+0x40/0x40 [15252.071553] ? hrtimer_interrupt+0x128/0x2c0 [15252.071557] ? smp_apic_timer_interrupt+0xa6/0x270 [15252.071565] ? apic_timer_interrupt+0xf/0x20 [15252.071571] [15252.071574] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15252.071581] do_raw_spin_lock+0x177/0x190 [15252.071586] _raw_spin_lock+0x21/0x30 [15252.071591] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15252.075151] ? do_raw_spin_unlock+0x75/0x190 [15252.075163] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15252.078825] ? do_wait_intr+0xf0/0xf0 [15252.078840] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15252.082165] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15252.085628] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15252.089820] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15252.090342] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15252.090401] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15252.090434] mdc_intent_lock+0x353/0x7e0 [mdc] [15252.090464] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15252.092386] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15252.094056] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15252.094068] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15252.094070] RDX: 0000000000004b94 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15252.094073] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15252.094075] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15252.094078] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15252.094053] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15252.094083] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15252.094088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15252.094090] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15252.094101] Call Trace: [15252.094110] [15252.094102] lmv_intent_open+0x338/0x1150 [lmv] [15252.094114] ? show_regs.cold.9+0x22/0x2f [15252.094128] ? watchdog_timer_fn.cold.6+0x62/0x10a [15252.094137] ? __hrtimer_run_queues+0x1c2/0x4d0 [15252.094145] ? watchdog+0x40/0x40 [15252.094150] ? hrtimer_interrupt+0x128/0x2c0 [15252.094154] ? smp_apic_timer_interrupt+0xa6/0x270 [15252.094165] ? apic_timer_interrupt+0xf/0x20 [15252.094171] [15252.094174] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15252.094182] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15252.094187] do_raw_spin_lock+0x177/0x190 [15252.094191] _raw_spin_lock+0x21/0x30 [15252.094197] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15252.094475] obd_mod_free+0x46/0x140 [ptlrpc] [15252.094683] mdc_commit_open+0x91/0xb0 [mdc] [15252.094713] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15252.094909] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15252.095105] after_reply+0xe2c/0x1780 [ptlrpc] [15252.094140] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15252.095774] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15252.095864] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15252.095899] ll_intent_lock+0xc5/0xaa0 [lustre] [15252.095989] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15252.096077] ll_lookup_it+0xe0e/0x2270 [lustre] [15252.096162] ? do_raw_spin_unlock+0x75/0x190 [15252.096169] ? _raw_spin_unlock+0x12/0x30 [15252.096175] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15252.096205] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15252.096233] ll_atomic_open+0x36a/0xd70 [lustre] [15252.096321] lookup_open+0x356/0xb40 [15252.096332] do_last+0x313/0xfc0 [15252.096337] ? nd_jump_root+0xe5/0x160 [15252.096342] ? path_init+0x437/0x520 [15252.096347] path_openat+0xf7/0x500 [15252.096352] ? do_raw_spin_unlock+0x75/0x190 [15252.096357] do_filp_open+0x99/0x140 [15252.096363] ? getname_flags+0x6e/0x330 [15252.096380] ? __check_object_size+0xff/0x256 [15252.096388] ? do_raw_spin_unlock+0x75/0x190 [15252.096393] ? _raw_spin_unlock+0x12/0x30 [15252.096397] do_sys_openat2+0x2b4/0x410 [15252.096402] do_sys_open+0x73/0xa0 [15252.096406] __x64_sys_openat+0x24/0x30 [15252.096410] do_syscall_64+0xc1/0x440 [15252.096417] entry_SYSCALL_64_after_hwframe+0x49/0xae [15252.096423] RIP: 0033:0x7f53425a5332 [15252.096432] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15252.096436] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15252.096441] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15252.096444] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15252.096446] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15252.096448] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15252.096451] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15252.098902] crc32c_intel [15252.113820] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15252.114301] ? __next_timer_interrupt+0x160/0x160 [15252.114314] ptlrpcd+0xa82/0xe40 [ptlrpc] [15252.114952] ? do_wait_intr+0xf0/0xf0 [15252.114976] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15252.115184] kthread+0x1d1/0x200 [15252.115190] ? set_kthread_struct+0x70/0x70 [15252.115194] ret_from_fork+0x1f/0x30 [15252.850950] ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15252.862249] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15252.870365] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15252.877584] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15252.883702] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15252.897527] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15252.904409] RAX: 0000000000001ed2 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15252.911980] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15252.915301] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15252.921908] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15252.928086] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15252.932529] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15252.939007] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15252.944697] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15252.947628] Call Trace: [15252.948574] [15252.949419] ? show_regs.cold.9+0x22/0x2f [15252.950713] ? watchdog_timer_fn.cold.6+0x62/0x10a [15252.952492] ? __hrtimer_run_queues+0x1c2/0x4d0 [15252.954417] ? watchdog+0x40/0x40 [15252.955874] ? hrtimer_interrupt+0x128/0x2c0 [15252.957764] ? smp_apic_timer_interrupt+0xa6/0x270 [15252.959773] ? apic_timer_interrupt+0xf/0x20 [15252.961574] [15252.962508] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15252.964819] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15252.969515] do_raw_spin_lock+0x177/0x190 [15252.972176] _raw_spin_lock+0x21/0x30 [15252.976155] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15252.980238] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15252.986190] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15252.992538] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15252.999035] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15253.001429] ? woken_wake_function+0x30/0x30 [15253.006710] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15253.013207] kthread+0x1d1/0x200 [15253.016211] ? set_kthread_struct+0x70/0x70 [15253.020022] ret_from_fork+0x1f/0x30 [15276.052013] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [15276.055156] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15276.100808] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15276.120168] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15276.126345] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15276.129915] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15276.136710] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15276.162162] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15276.168889] RAX: 000000000000561d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15276.176626] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15276.184439] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15276.187781] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15276.192512] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15276.196115] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15276.200877] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15276.205727] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15276.209588] Call Trace: [15276.210916] [15276.212319] ? show_regs.cold.9+0x22/0x2f [15276.214443] ? watchdog_timer_fn.cold.6+0x62/0x10a [15276.217453] ? __hrtimer_run_queues+0x1c2/0x4d0 [15276.226534] ? watchdog+0x40/0x40 [15276.227969] ? hrtimer_interrupt+0x128/0x2c0 [15276.230420] ? smp_apic_timer_interrupt+0xa6/0x270 [15276.239357] ? apic_timer_interrupt+0xf/0x20 [15276.241821] [15276.244163] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15276.251357] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15276.255491] ? newidle_balance.isra.93+0x207/0x580 [15276.260648] do_raw_spin_lock+0x177/0x190 [15276.266449] _raw_spin_lock+0x21/0x30 [15276.273473] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15276.277501] process_one_work+0x2c8/0x700 [15276.280082] worker_thread+0x296/0x6e0 [15276.282497] ? rescuer_thread+0x570/0x570 [15276.289524] kthread+0x1d1/0x200 [15276.292742] ? set_kthread_struct+0x70/0x70 [15276.297749] ret_from_fork+0x1f/0x30 [15278.563052] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12641s! [15278.565855] Showing busy workqueues and worker pools: [15278.573630] workqueue events: flags=0x0 [15278.576315] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15278.586510] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15278.602351] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15278.611494] pending: cpu_latency_work [ptlrpc] [15278.618808] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15278.629337] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15278.639686] workqueue events_long: flags=0x0 [15278.641074] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15278.649409] pending: lu_objects_flush [obdclass] [15278.652191] workqueue events_unbound: flags=0x2 [15278.653605] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15278.663679] pending: flush_memcg_stats_dwork, idle_cull_fn [15278.677275] workqueue events_power_efficient: flags=0x80 [15278.679221] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15278.696311] pending: check_lifetime, do_cache_clean [15278.704555] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15278.706809] pending: neigh_periodic_work, neigh_periodic_work [15278.723348] workqueue mm_percpu_wq: flags=0x8 [15278.724530] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15278.737315] pending: vmstat_update, lru_add_drain_per_cpu [15278.746337] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15278.762378] pending: vmstat_update, lru_add_drain_per_cpu [15278.769789] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15278.779710] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15278.795402] workqueue xprtiod: flags=0xa [15278.796912] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15278.807758] pending: xprt_autoclose, xs_error_handle [15278.809494] workqueue ipv6_addrconf: flags=0x40008 [15278.820315] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15278.826767] pending: addrconf_verify_work [15278.830693] workqueue knbd0-recv: flags=0x1a [15278.834709] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15278.846244] in-flight: 651:recv_work [15278.847620] workqueue ptlrpc_pinger: flags=0x2 [15278.861106] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15278.872815] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15278.887164] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12641s workers=4 idle: 15979 31824 8 [15278.898618] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15269s workers=2 idle: 654 [15280.058542] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [15280.065947] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15280.067148] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [15280.067480] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15280.067487] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15280.067550] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15280.067556] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15280.067559] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15280.067574] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15280.067578] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15280.067583] RAX: 000000000000650a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15280.067597] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15280.067600] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15280.067603] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15280.067606] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15280.067609] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15280.067612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15280.067614] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15280.067623] Call Trace: [15280.067630] [15280.067636] ? show_regs.cold.9+0x22/0x2f [15280.067645] ? watchdog_timer_fn.cold.6+0x62/0x10a [15280.067653] ? __hrtimer_run_queues+0x1c2/0x4d0 [15280.067657] ? watchdog+0x40/0x40 [15280.067661] ? hrtimer_interrupt+0x128/0x2c0 [15280.067664] ? smp_apic_timer_interrupt+0xa6/0x270 [15280.067673] ? apic_timer_interrupt+0xf/0x20 [15280.067678] [15280.067680] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15280.067685] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15280.067691] do_raw_spin_lock+0x177/0x190 [15280.067696] _raw_spin_lock+0x21/0x30 [15280.067700] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15280.067984] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15280.068171] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15280.068392] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15280.068603] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15280.068828] ? woken_wake_function+0x30/0x30 [15280.068838] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15280.076747] kthread+0x1d1/0x200 [15280.076759] ? set_kthread_struct+0x70/0x70 [15280.076764] ret_from_fork+0x1f/0x30 [15280.088077] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15280.088092] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15280.088095] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15280.088111] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15280.088115] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15280.088121] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15280.088123] RDX: 0000000000007c93 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15280.088125] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15280.088128] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15280.088130] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15280.088266] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15280.088268] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15280.088270] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15280.088278] Call Trace: [15280.088284] [15280.088291] ? show_regs.cold.9+0x22/0x2f [15280.088301] ? watchdog_timer_fn.cold.6+0x62/0x10a [15280.088309] ? __hrtimer_run_queues+0x1c2/0x4d0 [15280.088312] ? watchdog+0x40/0x40 [15280.088315] ? hrtimer_interrupt+0x128/0x2c0 [15280.088318] ? smp_apic_timer_interrupt+0xa6/0x270 [15280.088325] ? apic_timer_interrupt+0xf/0x20 [15280.088329] [15280.088331] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15280.088334] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15280.088337] do_raw_spin_lock+0x177/0x190 [15280.088340] _raw_spin_lock+0x21/0x30 [15280.088347] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15280.088586] obd_mod_free+0x46/0x140 [ptlrpc] [15280.091097] mdc_commit_open+0x91/0xb0 [mdc] [15280.091196] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15280.091363] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15280.091487] after_reply+0xe2c/0x1780 [ptlrpc] [15280.091610] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15280.095034] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15280.095104] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15280.095110] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15280.095112] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15280.095125] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15280.095128] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [15280.100533] ? __next_timer_interrupt+0x160/0x160 [15280.100552] ptlrpcd+0xa82/0xe40 [ptlrpc] [15280.127033] ORIG_RAX: ffffffffffffff13 [15280.127044] RAX: 0000000000007b83 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15280.127048] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15280.127050] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15280.127052] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15280.127054] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15280.127058] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15280.127061] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15280.127064] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15280.127072] Call Trace: [15280.127079] [15280.131736] ? do_wait_intr+0xf0/0xf0 [15280.131762] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15280.132137] kthread+0x1d1/0x200 [15280.132144] ? set_kthread_struct+0x70/0x70 [15280.132148] ret_from_fork+0x1f/0x30 [15280.515015] ? show_regs.cold.9+0x22/0x2f [15280.516189] ? watchdog_timer_fn.cold.6+0x62/0x10a [15280.520528] ? __hrtimer_run_queues+0x1c2/0x4d0 [15280.525309] ? watchdog+0x40/0x40 [15280.528729] ? hrtimer_interrupt+0x128/0x2c0 [15280.534844] ? smp_apic_timer_interrupt+0xa6/0x270 [15280.538416] ? apic_timer_interrupt+0xf/0x20 [15280.540011] [15280.541356] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15280.544634] do_raw_spin_lock+0x177/0x190 [15280.546458] _raw_spin_lock+0x21/0x30 [15280.548734] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15280.550319] ? do_raw_spin_unlock+0x75/0x190 [15280.551340] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15280.553079] ? do_wait_intr+0xf0/0xf0 [15280.554491] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15280.556320] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15280.558199] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15280.560078] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15280.561549] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15280.563192] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15280.564529] mdc_intent_lock+0x353/0x7e0 [mdc] [15280.566286] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15280.568117] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15280.569819] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15280.571428] lmv_intent_open+0x338/0x1150 [lmv] [15280.572715] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15280.574251] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15280.575753] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15280.576997] ll_intent_lock+0xc5/0xaa0 [lustre] [15280.578315] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15280.579817] ll_lookup_it+0xe0e/0x2270 [lustre] [15280.581166] ? do_raw_spin_unlock+0x75/0x190 [15280.582301] ? _raw_spin_unlock+0x12/0x30 [15280.583370] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15280.584787] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15280.586265] ll_atomic_open+0x36a/0xd70 [lustre] [15280.588017] lookup_open+0x356/0xb40 [15280.589476] do_last+0x313/0xfc0 [15280.590550] ? nd_jump_root+0xe5/0x160 [15280.592378] ? path_init+0x437/0x520 [15280.593359] path_openat+0xf7/0x500 [15280.594527] ? do_raw_spin_unlock+0x75/0x190 [15280.595755] do_filp_open+0x99/0x140 [15280.596807] ? getname_flags+0x6e/0x330 [15280.597760] ? __check_object_size+0xff/0x256 [15280.598773] ? do_raw_spin_unlock+0x75/0x190 [15280.600030] ? _raw_spin_unlock+0x12/0x30 [15280.601129] do_sys_openat2+0x2b4/0x410 [15280.602270] do_sys_open+0x73/0xa0 [15280.603161] __x64_sys_openat+0x24/0x30 [15280.604221] do_syscall_64+0xc1/0x440 [15280.605384] entry_SYSCALL_64_after_hwframe+0x49/0xae [15280.606743] RIP: 0033:0x7f53425a5332 [15280.607771] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15280.612744] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15280.614645] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15280.616487] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15280.618271] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15280.620104] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15280.621875] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15294.756962] rcu: INFO: rcu_sched self-detected stall on CPU [15294.765962] rcu: 2-....: (7317177 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [15294.782586] (t=13200330 jiffies g=274185 q=67370) [15294.786495] rcu: rcu_sched kthread starved for 12658696 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [15294.803241] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [15294.821338] rcu: RCU grace-period kthread stack dump: [15294.828711] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [15294.838725] Call Trace: [15294.839950] __schedule+0x351/0xcb0 [15294.847084] schedule+0xc0/0x180 [15294.848285] schedule_timeout+0xb4/0x190 [15294.856766] ? do_raw_spin_unlock+0x75/0x190 [15294.858740] ? __next_timer_interrupt+0x160/0x160 [15294.866211] ? prepare_to_swait_event+0xf3/0x140 [15294.867770] rcu_gp_kthread+0x8dd/0x15b0 [15294.873299] ? force_qs_rnp+0x2c0/0x2c0 [15294.875536] kthread+0x1d1/0x200 [15294.879386] ? set_kthread_struct+0x70/0x70 [15294.885982] ret_from_fork+0x1f/0x30 [15294.887347] rcu: Stack dump where RCU GP kthread last ran: [15294.893577] Sending NMI from CPU 2 to CPUs 3: [15294.911653] NMI backtrace for cpu 3 [15294.911662] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15294.911664] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15294.911676] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15294.911678] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15294.911681] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 [15294.911684] RAX: 0000000000007f08 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15294.911685] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15294.911686] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15294.911688] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15294.911689] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15294.911690] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15294.911691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15294.911693] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15294.911694] Call Trace: [15294.911695] [15294.911695] ? show_regs.cold.9+0x22/0x2f [15294.911696] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [15294.911697] ? nmi_cpu_backtrace_handler+0x15/0x40 [15294.911950] ? nmi_handle+0xbd/0x210 [15294.911954] ? default_do_nmi+0x4b/0x1b0 [15294.911955] ? do_nmi+0x280/0x380 [15294.911956] ? end_repeat_nmi+0x16/0x1f [15294.911957] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15294.911958] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15294.911959] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15294.911960] [15294.911961] do_raw_spin_lock+0x177/0x190 [15294.911962] _raw_spin_lock+0x21/0x30 [15294.911963] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15294.911964] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15294.911965] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15294.911966] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15294.911967] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15294.911968] ? woken_wake_function+0x30/0x30 [15294.911969] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15294.911970] kthread+0x1d1/0x200 [15294.911971] ? set_kthread_struct+0x70/0x70 [15294.911972] ret_from_fork+0x1f/0x30 [15294.914535] Sending NMI from CPU 2 to CPUs 1: [15295.097233] NMI backtrace for cpu 1 [15295.097241] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15295.097243] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15295.097245] RIP: 0010:debug_object_deactivate+0xdd/0x200 [15295.097247] Code: 15 c0 78 0c 03 48 8b 3f b8 01 00 00 00 48 83 c2 01 48 89 15 ad 78 0c 03 48 85 ff 74 20 83 c0 01 4c 3b 77 18 0f 84 a5 00 00 00 <48> 8b 3f 48 83 c2 01 48 89 15 8d 78 0c 03 48 85 ff 75 e0 39 05 5a [15295.097250] RSP: 0018:ffffbb5380d24eb0 EFLAGS: 00000006 [15295.097252] RAX: 0000000000000003 RBX: 00000000000487c0 RCX: 0000000000000002 [15295.097254] RDX: 00000000093f25a7 RSI: ffffffff92b096a0 RDI: ffff98a6c2b34f28 [15295.097256] RBP: ffffbb5380d24f08 R08: 0000000000000000 R09: 0000000000000000 [15295.097257] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff94335fa8 [15295.097259] R13: 0000000000000016 R14: ffff98a7020a5540 R15: ffffffff92b096a0 [15295.097261] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15295.097263] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15295.097264] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15295.097265] Call Trace: [15295.097266] [15295.097267] ? show_regs.cold.9+0x22/0x2f [15295.097268] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [15295.097270] ? nmi_cpu_backtrace_handler+0x15/0x40 [15295.097271] ? nmi_handle+0xbd/0x210 [15295.097272] ? default_do_nmi+0x4b/0x1b0 [15295.097273] ? do_nmi+0x280/0x380 [15295.097274] ? end_repeat_nmi+0x16/0x1f [15295.097275] ? debug_object_deactivate+0xdd/0x200 [15295.097276] ? debug_object_deactivate+0xdd/0x200 [15295.097278] ? debug_object_deactivate+0xdd/0x200 [15295.097279] [15295.097279] [15295.097280] __hrtimer_run_queues+0x115/0x4d0 [15295.097281] hrtimer_interrupt+0x128/0x2c0 [15295.097282] smp_apic_timer_interrupt+0xa6/0x270 [15295.097284] apic_timer_interrupt+0xf/0x20 [15295.097285] [15295.097286] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15295.097288] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15295.097290] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15295.097293] RAX: 0000000000005986 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15295.097294] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15295.097295] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15295.097297] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15295.097298] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15295.097305] do_raw_spin_lock+0x177/0x190 [15295.097306] _raw_spin_lock+0x21/0x30 [15295.097307] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15295.097308] ? do_raw_spin_unlock+0x75/0x190 [15295.097310] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15295.097311] ? do_wait_intr+0xf0/0xf0 [15295.097312] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15295.097314] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15295.097315] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15295.097317] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15295.097318] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15295.097320] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15295.097321] mdc_intent_lock+0x353/0x7e0 [mdc] [15295.097322] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15295.098454] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15295.098461] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15295.098462] lmv_intent_open+0x338/0x1150 [lmv] [15295.098463] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15295.098464] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15295.098465] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15295.098466] ll_intent_lock+0xc5/0xaa0 [lustre] [15295.098467] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15295.098468] ll_lookup_it+0xe0e/0x2270 [lustre] [15295.098469] ? do_raw_spin_unlock+0x75/0x190 [15295.098470] ? _raw_spin_unlock+0x12/0x30 [15295.098471] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15295.098473] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15295.098474] ll_atomic_open+0x36a/0xd70 [lustre] [15295.098475] lookup_open+0x356/0xb40 [15295.098475] do_last+0x313/0xfc0 [15295.098476] ? nd_jump_root+0xe5/0x160 [15295.098477] ? path_init+0x437/0x520 [15295.098478] path_openat+0xf7/0x500 [15295.098479] ? do_raw_spin_unlock+0x75/0x190 [15295.098481] do_filp_open+0x99/0x140 [15295.098482] ? getname_flags+0x6e/0x330 [15295.098483] ? __check_object_size+0xff/0x256 [15295.098484] ? do_raw_spin_unlock+0x75/0x190 [15295.098485] ? _raw_spin_unlock+0x12/0x30 [15295.098486] do_sys_openat2+0x2b4/0x410 [15295.098487] do_sys_open+0x73/0xa0 [15295.098488] __x64_sys_openat+0x24/0x30 [15295.098489] do_syscall_64+0xc1/0x440 [15295.098490] entry_SYSCALL_64_after_hwframe+0x49/0xae [15295.098491] RIP: 0033:0x7f53425a5332 [15295.098492] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15295.098495] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15295.098498] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15295.098499] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15295.098500] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15295.098502] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15295.098503] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15295.100307] NMI backtrace for cpu 2 [15295.597363] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15295.607490] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15295.612628] Call Trace: [15295.613569] [15295.614348] ? dump_stack+0xbb/0x10e [15295.622779] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [15295.626814] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [15295.628659] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [15295.630597] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [15295.632930] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [15295.635297] ? rcu_dump_cpu_stacks+0x11b/0x171 [15295.636736] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [15295.638500] ? update_process_times+0x6d/0xc0 [15295.640410] ? tick_sched_handle.isra.18+0x27/0x90 [15295.642910] ? tick_sched_timer+0x53/0xb0 [15295.644118] ? __hrtimer_run_queues+0x1c2/0x4d0 [15295.648077] ? tick_sched_handle.isra.18+0x90/0x90 [15295.650994] ? hrtimer_interrupt+0x128/0x2c0 [15295.652909] ? smp_apic_timer_interrupt+0xa6/0x270 [15295.654965] ? apic_timer_interrupt+0xf/0x20 [15295.656591] [15295.660797] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15295.662660] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15295.665572] ? do_raw_spin_lock+0x177/0x190 [15295.667140] ? _raw_spin_lock+0x21/0x30 [15295.669650] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15295.671569] ? obd_mod_free+0x46/0x140 [ptlrpc] [15295.675844] ? mdc_commit_open+0x91/0xb0 [mdc] [15295.677348] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15295.685061] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15295.687337] ? after_reply+0xe2c/0x1780 [ptlrpc] [15295.689597] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15295.693105] ? __next_timer_interrupt+0x160/0x160 [15295.694857] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [15295.698053] ? do_wait_intr+0xf0/0xf0 [15295.699825] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15295.702304] ? kthread+0x1d1/0x200 [15295.704174] ? set_kthread_struct+0x70/0x70 [15295.705977] ? ret_from_fork+0x1f/0x30 [15304.052206] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [15304.055807] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15304.097399] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15304.116611] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15304.121317] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15304.133385] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15304.142810] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15304.165829] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15304.171916] RAX: 0000000000003b08 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15304.178699] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15304.185419] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15304.192603] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15304.199859] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15304.207237] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15304.215300] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15304.220519] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15304.228201] Call Trace: [15304.229407] [15304.231043] ? show_regs.cold.9+0x22/0x2f [15304.240398] ? watchdog_timer_fn.cold.6+0x62/0x10a [15304.246823] ? __hrtimer_run_queues+0x1c2/0x4d0 [15304.248515] ? watchdog+0x40/0x40 [15304.252284] ? hrtimer_interrupt+0x128/0x2c0 [15304.254681] ? smp_apic_timer_interrupt+0xa6/0x270 [15304.257657] ? apic_timer_interrupt+0xf/0x20 [15304.263376] [15304.266221] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15304.272838] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15304.281223] ? newidle_balance.isra.93+0x207/0x580 [15304.285057] do_raw_spin_lock+0x177/0x190 [15304.287602] _raw_spin_lock+0x21/0x30 [15304.293749] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15304.302789] process_one_work+0x2c8/0x700 [15304.308405] worker_thread+0x296/0x6e0 [15304.313157] ? rescuer_thread+0x570/0x570 [15304.317690] kthread+0x1d1/0x200 [15304.319632] ? set_kthread_struct+0x70/0x70 [15304.325849] ret_from_fork+0x1f/0x30 [15309.282062] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12672s! [15309.289038] Showing busy workqueues and worker pools: [15309.292204] workqueue events: flags=0x0 [15309.294714] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15309.298975] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15309.312913] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15309.316841] pending: cpu_latency_work [ptlrpc] [15309.321904] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15309.326507] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15309.333948] workqueue events_long: flags=0x0 [15309.338827] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15309.345625] pending: lu_objects_flush [obdclass] [15309.351245] workqueue events_unbound: flags=0x2 [15309.355585] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15309.361723] pending: flush_memcg_stats_dwork, idle_cull_fn [15309.367688] workqueue events_power_efficient: flags=0x80 [15309.369551] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15309.380787] pending: check_lifetime, do_cache_clean [15309.388320] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15309.393240] pending: neigh_periodic_work, neigh_periodic_work [15309.397330] workqueue mm_percpu_wq: flags=0x8 [15309.399307] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15309.403136] pending: vmstat_update, lru_add_drain_per_cpu [15309.405959] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15309.409917] pending: vmstat_update, lru_add_drain_per_cpu [15309.412694] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15309.415688] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15309.421330] workqueue xprtiod: flags=0xa [15309.424748] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15309.429562] pending: xprt_autoclose, xs_error_handle [15309.432759] workqueue ipv6_addrconf: flags=0x40008 [15309.435173] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15309.440921] pending: addrconf_verify_work [15309.443742] workqueue knbd0-recv: flags=0x1a [15309.445469] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15309.452511] in-flight: 651:recv_work [15309.456485] workqueue ptlrpc_pinger: flags=0x2 [15309.460048] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15309.462036] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15309.470630] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12672s workers=4 idle: 15979 31824 8 [15309.486743] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15299s workers=2 idle: 654 [15320.055982] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [15320.058258] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [15320.058839] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15320.058845] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15320.058915] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15320.058921] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15320.058924] RIP: 0010:native_safe_halt+0x2e/0x50 [15320.058936] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15320.058941] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15320.058945] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15320.058947] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15320.058950] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15320.058952] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15320.058955] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15320.058957] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15320.058960] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15320.058962] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15320.058971] Call Trace: [15320.058977] [15320.058982] ? show_regs.cold.9+0x22/0x2f [15320.058993] ? watchdog_timer_fn.cold.6+0x62/0x10a [15320.059000] ? __hrtimer_run_queues+0x1c2/0x4d0 [15320.059005] ? watchdog+0x40/0x40 [15320.059009] ? hrtimer_interrupt+0x128/0x2c0 [15320.059014] ? smp_apic_timer_interrupt+0xa6/0x270 [15320.059021] ? apic_timer_interrupt+0xf/0x20 [15320.059026] [15320.059028] ? native_safe_halt+0x2e/0x50 [15320.059033] kvm_wait+0xcd/0xe0 [15320.059040] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15320.059049] do_raw_spin_lock+0x177/0x190 [15320.059054] _raw_spin_lock+0x21/0x30 [15320.059073] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15320.059366] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15320.059736] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15320.059945] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15320.060364] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15320.061950] ? woken_wake_function+0x30/0x30 [15320.061963] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15320.062238] kthread+0x1d1/0x200 [15320.062244] ? set_kthread_struct+0x70/0x70 [15320.062248] ret_from_fork+0x1f/0x30 [15320.065974] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15320.066055] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15320.066062] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15320.066065] RIP: 0010:native_safe_halt+0x2e/0x50 [15320.066078] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15320.066083] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15320.066087] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15320.066090] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15320.066092] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15320.066095] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15320.066097] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15320.066101] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15320.066104] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15320.066106] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15320.066116] Call Trace: [15320.066122] [15320.066127] ? show_regs.cold.9+0x22/0x2f [15320.066138] ? watchdog_timer_fn.cold.6+0x62/0x10a [15320.066144] ? __hrtimer_run_queues+0x1c2/0x4d0 [15320.066148] ? watchdog+0x40/0x40 [15320.066152] ? hrtimer_interrupt+0x128/0x2c0 [15320.066156] ? smp_apic_timer_interrupt+0xa6/0x270 [15320.066163] ? apic_timer_interrupt+0xf/0x20 [15320.066167] [15320.066233] ? native_safe_halt+0x2e/0x50 [15320.066238] kvm_wait+0xcd/0xe0 [15320.066246] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15320.066255] do_raw_spin_lock+0x177/0x190 [15320.066260] _raw_spin_lock+0x21/0x30 [15320.066265] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15320.067380] ? do_raw_spin_unlock+0x75/0x190 [15320.067388] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15320.067751] ? do_wait_intr+0xf0/0xf0 [15320.067761] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15320.067962] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15320.068188] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15320.068400] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15320.068938] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15320.068977] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15320.069006] mdc_intent_lock+0x353/0x7e0 [mdc] [15320.069034] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15320.069131] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15320.069393] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15320.069422] lmv_intent_open+0x338/0x1150 [lmv] [15320.069862] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15320.069955] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15320.070036] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15320.070059] ll_intent_lock+0xc5/0xaa0 [lustre] [15320.070145] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15320.070354] ll_lookup_it+0xe0e/0x2270 [lustre] [15320.070438] ? do_raw_spin_unlock+0x75/0x190 [15320.070722] ? _raw_spin_unlock+0x12/0x30 [15320.070730] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15320.070763] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15320.070783] ll_atomic_open+0x36a/0xd70 [lustre] [15320.070866] lookup_open+0x356/0xb40 [15320.070875] do_last+0x313/0xfc0 [15320.070880] ? nd_jump_root+0xe5/0x160 [15320.070884] ? path_init+0x437/0x520 [15320.070889] path_openat+0xf7/0x500 [15320.070894] ? do_raw_spin_unlock+0x75/0x190 [15320.070898] do_filp_open+0x99/0x140 [15320.070904] ? getname_flags+0x6e/0x330 [15320.070909] ? __check_object_size+0xff/0x256 [15320.070916] ? do_raw_spin_unlock+0x75/0x190 [15320.070920] ? _raw_spin_unlock+0x12/0x30 [15320.070924] do_sys_openat2+0x2b4/0x410 [15320.070927] do_sys_open+0x73/0xa0 [15320.070931] __x64_sys_openat+0x24/0x30 [15320.070934] do_syscall_64+0xc1/0x440 [15320.070940] entry_SYSCALL_64_after_hwframe+0x49/0xae [15320.070946] RIP: 0033:0x7f53425a5332 [15320.070954] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15320.070958] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15320.070963] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15320.070965] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15320.070967] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15320.070969] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15320.070970] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15320.765077] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15320.819100] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15320.837337] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15320.854715] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15320.859540] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15320.885475] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15320.898070] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15320.907229] RDX: 00000000000029b0 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15320.909690] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15320.923094] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15320.926831] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15320.941034] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15320.956423] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15320.966882] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15320.976842] Call Trace: [15320.977719] [15320.990601] ? show_regs.cold.9+0x22/0x2f [15320.993638] ? watchdog_timer_fn.cold.6+0x62/0x10a [15320.998597] ? __hrtimer_run_queues+0x1c2/0x4d0 [15321.000271] ? watchdog+0x40/0x40 [15321.009636] ? hrtimer_interrupt+0x128/0x2c0 [15321.011197] ? smp_apic_timer_interrupt+0xa6/0x270 [15321.018026] ? apic_timer_interrupt+0xf/0x20 [15321.025573] [15321.032047] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15321.041379] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15321.052831] do_raw_spin_lock+0x177/0x190 [15321.054305] _raw_spin_lock+0x21/0x30 [15321.068887] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15321.077770] obd_mod_free+0x46/0x140 [ptlrpc] [15321.094442] mdc_commit_open+0x91/0xb0 [mdc] [15321.101400] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15321.111066] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15321.120859] after_reply+0xe2c/0x1780 [ptlrpc] [15321.128949] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15321.136836] ? __next_timer_interrupt+0x160/0x160 [15321.143945] ptlrpcd+0xa82/0xe40 [ptlrpc] [15321.156845] ? do_wait_intr+0xf0/0xf0 [15321.164820] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15321.170015] kthread+0x1d1/0x200 [15321.180260] ? set_kthread_struct+0x70/0x70 [15321.186726] ret_from_fork+0x1f/0x30 [15336.052095] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [15336.055982] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15336.104548] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15336.116974] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15336.126377] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15336.135379] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15336.142139] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15336.170927] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15336.175367] RAX: 0000000000006484 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15336.182278] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15336.194505] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15336.211224] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15336.219287] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15336.221728] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15336.238465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15336.242749] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15336.248205] Call Trace: [15336.249446] [15336.254453] ? show_regs.cold.9+0x22/0x2f [15336.258453] ? watchdog_timer_fn.cold.6+0x62/0x10a [15336.264200] ? __hrtimer_run_queues+0x1c2/0x4d0 [15336.268688] ? watchdog+0x40/0x40 [15336.271192] ? hrtimer_interrupt+0x128/0x2c0 [15336.274065] ? smp_apic_timer_interrupt+0xa6/0x270 [15336.276332] ? apic_timer_interrupt+0xf/0x20 [15336.286757] [15336.288562] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15336.293869] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15336.298439] ? newidle_balance.isra.93+0x207/0x580 [15336.303186] do_raw_spin_lock+0x177/0x190 [15336.309963] _raw_spin_lock+0x21/0x30 [15336.317143] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15336.321429] process_one_work+0x2c8/0x700 [15336.324444] worker_thread+0x296/0x6e0 [15336.326443] ? rescuer_thread+0x570/0x570 [15336.337166] kthread+0x1d1/0x200 [15336.340361] ? set_kthread_struct+0x70/0x70 [15336.347743] ret_from_fork+0x1f/0x30 [15339.488425] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12702s! [15339.492632] Showing busy workqueues and worker pools: [15339.507125] workqueue events: flags=0x0 [15339.515383] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15339.523624] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15339.528672] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15339.540136] pending: cpu_latency_work [ptlrpc] [15339.547624] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15339.559147] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15339.587102] workqueue events_long: flags=0x0 [15339.596589] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15339.608348] pending: lu_objects_flush [obdclass] [15339.617735] workqueue events_unbound: flags=0x2 [15339.622593] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15339.628010] pending: flush_memcg_stats_dwork, idle_cull_fn [15339.635154] workqueue events_power_efficient: flags=0x80 [15339.646122] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15339.656289] pending: check_lifetime, do_cache_clean [15339.661777] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15339.671227] pending: neigh_periodic_work, neigh_periodic_work [15339.678746] workqueue mm_percpu_wq: flags=0x8 [15339.687521] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15339.704772] pending: vmstat_update, lru_add_drain_per_cpu [15339.718492] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15339.728629] pending: vmstat_update, lru_add_drain_per_cpu [15339.730034] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15339.742217] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15339.751054] workqueue xprtiod: flags=0xa [15339.761240] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15339.771985] pending: xprt_autoclose, xs_error_handle [15339.786382] workqueue ipv6_addrconf: flags=0x40008 [15339.796381] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15339.809789] pending: addrconf_verify_work [15339.819282] workqueue knbd0-recv: flags=0x1a [15339.824896] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15339.839735] in-flight: 651:recv_work [15339.841236] workqueue ptlrpc_pinger: flags=0x2 [15339.849519] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15339.855487] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15339.868321] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12702s workers=4 idle: 15979 31824 8 [15339.875508] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15330s workers=2 idle: 654 [15348.058612] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15348.060917] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [15348.061858] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [15348.061868] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15348.061949] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15348.061955] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15348.061958] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15348.061972] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15348.061976] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15348.061980] RAX: 0000000000000914 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15348.061982] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15348.061984] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15348.061986] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15348.061989] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15348.061991] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15348.061994] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15348.061996] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15348.062005] Call Trace: [15348.062010] [15348.062016] ? show_regs.cold.9+0x22/0x2f [15348.062027] ? watchdog_timer_fn.cold.6+0x62/0x10a [15348.062033] ? __hrtimer_run_queues+0x1c2/0x4d0 [15348.062037] ? watchdog+0x40/0x40 [15348.062041] ? hrtimer_interrupt+0x128/0x2c0 [15348.062045] ? smp_apic_timer_interrupt+0xa6/0x270 [15348.062055] ? apic_timer_interrupt+0xf/0x20 [15348.062060] [15348.062062] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15348.062069] do_raw_spin_lock+0x177/0x190 [15348.062074] _raw_spin_lock+0x21/0x30 [15348.062078] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15348.062374] ? do_raw_spin_unlock+0x75/0x190 [15348.062380] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15348.062832] ? do_wait_intr+0xf0/0xf0 [15348.062842] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15348.063053] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15348.063266] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15348.063773] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15348.063951] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15348.063990] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15348.064020] mdc_intent_lock+0x353/0x7e0 [mdc] [15348.064049] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15348.064133] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15348.065027] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15348.065061] lmv_intent_open+0x338/0x1150 [lmv] [15348.065095] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15348.065193] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15348.066024] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15348.066100] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15348.066107] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15348.066110] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15348.066122] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15348.066126] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15348.066131] RAX: 0000000000005494 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15348.066134] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15348.066136] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15348.066138] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15348.066141] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15348.066144] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15348.066147] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15348.066149] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15348.066158] Call Trace: [15348.066162] [15348.066166] ? show_regs.cold.9+0x22/0x2f [15348.066176] ? watchdog_timer_fn.cold.6+0x62/0x10a [15348.066182] ? __hrtimer_run_queues+0x1c2/0x4d0 [15348.066187] ? watchdog+0x40/0x40 [15348.066192] ? hrtimer_interrupt+0x128/0x2c0 [15348.066197] ? smp_apic_timer_interrupt+0xa6/0x270 [15348.066205] ? apic_timer_interrupt+0xf/0x20 [15348.066211] [15348.066214] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15348.066219] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15348.066225] do_raw_spin_lock+0x177/0x190 [15348.066229] _raw_spin_lock+0x21/0x30 [15348.066234] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15348.066984] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15348.067192] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15348.067835] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15348.068050] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15348.068266] ? woken_wake_function+0x30/0x30 [15348.068290] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15348.068777] kthread+0x1d1/0x200 [15348.068784] ? set_kthread_struct+0x70/0x70 [15348.068788] ret_from_fork+0x1f/0x30 [15348.078556] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15348.078632] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15348.078641] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15348.078644] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15348.078661] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15348.078666] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15348.078670] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15348.078672] RDX: 0000000000006592 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15348.078674] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15348.078676] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15348.078679] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15348.079859] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15348.079862] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15348.079865] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15348.079874] Call Trace: [15348.079880] [15348.079887] ? show_regs.cold.9+0x22/0x2f [15348.079898] ? watchdog_timer_fn.cold.6+0x62/0x10a [15348.079904] ? __hrtimer_run_queues+0x1c2/0x4d0 [15348.079908] ? watchdog+0x40/0x40 [15348.079913] ? hrtimer_interrupt+0x128/0x2c0 [15348.079917] ? smp_apic_timer_interrupt+0xa6/0x270 [15348.079925] ? apic_timer_interrupt+0xf/0x20 [15348.079929] [15348.079932] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15348.079938] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15348.079942] do_raw_spin_lock+0x177/0x190 [15348.079947] _raw_spin_lock+0x21/0x30 [15348.079953] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15348.080802] obd_mod_free+0x46/0x140 [ptlrpc] [15348.081014] mdc_commit_open+0x91/0xb0 [mdc] [15348.081049] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15348.081445] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15348.081643] after_reply+0xe2c/0x1780 [ptlrpc] [15348.081919] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15348.082112] ? __next_timer_interrupt+0x160/0x160 [15348.082126] ptlrpcd+0xa82/0xe40 [ptlrpc] [15348.082332] ? do_wait_intr+0xf0/0xf0 [15348.082341] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15348.082539] kthread+0x1d1/0x200 [15348.082546] ? set_kthread_struct+0x70/0x70 [15348.082550] ret_from_fork+0x1f/0x30 [15349.338360] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15349.344225] ll_intent_lock+0xc5/0xaa0 [lustre] [15349.347616] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15349.352516] ll_lookup_it+0xe0e/0x2270 [lustre] [15349.358254] ? do_raw_spin_unlock+0x75/0x190 [15349.362348] ? _raw_spin_unlock+0x12/0x30 [15349.365632] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15349.368976] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15349.370495] ll_atomic_open+0x36a/0xd70 [lustre] [15349.374649] lookup_open+0x356/0xb40 [15349.376525] do_last+0x313/0xfc0 [15349.377904] ? nd_jump_root+0xe5/0x160 [15349.379617] ? path_init+0x437/0x520 [15349.381603] path_openat+0xf7/0x500 [15349.383224] ? do_raw_spin_unlock+0x75/0x190 [15349.385133] do_filp_open+0x99/0x140 [15349.386381] ? getname_flags+0x6e/0x330 [15349.388502] ? __check_object_size+0xff/0x256 [15349.390120] ? do_raw_spin_unlock+0x75/0x190 [15349.391439] ? _raw_spin_unlock+0x12/0x30 [15349.392941] do_sys_openat2+0x2b4/0x410 [15349.394128] do_sys_open+0x73/0xa0 [15349.395479] __x64_sys_openat+0x24/0x30 [15349.397159] do_syscall_64+0xc1/0x440 [15349.398530] entry_SYSCALL_64_after_hwframe+0x49/0xae [15349.400426] RIP: 0033:0x7f53425a5332 [15349.401969] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15349.409897] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15349.413194] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15349.415551] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15349.418282] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15349.422118] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15349.425751] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15364.052033] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [15364.062755] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15364.142649] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15364.162032] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15364.173705] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15364.184594] RIP: 0010:native_safe_halt+0x2e/0x50 [15364.190072] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15364.213785] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [15364.224987] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15364.236782] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [15364.246672] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15364.256723] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15364.263975] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15364.269430] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15364.281434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15364.290383] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15364.299683] Call Trace: [15364.302650] [15364.304941] ? show_regs.cold.9+0x22/0x2f [15364.307882] ? watchdog_timer_fn.cold.6+0x62/0x10a [15364.315193] ? __hrtimer_run_queues+0x1c2/0x4d0 [15364.317338] ? watchdog+0x40/0x40 [15364.318668] ? hrtimer_interrupt+0x128/0x2c0 [15364.325772] ? smp_apic_timer_interrupt+0xa6/0x270 [15364.330637] ? apic_timer_interrupt+0xf/0x20 [15364.335792] [15364.338094] ? native_safe_halt+0x2e/0x50 [15364.339353] kvm_wait+0xcd/0xe0 [15364.341302] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15364.346705] ? newidle_balance.isra.93+0x207/0x580 [15364.348389] do_raw_spin_lock+0x177/0x190 [15364.358123] _raw_spin_lock+0x21/0x30 [15364.360755] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15364.366158] process_one_work+0x2c8/0x700 [15364.369878] worker_thread+0x296/0x6e0 [15364.375980] ? rescuer_thread+0x570/0x570 [15364.378892] kthread+0x1d1/0x200 [15364.381344] ? set_kthread_struct+0x70/0x70 [15364.385974] ret_from_fork+0x1f/0x30 [15370.208056] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12733s! [15370.215218] Showing busy workqueues and worker pools: [15370.224079] workqueue events: flags=0x0 [15370.225228] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15370.237014] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15370.260009] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15370.271906] pending: cpu_latency_work [ptlrpc] [15370.283501] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15370.294403] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15370.305829] workqueue events_long: flags=0x0 [15370.311713] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15370.322288] pending: lu_objects_flush [obdclass] [15370.331344] workqueue events_unbound: flags=0x2 [15370.336160] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15370.345648] pending: flush_memcg_stats_dwork, idle_cull_fn [15370.352440] workqueue events_power_efficient: flags=0x80 [15370.354192] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15370.375281] pending: check_lifetime, do_cache_clean [15370.380428] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15370.393893] pending: neigh_periodic_work, neigh_periodic_work [15370.404552] workqueue mm_percpu_wq: flags=0x8 [15370.409719] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15370.418923] pending: vmstat_update, lru_add_drain_per_cpu [15370.434281] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15370.439559] pending: vmstat_update, lru_add_drain_per_cpu [15370.444017] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15370.448021] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15370.453977] workqueue xprtiod: flags=0xa [15370.456636] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15370.463395] pending: xprt_autoclose, xs_error_handle [15370.465067] workqueue ipv6_addrconf: flags=0x40008 [15370.477236] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15370.484070] pending: addrconf_verify_work [15370.485384] workqueue knbd0-recv: flags=0x1a [15370.494625] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15370.498749] in-flight: 651:recv_work [15370.504535] workqueue ptlrpc_pinger: flags=0x2 [15370.509455] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15370.516013] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15370.523378] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12733s workers=4 idle: 15979 31824 8 [15370.538747] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15360s workers=2 idle: 654 [15376.055044] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [15376.057148] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [15376.058022] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [15376.058031] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15376.058113] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15376.058119] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15376.058123] Modules linked in: [15376.058123] RIP: 0010:native_safe_halt+0x2e/0x50 [15376.058139] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15376.058143] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [15376.058148] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15376.058150] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15376.058153] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15376.058155] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15376.058157] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15376.058161] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15376.058164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15376.058166] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15376.058176] Call Trace: [15376.058182] [15376.058187] ? show_regs.cold.9+0x22/0x2f [15376.058209] ? watchdog_timer_fn.cold.6+0x62/0x10a [15376.058217] ? __hrtimer_run_queues+0x1c2/0x4d0 [15376.058221] ? watchdog+0x40/0x40 [15376.058225] ? hrtimer_interrupt+0x128/0x2c0 [15376.058229] ? smp_apic_timer_interrupt+0xa6/0x270 [15376.058237] ? apic_timer_interrupt+0xf/0x20 [15376.058242] [15376.058244] ? native_safe_halt+0x2e/0x50 [15376.058249] kvm_wait+0xcd/0xe0 [15376.058256] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15376.058265] do_raw_spin_lock+0x177/0x190 [15376.058270] _raw_spin_lock+0x21/0x30 [15376.058275] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15376.058561] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15376.058782] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15376.059157] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15376.059373] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15376.059579] ? woken_wake_function+0x30/0x30 [15376.059588] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15376.060234] kthread+0x1d1/0x200 [15376.060243] ? set_kthread_struct+0x70/0x70 [15376.060247] ret_from_fork+0x1f/0x30 [15376.061081] Modules linked in: lustre(O) [15376.064871] lustre(O) [15376.103365] osp(O) [15376.108994] osp(O) [15376.113870] ofd(O) [15376.115519] ofd(O) [15376.116797] lod(O) mdt(O) [15376.130506] lod(O) [15376.133381] mdd(O) [15376.136705] mdt(O) [15376.142294] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15376.142359] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15376.142367] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15376.142370] RIP: 0010:native_safe_halt+0x2e/0x50 [15376.142382] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15376.142386] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15376.142390] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15376.142392] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15376.142395] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15376.142397] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15376.142399] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15376.142402] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15376.142405] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15376.142408] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15376.142417] Call Trace: [15376.142423] [15376.142429] ? show_regs.cold.9+0x22/0x2f [15376.142438] ? watchdog_timer_fn.cold.6+0x62/0x10a [15376.142444] ? __hrtimer_run_queues+0x1c2/0x4d0 [15376.142448] ? watchdog+0x40/0x40 [15376.142452] ? hrtimer_interrupt+0x128/0x2c0 [15376.142456] ? smp_apic_timer_interrupt+0xa6/0x270 [15376.142462] ? apic_timer_interrupt+0xf/0x20 [15376.142466] [15376.142468] ? native_safe_halt+0x2e/0x50 [15376.142472] kvm_wait+0xcd/0xe0 [15376.142479] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15376.142485] do_raw_spin_lock+0x177/0x190 [15376.142489] _raw_spin_lock+0x21/0x30 [15376.142494] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15376.142789] obd_mod_free+0x46/0x140 [ptlrpc] [15376.142976] mdc_commit_open+0x91/0xb0 [mdc] [15376.143012] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15376.151113] mdd(O) [15376.159951] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15376.160275] after_reply+0xe2c/0x1780 [ptlrpc] [15376.160470] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15376.160669] ? __next_timer_interrupt+0x160/0x160 [15376.160680] ptlrpcd+0xa82/0xe40 [ptlrpc] [15376.160893] ? do_wait_intr+0xf0/0xf0 [15376.160902] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15376.161115] kthread+0x1d1/0x200 [15376.161121] ? set_kthread_struct+0x70/0x70 [15376.161125] ret_from_fork+0x1f/0x30 [15376.401759] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15376.435756] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15376.441935] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15376.446166] RIP: 0010:native_safe_halt+0x2e/0x50 [15376.449349] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15376.460834] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15376.464350] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15376.467620] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15376.470665] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15376.473037] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15376.476107] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15376.478137] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15376.480548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15376.481881] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15376.483991] Call Trace: [15376.484671] [15376.485429] ? show_regs.cold.9+0x22/0x2f [15376.486481] ? watchdog_timer_fn.cold.6+0x62/0x10a [15376.487657] ? __hrtimer_run_queues+0x1c2/0x4d0 [15376.489459] ? watchdog+0x40/0x40 [15376.490816] ? hrtimer_interrupt+0x128/0x2c0 [15376.492980] ? smp_apic_timer_interrupt+0xa6/0x270 [15376.494946] ? apic_timer_interrupt+0xf/0x20 [15376.497020] [15376.497738] ? native_safe_halt+0x2e/0x50 [15376.500093] kvm_wait+0xcd/0xe0 [15376.501174] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15376.502522] do_raw_spin_lock+0x177/0x190 [15376.504148] _raw_spin_lock+0x21/0x30 [15376.504977] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15376.506453] ? do_raw_spin_unlock+0x75/0x190 [15376.509026] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15376.510495] ? do_wait_intr+0xf0/0xf0 [15376.511359] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15376.513143] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15376.514924] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15376.519291] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15376.521000] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15376.523113] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15376.524856] mdc_intent_lock+0x353/0x7e0 [mdc] [15376.526885] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15376.528257] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15376.531531] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15376.534072] lmv_intent_open+0x338/0x1150 [lmv] [15376.536964] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15376.539680] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15376.541123] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15376.542835] ll_intent_lock+0xc5/0xaa0 [lustre] [15376.544179] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15376.547438] ll_lookup_it+0xe0e/0x2270 [lustre] [15376.549432] ? do_raw_spin_unlock+0x75/0x190 [15376.551086] ? _raw_spin_unlock+0x12/0x30 [15376.552154] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15376.553658] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15376.555393] ll_atomic_open+0x36a/0xd70 [lustre] [15376.556797] lookup_open+0x356/0xb40 [15376.557862] do_last+0x313/0xfc0 [15376.559673] ? nd_jump_root+0xe5/0x160 [15376.561301] ? path_init+0x437/0x520 [15376.562104] path_openat+0xf7/0x500 [15376.563010] ? do_raw_spin_unlock+0x75/0x190 [15376.565095] do_filp_open+0x99/0x140 [15376.566466] ? getname_flags+0x6e/0x330 [15376.568024] ? __check_object_size+0xff/0x256 [15376.570017] ? do_raw_spin_unlock+0x75/0x190 [15376.571666] ? _raw_spin_unlock+0x12/0x30 [15376.573014] do_sys_openat2+0x2b4/0x410 [15376.574798] do_sys_open+0x73/0xa0 [15376.575916] __x64_sys_openat+0x24/0x30 [15376.578493] do_syscall_64+0xc1/0x440 [15376.579690] entry_SYSCALL_64_after_hwframe+0x49/0xae [15376.582525] RIP: 0033:0x7f53425a5332 [15376.584569] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15376.593821] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15376.597519] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15376.599569] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15376.602542] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15376.606601] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15376.608778] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15396.053380] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [15396.058937] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15396.093154] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15396.100130] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15396.105958] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15396.110178] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15396.112589] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15396.119667] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15396.122806] RAX: 00000000000047df RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15396.126781] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15396.135219] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15396.142912] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15396.150682] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15396.159628] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15396.166573] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15396.170309] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15396.173582] Call Trace: [15396.174763] [15396.176764] ? show_regs.cold.9+0x22/0x2f [15396.178321] ? watchdog_timer_fn.cold.6+0x62/0x10a [15396.180705] ? __hrtimer_run_queues+0x1c2/0x4d0 [15396.185423] ? watchdog+0x40/0x40 [15396.187733] ? hrtimer_interrupt+0x128/0x2c0 [15396.190259] ? smp_apic_timer_interrupt+0xa6/0x270 [15396.194012] ? apic_timer_interrupt+0xf/0x20 [15396.195910] [15396.199545] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15396.201774] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15396.205679] ? newidle_balance.isra.93+0x207/0x580 [15396.211216] do_raw_spin_lock+0x177/0x190 [15396.214423] _raw_spin_lock+0x21/0x30 [15396.216640] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15396.219623] process_one_work+0x2c8/0x700 [15396.224088] worker_thread+0x296/0x6e0 [15396.226478] ? rescuer_thread+0x570/0x570 [15396.228812] kthread+0x1d1/0x200 [15396.230489] ? set_kthread_struct+0x70/0x70 [15396.232640] ret_from_fork+0x1f/0x30 [15400.929674] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12763s! [15400.936845] Showing busy workqueues and worker pools: [15400.940499] workqueue events: flags=0x0 [15400.942088] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15400.944910] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15400.950587] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15400.956918] pending: cpu_latency_work [ptlrpc] [15400.960473] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15400.963089] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15400.968682] workqueue events_long: flags=0x0 [15400.973040] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15400.977400] pending: lu_objects_flush [obdclass] [15400.981975] workqueue events_unbound: flags=0x2 [15400.986107] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15400.994133] pending: flush_memcg_stats_dwork, idle_cull_fn [15401.001349] workqueue events_power_efficient: flags=0x80 [15401.006012] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15401.018798] pending: check_lifetime, do_cache_clean [15401.026322] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15401.034704] pending: neigh_periodic_work, neigh_periodic_work [15401.041612] workqueue mm_percpu_wq: flags=0x8 [15401.043253] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15401.050317] pending: vmstat_update, lru_add_drain_per_cpu [15401.057065] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15401.060575] pending: vmstat_update, lru_add_drain_per_cpu [15401.063217] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15401.066354] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15401.072222] workqueue xprtiod: flags=0xa [15401.073973] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15401.078305] pending: xprt_autoclose, xs_error_handle [15401.083700] workqueue ipv6_addrconf: flags=0x40008 [15401.086012] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15401.088610] pending: addrconf_verify_work [15401.093869] workqueue knbd0-recv: flags=0x1a [15401.098280] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15401.101334] in-flight: 651:recv_work [15401.103176] workqueue ptlrpc_pinger: flags=0x2 [15401.106845] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15401.110341] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15401.114994] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12764s workers=4 idle: 15979 31824 8 [15401.119845] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15391s workers=2 idle: 654 [15404.055018] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [15404.057318] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [15404.058329] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15404.058406] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15404.058412] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15404.058415] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15404.058428] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15404.058432] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15404.058436] RAX: 000000000000760a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15404.058438] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15404.058441] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15404.058443] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15404.058445] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15404.058448] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15404.058451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15404.058454] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15404.058463] Call Trace: [15404.058468] [15404.058474] ? show_regs.cold.9+0x22/0x2f [15404.058482] ? watchdog_timer_fn.cold.6+0x62/0x10a [15404.058488] ? __hrtimer_run_queues+0x1c2/0x4d0 [15404.058492] ? watchdog+0x40/0x40 [15404.058496] ? hrtimer_interrupt+0x128/0x2c0 [15404.058500] ? smp_apic_timer_interrupt+0xa6/0x270 [15404.058508] ? apic_timer_interrupt+0xf/0x20 [15404.058513] [15404.058516] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15404.058522] do_raw_spin_lock+0x177/0x190 [15404.058527] _raw_spin_lock+0x21/0x30 [15404.058532] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15404.058816] ? do_raw_spin_unlock+0x75/0x190 [15404.058822] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15404.059015] ? do_wait_intr+0xf0/0xf0 [15404.059023] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [15404.059031] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15404.059088] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15404.059092] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15404.059094] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15404.059102] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15404.059105] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15404.059109] RAX: 0000000000001a02 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15404.059111] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15404.059114] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15404.059116] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15404.059118] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15404.059121] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15404.059024] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15404.059126] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15404.059129] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15404.059136] Call Trace: [15404.059139] [15404.059141] ? show_regs.cold.9+0x22/0x2f [15404.059149] ? watchdog_timer_fn.cold.6+0x62/0x10a [15404.059153] ? __hrtimer_run_queues+0x1c2/0x4d0 [15404.059157] ? watchdog+0x40/0x40 [15404.059162] ? hrtimer_interrupt+0x128/0x2c0 [15404.059166] ? smp_apic_timer_interrupt+0xa6/0x270 [15404.059172] ? apic_timer_interrupt+0xf/0x20 [15404.059176] [15404.059180] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15404.059185] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15404.059190] do_raw_spin_lock+0x177/0x190 [15404.059195] _raw_spin_lock+0x21/0x30 [15404.059291] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15404.059555] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15404.059199] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15404.060052] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15404.060173] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15404.060218] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15404.060250] mdc_intent_lock+0x353/0x7e0 [mdc] [15404.060277] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15404.060361] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15404.060576] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15404.060678] lmv_intent_open+0x338/0x1150 [lmv] [15404.060718] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15404.060804] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15404.060886] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15404.060906] ll_intent_lock+0xc5/0xaa0 [lustre] [15404.061051] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15404.061132] ll_lookup_it+0xe0e/0x2270 [lustre] [15404.061213] ? do_raw_spin_unlock+0x75/0x190 [15404.061219] ? _raw_spin_unlock+0x12/0x30 [15404.061227] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15404.061258] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15404.061276] ll_atomic_open+0x36a/0xd70 [lustre] [15404.061359] lookup_open+0x356/0xb40 [15404.061368] do_last+0x313/0xfc0 [15404.061373] ? nd_jump_root+0xe5/0x160 [15404.061378] ? path_init+0x437/0x520 [15404.061382] path_openat+0xf7/0x500 [15404.061388] ? do_raw_spin_unlock+0x75/0x190 [15404.061393] do_filp_open+0x99/0x140 [15404.061400] ? getname_flags+0x6e/0x330 [15404.061404] ? __check_object_size+0xff/0x256 [15404.061411] ? do_raw_spin_unlock+0x75/0x190 [15404.061414] ? _raw_spin_unlock+0x12/0x30 [15404.061419] do_sys_openat2+0x2b4/0x410 [15404.061423] do_sys_open+0x73/0xa0 [15404.061427] __x64_sys_openat+0x24/0x30 [15404.061431] do_syscall_64+0xc1/0x440 [15404.061438] entry_SYSCALL_64_after_hwframe+0x49/0xae [15404.061444] RIP: 0033:0x7f53425a5332 [15404.061452] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15404.061457] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15404.061462] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15404.061464] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15404.061466] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15404.061469] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15404.061470] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15404.064390] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15404.064468] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15404.064486] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15404.064488] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15404.064501] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15404.064506] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15404.064509] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15404.064512] RDX: 0000000000001484 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15404.064514] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15404.064516] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15404.064518] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15404.064521] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15404.064523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15404.064525] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15404.064534] Call Trace: [15404.064538] [15404.064544] ? show_regs.cold.9+0x22/0x2f [15404.064553] ? watchdog_timer_fn.cold.6+0x62/0x10a [15404.064559] ? __hrtimer_run_queues+0x1c2/0x4d0 [15404.064563] ? watchdog+0x40/0x40 [15404.064567] ? hrtimer_interrupt+0x128/0x2c0 [15404.064571] ? smp_apic_timer_interrupt+0xa6/0x270 [15404.064578] ? apic_timer_interrupt+0xf/0x20 [15404.064583] [15404.064585] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15404.064589] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15404.064852] do_raw_spin_lock+0x177/0x190 [15404.064858] _raw_spin_lock+0x21/0x30 [15404.064864] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15404.065144] obd_mod_free+0x46/0x140 [ptlrpc] [15404.065332] mdc_commit_open+0x91/0xb0 [mdc] [15404.065370] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15404.065571] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15404.066123] after_reply+0xe2c/0x1780 [ptlrpc] [15404.102678] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15404.102977] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15404.103208] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15404.103423] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15404.108582] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15404.108993] ? __next_timer_interrupt+0x160/0x160 [15404.109004] ptlrpcd+0xa82/0xe40 [ptlrpc] [15404.109296] ? do_wait_intr+0xf0/0xf0 [15404.109306] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15404.109528] kthread+0x1d1/0x200 [15404.109534] ? set_kthread_struct+0x70/0x70 [15404.109538] ret_from_fork+0x1f/0x30 [15404.738610] ? woken_wake_function+0x30/0x30 [15404.740916] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15404.744325] kthread+0x1d1/0x200 [15404.746173] ? set_kthread_struct+0x70/0x70 [15404.748529] ret_from_fork+0x1f/0x30 [15428.065005] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [15428.068666] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15428.133616] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15428.142214] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15428.156580] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15428.162160] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15428.170159] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15428.202165] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15428.206574] RAX: 00000000000079bc RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15428.220603] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15428.227286] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15428.240111] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15428.250830] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15428.264171] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15428.275107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15428.280734] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15428.296961] Call Trace: [15428.298963] [15428.304054] ? show_regs.cold.9+0x22/0x2f [15428.310965] ? watchdog_timer_fn.cold.6+0x62/0x10a [15428.312929] ? __hrtimer_run_queues+0x1c2/0x4d0 [15428.314854] ? watchdog+0x40/0x40 [15428.316051] ? hrtimer_interrupt+0x128/0x2c0 [15428.318746] ? smp_apic_timer_interrupt+0xa6/0x270 [15428.321150] ? apic_timer_interrupt+0xf/0x20 [15428.324266] [15428.327518] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15428.331761] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15428.336899] ? newidle_balance.isra.93+0x207/0x580 [15428.339671] do_raw_spin_lock+0x177/0x190 [15428.343889] _raw_spin_lock+0x21/0x30 [15428.346625] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15428.352654] process_one_work+0x2c8/0x700 [15428.356018] worker_thread+0x296/0x6e0 [15428.357130] ? rescuer_thread+0x570/0x570 [15428.359064] kthread+0x1d1/0x200 [15428.363173] ? set_kthread_struct+0x70/0x70 [15428.371053] ret_from_fork+0x1f/0x30 [15431.138281] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12794s! [15431.141262] Showing busy workqueues and worker pools: [15431.143015] workqueue events: flags=0x0 [15431.156861] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15431.174601] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15431.181557] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15431.195743] pending: cpu_latency_work [ptlrpc] [15431.199792] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15431.214331] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15431.217192] workqueue events_long: flags=0x0 [15431.220130] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15431.222366] pending: lu_objects_flush [obdclass] [15431.226197] workqueue events_unbound: flags=0x2 [15431.229517] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15431.234933] pending: flush_memcg_stats_dwork, idle_cull_fn [15431.240119] workqueue events_power_efficient: flags=0x80 [15431.242565] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15431.249075] pending: check_lifetime, do_cache_clean [15431.254204] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15431.257945] pending: neigh_periodic_work, neigh_periodic_work [15431.261022] workqueue mm_percpu_wq: flags=0x8 [15431.267781] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15431.283935] pending: vmstat_update, lru_add_drain_per_cpu [15431.285850] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15431.295898] pending: vmstat_update, lru_add_drain_per_cpu [15431.304296] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15431.306652] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15431.321685] workqueue xprtiod: flags=0xa [15431.328157] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15431.331107] pending: xprt_autoclose, xs_error_handle [15431.334454] workqueue ipv6_addrconf: flags=0x40008 [15431.336359] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15431.342315] pending: addrconf_verify_work [15431.349857] workqueue knbd0-recv: flags=0x1a [15431.354051] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15431.356652] in-flight: 651:recv_work [15431.358492] workqueue ptlrpc_pinger: flags=0x2 [15431.360473] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15431.363094] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15431.365776] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12794s workers=4 idle: 15979 31824 8 [15431.369090] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15421s workers=2 idle: 654 [15432.057081] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [15432.058601] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [15432.058662] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [15432.058669] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15432.058743] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15432.058748] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15432.058751] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15432.058765] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15432.058770] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15432.058775] RAX: 000000000000345c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15432.058777] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15432.058778] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15432.058781] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15432.058783] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15432.058786] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15432.058789] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15432.058791] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15432.058799] Call Trace: [15432.058804] [15432.058810] ? show_regs.cold.9+0x22/0x2f [15432.058819] ? watchdog_timer_fn.cold.6+0x62/0x10a [15432.058824] ? __hrtimer_run_queues+0x1c2/0x4d0 [15432.058828] ? watchdog+0x40/0x40 [15432.058833] ? hrtimer_interrupt+0x128/0x2c0 [15432.058837] ? smp_apic_timer_interrupt+0xa6/0x270 [15432.058845] ? apic_timer_interrupt+0xf/0x20 [15432.058849] [15432.058852] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15432.058859] do_raw_spin_lock+0x177/0x190 [15432.058864] _raw_spin_lock+0x21/0x30 [15432.058869] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15432.061479] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15432.061564] ? do_raw_spin_unlock+0x75/0x190 [15432.061578] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15432.061566] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15432.061737] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15432.061740] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15432.061749] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15432.061753] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15432.061768] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15432.061770] RDX: 00000000000060b8 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15432.061773] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15432.061775] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15432.061777] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15432.061780] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15432.061783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15432.061785] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15432.061794] Call Trace: [15432.061798] [15432.061801] ? show_regs.cold.9+0x22/0x2f [15432.061810] ? watchdog_timer_fn.cold.6+0x62/0x10a [15432.061816] ? __hrtimer_run_queues+0x1c2/0x4d0 [15432.061821] ? watchdog+0x40/0x40 [15432.061826] ? hrtimer_interrupt+0x128/0x2c0 [15432.061829] ? smp_apic_timer_interrupt+0xa6/0x270 [15432.061836] ? apic_timer_interrupt+0xf/0x20 [15432.061841] [15432.061844] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15432.061852] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15432.061857] do_raw_spin_lock+0x177/0x190 [15432.061862] _raw_spin_lock+0x21/0x30 [15432.061873] ? do_wait_intr+0xf0/0xf0 [15432.061885] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15432.061872] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15432.062482] obd_mod_free+0x46/0x140 [ptlrpc] [15432.062706] mdc_commit_open+0x91/0xb0 [mdc] [15432.062755] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15432.065109] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15432.065364] after_reply+0xe2c/0x1780 [ptlrpc] [15432.065563] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15432.066113] ? __next_timer_interrupt+0x160/0x160 [15432.066124] ptlrpcd+0xa82/0xe40 [ptlrpc] [15432.066362] ? do_wait_intr+0xf0/0xf0 [15432.066373] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15432.066661] kthread+0x1d1/0x200 [15432.066666] ? set_kthread_struct+0x70/0x70 [15432.066671] ret_from_fork+0x1f/0x30 [15432.072752] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15432.072840] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15432.072848] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15432.072856] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15432.072870] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15432.072875] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15432.072884] RAX: 000000000000430d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15432.072886] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15432.072888] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15432.072890] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15432.072892] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15432.072896] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15432.072898] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15432.072901] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15432.072911] Call Trace: [15432.072917] [15432.072922] ? show_regs.cold.9+0x22/0x2f [15432.072933] ? watchdog_timer_fn.cold.6+0x62/0x10a [15432.072940] ? __hrtimer_run_queues+0x1c2/0x4d0 [15432.072944] ? watchdog+0x40/0x40 [15432.072949] ? hrtimer_interrupt+0x128/0x2c0 [15432.072953] ? smp_apic_timer_interrupt+0xa6/0x270 [15432.072961] ? apic_timer_interrupt+0xf/0x20 [15432.072966] [15432.072968] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15432.072974] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15432.072980] do_raw_spin_lock+0x177/0x190 [15432.072985] _raw_spin_lock+0x21/0x30 [15432.072990] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15432.073909] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15432.074117] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15432.074696] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15432.075607] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15432.075827] ? woken_wake_function+0x30/0x30 [15432.075838] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15432.076063] kthread+0x1d1/0x200 [15432.076069] ? set_kthread_struct+0x70/0x70 [15432.076073] ret_from_fork+0x1f/0x30 [15432.857687] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15432.864299] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15432.868267] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15432.871783] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15432.877753] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15432.881946] mdc_intent_lock+0x353/0x7e0 [mdc] [15432.885673] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15432.897727] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15432.902091] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15432.906906] lmv_intent_open+0x338/0x1150 [lmv] [15432.910302] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15432.912863] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15432.928880] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15432.932924] ll_intent_lock+0xc5/0xaa0 [lustre] [15432.938543] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15432.944830] ll_lookup_it+0xe0e/0x2270 [lustre] [15432.951539] ? do_raw_spin_unlock+0x75/0x190 [15432.954428] ? _raw_spin_unlock+0x12/0x30 [15432.957549] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15432.959469] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15432.968288] ll_atomic_open+0x36a/0xd70 [lustre] [15432.971675] lookup_open+0x356/0xb40 [15432.974483] do_last+0x313/0xfc0 [15432.977882] ? nd_jump_root+0xe5/0x160 [15432.979647] ? path_init+0x437/0x520 [15432.980974] path_openat+0xf7/0x500 [15432.982126] ? do_raw_spin_unlock+0x75/0x190 [15432.993903] do_filp_open+0x99/0x140 [15432.996999] ? getname_flags+0x6e/0x330 [15433.001691] ? __check_object_size+0xff/0x256 [15433.005908] ? do_raw_spin_unlock+0x75/0x190 [15433.008448] ? _raw_spin_unlock+0x12/0x30 [15433.011570] do_sys_openat2+0x2b4/0x410 [15433.015287] do_sys_open+0x73/0xa0 [15433.019386] __x64_sys_openat+0x24/0x30 [15433.021604] do_syscall_64+0xc1/0x440 [15433.028927] entry_SYSCALL_64_after_hwframe+0x49/0xae [15433.030851] RIP: 0033:0x7f53425a5332 [15433.036595] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15433.050710] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15433.058920] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15433.063399] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15433.070100] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15433.074668] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15433.081787] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15456.052091] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [15456.058069] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15456.088177] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15456.093317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15456.097239] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15456.101138] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15456.104227] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15456.111378] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15456.113638] RAX: 0000000000001590 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15456.116111] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15456.118964] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15456.121665] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15456.124149] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15456.127092] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15456.130575] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15456.133819] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15456.136778] Call Trace: [15456.137670] [15456.143079] ? show_regs.cold.9+0x22/0x2f [15456.144557] ? watchdog_timer_fn.cold.6+0x62/0x10a [15456.148171] ? __hrtimer_run_queues+0x1c2/0x4d0 [15456.150411] ? watchdog+0x40/0x40 [15456.156413] ? hrtimer_interrupt+0x128/0x2c0 [15456.158342] ? smp_apic_timer_interrupt+0xa6/0x270 [15456.163288] ? apic_timer_interrupt+0xf/0x20 [15456.171581] [15456.172596] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15456.182732] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15456.186675] ? newidle_balance.isra.93+0x207/0x580 [15456.196160] do_raw_spin_lock+0x177/0x190 [15456.200939] _raw_spin_lock+0x21/0x30 [15456.206104] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15456.214677] process_one_work+0x2c8/0x700 [15456.217277] worker_thread+0x296/0x6e0 [15456.221271] ? rescuer_thread+0x570/0x570 [15456.223725] kthread+0x1d1/0x200 [15456.229888] ? set_kthread_struct+0x70/0x70 [15456.231198] ret_from_fork+0x1f/0x30 [15460.056782] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [15460.057106] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [15460.059658] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [15460.059676] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15460.059775] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15460.059782] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15460.059789] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15460.059806] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15460.059810] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15460.059863] RAX: 0000000000003908 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15460.059865] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15460.059867] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15460.059869] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15460.059885] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15460.059889] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15460.059892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15460.059894] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15460.059903] Call Trace: [15460.059909] [15460.059915] ? show_regs.cold.9+0x22/0x2f [15460.059930] ? watchdog_timer_fn.cold.6+0x62/0x10a [15460.059944] ? __hrtimer_run_queues+0x1c2/0x4d0 [15460.059950] ? watchdog+0x40/0x40 [15460.059959] ? hrtimer_interrupt+0x128/0x2c0 [15460.059967] ? smp_apic_timer_interrupt+0xa6/0x270 [15460.059979] ? apic_timer_interrupt+0xf/0x20 [15460.059984] [15460.059988] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15460.060000] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15460.060010] do_raw_spin_lock+0x177/0x190 [15460.060016] _raw_spin_lock+0x21/0x30 [15460.060021] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15460.060388] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15460.060638] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15460.060938] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15460.061199] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15460.061481] ? woken_wake_function+0x30/0x30 [15460.061492] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15460.061761] kthread+0x1d1/0x200 [15460.061767] ? set_kthread_struct+0x70/0x70 [15460.061777] ret_from_fork+0x1f/0x30 [15460.063334] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15460.063388] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15460.063393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15460.063395] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15460.063405] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15460.063408] RSP: 0018:ffffbb5381503500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [15460.063411] RAX: 0000000000007f8c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15460.063413] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15460.063415] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15460.063416] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15460.063418] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15460.063420] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15460.063422] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15460.063423] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15460.063430] Call Trace: [15460.063433] [15460.063436] ? show_regs.cold.9+0x22/0x2f [15460.063443] ? watchdog_timer_fn.cold.6+0x62/0x10a [15460.063448] ? __hrtimer_run_queues+0x1c2/0x4d0 [15460.063452] ? watchdog+0x40/0x40 [15460.063457] ? hrtimer_interrupt+0x128/0x2c0 [15460.063461] ? smp_apic_timer_interrupt+0xa6/0x270 [15460.063467] ? apic_timer_interrupt+0xf/0x20 [15460.063471] [15460.063474] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15460.063480] do_raw_spin_lock+0x177/0x190 [15460.063484] _raw_spin_lock+0x21/0x30 [15460.063489] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15460.064185] ? do_raw_spin_unlock+0x75/0x190 [15460.064192] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15460.064373] ? do_wait_intr+0xf0/0xf0 [15460.064380] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15460.064537] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15460.064978] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15460.065847] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15460.065963] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15460.066003] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15460.066036] mdc_intent_lock+0x353/0x7e0 [mdc] [15460.066061] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15460.066804] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15460.078664] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15460.078763] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15460.078771] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15460.078774] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15460.078787] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15460.078791] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15460.078795] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15460.078798] RDX: 0000000000000e8f RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15460.078800] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15460.078802] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15460.078804] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15460.078807] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15460.078810] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15460.078812] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15460.078820] Call Trace: [15460.078825] [15460.078830] ? show_regs.cold.9+0x22/0x2f [15460.078839] ? watchdog_timer_fn.cold.6+0x62/0x10a [15460.078846] ? __hrtimer_run_queues+0x1c2/0x4d0 [15460.078850] ? watchdog+0x40/0x40 [15460.078853] ? hrtimer_interrupt+0x128/0x2c0 [15460.078857] ? smp_apic_timer_interrupt+0xa6/0x270 [15460.078865] ? apic_timer_interrupt+0xf/0x20 [15460.078869] [15460.078871] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15460.078876] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15460.078881] do_raw_spin_lock+0x177/0x190 [15460.078886] _raw_spin_lock+0x21/0x30 [15460.078892] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15460.079159] obd_mod_free+0x46/0x140 [ptlrpc] [15460.079351] mdc_commit_open+0x91/0xb0 [mdc] [15460.079389] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15460.079583] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15460.079808] after_reply+0xe2c/0x1780 [ptlrpc] [15460.080020] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15460.080226] ? __next_timer_interrupt+0x160/0x160 [15460.080235] ptlrpcd+0xa82/0xe40 [ptlrpc] [15460.081249] ? do_wait_intr+0xf0/0xf0 [15460.081260] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15460.081706] kthread+0x1d1/0x200 [15460.081712] ? set_kthread_struct+0x70/0x70 [15460.081716] ret_from_fork+0x1f/0x30 [15460.768898] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15460.771743] lmv_intent_open+0x338/0x1150 [lmv] [15460.775772] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15460.777489] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15460.780720] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15460.782323] ll_intent_lock+0xc5/0xaa0 [lustre] [15460.784764] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15460.786494] ll_lookup_it+0xe0e/0x2270 [lustre] [15460.788165] ? do_raw_spin_unlock+0x75/0x190 [15460.789228] ? _raw_spin_unlock+0x12/0x30 [15460.791260] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15460.794330] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15460.796392] ll_atomic_open+0x36a/0xd70 [lustre] [15460.798127] lookup_open+0x356/0xb40 [15460.799340] do_last+0x313/0xfc0 [15460.800230] ? nd_jump_root+0xe5/0x160 [15460.801434] ? path_init+0x437/0x520 [15460.802820] path_openat+0xf7/0x500 [15460.803586] ? do_raw_spin_unlock+0x75/0x190 [15460.804998] do_filp_open+0x99/0x140 [15460.806093] ? getname_flags+0x6e/0x330 [15460.807565] ? __check_object_size+0xff/0x256 [15460.808933] ? do_raw_spin_unlock+0x75/0x190 [15460.810309] ? _raw_spin_unlock+0x12/0x30 [15460.819245] do_sys_openat2+0x2b4/0x410 [15460.821264] do_sys_open+0x73/0xa0 [15460.822773] __x64_sys_openat+0x24/0x30 [15460.824566] do_syscall_64+0xc1/0x440 [15460.826181] entry_SYSCALL_64_after_hwframe+0x49/0xae [15460.828912] RIP: 0033:0x7f53425a5332 [15460.834281] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15460.858441] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15460.865305] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15460.874047] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15460.881179] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15460.890313] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15460.895931] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15461.860410] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12824s! [15461.864201] Showing busy workqueues and worker pools: [15461.870491] workqueue events: flags=0x0 [15461.872025] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15461.880491] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15461.887105] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15461.892956] pending: cpu_latency_work [ptlrpc] [15461.895973] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15461.899242] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15461.906369] workqueue events_long: flags=0x0 [15461.912973] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15461.918935] pending: lu_objects_flush [obdclass] [15461.921082] workqueue events_unbound: flags=0x2 [15461.923306] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15461.932342] pending: flush_memcg_stats_dwork, idle_cull_fn [15461.934995] workqueue events_power_efficient: flags=0x80 [15461.937799] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15461.940189] pending: check_lifetime, do_cache_clean [15461.944028] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15461.949021] pending: neigh_periodic_work, neigh_periodic_work [15461.953038] workqueue mm_percpu_wq: flags=0x8 [15461.956928] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15461.959491] pending: vmstat_update, lru_add_drain_per_cpu [15461.961821] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15461.965071] pending: vmstat_update, lru_add_drain_per_cpu [15461.967856] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15461.974429] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15461.977743] workqueue xprtiod: flags=0xa [15461.979822] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15461.982152] pending: xprt_autoclose, xs_error_handle [15461.984470] workqueue ipv6_addrconf: flags=0x40008 [15461.986076] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15461.988584] pending: addrconf_verify_work [15461.992268] workqueue knbd0-recv: flags=0x1a [15461.995125] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15461.999276] in-flight: 651:recv_work [15462.003288] workqueue ptlrpc_pinger: flags=0x2 [15462.005204] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15462.011258] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15462.016519] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12824s workers=4 idle: 15979 31824 8 [15462.020013] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15452s workers=2 idle: 654 [15474.759017] rcu: INFO: rcu_sched self-detected stall on CPU [15474.764858] rcu: 2-....: (7410722 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [15474.770664] (t=13380333 jiffies g=274185 q=67416) [15474.773207] rcu: rcu_sched kthread starved for 12838699 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [15474.778903] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [15474.783652] rcu: RCU grace-period kthread stack dump: [15474.787729] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [15474.792932] Call Trace: [15474.794660] __schedule+0x351/0xcb0 [15474.796611] schedule+0xc0/0x180 [15474.797755] schedule_timeout+0xb4/0x190 [15474.800172] ? do_raw_spin_unlock+0x75/0x190 [15474.802768] ? __next_timer_interrupt+0x160/0x160 [15474.805773] ? prepare_to_swait_event+0xf3/0x140 [15474.808398] rcu_gp_kthread+0x8dd/0x15b0 [15474.810870] ? force_qs_rnp+0x2c0/0x2c0 [15474.813174] kthread+0x1d1/0x200 [15474.814455] ? set_kthread_struct+0x70/0x70 [15474.816674] ret_from_fork+0x1f/0x30 [15474.818734] rcu: Stack dump where RCU GP kthread last ran: [15474.822493] Sending NMI from CPU 2 to CPUs 0: [15474.826092] NMI backtrace for cpu 0 [15474.826109] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15474.826112] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15474.826114] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15474.826116] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15474.826118] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15474.826120] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 [15474.826121] RAX: 0000000000007a0f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15474.826123] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15474.826124] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15474.826125] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15474.826126] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15474.826128] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15474.826129] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15474.826130] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15474.826131] Call Trace: [15474.826132] [15474.826133] ? show_regs.cold.9+0x22/0x2f [15474.826134] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [15474.826135] ? nmi_cpu_backtrace_handler+0x15/0x40 [15474.826137] ? nmi_handle+0xbd/0x210 [15474.826138] ? default_do_nmi+0x4b/0x1b0 [15474.826139] ? do_nmi+0x280/0x380 [15474.826140] ? end_repeat_nmi+0x16/0x1f [15474.826141] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15474.826142] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15474.826143] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15474.826145] [15474.826146] ? newidle_balance.isra.93+0x207/0x580 [15474.826147] do_raw_spin_lock+0x177/0x190 [15474.826148] _raw_spin_lock+0x21/0x30 [15474.826149] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15474.826150] process_one_work+0x2c8/0x700 [15474.826151] worker_thread+0x296/0x6e0 [15474.826152] ? rescuer_thread+0x570/0x570 [15474.826153] kthread+0x1d1/0x200 [15474.826154] ? set_kthread_struct+0x70/0x70 [15474.826155] ret_from_fork+0x1f/0x30 [15474.826671] Sending NMI from CPU 2 to CPUs 1: [15474.936440] NMI backtrace for cpu 1 [15474.936448] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15474.936451] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15474.936452] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15474.936457] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15474.936469] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [15474.936472] RAX: 0000000000003ffe RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15474.936474] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15474.936476] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15474.936477] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15474.936478] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15474.936479] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15474.936480] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15474.936482] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15474.936483] Call Trace: [15474.936483] [15474.936484] ? show_regs.cold.9+0x22/0x2f [15474.936485] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [15474.936487] ? nmi_cpu_backtrace_handler+0x15/0x40 [15474.936488] ? nmi_handle+0xbd/0x210 [15474.936489] ? default_do_nmi+0x4b/0x1b0 [15474.936490] ? do_nmi+0x280/0x380 [15474.936491] ? end_repeat_nmi+0x16/0x1f [15474.936492] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15474.936493] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15474.936494] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15474.936495] [15474.936496] do_raw_spin_lock+0x177/0x190 [15474.936498] _raw_spin_lock+0x21/0x30 [15474.936499] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15474.936500] ? do_raw_spin_unlock+0x75/0x190 [15474.936501] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15474.936502] ? do_wait_intr+0xf0/0xf0 [15474.936503] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15474.936504] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15474.936506] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15474.936507] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15474.936508] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15474.936510] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15474.936511] mdc_intent_lock+0x353/0x7e0 [mdc] [15474.936512] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15474.936514] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15474.936515] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15474.936516] lmv_intent_open+0x338/0x1150 [lmv] [15474.936518] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15474.936519] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15474.936520] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15474.936521] ll_intent_lock+0xc5/0xaa0 [lustre] [15474.936523] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15474.936524] ll_lookup_it+0xe0e/0x2270 [lustre] [15474.936525] ? do_raw_spin_unlock+0x75/0x190 [15474.936526] ? _raw_spin_unlock+0x12/0x30 [15474.936527] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15474.936528] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15474.936529] ll_atomic_open+0x36a/0xd70 [lustre] [15474.936530] lookup_open+0x356/0xb40 [15474.936531] do_last+0x313/0xfc0 [15474.936532] ? nd_jump_root+0xe5/0x160 [15474.936533] ? path_init+0x437/0x520 [15474.936534] path_openat+0xf7/0x500 [15474.936535] ? do_raw_spin_unlock+0x75/0x190 [15474.936536] do_filp_open+0x99/0x140 [15474.936537] ? getname_flags+0x6e/0x330 [15474.936538] ? __check_object_size+0xff/0x256 [15474.936539] ? do_raw_spin_unlock+0x75/0x190 [15474.936540] ? _raw_spin_unlock+0x12/0x30 [15474.936541] do_sys_openat2+0x2b4/0x410 [15474.936543] do_sys_open+0x73/0xa0 [15474.936544] __x64_sys_openat+0x24/0x30 [15474.936545] do_syscall_64+0xc1/0x440 [15474.936546] entry_SYSCALL_64_after_hwframe+0x49/0xae [15474.936547] RIP: 0033:0x7f53425a5332 [15474.936548] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15474.936551] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15474.936558] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15474.936559] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15474.936561] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15474.936562] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15474.936564] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15474.937343] NMI backtrace for cpu 2 [15475.115377] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15475.119284] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15475.122814] Call Trace: [15475.123721] [15475.124393] ? dump_stack+0xbb/0x10e [15475.125686] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [15475.127735] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [15475.129528] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [15475.131371] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [15475.133760] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [15475.135905] ? rcu_dump_cpu_stacks+0x11b/0x171 [15475.137813] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [15475.140008] ? update_process_times+0x6d/0xc0 [15475.142674] ? tick_sched_handle.isra.18+0x27/0x90 [15475.144819] ? tick_sched_timer+0x53/0xb0 [15475.146554] ? __hrtimer_run_queues+0x1c2/0x4d0 [15475.148522] ? tick_sched_handle.isra.18+0x90/0x90 [15475.150350] ? hrtimer_interrupt+0x128/0x2c0 [15475.152119] ? smp_apic_timer_interrupt+0xa6/0x270 [15475.153793] ? apic_timer_interrupt+0xf/0x20 [15475.155316] [15475.155871] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15475.158147] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15475.160339] ? do_raw_spin_lock+0x177/0x190 [15475.161820] ? _raw_spin_lock+0x21/0x30 [15475.162576] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15475.164647] ? obd_mod_free+0x46/0x140 [ptlrpc] [15475.166432] ? mdc_commit_open+0x91/0xb0 [mdc] [15475.168189] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15475.170799] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15475.173156] ? after_reply+0xe2c/0x1780 [ptlrpc] [15475.175498] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15475.178090] ? __next_timer_interrupt+0x160/0x160 [15475.179988] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [15475.182149] ? do_wait_intr+0xf0/0xf0 [15475.183789] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15475.186105] ? kthread+0x1d1/0x200 [15475.187449] ? set_kthread_struct+0x70/0x70 [15475.189547] ? ret_from_fork+0x1f/0x30 [15488.058014] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15488.062148] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15488.083121] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15488.089711] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15488.093531] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15488.096176] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15488.104895] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15488.108030] RAX: 0000000000003691 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15488.111472] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15488.114132] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15488.117465] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15488.120222] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15488.123016] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15488.127035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15488.129441] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15488.133097] Call Trace: [15488.135080] [15488.135873] ? show_regs.cold.9+0x22/0x2f [15488.137415] ? watchdog_timer_fn.cold.6+0x62/0x10a [15488.139751] ? __hrtimer_run_queues+0x1c2/0x4d0 [15488.141915] ? watchdog+0x40/0x40 [15488.143214] ? hrtimer_interrupt+0x128/0x2c0 [15488.144932] ? smp_apic_timer_interrupt+0xa6/0x270 [15488.147743] ? apic_timer_interrupt+0xf/0x20 [15488.149316] [15488.150191] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15488.152539] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15488.155219] do_raw_spin_lock+0x177/0x190 [15488.157252] _raw_spin_lock+0x21/0x30 [15488.158741] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15488.162111] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15488.165135] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15488.168054] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15488.171114] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15488.174514] ? woken_wake_function+0x30/0x30 [15488.176201] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15488.180149] kthread+0x1d1/0x200 [15488.181314] ? set_kthread_struct+0x70/0x70 [15488.182797] ret_from_fork+0x1f/0x30 [15492.065041] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12854s! [15492.083114] Showing busy workqueues and worker pools: [15492.088620] workqueue events: flags=0x0 [15492.093314] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15492.098608] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15492.112124] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15492.128002] pending: cpu_latency_work [ptlrpc] [15492.138735] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15492.144179] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15492.151110] workqueue events_long: flags=0x0 [15492.155836] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15492.160702] pending: lu_objects_flush [obdclass] [15492.169044] workqueue events_unbound: flags=0x2 [15492.170972] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15492.176316] pending: flush_memcg_stats_dwork, idle_cull_fn [15492.181885] workqueue events_power_efficient: flags=0x80 [15492.187735] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15492.192148] pending: check_lifetime, do_cache_clean [15492.194565] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15492.199115] pending: neigh_periodic_work, neigh_periodic_work [15492.210785] workqueue mm_percpu_wq: flags=0x8 [15492.212951] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15492.217055] pending: vmstat_update, lru_add_drain_per_cpu [15492.222538] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15492.227639] pending: vmstat_update, lru_add_drain_per_cpu [15492.235352] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15492.245260] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15492.247528] workqueue xprtiod: flags=0xa [15492.250793] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15492.254145] pending: xprt_autoclose, xs_error_handle [15492.265022] workqueue ipv6_addrconf: flags=0x40008 [15492.271424] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15492.283933] pending: addrconf_verify_work [15492.291514] workqueue knbd0-recv: flags=0x1a [15492.299299] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15492.317022] in-flight: 651:recv_work [15492.325761] workqueue ptlrpc_pinger: flags=0x2 [15492.334095] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15492.336344] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15492.345475] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12855s workers=4 idle: 15979 31824 8 [15492.351372] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15482s workers=2 idle: 654 [15500.056936] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [15500.064703] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [15500.065529] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15500.065597] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15500.065604] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15500.065608] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15500.065622] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15500.065626] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15500.065630] RAX: 0000000000000b7e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15500.065633] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15500.065636] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15500.065638] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15500.065640] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15500.065643] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15500.065646] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15500.065649] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15500.065714] Call Trace: [15500.065721] [15500.075489] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15500.076594] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15500.076602] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15500.076605] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15500.076618] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15500.076623] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15500.076628] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15500.076631] RDX: 0000000000001313 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15500.076633] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15500.076635] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15500.076637] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15500.076641] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15500.076644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15500.076646] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15500.076654] Call Trace: [15500.076660] [15500.076666] ? show_regs.cold.9+0x22/0x2f [15500.076676] ? watchdog_timer_fn.cold.6+0x62/0x10a [15500.076688] ? __hrtimer_run_queues+0x1c2/0x4d0 [15500.076693] ? watchdog+0x40/0x40 [15500.076697] ? hrtimer_interrupt+0x128/0x2c0 [15500.076701] ? smp_apic_timer_interrupt+0xa6/0x270 [15500.076717] ? apic_timer_interrupt+0xf/0x20 [15500.076721] [15500.076723] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15500.076729] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15500.076733] do_raw_spin_lock+0x177/0x190 [15500.076738] _raw_spin_lock+0x21/0x30 [15500.076745] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15500.077483] obd_mod_free+0x46/0x140 [ptlrpc] [15500.077719] mdc_commit_open+0x91/0xb0 [mdc] [15500.077763] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15500.078258] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15500.079127] after_reply+0xe2c/0x1780 [ptlrpc] [15500.079382] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15500.080384] ? __next_timer_interrupt+0x160/0x160 [15500.080400] ptlrpcd+0xa82/0xe40 [ptlrpc] [15500.081127] ? do_wait_intr+0xf0/0xf0 [15500.081146] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15500.081928] kthread+0x1d1/0x200 [15500.081935] ? set_kthread_struct+0x70/0x70 [15500.081940] ret_from_fork+0x1f/0x30 [15500.571146] ? show_regs.cold.9+0x22/0x2f [15500.572831] ? watchdog_timer_fn.cold.6+0x62/0x10a [15500.574584] ? __hrtimer_run_queues+0x1c2/0x4d0 [15500.583534] ? watchdog+0x40/0x40 [15500.588338] ? hrtimer_interrupt+0x128/0x2c0 [15500.590636] ? smp_apic_timer_interrupt+0xa6/0x270 [15500.592647] ? apic_timer_interrupt+0xf/0x20 [15500.595208] [15500.596580] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15500.603217] do_raw_spin_lock+0x177/0x190 [15500.605624] _raw_spin_lock+0x21/0x30 [15500.608156] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15500.615955] ? do_raw_spin_unlock+0x75/0x190 [15500.617613] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15500.622888] ? do_wait_intr+0xf0/0xf0 [15500.628282] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15500.630283] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15500.644292] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15500.652214] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15500.657822] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15500.661961] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15500.669571] mdc_intent_lock+0x353/0x7e0 [mdc] [15500.677462] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15500.690789] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15500.700412] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15500.709940] lmv_intent_open+0x338/0x1150 [lmv] [15500.717195] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15500.719652] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15500.722989] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15500.725606] ll_intent_lock+0xc5/0xaa0 [lustre] [15500.730389] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15500.737997] ll_lookup_it+0xe0e/0x2270 [lustre] [15500.741128] ? do_raw_spin_unlock+0x75/0x190 [15500.743343] ? _raw_spin_unlock+0x12/0x30 [15500.746341] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15500.757296] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15500.762138] ll_atomic_open+0x36a/0xd70 [lustre] [15500.777377] lookup_open+0x356/0xb40 [15500.780667] do_last+0x313/0xfc0 [15500.787374] ? nd_jump_root+0xe5/0x160 [15500.789441] ? path_init+0x437/0x520 [15500.795208] path_openat+0xf7/0x500 [15500.808412] ? do_raw_spin_unlock+0x75/0x190 [15500.818634] do_filp_open+0x99/0x140 [15500.820046] ? getname_flags+0x6e/0x330 [15500.821484] ? __check_object_size+0xff/0x256 [15500.826188] ? do_raw_spin_unlock+0x75/0x190 [15500.832320] ? _raw_spin_unlock+0x12/0x30 [15500.835291] do_sys_openat2+0x2b4/0x410 [15500.838266] do_sys_open+0x73/0xa0 [15500.841150] __x64_sys_openat+0x24/0x30 [15500.846326] do_syscall_64+0xc1/0x440 [15500.849131] entry_SYSCALL_64_after_hwframe+0x49/0xae [15500.852300] RIP: 0033:0x7f53425a5332 [15500.854093] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15500.867481] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15500.874346] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15500.882335] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15500.896660] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15500.902758] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15500.911105] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15516.068874] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15516.078710] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15516.121293] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15516.132595] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15516.140592] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15516.145982] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15516.165426] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15516.172950] RAX: 0000000000007a0a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15516.177121] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15516.181357] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15516.185081] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15516.190517] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15516.194138] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15516.199523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15516.204717] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15516.208642] Call Trace: [15516.210159] [15516.211388] ? show_regs.cold.9+0x22/0x2f [15516.213957] ? watchdog_timer_fn.cold.6+0x62/0x10a [15516.218857] ? __hrtimer_run_queues+0x1c2/0x4d0 [15516.221172] ? watchdog+0x40/0x40 [15516.222954] ? hrtimer_interrupt+0x128/0x2c0 [15516.225118] ? smp_apic_timer_interrupt+0xa6/0x270 [15516.227375] ? apic_timer_interrupt+0xf/0x20 [15516.228900] [15516.229483] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15516.231381] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15516.233627] do_raw_spin_lock+0x177/0x190 [15516.235479] _raw_spin_lock+0x21/0x30 [15516.237073] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15516.240291] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15516.242922] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15516.245278] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15516.247691] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15516.249960] ? woken_wake_function+0x30/0x30 [15516.251260] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15516.253274] kthread+0x1d1/0x200 [15516.254730] ? set_kthread_struct+0x70/0x70 [15516.256478] ret_from_fork+0x1f/0x30 [15520.052017] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [15520.059888] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15520.101285] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15520.117783] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15520.126216] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15520.133743] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15520.141283] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15520.158285] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15520.166264] RAX: 0000000000001241 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15520.171653] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15520.177581] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15520.184516] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15520.192491] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15520.194649] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15520.200752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15520.204150] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15520.207765] Call Trace: [15520.210717] [15520.212741] ? show_regs.cold.9+0x22/0x2f [15520.216309] ? watchdog_timer_fn.cold.6+0x62/0x10a [15520.219199] ? __hrtimer_run_queues+0x1c2/0x4d0 [15520.221330] ? watchdog+0x40/0x40 [15520.222840] ? hrtimer_interrupt+0x128/0x2c0 [15520.224781] ? smp_apic_timer_interrupt+0xa6/0x270 [15520.228247] ? apic_timer_interrupt+0xf/0x20 [15520.230142] [15520.231107] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15520.236087] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15520.238812] ? newidle_balance.isra.93+0x207/0x580 [15520.244542] do_raw_spin_lock+0x177/0x190 [15520.247285] _raw_spin_lock+0x21/0x30 [15520.250611] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15520.253872] process_one_work+0x2c8/0x700 [15520.256592] worker_thread+0x296/0x6e0 [15520.258524] ? rescuer_thread+0x570/0x570 [15520.260513] kthread+0x1d1/0x200 [15520.262596] ? set_kthread_struct+0x70/0x70 [15520.265264] ret_from_fork+0x1f/0x30 [15522.783224] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12885s! [15522.787920] Showing busy workqueues and worker pools: [15522.790385] workqueue events: flags=0x0 [15522.792279] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15522.795776] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15522.801310] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15522.805789] pending: cpu_latency_work [ptlrpc] [15522.808594] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15522.812462] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15522.817615] workqueue events_long: flags=0x0 [15522.819276] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15522.822598] pending: lu_objects_flush [obdclass] [15522.825234] workqueue events_unbound: flags=0x2 [15522.827462] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15522.830597] pending: flush_memcg_stats_dwork, idle_cull_fn [15522.833542] workqueue events_power_efficient: flags=0x80 [15522.836447] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15522.839885] pending: check_lifetime, do_cache_clean [15522.842172] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15522.844996] pending: neigh_periodic_work, neigh_periodic_work [15522.847755] workqueue mm_percpu_wq: flags=0x8 [15522.849848] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15522.852882] pending: vmstat_update, lru_add_drain_per_cpu [15522.855708] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15522.858878] pending: vmstat_update, lru_add_drain_per_cpu [15522.861986] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15522.864276] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15522.866964] workqueue xprtiod: flags=0xa [15522.868521] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15522.871216] pending: xprt_autoclose, xs_error_handle [15522.873453] workqueue ipv6_addrconf: flags=0x40008 [15522.875201] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15522.877842] pending: addrconf_verify_work [15522.879727] workqueue knbd0-recv: flags=0x1a [15522.881320] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15522.884477] in-flight: 651:recv_work [15522.886455] workqueue ptlrpc_pinger: flags=0x2 [15522.888479] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15522.891873] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15522.895208] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12885s workers=4 idle: 15979 31824 8 [15522.898865] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15513s workers=2 idle: 654 [15528.055018] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [15528.057763] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [15528.058795] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15528.058867] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15528.058874] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15528.058877] RIP: 0010:native_safe_halt+0x2e/0x50 [15528.058890] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15528.058894] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15528.058898] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15528.058901] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15528.058903] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15528.058905] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15528.058907] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15528.058910] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15528.058913] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15528.058915] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15528.058923] Call Trace: [15528.058928] [15528.058934] ? show_regs.cold.9+0x22/0x2f [15528.058944] ? watchdog_timer_fn.cold.6+0x62/0x10a [15528.059072] ? __hrtimer_run_queues+0x1c2/0x4d0 [15528.059077] ? watchdog+0x40/0x40 [15528.059081] ? hrtimer_interrupt+0x128/0x2c0 [15528.059085] ? smp_apic_timer_interrupt+0xa6/0x270 [15528.059092] ? apic_timer_interrupt+0xf/0x20 [15528.059097] [15528.059100] ? native_safe_halt+0x2e/0x50 [15528.059105] kvm_wait+0xcd/0xe0 [15528.059112] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15528.059121] do_raw_spin_lock+0x177/0x190 [15528.059126] _raw_spin_lock+0x21/0x30 [15528.059130] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15528.059778] ? do_raw_spin_unlock+0x75/0x190 [15528.059785] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15528.060002] ? do_wait_intr+0xf0/0xf0 [15528.060018] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15528.060216] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15528.060474] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15528.064248] Modules linked in: [15528.093827] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15528.101638] lustre(O) [15528.105072] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15528.114138] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15528.114213] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15528.114221] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15528.114224] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15528.114239] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15528.114243] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15528.114248] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15528.114251] RDX: 0000000000003f35 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15528.114253] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15528.114255] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15528.114258] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15528.114260] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15528.114263] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15528.114266] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15528.114275] Call Trace: [15528.114281] [15528.114287] ? show_regs.cold.9+0x22/0x2f [15528.114294] ? watchdog_timer_fn.cold.6+0x62/0x10a [15528.114301] ? __hrtimer_run_queues+0x1c2/0x4d0 [15528.114305] ? watchdog+0x40/0x40 [15528.114310] ? hrtimer_interrupt+0x128/0x2c0 [15528.114314] ? smp_apic_timer_interrupt+0xa6/0x270 [15528.114322] ? apic_timer_interrupt+0xf/0x20 [15528.114327] [15528.114329] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15528.114334] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15528.114339] do_raw_spin_lock+0x177/0x190 [15528.114344] _raw_spin_lock+0x21/0x30 [15528.114350] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15528.114621] obd_mod_free+0x46/0x140 [ptlrpc] [15528.114826] mdc_commit_open+0x91/0xb0 [mdc] [15528.114864] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15528.115246] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15528.115446] after_reply+0xe2c/0x1780 [ptlrpc] [15528.115658] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15528.115859] ? __next_timer_interrupt+0x160/0x160 [15528.116607] ptlrpcd+0xa82/0xe40 [ptlrpc] [15528.124258] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15528.124314] mdc_intent_lock+0x353/0x7e0 [mdc] [15528.124344] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15528.124458] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15528.124806] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15528.124837] lmv_intent_open+0x338/0x1150 [lmv] [15528.124868] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15528.124954] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15528.125600] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15528.125627] ll_intent_lock+0xc5/0xaa0 [lustre] [15528.125714] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15528.125795] ll_lookup_it+0xe0e/0x2270 [lustre] [15528.125880] ? do_raw_spin_unlock+0x75/0x190 [15528.125888] ? _raw_spin_unlock+0x12/0x30 [15528.125895] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15528.125919] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15528.125947] ll_atomic_open+0x36a/0xd70 [lustre] [15528.126254] lookup_open+0x356/0xb40 [15528.126263] do_last+0x313/0xfc0 [15528.126268] ? nd_jump_root+0xe5/0x160 [15528.126273] ? path_init+0x437/0x520 [15528.126277] path_openat+0xf7/0x500 [15528.126282] ? do_raw_spin_unlock+0x75/0x190 [15528.126287] do_filp_open+0x99/0x140 [15528.126293] ? getname_flags+0x6e/0x330 [15528.126297] ? __check_object_size+0xff/0x256 [15528.126304] ? do_raw_spin_unlock+0x75/0x190 [15528.126308] ? _raw_spin_unlock+0x12/0x30 [15528.126312] do_sys_openat2+0x2b4/0x410 [15528.126317] do_sys_open+0x73/0xa0 [15528.126322] __x64_sys_openat+0x24/0x30 [15528.126325] do_syscall_64+0xc1/0x440 [15528.126332] entry_SYSCALL_64_after_hwframe+0x49/0xae [15528.126337] RIP: 0033:0x7f53425a5332 [15528.126344] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15528.126348] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15528.126353] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15528.126356] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15528.126358] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15528.126361] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15528.126364] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15528.479404] ? do_wait_intr+0xf0/0xf0 [15528.487312] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15528.491022] kthread+0x1d1/0x200 [15528.494858] ? set_kthread_struct+0x70/0x70 [15528.498424] ret_from_fork+0x1f/0x30 [15544.058059] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15544.061451] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15544.081844] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15544.086811] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15544.090726] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15544.093711] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15544.101295] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15544.104473] RAX: 0000000000000fa8 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15544.107498] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15544.110560] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15544.113693] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15544.117476] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15544.120707] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15544.124486] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15544.127722] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15544.131458] Call Trace: [15544.132538] [15544.133521] ? show_regs.cold.9+0x22/0x2f [15544.135917] ? watchdog_timer_fn.cold.6+0x62/0x10a [15544.138846] ? __hrtimer_run_queues+0x1c2/0x4d0 [15544.141407] ? watchdog+0x40/0x40 [15544.143059] ? hrtimer_interrupt+0x128/0x2c0 [15544.146041] ? smp_apic_timer_interrupt+0xa6/0x270 [15544.148365] ? apic_timer_interrupt+0xf/0x20 [15544.150631] [15544.151681] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15544.154362] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15544.157460] do_raw_spin_lock+0x177/0x190 [15544.160364] _raw_spin_lock+0x21/0x30 [15544.161699] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15544.165767] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15544.169611] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15544.173509] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15544.176478] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15544.179401] ? woken_wake_function+0x30/0x30 [15544.181562] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15544.184377] kthread+0x1d1/0x200 [15544.185744] ? set_kthread_struct+0x70/0x70 [15544.188022] ret_from_fork+0x1f/0x30 [15548.053109] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [15548.063682] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15548.110756] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15548.123875] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15548.134369] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15548.139171] RIP: 0010:native_safe_halt+0x2e/0x50 [15548.142462] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15548.156123] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15548.160574] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15548.163653] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15548.169878] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15548.175916] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15548.183359] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15548.187967] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15548.194698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15548.202937] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15548.210645] Call Trace: [15548.217024] [15548.218815] ? show_regs.cold.9+0x22/0x2f [15548.232855] ? watchdog_timer_fn.cold.6+0x62/0x10a [15548.236648] ? __hrtimer_run_queues+0x1c2/0x4d0 [15548.242680] ? watchdog+0x40/0x40 [15548.244676] ? hrtimer_interrupt+0x128/0x2c0 [15548.250639] ? smp_apic_timer_interrupt+0xa6/0x270 [15548.252694] ? apic_timer_interrupt+0xf/0x20 [15548.259403] [15548.262601] ? native_safe_halt+0x2e/0x50 [15548.264903] kvm_wait+0xcd/0xe0 [15548.271085] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15548.277834] ? newidle_balance.isra.93+0x207/0x580 [15548.281082] do_raw_spin_lock+0x177/0x190 [15548.284110] _raw_spin_lock+0x21/0x30 [15548.286677] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15548.289089] process_one_work+0x2c8/0x700 [15548.290363] worker_thread+0x296/0x6e0 [15548.291661] ? rescuer_thread+0x570/0x570 [15548.293136] kthread+0x1d1/0x200 [15548.294404] ? set_kthread_struct+0x70/0x70 [15548.295847] ret_from_fork+0x1f/0x30 [15552.994054] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12915s! [15552.997462] Showing busy workqueues and worker pools: [15552.999700] workqueue events: flags=0x0 [15553.001394] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15553.007218] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15553.023777] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15553.028197] pending: cpu_latency_work [ptlrpc] [15553.031046] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15553.035049] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15553.048228] workqueue events_long: flags=0x0 [15553.052206] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15553.058920] pending: lu_objects_flush [obdclass] [15553.064348] workqueue events_unbound: flags=0x2 [15553.068555] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15553.071754] pending: flush_memcg_stats_dwork, idle_cull_fn [15553.080776] workqueue events_power_efficient: flags=0x80 [15553.086345] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15553.093691] pending: check_lifetime, do_cache_clean [15553.096626] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15553.106697] pending: neigh_periodic_work, neigh_periodic_work [15553.111435] workqueue mm_percpu_wq: flags=0x8 [15553.114072] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15553.119851] pending: vmstat_update, lru_add_drain_per_cpu [15553.124536] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15553.131624] pending: vmstat_update, lru_add_drain_per_cpu [15553.134443] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15553.137982] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15553.140151] workqueue xprtiod: flags=0xa [15553.142154] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15553.151880] pending: xprt_autoclose, xs_error_handle [15553.156776] workqueue ipv6_addrconf: flags=0x40008 [15553.160909] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15553.164416] pending: addrconf_verify_work [15553.166637] workqueue knbd0-recv: flags=0x1a [15553.169372] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15553.173340] in-flight: 651:recv_work [15553.175382] workqueue ptlrpc_pinger: flags=0x2 [15553.181900] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15553.186519] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15553.190066] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12916s workers=4 idle: 15979 31824 8 [15553.194352] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15543s workers=2 idle: 654 [15556.055009] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [15556.057014] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [15556.059510] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 [15556.063148] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15556.063299] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15556.063306] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15556.063309] RIP: 0010:native_safe_halt+0x2e/0x50 [15556.063321] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15556.063325] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15556.063329] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15556.063332] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15556.063334] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15556.063336] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15556.063338] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15556.063341] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15556.063343] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15556.063345] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15556.063352] Call Trace: [15556.063366] [15556.063371] ? show_regs.cold.9+0x22/0x2f [15556.063380] ? watchdog_timer_fn.cold.6+0x62/0x10a [15556.063386] ? __hrtimer_run_queues+0x1c2/0x4d0 [15556.063389] ? watchdog+0x40/0x40 [15556.063392] ? hrtimer_interrupt+0x128/0x2c0 [15556.063396] ? smp_apic_timer_interrupt+0xa6/0x270 [15556.063402] ? apic_timer_interrupt+0xf/0x20 [15556.063406] [15556.063408] ? native_safe_halt+0x2e/0x50 [15556.063411] kvm_wait+0xcd/0xe0 [15556.063417] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15556.063424] do_raw_spin_lock+0x177/0x190 [15556.063427] _raw_spin_lock+0x21/0x30 [15556.063432] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15556.063845] obd_mod_free+0x46/0x140 [ptlrpc] [15556.064082] mdc_commit_open+0x91/0xb0 [mdc] [15556.064135] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15556.064351] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15556.081772] dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15556.081814] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15556.081822] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15556.081825] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15556.081839] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15556.081844] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15556.081848] RAX: 0000000000000d23 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15556.081850] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15556.081853] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15556.081855] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15556.081857] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15556.081860] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15556.081863] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15556.081865] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15556.081875] Call Trace: [15556.081880] [15556.081886] ? show_regs.cold.9+0x22/0x2f [15556.081895] ? watchdog_timer_fn.cold.6+0x62/0x10a [15556.081901] ? __hrtimer_run_queues+0x1c2/0x4d0 [15556.081905] ? watchdog+0x40/0x40 [15556.081909] ? hrtimer_interrupt+0x128/0x2c0 [15556.081914] ? smp_apic_timer_interrupt+0xa6/0x270 [15556.081921] ? apic_timer_interrupt+0xf/0x20 [15556.081926] [15556.081929] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15556.081936] do_raw_spin_lock+0x177/0x190 [15556.081941] _raw_spin_lock+0x21/0x30 [15556.081946] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15556.148000] after_reply+0xe2c/0x1780 [ptlrpc] [15556.148454] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15556.150669] ? __next_timer_interrupt+0x160/0x160 [15556.150681] ptlrpcd+0xa82/0xe40 [ptlrpc] [15556.150937] ? do_wait_intr+0xf0/0xf0 [15556.150946] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15556.151421] kthread+0x1d1/0x200 [15556.151432] ? set_kthread_struct+0x70/0x70 [15556.151435] ret_from_fork+0x1f/0x30 [15556.355760] ? do_raw_spin_unlock+0x75/0x190 [15556.357665] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15556.360150] ? do_wait_intr+0xf0/0xf0 [15556.361824] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15556.367010] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15556.369483] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15556.374893] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15556.377164] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15556.382300] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15556.386196] mdc_intent_lock+0x353/0x7e0 [mdc] [15556.388340] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15556.390855] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15556.393537] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15556.396470] lmv_intent_open+0x338/0x1150 [lmv] [15556.398464] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15556.400993] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15556.402915] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15556.404137] ll_intent_lock+0xc5/0xaa0 [lustre] [15556.406280] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15556.409023] ll_lookup_it+0xe0e/0x2270 [lustre] [15556.411200] ? do_raw_spin_unlock+0x75/0x190 [15556.413361] ? _raw_spin_unlock+0x12/0x30 [15556.414945] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15556.418952] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15556.420998] ll_atomic_open+0x36a/0xd70 [lustre] [15556.423519] lookup_open+0x356/0xb40 [15556.425434] do_last+0x313/0xfc0 [15556.426788] ? nd_jump_root+0xe5/0x160 [15556.432066] ? path_init+0x437/0x520 [15556.433978] path_openat+0xf7/0x500 [15556.437930] ? do_raw_spin_unlock+0x75/0x190 [15556.439878] do_filp_open+0x99/0x140 [15556.441381] ? getname_flags+0x6e/0x330 [15556.443158] ? __check_object_size+0xff/0x256 [15556.446496] ? do_raw_spin_unlock+0x75/0x190 [15556.448171] ? _raw_spin_unlock+0x12/0x30 [15556.452611] do_sys_openat2+0x2b4/0x410 [15556.453761] do_sys_open+0x73/0xa0 [15556.456887] __x64_sys_openat+0x24/0x30 [15556.458720] do_syscall_64+0xc1/0x440 [15556.460623] entry_SYSCALL_64_after_hwframe+0x49/0xae [15556.463153] RIP: 0033:0x7f53425a5332 [15556.464980] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15556.473913] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15556.477668] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15556.482151] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15556.484623] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15556.487373] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15556.490671] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15572.058026] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15572.067271] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15572.102652] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15572.111612] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15572.119598] RIP: 0010:native_safe_halt+0x2e/0x50 [15572.123646] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15572.138079] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15572.143099] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15572.148211] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15572.155094] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15572.159261] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15572.164196] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15572.170712] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15572.175580] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15572.181330] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15572.186816] Call Trace: [15572.188161] [15572.189213] ? show_regs.cold.9+0x22/0x2f [15572.192719] ? watchdog_timer_fn.cold.6+0x62/0x10a [15572.194694] ? __hrtimer_run_queues+0x1c2/0x4d0 [15572.198851] ? watchdog+0x40/0x40 [15572.201744] ? hrtimer_interrupt+0x128/0x2c0 [15572.204470] ? smp_apic_timer_interrupt+0xa6/0x270 [15572.208287] ? apic_timer_interrupt+0xf/0x20 [15572.214231] [15572.215457] ? native_safe_halt+0x2e/0x50 [15572.220056] kvm_wait+0xcd/0xe0 [15572.222849] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15572.225620] do_raw_spin_lock+0x177/0x190 [15572.233630] _raw_spin_lock+0x21/0x30 [15572.235633] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15572.239813] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15572.246010] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15572.250625] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15572.253886] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15572.257125] ? woken_wake_function+0x30/0x30 [15572.259776] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15572.263703] kthread+0x1d1/0x200 [15572.267891] ? set_kthread_struct+0x70/0x70 [15572.270228] ret_from_fork+0x1f/0x30 [15580.054458] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [15580.057242] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15580.125985] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15580.139819] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15580.151567] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15580.155369] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15580.163027] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15580.187662] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15580.191565] RAX: 0000000000005f1b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15580.193942] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15580.208891] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15580.217133] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15580.219522] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15580.230893] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15580.239339] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15580.248546] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15580.257208] Call Trace: [15580.258091] [15580.259210] ? show_regs.cold.9+0x22/0x2f [15580.263806] ? watchdog_timer_fn.cold.6+0x62/0x10a [15580.275552] ? __hrtimer_run_queues+0x1c2/0x4d0 [15580.279119] ? watchdog+0x40/0x40 [15580.285881] ? hrtimer_interrupt+0x128/0x2c0 [15580.291428] ? smp_apic_timer_interrupt+0xa6/0x270 [15580.295276] ? apic_timer_interrupt+0xf/0x20 [15580.298050] [15580.298775] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15580.311772] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15580.316016] ? newidle_balance.isra.93+0x207/0x580 [15580.323358] do_raw_spin_lock+0x177/0x190 [15580.325428] _raw_spin_lock+0x21/0x30 [15580.329244] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15580.338282] process_one_work+0x2c8/0x700 [15580.343180] worker_thread+0x296/0x6e0 [15580.344514] ? rescuer_thread+0x570/0x570 [15580.354252] kthread+0x1d1/0x200 [15580.355466] ? set_kthread_struct+0x70/0x70 [15580.364094] ret_from_fork+0x1f/0x30 [15583.202051] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12946s! [15583.208703] Showing busy workqueues and worker pools: [15583.212257] workqueue events: flags=0x0 [15583.216179] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15583.225279] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15583.233213] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15583.237229] pending: cpu_latency_work [ptlrpc] [15583.239929] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15583.244357] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15583.249233] workqueue events_long: flags=0x0 [15583.251985] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15583.256284] pending: lu_objects_flush [obdclass] [15583.259933] workqueue events_unbound: flags=0x2 [15583.263219] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15583.269014] pending: flush_memcg_stats_dwork, idle_cull_fn [15583.273850] workqueue events_power_efficient: flags=0x80 [15583.276285] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15583.281201] pending: check_lifetime, do_cache_clean [15583.283930] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15583.287911] pending: neigh_periodic_work, neigh_periodic_work [15583.293140] workqueue mm_percpu_wq: flags=0x8 [15583.295717] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15583.299836] pending: vmstat_update, lru_add_drain_per_cpu [15583.306212] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15583.310836] pending: vmstat_update, lru_add_drain_per_cpu [15583.314178] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15583.319990] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15583.323344] workqueue xprtiod: flags=0xa [15583.326238] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15583.330021] pending: xprt_autoclose, xs_error_handle [15583.333091] workqueue ipv6_addrconf: flags=0x40008 [15583.335809] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15583.343107] pending: addrconf_verify_work [15583.347891] workqueue knbd0-recv: flags=0x1a [15583.354887] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15583.357940] in-flight: 651:recv_work [15583.361840] workqueue ptlrpc_pinger: flags=0x2 [15583.366344] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15583.370496] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15583.374199] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12946s workers=4 idle: 15979 31824 8 [15583.386896] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15573s workers=2 idle: 654 [15584.055073] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [15584.057029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [15584.058244] Modules linked in: lustre(O) [15584.062666] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15584.062837] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15584.062845] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15584.062848] RIP: 0010:native_safe_halt+0x2e/0x50 [15584.062863] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15584.062867] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15584.062872] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15584.062875] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15584.062877] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15584.062879] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15584.062882] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15584.062885] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15584.062888] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15584.062891] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15584.062900] Call Trace: [15584.062907] [15584.062913] ? show_regs.cold.9+0x22/0x2f [15584.062923] ? watchdog_timer_fn.cold.6+0x62/0x10a [15584.062931] ? __hrtimer_run_queues+0x1c2/0x4d0 [15584.062935] ? watchdog+0x40/0x40 [15584.062939] ? hrtimer_interrupt+0x128/0x2c0 [15584.062943] ? smp_apic_timer_interrupt+0xa6/0x270 [15584.062950] ? apic_timer_interrupt+0xf/0x20 [15584.062955] [15584.063035] ? native_safe_halt+0x2e/0x50 [15584.063042] kvm_wait+0xcd/0xe0 [15584.063050] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15584.063058] do_raw_spin_lock+0x177/0x190 [15584.063063] _raw_spin_lock+0x21/0x30 [15584.063069] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15584.063344] obd_mod_free+0x46/0x140 [ptlrpc] [15584.065299] osp(O) [15584.090133] mdc_commit_open+0x91/0xb0 [mdc] [15584.095120] ofd(O) [15584.100127] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15584.102227] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15584.102298] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15584.102305] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15584.102307] RIP: 0010:native_safe_halt+0x2e/0x50 [15584.102321] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15584.102326] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15584.102330] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15584.102333] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15584.102335] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15584.102337] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15584.102340] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15584.102343] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15584.102347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15584.102349] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15584.102358] Call Trace: [15584.102363] [15584.102368] ? show_regs.cold.9+0x22/0x2f [15584.102377] ? watchdog_timer_fn.cold.6+0x62/0x10a [15584.102383] ? __hrtimer_run_queues+0x1c2/0x4d0 [15584.102387] ? watchdog+0x40/0x40 [15584.102392] ? hrtimer_interrupt+0x128/0x2c0 [15584.102396] ? smp_apic_timer_interrupt+0xa6/0x270 [15584.102402] ? apic_timer_interrupt+0xf/0x20 [15584.102407] [15584.102410] ? native_safe_halt+0x2e/0x50 [15584.102414] kvm_wait+0xcd/0xe0 [15584.102439] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15584.102448] do_raw_spin_lock+0x177/0x190 [15584.102453] _raw_spin_lock+0x21/0x30 [15584.102458] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15584.102741] ? do_raw_spin_unlock+0x75/0x190 [15584.102748] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15584.102943] ? do_wait_intr+0xf0/0xf0 [15584.102952] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15584.103143] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15584.103355] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15584.103598] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15584.103717] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15584.103756] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15584.103786] mdc_intent_lock+0x353/0x7e0 [mdc] [15584.103812] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15584.103897] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15584.104119] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15584.104150] lmv_intent_open+0x338/0x1150 [lmv] [15584.104180] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15584.104263] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15584.104345] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15584.104367] ll_intent_lock+0xc5/0xaa0 [lustre] [15584.104453] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15584.104535] ll_lookup_it+0xe0e/0x2270 [lustre] [15584.125138] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15584.125412] after_reply+0xe2c/0x1780 [ptlrpc] [15584.133347] ? do_raw_spin_unlock+0x75/0x190 [15584.133369] ? _raw_spin_unlock+0x12/0x30 [15584.133378] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15584.133407] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15584.133663] ll_atomic_open+0x36a/0xd70 [lustre] [15584.133800] lookup_open+0x356/0xb40 [15584.133809] do_last+0x313/0xfc0 [15584.133814] ? nd_jump_root+0xe5/0x160 [15584.133819] ? path_init+0x437/0x520 [15584.133823] path_openat+0xf7/0x500 [15584.133828] ? do_raw_spin_unlock+0x75/0x190 [15584.133832] do_filp_open+0x99/0x140 [15584.133838] ? getname_flags+0x6e/0x330 [15584.133842] ? __check_object_size+0xff/0x256 [15584.133849] ? do_raw_spin_unlock+0x75/0x190 [15584.133852] ? _raw_spin_unlock+0x12/0x30 [15584.133857] do_sys_openat2+0x2b4/0x410 [15584.133862] do_sys_open+0x73/0xa0 [15584.133866] __x64_sys_openat+0x24/0x30 [15584.133870] do_syscall_64+0xc1/0x440 [15584.133876] entry_SYSCALL_64_after_hwframe+0x49/0xae [15584.133882] RIP: 0033:0x7f53425a5332 [15584.133935] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15584.133941] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15584.133947] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15584.133949] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15584.133952] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15584.133954] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15584.133957] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15584.508671] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15584.511717] ? __next_timer_interrupt+0x160/0x160 [15584.514058] ptlrpcd+0xa82/0xe40 [ptlrpc] [15584.516521] ? do_wait_intr+0xf0/0xf0 [15584.518265] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15584.521085] kthread+0x1d1/0x200 [15584.522700] ? set_kthread_struct+0x70/0x70 [15584.524534] ret_from_fork+0x1f/0x30 [15600.059143] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15600.066372] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15600.090968] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15600.095411] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15600.098901] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15600.100708] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15600.111134] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15600.114193] RAX: 0000000000005d53 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15600.116689] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15600.119158] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15600.121837] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15600.124396] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15600.127390] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15600.130562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15600.132608] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15600.135224] Call Trace: [15600.136347] [15600.137660] ? show_regs.cold.9+0x22/0x2f [15600.139168] ? watchdog_timer_fn.cold.6+0x62/0x10a [15600.140684] ? __hrtimer_run_queues+0x1c2/0x4d0 [15600.142480] ? watchdog+0x40/0x40 [15600.143611] ? hrtimer_interrupt+0x128/0x2c0 [15600.145095] ? smp_apic_timer_interrupt+0xa6/0x270 [15600.147050] ? apic_timer_interrupt+0xf/0x20 [15600.149023] [15600.150047] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15600.152159] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15600.154526] do_raw_spin_lock+0x177/0x190 [15600.155992] _raw_spin_lock+0x21/0x30 [15600.157242] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15600.160119] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15600.161691] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15600.163886] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15600.166051] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15600.168411] ? woken_wake_function+0x30/0x30 [15600.170363] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15600.172576] kthread+0x1d1/0x200 [15600.173849] ? set_kthread_struct+0x70/0x70 [15600.176090] ret_from_fork+0x1f/0x30 [15608.053050] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [15608.055828] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15608.114491] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15608.120903] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15608.133402] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15608.137712] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15608.141241] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15608.151983] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15608.159312] RAX: 000000000000596e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15608.162797] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15608.166725] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15608.168584] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15608.171366] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15608.183777] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15608.188438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15608.192402] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15608.197759] Call Trace: [15608.198962] [15608.206760] ? show_regs.cold.9+0x22/0x2f [15608.214855] ? watchdog_timer_fn.cold.6+0x62/0x10a [15608.220862] ? __hrtimer_run_queues+0x1c2/0x4d0 [15608.226949] ? watchdog+0x40/0x40 [15608.232835] ? hrtimer_interrupt+0x128/0x2c0 [15608.239422] ? smp_apic_timer_interrupt+0xa6/0x270 [15608.245369] ? apic_timer_interrupt+0xf/0x20 [15608.251725] [15608.252457] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15608.261814] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15608.267339] ? newidle_balance.isra.93+0x207/0x580 [15608.274892] do_raw_spin_lock+0x177/0x190 [15608.279972] _raw_spin_lock+0x21/0x30 [15608.285143] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15608.291447] process_one_work+0x2c8/0x700 [15608.293922] worker_thread+0x296/0x6e0 [15608.295105] ? rescuer_thread+0x570/0x570 [15608.298404] kthread+0x1d1/0x200 [15608.299936] ? set_kthread_struct+0x70/0x70 [15608.303381] ret_from_fork+0x1f/0x30 [15612.056865] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [15612.057349] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [15612.063259] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) [15612.071034] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15612.071146] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15612.071154] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15612.071156] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15612.071172] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15612.071177] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15612.071183] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15612.071186] RDX: 000000000000208e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15612.071189] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15612.071191] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15612.071194] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15612.071197] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15612.071199] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15612.071202] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15612.071210] Call Trace: [15612.071217] [15612.071228] ? show_regs.cold.9+0x22/0x2f [15612.071239] ? watchdog_timer_fn.cold.6+0x62/0x10a [15612.071245] ? __hrtimer_run_queues+0x1c2/0x4d0 [15612.071250] ? watchdog+0x40/0x40 [15612.071254] ? hrtimer_interrupt+0x128/0x2c0 [15612.071258] ? smp_apic_timer_interrupt+0xa6/0x270 [15612.071268] ? apic_timer_interrupt+0xf/0x20 [15612.071273] [15612.071276] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15612.071280] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15612.071285] do_raw_spin_lock+0x177/0x190 [15612.071290] _raw_spin_lock+0x21/0x30 [15612.071296] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15612.071549] obd_mod_free+0x46/0x140 [ptlrpc] [15612.071674] mdc_commit_open+0x91/0xb0 [mdc] [15612.071704] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15612.071827] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15612.071953] after_reply+0xe2c/0x1780 [ptlrpc] [15612.072445] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15612.072639] ? __next_timer_interrupt+0x160/0x160 [15612.072648] ptlrpcd+0xa82/0xe40 [ptlrpc] [15612.072838] ? do_wait_intr+0xf0/0xf0 [15612.072846] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15612.073043] kthread+0x1d1/0x200 [15612.073049] ? set_kthread_struct+0x70/0x70 [15612.073069] ret_from_fork+0x1f/0x30 [15612.274708] lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15612.295132] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15612.304298] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15612.311654] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15612.320683] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15612.332546] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15612.338711] RAX: 0000000000005b0f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15612.344578] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15612.347946] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15612.351884] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15612.357094] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15612.358788] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15612.373191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15612.377229] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15612.381640] Call Trace: [15612.383836] [15612.384474] ? show_regs.cold.9+0x22/0x2f [15612.385905] ? watchdog_timer_fn.cold.6+0x62/0x10a [15612.394944] ? __hrtimer_run_queues+0x1c2/0x4d0 [15612.399740] ? watchdog+0x40/0x40 [15612.400860] ? hrtimer_interrupt+0x128/0x2c0 [15612.409824] ? smp_apic_timer_interrupt+0xa6/0x270 [15612.413530] ? apic_timer_interrupt+0xf/0x20 [15612.416759] [15612.421549] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15612.427028] do_raw_spin_lock+0x177/0x190 [15612.434847] _raw_spin_lock+0x21/0x30 [15612.447806] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15612.450653] ? do_raw_spin_unlock+0x75/0x190 [15612.458965] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15612.469109] ? do_wait_intr+0xf0/0xf0 [15612.472838] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15612.482563] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15612.490972] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15612.504282] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15612.510520] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15612.513775] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15612.515435] mdc_intent_lock+0x353/0x7e0 [mdc] [15612.519157] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15612.524370] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15612.527670] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15612.529903] lmv_intent_open+0x338/0x1150 [lmv] [15612.533540] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15612.536731] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15612.542888] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15612.547287] ll_intent_lock+0xc5/0xaa0 [lustre] [15612.548951] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15612.550627] ll_lookup_it+0xe0e/0x2270 [lustre] [15612.553030] ? do_raw_spin_unlock+0x75/0x190 [15612.555308] ? _raw_spin_unlock+0x12/0x30 [15612.556998] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15612.559566] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15612.564199] ll_atomic_open+0x36a/0xd70 [lustre] [15612.566368] lookup_open+0x356/0xb40 [15612.568628] do_last+0x313/0xfc0 [15612.572986] ? nd_jump_root+0xe5/0x160 [15612.576901] ? path_init+0x437/0x520 [15612.578517] path_openat+0xf7/0x500 [15612.582735] ? do_raw_spin_unlock+0x75/0x190 [15612.585279] do_filp_open+0x99/0x140 [15612.588030] ? getname_flags+0x6e/0x330 [15612.589949] ? __check_object_size+0xff/0x256 [15612.592760] ? do_raw_spin_unlock+0x75/0x190 [15612.594299] ? _raw_spin_unlock+0x12/0x30 [15612.595650] do_sys_openat2+0x2b4/0x410 [15612.601929] do_sys_open+0x73/0xa0 [15612.604642] __x64_sys_openat+0x24/0x30 [15612.606747] do_syscall_64+0xc1/0x440 [15612.610473] entry_SYSCALL_64_after_hwframe+0x49/0xae [15612.613005] RIP: 0033:0x7f53425a5332 [15612.616176] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15612.629227] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15612.634877] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15612.640337] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15612.645551] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15612.653708] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15612.657245] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15613.408925] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 12976s! [15613.413533] Showing busy workqueues and worker pools: [15613.416408] workqueue events: flags=0x0 [15613.421822] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15613.427604] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15613.430703] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15613.436807] pending: cpu_latency_work [ptlrpc] [15613.439299] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15613.442299] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15613.446407] workqueue events_long: flags=0x0 [15613.457398] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15613.459514] pending: lu_objects_flush [obdclass] [15613.464534] workqueue events_unbound: flags=0x2 [15613.467629] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15613.471654] pending: flush_memcg_stats_dwork, idle_cull_fn [15613.475581] workqueue events_power_efficient: flags=0x80 [15613.478368] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15613.483397] pending: check_lifetime, do_cache_clean [15613.488357] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15613.494341] pending: neigh_periodic_work, neigh_periodic_work [15613.500718] workqueue mm_percpu_wq: flags=0x8 [15613.502496] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15613.507669] pending: vmstat_update, lru_add_drain_per_cpu [15613.509669] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15613.515390] pending: vmstat_update, lru_add_drain_per_cpu [15613.517226] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15613.522256] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15613.524726] workqueue xprtiod: flags=0xa [15613.526380] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15613.533775] pending: xprt_autoclose, xs_error_handle [15613.537886] workqueue ipv6_addrconf: flags=0x40008 [15613.540871] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15613.544988] pending: addrconf_verify_work [15613.546518] workqueue knbd0-recv: flags=0x1a [15613.552245] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15613.553957] in-flight: 651:recv_work [15613.555011] workqueue ptlrpc_pinger: flags=0x2 [15613.562197] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15613.564772] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15613.567561] pool 8: cpus=0-3 flags=0x4 nice=0 hung=12976s workers=4 idle: 15979 31824 8 [15613.570573] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15604s workers=2 idle: 654 [15628.059945] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15628.067708] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15628.092395] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15628.097124] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15628.102439] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15628.111350] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15628.129308] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15628.134566] RAX: 0000000000007f8e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15628.143321] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15628.147914] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15628.152170] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15628.155933] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15628.161002] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15628.164518] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15628.166970] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15628.170609] Call Trace: [15628.171666] [15628.173624] ? show_regs.cold.9+0x22/0x2f [15628.176122] ? watchdog_timer_fn.cold.6+0x62/0x10a [15628.180182] ? __hrtimer_run_queues+0x1c2/0x4d0 [15628.181939] ? watchdog+0x40/0x40 [15628.183018] ? hrtimer_interrupt+0x128/0x2c0 [15628.184431] ? smp_apic_timer_interrupt+0xa6/0x270 [15628.190486] ? apic_timer_interrupt+0xf/0x20 [15628.194115] [15628.195955] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15628.197720] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15628.203724] do_raw_spin_lock+0x177/0x190 [15628.206184] _raw_spin_lock+0x21/0x30 [15628.211607] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15628.215597] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15628.218532] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15628.227386] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15628.234273] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15628.236375] ? woken_wake_function+0x30/0x30 [15628.239582] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15628.247762] kthread+0x1d1/0x200 [15628.250346] ? set_kthread_struct+0x70/0x70 [15628.254076] ret_from_fork+0x1f/0x30 [15640.052186] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [15640.055586] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [15640.057357] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [15640.057368] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15640.057450] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15640.057455] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15640.057458] RIP: 0010:native_safe_halt+0x2e/0x50 [15640.057470] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15640.057474] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [15640.057480] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15640.057482] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15640.057485] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15640.057487] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15640.057490] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15640.057493] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15640.057496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15640.057498] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15640.057507] Call Trace: [15640.057511] [15640.057517] ? show_regs.cold.9+0x22/0x2f [15640.057526] ? watchdog_timer_fn.cold.6+0x62/0x10a [15640.057533] ? __hrtimer_run_queues+0x1c2/0x4d0 [15640.057537] ? watchdog+0x40/0x40 [15640.057542] ? hrtimer_interrupt+0x128/0x2c0 [15640.057547] ? smp_apic_timer_interrupt+0xa6/0x270 [15640.057562] ? apic_timer_interrupt+0xf/0x20 [15640.057575] [15640.057577] ? native_safe_halt+0x2e/0x50 [15640.057581] kvm_wait+0xcd/0xe0 [15640.057588] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15640.057594] do_raw_spin_lock+0x177/0x190 [15640.057598] _raw_spin_lock+0x21/0x30 [15640.057780] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15640.057842] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15640.057847] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15640.057856] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15640.057603] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15640.058524] obd_mod_free+0x46/0x140 [ptlrpc] [15640.059504] mdc_commit_open+0x91/0xb0 [mdc] [15640.059580] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15640.060145] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15640.060935] after_reply+0xe2c/0x1780 [ptlrpc] [15640.061215] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15640.061456] ? __next_timer_interrupt+0x160/0x160 [15640.061468] ptlrpcd+0xa82/0xe40 [ptlrpc] [15640.061868] ? do_wait_intr+0xf0/0xf0 [15640.061877] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15640.062093] kthread+0x1d1/0x200 [15640.062099] ? set_kthread_struct+0x70/0x70 [15640.062103] ret_from_fork+0x1f/0x30 [15640.063304] Modules linked in: lustre(O) [15640.066508] [15640.126649] osp(O) [15640.135186] RIP: 0010:native_safe_halt+0x2e/0x50 [15640.135208] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15640.135213] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15640.135218] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15640.135221] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15640.135223] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15640.135225] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15640.135227] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15640.135230] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15640.135233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15640.135235] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15640.135243] Call Trace: [15640.135249] [15640.135254] ? show_regs.cold.9+0x22/0x2f [15640.135265] ? watchdog_timer_fn.cold.6+0x62/0x10a [15640.135271] ? __hrtimer_run_queues+0x1c2/0x4d0 [15640.135275] ? watchdog+0x40/0x40 [15640.135279] ? hrtimer_interrupt+0x128/0x2c0 [15640.135283] ? smp_apic_timer_interrupt+0xa6/0x270 [15640.135290] ? apic_timer_interrupt+0xf/0x20 [15640.135295] [15640.135298] ? native_safe_halt+0x2e/0x50 [15640.135301] kvm_wait+0xcd/0xe0 [15640.135308] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15640.135316] ? newidle_balance.isra.93+0x207/0x580 [15640.135324] do_raw_spin_lock+0x177/0x190 [15640.135328] _raw_spin_lock+0x21/0x30 [15640.135333] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15640.135621] process_one_work+0x2c8/0x700 [15640.135628] worker_thread+0x296/0x6e0 [15640.135633] ? rescuer_thread+0x570/0x570 [15640.135637] kthread+0x1d1/0x200 [15640.135642] ? set_kthread_struct+0x70/0x70 [15640.135646] ret_from_fork+0x1f/0x30 [15640.578899] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15640.624898] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15640.632104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15640.643021] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15640.647524] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15640.669255] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15640.671456] RAX: 0000000000007e8d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15640.685488] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15640.690948] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15640.700477] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15640.706321] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15640.711981] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15640.721894] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15640.730889] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15640.741037] Call Trace: [15640.747741] [15640.748427] ? show_regs.cold.9+0x22/0x2f [15640.754974] ? watchdog_timer_fn.cold.6+0x62/0x10a [15640.764035] ? __hrtimer_run_queues+0x1c2/0x4d0 [15640.771518] ? watchdog+0x40/0x40 [15640.778540] ? hrtimer_interrupt+0x128/0x2c0 [15640.786280] ? smp_apic_timer_interrupt+0xa6/0x270 [15640.791205] ? apic_timer_interrupt+0xf/0x20 [15640.795277] [15640.796540] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15640.804676] do_raw_spin_lock+0x177/0x190 [15640.807511] _raw_spin_lock+0x21/0x30 [15640.814877] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15640.820894] ? do_raw_spin_unlock+0x75/0x190 [15640.827353] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15640.834316] ? do_wait_intr+0xf0/0xf0 [15640.837035] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15640.840442] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15640.849076] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15640.858126] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15640.859861] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15640.866513] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15640.872016] mdc_intent_lock+0x353/0x7e0 [mdc] [15640.876003] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15640.883813] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15640.885883] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15640.898288] lmv_intent_open+0x338/0x1150 [lmv] [15640.902362] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15640.906065] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15640.913299] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15640.914668] ll_intent_lock+0xc5/0xaa0 [lustre] [15640.921491] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15640.926734] ll_lookup_it+0xe0e/0x2270 [lustre] [15640.931475] ? do_raw_spin_unlock+0x75/0x190 [15640.932772] ? _raw_spin_unlock+0x12/0x30 [15640.942688] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15640.946888] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15640.953927] ll_atomic_open+0x36a/0xd70 [lustre] [15640.955984] lookup_open+0x356/0xb40 [15640.958912] do_last+0x313/0xfc0 [15640.960099] ? nd_jump_root+0xe5/0x160 [15640.965010] ? path_init+0x437/0x520 [15640.970890] path_openat+0xf7/0x500 [15640.972075] ? do_raw_spin_unlock+0x75/0x190 [15640.973438] do_filp_open+0x99/0x140 [15640.979965] ? getname_flags+0x6e/0x330 [15640.985255] ? __check_object_size+0xff/0x256 [15640.989903] ? do_raw_spin_unlock+0x75/0x190 [15640.993395] ? _raw_spin_unlock+0x12/0x30 [15641.003673] do_sys_openat2+0x2b4/0x410 [15641.009286] do_sys_open+0x73/0xa0 [15641.010957] __x64_sys_openat+0x24/0x30 [15641.014094] do_syscall_64+0xc1/0x440 [15641.018850] entry_SYSCALL_64_after_hwframe+0x49/0xae [15641.020481] RIP: 0033:0x7f53425a5332 [15641.028413] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15641.048434] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15641.055044] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15641.062937] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15641.068799] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15641.079017] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15641.088214] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15643.618090] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13006s! [15643.624400] Showing busy workqueues and worker pools: [15643.626486] workqueue events: flags=0x0 [15643.628845] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15643.637534] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15643.647441] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15643.651127] pending: cpu_latency_work [ptlrpc] [15643.653834] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15643.658625] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15643.666825] workqueue events_long: flags=0x0 [15643.668317] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15643.670930] pending: lu_objects_flush [obdclass] [15643.674161] workqueue events_unbound: flags=0x2 [15643.679280] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15643.682417] pending: flush_memcg_stats_dwork, idle_cull_fn [15643.685599] workqueue events_power_efficient: flags=0x80 [15643.688456] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15643.691368] pending: check_lifetime, do_cache_clean [15643.694729] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15643.696687] pending: neigh_periodic_work, neigh_periodic_work [15643.704353] workqueue mm_percpu_wq: flags=0x8 [15643.706633] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15643.710378] pending: vmstat_update, lru_add_drain_per_cpu [15643.718237] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15643.731147] pending: vmstat_update, lru_add_drain_per_cpu [15643.736075] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15643.739105] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15643.742599] workqueue xprtiod: flags=0xa [15643.744407] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15643.749606] pending: xprt_autoclose, xs_error_handle [15643.753823] workqueue ipv6_addrconf: flags=0x40008 [15643.756934] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15643.759399] pending: addrconf_verify_work [15643.765513] workqueue knbd0-recv: flags=0x1a [15643.767283] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15643.772231] in-flight: 651:recv_work [15643.774286] workqueue ptlrpc_pinger: flags=0x2 [15643.776606] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15643.779906] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15643.787966] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13006s workers=4 idle: 15979 31824 8 [15643.790216] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15634s workers=2 idle: 654 [15654.764792] rcu: INFO: rcu_sched self-detected stall on CPU [15654.767953] rcu: 2-....: (7522393 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [15654.781076] (t=13560338 jiffies g=274185 q=67440) [15654.784967] rcu: rcu_sched kthread starved for 13018704 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [15654.797643] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [15654.809562] rcu: RCU grace-period kthread stack dump: [15654.814632] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [15654.819610] Call Trace: [15654.822763] __schedule+0x351/0xcb0 [15654.824803] schedule+0xc0/0x180 [15654.827944] schedule_timeout+0xb4/0x190 [15654.830130] ? do_raw_spin_unlock+0x75/0x190 [15654.832053] ? __next_timer_interrupt+0x160/0x160 [15654.834629] ? prepare_to_swait_event+0xf3/0x140 [15654.838286] rcu_gp_kthread+0x8dd/0x15b0 [15654.841924] ? force_qs_rnp+0x2c0/0x2c0 [15654.852752] kthread+0x1d1/0x200 [15654.854078] ? set_kthread_struct+0x70/0x70 [15654.859665] ret_from_fork+0x1f/0x30 [15654.861506] rcu: Stack dump where RCU GP kthread last ran: [15654.867647] Sending NMI from CPU 2 to CPUs 1: [15654.871674] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [15654.875923] Sending NMI from CPU 2 to CPUs 1: [15654.886405] NMI backtrace for cpu 1 [15654.886411] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15654.886413] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15654.886414] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15654.886416] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15654.886418] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [15654.886421] RAX: 0000000000005b10 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15654.886422] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15654.886423] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15654.886425] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15654.886426] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15654.886427] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15654.886429] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15654.886430] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15654.886431] Call Trace: [15654.886433] [15654.886433] ? show_regs.cold.9+0x22/0x2f [15654.886434] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [15654.886435] ? nmi_cpu_backtrace_handler+0x15/0x40 [15654.886437] ? nmi_handle+0xbd/0x210 [15654.886438] ? default_do_nmi+0x4b/0x1b0 [15654.886439] ? do_nmi+0x280/0x380 [15654.886440] ? end_repeat_nmi+0x16/0x1f [15654.886440] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15654.886442] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15654.886443] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15654.886444] [15654.886444] do_raw_spin_lock+0x177/0x190 [15654.886445] _raw_spin_lock+0x21/0x30 [15654.886446] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15654.886447] ? do_raw_spin_unlock+0x75/0x190 [15654.886448] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15654.886449] ? do_wait_intr+0xf0/0xf0 [15654.886450] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15654.886451] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15654.886453] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15654.886454] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15654.886455] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15654.886456] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15654.886458] mdc_intent_lock+0x353/0x7e0 [mdc] [15654.886459] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15654.886460] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15654.886461] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15654.886463] lmv_intent_open+0x338/0x1150 [lmv] [15654.886464] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15654.886465] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15654.886466] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15654.886468] ll_intent_lock+0xc5/0xaa0 [lustre] [15654.886469] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15654.886470] ll_lookup_it+0xe0e/0x2270 [lustre] [15654.886471] ? do_raw_spin_unlock+0x75/0x190 [15654.886472] ? _raw_spin_unlock+0x12/0x30 [15654.886473] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15654.886475] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15654.886476] ll_atomic_open+0x36a/0xd70 [lustre] [15654.886477] lookup_open+0x356/0xb40 [15654.886478] do_last+0x313/0xfc0 [15654.886479] ? nd_jump_root+0xe5/0x160 [15654.886480] ? path_init+0x437/0x520 [15654.886481] path_openat+0xf7/0x500 [15654.886482] ? do_raw_spin_unlock+0x75/0x190 [15654.886483] do_filp_open+0x99/0x140 [15654.886484] ? getname_flags+0x6e/0x330 [15654.886485] ? __check_object_size+0xff/0x256 [15654.886487] ? do_raw_spin_unlock+0x75/0x190 [15654.886488] ? _raw_spin_unlock+0x12/0x30 [15654.886489] do_sys_openat2+0x2b4/0x410 [15654.886490] do_sys_open+0x73/0xa0 [15654.886491] __x64_sys_openat+0x24/0x30 [15654.886492] do_syscall_64+0xc1/0x440 [15654.886494] entry_SYSCALL_64_after_hwframe+0x49/0xae [15654.886495] RIP: 0033:0x7f53425a5332 [15654.886496] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15654.886498] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15654.886501] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15654.886502] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15654.886504] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15654.886505] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15654.886506] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15654.886913] NMI backtrace for cpu 2 [15655.234803] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15655.242480] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15655.251634] Call Trace: [15655.255279] [15655.256518] ? dump_stack+0xbb/0x10e [15655.260632] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [15655.264284] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [15655.268488] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [15655.273446] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [15655.277490] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [15655.282233] ? rcu_dump_cpu_stacks+0x11b/0x171 [15655.289754] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [15655.297897] ? update_process_times+0x6d/0xc0 [15655.301272] ? tick_sched_handle.isra.18+0x27/0x90 [15655.307029] ? tick_sched_timer+0x53/0xb0 [15655.309382] ? __hrtimer_run_queues+0x1c2/0x4d0 [15655.315777] ? tick_sched_handle.isra.18+0x90/0x90 [15655.323644] ? hrtimer_interrupt+0x128/0x2c0 [15655.326341] ? smp_apic_timer_interrupt+0xa6/0x270 [15655.335639] ? apic_timer_interrupt+0xf/0x20 [15655.338886] [15655.347668] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15655.352693] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15655.362774] ? do_raw_spin_lock+0x177/0x190 [15655.366033] ? _raw_spin_lock+0x21/0x30 [15655.369417] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15655.375368] ? obd_mod_free+0x46/0x140 [ptlrpc] [15655.382438] ? mdc_commit_open+0x91/0xb0 [mdc] [15655.385203] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15655.392895] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15655.397196] ? after_reply+0xe2c/0x1780 [ptlrpc] [15655.407973] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15655.422781] ? __next_timer_interrupt+0x160/0x160 [15655.426357] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [15655.433825] ? do_wait_intr+0xf0/0xf0 [15655.437202] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15655.440727] ? kthread+0x1d1/0x200 [15655.448027] ? set_kthread_struct+0x70/0x70 [15655.449629] ? ret_from_fork+0x1f/0x30 [15656.058018] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15656.062094] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15656.095217] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15656.100353] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15656.104481] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15656.107212] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15656.118722] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [15656.125213] RAX: 00000000000078b5 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15656.133238] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15656.140111] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15656.153840] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15656.159984] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15656.171017] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15656.177747] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15656.183757] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15656.189222] Call Trace: [15656.192674] [15656.194004] ? show_regs.cold.9+0x22/0x2f [15656.197945] ? watchdog_timer_fn.cold.6+0x62/0x10a [15656.205207] ? __hrtimer_run_queues+0x1c2/0x4d0 [15656.207508] ? watchdog+0x40/0x40 [15656.211658] ? hrtimer_interrupt+0x128/0x2c0 [15656.214612] ? smp_apic_timer_interrupt+0xa6/0x270 [15656.220537] ? apic_timer_interrupt+0xf/0x20 [15656.226518] [15656.227212] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15656.231893] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15656.238846] do_raw_spin_lock+0x177/0x190 [15656.244935] _raw_spin_lock+0x21/0x30 [15656.248261] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15656.256685] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15656.263503] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15656.271028] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15656.284525] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15656.290731] ? woken_wake_function+0x30/0x30 [15656.294611] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15656.305138] kthread+0x1d1/0x200 [15656.306117] ? set_kthread_struct+0x70/0x70 [15656.316597] ret_from_fork+0x1f/0x30 [15668.058181] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [15668.066021] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15668.105141] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15668.113954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15668.122226] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15668.133951] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15668.140899] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15668.160853] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15668.170906] RAX: 0000000000006b90 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15668.176912] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15668.188286] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15668.197440] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15668.204527] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15668.213167] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15668.219907] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15668.222992] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15668.226955] Call Trace: [15668.229014] [15668.230285] ? show_regs.cold.9+0x22/0x2f [15668.232777] ? watchdog_timer_fn.cold.6+0x62/0x10a [15668.237982] ? __hrtimer_run_queues+0x1c2/0x4d0 [15668.242125] ? watchdog+0x40/0x40 [15668.244302] ? hrtimer_interrupt+0x128/0x2c0 [15668.248501] ? smp_apic_timer_interrupt+0xa6/0x270 [15668.253056] ? apic_timer_interrupt+0xf/0x20 [15668.255982] [15668.256758] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15668.266538] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15668.270946] ? newidle_balance.isra.93+0x207/0x580 [15668.273589] do_raw_spin_lock+0x177/0x190 [15668.274881] _raw_spin_lock+0x21/0x30 [15668.276985] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15668.281454] process_one_work+0x2c8/0x700 [15668.283179] worker_thread+0x296/0x6e0 [15668.287302] ? rescuer_thread+0x570/0x570 [15668.289436] kthread+0x1d1/0x200 [15668.290949] ? set_kthread_struct+0x70/0x70 [15668.292841] ret_from_fork+0x1f/0x30 [15673.829566] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13036s! [15673.837268] Showing busy workqueues and worker pools: [15673.843551] workqueue events: flags=0x0 [15673.844711] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15673.849203] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15673.853268] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15673.855348] pending: cpu_latency_work [ptlrpc] [15673.857564] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15673.860738] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15673.864330] workqueue events_long: flags=0x0 [15673.865774] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15673.868330] pending: lu_objects_flush [obdclass] [15673.872788] workqueue events_unbound: flags=0x2 [15673.874508] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15673.876987] pending: flush_memcg_stats_dwork, idle_cull_fn [15673.880583] workqueue events_power_efficient: flags=0x80 [15673.883300] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15673.886105] pending: check_lifetime, do_cache_clean [15673.888696] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15673.891900] pending: neigh_periodic_work, neigh_periodic_work [15673.894294] workqueue mm_percpu_wq: flags=0x8 [15673.896338] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15673.899424] pending: vmstat_update, lru_add_drain_per_cpu [15673.901992] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15673.904387] pending: vmstat_update, lru_add_drain_per_cpu [15673.906404] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15673.908680] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15673.911510] workqueue xprtiod: flags=0xa [15673.913287] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15673.917595] pending: xprt_autoclose, xs_error_handle [15673.920470] workqueue ipv6_addrconf: flags=0x40008 [15673.922274] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15673.929354] pending: addrconf_verify_work [15673.932323] workqueue knbd0-recv: flags=0x1a [15673.933954] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15673.936744] in-flight: 651:recv_work [15673.938853] workqueue ptlrpc_pinger: flags=0x2 [15673.940643] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15673.943122] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15673.946111] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13036s workers=4 idle: 15979 31824 8 [15673.949439] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15664s workers=2 idle: 654 [15680.057653] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [15680.062886] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [15680.063328] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15680.063404] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15680.063409] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15680.064165] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15680.064180] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15680.064185] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15680.064189] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15680.064192] RDX: 0000000000007f2b RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15680.064194] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15680.064196] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15680.064199] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15680.064202] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15680.064204] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15680.064206] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15680.064214] Call Trace: [15680.064220] [15680.064228] ? show_regs.cold.9+0x22/0x2f [15680.064237] ? watchdog_timer_fn.cold.6+0x62/0x10a [15680.064244] ? __hrtimer_run_queues+0x1c2/0x4d0 [15680.064248] ? watchdog+0x40/0x40 [15680.064253] ? hrtimer_interrupt+0x128/0x2c0 [15680.064256] ? smp_apic_timer_interrupt+0xa6/0x270 [15680.064264] ? apic_timer_interrupt+0xf/0x20 [15680.064268] [15680.064271] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15680.064275] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15680.064280] do_raw_spin_lock+0x177/0x190 [15680.064285] _raw_spin_lock+0x21/0x30 [15680.064292] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15680.065048] obd_mod_free+0x46/0x140 [ptlrpc] [15680.065252] mdc_commit_open+0x91/0xb0 [mdc] [15680.065295] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15680.065709] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15680.065974] after_reply+0xe2c/0x1780 [ptlrpc] [15680.066173] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15680.066374] ? __next_timer_interrupt+0x160/0x160 [15680.066384] ptlrpcd+0xa82/0xe40 [ptlrpc] [15680.066606] ? do_wait_intr+0xf0/0xf0 [15680.066615] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15680.067060] kthread+0x1d1/0x200 [15680.067066] ? set_kthread_struct+0x70/0x70 [15680.067070] ret_from_fork+0x1f/0x30 [15680.293078] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15680.312385] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15680.316594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15680.324817] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15680.332495] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15680.348403] RSP: 0018:ffffbb5381503500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [15680.355819] RAX: 0000000000007a87 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15680.363343] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15680.368048] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15680.371129] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15680.373491] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15680.376546] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15680.380336] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15680.383223] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15680.386502] Call Trace: [15680.387542] [15680.388523] ? show_regs.cold.9+0x22/0x2f [15680.390419] ? watchdog_timer_fn.cold.6+0x62/0x10a [15680.396084] ? __hrtimer_run_queues+0x1c2/0x4d0 [15680.398139] ? watchdog+0x40/0x40 [15680.404197] ? hrtimer_interrupt+0x128/0x2c0 [15680.407595] ? smp_apic_timer_interrupt+0xa6/0x270 [15680.409854] ? apic_timer_interrupt+0xf/0x20 [15680.413650] [15680.415608] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15680.419043] do_raw_spin_lock+0x177/0x190 [15680.422071] _raw_spin_lock+0x21/0x30 [15680.424042] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15680.430573] ? do_raw_spin_unlock+0x75/0x190 [15680.433320] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15680.437432] ? do_wait_intr+0xf0/0xf0 [15680.442605] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15680.448746] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15680.454391] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15680.464379] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15680.473536] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15680.481919] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15680.485145] mdc_intent_lock+0x353/0x7e0 [mdc] [15680.488582] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15680.492203] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15680.497094] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15680.503131] lmv_intent_open+0x338/0x1150 [lmv] [15680.515386] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15680.523652] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15680.533506] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15680.547619] ll_intent_lock+0xc5/0xaa0 [lustre] [15680.558742] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15680.560627] ll_lookup_it+0xe0e/0x2270 [lustre] [15680.572314] ? do_raw_spin_unlock+0x75/0x190 [15680.576727] ? _raw_spin_unlock+0x12/0x30 [15680.582882] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15680.587233] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15680.592720] ll_atomic_open+0x36a/0xd70 [lustre] [15680.595709] lookup_open+0x356/0xb40 [15680.600548] do_last+0x313/0xfc0 [15680.601649] ? nd_jump_root+0xe5/0x160 [15680.602822] ? path_init+0x437/0x520 [15680.606545] path_openat+0xf7/0x500 [15680.611604] ? do_raw_spin_unlock+0x75/0x190 [15680.613794] do_filp_open+0x99/0x140 [15680.616597] ? getname_flags+0x6e/0x330 [15680.626293] ? __check_object_size+0xff/0x256 [15680.627787] ? do_raw_spin_unlock+0x75/0x190 [15680.631724] ? _raw_spin_unlock+0x12/0x30 [15680.636581] do_sys_openat2+0x2b4/0x410 [15680.638728] do_sys_open+0x73/0xa0 [15680.645745] __x64_sys_openat+0x24/0x30 [15680.651186] do_syscall_64+0xc1/0x440 [15680.654222] entry_SYSCALL_64_after_hwframe+0x49/0xae [15680.660509] RIP: 0033:0x7f53425a5332 [15680.664342] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15680.681938] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15680.689506] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15680.696832] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15680.703729] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15680.706786] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15680.719171] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15684.058083] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15684.063363] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15684.108534] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15684.111772] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15684.120765] RIP: 0010:native_safe_halt+0x2e/0x50 [15684.124369] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15684.139761] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [15684.148005] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15684.155019] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15684.162102] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15684.169240] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15684.172499] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15684.179854] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15684.186927] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15684.193011] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15684.199721] Call Trace: [15684.201543] [15684.205132] ? show_regs.cold.9+0x22/0x2f [15684.210108] ? watchdog_timer_fn.cold.6+0x62/0x10a [15684.215657] ? __hrtimer_run_queues+0x1c2/0x4d0 [15684.221711] ? watchdog+0x40/0x40 [15684.223312] ? hrtimer_interrupt+0x128/0x2c0 [15684.225563] ? smp_apic_timer_interrupt+0xa6/0x270 [15684.230346] ? apic_timer_interrupt+0xf/0x20 [15684.232555] [15684.233289] ? native_safe_halt+0x2e/0x50 [15684.234591] kvm_wait+0xcd/0xe0 [15684.236276] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15684.238532] do_raw_spin_lock+0x177/0x190 [15684.240238] _raw_spin_lock+0x21/0x30 [15684.245084] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15684.249022] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15684.257684] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15684.263740] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15684.268528] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15684.273552] ? woken_wake_function+0x30/0x30 [15684.277571] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15684.284075] kthread+0x1d1/0x200 [15684.285776] ? set_kthread_struct+0x70/0x70 [15684.291484] ret_from_fork+0x1f/0x30 [15700.052014] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [15700.060847] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15700.096786] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15700.116675] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15700.121969] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15700.127466] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15700.129295] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15700.144008] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15700.146766] RAX: 00000000000066eb RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15700.151684] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15700.162764] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15700.168023] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15700.173234] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15700.176659] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15700.180737] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15700.182891] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15700.186267] Call Trace: [15700.187623] [15700.188601] ? show_regs.cold.9+0x22/0x2f [15700.190343] ? watchdog_timer_fn.cold.6+0x62/0x10a [15700.195805] ? __hrtimer_run_queues+0x1c2/0x4d0 [15700.198210] ? watchdog+0x40/0x40 [15700.199707] ? hrtimer_interrupt+0x128/0x2c0 [15700.201970] ? smp_apic_timer_interrupt+0xa6/0x270 [15700.204509] ? apic_timer_interrupt+0xf/0x20 [15700.211502] [15700.212472] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15700.217857] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15700.220790] ? newidle_balance.isra.93+0x207/0x580 [15700.226574] do_raw_spin_lock+0x177/0x190 [15700.233838] _raw_spin_lock+0x21/0x30 [15700.237815] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15700.241004] process_one_work+0x2c8/0x700 [15700.243826] worker_thread+0x296/0x6e0 [15700.246170] ? rescuer_thread+0x570/0x570 [15700.248490] kthread+0x1d1/0x200 [15700.250233] ? set_kthread_struct+0x70/0x70 [15700.252920] ret_from_fork+0x1f/0x30 [15704.034762] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13066s! [15704.041074] Showing busy workqueues and worker pools: [15704.050077] workqueue events: flags=0x0 [15704.052033] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15704.055033] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15704.060989] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15704.066056] pending: cpu_latency_work [ptlrpc] [15704.067875] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15704.072097] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15704.076813] workqueue events_long: flags=0x0 [15704.079266] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15704.081687] pending: lu_objects_flush [obdclass] [15704.085808] workqueue events_unbound: flags=0x2 [15704.091803] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15704.094535] pending: flush_memcg_stats_dwork, idle_cull_fn [15704.102096] workqueue events_power_efficient: flags=0x80 [15704.109322] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15704.114197] pending: check_lifetime, do_cache_clean [15704.119007] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15704.124006] pending: neigh_periodic_work, neigh_periodic_work [15704.129182] workqueue mm_percpu_wq: flags=0x8 [15704.134934] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15704.142923] pending: vmstat_update, lru_add_drain_per_cpu [15704.152743] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15704.163847] pending: vmstat_update, lru_add_drain_per_cpu [15704.171133] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15704.179635] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15704.184199] workqueue xprtiod: flags=0xa [15704.192689] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15704.197224] pending: xprt_autoclose, xs_error_handle [15704.200650] workqueue ipv6_addrconf: flags=0x40008 [15704.205246] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15704.212347] pending: addrconf_verify_work [15704.215152] workqueue knbd0-recv: flags=0x1a [15704.219848] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15704.225936] in-flight: 651:recv_work [15704.227346] workqueue ptlrpc_pinger: flags=0x2 [15704.230086] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15704.234279] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15704.238039] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13067s workers=4 idle: 15979 31824 8 [15704.241606] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15694s workers=2 idle: 654 [15708.055049] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [15708.057025] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [15708.058307] Modules linked in: [15708.061338] Modules linked in: [15708.061341] lustre(O) osp(O) [15708.062365] lustre(O) [15708.063406] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15708.063549] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15708.063555] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15708.063558] RIP: 0010:native_safe_halt+0x2e/0x50 [15708.063581] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15708.063585] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15708.063589] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15708.063591] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15708.063593] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15708.063595] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15708.063597] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15708.063600] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15708.063602] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15708.063604] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15708.063612] Call Trace: [15708.063624] [15708.063630] ? show_regs.cold.9+0x22/0x2f [15708.063639] ? watchdog_timer_fn.cold.6+0x62/0x10a [15708.063645] ? __hrtimer_run_queues+0x1c2/0x4d0 [15708.063650] ? watchdog+0x40/0x40 [15708.063654] ? hrtimer_interrupt+0x128/0x2c0 [15708.063658] ? smp_apic_timer_interrupt+0xa6/0x270 [15708.063665] ? apic_timer_interrupt+0xf/0x20 [15708.063669] [15708.063673] ? native_safe_halt+0x2e/0x50 [15708.063677] kvm_wait+0xcd/0xe0 [15708.063683] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15708.063692] do_raw_spin_lock+0x177/0x190 [15708.063696] _raw_spin_lock+0x21/0x30 [15708.063700] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15708.063988] ? do_raw_spin_unlock+0x75/0x190 [15708.063995] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15708.064189] ? do_wait_intr+0xf0/0xf0 [15708.064198] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15708.064400] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15708.064612] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15708.064823] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15708.064934] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15708.064972] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15708.065001] mdc_intent_lock+0x353/0x7e0 [mdc] [15708.065031] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15708.065115] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15708.065326] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15708.065357] lmv_intent_open+0x338/0x1150 [lmv] [15708.065865] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15708.065956] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15708.066041] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15708.066063] ll_intent_lock+0xc5/0xaa0 [lustre] [15708.066147] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15708.066227] ll_lookup_it+0xe0e/0x2270 [lustre] [15708.066310] ? do_raw_spin_unlock+0x75/0x190 [15708.066317] ? _raw_spin_unlock+0x12/0x30 [15708.066323] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15708.066353] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15708.068290] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15708.068362] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15708.068369] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15708.068371] RIP: 0010:native_safe_halt+0x2e/0x50 [15708.099682] ll_atomic_open+0x36a/0xd70 [lustre] [15708.099828] lookup_open+0x356/0xb40 [15708.099839] do_last+0x313/0xfc0 [15708.099845] ? nd_jump_root+0xe5/0x160 [15708.099850] ? path_init+0x437/0x520 [15708.099854] path_openat+0xf7/0x500 [15708.099860] ? do_raw_spin_unlock+0x75/0x190 [15708.099867] do_filp_open+0x99/0x140 [15708.099874] ? getname_flags+0x6e/0x330 [15708.099879] ? __check_object_size+0xff/0x256 [15708.099886] ? do_raw_spin_unlock+0x75/0x190 [15708.099890] ? _raw_spin_unlock+0x12/0x30 [15708.099896] do_sys_openat2+0x2b4/0x410 [15708.099901] do_sys_open+0x73/0xa0 [15708.099907] __x64_sys_openat+0x24/0x30 [15708.099910] do_syscall_64+0xc1/0x440 [15708.099917] entry_SYSCALL_64_after_hwframe+0x49/0xae [15708.099924] RIP: 0033:0x7f53425a5332 [15708.099931] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15708.099936] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15708.099941] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15708.099944] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15708.099946] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15708.099948] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15708.099951] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15708.342565] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15708.351264] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15708.355253] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15708.358613] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15708.362284] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15708.365503] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15708.369218] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15708.373119] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15708.377434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15708.380167] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15708.382260] Call Trace: [15708.384083] [15708.385120] ? show_regs.cold.9+0x22/0x2f [15708.387301] ? watchdog_timer_fn.cold.6+0x62/0x10a [15708.390905] ? __hrtimer_run_queues+0x1c2/0x4d0 [15708.394055] ? watchdog+0x40/0x40 [15708.396683] ? hrtimer_interrupt+0x128/0x2c0 [15708.398975] ? smp_apic_timer_interrupt+0xa6/0x270 [15708.404707] ? apic_timer_interrupt+0xf/0x20 [15708.408602] [15708.410250] ? native_safe_halt+0x2e/0x50 [15708.412158] kvm_wait+0xcd/0xe0 [15708.417385] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15708.421282] do_raw_spin_lock+0x177/0x190 [15708.427836] _raw_spin_lock+0x21/0x30 [15708.429939] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15708.435598] obd_mod_free+0x46/0x140 [ptlrpc] [15708.443063] mdc_commit_open+0x91/0xb0 [mdc] [15708.454325] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15708.466207] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15708.475377] after_reply+0xe2c/0x1780 [ptlrpc] [15708.483322] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15708.488737] ? __next_timer_interrupt+0x160/0x160 [15708.490164] ptlrpcd+0xa82/0xe40 [ptlrpc] [15708.492602] ? do_wait_intr+0xf0/0xf0 [15708.495311] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15708.500528] kthread+0x1d1/0x200 [15708.502243] ? set_kthread_struct+0x70/0x70 [15708.504291] ret_from_fork+0x1f/0x30 [15712.058505] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15712.071554] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15712.120792] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15712.129976] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15712.148849] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15712.161917] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15712.176876] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15712.188289] RAX: 00000000000012ae RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15712.196718] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15712.200849] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15712.209498] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15712.213868] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15712.218137] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15712.226033] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15712.229709] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15712.240674] Call Trace: [15712.241583] [15712.242326] ? show_regs.cold.9+0x22/0x2f [15712.245774] ? watchdog_timer_fn.cold.6+0x62/0x10a [15712.249781] ? __hrtimer_run_queues+0x1c2/0x4d0 [15712.253884] ? watchdog+0x40/0x40 [15712.256794] ? hrtimer_interrupt+0x128/0x2c0 [15712.259274] ? smp_apic_timer_interrupt+0xa6/0x270 [15712.264927] ? apic_timer_interrupt+0xf/0x20 [15712.267135] [15712.270758] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15712.279164] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15712.289049] do_raw_spin_lock+0x177/0x190 [15712.292468] _raw_spin_lock+0x21/0x30 [15712.294489] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15712.301050] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15712.304006] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15712.308726] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15712.315447] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15712.320876] ? woken_wake_function+0x30/0x30 [15712.325501] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15712.336006] kthread+0x1d1/0x200 [15712.340175] ? set_kthread_struct+0x70/0x70 [15712.347712] ret_from_fork+0x1f/0x30 [15732.055702] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [15732.058322] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15732.073848] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15732.079527] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15732.083080] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15732.085507] RIP: 0010:native_safe_halt+0x2e/0x50 [15732.086947] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15732.093810] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15732.096660] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15732.099352] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [15732.102510] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15732.104867] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15732.107772] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15732.110465] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15732.113239] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15732.115189] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15732.117448] Call Trace: [15732.118427] [15732.119383] ? show_regs.cold.9+0x22/0x2f [15732.120772] ? watchdog_timer_fn.cold.6+0x62/0x10a [15732.122462] ? __hrtimer_run_queues+0x1c2/0x4d0 [15732.123906] ? watchdog+0x40/0x40 [15732.124902] ? hrtimer_interrupt+0x128/0x2c0 [15732.126206] ? smp_apic_timer_interrupt+0xa6/0x270 [15732.127718] ? apic_timer_interrupt+0xf/0x20 [15732.128920] [15732.129650] ? native_safe_halt+0x2e/0x50 [15732.131171] kvm_wait+0xcd/0xe0 [15732.132275] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15732.134134] ? newidle_balance.isra.93+0x207/0x580 [15732.135897] do_raw_spin_lock+0x177/0x190 [15732.137389] _raw_spin_lock+0x21/0x30 [15732.138512] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15732.140762] process_one_work+0x2c8/0x700 [15732.142486] worker_thread+0x296/0x6e0 [15732.143870] ? rescuer_thread+0x570/0x570 [15732.145335] kthread+0x1d1/0x200 [15732.146731] ? set_kthread_struct+0x70/0x70 [15732.148517] ret_from_fork+0x1f/0x30 [15734.754994] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13097s! [15734.763287] Showing busy workqueues and worker pools: [15734.766625] workqueue events: flags=0x0 [15734.770894] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15734.785308] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15734.821140] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15734.835390] pending: cpu_latency_work [ptlrpc] [15734.840013] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15734.848762] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15734.864506] workqueue events_long: flags=0x0 [15734.873281] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15734.882793] pending: lu_objects_flush [obdclass] [15734.890890] workqueue events_unbound: flags=0x2 [15734.896359] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15734.901096] pending: flush_memcg_stats_dwork, idle_cull_fn [15734.907162] workqueue events_power_efficient: flags=0x80 [15734.917545] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15734.930638] pending: check_lifetime, do_cache_clean [15734.935114] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15734.940348] pending: neigh_periodic_work, neigh_periodic_work [15734.943841] workqueue mm_percpu_wq: flags=0x8 [15734.945848] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15734.949968] pending: vmstat_update, lru_add_drain_per_cpu [15734.953247] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15734.957712] pending: vmstat_update, lru_add_drain_per_cpu [15734.964652] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15734.971718] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15734.977696] workqueue xprtiod: flags=0xa [15734.979501] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15734.988198] pending: xprt_autoclose, xs_error_handle [15734.990760] workqueue ipv6_addrconf: flags=0x40008 [15734.993116] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15734.996396] pending: addrconf_verify_work [15735.001327] workqueue knbd0-recv: flags=0x1a [15735.006548] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15735.011983] in-flight: 651:recv_work [15735.014121] workqueue ptlrpc_pinger: flags=0x2 [15735.016274] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15735.019235] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15735.022645] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13097s workers=4 idle: 15979 31824 8 [15735.026565] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15725s workers=2 idle: 654 [15736.055021] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [15736.057059] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [15736.058661] Modules linked in: [15736.061030] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15736.061121] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15736.061130] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15736.061133] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15736.061148] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15736.061153] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15736.061158] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15736.061160] RDX: 0000000000004659 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15736.061162] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15736.061164] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15736.061166] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15736.061169] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15736.061172] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15736.061174] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15736.061181] Call Trace: [15736.061187] [15736.061192] ? show_regs.cold.9+0x22/0x2f [15736.061200] ? watchdog_timer_fn.cold.6+0x62/0x10a [15736.061207] ? __hrtimer_run_queues+0x1c2/0x4d0 [15736.061210] ? watchdog+0x40/0x40 [15736.061214] ? hrtimer_interrupt+0x128/0x2c0 [15736.061217] ? smp_apic_timer_interrupt+0xa6/0x270 [15736.061225] ? apic_timer_interrupt+0xf/0x20 [15736.061229] [15736.061232] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15736.061237] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15736.061241] do_raw_spin_lock+0x177/0x190 [15736.061246] _raw_spin_lock+0x21/0x30 [15736.061252] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15736.062862] lustre(O) [15736.077278] obd_mod_free+0x46/0x140 [ptlrpc] [15736.081451] osp(O) [15736.084681] mdc_commit_open+0x91/0xb0 [mdc] [15736.084730] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15736.085001] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15736.085264] after_reply+0xe2c/0x1780 [ptlrpc] [15736.088699] ofd(O) [15736.098174] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15736.098451] ? __next_timer_interrupt+0x160/0x160 [15736.098463] ptlrpcd+0xa82/0xe40 [ptlrpc] [15736.099434] ? do_wait_intr+0xf0/0xf0 [15736.099444] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15736.099833] kthread+0x1d1/0x200 [15736.099840] ? set_kthread_struct+0x70/0x70 [15736.099845] ret_from_fork+0x1f/0x30 [15736.219704] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15736.256688] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15736.265856] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15736.274196] RIP: 0010:native_safe_halt+0x2e/0x50 [15736.276860] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15736.286392] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15736.297841] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15736.301756] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15736.308500] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15736.310508] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15736.314008] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15736.318061] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15736.324627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15736.327458] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15736.331350] Call Trace: [15736.332664] [15736.333736] ? show_regs.cold.9+0x22/0x2f [15736.335799] ? watchdog_timer_fn.cold.6+0x62/0x10a [15736.338360] ? __hrtimer_run_queues+0x1c2/0x4d0 [15736.340694] ? watchdog+0x40/0x40 [15736.342217] ? hrtimer_interrupt+0x128/0x2c0 [15736.344016] ? smp_apic_timer_interrupt+0xa6/0x270 [15736.346059] ? apic_timer_interrupt+0xf/0x20 [15736.350246] [15736.351229] ? native_safe_halt+0x2e/0x50 [15736.353487] kvm_wait+0xcd/0xe0 [15736.357199] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15736.359755] do_raw_spin_lock+0x177/0x190 [15736.365108] _raw_spin_lock+0x21/0x30 [15736.368448] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15736.374401] ? do_raw_spin_unlock+0x75/0x190 [15736.377185] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15736.381184] ? do_wait_intr+0xf0/0xf0 [15736.383737] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15736.387404] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15736.389818] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15736.397229] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15736.403263] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15736.406706] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15736.410758] mdc_intent_lock+0x353/0x7e0 [mdc] [15736.413834] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15736.418434] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15736.423399] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15736.430047] lmv_intent_open+0x338/0x1150 [lmv] [15736.431987] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15736.435040] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15736.438328] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15736.440772] ll_intent_lock+0xc5/0xaa0 [lustre] [15736.447240] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15736.450415] ll_lookup_it+0xe0e/0x2270 [lustre] [15736.454024] ? do_raw_spin_unlock+0x75/0x190 [15736.457779] ? _raw_spin_unlock+0x12/0x30 [15736.461694] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15736.466673] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15736.470135] ll_atomic_open+0x36a/0xd70 [lustre] [15736.473115] lookup_open+0x356/0xb40 [15736.478204] do_last+0x313/0xfc0 [15736.479354] ? nd_jump_root+0xe5/0x160 [15736.480684] ? path_init+0x437/0x520 [15736.486594] path_openat+0xf7/0x500 [15736.489643] ? do_raw_spin_unlock+0x75/0x190 [15736.492143] do_filp_open+0x99/0x140 [15736.494749] ? getname_flags+0x6e/0x330 [15736.501673] ? __check_object_size+0xff/0x256 [15736.506340] ? do_raw_spin_unlock+0x75/0x190 [15736.509757] ? _raw_spin_unlock+0x12/0x30 [15736.515839] do_sys_openat2+0x2b4/0x410 [15736.519457] do_sys_open+0x73/0xa0 [15736.523189] __x64_sys_openat+0x24/0x30 [15736.527091] do_syscall_64+0xc1/0x440 [15736.530145] entry_SYSCALL_64_after_hwframe+0x49/0xae [15736.533152] RIP: 0033:0x7f53425a5332 [15736.537137] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15736.551982] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15736.557380] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15736.566115] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15736.573074] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15736.578963] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15736.581809] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15740.059246] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [15740.065374] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15740.105249] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15740.117032] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15740.141434] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15740.146830] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15740.161271] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15740.165480] RAX: 000000000000798c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15740.171600] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15740.177211] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15740.180795] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15740.183405] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15740.186345] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15740.193282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15740.195370] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15740.204208] Call Trace: [15740.205914] [15740.207143] ? show_regs.cold.9+0x22/0x2f [15740.208918] ? watchdog_timer_fn.cold.6+0x62/0x10a [15740.211163] ? __hrtimer_run_queues+0x1c2/0x4d0 [15740.213363] ? watchdog+0x40/0x40 [15740.216098] ? hrtimer_interrupt+0x128/0x2c0 [15740.218383] ? smp_apic_timer_interrupt+0xa6/0x270 [15740.223477] ? apic_timer_interrupt+0xf/0x20 [15740.225444] [15740.226906] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15740.233356] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15740.237733] do_raw_spin_lock+0x177/0x190 [15740.243251] _raw_spin_lock+0x21/0x30 [15740.245636] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15740.249990] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15740.254302] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15740.258722] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15740.265589] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15740.273350] ? woken_wake_function+0x30/0x30 [15740.279705] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15740.284623] kthread+0x1d1/0x200 [15740.287352] ? set_kthread_struct+0x70/0x70 [15740.294656] ret_from_fork+0x1f/0x30 [15760.052621] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [15760.057031] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15760.090145] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15760.095612] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15760.099740] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15760.103007] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15760.105614] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15760.122740] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15760.127086] RAX: 0000000000007d88 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15760.131960] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15760.135580] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15760.139961] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15760.144363] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15760.148609] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15760.152153] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15760.160711] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15760.167676] Call Trace: [15760.168834] [15760.169549] ? show_regs.cold.9+0x22/0x2f [15760.174342] ? watchdog_timer_fn.cold.6+0x62/0x10a [15760.177200] ? __hrtimer_run_queues+0x1c2/0x4d0 [15760.179747] ? watchdog+0x40/0x40 [15760.185743] ? hrtimer_interrupt+0x128/0x2c0 [15760.188653] ? smp_apic_timer_interrupt+0xa6/0x270 [15760.190379] ? apic_timer_interrupt+0xf/0x20 [15760.193084] [15760.194297] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15760.205964] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15760.210136] ? newidle_balance.isra.93+0x207/0x580 [15760.218563] do_raw_spin_lock+0x177/0x190 [15760.223722] _raw_spin_lock+0x21/0x30 [15760.228592] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15760.233939] process_one_work+0x2c8/0x700 [15760.235315] worker_thread+0x296/0x6e0 [15760.237565] ? rescuer_thread+0x570/0x570 [15760.245723] kthread+0x1d1/0x200 [15760.250662] ? set_kthread_struct+0x70/0x70 [15760.255205] ret_from_fork+0x1f/0x30 [15764.055312] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [15764.058194] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [15764.058352] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15764.058443] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15764.058450] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15764.058453] RIP: 0010:native_safe_halt+0x2e/0x50 [15764.058468] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15764.058473] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15764.058477] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15764.058479] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15764.058481] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15764.058484] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15764.058486] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15764.058489] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15764.058492] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15764.058494] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15764.058503] Call Trace: [15764.058510] [15764.058516] ? show_regs.cold.9+0x22/0x2f [15764.058526] ? watchdog_timer_fn.cold.6+0x62/0x10a [15764.058532] ? __hrtimer_run_queues+0x1c2/0x4d0 [15764.058537] ? watchdog+0x40/0x40 [15764.058541] ? hrtimer_interrupt+0x128/0x2c0 [15764.058545] ? smp_apic_timer_interrupt+0xa6/0x270 [15764.058552] ? apic_timer_interrupt+0xf/0x20 [15764.058557] [15764.058560] ? native_safe_halt+0x2e/0x50 [15764.058565] kvm_wait+0xcd/0xe0 [15764.058572] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15764.058581] do_raw_spin_lock+0x177/0x190 [15764.058587] _raw_spin_lock+0x21/0x30 [15764.058591] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15764.058986] ? do_raw_spin_unlock+0x75/0x190 [15764.058993] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15764.059353] ? do_wait_intr+0xf0/0xf0 [15764.059363] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15764.059684] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15764.059965] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15764.060371] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15764.060673] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15764.060714] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15764.060742] mdc_intent_lock+0x353/0x7e0 [mdc] [15764.060768] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15764.060860] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15764.061155] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15764.061186] lmv_intent_open+0x338/0x1150 [lmv] [15764.061214] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15764.061304] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15764.061657] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15764.061686] ll_intent_lock+0xc5/0xaa0 [lustre] [15764.061785] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15764.061871] ll_lookup_it+0xe0e/0x2270 [lustre] [15764.061955] ? do_raw_spin_unlock+0x75/0x190 [15764.061964] ? _raw_spin_unlock+0x12/0x30 [15764.061981] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15764.062021] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15764.062044] ll_atomic_open+0x36a/0xd70 [lustre] [15764.062128] lookup_open+0x356/0xb40 [15764.062139] do_last+0x313/0xfc0 [15764.062144] ? nd_jump_root+0xe5/0x160 [15764.062149] ? path_init+0x437/0x520 [15764.062154] path_openat+0xf7/0x500 [15764.062159] ? do_raw_spin_unlock+0x75/0x190 [15764.062164] do_filp_open+0x99/0x140 [15764.062171] ? getname_flags+0x6e/0x330 [15764.062175] ? __check_object_size+0xff/0x256 [15764.062183] ? do_raw_spin_unlock+0x75/0x190 [15764.062188] ? _raw_spin_unlock+0x12/0x30 [15764.062192] do_sys_openat2+0x2b4/0x410 [15764.062197] do_sys_open+0x73/0xa0 [15764.062202] __x64_sys_openat+0x24/0x30 [15764.062206] do_syscall_64+0xc1/0x440 [15764.062213] entry_SYSCALL_64_after_hwframe+0x49/0xae [15764.062220] RIP: 0033:0x7f53425a5332 [15764.062226] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15764.062231] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15764.062236] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15764.062239] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15764.062242] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15764.062244] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15764.062246] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15764.378538] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15764.411932] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15764.417715] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15764.421496] RIP: 0010:native_safe_halt+0x2e/0x50 [15764.423321] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15764.430747] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15764.433520] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15764.436370] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15764.439078] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15764.441662] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15764.444232] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15764.447278] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15764.451381] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15764.457475] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15764.464534] Call Trace: [15764.466353] [15764.468298] ? show_regs.cold.9+0x22/0x2f [15764.472227] ? watchdog_timer_fn.cold.6+0x62/0x10a [15764.477561] ? __hrtimer_run_queues+0x1c2/0x4d0 [15764.481321] ? watchdog+0x40/0x40 [15764.482921] ? hrtimer_interrupt+0x128/0x2c0 [15764.484906] ? smp_apic_timer_interrupt+0xa6/0x270 [15764.487303] ? apic_timer_interrupt+0xf/0x20 [15764.489171] [15764.490153] ? native_safe_halt+0x2e/0x50 [15764.491768] kvm_wait+0xcd/0xe0 [15764.493281] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15764.495541] do_raw_spin_lock+0x177/0x190 [15764.497190] _raw_spin_lock+0x21/0x30 [15764.498479] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15764.500288] obd_mod_free+0x46/0x140 [ptlrpc] [15764.501859] mdc_commit_open+0x91/0xb0 [mdc] [15764.503323] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15764.508027] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15764.511118] after_reply+0xe2c/0x1780 [ptlrpc] [15764.516334] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15764.519683] ? __next_timer_interrupt+0x160/0x160 [15764.521966] ptlrpcd+0xa82/0xe40 [ptlrpc] [15764.523911] ? do_wait_intr+0xf0/0xf0 [15764.528910] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15764.531533] kthread+0x1d1/0x200 [15764.532952] ? set_kthread_struct+0x70/0x70 [15764.535214] ret_from_fork+0x1f/0x30 [15765.472034] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13128s! [15765.474619] Showing busy workqueues and worker pools: [15765.479796] workqueue events: flags=0x0 [15765.480974] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15765.485361] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15765.492346] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15765.493881] pending: cpu_latency_work [ptlrpc] [15765.496727] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15765.500525] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15765.505790] workqueue events_long: flags=0x0 [15765.508597] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15765.511076] pending: lu_objects_flush [obdclass] [15765.513256] workqueue events_unbound: flags=0x2 [15765.514910] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15765.517442] pending: flush_memcg_stats_dwork, idle_cull_fn [15765.519646] workqueue events_power_efficient: flags=0x80 [15765.521909] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15765.524717] pending: check_lifetime, do_cache_clean [15765.526782] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15765.529155] pending: neigh_periodic_work, neigh_periodic_work [15765.531304] workqueue mm_percpu_wq: flags=0x8 [15765.532900] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15765.535828] pending: vmstat_update, lru_add_drain_per_cpu [15765.538153] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15765.546900] pending: vmstat_update, lru_add_drain_per_cpu [15765.550739] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15765.553059] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15765.556126] workqueue xprtiod: flags=0xa [15765.560920] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15765.562456] pending: xprt_autoclose, xs_error_handle [15765.564735] workqueue ipv6_addrconf: flags=0x40008 [15765.567038] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15765.571755] pending: addrconf_verify_work [15765.572814] workqueue knbd0-recv: flags=0x1a [15765.574674] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15765.576819] in-flight: 651:recv_work [15765.578306] workqueue ptlrpc_pinger: flags=0x2 [15765.580522] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15765.582969] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15765.585918] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13128s workers=4 idle: 15979 31824 8 [15765.589604] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15756s workers=2 idle: 654 [15768.060465] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [15768.067590] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15768.099475] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15768.108856] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15768.117679] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15768.123039] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15768.141635] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15768.151045] RAX: 0000000000007e86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15768.158187] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15768.163404] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15768.167930] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15768.171968] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15768.175410] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15768.180195] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15768.183098] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15768.186643] Call Trace: [15768.187924] [15768.189119] ? show_regs.cold.9+0x22/0x2f [15768.191374] ? watchdog_timer_fn.cold.6+0x62/0x10a [15768.194219] ? __hrtimer_run_queues+0x1c2/0x4d0 [15768.198084] ? watchdog+0x40/0x40 [15768.199747] ? hrtimer_interrupt+0x128/0x2c0 [15768.202216] ? smp_apic_timer_interrupt+0xa6/0x270 [15768.205134] ? apic_timer_interrupt+0xf/0x20 [15768.207549] [15768.208786] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15768.211482] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15768.213782] do_raw_spin_lock+0x177/0x190 [15768.216153] _raw_spin_lock+0x21/0x30 [15768.218261] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15768.222059] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15768.225183] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15768.233847] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15768.239574] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15768.242818] ? woken_wake_function+0x30/0x30 [15768.246101] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15768.249964] kthread+0x1d1/0x200 [15768.251717] ? set_kthread_struct+0x70/0x70 [15768.254213] ret_from_fork+0x1f/0x30 [15792.054971] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [15792.055977] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [15792.057925] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [15792.057935] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15792.058005] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15792.058009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15792.058220] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15792.058233] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15792.058236] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15792.058240] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15792.058242] RDX: 00000000000015d9 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15792.058243] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15792.058245] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15792.058247] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15792.058249] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15792.058251] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15792.058252] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15792.058259] Call Trace: [15792.058263] [15792.058268] ? show_regs.cold.9+0x22/0x2f [15792.058276] ? watchdog_timer_fn.cold.6+0x62/0x10a [15792.058280] ? __hrtimer_run_queues+0x1c2/0x4d0 [15792.058283] ? watchdog+0x40/0x40 [15792.058287] ? hrtimer_interrupt+0x128/0x2c0 [15792.058291] ? smp_apic_timer_interrupt+0xa6/0x270 [15792.058299] ? apic_timer_interrupt+0xf/0x20 [15792.058303] [15792.058306] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15792.058310] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15792.058315] do_raw_spin_lock+0x177/0x190 [15792.058320] _raw_spin_lock+0x21/0x30 [15792.058326] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15792.058810] obd_mod_free+0x46/0x140 [ptlrpc] [15792.059018] mdc_commit_open+0x91/0xb0 [mdc] [15792.059056] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15792.059401] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15792.062009] after_reply+0xe2c/0x1780 [ptlrpc] [15792.062311] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15792.062626] ? __next_timer_interrupt+0x160/0x160 [15792.062637] ptlrpcd+0xa82/0xe40 [ptlrpc] [15792.062877] ? do_wait_intr+0xf0/0xf0 [15792.062884] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15792.063217] kthread+0x1d1/0x200 [15792.063222] ? set_kthread_struct+0x70/0x70 [15792.063225] ret_from_fork+0x1f/0x30 [15792.065010] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15792.065089] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15792.065096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15792.065102] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15792.065353] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15792.065362] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15792.065366] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15792.065371] RAX: 0000000000005984 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15792.065373] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15792.065375] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15792.066026] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15792.066031] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15792.066034] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15792.066037] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15792.066041] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15792.066049] Call Trace: [15792.066054] [15792.066059] ? show_regs.cold.9+0x22/0x2f [15792.066069] ? watchdog_timer_fn.cold.6+0x62/0x10a [15792.066075] ? __hrtimer_run_queues+0x1c2/0x4d0 [15792.066079] ? watchdog+0x40/0x40 [15792.066083] ? hrtimer_interrupt+0x128/0x2c0 [15792.066087] ? smp_apic_timer_interrupt+0xa6/0x270 [15792.066097] ? apic_timer_interrupt+0xf/0x20 [15792.066101] [15792.066104] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15792.066109] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15792.066112] ? newidle_balance.isra.93+0x207/0x580 [15792.066118] do_raw_spin_lock+0x177/0x190 [15792.066123] _raw_spin_lock+0x21/0x30 [15792.066127] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15792.066348] process_one_work+0x2c8/0x700 [15792.066356] worker_thread+0x296/0x6e0 [15792.066361] ? rescuer_thread+0x570/0x570 [15792.066364] kthread+0x1d1/0x200 [15792.066369] ? set_kthread_struct+0x70/0x70 [15792.066373] ret_from_fork+0x1f/0x30 [15792.624410] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15792.653104] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15792.659714] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15792.665612] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15792.667735] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15792.685514] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15792.694106] RAX: 0000000000004f12 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15792.697359] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15792.704836] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15792.715707] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15792.724054] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15792.733704] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15792.742443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15792.748101] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15792.751423] Call Trace: [15792.754461] [15792.758071] ? show_regs.cold.9+0x22/0x2f [15792.768513] ? watchdog_timer_fn.cold.6+0x62/0x10a [15792.775474] ? __hrtimer_run_queues+0x1c2/0x4d0 [15792.780726] ? watchdog+0x40/0x40 [15792.786698] ? hrtimer_interrupt+0x128/0x2c0 [15792.794806] ? smp_apic_timer_interrupt+0xa6/0x270 [15792.796448] ? apic_timer_interrupt+0xf/0x20 [15792.803092] [15792.804485] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15792.811573] do_raw_spin_lock+0x177/0x190 [15792.821421] _raw_spin_lock+0x21/0x30 [15792.826393] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15792.835424] ? do_raw_spin_unlock+0x75/0x190 [15792.836821] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15792.842860] ? do_wait_intr+0xf0/0xf0 [15792.844710] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15792.851253] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15792.858390] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15792.862457] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15792.865450] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15792.869978] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15792.871876] mdc_intent_lock+0x353/0x7e0 [mdc] [15792.881104] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15792.895443] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15792.902465] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15792.906786] lmv_intent_open+0x338/0x1150 [lmv] [15792.909187] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15792.911807] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15792.914309] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15792.916328] ll_intent_lock+0xc5/0xaa0 [lustre] [15792.923596] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15792.932954] ll_lookup_it+0xe0e/0x2270 [lustre] [15792.937770] ? do_raw_spin_unlock+0x75/0x190 [15792.943719] ? _raw_spin_unlock+0x12/0x30 [15792.945108] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15792.955121] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15792.960586] ll_atomic_open+0x36a/0xd70 [lustre] [15792.965406] lookup_open+0x356/0xb40 [15792.967858] do_last+0x313/0xfc0 [15792.969749] ? nd_jump_root+0xe5/0x160 [15792.971419] ? path_init+0x437/0x520 [15792.974718] path_openat+0xf7/0x500 [15792.975975] ? do_raw_spin_unlock+0x75/0x190 [15792.985638] do_filp_open+0x99/0x140 [15792.991032] ? getname_flags+0x6e/0x330 [15792.999569] ? __check_object_size+0xff/0x256 [15793.004372] ? do_raw_spin_unlock+0x75/0x190 [15793.012728] ? _raw_spin_unlock+0x12/0x30 [15793.022280] do_sys_openat2+0x2b4/0x410 [15793.028001] do_sys_open+0x73/0xa0 [15793.035143] __x64_sys_openat+0x24/0x30 [15793.043202] do_syscall_64+0xc1/0x440 [15793.048712] entry_SYSCALL_64_after_hwframe+0x49/0xae [15793.061167] RIP: 0033:0x7f53425a5332 [15793.065220] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15793.092717] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15793.096400] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15793.103246] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15793.108350] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15793.114629] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15793.117951] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15795.681033] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13158s! [15795.689323] Showing busy workqueues and worker pools: [15795.693124] workqueue events: flags=0x0 [15795.696434] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15795.701676] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15795.716961] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15795.727987] pending: cpu_latency_work [ptlrpc] [15795.730481] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15795.743178] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15795.759973] workqueue events_long: flags=0x0 [15795.767052] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15795.777926] pending: lu_objects_flush [obdclass] [15795.779799] workqueue events_unbound: flags=0x2 [15795.787728] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15795.798165] pending: flush_memcg_stats_dwork, idle_cull_fn [15795.806758] workqueue events_power_efficient: flags=0x80 [15795.808770] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15795.822104] pending: check_lifetime, do_cache_clean [15795.825414] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15795.834476] pending: neigh_periodic_work, neigh_periodic_work [15795.844784] workqueue mm_percpu_wq: flags=0x8 [15795.851513] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15795.857306] pending: vmstat_update, lru_add_drain_per_cpu [15795.863221] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15795.867043] pending: vmstat_update, lru_add_drain_per_cpu [15795.873477] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15795.880454] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15795.893480] workqueue xprtiod: flags=0xa [15795.900907] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15795.907003] pending: xprt_autoclose, xs_error_handle [15795.912117] workqueue ipv6_addrconf: flags=0x40008 [15795.917240] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15795.922214] pending: addrconf_verify_work [15795.925377] workqueue knbd0-recv: flags=0x1a [15795.926646] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15795.933136] in-flight: 651:recv_work [15795.936175] workqueue ptlrpc_pinger: flags=0x2 [15795.940182] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15795.943464] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15795.949052] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13158s workers=4 idle: 15979 31824 8 [15795.959280] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15786s workers=2 idle: 654 [15796.062194] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [15796.067163] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15796.112927] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15796.122500] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15796.131977] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15796.135426] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15796.159793] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15796.167348] RAX: 0000000000007f83 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15796.169063] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15796.170664] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15796.179550] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15796.184148] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15796.193128] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15796.197999] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15796.201453] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15796.207416] Call Trace: [15796.210330] [15796.212112] ? show_regs.cold.9+0x22/0x2f [15796.213040] ? watchdog_timer_fn.cold.6+0x62/0x10a [15796.214182] ? __hrtimer_run_queues+0x1c2/0x4d0 [15796.219229] ? watchdog+0x40/0x40 [15796.221134] ? hrtimer_interrupt+0x128/0x2c0 [15796.226458] ? smp_apic_timer_interrupt+0xa6/0x270 [15796.229412] ? apic_timer_interrupt+0xf/0x20 [15796.230825] [15796.233504] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15796.241760] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15796.245075] do_raw_spin_lock+0x177/0x190 [15796.249400] _raw_spin_lock+0x21/0x30 [15796.255163] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15796.258331] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15796.262457] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15796.265165] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15796.271236] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15796.276396] ? woken_wake_function+0x30/0x30 [15796.278662] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15796.282409] kthread+0x1d1/0x200 [15796.283433] ? set_kthread_struct+0x70/0x70 [15796.289306] ret_from_fork+0x1f/0x30 [15820.052047] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [15820.055036] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [15820.056733] Modules linked in: [15820.057077] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [15820.057084] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15820.057156] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15820.057160] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15820.057163] RIP: 0010:native_safe_halt+0x2e/0x50 [15820.057176] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15820.057180] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15820.057184] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15820.057187] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15820.057189] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15820.057191] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15820.057194] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15820.057197] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15820.057207] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15820.057209] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15820.057217] Call Trace: [15820.057228] [15820.057233] ? show_regs.cold.9+0x22/0x2f [15820.057242] ? watchdog_timer_fn.cold.6+0x62/0x10a [15820.057249] ? __hrtimer_run_queues+0x1c2/0x4d0 [15820.057253] ? watchdog+0x40/0x40 [15820.057258] ? hrtimer_interrupt+0x128/0x2c0 [15820.057262] ? smp_apic_timer_interrupt+0xa6/0x270 [15820.057269] ? apic_timer_interrupt+0xf/0x20 [15820.057274] [15820.057278] ? native_safe_halt+0x2e/0x50 [15820.057282] kvm_wait+0xcd/0xe0 [15820.057313] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15820.057320] do_raw_spin_lock+0x177/0x190 [15820.057325] _raw_spin_lock+0x21/0x30 [15820.057330] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15820.057655] obd_mod_free+0x46/0x140 [ptlrpc] [15820.057863] mdc_commit_open+0x91/0xb0 [mdc] [15820.057900] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15820.058109] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15820.058335] after_reply+0xe2c/0x1780 [ptlrpc] [15820.058596] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15820.058834] ? __next_timer_interrupt+0x160/0x160 [15820.058843] ptlrpcd+0xa82/0xe40 [ptlrpc] [15820.059073] ? do_wait_intr+0xf0/0xf0 [15820.059082] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15820.059297] kthread+0x1d1/0x200 [15820.059303] ? set_kthread_struct+0x70/0x70 [15820.059307] ret_from_fork+0x1f/0x30 [15820.059317] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [15820.061804] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15820.061856] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15820.061905] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15820.061910] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15820.062246] RIP: 0010:native_safe_halt+0x2e/0x50 [15820.062318] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15820.062322] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15820.062388] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15820.062393] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [15820.062396] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15820.062398] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15820.062400] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15820.062403] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15820.062405] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15820.062469] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15820.062478] Call Trace: [15820.062483] [15820.062489] ? show_regs.cold.9+0x22/0x2f [15820.062499] ? watchdog_timer_fn.cold.6+0x62/0x10a [15820.062505] ? __hrtimer_run_queues+0x1c2/0x4d0 [15820.062517] ? watchdog+0x40/0x40 [15820.062521] ? hrtimer_interrupt+0x128/0x2c0 [15820.062525] ? smp_apic_timer_interrupt+0xa6/0x270 [15820.062532] ? apic_timer_interrupt+0xf/0x20 [15820.062536] [15820.062539] ? native_safe_halt+0x2e/0x50 [15820.062549] kvm_wait+0xcd/0xe0 [15820.062557] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15820.062563] ? newidle_balance.isra.93+0x207/0x580 [15820.062570] do_raw_spin_lock+0x177/0x190 [15820.062579] _raw_spin_lock+0x21/0x30 [15820.062584] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15820.063114] process_one_work+0x2c8/0x700 [15820.063123] worker_thread+0x296/0x6e0 [15820.063128] ? rescuer_thread+0x570/0x570 [15820.063132] kthread+0x1d1/0x200 [15820.063136] ? set_kthread_struct+0x70/0x70 [15820.063205] ret_from_fork+0x1f/0x30 [15820.384052] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15820.419642] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15820.425537] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15820.433259] RIP: 0010:native_safe_halt+0x2e/0x50 [15820.435390] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15820.448835] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15820.456258] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15820.464250] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15820.471016] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15820.474022] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15820.477591] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15820.485908] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15820.491753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15820.494915] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15820.501568] Call Trace: [15820.502856] [15820.505368] ? show_regs.cold.9+0x22/0x2f [15820.507280] ? watchdog_timer_fn.cold.6+0x62/0x10a [15820.511085] ? __hrtimer_run_queues+0x1c2/0x4d0 [15820.515862] ? watchdog+0x40/0x40 [15820.517888] ? hrtimer_interrupt+0x128/0x2c0 [15820.520971] ? smp_apic_timer_interrupt+0xa6/0x270 [15820.522680] ? apic_timer_interrupt+0xf/0x20 [15820.530233] [15820.531709] ? native_safe_halt+0x2e/0x50 [15820.535911] kvm_wait+0xcd/0xe0 [15820.540898] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15820.545374] do_raw_spin_lock+0x177/0x190 [15820.552706] _raw_spin_lock+0x21/0x30 [15820.554281] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15820.558068] ? do_raw_spin_unlock+0x75/0x190 [15820.561991] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15820.564424] ? do_wait_intr+0xf0/0xf0 [15820.566097] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15820.569034] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15820.572706] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15820.578557] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15820.584950] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15820.587194] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15820.591524] mdc_intent_lock+0x353/0x7e0 [mdc] [15820.596260] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15820.598586] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15820.602540] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15820.607368] lmv_intent_open+0x338/0x1150 [lmv] [15820.611502] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15820.620107] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15820.622561] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15820.627065] ll_intent_lock+0xc5/0xaa0 [lustre] [15820.628994] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15820.631844] ll_lookup_it+0xe0e/0x2270 [lustre] [15820.634160] ? do_raw_spin_unlock+0x75/0x190 [15820.638369] ? _raw_spin_unlock+0x12/0x30 [15820.640226] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15820.642490] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15820.644488] ll_atomic_open+0x36a/0xd70 [lustre] [15820.646555] lookup_open+0x356/0xb40 [15820.649147] do_last+0x313/0xfc0 [15820.651059] ? nd_jump_root+0xe5/0x160 [15820.652919] ? path_init+0x437/0x520 [15820.654320] path_openat+0xf7/0x500 [15820.655947] ? do_raw_spin_unlock+0x75/0x190 [15820.658023] do_filp_open+0x99/0x140 [15820.659210] ? getname_flags+0x6e/0x330 [15820.661002] ? __check_object_size+0xff/0x256 [15820.662533] ? do_raw_spin_unlock+0x75/0x190 [15820.664240] ? _raw_spin_unlock+0x12/0x30 [15820.665603] do_sys_openat2+0x2b4/0x410 [15820.667198] do_sys_open+0x73/0xa0 [15820.668620] __x64_sys_openat+0x24/0x30 [15820.670293] do_syscall_64+0xc1/0x440 [15820.671929] entry_SYSCALL_64_after_hwframe+0x49/0xae [15820.673865] RIP: 0033:0x7f53425a5332 [15820.675391] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15820.683571] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15820.687310] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15820.690626] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15820.693862] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15820.696681] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15820.699695] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15824.058023] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [15824.061279] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15824.120107] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15824.134881] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15824.145069] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15824.152006] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15824.176742] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15824.184207] RAX: 0000000000000b2c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15824.195289] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15824.202430] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15824.209243] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15824.223251] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15824.235244] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15824.243661] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15824.249240] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15824.257847] Call Trace: [15824.260050] [15824.262028] ? show_regs.cold.9+0x22/0x2f [15824.264856] ? watchdog_timer_fn.cold.6+0x62/0x10a [15824.272144] ? __hrtimer_run_queues+0x1c2/0x4d0 [15824.276131] ? watchdog+0x40/0x40 [15824.280961] ? hrtimer_interrupt+0x128/0x2c0 [15824.284219] ? smp_apic_timer_interrupt+0xa6/0x270 [15824.291134] ? apic_timer_interrupt+0xf/0x20 [15824.292614] [15824.294421] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15824.297051] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15824.301296] do_raw_spin_lock+0x177/0x190 [15824.303268] _raw_spin_lock+0x21/0x30 [15824.305178] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15824.310585] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15824.322719] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15824.329302] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15824.336997] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15824.343184] ? woken_wake_function+0x30/0x30 [15824.350064] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15824.355089] kthread+0x1d1/0x200 [15824.359936] ? set_kthread_struct+0x70/0x70 [15824.366012] ret_from_fork+0x1f/0x30 [15826.404608] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13189s! [15826.408506] Showing busy workqueues and worker pools: [15826.410737] workqueue events: flags=0x0 [15826.412230] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15826.416015] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15826.420541] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15826.423099] pending: cpu_latency_work [ptlrpc] [15826.427764] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15826.431754] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15826.444273] workqueue events_long: flags=0x0 [15826.450166] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15826.453320] pending: lu_objects_flush [obdclass] [15826.457178] workqueue events_unbound: flags=0x2 [15826.460220] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15826.467606] pending: flush_memcg_stats_dwork, idle_cull_fn [15826.469596] workqueue events_power_efficient: flags=0x80 [15826.472452] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15826.477326] pending: check_lifetime, do_cache_clean [15826.481209] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15826.486704] pending: neigh_periodic_work, neigh_periodic_work [15826.489483] workqueue mm_percpu_wq: flags=0x8 [15826.491497] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15826.495825] pending: vmstat_update, lru_add_drain_per_cpu [15826.498385] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15826.501727] pending: vmstat_update, lru_add_drain_per_cpu [15826.507132] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15826.517995] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15826.523830] workqueue xprtiod: flags=0xa [15826.526014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15826.531293] pending: xprt_autoclose, xs_error_handle [15826.534775] workqueue ipv6_addrconf: flags=0x40008 [15826.537032] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15826.540846] pending: addrconf_verify_work [15826.543168] workqueue knbd0-recv: flags=0x1a [15826.544930] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15826.547701] in-flight: 651:recv_work [15826.549477] workqueue ptlrpc_pinger: flags=0x2 [15826.551055] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15826.553926] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15826.558751] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13189s workers=4 idle: 15979 31824 8 [15826.568144] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15817s workers=2 idle: 654 [15834.774288] rcu: INFO: rcu_sched self-detected stall on CPU [15834.783459] rcu: 2-....: (7637707 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [15834.797913] (t=13740348 jiffies g=274185 q=67449) [15834.808775] rcu: rcu_sched kthread starved for 13198714 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [15834.821889] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [15834.845662] rcu: RCU grace-period kthread stack dump: [15834.857469] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [15834.872874] Call Trace: [15834.874438] __schedule+0x351/0xcb0 [15834.879521] schedule+0xc0/0x180 [15834.887109] schedule_timeout+0xb4/0x190 [15834.894476] ? do_raw_spin_unlock+0x75/0x190 [15834.906354] ? __next_timer_interrupt+0x160/0x160 [15834.913568] ? prepare_to_swait_event+0xf3/0x140 [15834.915319] rcu_gp_kthread+0x8dd/0x15b0 [15834.916683] ? force_qs_rnp+0x2c0/0x2c0 [15834.920370] kthread+0x1d1/0x200 [15834.922724] ? set_kthread_struct+0x70/0x70 [15834.931871] ret_from_fork+0x1f/0x30 [15834.940561] rcu: Stack dump where RCU GP kthread last ran: [15834.946033] Sending NMI from CPU 2 to CPUs 3: [15834.951830] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [15834.956866] Sending NMI from CPU 2 to CPUs 1: [15834.972512] NMI backtrace for cpu 1 [15834.972519] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15834.972521] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15834.972523] RIP: 0010:__pv_queued_spin_lock_slowpath+0x320/0x6d0 [15834.972524] Code: 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 <48> 83 05 18 d4 dc 02 01 eb e0 48 83 05 06 d4 dc 02 01 48 83 05 be [15834.972527] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [15834.972529] RAX: 0000000000002d19 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15834.972531] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15834.972532] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15834.972533] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15834.972535] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15834.972536] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15834.972537] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15834.972538] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15834.972539] Call Trace: [15834.972540] [15834.972541] ? show_regs.cold.9+0x22/0x2f [15834.972542] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [15834.972544] ? nmi_cpu_backtrace_handler+0x15/0x40 [15834.972545] ? nmi_handle+0xbd/0x210 [15834.972546] ? default_do_nmi+0x4b/0x1b0 [15834.972547] ? do_nmi+0x280/0x380 [15834.972548] ? end_repeat_nmi+0x16/0x1f [15834.972549] ? __pv_queued_spin_lock_slowpath+0x320/0x6d0 [15834.972550] ? __pv_queued_spin_lock_slowpath+0x320/0x6d0 [15834.972552] ? __pv_queued_spin_lock_slowpath+0x320/0x6d0 [15834.972553] [15834.972554] do_raw_spin_lock+0x177/0x190 [15834.972555] _raw_spin_lock+0x21/0x30 [15834.972556] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15834.972557] ? do_raw_spin_unlock+0x75/0x190 [15834.972558] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15834.972559] ? do_wait_intr+0xf0/0xf0 [15834.972561] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15834.972562] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15834.972563] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15834.972564] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15834.972565] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15834.972566] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15834.972568] mdc_intent_lock+0x353/0x7e0 [mdc] [15834.972569] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15834.972570] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15834.972571] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15834.972572] lmv_intent_open+0x338/0x1150 [lmv] [15834.972574] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15834.972575] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15834.972576] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15834.972577] ll_intent_lock+0xc5/0xaa0 [lustre] [15834.972578] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15834.972579] ll_lookup_it+0xe0e/0x2270 [lustre] [15834.972580] ? do_raw_spin_unlock+0x75/0x190 [15834.972581] ? _raw_spin_unlock+0x12/0x30 [15834.972582] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15834.972583] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15834.972584] ll_atomic_open+0x36a/0xd70 [lustre] [15834.972585] lookup_open+0x356/0xb40 [15834.972586] do_last+0x313/0xfc0 [15834.972587] ? nd_jump_root+0xe5/0x160 [15834.972588] ? path_init+0x437/0x520 [15834.972589] path_openat+0xf7/0x500 [15834.972590] ? do_raw_spin_unlock+0x75/0x190 [15834.972591] do_filp_open+0x99/0x140 [15834.972592] ? getname_flags+0x6e/0x330 [15834.972593] ? __check_object_size+0xff/0x256 [15834.972594] ? do_raw_spin_unlock+0x75/0x190 [15834.972595] ? _raw_spin_unlock+0x12/0x30 [15834.972596] do_sys_openat2+0x2b4/0x410 [15834.972598] do_sys_open+0x73/0xa0 [15834.972599] __x64_sys_openat+0x24/0x30 [15834.972600] do_syscall_64+0xc1/0x440 [15834.972601] entry_SYSCALL_64_after_hwframe+0x49/0xae [15834.972602] RIP: 0033:0x7f53425a5332 [15834.972604] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15834.972606] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15834.972609] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15834.972611] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15834.972612] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15834.972613] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15834.972615] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15834.973333] NMI backtrace for cpu 2 [15835.335543] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15835.343637] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15835.350556] Call Trace: [15835.352465] [15835.354024] ? dump_stack+0xbb/0x10e [15835.356905] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [15835.359932] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [15835.362857] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [15835.365628] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [15835.368638] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [15835.372043] ? rcu_dump_cpu_stacks+0x11b/0x171 [15835.374306] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [15835.377981] ? update_process_times+0x6d/0xc0 [15835.383856] ? tick_sched_handle.isra.18+0x27/0x90 [15835.389596] ? tick_sched_timer+0x53/0xb0 [15835.394006] ? __hrtimer_run_queues+0x1c2/0x4d0 [15835.397403] ? tick_sched_handle.isra.18+0x90/0x90 [15835.402512] ? hrtimer_interrupt+0x128/0x2c0 [15835.404983] ? smp_apic_timer_interrupt+0xa6/0x270 [15835.407505] ? apic_timer_interrupt+0xf/0x20 [15835.409853] [15835.410742] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15835.413535] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15835.416423] ? do_raw_spin_lock+0x177/0x190 [15835.418648] ? _raw_spin_lock+0x21/0x30 [15835.423552] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15835.429576] ? obd_mod_free+0x46/0x140 [ptlrpc] [15835.432162] ? mdc_commit_open+0x91/0xb0 [mdc] [15835.435959] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15835.441113] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15835.449272] ? after_reply+0xe2c/0x1780 [ptlrpc] [15835.453998] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15835.458261] ? __next_timer_interrupt+0x160/0x160 [15835.462919] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [15835.467448] ? do_wait_intr+0xf0/0xf0 [15835.470775] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15835.473890] ? kthread+0x1d1/0x200 [15835.475399] ? set_kthread_struct+0x70/0x70 [15835.477878] ? ret_from_fork+0x1f/0x30 [15852.053840] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [15852.062562] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15852.100385] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15852.106879] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15852.110825] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15852.114450] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15852.117253] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15852.125390] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [15852.128466] RAX: 0000000000007e53 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15852.131940] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15852.135089] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15852.137570] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15852.140837] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15852.144419] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15852.148783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15852.151742] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15852.155409] Call Trace: [15852.156786] [15852.157804] ? show_regs.cold.9+0x22/0x2f [15852.160135] ? watchdog_timer_fn.cold.6+0x62/0x10a [15852.162733] ? __hrtimer_run_queues+0x1c2/0x4d0 [15852.165263] ? watchdog+0x40/0x40 [15852.167177] ? hrtimer_interrupt+0x128/0x2c0 [15852.169371] ? smp_apic_timer_interrupt+0xa6/0x270 [15852.171689] ? apic_timer_interrupt+0xf/0x20 [15852.173777] [15852.174980] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15852.177829] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15852.180515] ? newidle_balance.isra.93+0x207/0x580 [15852.182998] do_raw_spin_lock+0x177/0x190 [15852.184891] _raw_spin_lock+0x21/0x30 [15852.186759] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15852.189746] process_one_work+0x2c8/0x700 [15852.192202] worker_thread+0x296/0x6e0 [15852.194127] ? rescuer_thread+0x570/0x570 [15852.196172] kthread+0x1d1/0x200 [15852.197667] ? set_kthread_struct+0x70/0x70 [15852.199814] ret_from_fork+0x1f/0x30 [15856.607770] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13219s! [15856.615114] Showing busy workqueues and worker pools: [15856.624072] workqueue events: flags=0x0 [15856.631858] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15856.639485] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15856.655111] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15856.662525] pending: cpu_latency_work [ptlrpc] [15856.666851] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15856.672426] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15856.677468] workqueue events_long: flags=0x0 [15856.680865] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15856.684231] pending: lu_objects_flush [obdclass] [15856.686312] workqueue events_unbound: flags=0x2 [15856.688672] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15856.693646] pending: flush_memcg_stats_dwork, idle_cull_fn [15856.696046] workqueue events_power_efficient: flags=0x80 [15856.698934] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15856.707690] pending: check_lifetime, do_cache_clean [15856.717237] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15856.731724] pending: neigh_periodic_work, neigh_periodic_work [15856.736455] workqueue mm_percpu_wq: flags=0x8 [15856.738365] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15856.741630] pending: vmstat_update, lru_add_drain_per_cpu [15856.744112] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15856.746895] pending: vmstat_update, lru_add_drain_per_cpu [15856.749348] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15856.753945] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15856.761460] workqueue xprtiod: flags=0xa [15856.766312] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15856.771888] pending: xprt_autoclose, xs_error_handle [15856.775152] workqueue ipv6_addrconf: flags=0x40008 [15856.777396] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15856.780359] pending: addrconf_verify_work [15856.782980] workqueue knbd0-recv: flags=0x1a [15856.788580] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15856.796780] in-flight: 651:recv_work [15856.808180] workqueue ptlrpc_pinger: flags=0x2 [15856.809729] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15856.827387] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15856.833933] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13219s workers=4 idle: 15979 31824 8 [15856.838060] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15847s workers=2 idle: 654 [15860.055354] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [15860.058781] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15860.058798] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [15860.058806] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15860.058875] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15860.058881] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15860.058883] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15860.058898] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15860.058902] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15860.058906] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15860.058908] RDX: 0000000000001ade RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15860.058911] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15860.058913] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15860.058915] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15860.058918] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15860.058921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15860.058923] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15860.058933] Call Trace: [15860.058938] [15860.058945] ? show_regs.cold.9+0x22/0x2f [15860.058954] ? watchdog_timer_fn.cold.6+0x62/0x10a [15860.058961] ? __hrtimer_run_queues+0x1c2/0x4d0 [15860.058965] ? watchdog+0x40/0x40 [15860.058968] ? hrtimer_interrupt+0x128/0x2c0 [15860.060559] ? smp_apic_timer_interrupt+0xa6/0x270 [15860.060569] ? apic_timer_interrupt+0xf/0x20 [15860.060574] [15860.060579] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15860.060587] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15860.060591] do_raw_spin_lock+0x177/0x190 [15860.060597] _raw_spin_lock+0x21/0x30 [15860.060605] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15860.062502] obd_mod_free+0x46/0x140 [ptlrpc] [15860.062735] mdc_commit_open+0x91/0xb0 [mdc] [15860.062780] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15860.063198] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15860.063350] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15860.063495] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15860.063501] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15860.063504] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15860.063515] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15860.063518] RSP: 0018:ffffbb5381503500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [15860.063523] RAX: 0000000000007008 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15860.063422] after_reply+0xe2c/0x1780 [ptlrpc] [15860.063528] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15860.063530] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15860.063533] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15860.063535] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15860.063539] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15860.063542] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15860.063544] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15860.063552] Call Trace: [15860.063556] [15860.063558] ? show_regs.cold.9+0x22/0x2f [15860.063567] ? watchdog_timer_fn.cold.6+0x62/0x10a [15860.063573] ? __hrtimer_run_queues+0x1c2/0x4d0 [15860.063577] ? watchdog+0x40/0x40 [15860.063582] ? hrtimer_interrupt+0x128/0x2c0 [15860.063586] ? smp_apic_timer_interrupt+0xa6/0x270 [15860.063594] ? apic_timer_interrupt+0xf/0x20 [15860.063599] [15860.063602] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15860.063608] do_raw_spin_lock+0x177/0x190 [15860.063612] _raw_spin_lock+0x21/0x30 [15860.063617] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15860.063952] ? do_raw_spin_unlock+0x75/0x190 [15860.063958] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15860.063632] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15860.065348] ? do_wait_intr+0xf0/0xf0 [15860.065360] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15860.065667] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15860.065899] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15860.066128] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15860.066257] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15860.066296] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15860.066325] mdc_intent_lock+0x353/0x7e0 [mdc] [15860.066354] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15860.067181] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15860.067567] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15860.067610] lmv_intent_open+0x338/0x1150 [lmv] [15860.067644] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15860.067752] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15860.067884] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15860.067973] ll_intent_lock+0xc5/0xaa0 [lustre] [15860.068207] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15860.068297] ll_lookup_it+0xe0e/0x2270 [lustre] [15860.068387] ? do_raw_spin_unlock+0x75/0x190 [15860.068397] ? _raw_spin_unlock+0x12/0x30 [15860.068404] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15860.068503] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15860.068525] ll_atomic_open+0x36a/0xd70 [lustre] [15860.068611] lookup_open+0x356/0xb40 [15860.068619] do_last+0x313/0xfc0 [15860.068624] ? nd_jump_root+0xe5/0x160 [15860.068629] ? path_init+0x437/0x520 [15860.068634] path_openat+0xf7/0x500 [15860.068639] ? do_raw_spin_unlock+0x75/0x190 [15860.068644] do_filp_open+0x99/0x140 [15860.068651] ? getname_flags+0x6e/0x330 [15860.068655] ? __check_object_size+0xff/0x256 [15860.068663] ? do_raw_spin_unlock+0x75/0x190 [15860.068667] ? _raw_spin_unlock+0x12/0x30 [15860.068672] do_sys_openat2+0x2b4/0x410 [15860.068676] do_sys_open+0x73/0xa0 [15860.068680] __x64_sys_openat+0x24/0x30 [15860.068684] do_syscall_64+0xc1/0x440 [15860.068692] entry_SYSCALL_64_after_hwframe+0x49/0xae [15860.068698] RIP: 0033:0x7f53425a5332 [15860.068756] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15860.068846] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15860.068917] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15860.068923] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15860.068925] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15860.068934] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15860.068937] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15860.068941] RAX: 000000000000261b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15860.068943] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15860.068945] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15860.068947] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15860.068949] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15860.068953] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15860.068956] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15860.068958] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15860.068967] Call Trace: [15860.068971] [15860.068974] ? show_regs.cold.9+0x22/0x2f [15860.068985] ? watchdog_timer_fn.cold.6+0x62/0x10a [15860.068991] ? __hrtimer_run_queues+0x1c2/0x4d0 [15860.068995] ? watchdog+0x40/0x40 [15860.068999] ? hrtimer_interrupt+0x128/0x2c0 [15860.069003] ? smp_apic_timer_interrupt+0xa6/0x270 [15860.069009] ? apic_timer_interrupt+0xf/0x20 [15860.069015] [15860.069018] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15860.069022] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15860.069028] do_raw_spin_lock+0x177/0x190 [15860.069032] _raw_spin_lock+0x21/0x30 [15860.069037] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15860.069435] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15860.069647] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15860.069911] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15860.070116] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15860.071821] ? woken_wake_function+0x30/0x30 [15860.071833] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15860.073913] kthread+0x1d1/0x200 [15860.073922] ? set_kthread_struct+0x70/0x70 [15860.073926] ret_from_fork+0x1f/0x30 [15860.076309] ? __next_timer_interrupt+0x160/0x160 [15860.096632] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15860.096646] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15860.096649] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15860.096651] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15860.096654] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15860.096655] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15860.504387] ptlrpcd+0xa82/0xe40 [ptlrpc] [15860.506330] ? do_wait_intr+0xf0/0xf0 [15860.507619] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15860.509957] kthread+0x1d1/0x200 [15860.511106] ? set_kthread_struct+0x70/0x70 [15860.512450] ret_from_fork+0x1f/0x30 [15884.052284] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [15884.061971] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15884.115922] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15884.126838] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15884.134171] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15884.140087] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15884.146026] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15884.165082] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15884.174083] RAX: 0000000000001584 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15884.178110] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15884.184275] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15884.192642] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15884.198170] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15884.205675] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15884.214565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15884.220927] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15884.227108] Call Trace: [15884.229242] [15884.236865] ? show_regs.cold.9+0x22/0x2f [15884.242983] ? watchdog_timer_fn.cold.6+0x62/0x10a [15884.250094] ? __hrtimer_run_queues+0x1c2/0x4d0 [15884.257679] ? watchdog+0x40/0x40 [15884.260224] ? hrtimer_interrupt+0x128/0x2c0 [15884.265580] ? smp_apic_timer_interrupt+0xa6/0x270 [15884.269841] ? apic_timer_interrupt+0xf/0x20 [15884.274119] [15884.276175] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15884.281215] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15884.285021] ? newidle_balance.isra.93+0x207/0x580 [15884.287015] do_raw_spin_lock+0x177/0x190 [15884.288554] _raw_spin_lock+0x21/0x30 [15884.289780] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15884.292148] process_one_work+0x2c8/0x700 [15884.294079] worker_thread+0x296/0x6e0 [15884.295554] ? rescuer_thread+0x570/0x570 [15884.296908] kthread+0x1d1/0x200 [15884.297949] ? set_kthread_struct+0x70/0x70 [15884.299940] ret_from_fork+0x1f/0x30 [15887.331322] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13250s! [15887.335027] Showing busy workqueues and worker pools: [15887.348915] workqueue events: flags=0x0 [15887.353110] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15887.358227] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15887.370781] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15887.375119] pending: cpu_latency_work [ptlrpc] [15887.379798] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15887.389515] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15887.406785] workqueue events_long: flags=0x0 [15887.411321] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15887.420174] pending: lu_objects_flush [obdclass] [15887.433379] workqueue events_unbound: flags=0x2 [15887.435951] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15887.443947] pending: flush_memcg_stats_dwork, idle_cull_fn [15887.449030] workqueue events_power_efficient: flags=0x80 [15887.452035] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15887.455740] pending: check_lifetime, do_cache_clean [15887.458104] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15887.465844] pending: neigh_periodic_work, neigh_periodic_work [15887.469740] workqueue mm_percpu_wq: flags=0x8 [15887.475580] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15887.487096] pending: vmstat_update, lru_add_drain_per_cpu [15887.492031] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15887.499163] pending: vmstat_update, lru_add_drain_per_cpu [15887.502959] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15887.509036] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15887.513520] workqueue xprtiod: flags=0xa [15887.517734] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15887.520572] pending: xprt_autoclose, xs_error_handle [15887.523392] workqueue ipv6_addrconf: flags=0x40008 [15887.526367] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15887.530697] pending: addrconf_verify_work [15887.532466] workqueue knbd0-recv: flags=0x1a [15887.535250] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15887.546599] in-flight: 651:recv_work [15887.549052] workqueue ptlrpc_pinger: flags=0x2 [15887.551205] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15887.555202] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15887.558495] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13250s workers=4 idle: 15979 31824 8 [15887.562508] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15878s workers=2 idle: 654 [15888.055029] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [15888.058084] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15888.058094] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [15888.058099] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15888.058172] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15888.058177] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15888.058180] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15888.058193] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15888.058197] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15888.058201] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15888.058206] RDX: 00000000000071c2 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15888.058208] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15888.058211] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15888.058213] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15888.058216] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15888.058218] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15888.058221] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15888.058236] Call Trace: [15888.058241] [15888.058246] ? show_regs.cold.9+0x22/0x2f [15888.058254] ? watchdog_timer_fn.cold.6+0x62/0x10a [15888.058260] ? __hrtimer_run_queues+0x1c2/0x4d0 [15888.058263] ? watchdog+0x40/0x40 [15888.058267] ? hrtimer_interrupt+0x128/0x2c0 [15888.058271] ? smp_apic_timer_interrupt+0xa6/0x270 [15888.058279] ? apic_timer_interrupt+0xf/0x20 [15888.058288] [15888.058290] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15888.058295] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15888.058299] do_raw_spin_lock+0x177/0x190 [15888.058304] _raw_spin_lock+0x21/0x30 [15888.058309] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15888.058581] obd_mod_free+0x46/0x140 [ptlrpc] [15888.059739] mdc_commit_open+0x91/0xb0 [mdc] [15888.059774] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15888.060513] Modules linked in: [15888.059984] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15888.060748] after_reply+0xe2c/0x1780 [ptlrpc] [15888.060941] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15888.061753] ? __next_timer_interrupt+0x160/0x160 [15888.061766] ptlrpcd+0xa82/0xe40 [ptlrpc] [15888.062099] ? do_wait_intr+0xf0/0xf0 [15888.062109] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15888.062331] kthread+0x1d1/0x200 [15888.062337] ? set_kthread_struct+0x70/0x70 [15888.062341] ret_from_fork+0x1f/0x30 [15888.063603] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15888.063674] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15888.063681] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15888.063684] RIP: 0010:native_safe_halt+0x2e/0x50 [15888.063693] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15888.063697] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15888.063702] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15888.063704] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15888.063707] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15888.063709] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15888.063711] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15888.063715] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15888.063718] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15888.063720] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15888.063729] Call Trace: [15888.063734] [15888.063738] ? show_regs.cold.9+0x22/0x2f [15888.063748] ? watchdog_timer_fn.cold.6+0x62/0x10a [15888.068613] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15888.068805] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15888.068811] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15888.068814] RIP: 0010:native_safe_halt+0x2e/0x50 [15888.068827] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15888.068832] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15888.068837] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15888.068839] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15888.068842] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15888.068844] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15888.068846] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15888.068850] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15888.068853] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15888.068856] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15888.068865] Call Trace: [15888.068870] [15888.068876] ? show_regs.cold.9+0x22/0x2f [15888.068887] ? watchdog_timer_fn.cold.6+0x62/0x10a [15888.068894] ? __hrtimer_run_queues+0x1c2/0x4d0 [15888.068898] ? watchdog+0x40/0x40 [15888.068903] ? hrtimer_interrupt+0x128/0x2c0 [15888.068907] ? smp_apic_timer_interrupt+0xa6/0x270 [15888.068914] ? apic_timer_interrupt+0xf/0x20 [15888.068919] [15888.068922] ? native_safe_halt+0x2e/0x50 [15888.068926] kvm_wait+0xcd/0xe0 [15888.068933] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15888.068942] do_raw_spin_lock+0x177/0x190 [15888.068946] _raw_spin_lock+0x21/0x30 [15888.068950] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15888.069340] ? do_raw_spin_unlock+0x75/0x190 [15888.069346] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15888.069538] ? do_wait_intr+0xf0/0xf0 [15888.069546] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15888.070057] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15888.070435] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15888.070786] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15888.070899] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15888.070940] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15888.070969] mdc_intent_lock+0x353/0x7e0 [mdc] [15888.070996] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15888.071135] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15888.071421] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15888.071451] lmv_intent_open+0x338/0x1150 [lmv] [15888.071483] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15888.097657] ? __hrtimer_run_queues+0x1c2/0x4d0 [15888.097677] ? watchdog+0x40/0x40 [15888.097683] ? hrtimer_interrupt+0x128/0x2c0 [15888.097688] ? smp_apic_timer_interrupt+0xa6/0x270 [15888.097698] ? apic_timer_interrupt+0xf/0x20 [15888.097703] [15888.097708] ? native_safe_halt+0x2e/0x50 [15888.097712] kvm_wait+0xcd/0xe0 [15888.097721] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15888.097730] do_raw_spin_lock+0x177/0x190 [15888.097735] _raw_spin_lock+0x21/0x30 [15888.097739] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15888.107184] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15888.107479] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15888.107512] ll_intent_lock+0xc5/0xaa0 [lustre] [15888.107753] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15888.107841] ll_lookup_it+0xe0e/0x2270 [lustre] [15888.107929] ? do_raw_spin_unlock+0x75/0x190 [15888.107937] ? _raw_spin_unlock+0x12/0x30 [15888.107943] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15888.107970] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15888.107995] ll_atomic_open+0x36a/0xd70 [lustre] [15888.109133] lookup_open+0x356/0xb40 [15888.109144] do_last+0x313/0xfc0 [15888.109150] ? nd_jump_root+0xe5/0x160 [15888.109155] ? path_init+0x437/0x520 [15888.109160] path_openat+0xf7/0x500 [15888.109165] ? do_raw_spin_unlock+0x75/0x190 [15888.109171] do_filp_open+0x99/0x140 [15888.109179] ? getname_flags+0x6e/0x330 [15888.109184] ? __check_object_size+0xff/0x256 [15888.109388] ? do_raw_spin_unlock+0x75/0x190 [15888.109393] ? _raw_spin_unlock+0x12/0x30 [15888.109399] do_sys_openat2+0x2b4/0x410 [15888.109404] do_sys_open+0x73/0xa0 [15888.109408] __x64_sys_openat+0x24/0x30 [15888.109413] do_syscall_64+0xc1/0x440 [15888.109419] entry_SYSCALL_64_after_hwframe+0x49/0xae [15888.109426] RIP: 0033:0x7f53425a5332 [15888.109435] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15888.109440] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15888.109446] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15888.109449] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15888.109451] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15888.109454] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15888.109456] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15888.520715] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15888.523030] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15888.525562] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15888.528449] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15888.530954] ? woken_wake_function+0x30/0x30 [15888.532704] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15888.535839] kthread+0x1d1/0x200 [15888.537784] ? set_kthread_struct+0x70/0x70 [15888.539615] ret_from_fork+0x1f/0x30 [15912.053210] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [15912.060555] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15912.093328] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15912.103734] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15912.109761] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15912.118797] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15912.123468] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15912.151665] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15912.161967] RAX: 0000000000007306 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15912.169027] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15912.173911] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15912.181842] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15912.198060] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15912.204179] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15912.212090] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15912.214896] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15912.219570] Call Trace: [15912.220993] [15912.222608] ? show_regs.cold.9+0x22/0x2f [15912.225993] ? watchdog_timer_fn.cold.6+0x62/0x10a [15912.229571] ? __hrtimer_run_queues+0x1c2/0x4d0 [15912.232630] ? watchdog+0x40/0x40 [15912.235435] ? hrtimer_interrupt+0x128/0x2c0 [15912.237771] ? smp_apic_timer_interrupt+0xa6/0x270 [15912.243511] ? apic_timer_interrupt+0xf/0x20 [15912.249079] [15912.250507] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15912.253950] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15912.256745] ? newidle_balance.isra.93+0x207/0x580 [15912.259447] do_raw_spin_lock+0x177/0x190 [15912.261279] _raw_spin_lock+0x21/0x30 [15912.263091] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15912.266680] process_one_work+0x2c8/0x700 [15912.268788] worker_thread+0x296/0x6e0 [15912.270926] ? rescuer_thread+0x570/0x570 [15912.278873] kthread+0x1d1/0x200 [15912.283620] ? set_kthread_struct+0x70/0x70 [15912.292874] ret_from_fork+0x1f/0x30 [15916.055871] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [15916.057215] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [15916.060227] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [15916.060244] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15916.060332] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15916.060338] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15916.060341] RIP: 0010:native_safe_halt+0x2e/0x50 [15916.060354] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15916.060362] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15916.060367] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15916.060370] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15916.060372] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15916.060374] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15916.060376] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15916.060379] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15916.060385] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15916.060388] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15916.060396] Call Trace: [15916.060402] [15916.060407] ? show_regs.cold.9+0x22/0x2f [15916.060421] ? watchdog_timer_fn.cold.6+0x62/0x10a [15916.060431] ? __hrtimer_run_queues+0x1c2/0x4d0 [15916.060436] ? watchdog+0x40/0x40 [15916.060441] ? hrtimer_interrupt+0x128/0x2c0 [15916.060446] ? smp_apic_timer_interrupt+0xa6/0x270 [15916.060457] ? apic_timer_interrupt+0xf/0x20 [15916.060465] [15916.060468] ? native_safe_halt+0x2e/0x50 [15916.060472] kvm_wait+0xcd/0xe0 [15916.060480] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15916.060488] do_raw_spin_lock+0x177/0x190 [15916.060493] _raw_spin_lock+0x21/0x30 [15916.060502] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15916.060975] Modules linked in: [15916.060853] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15916.060984] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15916.062477] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15916.062486] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15916.062489] RIP: 0010:native_safe_halt+0x2e/0x50 [15916.062504] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15916.062509] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15916.062514] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15916.062517] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15916.062519] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15916.062522] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15916.062525] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15916.062529] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15916.062532] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15916.062535] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15916.062544] Call Trace: [15916.062550] [15916.062556] ? show_regs.cold.9+0x22/0x2f [15916.062568] ? watchdog_timer_fn.cold.6+0x62/0x10a [15916.062575] ? __hrtimer_run_queues+0x1c2/0x4d0 [15916.062580] ? watchdog+0x40/0x40 [15916.062584] ? hrtimer_interrupt+0x128/0x2c0 [15916.062589] ? smp_apic_timer_interrupt+0xa6/0x270 [15916.062442] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15916.062601] ? apic_timer_interrupt+0xf/0x20 [15916.062609] [15916.062612] ? native_safe_halt+0x2e/0x50 [15916.062616] kvm_wait+0xcd/0xe0 [15916.062624] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15916.062634] do_raw_spin_lock+0x177/0x190 [15916.062639] _raw_spin_lock+0x21/0x30 [15916.062806] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15916.063276] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15916.063527] ? woken_wake_function+0x30/0x30 [15916.063538] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15916.063760] kthread+0x1d1/0x200 [15916.063767] ? set_kthread_struct+0x70/0x70 [15916.063771] ret_from_fork+0x1f/0x30 [15916.062643] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15916.069591] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15916.069684] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15916.069693] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15916.069697] RIP: 0010:native_safe_halt+0x2e/0x50 [15916.069713] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [15916.069718] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [15916.069722] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15916.069725] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15916.069728] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15916.070935] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15916.070940] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15916.070944] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15916.070946] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15916.070949] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15916.070958] Call Trace: [15916.070968] [15916.070976] ? show_regs.cold.9+0x22/0x2f [15916.070991] ? watchdog_timer_fn.cold.6+0x62/0x10a [15916.070999] ? __hrtimer_run_queues+0x1c2/0x4d0 [15916.071004] ? watchdog+0x40/0x40 [15916.071008] ? hrtimer_interrupt+0x128/0x2c0 [15916.071012] ? smp_apic_timer_interrupt+0xa6/0x270 [15916.071023] ? apic_timer_interrupt+0xf/0x20 [15916.071027] [15916.071030] ? native_safe_halt+0x2e/0x50 [15916.080778] ? do_raw_spin_unlock+0x75/0x190 [15916.080810] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15916.109597] kvm_wait+0xcd/0xe0 [15916.109630] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15916.109640] do_raw_spin_lock+0x177/0x190 [15916.109645] _raw_spin_lock+0x21/0x30 [15916.109658] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15916.114999] ? do_wait_intr+0xf0/0xf0 [15916.115029] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15916.115562] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15916.115844] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15916.116129] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15916.116256] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15916.121444] obd_mod_free+0x46/0x140 [ptlrpc] [15916.121835] mdc_commit_open+0x91/0xb0 [mdc] [15916.121878] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15916.126176] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15916.136712] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15916.138514] mdc_intent_lock+0x353/0x7e0 [mdc] [15916.138547] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15916.138675] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15916.139448] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15916.139485] lmv_intent_open+0x338/0x1150 [lmv] [15916.139528] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15916.139633] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15916.139716] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15916.152305] after_reply+0xe2c/0x1780 [ptlrpc] [15916.158515] ll_intent_lock+0xc5/0xaa0 [lustre] [15916.158668] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15916.158763] ll_lookup_it+0xe0e/0x2270 [lustre] [15916.158860] ? do_raw_spin_unlock+0x75/0x190 [15916.158872] ? _raw_spin_unlock+0x12/0x30 [15916.158881] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15916.158918] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15916.158939] ll_atomic_open+0x36a/0xd70 [lustre] [15916.159025] lookup_open+0x356/0xb40 [15916.159036] do_last+0x313/0xfc0 [15916.159042] ? nd_jump_root+0xe5/0x160 [15916.159047] ? path_init+0x437/0x520 [15916.159051] path_openat+0xf7/0x500 [15916.159056] ? do_raw_spin_unlock+0x75/0x190 [15916.159061] do_filp_open+0x99/0x140 [15916.159067] ? getname_flags+0x6e/0x330 [15916.159071] ? __check_object_size+0xff/0x256 [15916.159079] ? do_raw_spin_unlock+0x75/0x190 [15916.159083] ? _raw_spin_unlock+0x12/0x30 [15916.159088] do_sys_openat2+0x2b4/0x410 [15916.159093] do_sys_open+0x73/0xa0 [15916.159097] __x64_sys_openat+0x24/0x30 [15916.159101] do_syscall_64+0xc1/0x440 [15916.159108] entry_SYSCALL_64_after_hwframe+0x49/0xae [15916.159116] RIP: 0033:0x7f53425a5332 [15916.159125] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15916.159129] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15916.159135] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15916.159138] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15916.159141] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15916.159143] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15916.159146] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15917.106574] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15917.109603] ? __next_timer_interrupt+0x160/0x160 [15917.112343] ptlrpcd+0xa82/0xe40 [ptlrpc] [15917.118549] ? do_wait_intr+0xf0/0xf0 [15917.127330] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15917.133091] kthread+0x1d1/0x200 [15917.135172] ? set_kthread_struct+0x70/0x70 [15917.138134] ret_from_fork+0x1f/0x30 [15918.051033] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13280s! [15918.063231] Showing busy workqueues and worker pools: [15918.068035] workqueue events: flags=0x0 [15918.070146] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15918.074074] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15918.083186] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15918.091968] pending: cpu_latency_work [ptlrpc] [15918.095289] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15918.102852] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15918.118247] workqueue events_long: flags=0x0 [15918.121643] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15918.125185] pending: lu_objects_flush [obdclass] [15918.128184] workqueue events_unbound: flags=0x2 [15918.133836] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15918.136854] pending: flush_memcg_stats_dwork, idle_cull_fn [15918.142912] workqueue events_power_efficient: flags=0x80 [15918.148255] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15918.153406] pending: check_lifetime, do_cache_clean [15918.156894] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15918.162831] pending: neigh_periodic_work, neigh_periodic_work [15918.165629] workqueue mm_percpu_wq: flags=0x8 [15918.169387] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15918.175489] pending: vmstat_update, lru_add_drain_per_cpu [15918.181204] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15918.186243] pending: vmstat_update, lru_add_drain_per_cpu [15918.191656] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15918.196956] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15918.203635] workqueue xprtiod: flags=0xa [15918.205515] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15918.210570] pending: xprt_autoclose, xs_error_handle [15918.218326] workqueue ipv6_addrconf: flags=0x40008 [15918.222360] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15918.229342] pending: addrconf_verify_work [15918.232155] workqueue knbd0-recv: flags=0x1a [15918.236279] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15918.242849] in-flight: 651:recv_work [15918.246265] workqueue ptlrpc_pinger: flags=0x2 [15918.248394] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15918.253379] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15918.258430] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13281s workers=4 idle: 15979 31824 8 [15918.265783] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15908s workers=2 idle: 654 [15944.056252] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [15944.057009] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [15944.058023] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [15944.058032] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15944.058167] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15944.058172] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15944.058175] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15944.058189] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15944.058192] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15944.058196] RAX: 0000000000001515 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15944.058198] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15944.058201] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15944.058203] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15944.058206] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15944.058209] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15944.058212] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15944.058215] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15944.058224] Call Trace: [15944.058230] [15944.058235] ? show_regs.cold.9+0x22/0x2f [15944.058257] ? watchdog_timer_fn.cold.6+0x62/0x10a [15944.058263] ? __hrtimer_run_queues+0x1c2/0x4d0 [15944.058268] ? watchdog+0x40/0x40 [15944.058273] ? hrtimer_interrupt+0x128/0x2c0 [15944.058277] ? smp_apic_timer_interrupt+0xa6/0x270 [15944.058285] ? apic_timer_interrupt+0xf/0x20 [15944.058290] [15944.058293] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15944.058298] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15944.058304] do_raw_spin_lock+0x177/0x190 [15944.058310] _raw_spin_lock+0x21/0x30 [15944.058314] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15944.058655] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15944.058909] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15944.059488] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15944.059696] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15944.059977] ? woken_wake_function+0x30/0x30 [15944.059987] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15944.060305] kthread+0x1d1/0x200 [15944.060312] ? set_kthread_struct+0x70/0x70 [15944.060316] ret_from_fork+0x1f/0x30 [15944.061346] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [15944.061357] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15944.061442] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15944.061447] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15944.061454] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15944.064667] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15944.064679] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15944.064683] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15944.064686] RAX: 00000000000018d0 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15944.064688] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15944.064690] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15944.064691] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15944.064693] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15944.064696] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15944.064698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15944.064699] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15944.064707] Call Trace: [15944.064711] [15944.064716] ? show_regs.cold.9+0x22/0x2f [15944.064724] ? watchdog_timer_fn.cold.6+0x62/0x10a [15944.064730] ? __hrtimer_run_queues+0x1c2/0x4d0 [15944.064734] ? watchdog+0x40/0x40 [15944.064738] ? hrtimer_interrupt+0x128/0x2c0 [15944.064743] ? smp_apic_timer_interrupt+0xa6/0x270 [15944.064763] ? apic_timer_interrupt+0xf/0x20 [15944.067649] [15944.067660] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15944.067672] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15944.067676] ? newidle_balance.isra.93+0x207/0x580 [15944.067683] do_raw_spin_lock+0x177/0x190 [15944.067689] _raw_spin_lock+0x21/0x30 [15944.067696] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15944.068638] process_one_work+0x2c8/0x700 [15944.068647] worker_thread+0x296/0x6e0 [15944.068652] ? rescuer_thread+0x570/0x570 [15944.068656] kthread+0x1d1/0x200 [15944.068660] ? set_kthread_struct+0x70/0x70 [15944.068664] ret_from_fork+0x1f/0x30 [15944.070710] Modules linked in: [15944.074659] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15944.074751] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15944.074760] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15944.074763] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15944.074778] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15944.074783] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15944.074788] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15944.074790] RDX: 00000000000041fa RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15944.074793] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15944.074795] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15944.074798] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15944.074802] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15944.074805] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15944.074807] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15944.074823] Call Trace: [15944.074829] [15944.074835] ? show_regs.cold.9+0x22/0x2f [15944.074845] ? watchdog_timer_fn.cold.6+0x62/0x10a [15944.074852] ? __hrtimer_run_queues+0x1c2/0x4d0 [15944.074856] ? watchdog+0x40/0x40 [15944.074861] ? hrtimer_interrupt+0x128/0x2c0 [15944.074866] ? smp_apic_timer_interrupt+0xa6/0x270 [15944.074875] ? apic_timer_interrupt+0xf/0x20 [15944.074880] [15944.074884] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15944.074889] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15944.074895] do_raw_spin_lock+0x177/0x190 [15944.074900] _raw_spin_lock+0x21/0x30 [15944.074906] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15944.075169] obd_mod_free+0x46/0x140 [ptlrpc] [15944.075506] mdc_commit_open+0x91/0xb0 [mdc] [15944.075547] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15944.075735] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15944.076014] after_reply+0xe2c/0x1780 [ptlrpc] [15944.076216] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15944.077040] ? __next_timer_interrupt+0x160/0x160 [15944.077051] ptlrpcd+0xa82/0xe40 [ptlrpc] [15944.077273] ? do_wait_intr+0xf0/0xf0 [15944.077282] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15944.088823] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15944.088894] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15944.088901] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15944.088905] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15944.088919] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15944.088924] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [15944.088928] RAX: 0000000000007f86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15944.088930] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15944.088933] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15944.088934] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15944.088937] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15944.088940] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15944.088943] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15944.088946] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15944.088956] Call Trace: [15944.088962] [15944.088968] ? show_regs.cold.9+0x22/0x2f [15944.088980] ? watchdog_timer_fn.cold.6+0x62/0x10a [15944.088987] ? __hrtimer_run_queues+0x1c2/0x4d0 [15944.088991] ? watchdog+0x40/0x40 [15944.088996] ? hrtimer_interrupt+0x128/0x2c0 [15944.089000] ? smp_apic_timer_interrupt+0xa6/0x270 [15944.089009] ? apic_timer_interrupt+0xf/0x20 [15944.089013] [15944.089017] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15944.089023] do_raw_spin_lock+0x177/0x190 [15944.089027] _raw_spin_lock+0x21/0x30 [15944.089031] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15944.089438] ? do_raw_spin_unlock+0x75/0x190 [15944.089444] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15944.089641] ? do_wait_intr+0xf0/0xf0 [15944.089650] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15944.089857] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15944.090067] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15944.114672] kthread+0x1d1/0x200 [15944.114692] ? set_kthread_struct+0x70/0x70 [15944.114697] ret_from_fork+0x1f/0x30 [15945.398736] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15945.405506] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15945.407783] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15945.414096] mdc_intent_lock+0x353/0x7e0 [mdc] [15945.415692] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15945.421251] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15945.429818] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15945.438585] lmv_intent_open+0x338/0x1150 [lmv] [15945.442140] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15945.450503] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15945.452097] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15945.458678] ll_intent_lock+0xc5/0xaa0 [lustre] [15945.460993] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15945.468475] ll_lookup_it+0xe0e/0x2270 [lustre] [15945.472503] ? do_raw_spin_unlock+0x75/0x190 [15945.474032] ? _raw_spin_unlock+0x12/0x30 [15945.481518] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15945.488177] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15945.494449] ll_atomic_open+0x36a/0xd70 [lustre] [15945.503327] lookup_open+0x356/0xb40 [15945.507758] do_last+0x313/0xfc0 [15945.513686] ? nd_jump_root+0xe5/0x160 [15945.519855] ? path_init+0x437/0x520 [15945.527371] path_openat+0xf7/0x500 [15945.531780] ? do_raw_spin_unlock+0x75/0x190 [15945.540785] do_filp_open+0x99/0x140 [15945.546791] ? getname_flags+0x6e/0x330 [15945.555697] ? __check_object_size+0xff/0x256 [15945.557783] ? do_raw_spin_unlock+0x75/0x190 [15945.559690] ? _raw_spin_unlock+0x12/0x30 [15945.563371] do_sys_openat2+0x2b4/0x410 [15945.568318] do_sys_open+0x73/0xa0 [15945.570751] __x64_sys_openat+0x24/0x30 [15945.579788] do_syscall_64+0xc1/0x440 [15945.581940] entry_SYSCALL_64_after_hwframe+0x49/0xae [15945.584460] RIP: 0033:0x7f53425a5332 [15945.586872] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15945.604460] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15945.610394] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15945.617189] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15945.621677] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15945.629709] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15945.639839] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15948.768187] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13311s! [15948.780732] Showing busy workqueues and worker pools: [15948.790088] workqueue events: flags=0x0 [15948.799084] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15948.817143] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15948.846893] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15948.862491] pending: cpu_latency_work [ptlrpc] [15948.880010] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15948.898482] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15948.934694] workqueue events_long: flags=0x0 [15948.954635] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15948.974489] pending: lu_objects_flush [obdclass] [15948.987150] workqueue events_unbound: flags=0x2 [15949.000304] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15949.017370] pending: flush_memcg_stats_dwork, idle_cull_fn [15949.037723] workqueue events_power_efficient: flags=0x80 [15949.053449] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15949.071189] pending: check_lifetime, do_cache_clean [15949.094195] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15949.112487] pending: neigh_periodic_work, neigh_periodic_work [15949.131361] workqueue mm_percpu_wq: flags=0x8 [15949.136406] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15949.146898] pending: vmstat_update, lru_add_drain_per_cpu [15949.150658] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15949.161880] pending: vmstat_update, lru_add_drain_per_cpu [15949.165047] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15949.170109] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15949.176412] workqueue xprtiod: flags=0xa [15949.177956] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15949.182873] pending: xprt_autoclose, xs_error_handle [15949.185480] workqueue ipv6_addrconf: flags=0x40008 [15949.187476] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15949.190835] pending: addrconf_verify_work [15949.192351] workqueue knbd0-recv: flags=0x1a [15949.203918] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15949.224401] in-flight: 651:recv_work [15949.225796] workqueue ptlrpc_pinger: flags=0x2 [15949.231529] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15949.235602] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15949.239728] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13312s workers=4 idle: 15979 31824 8 [15949.246311] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15939s workers=2 idle: 654 [15972.055914] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [15972.062701] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [15972.065520] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [15972.065534] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15972.065611] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15972.065616] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15972.065619] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15972.065633] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15972.065637] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15972.065642] RAX: 0000000000007604 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15972.065644] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [15972.065647] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [15972.065649] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [15972.065652] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [15972.065655] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [15972.065659] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15972.065661] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [15972.065670] Call Trace: [15972.065675] [15972.065681] ? show_regs.cold.9+0x22/0x2f [15972.065690] ? watchdog_timer_fn.cold.6+0x62/0x10a [15972.065697] ? __hrtimer_run_queues+0x1c2/0x4d0 [15972.065701] ? watchdog+0x40/0x40 [15972.065705] ? hrtimer_interrupt+0x128/0x2c0 [15972.065709] ? smp_apic_timer_interrupt+0xa6/0x270 [15972.065718] ? apic_timer_interrupt+0xf/0x20 [15972.065722] [15972.065726] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15972.065731] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15972.065737] do_raw_spin_lock+0x177/0x190 [15972.065742] _raw_spin_lock+0x21/0x30 [15972.065747] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [15972.066168] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [15972.066365] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [15972.067162] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15972.067234] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15972.067241] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15972.067243] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15972.067253] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15972.067258] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15972.067263] RAX: 0000000000005e14 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15972.067265] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [15972.067268] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [15972.067270] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [15972.067273] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [15972.067276] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [15972.067280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15972.067282] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [15972.067291] Call Trace: [15972.067294] [15972.067298] ? show_regs.cold.9+0x22/0x2f [15972.067305] ? watchdog_timer_fn.cold.6+0x62/0x10a [15972.067311] ? __hrtimer_run_queues+0x1c2/0x4d0 [15972.067315] ? watchdog+0x40/0x40 [15972.067319] ? hrtimer_interrupt+0x128/0x2c0 [15972.067324] ? smp_apic_timer_interrupt+0xa6/0x270 [15972.067331] ? apic_timer_interrupt+0xf/0x20 [15972.067336] [15972.067339] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15972.067346] do_raw_spin_lock+0x177/0x190 [15972.067351] _raw_spin_lock+0x21/0x30 [15972.067356] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [15972.067596] ? do_raw_spin_unlock+0x75/0x190 [15972.067602] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [15972.067814] ? do_wait_intr+0xf0/0xf0 [15972.067823] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [15972.068031] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [15972.068506] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [15972.068281] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15972.069407] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [15972.069630] ? woken_wake_function+0x30/0x30 [15972.069638] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [15972.069857] kthread+0x1d1/0x200 [15972.069862] ? set_kthread_struct+0x70/0x70 [15972.069866] ret_from_fork+0x1f/0x30 [15972.069938] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15972.070142] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15972.070186] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [15972.070217] mdc_intent_lock+0x353/0x7e0 [mdc] [15972.070247] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15972.070339] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [15972.076227] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15972.076304] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15972.076311] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15972.076314] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [15972.076971] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [15972.076976] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15972.076980] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [15972.076983] RDX: 000000000000511d RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [15972.076985] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [15972.076987] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [15972.076989] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [15972.076992] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [15972.076995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15972.076997] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [15972.077006] Call Trace: [15972.077011] [15972.077017] ? show_regs.cold.9+0x22/0x2f [15972.077027] ? watchdog_timer_fn.cold.6+0x62/0x10a [15972.077033] ? __hrtimer_run_queues+0x1c2/0x4d0 [15972.077037] ? watchdog+0x40/0x40 [15972.077042] ? hrtimer_interrupt+0x128/0x2c0 [15972.077045] ? smp_apic_timer_interrupt+0xa6/0x270 [15972.077053] ? apic_timer_interrupt+0xf/0x20 [15972.077057] [15972.077060] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [15972.077065] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [15972.077069] do_raw_spin_lock+0x177/0x190 [15972.077073] _raw_spin_lock+0x21/0x30 [15972.077079] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [15972.077359] obd_mod_free+0x46/0x140 [ptlrpc] [15972.077742] mdc_commit_open+0x91/0xb0 [mdc] [15972.077795] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [15972.078096] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [15972.078479] after_reply+0xe2c/0x1780 [ptlrpc] [15972.078699] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [15972.085871] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [15972.085927] lmv_intent_open+0x338/0x1150 [lmv] [15972.085972] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15972.086084] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15972.086174] lmv_intent_lock+0x1dd/0x4c0 [lmv] [15972.086199] ll_intent_lock+0xc5/0xaa0 [lustre] [15972.086291] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [15972.087038] ll_lookup_it+0xe0e/0x2270 [lustre] [15972.087129] ? do_raw_spin_unlock+0x75/0x190 [15972.087139] ? _raw_spin_unlock+0x12/0x30 [15972.087146] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [15972.087169] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [15972.087189] ll_atomic_open+0x36a/0xd70 [lustre] [15972.087276] lookup_open+0x356/0xb40 [15972.087286] do_last+0x313/0xfc0 [15972.087290] ? nd_jump_root+0xe5/0x160 [15972.087295] ? path_init+0x437/0x520 [15972.087299] path_openat+0xf7/0x500 [15972.087304] ? do_raw_spin_unlock+0x75/0x190 [15972.087310] do_filp_open+0x99/0x140 [15972.087317] ? getname_flags+0x6e/0x330 [15972.087322] ? __check_object_size+0xff/0x256 [15972.087330] ? do_raw_spin_unlock+0x75/0x190 [15972.087335] ? _raw_spin_unlock+0x12/0x30 [15972.087340] do_sys_openat2+0x2b4/0x410 [15972.087357] do_sys_open+0x73/0xa0 [15972.087814] __x64_sys_openat+0x24/0x30 [15972.087823] do_syscall_64+0xc1/0x440 [15972.087831] entry_SYSCALL_64_after_hwframe+0x49/0xae [15972.087838] RIP: 0033:0x7f53425a5332 [15972.087846] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [15972.087850] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [15972.087856] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [15972.087858] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [15972.087861] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [15972.087863] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [15972.087866] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [15973.207666] ? __next_timer_interrupt+0x160/0x160 [15973.210430] ptlrpcd+0xa82/0xe40 [ptlrpc] [15973.216861] ? do_wait_intr+0xf0/0xf0 [15973.218873] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [15973.226508] kthread+0x1d1/0x200 [15973.229964] ? set_kthread_struct+0x70/0x70 [15973.237008] ret_from_fork+0x1f/0x30 [15976.052013] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [15976.062301] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [15976.106824] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [15976.113636] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [15976.120985] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [15976.124709] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15976.130621] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [15976.147354] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [15976.154196] RAX: 0000000000004bcd RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [15976.167072] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [15976.175969] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [15976.186038] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [15976.204381] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [15976.214777] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [15976.224854] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15976.236954] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [15976.253009] Call Trace: [15976.257672] [15976.260627] ? show_regs.cold.9+0x22/0x2f [15976.263021] ? watchdog_timer_fn.cold.6+0x62/0x10a [15976.269174] ? __hrtimer_run_queues+0x1c2/0x4d0 [15976.272530] ? watchdog+0x40/0x40 [15976.274598] ? hrtimer_interrupt+0x128/0x2c0 [15976.282269] ? smp_apic_timer_interrupt+0xa6/0x270 [15976.289669] ? apic_timer_interrupt+0xf/0x20 [15976.300222] [15976.300955] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [15976.305670] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [15976.308424] ? newidle_balance.isra.93+0x207/0x580 [15976.318019] do_raw_spin_lock+0x177/0x190 [15976.321471] _raw_spin_lock+0x21/0x30 [15976.333178] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [15976.348971] process_one_work+0x2c8/0x700 [15976.356059] worker_thread+0x296/0x6e0 [15976.361466] ? rescuer_thread+0x570/0x570 [15976.365614] kthread+0x1d1/0x200 [15976.367119] ? set_kthread_struct+0x70/0x70 [15976.369933] ret_from_fork+0x1f/0x30 [15979.487313] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13342s! [15979.490758] Showing busy workqueues and worker pools: [15979.498179] workqueue events: flags=0x0 [15979.502905] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [15979.512056] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [15979.530521] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15979.532796] pending: cpu_latency_work [ptlrpc] [15979.544019] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [15979.553110] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [15979.563012] workqueue events_long: flags=0x0 [15979.569492] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [15979.577224] pending: lu_objects_flush [obdclass] [15979.585048] workqueue events_unbound: flags=0x2 [15979.591970] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [15979.602328] pending: flush_memcg_stats_dwork, idle_cull_fn [15979.610770] workqueue events_power_efficient: flags=0x80 [15979.617125] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15979.628041] pending: check_lifetime, do_cache_clean [15979.637719] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15979.647049] pending: neigh_periodic_work, neigh_periodic_work [15979.655855] workqueue mm_percpu_wq: flags=0x8 [15979.662744] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15979.670490] pending: vmstat_update, lru_add_drain_per_cpu [15979.672544] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [15979.684133] pending: vmstat_update, lru_add_drain_per_cpu [15979.692125] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [15979.702825] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [15979.713329] workqueue xprtiod: flags=0xa [15979.714732] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [15979.724033] pending: xprt_autoclose, xs_error_handle [15979.736553] workqueue ipv6_addrconf: flags=0x40008 [15979.744207] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [15979.756509] pending: addrconf_verify_work [15979.758121] workqueue knbd0-recv: flags=0x1a [15979.770454] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [15979.776042] in-flight: 651:recv_work [15979.783871] workqueue ptlrpc_pinger: flags=0x2 [15979.792639] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [15979.800909] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [15979.813515] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13342s workers=4 idle: 15979 31824 8 [15979.816528] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=15970s workers=2 idle: 654 [16000.056123] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [16000.061190] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [16000.061954] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [16000.061961] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16000.062087] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16000.062092] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16000.062094] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16000.062108] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16000.062113] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16000.062117] RAX: 0000000000003a89 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16000.062120] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16000.062122] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16000.062124] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16000.062127] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16000.062130] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16000.062133] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16000.062136] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16000.062145] Call Trace: [16000.062151] [16000.062156] ? show_regs.cold.9+0x22/0x2f [16000.062164] ? watchdog_timer_fn.cold.6+0x62/0x10a [16000.062170] ? __hrtimer_run_queues+0x1c2/0x4d0 [16000.062174] ? watchdog+0x40/0x40 [16000.062178] ? hrtimer_interrupt+0x128/0x2c0 [16000.062183] ? smp_apic_timer_interrupt+0xa6/0x270 [16000.062191] ? apic_timer_interrupt+0xf/0x20 [16000.062196] [16000.062199] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16000.062204] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16000.062210] do_raw_spin_lock+0x177/0x190 [16000.062215] _raw_spin_lock+0x21/0x30 [16000.062220] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16000.062508] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16000.065010] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16000.065292] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16000.065543] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16000.065890] ? woken_wake_function+0x30/0x30 [16000.065974] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16000.066214] kthread+0x1d1/0x200 [16000.066221] ? set_kthread_struct+0x70/0x70 [16000.066225] ret_from_fork+0x1f/0x30 [16000.066693] Modules linked in: lustre(O) [16000.083918] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16000.083996] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16000.084004] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16000.084007] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16000.084020] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16000.084024] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16000.084029] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16000.084032] RDX: 0000000000006ab1 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16000.084034] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16000.084036] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16000.084038] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16000.084041] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16000.084044] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16000.084046] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16000.084056] Call Trace: [16000.084076] [16000.084083] ? show_regs.cold.9+0x22/0x2f [16000.084092] ? watchdog_timer_fn.cold.6+0x62/0x10a [16000.084098] ? __hrtimer_run_queues+0x1c2/0x4d0 [16000.084103] ? watchdog+0x40/0x40 [16000.084107] ? hrtimer_interrupt+0x128/0x2c0 [16000.084111] ? smp_apic_timer_interrupt+0xa6/0x270 [16000.095732] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16000.095815] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16000.095823] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16000.095826] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16000.095840] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16000.095845] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16000.095849] RAX: 0000000000002d8e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16000.095851] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16000.095853] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16000.095855] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16000.095858] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16000.095861] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16000.095864] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16000.095867] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16000.095876] Call Trace: [16000.095881] [16000.095886] ? show_regs.cold.9+0x22/0x2f [16000.095895] ? watchdog_timer_fn.cold.6+0x62/0x10a [16000.095902] ? __hrtimer_run_queues+0x1c2/0x4d0 [16000.095906] ? watchdog+0x40/0x40 [16000.095911] ? hrtimer_interrupt+0x128/0x2c0 [16000.095915] ? smp_apic_timer_interrupt+0xa6/0x270 [16000.095923] ? apic_timer_interrupt+0xf/0x20 [16000.095928] [16000.095931] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16000.095938] do_raw_spin_lock+0x177/0x190 [16000.095943] _raw_spin_lock+0x21/0x30 [16000.095948] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16000.150456] ? apic_timer_interrupt+0xf/0x20 [16000.150478] [16000.150484] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16000.150491] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16000.150495] do_raw_spin_lock+0x177/0x190 [16000.150501] _raw_spin_lock+0x21/0x30 [16000.150508] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16000.161762] ? do_raw_spin_unlock+0x75/0x190 [16000.161783] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16000.176134] obd_mod_free+0x46/0x140 [ptlrpc] [16000.176430] mdc_commit_open+0x91/0xb0 [mdc] [16000.176467] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16000.176659] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16000.176866] after_reply+0xe2c/0x1780 [ptlrpc] [16000.177081] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16000.177289] ? __next_timer_interrupt+0x160/0x160 [16000.177301] ptlrpcd+0xa82/0xe40 [ptlrpc] [16000.177506] ? do_wait_intr+0xf0/0xf0 [16000.177515] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16000.177924] kthread+0x1d1/0x200 [16000.177931] ? set_kthread_struct+0x70/0x70 [16000.177935] ret_from_fork+0x1f/0x30 [16000.652170] ? do_wait_intr+0xf0/0xf0 [16000.654849] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16000.659937] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16000.662066] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16000.664656] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16000.666502] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16000.672312] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16000.677026] mdc_intent_lock+0x353/0x7e0 [mdc] [16000.680435] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16000.683172] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16000.686786] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16000.691738] lmv_intent_open+0x338/0x1150 [lmv] [16000.694898] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16000.697863] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16000.700555] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16000.702944] ll_intent_lock+0xc5/0xaa0 [lustre] [16000.705042] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16000.707842] ll_lookup_it+0xe0e/0x2270 [lustre] [16000.714975] ? do_raw_spin_unlock+0x75/0x190 [16000.719785] ? _raw_spin_unlock+0x12/0x30 [16000.723396] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16000.728203] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16000.732196] ll_atomic_open+0x36a/0xd70 [lustre] [16000.733694] lookup_open+0x356/0xb40 [16000.736180] do_last+0x313/0xfc0 [16000.737515] ? nd_jump_root+0xe5/0x160 [16000.740850] ? path_init+0x437/0x520 [16000.742012] path_openat+0xf7/0x500 [16000.743202] ? do_raw_spin_unlock+0x75/0x190 [16000.748712] do_filp_open+0x99/0x140 [16000.750503] ? getname_flags+0x6e/0x330 [16000.751780] ? __check_object_size+0xff/0x256 [16000.754013] ? do_raw_spin_unlock+0x75/0x190 [16000.756231] ? _raw_spin_unlock+0x12/0x30 [16000.758058] do_sys_openat2+0x2b4/0x410 [16000.761018] do_sys_open+0x73/0xa0 [16000.765792] __x64_sys_openat+0x24/0x30 [16000.767854] do_syscall_64+0xc1/0x440 [16000.770049] entry_SYSCALL_64_after_hwframe+0x49/0xae [16000.772889] RIP: 0033:0x7f53425a5332 [16000.780428] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16000.805851] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16000.822302] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16000.829854] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16000.841754] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16000.853013] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16000.865517] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16004.052020] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [16004.054131] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16004.067153] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16004.070425] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16004.073026] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16004.075258] RIP: 0010:native_safe_halt+0x2e/0x50 [16004.076707] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16004.082185] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16004.084212] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16004.086395] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [16004.088824] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16004.090827] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16004.093052] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16004.095172] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16004.097681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16004.099255] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16004.101263] Call Trace: [16004.101956] [16004.102471] ? show_regs.cold.9+0x22/0x2f [16004.103642] ? watchdog_timer_fn.cold.6+0x62/0x10a [16004.104919] ? __hrtimer_run_queues+0x1c2/0x4d0 [16004.106176] ? watchdog+0x40/0x40 [16004.107071] ? hrtimer_interrupt+0x128/0x2c0 [16004.108294] ? smp_apic_timer_interrupt+0xa6/0x270 [16004.109571] ? apic_timer_interrupt+0xf/0x20 [16004.110749] [16004.111409] ? native_safe_halt+0x2e/0x50 [16004.112611] kvm_wait+0xcd/0xe0 [16004.113570] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16004.118135] ? newidle_balance.isra.93+0x207/0x580 [16004.120126] do_raw_spin_lock+0x177/0x190 [16004.122002] _raw_spin_lock+0x21/0x30 [16004.124128] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16004.126948] process_one_work+0x2c8/0x700 [16004.128560] worker_thread+0x296/0x6e0 [16004.132360] ? rescuer_thread+0x570/0x570 [16004.133813] kthread+0x1d1/0x200 [16004.135316] ? set_kthread_struct+0x70/0x70 [16004.138032] ret_from_fork+0x1f/0x30 [16010.218382] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13373s! [16010.224749] Showing busy workqueues and worker pools: [16010.228205] workqueue events: flags=0x0 [16010.230909] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16010.244004] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16010.260367] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16010.269277] pending: cpu_latency_work [ptlrpc] [16010.273804] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16010.277463] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16010.282791] workqueue events_long: flags=0x0 [16010.287345] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16010.293040] pending: lu_objects_flush [obdclass] [16010.302161] workqueue events_unbound: flags=0x2 [16010.307713] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16010.310376] pending: flush_memcg_stats_dwork, idle_cull_fn [16010.316426] workqueue events_power_efficient: flags=0x80 [16010.321656] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16010.328556] pending: check_lifetime, do_cache_clean [16010.332755] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16010.338978] pending: neigh_periodic_work, neigh_periodic_work [16010.344475] workqueue mm_percpu_wq: flags=0x8 [16010.345918] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16010.349186] pending: vmstat_update, lru_add_drain_per_cpu [16010.363674] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16010.366364] pending: vmstat_update, lru_add_drain_per_cpu [16010.372985] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16010.380553] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16010.384070] workqueue xprtiod: flags=0xa [16010.386259] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16010.389171] pending: xprt_autoclose, xs_error_handle [16010.391334] workqueue ipv6_addrconf: flags=0x40008 [16010.393596] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16010.396399] pending: addrconf_verify_work [16010.398877] workqueue knbd0-recv: flags=0x1a [16010.400594] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16010.403782] in-flight: 651:recv_work [16010.405349] workqueue ptlrpc_pinger: flags=0x2 [16010.407139] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16010.409830] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16010.412388] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13373s workers=4 idle: 15979 31824 8 [16010.418509] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16000s workers=2 idle: 654 [16014.777033] rcu: INFO: rcu_sched self-detected stall on CPU [16014.779617] rcu: 2-....: (7736668 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [16014.784816] (t=13920351 jiffies g=274185 q=67532) [16014.788877] rcu: rcu_sched kthread starved for 13378717 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [16014.795820] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [16014.802494] rcu: RCU grace-period kthread stack dump: [16014.805782] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [16014.814559] Call Trace: [16014.816602] __schedule+0x351/0xcb0 [16014.818496] schedule+0xc0/0x180 [16014.820379] schedule_timeout+0xb4/0x190 [16014.825496] ? do_raw_spin_unlock+0x75/0x190 [16014.830762] ? __next_timer_interrupt+0x160/0x160 [16014.834057] ? prepare_to_swait_event+0xf3/0x140 [16014.837389] rcu_gp_kthread+0x8dd/0x15b0 [16014.839894] ? force_qs_rnp+0x2c0/0x2c0 [16014.842719] kthread+0x1d1/0x200 [16014.845461] ? set_kthread_struct+0x70/0x70 [16014.851810] ret_from_fork+0x1f/0x30 [16014.854575] rcu: Stack dump where RCU GP kthread last ran: [16014.857723] NMI backtrace for cpu 2 [16014.859851] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16014.871316] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16014.877790] Call Trace: [16014.878939] [16014.881397] ? dump_stack+0xbb/0x10e [16014.888763] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16014.890514] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16014.893436] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16014.897456] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16014.907399] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16014.911725] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [16014.916276] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [16014.919466] ? update_process_times+0x6d/0xc0 [16014.922913] ? tick_sched_handle.isra.18+0x27/0x90 [16014.926891] ? tick_sched_timer+0x53/0xb0 [16014.931751] ? __hrtimer_run_queues+0x1c2/0x4d0 [16014.934608] ? tick_sched_handle.isra.18+0x90/0x90 [16014.941317] ? hrtimer_interrupt+0x128/0x2c0 [16014.947581] ? smp_apic_timer_interrupt+0xa6/0x270 [16014.949603] ? apic_timer_interrupt+0xf/0x20 [16014.952270] [16014.953454] ? native_safe_halt+0x2e/0x50 [16014.957439] ? kvm_wait+0xcd/0xe0 [16014.959436] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16014.962892] ? do_raw_spin_lock+0x177/0x190 [16014.965719] ? _raw_spin_lock+0x21/0x30 [16014.969383] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16014.975560] ? obd_mod_free+0x46/0x140 [ptlrpc] [16014.980285] ? mdc_commit_open+0x91/0xb0 [mdc] [16014.982676] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16014.986525] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16014.991919] ? after_reply+0xe2c/0x1780 [ptlrpc] [16014.995870] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16015.002803] ? __next_timer_interrupt+0x160/0x160 [16015.004565] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16015.007717] ? do_wait_intr+0xf0/0xf0 [16015.011814] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16015.018322] ? kthread+0x1d1/0x200 [16015.020679] ? set_kthread_struct+0x70/0x70 [16015.024635] ? ret_from_fork+0x1f/0x30 [16015.025939] Sending NMI from CPU 2 to CPUs 1: [16015.028991] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [16015.033658] NMI backtrace for cpu 2 [16015.042502] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16015.049515] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16015.057505] Call Trace: [16015.058368] [16015.060915] ? dump_stack+0xbb/0x10e [16015.062177] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16015.063941] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16015.069944] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16015.074807] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16015.079820] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16015.083364] ? rcu_dump_cpu_stacks+0x11b/0x171 [16015.085736] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [16015.099956] ? update_process_times+0x6d/0xc0 [16015.103495] ? tick_sched_handle.isra.18+0x27/0x90 [16015.111829] ? tick_sched_timer+0x53/0xb0 [16015.119505] ? __hrtimer_run_queues+0x1c2/0x4d0 [16015.126175] ? tick_sched_handle.isra.18+0x90/0x90 [16015.130934] ? hrtimer_interrupt+0x128/0x2c0 [16015.133995] ? smp_apic_timer_interrupt+0xa6/0x270 [16015.136937] ? apic_timer_interrupt+0xf/0x20 [16015.139852] [16015.141910] ? native_safe_halt+0x2e/0x50 [16015.152288] ? kvm_wait+0xcd/0xe0 [16015.155858] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16015.159503] ? do_raw_spin_lock+0x177/0x190 [16015.160982] ? _raw_spin_lock+0x21/0x30 [16015.173856] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16015.181738] ? obd_mod_free+0x46/0x140 [ptlrpc] [16015.186334] ? mdc_commit_open+0x91/0xb0 [mdc] [16015.192721] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16015.202636] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16015.204815] ? after_reply+0xe2c/0x1780 [ptlrpc] [16015.213561] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16015.223896] ? __next_timer_interrupt+0x160/0x160 [16015.225502] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16015.234397] ? do_wait_intr+0xf0/0xf0 [16015.238540] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16015.245517] ? kthread+0x1d1/0x200 [16015.247951] ? set_kthread_struct+0x70/0x70 [16015.253498] ? ret_from_fork+0x1f/0x30 [16028.061495] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [16028.077091] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16028.120334] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16028.127591] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16028.130937] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16028.135458] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16028.147075] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16028.150505] RAX: 0000000000007f17 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16028.153326] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16028.155558] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16028.158481] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16028.161292] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16028.165049] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16028.167968] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16028.171156] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16028.174089] Call Trace: [16028.174948] [16028.175611] ? show_regs.cold.9+0x22/0x2f [16028.177141] ? watchdog_timer_fn.cold.6+0x62/0x10a [16028.178669] ? __hrtimer_run_queues+0x1c2/0x4d0 [16028.180219] ? watchdog+0x40/0x40 [16028.182601] ? hrtimer_interrupt+0x128/0x2c0 [16028.184214] ? smp_apic_timer_interrupt+0xa6/0x270 [16028.186186] ? apic_timer_interrupt+0xf/0x20 [16028.187664] [16028.188532] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16028.190690] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16028.194108] do_raw_spin_lock+0x177/0x190 [16028.195233] _raw_spin_lock+0x21/0x30 [16028.196527] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16028.198845] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16028.201115] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16028.203555] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16028.205844] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16028.207986] ? woken_wake_function+0x30/0x30 [16028.209463] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16028.211892] kthread+0x1d1/0x200 [16028.213353] ? set_kthread_struct+0x70/0x70 [16028.214975] ret_from_fork+0x1f/0x30 [16036.053005] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [16036.061585] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16036.113028] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16036.127596] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16036.139808] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16036.145715] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16036.150505] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16036.169267] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16036.174989] RAX: 000000000000033a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16036.177380] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16036.191477] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16036.200452] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16036.206759] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16036.217505] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16036.224258] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16036.226137] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16036.234344] Call Trace: [16036.235100] [16036.239884] ? show_regs.cold.9+0x22/0x2f [16036.243379] ? watchdog_timer_fn.cold.6+0x62/0x10a [16036.248778] ? __hrtimer_run_queues+0x1c2/0x4d0 [16036.254573] ? watchdog+0x40/0x40 [16036.258434] ? hrtimer_interrupt+0x128/0x2c0 [16036.263585] ? smp_apic_timer_interrupt+0xa6/0x270 [16036.265595] ? apic_timer_interrupt+0xf/0x20 [16036.272290] [16036.274463] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16036.281503] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16036.291211] ? newidle_balance.isra.93+0x207/0x580 [16036.297506] do_raw_spin_lock+0x177/0x190 [16036.298762] _raw_spin_lock+0x21/0x30 [16036.303936] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16036.305865] process_one_work+0x2c8/0x700 [16036.315199] worker_thread+0x296/0x6e0 [16036.318591] ? rescuer_thread+0x570/0x570 [16036.327023] kthread+0x1d1/0x200 [16036.328078] ? set_kthread_struct+0x70/0x70 [16036.329441] ret_from_fork+0x1f/0x30 [16040.057905] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [16040.057963] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [16040.065040] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16040.065121] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16040.065128] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16040.065136] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16040.065153] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16040.065158] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16040.065163] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16040.065165] RDX: 000000000000026f RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16040.065168] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16040.065170] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16040.065172] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16040.065175] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16040.065177] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16040.065179] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16040.065188] Call Trace: [16040.065193] [16040.065199] ? show_regs.cold.9+0x22/0x2f [16040.065209] ? watchdog_timer_fn.cold.6+0x62/0x10a [16040.065216] ? __hrtimer_run_queues+0x1c2/0x4d0 [16040.065220] ? watchdog+0x40/0x40 [16040.065224] ? hrtimer_interrupt+0x128/0x2c0 [16040.065229] ? smp_apic_timer_interrupt+0xa6/0x270 [16040.065238] ? apic_timer_interrupt+0xf/0x20 [16040.065243] [16040.065246] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16040.065256] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16040.065265] do_raw_spin_lock+0x177/0x190 [16040.065270] _raw_spin_lock+0x21/0x30 [16040.065277] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16040.065558] obd_mod_free+0x46/0x140 [ptlrpc] [16040.065809] mdc_commit_open+0x91/0xb0 [mdc] [16040.065844] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16040.066054] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16040.066485] after_reply+0xe2c/0x1780 [ptlrpc] [16040.066958] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16040.067314] ? __next_timer_interrupt+0x160/0x160 [16040.067324] ptlrpcd+0xa82/0xe40 [ptlrpc] [16040.067546] ? do_wait_intr+0xf0/0xf0 [16040.067556] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16040.074456] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16040.074534] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16040.074542] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16040.074545] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16040.074647] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16040.074653] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16040.074658] RAX: 0000000000003899 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16040.074661] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16040.074663] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16040.074665] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16040.074667] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16040.074671] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16040.074674] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16040.074676] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16040.074684] Call Trace: [16040.074690] [16040.074695] ? show_regs.cold.9+0x22/0x2f [16040.074705] ? watchdog_timer_fn.cold.6+0x62/0x10a [16040.074711] ? __hrtimer_run_queues+0x1c2/0x4d0 [16040.074716] ? watchdog+0x40/0x40 [16040.074720] ? hrtimer_interrupt+0x128/0x2c0 [16040.074724] ? smp_apic_timer_interrupt+0xa6/0x270 [16040.074732] ? apic_timer_interrupt+0xf/0x20 [16040.074737] [16040.074740] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16040.074747] do_raw_spin_lock+0x177/0x190 [16040.074752] _raw_spin_lock+0x21/0x30 [16040.074757] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16040.075045] ? do_raw_spin_unlock+0x75/0x190 [16040.075051] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16040.075416] ? do_wait_intr+0xf0/0xf0 [16040.075426] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16040.076140] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16040.076515] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16040.076953] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16040.077072] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16040.077112] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16040.077142] mdc_intent_lock+0x353/0x7e0 [mdc] [16040.106757] kthread+0x1d1/0x200 [16040.106775] ? set_kthread_struct+0x70/0x70 [16040.106780] ret_from_fork+0x1f/0x30 [16040.437792] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16040.442115] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16040.453803] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16040.457245] lmv_intent_open+0x338/0x1150 [lmv] [16040.459776] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16040.466526] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16040.469803] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16040.472270] ll_intent_lock+0xc5/0xaa0 [lustre] [16040.478175] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16040.481058] ll_lookup_it+0xe0e/0x2270 [lustre] [16040.484021] ? do_raw_spin_unlock+0x75/0x190 [16040.485312] ? _raw_spin_unlock+0x12/0x30 [16040.491903] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16040.494475] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16040.499820] ll_atomic_open+0x36a/0xd70 [lustre] [16040.506060] lookup_open+0x356/0xb40 [16040.508000] do_last+0x313/0xfc0 [16040.511033] ? nd_jump_root+0xe5/0x160 [16040.516132] ? path_init+0x437/0x520 [16040.520448] path_openat+0xf7/0x500 [16040.526364] ? do_raw_spin_unlock+0x75/0x190 [16040.527684] do_filp_open+0x99/0x140 [16040.532284] ? getname_flags+0x6e/0x330 [16040.538382] ? __check_object_size+0xff/0x256 [16040.541077] ? do_raw_spin_unlock+0x75/0x190 [16040.542965] ? _raw_spin_unlock+0x12/0x30 [16040.547887] do_sys_openat2+0x2b4/0x410 [16040.550300] do_sys_open+0x73/0xa0 [16040.552459] __x64_sys_openat+0x24/0x30 [16040.554011] do_syscall_64+0xc1/0x440 [16040.558643] entry_SYSCALL_64_after_hwframe+0x49/0xae [16040.561854] RIP: 0033:0x7f53425a5332 [16040.565512] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16040.575604] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16040.583059] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16040.586454] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16040.589493] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16040.596414] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16040.602519] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16040.937372] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13403s! [16040.945315] Showing busy workqueues and worker pools: [16040.947974] workqueue events: flags=0x0 [16040.950195] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16040.954125] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16040.969158] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16040.981977] pending: cpu_latency_work [ptlrpc] [16040.985069] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16040.991759] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16041.006196] workqueue events_long: flags=0x0 [16041.010853] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16041.018618] pending: lu_objects_flush [obdclass] [16041.021028] workqueue events_unbound: flags=0x2 [16041.022808] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16041.028085] pending: flush_memcg_stats_dwork, idle_cull_fn [16041.031168] workqueue events_power_efficient: flags=0x80 [16041.036031] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16041.038714] pending: check_lifetime, do_cache_clean [16041.040146] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16041.045247] pending: neigh_periodic_work, neigh_periodic_work [16041.049912] workqueue mm_percpu_wq: flags=0x8 [16041.051749] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16041.055893] pending: vmstat_update, lru_add_drain_per_cpu [16041.062216] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16041.065120] pending: vmstat_update, lru_add_drain_per_cpu [16041.072015] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16041.078034] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16041.089397] workqueue xprtiod: flags=0xa [16041.091068] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16041.095886] pending: xprt_autoclose, xs_error_handle [16041.098160] workqueue ipv6_addrconf: flags=0x40008 [16041.103013] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16041.106880] pending: addrconf_verify_work [16041.108478] workqueue knbd0-recv: flags=0x1a [16041.115808] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16041.126531] in-flight: 651:recv_work [16041.130148] workqueue ptlrpc_pinger: flags=0x2 [16041.134338] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16041.139401] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16041.143250] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13404s workers=4 idle: 15979 31824 8 [16041.150210] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16031s workers=2 idle: 654 [16056.058105] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16056.063967] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16056.085902] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16056.089960] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16056.094735] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16056.097363] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16056.112835] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16056.119807] RAX: 00000000000034ac RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16056.124651] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16056.131468] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16056.139234] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16056.140972] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16056.145097] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16056.148925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16056.151639] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16056.154380] Call Trace: [16056.155091] [16056.155791] ? show_regs.cold.9+0x22/0x2f [16056.157815] ? watchdog_timer_fn.cold.6+0x62/0x10a [16056.159739] ? __hrtimer_run_queues+0x1c2/0x4d0 [16056.161261] ? watchdog+0x40/0x40 [16056.162150] ? hrtimer_interrupt+0x128/0x2c0 [16056.163556] ? smp_apic_timer_interrupt+0xa6/0x270 [16056.164941] ? apic_timer_interrupt+0xf/0x20 [16056.166325] [16056.167129] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16056.169123] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16056.170872] do_raw_spin_lock+0x177/0x190 [16056.172001] _raw_spin_lock+0x21/0x30 [16056.173406] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16056.176381] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16056.179066] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16056.181869] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16056.185386] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16056.187603] ? woken_wake_function+0x30/0x30 [16056.189148] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16056.191695] kthread+0x1d1/0x200 [16056.192717] ? set_kthread_struct+0x70/0x70 [16056.193953] ret_from_fork+0x1f/0x30 [16068.052778] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [16068.057019] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [16068.057704] Modules linked in: [16068.058169] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [16068.058176] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16068.058249] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16068.058255] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16068.058258] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16068.058272] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16068.058277] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16068.058281] RAX: 0000000000003d54 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16068.058284] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16068.058287] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16068.058289] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16068.058291] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16068.058294] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16068.058304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16068.058307] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16068.058314] Call Trace: [16068.058320] [16068.058326] ? show_regs.cold.9+0x22/0x2f [16068.058340] ? watchdog_timer_fn.cold.6+0x62/0x10a [16068.058347] ? __hrtimer_run_queues+0x1c2/0x4d0 [16068.058351] ? watchdog+0x40/0x40 [16068.058356] ? hrtimer_interrupt+0x128/0x2c0 [16068.058360] ? smp_apic_timer_interrupt+0xa6/0x270 [16068.058367] ? apic_timer_interrupt+0xf/0x20 [16068.058372] [16068.058375] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16068.058382] do_raw_spin_lock+0x177/0x190 [16068.058387] _raw_spin_lock+0x21/0x30 [16068.058392] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16068.058985] ? do_raw_spin_unlock+0x75/0x190 [16068.058992] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16068.059339] ? do_wait_intr+0xf0/0xf0 [16068.059348] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16068.059563] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16068.060433] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) [16068.061980] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16068.062055] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16068.062063] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16068.062107] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16068.062070] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16068.063003] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16068.063214] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16068.063264] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16068.063293] mdc_intent_lock+0x353/0x7e0 [mdc] [16068.063324] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16068.063419] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16068.063901] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16068.063939] lmv_intent_open+0x338/0x1150 [lmv] [16068.063970] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16068.064085] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16068.064188] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16068.064216] ll_intent_lock+0xc5/0xaa0 [lustre] [16068.064311] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16068.064398] ll_lookup_it+0xe0e/0x2270 [lustre] [16068.064498] ? do_raw_spin_unlock+0x75/0x190 [16068.064507] ? _raw_spin_unlock+0x12/0x30 [16068.064513] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16068.064542] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16068.064559] ll_atomic_open+0x36a/0xd70 [lustre] [16068.064642] lookup_open+0x356/0xb40 [16068.064769] do_last+0x313/0xfc0 [16068.064778] ? nd_jump_root+0xe5/0x160 [16068.064783] ? path_init+0x437/0x520 [16068.064788] path_openat+0xf7/0x500 [16068.064793] ? do_raw_spin_unlock+0x75/0x190 [16068.064799] do_filp_open+0x99/0x140 [16068.064805] ? getname_flags+0x6e/0x330 [16068.064810] ? __check_object_size+0xff/0x256 [16068.064817] ? do_raw_spin_unlock+0x75/0x190 [16068.064822] ? _raw_spin_unlock+0x12/0x30 [16068.064827] do_sys_openat2+0x2b4/0x410 [16068.064831] do_sys_open+0x73/0xa0 [16068.064835] __x64_sys_openat+0x24/0x30 [16068.064844] do_syscall_64+0xc1/0x440 [16068.064852] entry_SYSCALL_64_after_hwframe+0x49/0xae [16068.064858] RIP: 0033:0x7f53425a5332 [16068.064866] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16068.064870] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16068.064879] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16068.064882] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16068.064884] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16068.064886] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16068.064889] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16068.066643] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16068.066684] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16068.066690] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16068.066693] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16068.066702] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16068.066706] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16068.066710] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16068.066712] RDX: 00000000000011b3 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16068.066714] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16068.066717] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16068.066719] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16068.066722] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16068.066724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16068.066727] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16068.066734] Call Trace: [16068.066738] [16068.066742] ? show_regs.cold.9+0x22/0x2f [16068.066750] ? watchdog_timer_fn.cold.6+0x62/0x10a [16068.066756] ? __hrtimer_run_queues+0x1c2/0x4d0 [16068.066760] ? watchdog+0x40/0x40 [16068.066764] ? hrtimer_interrupt+0x128/0x2c0 [16068.066767] ? smp_apic_timer_interrupt+0xa6/0x270 [16068.066774] ? apic_timer_interrupt+0xf/0x20 [16068.066778] [16068.066781] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16068.066785] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16068.066789] do_raw_spin_lock+0x177/0x190 [16068.066793] _raw_spin_lock+0x21/0x30 [16068.066798] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16068.067093] obd_mod_free+0x46/0x140 [ptlrpc] [16068.067438] mdc_commit_open+0x91/0xb0 [mdc] [16068.067541] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16068.067783] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16068.068134] after_reply+0xe2c/0x1780 [ptlrpc] [16068.068364] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16068.068568] ? __next_timer_interrupt+0x160/0x160 [16068.068577] ptlrpcd+0xa82/0xe40 [ptlrpc] [16068.069048] ? do_wait_intr+0xf0/0xf0 [16068.069058] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16068.069333] kthread+0x1d1/0x200 [16068.069339] ? set_kthread_struct+0x70/0x70 [16068.069343] ret_from_fork+0x1f/0x30 [16068.519725] [16068.522280] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16068.523972] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16068.541388] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16068.546052] RAX: 000000000000358e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16068.548920] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16068.552392] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16068.554847] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16068.557552] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16068.560295] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16068.565877] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16068.568323] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16068.571168] Call Trace: [16068.572163] [16068.572802] ? show_regs.cold.9+0x22/0x2f [16068.574118] ? watchdog_timer_fn.cold.6+0x62/0x10a [16068.575969] ? __hrtimer_run_queues+0x1c2/0x4d0 [16068.577483] ? watchdog+0x40/0x40 [16068.578433] ? hrtimer_interrupt+0x128/0x2c0 [16068.579662] ? smp_apic_timer_interrupt+0xa6/0x270 [16068.581863] ? apic_timer_interrupt+0xf/0x20 [16068.583702] [16068.584428] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16068.586498] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16068.588289] ? newidle_balance.isra.93+0x207/0x580 [16068.589934] do_raw_spin_lock+0x177/0x190 [16068.591461] _raw_spin_lock+0x21/0x30 [16068.592655] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16068.594527] process_one_work+0x2c8/0x700 [16068.595923] worker_thread+0x296/0x6e0 [16068.597167] ? rescuer_thread+0x570/0x570 [16068.598251] kthread+0x1d1/0x200 [16068.598981] ? set_kthread_struct+0x70/0x70 [16068.600285] ret_from_fork+0x1f/0x30 [16071.648709] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13434s! [16071.651865] Showing busy workqueues and worker pools: [16071.655592] workqueue events: flags=0x0 [16071.659259] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16071.670601] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16071.680871] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16071.689208] pending: cpu_latency_work [ptlrpc] [16071.695796] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16071.699166] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16071.705628] workqueue events_long: flags=0x0 [16071.712953] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16071.718363] pending: lu_objects_flush [obdclass] [16071.723335] workqueue events_unbound: flags=0x2 [16071.725652] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16071.730986] pending: flush_memcg_stats_dwork, idle_cull_fn [16071.735644] workqueue events_power_efficient: flags=0x80 [16071.738529] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16071.741346] pending: check_lifetime, do_cache_clean [16071.744201] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16071.749101] pending: neigh_periodic_work, neigh_periodic_work [16071.751561] workqueue mm_percpu_wq: flags=0x8 [16071.752850] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16071.757359] pending: vmstat_update, lru_add_drain_per_cpu [16071.759309] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16071.762518] pending: vmstat_update, lru_add_drain_per_cpu [16071.765123] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16071.769495] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16071.779403] workqueue xprtiod: flags=0xa [16071.782752] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16071.787456] pending: xprt_autoclose, xs_error_handle [16071.792414] workqueue ipv6_addrconf: flags=0x40008 [16071.796338] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16071.800668] pending: addrconf_verify_work [16071.802556] workqueue knbd0-recv: flags=0x1a [16071.804500] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16071.807692] in-flight: 651:recv_work [16071.809773] workqueue ptlrpc_pinger: flags=0x2 [16071.812238] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16071.816470] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16071.819481] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13434s workers=4 idle: 15979 31824 8 [16071.831533] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16062s workers=2 idle: 654 [16084.058023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16084.061448] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16084.101101] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16084.111590] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16084.123682] RIP: 0010:native_safe_halt+0x2e/0x50 [16084.126549] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16084.139191] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16084.143329] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16084.148068] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16084.154467] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16084.156312] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16084.166172] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16084.171293] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16084.176434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16084.179542] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16084.181777] Call Trace: [16084.182696] [16084.183222] ? show_regs.cold.9+0x22/0x2f [16084.184556] ? watchdog_timer_fn.cold.6+0x62/0x10a [16084.186594] ? __hrtimer_run_queues+0x1c2/0x4d0 [16084.188541] ? watchdog+0x40/0x40 [16084.190147] ? hrtimer_interrupt+0x128/0x2c0 [16084.195484] ? smp_apic_timer_interrupt+0xa6/0x270 [16084.198170] ? apic_timer_interrupt+0xf/0x20 [16084.199765] [16084.201727] ? native_safe_halt+0x2e/0x50 [16084.204610] kvm_wait+0xcd/0xe0 [16084.206220] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16084.208914] do_raw_spin_lock+0x177/0x190 [16084.210608] _raw_spin_lock+0x21/0x30 [16084.213352] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16084.216410] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16084.226122] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16084.231729] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16084.239770] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16084.245921] ? woken_wake_function+0x30/0x30 [16084.247435] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16084.265397] kthread+0x1d1/0x200 [16084.268234] ? set_kthread_struct+0x70/0x70 [16084.273575] ret_from_fork+0x1f/0x30 [16096.054959] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [16096.056093] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [16096.060912] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16096.060991] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16096.060998] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16096.061005] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16096.062385] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16096.062402] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16096.062407] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16096.062411] RAX: 0000000000005626 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16096.062414] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16096.062416] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16096.062418] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16096.062420] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16096.062424] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16096.062427] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16096.062430] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16096.062438] Call Trace: [16096.062444] [16096.062450] ? show_regs.cold.9+0x22/0x2f [16096.062460] ? watchdog_timer_fn.cold.6+0x62/0x10a [16096.062466] ? __hrtimer_run_queues+0x1c2/0x4d0 [16096.062471] ? watchdog+0x40/0x40 [16096.062475] ? hrtimer_interrupt+0x128/0x2c0 [16096.062479] ? smp_apic_timer_interrupt+0xa6/0x270 [16096.062488] ? apic_timer_interrupt+0xf/0x20 [16096.062492] [16096.062495] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16096.062500] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16096.062504] ? newidle_balance.isra.93+0x207/0x580 [16096.062510] do_raw_spin_lock+0x177/0x190 [16096.062515] _raw_spin_lock+0x21/0x30 [16096.062520] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16096.062740] process_one_work+0x2c8/0x700 [16096.062748] worker_thread+0x296/0x6e0 [16096.062752] ? rescuer_thread+0x570/0x570 [16096.062755] kthread+0x1d1/0x200 [16096.062759] ? set_kthread_struct+0x70/0x70 [16096.062763] ret_from_fork+0x1f/0x30 [16096.062883] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [16096.062890] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16096.062953] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16096.062958] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16096.062960] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16096.062968] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16096.062971] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16096.062975] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16096.062977] RDX: 0000000000007e8d RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16096.062979] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16096.062981] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16096.062983] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16096.062986] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16096.062989] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16096.062991] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16096.062998] Call Trace: [16096.063001] [16096.063004] ? show_regs.cold.9+0x22/0x2f [16096.063011] ? watchdog_timer_fn.cold.6+0x62/0x10a [16096.063016] ? __hrtimer_run_queues+0x1c2/0x4d0 [16096.063019] ? watchdog+0x40/0x40 [16096.063024] ? hrtimer_interrupt+0x128/0x2c0 [16096.063028] ? smp_apic_timer_interrupt+0xa6/0x270 [16096.063033] ? apic_timer_interrupt+0xf/0x20 [16096.063037] [16096.063039] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16096.063044] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16096.063048] do_raw_spin_lock+0x177/0x190 [16096.063053] _raw_spin_lock+0x21/0x30 [16096.063057] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16096.063293] obd_mod_free+0x46/0x140 [ptlrpc] [16096.081652] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16096.081969] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16096.082664] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16096.082667] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16096.082680] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16096.082684] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16096.082688] RAX: 0000000000007e16 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16096.082690] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16096.082692] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16096.082694] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16096.082697] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16096.082699] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16096.082702] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16096.082704] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16096.082713] Call Trace: [16096.082718] [16096.082724] ? show_regs.cold.9+0x22/0x2f [16096.082733] ? watchdog_timer_fn.cold.6+0x62/0x10a [16096.082739] ? __hrtimer_run_queues+0x1c2/0x4d0 [16096.082743] ? watchdog+0x40/0x40 [16096.082747] ? hrtimer_interrupt+0x128/0x2c0 [16096.082751] ? smp_apic_timer_interrupt+0xa6/0x270 [16096.082758] ? apic_timer_interrupt+0xf/0x20 [16096.082763] [16096.082765] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16096.082772] do_raw_spin_lock+0x177/0x190 [16096.082776] _raw_spin_lock+0x21/0x30 [16096.082781] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16096.083062] ? do_raw_spin_unlock+0x75/0x190 [16096.083068] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16096.083258] ? do_wait_intr+0xf0/0xf0 [16096.083267] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16096.083462] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16096.083686] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16096.083894] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16096.084008] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16096.084045] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16096.084074] mdc_intent_lock+0x353/0x7e0 [mdc] [16096.084104] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16096.084185] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16096.084398] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16096.084426] lmv_intent_open+0x338/0x1150 [lmv] [16096.084454] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16096.084542] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16096.084636] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16096.084658] ll_intent_lock+0xc5/0xaa0 [lustre] [16096.084742] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16096.084824] ll_lookup_it+0xe0e/0x2270 [lustre] [16096.084905] ? do_raw_spin_unlock+0x75/0x190 [16096.084912] ? _raw_spin_unlock+0x12/0x30 [16096.084918] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16096.084944] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16096.084963] ll_atomic_open+0x36a/0xd70 [lustre] [16096.085045] lookup_open+0x356/0xb40 [16096.085055] do_last+0x313/0xfc0 [16096.085060] ? nd_jump_root+0xe5/0x160 [16096.085064] ? path_init+0x437/0x520 [16096.085070] path_openat+0xf7/0x500 [16096.085075] ? do_raw_spin_unlock+0x75/0x190 [16096.085079] do_filp_open+0x99/0x140 [16096.085085] ? getname_flags+0x6e/0x330 [16096.085090] ? __check_object_size+0xff/0x256 [16096.085097] ? do_raw_spin_unlock+0x75/0x190 [16096.085101] ? _raw_spin_unlock+0x12/0x30 [16096.085106] do_sys_openat2+0x2b4/0x410 [16096.085110] do_sys_open+0x73/0xa0 [16096.085113] __x64_sys_openat+0x24/0x30 [16096.085117] do_syscall_64+0xc1/0x440 [16096.085123] entry_SYSCALL_64_after_hwframe+0x49/0xae [16096.085130] RIP: 0033:0x7f53425a5332 [16096.085138] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16096.085142] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16096.085147] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16096.085149] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16096.085152] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16096.085154] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16096.085156] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16096.841915] mdc_commit_open+0x91/0xb0 [mdc] [16096.843359] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16096.845312] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16096.846859] after_reply+0xe2c/0x1780 [ptlrpc] [16096.847987] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16096.849841] ? __next_timer_interrupt+0x160/0x160 [16096.851227] ptlrpcd+0xa82/0xe40 [ptlrpc] [16096.853253] ? do_wait_intr+0xf0/0xf0 [16096.854444] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16096.856431] kthread+0x1d1/0x200 [16096.857476] ? set_kthread_struct+0x70/0x70 [16096.858776] ret_from_fork+0x1f/0x30 [16101.863042] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13464s! [16101.878693] Showing busy workqueues and worker pools: [16101.880492] workqueue events: flags=0x0 [16101.881463] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16101.891784] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16101.911028] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16101.918895] pending: cpu_latency_work [ptlrpc] [16101.925387] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16101.935243] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16101.941293] workqueue events_long: flags=0x0 [16101.948366] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16101.953078] pending: lu_objects_flush [obdclass] [16101.965330] workqueue events_unbound: flags=0x2 [16101.975015] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16101.984372] pending: flush_memcg_stats_dwork, idle_cull_fn [16101.993878] workqueue events_power_efficient: flags=0x80 [16101.997931] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16102.001260] pending: check_lifetime, do_cache_clean [16102.005310] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16102.013622] pending: neigh_periodic_work, neigh_periodic_work [16102.015927] workqueue mm_percpu_wq: flags=0x8 [16102.025657] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16102.038881] pending: vmstat_update, lru_add_drain_per_cpu [16102.043958] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16102.050040] pending: vmstat_update, lru_add_drain_per_cpu [16102.052130] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16102.063846] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16102.067614] workqueue xprtiod: flags=0xa [16102.077255] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16102.081850] pending: xprt_autoclose, xs_error_handle [16102.085065] workqueue ipv6_addrconf: flags=0x40008 [16102.089880] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16102.097890] pending: addrconf_verify_work [16102.101033] workqueue knbd0-recv: flags=0x1a [16102.113736] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16102.118769] in-flight: 651:recv_work [16102.125226] workqueue ptlrpc_pinger: flags=0x2 [16102.129485] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16102.135665] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16102.142801] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13465s workers=4 idle: 15979 31824 8 [16102.149408] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16092s workers=2 idle: 654 [16112.059083] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16112.066084] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16112.090263] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16112.094983] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16112.099055] RIP: 0010:native_safe_halt+0x2e/0x50 [16112.100951] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16112.109214] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16112.112708] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16112.115666] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16112.118477] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16112.121800] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16112.124614] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16112.127256] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16112.130619] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16112.133051] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16112.135109] Call Trace: [16112.136380] [16112.137190] ? show_regs.cold.9+0x22/0x2f [16112.139513] ? watchdog_timer_fn.cold.6+0x62/0x10a [16112.141281] ? __hrtimer_run_queues+0x1c2/0x4d0 [16112.143872] ? watchdog+0x40/0x40 [16112.146198] ? hrtimer_interrupt+0x128/0x2c0 [16112.147978] ? smp_apic_timer_interrupt+0xa6/0x270 [16112.150981] ? apic_timer_interrupt+0xf/0x20 [16112.153276] [16112.154869] ? native_safe_halt+0x2e/0x50 [16112.157910] kvm_wait+0xcd/0xe0 [16112.161457] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16112.164715] do_raw_spin_lock+0x177/0x190 [16112.169107] _raw_spin_lock+0x21/0x30 [16112.173712] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16112.177653] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16112.181191] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16112.186905] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16112.189770] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16112.192563] ? woken_wake_function+0x30/0x30 [16112.194962] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16112.197423] kthread+0x1d1/0x200 [16112.198903] ? set_kthread_struct+0x70/0x70 [16112.200474] ret_from_fork+0x1f/0x30 [16124.057086] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [16124.060085] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [16124.063022] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16124.063115] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16124.063123] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16124.063126] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16124.063140] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16124.065809] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16124.065822] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16124.065915] RDX: 00000000000063e6 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16124.065919] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16124.065921] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16124.065924] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16124.065988] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16124.065993] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16124.065995] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16124.066003] Call Trace: [16124.066009] [16124.066015] ? show_regs.cold.9+0x22/0x2f [16124.066027] ? watchdog_timer_fn.cold.6+0x62/0x10a [16124.066033] ? __hrtimer_run_queues+0x1c2/0x4d0 [16124.066037] ? watchdog+0x40/0x40 [16124.066041] ? hrtimer_interrupt+0x128/0x2c0 [16124.066044] ? smp_apic_timer_interrupt+0xa6/0x270 [16124.066052] ? apic_timer_interrupt+0xf/0x20 [16124.066057] [16124.066060] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16124.066065] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16124.066069] do_raw_spin_lock+0x177/0x190 [16124.066356] _raw_spin_lock+0x21/0x30 [16124.066364] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16124.068662] obd_mod_free+0x46/0x140 [ptlrpc] [16124.068895] mdc_commit_open+0x91/0xb0 [mdc] [16124.076360] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16124.076462] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16124.076470] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16124.076473] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16124.076491] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16124.076495] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16124.076500] RAX: 000000000000781a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16124.076502] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16124.076505] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16124.076507] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16124.076509] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16124.076512] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16124.076515] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16124.076517] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16124.076527] Call Trace: [16124.076532] [16124.076538] ? show_regs.cold.9+0x22/0x2f [16124.076548] ? watchdog_timer_fn.cold.6+0x62/0x10a [16124.076554] ? __hrtimer_run_queues+0x1c2/0x4d0 [16124.076559] ? watchdog+0x40/0x40 [16124.076563] ? hrtimer_interrupt+0x128/0x2c0 [16124.076567] ? smp_apic_timer_interrupt+0xa6/0x270 [16124.076586] ? apic_timer_interrupt+0xf/0x20 [16124.076591] [16124.076594] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16124.076601] do_raw_spin_lock+0x177/0x190 [16124.076611] _raw_spin_lock+0x21/0x30 [16124.076616] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16124.077138] ? do_raw_spin_unlock+0x75/0x190 [16124.077150] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16124.077460] ? do_wait_intr+0xf0/0xf0 [16124.077469] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16124.077991] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16124.078832] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16124.079111] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16124.079289] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16124.079344] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16124.079451] mdc_intent_lock+0x353/0x7e0 [mdc] [16124.079490] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16124.080126] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16124.080411] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16124.080453] lmv_intent_open+0x338/0x1150 [lmv] [16124.080557] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16124.082268] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16124.082973] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16124.083014] ll_intent_lock+0xc5/0xaa0 [lustre] [16124.121653] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16124.122211] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16124.122961] after_reply+0xe2c/0x1780 [ptlrpc] [16124.131125] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16124.131359] ll_lookup_it+0xe0e/0x2270 [lustre] [16124.132009] ? do_raw_spin_unlock+0x75/0x190 [16124.132018] ? _raw_spin_unlock+0x12/0x30 [16124.132025] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16124.132053] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16124.133183] ll_atomic_open+0x36a/0xd70 [lustre] [16124.133503] lookup_open+0x356/0xb40 [16124.133513] do_last+0x313/0xfc0 [16124.133519] ? nd_jump_root+0xe5/0x160 [16124.133524] ? path_init+0x437/0x520 [16124.133534] path_openat+0xf7/0x500 [16124.133542] ? do_raw_spin_unlock+0x75/0x190 [16124.133551] do_filp_open+0x99/0x140 [16124.133558] ? getname_flags+0x6e/0x330 [16124.133563] ? __check_object_size+0xff/0x256 [16124.133570] ? do_raw_spin_unlock+0x75/0x190 [16124.133575] ? _raw_spin_unlock+0x12/0x30 [16124.133584] do_sys_openat2+0x2b4/0x410 [16124.133589] do_sys_open+0x73/0xa0 [16124.133597] __x64_sys_openat+0x24/0x30 [16124.133601] do_syscall_64+0xc1/0x440 [16124.133608] entry_SYSCALL_64_after_hwframe+0x49/0xae [16124.133614] RIP: 0033:0x7f53425a5332 [16124.133626] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16124.133630] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16124.133635] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16124.133638] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16124.133640] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16124.133653] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16124.133656] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16124.577862] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16124.580652] ? __next_timer_interrupt+0x160/0x160 [16124.583270] ptlrpcd+0xa82/0xe40 [ptlrpc] [16124.585140] ? do_wait_intr+0xf0/0xf0 [16124.587385] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16124.589566] kthread+0x1d1/0x200 [16124.591076] ? set_kthread_struct+0x70/0x70 [16124.593337] ret_from_fork+0x1f/0x30 [16128.055005] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [16128.058842] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16128.110632] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16128.118958] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16128.126006] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16128.131870] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16128.135803] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16128.157029] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16128.163189] RAX: 00000000000020df RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16128.168799] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16128.177201] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16128.187995] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16128.197389] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16128.205614] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16128.214471] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16128.216944] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16128.229940] Call Trace: [16128.235130] [16128.236718] ? show_regs.cold.9+0x22/0x2f [16128.244582] ? watchdog_timer_fn.cold.6+0x62/0x10a [16128.249620] ? __hrtimer_run_queues+0x1c2/0x4d0 [16128.262744] ? watchdog+0x40/0x40 [16128.271805] ? hrtimer_interrupt+0x128/0x2c0 [16128.281009] ? smp_apic_timer_interrupt+0xa6/0x270 [16128.290874] ? apic_timer_interrupt+0xf/0x20 [16128.300928] [16128.302987] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16128.325764] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16128.347823] ? newidle_balance.isra.93+0x207/0x580 [16128.349522] do_raw_spin_lock+0x177/0x190 [16128.364012] _raw_spin_lock+0x21/0x30 [16128.367367] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16128.381229] process_one_work+0x2c8/0x700 [16128.398147] worker_thread+0x296/0x6e0 [16128.402195] ? rescuer_thread+0x570/0x570 [16128.429615] kthread+0x1d1/0x200 [16128.434877] ? set_kthread_struct+0x70/0x70 [16128.448391] ret_from_fork+0x1f/0x30 [16132.575046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13495s! [16132.579684] Showing busy workqueues and worker pools: [16132.583793] workqueue events: flags=0x0 [16132.585804] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16132.589301] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16132.595400] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16132.598666] pending: cpu_latency_work [ptlrpc] [16132.601931] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16132.606402] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16132.613404] workqueue events_long: flags=0x0 [16132.615833] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16132.619812] pending: lu_objects_flush [obdclass] [16132.623143] workqueue events_unbound: flags=0x2 [16132.626490] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16132.630259] pending: flush_memcg_stats_dwork, idle_cull_fn [16132.632988] workqueue events_power_efficient: flags=0x80 [16132.636600] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16132.640452] pending: check_lifetime, do_cache_clean [16132.643491] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16132.647379] pending: neigh_periodic_work, neigh_periodic_work [16132.650491] workqueue mm_percpu_wq: flags=0x8 [16132.653038] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16132.657740] pending: vmstat_update, lru_add_drain_per_cpu [16132.661402] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16132.664908] pending: vmstat_update, lru_add_drain_per_cpu [16132.668263] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16132.671954] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16132.675425] workqueue xprtiod: flags=0xa [16132.676900] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16132.680892] pending: xprt_autoclose, xs_error_handle [16132.684399] workqueue ipv6_addrconf: flags=0x40008 [16132.687828] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16132.691460] pending: addrconf_verify_work [16132.695249] workqueue knbd0-recv: flags=0x1a [16132.697511] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16132.701209] in-flight: 651:recv_work [16132.703380] workqueue ptlrpc_pinger: flags=0x2 [16132.705605] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16132.709452] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16132.712781] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13495s workers=4 idle: 15979 31824 8 [16132.716875] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16123s workers=2 idle: 654 [16140.061667] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16140.067529] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16140.110245] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16140.125874] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16140.138452] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16140.140251] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16140.156738] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16140.160952] RAX: 0000000000000885 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16140.164749] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16140.170261] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16140.175787] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16140.179262] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16140.185488] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16140.190137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16140.195080] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16140.200661] Call Trace: [16140.202867] [16140.203689] ? show_regs.cold.9+0x22/0x2f [16140.205318] ? watchdog_timer_fn.cold.6+0x62/0x10a [16140.209896] ? __hrtimer_run_queues+0x1c2/0x4d0 [16140.212621] ? watchdog+0x40/0x40 [16140.214281] ? hrtimer_interrupt+0x128/0x2c0 [16140.218665] ? smp_apic_timer_interrupt+0xa6/0x270 [16140.222795] ? apic_timer_interrupt+0xf/0x20 [16140.226672] [16140.228954] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16140.233734] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16140.238537] do_raw_spin_lock+0x177/0x190 [16140.242101] _raw_spin_lock+0x21/0x30 [16140.243402] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16140.245624] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16140.258652] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16140.264034] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16140.269942] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16140.271886] ? woken_wake_function+0x30/0x30 [16140.275138] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16140.280122] kthread+0x1d1/0x200 [16140.283310] ? set_kthread_struct+0x70/0x70 [16140.287701] ret_from_fork+0x1f/0x30 [16152.055893] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [16152.057934] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [16152.059683] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16152.059768] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16152.059776] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16152.059778] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16152.060002] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16152.060009] RSP: 0018:ffffbb5381503500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [16152.060014] RAX: 0000000000007588 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16152.060017] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16152.060019] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16152.060021] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16152.060024] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16152.060027] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16152.060029] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16152.060031] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16152.060040] Call Trace: [16152.060046] [16152.060051] ? show_regs.cold.9+0x22/0x2f [16152.060062] ? watchdog_timer_fn.cold.6+0x62/0x10a [16152.060068] ? __hrtimer_run_queues+0x1c2/0x4d0 [16152.060072] ? watchdog+0x40/0x40 [16152.060077] ? hrtimer_interrupt+0x128/0x2c0 [16152.060081] ? smp_apic_timer_interrupt+0xa6/0x270 [16152.060090] ? apic_timer_interrupt+0xf/0x20 [16152.060095] [16152.060098] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16152.060105] do_raw_spin_lock+0x177/0x190 [16152.060110] _raw_spin_lock+0x21/0x30 [16152.060115] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16152.060468] ? do_raw_spin_unlock+0x75/0x190 [16152.060475] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16152.065378] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16152.065464] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16152.065472] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16152.065475] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16152.065489] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16152.065494] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16152.065498] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16152.065501] RDX: 0000000000001cae RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16152.065504] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16152.065506] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16152.065509] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16152.065511] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16152.065514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16152.065516] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16152.065525] Call Trace: [16152.065531] [16152.065537] ? show_regs.cold.9+0x22/0x2f [16152.065547] ? watchdog_timer_fn.cold.6+0x62/0x10a [16152.065553] ? __hrtimer_run_queues+0x1c2/0x4d0 [16152.065559] ? watchdog+0x40/0x40 [16152.065563] ? hrtimer_interrupt+0x128/0x2c0 [16152.065567] ? smp_apic_timer_interrupt+0xa6/0x270 [16152.065575] ? apic_timer_interrupt+0xf/0x20 [16152.065579] [16152.065582] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16152.065586] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16152.065590] do_raw_spin_lock+0x177/0x190 [16152.065595] _raw_spin_lock+0x21/0x30 [16152.065663] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16152.065988] obd_mod_free+0x46/0x140 [ptlrpc] [16152.066187] mdc_commit_open+0x91/0xb0 [mdc] [16152.066232] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16152.066672] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16152.066931] after_reply+0xe2c/0x1780 [ptlrpc] [16152.067124] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16152.067309] ? __next_timer_interrupt+0x160/0x160 [16152.067318] ptlrpcd+0xa82/0xe40 [ptlrpc] [16152.067513] ? do_wait_intr+0xf0/0xf0 [16152.067521] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16152.092724] ? do_wait_intr+0xf0/0xf0 [16152.092750] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16152.093022] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16152.093236] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16152.098369] kthread+0x1d1/0x200 [16152.098393] ? set_kthread_struct+0x70/0x70 [16152.098398] ret_from_fork+0x1f/0x30 [16152.418853] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16152.422643] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16152.425480] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16152.427939] mdc_intent_lock+0x353/0x7e0 [mdc] [16152.430633] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16152.433282] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16152.436499] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16152.440507] lmv_intent_open+0x338/0x1150 [lmv] [16152.442325] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16152.449311] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16152.455216] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16152.458431] ll_intent_lock+0xc5/0xaa0 [lustre] [16152.462942] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16152.469917] ll_lookup_it+0xe0e/0x2270 [lustre] [16152.472939] ? do_raw_spin_unlock+0x75/0x190 [16152.477359] ? _raw_spin_unlock+0x12/0x30 [16152.479000] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16152.480998] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16152.483508] ll_atomic_open+0x36a/0xd70 [lustre] [16152.485796] lookup_open+0x356/0xb40 [16152.487586] do_last+0x313/0xfc0 [16152.488657] ? nd_jump_root+0xe5/0x160 [16152.490481] ? path_init+0x437/0x520 [16152.493100] path_openat+0xf7/0x500 [16152.494172] ? do_raw_spin_unlock+0x75/0x190 [16152.495758] do_filp_open+0x99/0x140 [16152.497015] ? getname_flags+0x6e/0x330 [16152.498675] ? __check_object_size+0xff/0x256 [16152.500223] ? do_raw_spin_unlock+0x75/0x190 [16152.502510] ? _raw_spin_unlock+0x12/0x30 [16152.506564] do_sys_openat2+0x2b4/0x410 [16152.509168] do_sys_open+0x73/0xa0 [16152.512372] __x64_sys_openat+0x24/0x30 [16152.520350] do_syscall_64+0xc1/0x440 [16152.522483] entry_SYSCALL_64_after_hwframe+0x49/0xae [16152.528028] RIP: 0033:0x7f53425a5332 [16152.530014] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16152.541072] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16152.544564] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16152.549201] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16152.552826] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16152.556041] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16152.562867] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16160.056498] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [16160.061434] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16160.097458] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16160.105242] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16160.111835] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16160.117441] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16160.131015] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16160.145210] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16160.150733] RAX: 00000000000030db RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16160.153987] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16160.160044] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16160.163185] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16160.165270] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16160.167726] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16160.171246] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16160.176833] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16160.184131] Call Trace: [16160.185398] [16160.186702] ? show_regs.cold.9+0x22/0x2f [16160.191690] ? watchdog_timer_fn.cold.6+0x62/0x10a [16160.196260] ? __hrtimer_run_queues+0x1c2/0x4d0 [16160.198320] ? watchdog+0x40/0x40 [16160.200509] ? hrtimer_interrupt+0x128/0x2c0 [16160.202396] ? smp_apic_timer_interrupt+0xa6/0x270 [16160.205335] ? apic_timer_interrupt+0xf/0x20 [16160.206942] [16160.207454] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16160.209571] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16160.212190] ? newidle_balance.isra.93+0x207/0x580 [16160.214719] do_raw_spin_lock+0x177/0x190 [16160.216728] _raw_spin_lock+0x21/0x30 [16160.218306] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16160.220898] process_one_work+0x2c8/0x700 [16160.224041] worker_thread+0x296/0x6e0 [16160.225767] ? rescuer_thread+0x570/0x570 [16160.226803] kthread+0x1d1/0x200 [16160.228131] ? set_kthread_struct+0x70/0x70 [16160.230007] ret_from_fork+0x1f/0x30 [16162.787054] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13525s! [16162.792182] Showing busy workqueues and worker pools: [16162.794110] workqueue events: flags=0x0 [16162.795918] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16162.798619] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16162.803587] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16162.806045] pending: cpu_latency_work [ptlrpc] [16162.808567] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16162.811349] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16162.815272] workqueue events_long: flags=0x0 [16162.816859] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16162.819511] pending: lu_objects_flush [obdclass] [16162.821565] workqueue events_unbound: flags=0x2 [16162.823319] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16162.826015] pending: flush_memcg_stats_dwork, idle_cull_fn [16162.828644] workqueue events_power_efficient: flags=0x80 [16162.830865] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16162.833670] pending: check_lifetime, do_cache_clean [16162.835646] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16162.841310] pending: neigh_periodic_work, neigh_periodic_work [16162.844218] workqueue mm_percpu_wq: flags=0x8 [16162.849296] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16162.855041] pending: vmstat_update, lru_add_drain_per_cpu [16162.864602] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16162.867078] pending: vmstat_update, lru_add_drain_per_cpu [16162.871880] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16162.874871] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16162.877762] workqueue xprtiod: flags=0xa [16162.879337] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16162.882047] pending: xprt_autoclose, xs_error_handle [16162.884297] workqueue ipv6_addrconf: flags=0x40008 [16162.886356] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16162.889071] pending: addrconf_verify_work [16162.891168] workqueue knbd0-recv: flags=0x1a [16162.894933] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16162.897896] in-flight: 651:recv_work [16162.899841] workqueue ptlrpc_pinger: flags=0x2 [16162.901875] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16162.904859] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16162.907673] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13525s workers=4 idle: 15979 31824 8 [16162.911019] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16153s workers=2 idle: 654 [16168.058025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16168.060761] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16168.088128] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16168.095468] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16168.098306] RIP: 0010:native_safe_halt+0x2e/0x50 [16168.101026] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16168.111836] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16168.119723] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16168.122227] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16168.129212] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16168.133681] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16168.138112] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16168.141755] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16168.145052] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16168.147154] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16168.151235] Call Trace: [16168.153113] [16168.155237] ? show_regs.cold.9+0x22/0x2f [16168.157567] ? watchdog_timer_fn.cold.6+0x62/0x10a [16168.160122] ? __hrtimer_run_queues+0x1c2/0x4d0 [16168.161712] ? watchdog+0x40/0x40 [16168.164243] ? hrtimer_interrupt+0x128/0x2c0 [16168.166909] ? smp_apic_timer_interrupt+0xa6/0x270 [16168.169527] ? apic_timer_interrupt+0xf/0x20 [16168.171617] [16168.173380] ? native_safe_halt+0x2e/0x50 [16168.175288] kvm_wait+0xcd/0xe0 [16168.176298] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16168.182908] do_raw_spin_lock+0x177/0x190 [16168.185004] _raw_spin_lock+0x21/0x30 [16168.187704] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16168.192446] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16168.196107] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16168.200278] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16168.203047] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16168.206093] ? woken_wake_function+0x30/0x30 [16168.208398] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16168.212048] kthread+0x1d1/0x200 [16168.213571] ? set_kthread_struct+0x70/0x70 [16168.215773] ret_from_fork+0x1f/0x30 [16180.055076] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [16180.057331] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [16180.057976] Modules linked in: [16180.061628] Modules linked in: [16180.061638] lustre(O) [16180.062956] lustre(O) [16180.064475] osp(O) [16180.065749] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16180.065799] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16180.065804] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16180.065806] RIP: 0010:native_safe_halt+0x2e/0x50 [16180.065820] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16180.069060] ofd(O) [16180.069944] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 [16180.097593] lod(O) [16180.102461] ORIG_RAX: ffffffffffffff13 [16180.110888] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16180.110969] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16180.110978] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16180.110982] RIP: 0010:native_safe_halt+0x2e/0x50 [16180.110999] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16180.111003] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16180.111008] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16180.111010] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16180.111012] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16180.111015] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16180.111018] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16180.111022] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16180.111025] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16180.111028] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16180.111037] Call Trace: [16180.111046] [16180.111054] ? show_regs.cold.9+0x22/0x2f [16180.111066] ? watchdog_timer_fn.cold.6+0x62/0x10a [16180.111073] ? __hrtimer_run_queues+0x1c2/0x4d0 [16180.111077] ? watchdog+0x40/0x40 [16180.111082] ? hrtimer_interrupt+0x128/0x2c0 [16180.111086] ? smp_apic_timer_interrupt+0xa6/0x270 [16180.111093] ? apic_timer_interrupt+0xf/0x20 [16180.111098] [16180.111101] ? native_safe_halt+0x2e/0x50 [16180.111105] kvm_wait+0xcd/0xe0 [16180.111127] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16180.111138] do_raw_spin_lock+0x177/0x190 [16180.111142] _raw_spin_lock+0x21/0x30 [16180.111146] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16180.113412] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16180.113420] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16180.113422] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16180.113425] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16180.113427] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16180.113430] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16180.113433] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16180.113436] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16180.113444] Call Trace: [16180.113537] [16180.113543] ? show_regs.cold.9+0x22/0x2f [16180.113556] ? watchdog_timer_fn.cold.6+0x62/0x10a [16180.113563] ? __hrtimer_run_queues+0x1c2/0x4d0 [16180.113567] ? watchdog+0x40/0x40 [16180.113572] ? hrtimer_interrupt+0x128/0x2c0 [16180.113577] ? smp_apic_timer_interrupt+0xa6/0x270 [16180.127982] ? do_raw_spin_unlock+0x75/0x190 [16180.129238] ? apic_timer_interrupt+0xf/0x20 [16180.129258] [16180.129263] ? native_safe_halt+0x2e/0x50 [16180.129268] kvm_wait+0xcd/0xe0 [16180.129278] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16180.129286] do_raw_spin_lock+0x177/0x190 [16180.129291] _raw_spin_lock+0x21/0x30 [16180.129297] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16180.129654] obd_mod_free+0x46/0x140 [ptlrpc] [16180.129903] mdc_commit_open+0x91/0xb0 [mdc] [16180.129951] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16180.130182] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16180.130416] after_reply+0xe2c/0x1780 [ptlrpc] [16180.130647] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16180.130858] ? __next_timer_interrupt+0x160/0x160 [16180.130868] ptlrpcd+0xa82/0xe40 [ptlrpc] [16180.131089] ? do_wait_intr+0xf0/0xf0 [16180.131099] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16180.131314] kthread+0x1d1/0x200 [16180.131327] ? set_kthread_struct+0x70/0x70 [16180.131335] ret_from_fork+0x1f/0x30 [16180.512884] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16180.514466] ? do_wait_intr+0xf0/0xf0 [16180.515448] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16180.529945] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16180.531446] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16180.534734] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16180.537817] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16180.540747] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16180.542052] mdc_intent_lock+0x353/0x7e0 [mdc] [16180.546684] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16180.556401] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16180.558297] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16180.565415] lmv_intent_open+0x338/0x1150 [lmv] [16180.569023] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16180.576187] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16180.580548] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16180.585664] ll_intent_lock+0xc5/0xaa0 [lustre] [16180.588082] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16180.593290] ll_lookup_it+0xe0e/0x2270 [lustre] [16180.595450] ? do_raw_spin_unlock+0x75/0x190 [16180.603439] ? _raw_spin_unlock+0x12/0x30 [16180.609454] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16180.612457] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16180.615499] ll_atomic_open+0x36a/0xd70 [lustre] [16180.620089] lookup_open+0x356/0xb40 [16180.622460] do_last+0x313/0xfc0 [16180.623887] ? nd_jump_root+0xe5/0x160 [16180.629770] ? path_init+0x437/0x520 [16180.632264] path_openat+0xf7/0x500 [16180.635695] ? do_raw_spin_unlock+0x75/0x190 [16180.639296] do_filp_open+0x99/0x140 [16180.640893] ? getname_flags+0x6e/0x330 [16180.643867] ? __check_object_size+0xff/0x256 [16180.648433] ? do_raw_spin_unlock+0x75/0x190 [16180.652964] ? _raw_spin_unlock+0x12/0x30 [16180.657388] do_sys_openat2+0x2b4/0x410 [16180.659201] do_sys_open+0x73/0xa0 [16180.660358] __x64_sys_openat+0x24/0x30 [16180.663491] do_syscall_64+0xc1/0x440 [16180.665093] entry_SYSCALL_64_after_hwframe+0x49/0xae [16180.667231] RIP: 0033:0x7f53425a5332 [16180.669870] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16180.684426] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16180.686386] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16180.688358] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16180.689915] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16180.691968] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16180.693684] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16188.052316] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [16188.057328] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16188.085438] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16188.089927] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16188.095926] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16188.098941] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16188.101895] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16188.111500] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16188.114635] RAX: 0000000000007d8c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16188.118255] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16188.120565] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16188.123758] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16188.127734] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16188.130568] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16188.135385] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16188.138463] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16188.142282] Call Trace: [16188.143540] [16188.145211] ? show_regs.cold.9+0x22/0x2f [16188.147403] ? watchdog_timer_fn.cold.6+0x62/0x10a [16188.149806] ? __hrtimer_run_queues+0x1c2/0x4d0 [16188.152341] ? watchdog+0x40/0x40 [16188.154518] ? hrtimer_interrupt+0x128/0x2c0 [16188.156840] ? smp_apic_timer_interrupt+0xa6/0x270 [16188.159459] ? apic_timer_interrupt+0xf/0x20 [16188.161813] [16188.163031] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16188.165184] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16188.167448] ? newidle_balance.isra.93+0x207/0x580 [16188.169635] do_raw_spin_lock+0x177/0x190 [16188.171678] _raw_spin_lock+0x21/0x30 [16188.176389] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16188.180579] process_one_work+0x2c8/0x700 [16188.184292] worker_thread+0x296/0x6e0 [16188.186383] ? rescuer_thread+0x570/0x570 [16188.191419] kthread+0x1d1/0x200 [16188.194630] ? set_kthread_struct+0x70/0x70 [16188.196162] ret_from_fork+0x1f/0x30 [16192.992390] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13555s! [16192.998394] Showing busy workqueues and worker pools: [16193.001136] workqueue events: flags=0x0 [16193.003092] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16193.007650] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16193.015267] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16193.020664] pending: cpu_latency_work [ptlrpc] [16193.024294] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16193.027422] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16193.031737] workqueue events_long: flags=0x0 [16193.034337] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16193.038062] pending: lu_objects_flush [obdclass] [16193.040057] workqueue events_unbound: flags=0x2 [16193.041750] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16193.047389] pending: flush_memcg_stats_dwork, idle_cull_fn [16193.050626] workqueue events_power_efficient: flags=0x80 [16193.056805] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16193.062837] pending: check_lifetime, do_cache_clean [16193.067689] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16193.074861] pending: neigh_periodic_work, neigh_periodic_work [16193.081390] workqueue mm_percpu_wq: flags=0x8 [16193.082911] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16193.085870] pending: vmstat_update, lru_add_drain_per_cpu [16193.089975] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16193.095621] pending: vmstat_update, lru_add_drain_per_cpu [16193.102481] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16193.109279] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16193.116180] workqueue xprtiod: flags=0xa [16193.121651] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16193.130098] pending: xprt_autoclose, xs_error_handle [16193.132941] workqueue ipv6_addrconf: flags=0x40008 [16193.135728] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16193.140156] pending: addrconf_verify_work [16193.142713] workqueue knbd0-recv: flags=0x1a [16193.145140] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16193.148627] in-flight: 651:recv_work [16193.153159] workqueue ptlrpc_pinger: flags=0x2 [16193.154694] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16193.170844] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16193.181338] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13556s workers=4 idle: 15979 31824 8 [16193.195717] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16183s workers=2 idle: 654 [16194.780114] rcu: INFO: rcu_sched self-detected stall on CPU [16194.782835] rcu: 2-....: (7839675 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [16194.786395] (t=14100354 jiffies g=274185 q=67568) [16194.789861] rcu: rcu_sched kthread starved for 13558720 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [16194.803584] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [16194.809568] rcu: RCU grace-period kthread stack dump: [16194.811835] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [16194.817137] Call Trace: [16194.820231] __schedule+0x351/0xcb0 [16194.823524] schedule+0xc0/0x180 [16194.826024] schedule_timeout+0xb4/0x190 [16194.828575] ? do_raw_spin_unlock+0x75/0x190 [16194.830579] ? __next_timer_interrupt+0x160/0x160 [16194.832175] ? prepare_to_swait_event+0xf3/0x140 [16194.835201] rcu_gp_kthread+0x8dd/0x15b0 [16194.837115] ? force_qs_rnp+0x2c0/0x2c0 [16194.838745] kthread+0x1d1/0x200 [16194.840601] ? set_kthread_struct+0x70/0x70 [16194.842834] ret_from_fork+0x1f/0x30 [16194.846590] rcu: Stack dump where RCU GP kthread last ran: [16194.849821] Sending NMI from CPU 2 to CPUs 1: [16194.856282] NMI backtrace for cpu 1 [16194.856288] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16194.856291] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16194.856293] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16194.856295] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16194.856297] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [16194.856304] RAX: 0000000000002fb8 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16194.856306] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16194.856307] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16194.856312] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16194.856313] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16194.856315] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16194.856316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16194.856318] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16194.856319] Call Trace: [16194.856320] [16194.856321] ? show_regs.cold.9+0x22/0x2f [16194.856322] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [16194.856324] ? nmi_cpu_backtrace_handler+0x15/0x40 [16194.856325] ? nmi_handle+0xbd/0x210 [16194.856326] ? default_do_nmi+0x4b/0x1b0 [16194.856328] ? do_nmi+0x280/0x380 [16194.856328] ? end_repeat_nmi+0x16/0x1f [16194.856330] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16194.856331] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16194.856333] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16194.856339] [16194.856341] do_raw_spin_lock+0x177/0x190 [16194.856342] _raw_spin_lock+0x21/0x30 [16194.856343] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16194.856344] ? do_raw_spin_unlock+0x75/0x190 [16194.856346] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16194.856347] ? do_wait_intr+0xf0/0xf0 [16194.856348] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16194.856349] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16194.856351] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16194.856352] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16194.856353] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16194.856354] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16194.856356] mdc_intent_lock+0x353/0x7e0 [mdc] [16194.856440] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16194.856557] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16194.856558] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16194.856564] lmv_intent_open+0x338/0x1150 [lmv] [16194.856565] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16194.856566] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16194.856567] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16194.856569] ll_intent_lock+0xc5/0xaa0 [lustre] [16194.856571] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16194.856572] ll_lookup_it+0xe0e/0x2270 [lustre] [16194.856573] ? do_raw_spin_unlock+0x75/0x190 [16194.856574] ? _raw_spin_unlock+0x12/0x30 [16194.856575] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16194.856576] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16194.856577] ll_atomic_open+0x36a/0xd70 [lustre] [16194.856578] lookup_open+0x356/0xb40 [16194.856579] do_last+0x313/0xfc0 [16194.856580] ? nd_jump_root+0xe5/0x160 [16194.856581] ? path_init+0x437/0x520 [16194.856582] path_openat+0xf7/0x500 [16194.856586] ? do_raw_spin_unlock+0x75/0x190 [16194.856587] do_filp_open+0x99/0x140 [16194.856587] ? getname_flags+0x6e/0x330 [16194.856589] ? __check_object_size+0xff/0x256 [16194.856592] ? do_raw_spin_unlock+0x75/0x190 [16194.856594] ? _raw_spin_unlock+0x12/0x30 [16194.856595] do_sys_openat2+0x2b4/0x410 [16194.856596] do_sys_open+0x73/0xa0 [16194.856597] __x64_sys_openat+0x24/0x30 [16194.856598] do_syscall_64+0xc1/0x440 [16194.856599] entry_SYSCALL_64_after_hwframe+0x49/0xae [16194.856600] RIP: 0033:0x7f53425a5332 [16194.856601] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16194.856603] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16194.856606] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16194.856607] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16194.856608] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16194.856610] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16194.856614] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16194.856636] Sending NMI from CPU 2 to CPUs 1: [16195.138553] NMI backtrace for cpu 1 [16195.138560] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16195.138563] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16195.138564] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16195.138566] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16195.138568] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [16195.138571] RAX: 0000000000004186 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16195.138573] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16195.138574] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16195.138576] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16195.138577] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16195.138579] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16195.138580] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16195.138581] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16195.138588] Call Trace: [16195.138589] [16195.138590] ? show_regs.cold.9+0x22/0x2f [16195.138591] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [16195.138814] ? nmi_cpu_backtrace_handler+0x15/0x40 [16195.138817] ? nmi_handle+0xbd/0x210 [16195.138818] ? default_do_nmi+0x4b/0x1b0 [16195.138819] ? do_nmi+0x280/0x380 [16195.138820] ? end_repeat_nmi+0x16/0x1f [16195.138821] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16195.138822] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16195.138823] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16195.138824] [16195.138825] do_raw_spin_lock+0x177/0x190 [16195.138826] _raw_spin_lock+0x21/0x30 [16195.138827] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16195.138828] ? do_raw_spin_unlock+0x75/0x190 [16195.138829] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16195.138830] ? do_wait_intr+0xf0/0xf0 [16195.138831] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16195.138832] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16195.138833] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16195.138834] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16195.138836] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16195.138837] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16195.138838] mdc_intent_lock+0x353/0x7e0 [mdc] [16195.138839] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16195.138840] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16195.138842] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16195.138843] lmv_intent_open+0x338/0x1150 [lmv] [16195.138844] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16195.138854] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16195.138855] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16195.138856] ll_intent_lock+0xc5/0xaa0 [lustre] [16195.138857] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16195.138859] ll_lookup_it+0xe0e/0x2270 [lustre] [16195.138860] ? do_raw_spin_unlock+0x75/0x190 [16195.138861] ? _raw_spin_unlock+0x12/0x30 [16195.138862] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16195.138863] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16195.138864] ll_atomic_open+0x36a/0xd70 [lustre] [16195.138865] lookup_open+0x356/0xb40 [16195.138866] do_last+0x313/0xfc0 [16195.138867] ? nd_jump_root+0xe5/0x160 [16195.138868] ? path_init+0x437/0x520 [16195.138869] path_openat+0xf7/0x500 [16195.138870] ? do_raw_spin_unlock+0x75/0x190 [16195.138871] do_filp_open+0x99/0x140 [16195.138872] ? getname_flags+0x6e/0x330 [16195.138873] ? __check_object_size+0xff/0x256 [16195.138875] ? do_raw_spin_unlock+0x75/0x190 [16195.138876] ? _raw_spin_unlock+0x12/0x30 [16195.138877] do_sys_openat2+0x2b4/0x410 [16195.138878] do_sys_open+0x73/0xa0 [16195.138879] __x64_sys_openat+0x24/0x30 [16195.138880] do_syscall_64+0xc1/0x440 [16195.138922] entry_SYSCALL_64_after_hwframe+0x49/0xae [16195.138923] RIP: 0033:0x7f53425a5332 [16195.138924] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16195.138927] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16195.138930] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16195.138931] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16195.138932] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16195.138934] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16195.138935] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16195.138941] NMI backtrace for cpu 2 [16195.138946] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16195.138951] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16195.138954] Call Trace: [16195.138959] [16195.138962] ? dump_stack+0xbb/0x10e [16195.138970] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16195.138976] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16195.328193] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16195.330474] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16195.332803] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16195.334445] ? rcu_dump_cpu_stacks+0x11b/0x171 [16195.338349] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [16195.340257] ? update_process_times+0x6d/0xc0 [16195.342439] ? tick_sched_handle.isra.18+0x27/0x90 [16195.344300] ? tick_sched_timer+0x53/0xb0 [16195.345959] ? __hrtimer_run_queues+0x1c2/0x4d0 [16195.347946] ? tick_sched_handle.isra.18+0x90/0x90 [16195.350537] ? hrtimer_interrupt+0x128/0x2c0 [16195.352434] ? smp_apic_timer_interrupt+0xa6/0x270 [16195.353809] ? apic_timer_interrupt+0xf/0x20 [16195.356776] [16195.357686] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16195.360110] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16195.362408] ? do_raw_spin_lock+0x177/0x190 [16195.364114] ? _raw_spin_lock+0x21/0x30 [16195.365698] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16195.368105] ? obd_mod_free+0x46/0x140 [ptlrpc] [16195.370316] ? mdc_commit_open+0x91/0xb0 [mdc] [16195.373452] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16195.376177] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16195.379498] ? after_reply+0xe2c/0x1780 [ptlrpc] [16195.381968] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16195.384463] ? __next_timer_interrupt+0x160/0x160 [16195.388124] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16195.390217] ? do_wait_intr+0xf0/0xf0 [16195.391596] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16195.394110] ? kthread+0x1d1/0x200 [16195.396219] ? set_kthread_struct+0x70/0x70 [16195.397449] ? ret_from_fork+0x1f/0x30 [16196.061802] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16196.071670] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16196.124780] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16196.139304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16196.149213] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16196.152421] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16196.177844] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16196.183030] RAX: 000000000000757b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16196.189696] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16196.198953] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16196.206309] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16196.210378] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16196.219772] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16196.223333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16196.227130] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16196.233537] Call Trace: [16196.234397] [16196.235495] ? show_regs.cold.9+0x22/0x2f [16196.237037] ? watchdog_timer_fn.cold.6+0x62/0x10a [16196.239340] ? __hrtimer_run_queues+0x1c2/0x4d0 [16196.242320] ? watchdog+0x40/0x40 [16196.245252] ? hrtimer_interrupt+0x128/0x2c0 [16196.250229] ? smp_apic_timer_interrupt+0xa6/0x270 [16196.258156] ? apic_timer_interrupt+0xf/0x20 [16196.269461] [16196.271123] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16196.279092] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16196.283068] do_raw_spin_lock+0x177/0x190 [16196.284401] _raw_spin_lock+0x21/0x30 [16196.285476] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16196.289968] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16196.296705] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16196.300905] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16196.305976] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16196.311988] ? woken_wake_function+0x30/0x30 [16196.315124] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16196.317807] kthread+0x1d1/0x200 [16196.319670] ? set_kthread_struct+0x70/0x70 [16196.322135] ret_from_fork+0x1f/0x30 [16220.052028] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [16220.055084] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16220.057775] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [16220.057916] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [16220.057923] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16220.058002] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16220.058157] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16220.058160] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16220.058180] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16220.058186] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16220.058191] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16220.058194] RDX: 0000000000002cc8 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16220.058197] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16220.058204] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16220.058206] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16220.058213] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16220.058216] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16220.058219] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16220.058233] Call Trace: [16220.058239] [16220.058245] ? show_regs.cold.9+0x22/0x2f [16220.058259] ? watchdog_timer_fn.cold.6+0x62/0x10a [16220.058266] ? __hrtimer_run_queues+0x1c2/0x4d0 [16220.058274] ? watchdog+0x40/0x40 [16220.058279] ? hrtimer_interrupt+0x128/0x2c0 [16220.058284] ? smp_apic_timer_interrupt+0xa6/0x270 [16220.058292] ? apic_timer_interrupt+0xf/0x20 [16220.058297] [16220.058301] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16220.058311] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16220.058315] do_raw_spin_lock+0x177/0x190 [16220.058320] _raw_spin_lock+0x21/0x30 [16220.058326] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16220.058636] obd_mod_free+0x46/0x140 [ptlrpc] [16220.058849] mdc_commit_open+0x91/0xb0 [mdc] [16220.058884] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16220.059100] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16220.060533] after_reply+0xe2c/0x1780 [ptlrpc] [16220.061358] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16220.062762] ? __next_timer_interrupt+0x160/0x160 [16220.062773] ptlrpcd+0xa82/0xe40 [ptlrpc] [16220.062992] ? do_wait_intr+0xf0/0xf0 [16220.063002] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16220.063196] kthread+0x1d1/0x200 [16220.063202] ? set_kthread_struct+0x70/0x70 [16220.063206] ret_from_fork+0x1f/0x30 [16220.131529] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16220.131545] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16220.131552] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16220.135169] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16220.135184] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16220.135188] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16220.135194] RAX: 000000000000618b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16220.135196] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16220.135198] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16220.135200] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16220.135202] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16220.135206] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16220.135209] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16220.135211] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16220.136604] Call Trace: [16220.140585] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16220.140662] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16220.140670] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16220.140673] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16220.140687] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16220.140691] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16220.140696] RAX: 0000000000007d8a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16220.140698] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16220.140700] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16220.140702] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16220.140705] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16220.140708] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16220.140711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16220.140713] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16220.140723] Call Trace: [16220.140728] [16220.140734] ? show_regs.cold.9+0x22/0x2f [16220.140744] ? watchdog_timer_fn.cold.6+0x62/0x10a [16220.140750] ? __hrtimer_run_queues+0x1c2/0x4d0 [16220.140755] ? watchdog+0x40/0x40 [16220.140759] ? hrtimer_interrupt+0x128/0x2c0 [16220.140762] ? smp_apic_timer_interrupt+0xa6/0x270 [16220.140770] ? apic_timer_interrupt+0xf/0x20 [16220.140774] [16220.140776] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16220.140782] do_raw_spin_lock+0x177/0x190 [16220.140786] _raw_spin_lock+0x21/0x30 [16220.140791] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16220.141621] ? do_raw_spin_unlock+0x75/0x190 [16220.141628] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16220.141836] ? do_wait_intr+0xf0/0xf0 [16220.141844] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16220.142045] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16220.142319] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16220.142641] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16220.142765] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16220.142804] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16220.142835] mdc_intent_lock+0x353/0x7e0 [mdc] [16220.142863] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16220.142950] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16220.143231] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16220.143257] lmv_intent_open+0x338/0x1150 [lmv] [16220.143285] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16220.143369] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16220.143646] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16220.143672] ll_intent_lock+0xc5/0xaa0 [lustre] [16220.143767] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16220.143860] ll_lookup_it+0xe0e/0x2270 [lustre] [16220.143957] ? do_raw_spin_unlock+0x75/0x190 [16220.143964] ? _raw_spin_unlock+0x12/0x30 [16220.143976] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16220.143999] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16220.144025] ll_atomic_open+0x36a/0xd70 [lustre] [16220.144121] lookup_open+0x356/0xb40 [16220.144184] do_last+0x313/0xfc0 [16220.144189] ? nd_jump_root+0xe5/0x160 [16220.144194] ? path_init+0x437/0x520 [16220.144199] path_openat+0xf7/0x500 [16220.144203] ? do_raw_spin_unlock+0x75/0x190 [16220.144208] do_filp_open+0x99/0x140 [16220.144341] ? getname_flags+0x6e/0x330 [16220.144347] ? __check_object_size+0xff/0x256 [16220.144354] ? do_raw_spin_unlock+0x75/0x190 [16220.144358] ? _raw_spin_unlock+0x12/0x30 [16220.144362] do_sys_openat2+0x2b4/0x410 [16220.144366] do_sys_open+0x73/0xa0 [16220.144371] __x64_sys_openat+0x24/0x30 [16220.144374] do_syscall_64+0xc1/0x440 [16220.144381] entry_SYSCALL_64_after_hwframe+0x49/0xae [16220.144387] RIP: 0033:0x7f53425a5332 [16220.144394] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16220.144398] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16220.146569] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16220.146574] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16220.146576] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16220.146578] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16220.146581] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16220.913709] [16220.915556] ? show_regs.cold.9+0x22/0x2f [16220.917543] ? watchdog_timer_fn.cold.6+0x62/0x10a [16220.919713] ? __hrtimer_run_queues+0x1c2/0x4d0 [16220.921755] ? watchdog+0x40/0x40 [16220.923922] ? hrtimer_interrupt+0x128/0x2c0 [16220.928674] ? smp_apic_timer_interrupt+0xa6/0x270 [16220.931357] ? apic_timer_interrupt+0xf/0x20 [16220.935031] [16220.937614] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16220.941941] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16220.943892] ? newidle_balance.isra.93+0x207/0x580 [16220.951175] do_raw_spin_lock+0x177/0x190 [16220.952719] _raw_spin_lock+0x21/0x30 [16220.954618] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16220.957089] process_one_work+0x2c8/0x700 [16220.958303] worker_thread+0x296/0x6e0 [16220.960255] ? rescuer_thread+0x570/0x570 [16220.961833] kthread+0x1d1/0x200 [16220.963261] ? set_kthread_struct+0x70/0x70 [16220.965237] ret_from_fork+0x1f/0x30 [16223.712038] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13586s! [16223.716409] Showing busy workqueues and worker pools: [16223.718813] workqueue events: flags=0x0 [16223.720548] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16223.724004] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16223.728099] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16223.730972] pending: cpu_latency_work [ptlrpc] [16223.733642] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16223.736220] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16223.739743] workqueue events_long: flags=0x0 [16223.741246] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16223.743829] pending: lu_objects_flush [obdclass] [16223.747637] workqueue events_unbound: flags=0x2 [16223.749183] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16223.753498] pending: flush_memcg_stats_dwork, idle_cull_fn [16223.760611] workqueue events_power_efficient: flags=0x80 [16223.769153] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16223.775536] pending: check_lifetime, do_cache_clean [16223.777404] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16223.782729] pending: neigh_periodic_work, neigh_periodic_work [16223.784701] workqueue mm_percpu_wq: flags=0x8 [16223.786010] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16223.790547] pending: vmstat_update, lru_add_drain_per_cpu [16223.794726] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16223.797766] pending: vmstat_update, lru_add_drain_per_cpu [16223.803624] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16223.812894] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16223.814433] workqueue xprtiod: flags=0xa [16223.815370] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16223.823212] pending: xprt_autoclose, xs_error_handle [16223.833562] workqueue ipv6_addrconf: flags=0x40008 [16223.835203] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16223.846962] pending: addrconf_verify_work [16223.848484] workqueue knbd0-recv: flags=0x1a [16223.852832] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16223.855094] in-flight: 651:recv_work [16223.856920] workqueue ptlrpc_pinger: flags=0x2 [16223.859893] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16223.865044] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16223.868597] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13586s workers=4 idle: 15979 31824 8 [16223.871868] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16214s workers=2 idle: 654 [16224.059949] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16224.068571] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16224.097226] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16224.103915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16224.109424] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16224.113488] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16224.140868] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16224.151568] RAX: 0000000000007f89 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16224.156824] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16224.162332] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16224.170291] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16224.174944] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16224.180259] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16224.184134] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16224.189591] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16224.195508] Call Trace: [16224.196532] [16224.197661] ? show_regs.cold.9+0x22/0x2f [16224.199314] ? watchdog_timer_fn.cold.6+0x62/0x10a [16224.201333] ? __hrtimer_run_queues+0x1c2/0x4d0 [16224.203206] ? watchdog+0x40/0x40 [16224.204562] ? hrtimer_interrupt+0x128/0x2c0 [16224.206444] ? smp_apic_timer_interrupt+0xa6/0x270 [16224.208414] ? apic_timer_interrupt+0xf/0x20 [16224.210110] [16224.210988] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16224.213138] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16224.215248] do_raw_spin_lock+0x177/0x190 [16224.216933] _raw_spin_lock+0x21/0x30 [16224.218784] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16224.221705] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16224.224066] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16224.226534] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16224.228929] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16224.231249] ? woken_wake_function+0x30/0x30 [16224.233050] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16224.235494] kthread+0x1d1/0x200 [16224.236277] ? set_kthread_struct+0x70/0x70 [16224.237415] ret_from_fork+0x1f/0x30 [16248.055009] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [16248.056216] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [16248.057842] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16248.057912] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16248.058036] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16248.058039] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16248.058051] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16248.058056] RSP: 0018:ffffbb5381503500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [16248.058105] RAX: 00000000000023b4 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16248.058108] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16248.058111] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16248.058113] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16248.058115] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16248.058118] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16248.058122] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16248.058125] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16248.058133] Call Trace: [16248.058139] [16248.058144] ? show_regs.cold.9+0x22/0x2f [16248.058154] ? watchdog_timer_fn.cold.6+0x62/0x10a [16248.058160] ? __hrtimer_run_queues+0x1c2/0x4d0 [16248.058164] ? watchdog+0x40/0x40 [16248.058168] ? hrtimer_interrupt+0x128/0x2c0 [16248.058172] ? smp_apic_timer_interrupt+0xa6/0x270 [16248.058181] ? apic_timer_interrupt+0xf/0x20 [16248.058186] [16248.058189] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16248.058196] do_raw_spin_lock+0x177/0x190 [16248.058201] _raw_spin_lock+0x21/0x30 [16248.058205] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16248.058467] ? do_raw_spin_unlock+0x75/0x190 [16248.058474] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16248.058661] ? do_wait_intr+0xf0/0xf0 [16248.059833] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16248.060755] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [16248.060763] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16248.060839] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16248.060844] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16248.060847] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16248.060857] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16248.060861] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16248.060865] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16248.060868] RDX: 0000000000006e0a RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16248.060870] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16248.060873] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16248.060876] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16248.060879] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16248.060882] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16248.060885] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16248.061311] Call Trace: [16248.061316] [16248.061319] ? show_regs.cold.9+0x22/0x2f [16248.061328] ? watchdog_timer_fn.cold.6+0x62/0x10a [16248.061334] ? __hrtimer_run_queues+0x1c2/0x4d0 [16248.061338] ? watchdog+0x40/0x40 [16248.061342] ? hrtimer_interrupt+0x128/0x2c0 [16248.061346] ? smp_apic_timer_interrupt+0xa6/0x270 [16248.061353] ? apic_timer_interrupt+0xf/0x20 [16248.061357] [16248.061360] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16248.061364] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16248.061369] do_raw_spin_lock+0x177/0x190 [16248.061373] _raw_spin_lock+0x21/0x30 [16248.061379] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16248.061621] obd_mod_free+0x46/0x140 [ptlrpc] [16248.062784] mdc_commit_open+0x91/0xb0 [mdc] [16248.062875] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16248.063465] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16248.063739] after_reply+0xe2c/0x1780 [ptlrpc] [16248.064176] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) [16248.064382] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16248.064589] ? __next_timer_interrupt+0x160/0x160 [16248.064598] ptlrpcd+0xa82/0xe40 [ptlrpc] [16248.067492] ? do_wait_intr+0xf0/0xf0 [16248.067504] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16248.068356] kthread+0x1d1/0x200 [16248.068364] ? set_kthread_struct+0x70/0x70 [16248.068369] ret_from_fork+0x1f/0x30 [16248.103686] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16248.104248] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16248.104819] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16248.105402] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16248.105456] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16248.105495] mdc_intent_lock+0x353/0x7e0 [mdc] [16248.105525] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16248.105833] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16248.122851] ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16248.122903] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16248.122912] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16248.122936] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16248.123227] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16248.123237] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16248.123241] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16248.123246] RAX: 0000000000001806 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16248.123249] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16248.123251] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16248.123253] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16248.123255] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16248.125377] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16248.125384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16248.125387] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16248.125396] Call Trace: [16248.125403] [16248.125409] ? show_regs.cold.9+0x22/0x2f [16248.125426] ? watchdog_timer_fn.cold.6+0x62/0x10a [16248.125436] ? __hrtimer_run_queues+0x1c2/0x4d0 [16248.125441] ? watchdog+0x40/0x40 [16248.125449] ? hrtimer_interrupt+0x128/0x2c0 [16248.125453] ? smp_apic_timer_interrupt+0xa6/0x270 [16248.125461] ? apic_timer_interrupt+0xf/0x20 [16248.125465] [16248.125468] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16248.125474] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16248.125478] ? newidle_balance.isra.93+0x207/0x580 [16248.125485] do_raw_spin_lock+0x177/0x190 [16248.125489] _raw_spin_lock+0x21/0x30 [16248.125494] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16248.125784] process_one_work+0x2c8/0x700 [16248.125792] worker_thread+0x296/0x6e0 [16248.125796] ? rescuer_thread+0x570/0x570 [16248.125799] kthread+0x1d1/0x200 [16248.125803] ? set_kthread_struct+0x70/0x70 [16248.125807] ret_from_fork+0x1f/0x30 [16248.845870] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16248.851485] lmv_intent_open+0x338/0x1150 [lmv] [16248.853362] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16248.856033] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16248.858571] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16248.860611] ll_intent_lock+0xc5/0xaa0 [lustre] [16248.862464] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16248.869369] ll_lookup_it+0xe0e/0x2270 [lustre] [16248.873524] ? do_raw_spin_unlock+0x75/0x190 [16248.876449] ? _raw_spin_unlock+0x12/0x30 [16248.881513] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16248.884152] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16248.892465] ll_atomic_open+0x36a/0xd70 [lustre] [16248.895582] lookup_open+0x356/0xb40 [16248.898652] do_last+0x313/0xfc0 [16248.902510] ? nd_jump_root+0xe5/0x160 [16248.908682] ? path_init+0x437/0x520 [16248.913307] path_openat+0xf7/0x500 [16248.914398] ? do_raw_spin_unlock+0x75/0x190 [16248.915515] do_filp_open+0x99/0x140 [16248.923173] ? getname_flags+0x6e/0x330 [16248.925296] ? __check_object_size+0xff/0x256 [16248.928400] ? do_raw_spin_unlock+0x75/0x190 [16248.933205] ? _raw_spin_unlock+0x12/0x30 [16248.936123] do_sys_openat2+0x2b4/0x410 [16248.937653] do_sys_open+0x73/0xa0 [16248.940930] __x64_sys_openat+0x24/0x30 [16248.942470] do_syscall_64+0xc1/0x440 [16248.945555] entry_SYSCALL_64_after_hwframe+0x49/0xae [16248.953137] RIP: 0033:0x7f53425a5332 [16248.958754] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16248.977876] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16248.984344] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16248.991169] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16248.997867] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16249.004612] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16249.011779] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16252.069163] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16252.078208] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16252.135386] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16252.147675] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16252.161501] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16252.165356] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16252.192927] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16252.202775] RAX: 0000000000007a85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16252.211749] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16252.217917] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16252.227900] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16252.239178] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16252.245249] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16252.254441] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16252.260168] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16252.271218] Call Trace: [16252.272917] [16252.275939] ? show_regs.cold.9+0x22/0x2f [16252.280711] ? watchdog_timer_fn.cold.6+0x62/0x10a [16252.288991] ? __hrtimer_run_queues+0x1c2/0x4d0 [16252.290439] ? watchdog+0x40/0x40 [16252.291409] ? hrtimer_interrupt+0x128/0x2c0 [16252.300295] ? smp_apic_timer_interrupt+0xa6/0x270 [16252.304456] ? apic_timer_interrupt+0xf/0x20 [16252.310803] [16252.311540] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16252.313235] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16252.327403] do_raw_spin_lock+0x177/0x190 [16252.335539] _raw_spin_lock+0x21/0x30 [16252.342387] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16252.350589] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16252.359980] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16252.369114] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16252.379541] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16252.395444] ? woken_wake_function+0x30/0x30 [16252.401008] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16252.416036] kthread+0x1d1/0x200 [16252.418972] ? set_kthread_struct+0x70/0x70 [16252.430115] ret_from_fork+0x1f/0x30 [16253.920040] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13616s! [16253.925531] Showing busy workqueues and worker pools: [16253.928745] workqueue events: flags=0x0 [16253.934420] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16253.940152] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16253.946671] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16253.950574] pending: cpu_latency_work [ptlrpc] [16253.953698] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16253.957467] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16253.962920] workqueue events_long: flags=0x0 [16253.966432] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16253.969606] pending: lu_objects_flush [obdclass] [16253.973106] workqueue events_unbound: flags=0x2 [16253.975621] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16253.979093] pending: flush_memcg_stats_dwork, idle_cull_fn [16253.982066] workqueue events_power_efficient: flags=0x80 [16253.985539] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16253.989582] pending: check_lifetime, do_cache_clean [16253.995163] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16253.999487] pending: neigh_periodic_work, neigh_periodic_work [16254.003340] workqueue mm_percpu_wq: flags=0x8 [16254.011608] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16254.019401] pending: vmstat_update, lru_add_drain_per_cpu [16254.026661] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16254.035516] pending: vmstat_update, lru_add_drain_per_cpu [16254.039680] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16254.044327] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16254.055692] workqueue xprtiod: flags=0xa [16254.062700] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16254.070671] pending: xprt_autoclose, xs_error_handle [16254.079015] workqueue ipv6_addrconf: flags=0x40008 [16254.081579] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16254.085892] pending: addrconf_verify_work [16254.088709] workqueue knbd0-recv: flags=0x1a [16254.094632] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16254.099553] in-flight: 651:recv_work [16254.105078] workqueue ptlrpc_pinger: flags=0x2 [16254.110054] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16254.113905] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16254.118417] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13617s workers=4 idle: 15979 31824 8 [16254.127364] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16244s workers=2 idle: 654 [16276.055068] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [16276.058369] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [16276.058961] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16276.059030] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16276.059034] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16276.059037] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16276.059049] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16276.059052] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16276.059055] RAX: 0000000000005a3f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16276.059057] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16276.059058] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16276.059060] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16276.059062] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16276.059064] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16276.059066] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16276.059067] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16276.059074] Call Trace: [16276.059078] [16276.059083] ? show_regs.cold.9+0x22/0x2f [16276.059090] ? watchdog_timer_fn.cold.6+0x62/0x10a [16276.059094] ? __hrtimer_run_queues+0x1c2/0x4d0 [16276.059097] ? watchdog+0x40/0x40 [16276.059099] ? hrtimer_interrupt+0x128/0x2c0 [16276.059102] ? smp_apic_timer_interrupt+0xa6/0x270 [16276.059108] ? apic_timer_interrupt+0xf/0x20 [16276.059111] [16276.059113] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16276.059118] do_raw_spin_lock+0x177/0x190 [16276.059121] _raw_spin_lock+0x21/0x30 [16276.059125] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16276.065870] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16276.065948] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16276.065955] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16276.065958] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16276.065972] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16276.065976] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16276.065981] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16276.065983] RDX: 00000000000053d9 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16276.065985] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16276.065987] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16276.065989] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16276.065992] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16276.065995] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16276.065998] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16276.066006] Call Trace: [16276.066012] [16276.066018] ? show_regs.cold.9+0x22/0x2f [16276.066029] ? watchdog_timer_fn.cold.6+0x62/0x10a [16276.066036] ? __hrtimer_run_queues+0x1c2/0x4d0 [16276.066041] ? watchdog+0x40/0x40 [16276.066044] ? hrtimer_interrupt+0x128/0x2c0 [16276.066048] ? smp_apic_timer_interrupt+0xa6/0x270 [16276.066055] ? apic_timer_interrupt+0xf/0x20 [16276.066060] [16276.066063] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16276.066151] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16276.066157] do_raw_spin_lock+0x177/0x190 [16276.066162] _raw_spin_lock+0x21/0x30 [16276.066168] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16276.068585] obd_mod_free+0x46/0x140 [ptlrpc] [16276.068856] mdc_commit_open+0x91/0xb0 [mdc] [16276.068895] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16276.069244] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16276.069470] after_reply+0xe2c/0x1780 [ptlrpc] [16276.069668] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16276.069980] ? __next_timer_interrupt+0x160/0x160 [16276.069990] ptlrpcd+0xa82/0xe40 [ptlrpc] [16276.070588] ? do_wait_intr+0xf0/0xf0 [16276.070796] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16276.111337] ? do_raw_spin_unlock+0x75/0x190 [16276.111358] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16276.111654] ? do_wait_intr+0xf0/0xf0 [16276.111664] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16276.112606] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16276.113367] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16276.124420] kthread+0x1d1/0x200 [16276.124454] ? set_kthread_struct+0x70/0x70 [16276.124459] ret_from_fork+0x1f/0x30 [16276.560550] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16276.568108] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16276.570630] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16276.578231] mdc_intent_lock+0x353/0x7e0 [mdc] [16276.582286] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16276.585854] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16276.593722] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16276.602579] lmv_intent_open+0x338/0x1150 [lmv] [16276.607237] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16276.614092] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16276.616525] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16276.620746] ll_intent_lock+0xc5/0xaa0 [lustre] [16276.623785] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16276.626517] ll_lookup_it+0xe0e/0x2270 [lustre] [16276.633088] ? do_raw_spin_unlock+0x75/0x190 [16276.639249] ? _raw_spin_unlock+0x12/0x30 [16276.642963] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16276.647946] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16276.650899] ll_atomic_open+0x36a/0xd70 [lustre] [16276.652366] lookup_open+0x356/0xb40 [16276.657512] do_last+0x313/0xfc0 [16276.659041] ? nd_jump_root+0xe5/0x160 [16276.662527] ? path_init+0x437/0x520 [16276.671236] path_openat+0xf7/0x500 [16276.675402] ? do_raw_spin_unlock+0x75/0x190 [16276.680354] do_filp_open+0x99/0x140 [16276.684637] ? getname_flags+0x6e/0x330 [16276.689479] ? __check_object_size+0xff/0x256 [16276.692984] ? do_raw_spin_unlock+0x75/0x190 [16276.697584] ? _raw_spin_unlock+0x12/0x30 [16276.699525] do_sys_openat2+0x2b4/0x410 [16276.704523] do_sys_open+0x73/0xa0 [16276.707140] __x64_sys_openat+0x24/0x30 [16276.710078] do_syscall_64+0xc1/0x440 [16276.716025] entry_SYSCALL_64_after_hwframe+0x49/0xae [16276.718817] RIP: 0033:0x7f53425a5332 [16276.723238] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16276.740466] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16276.746692] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16276.756535] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16276.772578] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16276.776136] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16276.779881] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16280.052030] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [16280.056390] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16280.058020] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16280.081179] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16280.086280] Modules linked in: lustre(O) osp(O) [16280.092288] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16280.092305] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16280.092636] RIP: 0010:native_safe_halt+0x2e/0x50 [16280.092648] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16280.092653] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16280.092658] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16280.092661] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [16280.092663] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16280.092665] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16280.092668] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16280.092671] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16280.092674] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16280.092677] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16280.092686] Call Trace: [16280.092693] [16280.092698] ? show_regs.cold.9+0x22/0x2f [16280.092707] ? watchdog_timer_fn.cold.6+0x62/0x10a [16280.092714] ? __hrtimer_run_queues+0x1c2/0x4d0 [16280.092718] ? watchdog+0x40/0x40 [16280.092722] ? hrtimer_interrupt+0x128/0x2c0 [16280.092726] ? smp_apic_timer_interrupt+0xa6/0x270 [16280.092733] ? apic_timer_interrupt+0xf/0x20 [16280.092744] [16280.092837] ? native_safe_halt+0x2e/0x50 [16280.092844] kvm_wait+0xcd/0xe0 [16280.092852] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16280.092863] ? newidle_balance.isra.93+0x207/0x580 [16280.092869] do_raw_spin_lock+0x177/0x190 [16280.092874] _raw_spin_lock+0x21/0x30 [16280.092879] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16280.097261] ofd(O) [16280.102435] process_one_work+0x2c8/0x700 [16280.106143] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16280.106228] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16280.106236] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16280.106240] RIP: 0010:native_safe_halt+0x2e/0x50 [16280.106254] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16280.106258] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16280.106263] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16280.106265] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16280.106268] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16280.106269] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16280.106272] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16280.106275] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16280.106277] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16280.106279] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16280.106287] Call Trace: [16280.106292] [16280.106297] ? show_regs.cold.9+0x22/0x2f [16280.106312] ? watchdog_timer_fn.cold.6+0x62/0x10a [16280.106319] ? __hrtimer_run_queues+0x1c2/0x4d0 [16280.106323] ? watchdog+0x40/0x40 [16280.106328] ? hrtimer_interrupt+0x128/0x2c0 [16280.106331] ? smp_apic_timer_interrupt+0xa6/0x270 [16280.106341] ? apic_timer_interrupt+0xf/0x20 [16280.106346] [16280.106349] ? native_safe_halt+0x2e/0x50 [16280.106353] kvm_wait+0xcd/0xe0 [16280.106432] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16280.106443] do_raw_spin_lock+0x177/0x190 [16280.106447] _raw_spin_lock+0x21/0x30 [16280.106458] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16280.106877] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16280.111244] worker_thread+0x296/0x6e0 [16280.111266] ? rescuer_thread+0x570/0x570 [16280.111270] kthread+0x1d1/0x200 [16280.120230] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16280.127574] ? set_kthread_struct+0x70/0x70 [16280.143885] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16280.147410] ret_from_fork+0x1f/0x30 [16280.417045] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16280.419412] ? woken_wake_function+0x30/0x30 [16280.421201] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16280.423831] kthread+0x1d1/0x200 [16280.425243] ? set_kthread_struct+0x70/0x70 [16280.426838] ret_from_fork+0x1f/0x30 [16284.127078] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13647s! [16284.131188] Showing busy workqueues and worker pools: [16284.134109] workqueue events: flags=0x0 [16284.136684] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16284.140304] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16284.145820] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16284.148444] pending: cpu_latency_work [ptlrpc] [16284.151077] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16284.154212] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16284.160061] workqueue events_long: flags=0x0 [16284.162015] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16284.165035] pending: lu_objects_flush [obdclass] [16284.168047] workqueue events_unbound: flags=0x2 [16284.170199] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16284.173514] pending: flush_memcg_stats_dwork, idle_cull_fn [16284.177930] workqueue events_power_efficient: flags=0x80 [16284.183187] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16284.188061] pending: check_lifetime, do_cache_clean [16284.192196] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16284.194381] pending: neigh_periodic_work, neigh_periodic_work [16284.196047] workqueue mm_percpu_wq: flags=0x8 [16284.197602] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16284.199707] pending: vmstat_update, lru_add_drain_per_cpu [16284.201703] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16284.204245] pending: vmstat_update, lru_add_drain_per_cpu [16284.207564] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16284.209966] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16284.212165] workqueue xprtiod: flags=0xa [16284.213510] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16284.215844] pending: xprt_autoclose, xs_error_handle [16284.217238] workqueue ipv6_addrconf: flags=0x40008 [16284.218787] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16284.220922] pending: addrconf_verify_work [16284.222483] workqueue knbd0-recv: flags=0x1a [16284.227159] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16284.233882] in-flight: 651:recv_work [16284.235131] workqueue ptlrpc_pinger: flags=0x2 [16284.237949] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16284.242605] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16284.247465] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13647s workers=4 idle: 15979 31824 8 [16284.254565] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16274s workers=2 idle: 654 [16304.055012] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [16304.057723] Modules linked in: [16304.057724] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [16304.057732] Modules linked in: [16304.058814] lustre(O) [16304.062858] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16304.062932] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16304.062949] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16304.062952] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16304.062966] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16304.062971] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16304.062976] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16304.062979] RDX: 0000000000007f81 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16304.062981] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16304.062983] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16304.062985] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16304.062989] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16304.062992] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16304.062994] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16304.063003] Call Trace: [16304.063009] [16304.063015] ? show_regs.cold.9+0x22/0x2f [16304.063025] ? watchdog_timer_fn.cold.6+0x62/0x10a [16304.063032] ? __hrtimer_run_queues+0x1c2/0x4d0 [16304.063036] ? watchdog+0x40/0x40 [16304.063041] ? hrtimer_interrupt+0x128/0x2c0 [16304.063045] ? smp_apic_timer_interrupt+0xa6/0x270 [16304.063053] ? apic_timer_interrupt+0xf/0x20 [16304.063057] [16304.063060] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16304.063066] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16304.063071] do_raw_spin_lock+0x177/0x190 [16304.063077] _raw_spin_lock+0x21/0x30 [16304.063410] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16304.071416] osp(O) [16304.072262] obd_mod_free+0x46/0x140 [ptlrpc] [16304.072640] mdc_commit_open+0x91/0xb0 [mdc] [16304.072681] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16304.073396] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16304.119796] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16304.119869] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16304.119876] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16304.119880] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16304.133616] after_reply+0xe2c/0x1780 [ptlrpc] [16304.142398] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16304.142411] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16304.142418] RAX: 00000000000008fa RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16304.142420] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16304.142423] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16304.142424] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16304.142427] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16304.142501] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16304.142504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16304.142506] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16304.142514] Call Trace: [16304.142519] [16304.142525] ? show_regs.cold.9+0x22/0x2f [16304.142539] ? watchdog_timer_fn.cold.6+0x62/0x10a [16304.142545] ? __hrtimer_run_queues+0x1c2/0x4d0 [16304.142549] ? watchdog+0x40/0x40 [16304.142554] ? hrtimer_interrupt+0x128/0x2c0 [16304.142557] ? smp_apic_timer_interrupt+0xa6/0x270 [16304.142568] ? apic_timer_interrupt+0xf/0x20 [16304.142573] [16304.142576] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16304.142586] do_raw_spin_lock+0x177/0x190 [16304.142591] _raw_spin_lock+0x21/0x30 [16304.142596] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16304.143187] ? do_raw_spin_unlock+0x75/0x190 [16304.143194] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16304.143389] ? do_wait_intr+0xf0/0xf0 [16304.143397] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16304.143619] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16304.143911] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16304.144362] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16304.144557] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16304.144595] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16304.144623] mdc_intent_lock+0x353/0x7e0 [mdc] [16304.144648] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16304.147339] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16304.147641] ? __next_timer_interrupt+0x160/0x160 [16304.147651] ptlrpcd+0xa82/0xe40 [ptlrpc] [16304.147853] ? do_wait_intr+0xf0/0xf0 [16304.147861] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16304.148209] kthread+0x1d1/0x200 [16304.148215] ? set_kthread_struct+0x70/0x70 [16304.148219] ret_from_fork+0x1f/0x30 [16304.332391] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16304.336313] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16304.339844] lmv_intent_open+0x338/0x1150 [lmv] [16304.343548] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16304.345907] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16304.348389] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16304.351530] ll_intent_lock+0xc5/0xaa0 [lustre] [16304.353682] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16304.356164] ll_lookup_it+0xe0e/0x2270 [lustre] [16304.358410] ? do_raw_spin_unlock+0x75/0x190 [16304.360348] ? _raw_spin_unlock+0x12/0x30 [16304.362131] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16304.364411] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16304.366725] ll_atomic_open+0x36a/0xd70 [lustre] [16304.368805] lookup_open+0x356/0xb40 [16304.370088] do_last+0x313/0xfc0 [16304.371291] ? nd_jump_root+0xe5/0x160 [16304.373318] ? path_init+0x437/0x520 [16304.374833] path_openat+0xf7/0x500 [16304.376366] ? do_raw_spin_unlock+0x75/0x190 [16304.378338] do_filp_open+0x99/0x140 [16304.380133] ? getname_flags+0x6e/0x330 [16304.381897] ? __check_object_size+0xff/0x256 [16304.384042] ? do_raw_spin_unlock+0x75/0x190 [16304.385812] ? _raw_spin_unlock+0x12/0x30 [16304.387598] do_sys_openat2+0x2b4/0x410 [16304.389592] do_sys_open+0x73/0xa0 [16304.391217] __x64_sys_openat+0x24/0x30 [16304.393413] do_syscall_64+0xc1/0x440 [16304.395269] entry_SYSCALL_64_after_hwframe+0x49/0xae [16304.397631] RIP: 0033:0x7f53425a5332 [16304.399415] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16304.407875] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16304.411286] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16304.414753] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16304.417854] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16304.421313] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16304.424706] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16308.058025] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [16308.061025] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16308.085863] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16308.097198] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16308.103383] RIP: 0010:native_safe_halt+0x2e/0x50 [16308.107030] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16308.118767] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16308.126536] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16308.129582] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16308.135558] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16308.139184] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16308.146468] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16308.148451] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16308.155980] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16308.158301] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16308.165957] Call Trace: [16308.166772] [16308.167443] ? show_regs.cold.9+0x22/0x2f [16308.168954] ? watchdog_timer_fn.cold.6+0x62/0x10a [16308.170664] ? __hrtimer_run_queues+0x1c2/0x4d0 [16308.172403] ? watchdog+0x40/0x40 [16308.173520] ? hrtimer_interrupt+0x128/0x2c0 [16308.175198] ? smp_apic_timer_interrupt+0xa6/0x270 [16308.177172] ? apic_timer_interrupt+0xf/0x20 [16308.179114] [16308.179848] ? native_safe_halt+0x2e/0x50 [16308.181605] kvm_wait+0xcd/0xe0 [16308.183078] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16308.185418] do_raw_spin_lock+0x177/0x190 [16308.187232] _raw_spin_lock+0x21/0x30 [16308.188584] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16308.191340] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16308.194040] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16308.196525] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16308.199217] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16308.201750] ? woken_wake_function+0x30/0x30 [16308.203388] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16308.205816] kthread+0x1d1/0x200 [16308.207275] ? set_kthread_struct+0x70/0x70 [16308.208863] ret_from_fork+0x1f/0x30 [16312.052033] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [16312.057653] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16312.088414] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16312.098924] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16312.103335] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16312.107646] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16312.114407] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16312.127185] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16312.130059] RAX: 0000000000003c08 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16312.132608] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16312.136567] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16312.139506] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16312.142470] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16312.144826] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16312.147878] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16312.149616] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16312.151704] Call Trace: [16312.152551] [16312.153269] ? show_regs.cold.9+0x22/0x2f [16312.154744] ? watchdog_timer_fn.cold.6+0x62/0x10a [16312.156260] ? __hrtimer_run_queues+0x1c2/0x4d0 [16312.157593] ? watchdog+0x40/0x40 [16312.158561] ? hrtimer_interrupt+0x128/0x2c0 [16312.160128] ? smp_apic_timer_interrupt+0xa6/0x270 [16312.161837] ? apic_timer_interrupt+0xf/0x20 [16312.163267] [16312.163843] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16312.165547] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16312.167301] ? newidle_balance.isra.93+0x207/0x580 [16312.168660] do_raw_spin_lock+0x177/0x190 [16312.169888] _raw_spin_lock+0x21/0x30 [16312.171078] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16312.172861] process_one_work+0x2c8/0x700 [16312.174196] worker_thread+0x296/0x6e0 [16312.175265] ? rescuer_thread+0x570/0x570 [16312.176544] kthread+0x1d1/0x200 [16312.177598] ? set_kthread_struct+0x70/0x70 [16312.178955] ret_from_fork+0x1f/0x30 [16314.335538] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13677s! [16314.340593] Showing busy workqueues and worker pools: [16314.345586] workqueue events: flags=0x0 [16314.346670] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16314.354456] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16314.360453] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16314.365020] pending: cpu_latency_work [ptlrpc] [16314.368612] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16314.370393] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16314.375307] workqueue events_long: flags=0x0 [16314.377111] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16314.379556] pending: lu_objects_flush [obdclass] [16314.381270] workqueue events_unbound: flags=0x2 [16314.385464] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16314.388120] pending: flush_memcg_stats_dwork, idle_cull_fn [16314.394634] workqueue events_power_efficient: flags=0x80 [16314.396658] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16314.399371] pending: check_lifetime, do_cache_clean [16314.403217] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16314.405014] pending: neigh_periodic_work, neigh_periodic_work [16314.407106] workqueue mm_percpu_wq: flags=0x8 [16314.408450] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16314.410955] pending: vmstat_update, lru_add_drain_per_cpu [16314.414732] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16314.418228] pending: vmstat_update, lru_add_drain_per_cpu [16314.420885] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16314.425047] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16314.428289] workqueue xprtiod: flags=0xa [16314.429481] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16314.431600] pending: xprt_autoclose, xs_error_handle [16314.434014] workqueue ipv6_addrconf: flags=0x40008 [16314.435842] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16314.437998] pending: addrconf_verify_work [16314.439710] workqueue knbd0-recv: flags=0x1a [16314.441633] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16314.444056] in-flight: 651:recv_work [16314.445619] workqueue ptlrpc_pinger: flags=0x2 [16314.447064] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16314.449140] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16314.451426] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13677s workers=4 idle: 15979 31824 8 [16314.454021] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16304s workers=2 idle: 654 [16332.055026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [16332.057202] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [16332.059130] Modules linked in: lustre(O) [16332.061765] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16332.061840] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16332.061847] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16332.061851] RIP: 0010:native_safe_halt+0x2e/0x50 [16332.061864] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16332.061868] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16332.061872] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16332.061875] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16332.061877] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16332.061879] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16332.061882] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16332.061886] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16332.061888] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16332.061891] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16332.061899] Call Trace: [16332.061905] [16332.061910] ? show_regs.cold.9+0x22/0x2f [16332.061920] ? watchdog_timer_fn.cold.6+0x62/0x10a [16332.061928] ? __hrtimer_run_queues+0x1c2/0x4d0 [16332.061932] ? watchdog+0x40/0x40 [16332.061936] ? hrtimer_interrupt+0x128/0x2c0 [16332.061940] ? smp_apic_timer_interrupt+0xa6/0x270 [16332.061947] ? apic_timer_interrupt+0xf/0x20 [16332.061952] [16332.061955] ? native_safe_halt+0x2e/0x50 [16332.061960] kvm_wait+0xcd/0xe0 [16332.061967] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16332.061975] do_raw_spin_lock+0x177/0x190 [16332.061980] _raw_spin_lock+0x21/0x30 [16332.061986] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16332.062260] obd_mod_free+0x46/0x140 [ptlrpc] [16332.062635] mdc_commit_open+0x91/0xb0 [mdc] [16332.062729] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16332.062942] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16332.063154] after_reply+0xe2c/0x1780 [ptlrpc] [16332.063361] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16332.068271] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16332.068342] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16332.068349] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16332.068352] RIP: 0010:native_safe_halt+0x2e/0x50 [16332.068364] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16332.068368] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16332.068372] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16332.068374] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16332.068377] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16332.068379] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16332.068381] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16332.068384] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16332.068387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16332.068390] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16332.068399] Call Trace: [16332.068404] [16332.068409] ? show_regs.cold.9+0x22/0x2f [16332.068418] ? watchdog_timer_fn.cold.6+0x62/0x10a [16332.068424] ? __hrtimer_run_queues+0x1c2/0x4d0 [16332.068429] ? watchdog+0x40/0x40 [16332.068433] ? hrtimer_interrupt+0x128/0x2c0 [16332.068437] ? smp_apic_timer_interrupt+0xa6/0x270 [16332.068443] ? apic_timer_interrupt+0xf/0x20 [16332.103087] ? __next_timer_interrupt+0x160/0x160 [16332.103110] ptlrpcd+0xa82/0xe40 [ptlrpc] [16332.103382] ? do_wait_intr+0xf0/0xf0 [16332.103392] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16332.115315] [16332.115325] ? native_safe_halt+0x2e/0x50 [16332.115335] kvm_wait+0xcd/0xe0 [16332.115342] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16332.115350] do_raw_spin_lock+0x177/0x190 [16332.115354] _raw_spin_lock+0x21/0x30 [16332.115357] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16332.115923] ? do_raw_spin_unlock+0x75/0x190 [16332.115930] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16332.116134] ? do_wait_intr+0xf0/0xf0 [16332.116144] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16332.116355] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16332.124223] kthread+0x1d1/0x200 [16332.124244] ? set_kthread_struct+0x70/0x70 [16332.124249] ret_from_fork+0x1f/0x30 [16332.507645] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16332.510905] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16332.514163] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16332.516103] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16332.518706] mdc_intent_lock+0x353/0x7e0 [mdc] [16332.520166] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16332.528640] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16332.532752] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16332.534610] lmv_intent_open+0x338/0x1150 [lmv] [16332.536984] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16332.542054] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16332.544864] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16332.546900] ll_intent_lock+0xc5/0xaa0 [lustre] [16332.548639] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16332.556821] ll_lookup_it+0xe0e/0x2270 [lustre] [16332.558288] ? do_raw_spin_unlock+0x75/0x190 [16332.562804] ? _raw_spin_unlock+0x12/0x30 [16332.565425] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16332.570634] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16332.575716] ll_atomic_open+0x36a/0xd70 [lustre] [16332.578505] lookup_open+0x356/0xb40 [16332.580376] do_last+0x313/0xfc0 [16332.581727] ? nd_jump_root+0xe5/0x160 [16332.583237] ? path_init+0x437/0x520 [16332.584666] path_openat+0xf7/0x500 [16332.586158] ? do_raw_spin_unlock+0x75/0x190 [16332.587823] do_filp_open+0x99/0x140 [16332.589218] ? getname_flags+0x6e/0x330 [16332.590906] ? __check_object_size+0xff/0x256 [16332.592662] ? do_raw_spin_unlock+0x75/0x190 [16332.594344] ? _raw_spin_unlock+0x12/0x30 [16332.595912] do_sys_openat2+0x2b4/0x410 [16332.597641] do_sys_open+0x73/0xa0 [16332.599094] __x64_sys_openat+0x24/0x30 [16332.600520] do_syscall_64+0xc1/0x440 [16332.601934] entry_SYSCALL_64_after_hwframe+0x49/0xae [16332.604210] RIP: 0033:0x7f53425a5332 [16332.605641] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16332.613230] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16332.616820] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16332.619888] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16332.622984] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16332.626195] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16332.628986] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16336.058344] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [16336.062024] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16336.097858] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16336.105929] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16336.110108] RIP: 0010:native_safe_halt+0x2e/0x50 [16336.113145] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16336.124176] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16336.127019] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16336.130144] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16336.137016] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16336.140911] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16336.144588] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16336.148032] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16336.151454] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16336.154257] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16336.157525] Call Trace: [16336.158546] [16336.159446] ? show_regs.cold.9+0x22/0x2f [16336.161717] ? watchdog_timer_fn.cold.6+0x62/0x10a [16336.164516] ? __hrtimer_run_queues+0x1c2/0x4d0 [16336.166564] ? watchdog+0x40/0x40 [16336.168415] ? hrtimer_interrupt+0x128/0x2c0 [16336.170680] ? smp_apic_timer_interrupt+0xa6/0x270 [16336.172193] ? apic_timer_interrupt+0xf/0x20 [16336.174138] [16336.175158] ? native_safe_halt+0x2e/0x50 [16336.177058] kvm_wait+0xcd/0xe0 [16336.178517] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16336.181070] do_raw_spin_lock+0x177/0x190 [16336.182892] _raw_spin_lock+0x21/0x30 [16336.184325] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16336.187253] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16336.189918] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16336.192722] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16336.195328] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16336.198002] ? woken_wake_function+0x30/0x30 [16336.200080] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16336.203229] kthread+0x1d1/0x200 [16336.204874] ? set_kthread_struct+0x70/0x70 [16336.207102] ret_from_fork+0x1f/0x30 [16340.052060] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [16340.056038] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16340.074292] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16340.078452] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16340.081611] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16340.086834] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16340.089737] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16340.096864] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16340.103796] RAX: 0000000000004c30 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16340.109506] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16340.113926] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16340.117919] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16340.122087] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16340.127714] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16340.132438] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16340.135818] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16340.139683] Call Trace: [16340.140713] [16340.141533] ? show_regs.cold.9+0x22/0x2f [16340.143061] ? watchdog_timer_fn.cold.6+0x62/0x10a [16340.145256] ? __hrtimer_run_queues+0x1c2/0x4d0 [16340.146946] ? watchdog+0x40/0x40 [16340.148554] ? hrtimer_interrupt+0x128/0x2c0 [16340.150159] ? smp_apic_timer_interrupt+0xa6/0x270 [16340.151996] ? apic_timer_interrupt+0xf/0x20 [16340.153738] [16340.154671] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16340.157521] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16340.160404] ? newidle_balance.isra.93+0x207/0x580 [16340.162292] do_raw_spin_lock+0x177/0x190 [16340.163952] _raw_spin_lock+0x21/0x30 [16340.165450] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16340.169460] process_one_work+0x2c8/0x700 [16340.171058] worker_thread+0x296/0x6e0 [16340.173452] ? rescuer_thread+0x570/0x570 [16340.175716] kthread+0x1d1/0x200 [16340.177179] ? set_kthread_struct+0x70/0x70 [16340.179163] ret_from_fork+0x1f/0x30 [16344.543787] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13707s! [16344.546925] Showing busy workqueues and worker pools: [16344.548888] workqueue events: flags=0x0 [16344.550538] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16344.553075] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16344.559113] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16344.562086] pending: cpu_latency_work [ptlrpc] [16344.564433] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16344.567103] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16344.570895] workqueue events_long: flags=0x0 [16344.572456] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16344.575326] pending: lu_objects_flush [obdclass] [16344.577880] workqueue events_unbound: flags=0x2 [16344.579754] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16344.582170] pending: flush_memcg_stats_dwork, idle_cull_fn [16344.584523] workqueue events_power_efficient: flags=0x80 [16344.586435] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16344.589482] pending: check_lifetime, do_cache_clean [16344.591841] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16344.594646] pending: neigh_periodic_work, neigh_periodic_work [16344.597666] workqueue mm_percpu_wq: flags=0x8 [16344.599531] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16344.602865] pending: vmstat_update, lru_add_drain_per_cpu [16344.605051] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16344.607798] pending: vmstat_update, lru_add_drain_per_cpu [16344.609871] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16344.613018] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16344.616097] workqueue xprtiod: flags=0xa [16344.617591] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16344.620102] pending: xprt_autoclose, xs_error_handle [16344.623441] workqueue ipv6_addrconf: flags=0x40008 [16344.625432] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16344.627888] pending: addrconf_verify_work [16344.629666] workqueue knbd0-recv: flags=0x1a [16344.631425] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16344.634590] in-flight: 651:recv_work [16344.636583] workqueue ptlrpc_pinger: flags=0x2 [16344.638667] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16344.641146] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16344.643826] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13707s workers=4 idle: 15979 31824 8 [16344.646724] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16335s workers=2 idle: 654 [16360.056512] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [16360.061195] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [16360.062178] Modules linked in: lustre(O) [16360.064743] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16360.064817] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16360.064824] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16360.064828] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16360.064843] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16360.064847] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16360.064852] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16360.064854] RDX: 00000000000010cf RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16360.064857] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16360.064858] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16360.064861] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16360.064864] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16360.064866] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16360.064868] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16360.064877] Call Trace: [16360.064883] [16360.064890] ? show_regs.cold.9+0x22/0x2f [16360.064903] ? watchdog_timer_fn.cold.6+0x62/0x10a [16360.064910] ? __hrtimer_run_queues+0x1c2/0x4d0 [16360.064914] ? watchdog+0x40/0x40 [16360.064919] ? hrtimer_interrupt+0x128/0x2c0 [16360.064924] ? smp_apic_timer_interrupt+0xa6/0x270 [16360.064934] ? apic_timer_interrupt+0xf/0x20 [16360.064938] [16360.064941] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16360.064947] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16360.064951] do_raw_spin_lock+0x177/0x190 [16360.065072] _raw_spin_lock+0x21/0x30 [16360.065079] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16360.065371] obd_mod_free+0x46/0x140 [ptlrpc] [16360.065833] mdc_commit_open+0x91/0xb0 [mdc] [16360.065873] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16360.066086] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16360.066284] after_reply+0xe2c/0x1780 [ptlrpc] [16360.068434] osp(O) [16360.087679] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16360.091160] ofd(O) lod(O) mdt(O) [16360.094872] ? __next_timer_interrupt+0x160/0x160 [16360.094893] ptlrpcd+0xa82/0xe40 [ptlrpc] [16360.095362] ? do_wait_intr+0xf0/0xf0 [16360.095376] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16360.097927] mdd(O) [16360.107371] kthread+0x1d1/0x200 [16360.107395] ? set_kthread_struct+0x70/0x70 [16360.107401] ret_from_fork+0x1f/0x30 [16360.223099] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16360.262222] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16360.270056] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16360.280802] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16360.286931] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16360.301229] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16360.303735] RAX: 0000000000007e0c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16360.313876] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16360.317712] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16360.325718] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16360.330105] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16360.333873] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16360.338329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16360.344781] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16360.350971] Call Trace: [16360.351885] [16360.352573] ? show_regs.cold.9+0x22/0x2f [16360.353867] ? watchdog_timer_fn.cold.6+0x62/0x10a [16360.362461] ? __hrtimer_run_queues+0x1c2/0x4d0 [16360.363861] ? watchdog+0x40/0x40 [16360.372696] ? hrtimer_interrupt+0x128/0x2c0 [16360.374720] ? smp_apic_timer_interrupt+0xa6/0x270 [16360.379267] ? apic_timer_interrupt+0xf/0x20 [16360.382241] [16360.384183] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16360.389882] do_raw_spin_lock+0x177/0x190 [16360.392119] _raw_spin_lock+0x21/0x30 [16360.394015] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16360.402405] ? do_raw_spin_unlock+0x75/0x190 [16360.407223] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16360.413297] ? do_wait_intr+0xf0/0xf0 [16360.418625] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16360.425363] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16360.429469] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16360.436911] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16360.440298] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16360.444694] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16360.446845] mdc_intent_lock+0x353/0x7e0 [mdc] [16360.449477] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16360.459690] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16360.467446] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16360.472456] lmv_intent_open+0x338/0x1150 [lmv] [16360.478718] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16360.482378] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16360.487348] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16360.491246] ll_intent_lock+0xc5/0xaa0 [lustre] [16360.497422] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16360.506065] ll_lookup_it+0xe0e/0x2270 [lustre] [16360.511533] ? do_raw_spin_unlock+0x75/0x190 [16360.518309] ? _raw_spin_unlock+0x12/0x30 [16360.523151] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16360.528408] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16360.532454] ll_atomic_open+0x36a/0xd70 [lustre] [16360.537479] lookup_open+0x356/0xb40 [16360.544504] do_last+0x313/0xfc0 [16360.545660] ? nd_jump_root+0xe5/0x160 [16360.553106] ? path_init+0x437/0x520 [16360.557392] path_openat+0xf7/0x500 [16360.561337] ? do_raw_spin_unlock+0x75/0x190 [16360.565504] do_filp_open+0x99/0x140 [16360.569960] ? getname_flags+0x6e/0x330 [16360.571678] ? __check_object_size+0xff/0x256 [16360.576602] ? do_raw_spin_unlock+0x75/0x190 [16360.578355] ? _raw_spin_unlock+0x12/0x30 [16360.581491] do_sys_openat2+0x2b4/0x410 [16360.583838] do_sys_open+0x73/0xa0 [16360.586720] __x64_sys_openat+0x24/0x30 [16360.596166] do_syscall_64+0xc1/0x440 [16360.603393] entry_SYSCALL_64_after_hwframe+0x49/0xae [16360.605219] RIP: 0033:0x7f53425a5332 [16360.606503] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16360.617597] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16360.620804] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16360.623838] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16360.626198] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16360.630820] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16360.633790] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16364.058937] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [16364.062288] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16364.082874] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16364.087634] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16364.091303] RIP: 0010:native_safe_halt+0x2e/0x50 [16364.092930] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16364.100354] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16364.103121] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16364.106028] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16364.108752] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16364.111293] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16364.114262] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16364.116818] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16364.120071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16364.122380] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16364.125175] Call Trace: [16364.126377] [16364.127022] ? show_regs.cold.9+0x22/0x2f [16364.128613] ? watchdog_timer_fn.cold.6+0x62/0x10a [16364.130564] ? __hrtimer_run_queues+0x1c2/0x4d0 [16364.132820] ? watchdog+0x40/0x40 [16364.134413] ? hrtimer_interrupt+0x128/0x2c0 [16364.137089] ? smp_apic_timer_interrupt+0xa6/0x270 [16364.139095] ? apic_timer_interrupt+0xf/0x20 [16364.143644] [16364.144481] ? native_safe_halt+0x2e/0x50 [16364.146638] kvm_wait+0xcd/0xe0 [16364.148119] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16364.151819] do_raw_spin_lock+0x177/0x190 [16364.154214] _raw_spin_lock+0x21/0x30 [16364.156514] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16364.160012] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16364.162245] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16364.164617] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16364.166854] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16364.169284] ? woken_wake_function+0x30/0x30 [16364.170835] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16364.173427] kthread+0x1d1/0x200 [16364.174647] ? set_kthread_struct+0x70/0x70 [16364.176780] ret_from_fork+0x1f/0x30 [16372.055239] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [16372.060968] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16372.110744] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16372.125160] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16372.136959] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16372.143976] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16372.149520] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16372.162911] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16372.166803] RAX: 0000000000007c84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16372.169657] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16372.174599] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16372.178848] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16372.180745] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16372.182974] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16372.185273] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16372.187826] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16372.191066] Call Trace: [16372.192130] [16372.192929] ? show_regs.cold.9+0x22/0x2f [16372.194337] ? watchdog_timer_fn.cold.6+0x62/0x10a [16372.197057] ? __hrtimer_run_queues+0x1c2/0x4d0 [16372.198936] ? watchdog+0x40/0x40 [16372.200694] ? hrtimer_interrupt+0x128/0x2c0 [16372.204236] ? smp_apic_timer_interrupt+0xa6/0x270 [16372.208538] ? apic_timer_interrupt+0xf/0x20 [16372.216895] [16372.219883] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16372.224999] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16372.229073] ? newidle_balance.isra.93+0x207/0x580 [16372.232141] do_raw_spin_lock+0x177/0x190 [16372.234432] _raw_spin_lock+0x21/0x30 [16372.239971] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16372.250971] process_one_work+0x2c8/0x700 [16372.256037] worker_thread+0x296/0x6e0 [16372.261992] ? rescuer_thread+0x570/0x570 [16372.266824] kthread+0x1d1/0x200 [16372.271461] ? set_kthread_struct+0x70/0x70 [16372.273427] ret_from_fork+0x1f/0x30 [16374.751936] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13737s! [16374.756307] Showing busy workqueues and worker pools: [16374.758848] workqueue events: flags=0x0 [16374.760124] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16374.763151] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16374.769155] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16374.771883] pending: cpu_latency_work [ptlrpc] [16374.774395] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16374.778122] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16374.782760] workqueue events_long: flags=0x0 [16374.783023] rcu: INFO: rcu_sched self-detected stall on CPU [16374.784491] pwq 0: [16374.788346] rcu: 2-....: (7952725 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [16374.788365] (t=14280357 jiffies g=274185 q=67621) [16374.788370] rcu: rcu_sched kthread starved for 13738723 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [16374.788374] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [16374.788376] rcu: RCU grace-period kthread stack dump: [16374.788378] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [16374.788386] Call Trace: [16374.788394] __schedule+0x351/0xcb0 [16374.788409] schedule+0xc0/0x180 [16374.788414] schedule_timeout+0xb4/0x190 [16374.788419] ? do_raw_spin_unlock+0x75/0x190 [16374.788426] ? __next_timer_interrupt+0x160/0x160 [16374.788434] ? prepare_to_swait_event+0xf3/0x140 [16374.788441] rcu_gp_kthread+0x8dd/0x15b0 [16374.788447] ? force_qs_rnp+0x2c0/0x2c0 [16374.788451] kthread+0x1d1/0x200 [16374.788455] ? set_kthread_struct+0x70/0x70 [16374.788466] ret_from_fork+0x1f/0x30 [16374.788474] rcu: Stack dump where RCU GP kthread last ran: [16374.788476] Sending NMI from CPU 2 to CPUs 1: [16374.788562] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [16374.789734] Sending NMI from CPU 2 to CPUs 1: [16374.791781] cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16374.791798] pending: lu_objects_flush [obdclass] [16374.791986] workqueue events_unbound: flags=0x2 [16374.791990] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16374.792009] pending: flush_memcg_stats_dwork, idle_cull_fn [16374.792034] workqueue events_power_efficient: flags=0x80 [16374.792038] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16374.792046] pending: check_lifetime, do_cache_clean [16374.792058] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16374.792066] pending: neigh_periodic_work, neigh_periodic_work [16374.792079] workqueue mm_percpu_wq: flags=0x8 [16374.792082] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16374.792088] pending: vmstat_update, lru_add_drain_per_cpu [16374.792100] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16374.792106] pending: vmstat_update, lru_add_drain_per_cpu [16374.792114] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16374.792120] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16374.792138] workqueue xprtiod: flags=0xa [16374.792145] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16374.792150] pending: xprt_autoclose, xs_error_handle [16374.792166] workqueue ipv6_addrconf: flags=0x40008 [16374.792168] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16374.792175] pending: addrconf_verify_work [16374.792186] workqueue knbd0-recv: flags=0x1a [16374.792188] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16374.792195] in-flight: 651:recv_work [16374.792205] workqueue ptlrpc_pinger: flags=0x2 [16374.792207] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16374.792212] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16374.792471] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13737s workers=4 idle: 15979 31824 8 [16374.792483] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16365s workers=2 idle: 654 [16374.934534] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [16374.937410] NMI backtrace for cpu 2 [16374.939167] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16374.943867] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16374.947652] Call Trace: [16374.948896] [16374.950123] ? dump_stack+0xbb/0x10e [16374.952457] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16374.956262] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16374.960548] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16374.965942] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16374.971902] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16374.975505] ? rcu_dump_cpu_stacks+0x11b/0x171 [16374.979488] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [16374.981803] ? update_process_times+0x6d/0xc0 [16374.984796] ? tick_sched_handle.isra.18+0x27/0x90 [16374.986972] ? tick_sched_timer+0x53/0xb0 [16374.988968] ? __hrtimer_run_queues+0x1c2/0x4d0 [16374.991900] ? tick_sched_handle.isra.18+0x90/0x90 [16374.993638] ? hrtimer_interrupt+0x128/0x2c0 [16374.994855] ? smp_apic_timer_interrupt+0xa6/0x270 [16374.996126] ? apic_timer_interrupt+0xf/0x20 [16374.997332] [16374.999501] ? native_safe_halt+0x2e/0x50 [16375.000929] ? kvm_wait+0xcd/0xe0 [16375.002082] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16375.004896] ? do_raw_spin_lock+0x177/0x190 [16375.006603] ? _raw_spin_lock+0x21/0x30 [16375.008265] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16375.009921] ? obd_mod_free+0x46/0x140 [ptlrpc] [16375.011610] ? mdc_commit_open+0x91/0xb0 [mdc] [16375.016659] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16375.023216] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16375.025472] ? after_reply+0xe2c/0x1780 [ptlrpc] [16375.028449] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16375.030445] ? __next_timer_interrupt+0x160/0x160 [16375.032279] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16375.034170] ? do_wait_intr+0xf0/0xf0 [16375.035765] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16375.038268] ? kthread+0x1d1/0x200 [16375.039988] ? set_kthread_struct+0x70/0x70 [16375.041385] ? ret_from_fork+0x1f/0x30 [16392.058071] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [16392.062075] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16392.107198] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16392.114111] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16392.126554] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16392.130517] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16392.147708] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16392.153092] RAX: 0000000000005164 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16392.157338] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16392.163066] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16392.167573] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16392.172099] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16392.175804] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16392.186622] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16392.195562] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16392.203863] Call Trace: [16392.205364] [16392.206745] ? show_regs.cold.9+0x22/0x2f [16392.208738] ? watchdog_timer_fn.cold.6+0x62/0x10a [16392.212117] ? __hrtimer_run_queues+0x1c2/0x4d0 [16392.219491] ? watchdog+0x40/0x40 [16392.222948] ? hrtimer_interrupt+0x128/0x2c0 [16392.227941] ? smp_apic_timer_interrupt+0xa6/0x270 [16392.232606] ? apic_timer_interrupt+0xf/0x20 [16392.238651] [16392.239667] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16392.244280] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16392.250868] do_raw_spin_lock+0x177/0x190 [16392.252214] _raw_spin_lock+0x21/0x30 [16392.254628] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16392.258376] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16392.260734] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16392.264295] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16392.269238] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16392.274322] ? woken_wake_function+0x30/0x30 [16392.277647] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16392.287297] kthread+0x1d1/0x200 [16392.290505] ? set_kthread_struct+0x70/0x70 [16392.293700] ret_from_fork+0x1f/0x30 [16400.056090] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [16400.057185] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [16400.058072] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [16400.058080] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16400.058206] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16400.058212] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16400.058215] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16400.058229] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16400.058234] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16400.058238] RAX: 0000000000006286 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16400.058241] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16400.058243] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16400.058245] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16400.058248] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16400.058251] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16400.058407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16400.058409] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16400.058418] Call Trace: [16400.058424] [16400.058430] ? show_regs.cold.9+0x22/0x2f [16400.058442] ? watchdog_timer_fn.cold.6+0x62/0x10a [16400.058449] ? __hrtimer_run_queues+0x1c2/0x4d0 [16400.058453] ? watchdog+0x40/0x40 [16400.058457] ? hrtimer_interrupt+0x128/0x2c0 [16400.058461] ? smp_apic_timer_interrupt+0xa6/0x270 [16400.058470] ? apic_timer_interrupt+0xf/0x20 [16400.058476] [16400.058479] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16400.058486] do_raw_spin_lock+0x177/0x190 [16400.058492] _raw_spin_lock+0x21/0x30 [16400.058497] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16400.058903] ? do_raw_spin_unlock+0x75/0x190 [16400.058909] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16400.059167] ? do_wait_intr+0xf0/0xf0 [16400.059176] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16400.059516] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16400.059786] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16400.059994] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16400.060343] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16400.060448] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16400.060477] mdc_intent_lock+0x353/0x7e0 [mdc] [16400.060506] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16400.061562] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16400.062960] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16400.062998] lmv_intent_open+0x338/0x1150 [lmv] [16400.063036] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16400.063159] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16400.063241] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16400.063277] ll_intent_lock+0xc5/0xaa0 [lustre] [16400.065085] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16400.065162] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16400.065170] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16400.065702] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16400.066039] ll_lookup_it+0xe0e/0x2270 [lustre] [16400.065177] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16400.067188] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16400.067203] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16400.067207] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16400.067212] RAX: 00000000000075a1 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16400.067214] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16400.067216] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16400.067219] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16400.067221] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16400.067224] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16400.067227] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16400.067229] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16400.067240] Call Trace: [16400.067245] [16400.067249] ? show_regs.cold.9+0x22/0x2f [16400.067295] ? watchdog_timer_fn.cold.6+0x62/0x10a [16400.067301] ? __hrtimer_run_queues+0x1c2/0x4d0 [16400.067306] ? watchdog+0x40/0x40 [16400.067310] ? hrtimer_interrupt+0x128/0x2c0 [16400.067314] ? smp_apic_timer_interrupt+0xa6/0x270 [16400.067322] ? apic_timer_interrupt+0xf/0x20 [16400.067327] [16400.067330] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16400.067335] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16400.067338] ? newidle_balance.isra.93+0x207/0x580 [16400.067344] do_raw_spin_lock+0x177/0x190 [16400.067349] _raw_spin_lock+0x21/0x30 [16400.067354] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16400.070724] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16400.070801] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16400.070808] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16400.070811] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16400.070823] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16400.070827] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16400.070832] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16400.070835] RDX: 000000000000509e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16400.070837] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16400.070840] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16400.070843] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16400.070846] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16400.070849] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16400.070852] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16400.070860] Call Trace: [16400.070864] [16400.070868] ? show_regs.cold.9+0x22/0x2f [16400.070877] ? watchdog_timer_fn.cold.6+0x62/0x10a [16400.070882] ? __hrtimer_run_queues+0x1c2/0x4d0 [16400.070886] ? watchdog+0x40/0x40 [16400.070890] ? hrtimer_interrupt+0x128/0x2c0 [16400.070894] ? smp_apic_timer_interrupt+0xa6/0x270 [16400.070902] ? apic_timer_interrupt+0xf/0x20 [16400.070907] [16400.070910] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16400.070915] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16400.070920] do_raw_spin_lock+0x177/0x190 [16400.070924] _raw_spin_lock+0x21/0x30 [16400.069980] process_one_work+0x2c8/0x700 [16400.071336] worker_thread+0x296/0x6e0 [16400.071341] ? rescuer_thread+0x570/0x570 [16400.071346] kthread+0x1d1/0x200 [16400.071350] ? set_kthread_struct+0x70/0x70 [16400.071354] ret_from_fork+0x1f/0x30 [16400.070930] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16400.071710] obd_mod_free+0x46/0x140 [ptlrpc] [16400.071909] mdc_commit_open+0x91/0xb0 [mdc] [16400.071950] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16400.072186] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16400.082000] ? do_raw_spin_unlock+0x75/0x190 [16400.082024] ? _raw_spin_unlock+0x12/0x30 [16400.082031] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16400.082060] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16400.082082] ll_atomic_open+0x36a/0xd70 [lustre] [16400.082202] lookup_open+0x356/0xb40 [16400.082212] do_last+0x313/0xfc0 [16400.082217] ? nd_jump_root+0xe5/0x160 [16400.082222] ? path_init+0x437/0x520 [16400.082227] path_openat+0xf7/0x500 [16400.082232] ? do_raw_spin_unlock+0x75/0x190 [16400.082237] do_filp_open+0x99/0x140 [16400.082244] ? getname_flags+0x6e/0x330 [16400.082249] ? __check_object_size+0xff/0x256 [16400.083044] ? do_raw_spin_unlock+0x75/0x190 [16400.083049] ? _raw_spin_unlock+0x12/0x30 [16400.083054] do_sys_openat2+0x2b4/0x410 [16400.083058] do_sys_open+0x73/0xa0 [16400.084936] __x64_sys_openat+0x24/0x30 [16400.084941] do_syscall_64+0xc1/0x440 [16400.084947] entry_SYSCALL_64_after_hwframe+0x49/0xae [16400.084955] RIP: 0033:0x7f53425a5332 [16400.084964] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16400.084968] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16400.084973] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16400.084975] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16400.084977] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16400.084979] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16400.084981] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16400.939415] after_reply+0xe2c/0x1780 [ptlrpc] [16400.941748] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16400.946600] ? __next_timer_interrupt+0x160/0x160 [16400.948201] ptlrpcd+0xa82/0xe40 [ptlrpc] [16400.951979] ? do_wait_intr+0xf0/0xf0 [16400.954512] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16400.956910] kthread+0x1d1/0x200 [16400.958612] ? set_kthread_struct+0x70/0x70 [16400.960129] ret_from_fork+0x1f/0x30 [16404.973762] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13767s! [16404.979863] Showing busy workqueues and worker pools: [16404.985405] workqueue events: flags=0x0 [16404.987807] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16404.991984] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16405.003234] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16405.014096] pending: cpu_latency_work [ptlrpc] [16405.020017] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16405.023006] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16405.033022] workqueue events_long: flags=0x0 [16405.036042] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16405.040249] pending: lu_objects_flush [obdclass] [16405.050700] workqueue events_unbound: flags=0x2 [16405.057413] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16405.061142] pending: flush_memcg_stats_dwork, idle_cull_fn [16405.066101] workqueue events_power_efficient: flags=0x80 [16405.071829] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16405.077025] pending: check_lifetime, do_cache_clean [16405.084979] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16405.089762] pending: neigh_periodic_work, neigh_periodic_work [16405.099078] workqueue mm_percpu_wq: flags=0x8 [16405.103582] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16405.111143] pending: vmstat_update, lru_add_drain_per_cpu [16405.120122] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16405.131760] pending: vmstat_update, lru_add_drain_per_cpu [16405.142105] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16405.151893] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16405.161352] workqueue xprtiod: flags=0xa [16405.162835] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16405.171973] pending: xprt_autoclose, xs_error_handle [16405.185450] workqueue ipv6_addrconf: flags=0x40008 [16405.189701] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16405.192516] pending: addrconf_verify_work [16405.193759] workqueue knbd0-recv: flags=0x1a [16405.201684] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16405.203936] in-flight: 651:recv_work [16405.213532] workqueue ptlrpc_pinger: flags=0x2 [16405.219703] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16405.221959] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16405.239363] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13768s workers=4 idle: 15979 31824 8 [16405.248970] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16395s workers=2 idle: 654 [16420.059612] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16420.072797] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16420.111500] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16420.126755] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16420.131364] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16420.138364] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16420.153610] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16420.155805] RAX: 0000000000003f85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16420.159583] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16420.164859] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16420.169098] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16420.174805] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16420.179498] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16420.184603] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16420.190622] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16420.197118] Call Trace: [16420.197893] [16420.205118] ? show_regs.cold.9+0x22/0x2f [16420.211702] ? watchdog_timer_fn.cold.6+0x62/0x10a [16420.219789] ? __hrtimer_run_queues+0x1c2/0x4d0 [16420.228232] ? watchdog+0x40/0x40 [16420.229453] ? hrtimer_interrupt+0x128/0x2c0 [16420.230768] ? smp_apic_timer_interrupt+0xa6/0x270 [16420.244771] ? apic_timer_interrupt+0xf/0x20 [16420.248422] [16420.249504] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16420.252265] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16420.255663] do_raw_spin_lock+0x177/0x190 [16420.256835] _raw_spin_lock+0x21/0x30 [16420.259583] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16420.266157] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16420.277897] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16420.279486] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16420.280894] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16420.287668] ? woken_wake_function+0x30/0x30 [16420.292081] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16420.299534] kthread+0x1d1/0x200 [16420.304159] ? set_kthread_struct+0x70/0x70 [16420.315395] ret_from_fork+0x1f/0x30 [16428.060198] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [16428.060718] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [16428.063002] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) [16428.073266] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16428.073528] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16428.073535] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16428.073538] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16428.073552] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16428.073556] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16428.073561] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16428.073564] RDX: 0000000000007c74 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16428.073566] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16428.073568] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16428.073571] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16428.073575] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16428.073577] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16428.073580] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16428.073588] Call Trace: [16428.073809] [16428.073815] ? show_regs.cold.9+0x22/0x2f [16428.073824] ? watchdog_timer_fn.cold.6+0x62/0x10a [16428.073829] ? __hrtimer_run_queues+0x1c2/0x4d0 [16428.073832] ? watchdog+0x40/0x40 [16428.073837] ? hrtimer_interrupt+0x128/0x2c0 [16428.073840] ? smp_apic_timer_interrupt+0xa6/0x270 [16428.073847] ? apic_timer_interrupt+0xf/0x20 [16428.073851] [16428.073854] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16428.073858] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16428.073862] do_raw_spin_lock+0x177/0x190 [16428.073867] _raw_spin_lock+0x21/0x30 [16428.073873] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16428.074257] obd_mod_free+0x46/0x140 [ptlrpc] [16428.074478] mdc_commit_open+0x91/0xb0 [mdc] [16428.090228] ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16428.090285] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16428.090292] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16428.090295] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16428.090310] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16428.090315] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16428.090320] RAX: 00000000000052f5 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16428.090322] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16428.090325] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16428.090327] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16428.090330] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16428.090333] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16428.090335] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16428.090338] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16428.090357] Call Trace: [16428.090365] [16428.090372] ? show_regs.cold.9+0x22/0x2f [16428.090383] ? watchdog_timer_fn.cold.6+0x62/0x10a [16428.090389] ? __hrtimer_run_queues+0x1c2/0x4d0 [16428.090393] ? watchdog+0x40/0x40 [16428.090398] ? hrtimer_interrupt+0x128/0x2c0 [16428.090402] ? smp_apic_timer_interrupt+0xa6/0x270 [16428.090412] ? apic_timer_interrupt+0xf/0x20 [16428.090417] [16428.090420] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16428.090426] do_raw_spin_lock+0x177/0x190 [16428.090431] _raw_spin_lock+0x21/0x30 [16428.090437] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16428.090715] ? do_raw_spin_unlock+0x75/0x190 [16428.090721] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16428.090930] ? do_wait_intr+0xf0/0xf0 [16428.090938] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16428.091134] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16428.091348] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16428.091552] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16428.091656] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16428.091698] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16428.091727] mdc_intent_lock+0x353/0x7e0 [mdc] [16428.091755] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16428.092471] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16428.092708] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16428.092744] lmv_intent_open+0x338/0x1150 [lmv] [16428.092858] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16428.092946] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16428.093027] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16428.093052] ll_intent_lock+0xc5/0xaa0 [lustre] [16428.093139] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16428.093222] ll_lookup_it+0xe0e/0x2270 [lustre] [16428.093305] ? do_raw_spin_unlock+0x75/0x190 [16428.093314] ? _raw_spin_unlock+0x12/0x30 [16428.093321] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16428.093350] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16428.093370] ll_atomic_open+0x36a/0xd70 [lustre] [16428.093452] lookup_open+0x356/0xb40 [16428.093462] do_last+0x313/0xfc0 [16428.093466] ? nd_jump_root+0xe5/0x160 [16428.093471] ? path_init+0x437/0x520 [16428.093475] path_openat+0xf7/0x500 [16428.093480] ? do_raw_spin_unlock+0x75/0x190 [16428.093484] do_filp_open+0x99/0x140 [16428.093491] ? getname_flags+0x6e/0x330 [16428.093495] ? __check_object_size+0xff/0x256 [16428.093503] ? do_raw_spin_unlock+0x75/0x190 [16428.093507] ? _raw_spin_unlock+0x12/0x30 [16428.093512] do_sys_openat2+0x2b4/0x410 [16428.093516] do_sys_open+0x73/0xa0 [16428.093520] __x64_sys_openat+0x24/0x30 [16428.093524] do_syscall_64+0xc1/0x440 [16428.093531] entry_SYSCALL_64_after_hwframe+0x49/0xae [16428.093537] RIP: 0033:0x7f53425a5332 [16428.093544] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16428.093548] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16428.093554] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16428.093556] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16428.093558] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16428.093560] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16428.093563] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16428.629191] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16428.631980] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16428.634607] after_reply+0xe2c/0x1780 [ptlrpc] [16428.636447] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16428.641221] ? __next_timer_interrupt+0x160/0x160 [16428.644354] ptlrpcd+0xa82/0xe40 [ptlrpc] [16428.646568] ? do_wait_intr+0xf0/0xf0 [16428.650242] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16428.653356] kthread+0x1d1/0x200 [16428.654675] ? set_kthread_struct+0x70/0x70 [16428.656165] ret_from_fork+0x1f/0x30 [16432.054804] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [16432.060585] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16432.112675] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16432.126008] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16432.136672] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16432.143132] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16432.147350] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16432.169008] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16432.177868] RAX: 0000000000002984 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16432.183035] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16432.190250] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16432.201626] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16432.208667] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16432.217974] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16432.231487] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16432.239971] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16432.252312] Call Trace: [16432.252967] [16432.253530] ? show_regs.cold.9+0x22/0x2f [16432.257782] ? watchdog_timer_fn.cold.6+0x62/0x10a [16432.259319] ? __hrtimer_run_queues+0x1c2/0x4d0 [16432.267746] ? watchdog+0x40/0x40 [16432.268676] ? hrtimer_interrupt+0x128/0x2c0 [16432.269702] ? smp_apic_timer_interrupt+0xa6/0x270 [16432.274063] ? apic_timer_interrupt+0xf/0x20 [16432.276057] [16432.276644] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16432.278328] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16432.281626] ? newidle_balance.isra.93+0x207/0x580 [16432.284247] do_raw_spin_lock+0x177/0x190 [16432.285331] _raw_spin_lock+0x21/0x30 [16432.290971] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16432.294494] process_one_work+0x2c8/0x700 [16432.296005] worker_thread+0x296/0x6e0 [16432.297186] ? rescuer_thread+0x570/0x570 [16432.298048] kthread+0x1d1/0x200 [16432.299127] ? set_kthread_struct+0x70/0x70 [16432.300777] ret_from_fork+0x1f/0x30 [16435.680652] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13798s! [16435.688376] Showing busy workqueues and worker pools: [16435.690484] workqueue events: flags=0x0 [16435.693148] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16435.704232] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16435.709499] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16435.713233] pending: cpu_latency_work [ptlrpc] [16435.716790] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16435.721090] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16435.727081] workqueue events_long: flags=0x0 [16435.729086] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16435.731805] pending: lu_objects_flush [obdclass] [16435.733900] workqueue events_unbound: flags=0x2 [16435.735480] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16435.739496] pending: flush_memcg_stats_dwork, idle_cull_fn [16435.742404] workqueue events_power_efficient: flags=0x80 [16435.746936] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16435.750524] pending: check_lifetime, do_cache_clean [16435.752906] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16435.755714] pending: neigh_periodic_work, neigh_periodic_work [16435.759260] workqueue mm_percpu_wq: flags=0x8 [16435.761166] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16435.764506] pending: vmstat_update, lru_add_drain_per_cpu [16435.767049] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16435.769839] pending: vmstat_update, lru_add_drain_per_cpu [16435.772580] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16435.776697] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16435.781162] workqueue xprtiod: flags=0xa [16435.783617] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16435.786575] pending: xprt_autoclose, xs_error_handle [16435.790180] workqueue ipv6_addrconf: flags=0x40008 [16435.792239] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16435.795199] pending: addrconf_verify_work [16435.801870] workqueue knbd0-recv: flags=0x1a [16435.807097] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16435.809176] in-flight: 651:recv_work [16435.813490] workqueue ptlrpc_pinger: flags=0x2 [16435.816085] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16435.818420] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16435.820897] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13798s workers=4 idle: 15979 31824 8 [16435.823843] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16426s workers=2 idle: 654 [16448.058960] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16448.063690] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16448.087711] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16448.093986] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16448.098813] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16448.101182] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16448.112760] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16448.116729] RAX: 00000000000037f1 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16448.120232] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16448.123472] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16448.126499] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16448.128761] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16448.133493] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16448.137758] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16448.140613] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16448.144821] Call Trace: [16448.146334] [16448.147177] ? show_regs.cold.9+0x22/0x2f [16448.149683] ? watchdog_timer_fn.cold.6+0x62/0x10a [16448.152076] ? __hrtimer_run_queues+0x1c2/0x4d0 [16448.154959] ? watchdog+0x40/0x40 [16448.156543] ? hrtimer_interrupt+0x128/0x2c0 [16448.159206] ? smp_apic_timer_interrupt+0xa6/0x270 [16448.161727] ? apic_timer_interrupt+0xf/0x20 [16448.164156] [16448.165344] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16448.168110] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16448.172186] do_raw_spin_lock+0x177/0x190 [16448.175528] _raw_spin_lock+0x21/0x30 [16448.177633] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16448.179734] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16448.181971] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16448.184598] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16448.187666] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16448.190205] ? woken_wake_function+0x30/0x30 [16448.194270] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16448.196229] kthread+0x1d1/0x200 [16448.199072] ? set_kthread_struct+0x70/0x70 [16448.201613] ret_from_fork+0x1f/0x30 [16456.056032] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [16456.060192] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [16456.063220] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 [16456.066980] Modules linked in: [16456.066988] auth_rpcgss [16456.076152] lustre(O) [16456.077507] nfsv4 [16456.078801] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16456.078885] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16456.078895] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16456.078898] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16456.078919] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16456.078924] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16456.078929] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16456.078932] RDX: 0000000000004aac RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16456.078934] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16456.078936] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16456.078938] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16456.078942] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16456.078944] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16456.078947] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16456.078956] Call Trace: [16456.078965] [16456.078972] ? show_regs.cold.9+0x22/0x2f [16456.078985] ? watchdog_timer_fn.cold.6+0x62/0x10a [16456.078992] ? __hrtimer_run_queues+0x1c2/0x4d0 [16456.078997] ? watchdog+0x40/0x40 [16456.079001] ? hrtimer_interrupt+0x128/0x2c0 [16456.079005] ? smp_apic_timer_interrupt+0xa6/0x270 [16456.079015] ? apic_timer_interrupt+0xf/0x20 [16456.079020] [16456.079024] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16456.079030] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16456.079034] do_raw_spin_lock+0x177/0x190 [16456.079039] _raw_spin_lock+0x21/0x30 [16456.079047] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16456.081510] dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16456.081549] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16456.081555] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16456.081558] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16456.081571] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16456.081576] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16456.081581] RAX: 0000000000001617 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16456.081583] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16456.081586] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16456.081588] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16456.081591] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16456.081595] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16456.081598] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16456.081600] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16456.081610] Call Trace: [16456.081616] [16456.081622] ? show_regs.cold.9+0x22/0x2f [16456.081632] ? watchdog_timer_fn.cold.6+0x62/0x10a [16456.081639] ? __hrtimer_run_queues+0x1c2/0x4d0 [16456.081644] ? watchdog+0x40/0x40 [16456.081648] ? hrtimer_interrupt+0x128/0x2c0 [16456.081653] ? smp_apic_timer_interrupt+0xa6/0x270 [16456.081662] ? apic_timer_interrupt+0xf/0x20 [16456.081667] [16456.081669] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16456.081675] do_raw_spin_lock+0x177/0x190 [16456.081680] _raw_spin_lock+0x21/0x30 [16456.081684] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16456.082169] ? do_raw_spin_unlock+0x75/0x190 [16456.082176] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16456.082388] ? do_wait_intr+0xf0/0xf0 [16456.082401] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16456.082600] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16456.083018] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16456.083238] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16456.083716] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16456.083758] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16456.083784] mdc_intent_lock+0x353/0x7e0 [mdc] [16456.083877] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16456.084045] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16456.084608] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16456.084641] lmv_intent_open+0x338/0x1150 [lmv] [16456.084679] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16456.084839] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16456.085005] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16456.085033] ll_intent_lock+0xc5/0xaa0 [lustre] [16456.085126] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16456.085220] ll_lookup_it+0xe0e/0x2270 [lustre] [16456.085321] ? do_raw_spin_unlock+0x75/0x190 [16456.085332] ? _raw_spin_unlock+0x12/0x30 [16456.085341] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16456.085377] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16456.085398] ll_atomic_open+0x36a/0xd70 [lustre] [16456.085489] lookup_open+0x356/0xb40 [16456.085499] do_last+0x313/0xfc0 [16456.085504] ? nd_jump_root+0xe5/0x160 [16456.085509] ? path_init+0x437/0x520 [16456.085513] path_openat+0xf7/0x500 [16456.085519] ? do_raw_spin_unlock+0x75/0x190 [16456.085523] do_filp_open+0x99/0x140 [16456.085530] ? getname_flags+0x6e/0x330 [16456.085535] ? __check_object_size+0xff/0x256 [16456.085543] ? do_raw_spin_unlock+0x75/0x190 [16456.085547] ? _raw_spin_unlock+0x12/0x30 [16456.085552] do_sys_openat2+0x2b4/0x410 [16456.085556] do_sys_open+0x73/0xa0 [16456.085561] __x64_sys_openat+0x24/0x30 [16456.085564] do_syscall_64+0xc1/0x440 [16456.085571] entry_SYSCALL_64_after_hwframe+0x49/0xae [16456.085578] RIP: 0033:0x7f53425a5332 [16456.087210] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16456.107757] obd_mod_free+0x46/0x140 [ptlrpc] [16456.110890] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 [16456.114409] mdc_commit_open+0x91/0xb0 [mdc] [16456.115984] ORIG_RAX: 0000000000000101 [16456.124066] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16456.127230] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16456.127244] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16456.127246] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16456.127248] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16456.127250] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16456.469029] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16456.476291] after_reply+0xe2c/0x1780 [ptlrpc] [16456.480351] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16456.486027] ? __next_timer_interrupt+0x160/0x160 [16456.491534] ptlrpcd+0xa82/0xe40 [ptlrpc] [16456.500439] ? do_wait_intr+0xf0/0xf0 [16456.502943] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16456.511547] kthread+0x1d1/0x200 [16456.517982] ? set_kthread_struct+0x70/0x70 [16456.520021] ret_from_fork+0x1f/0x30 [16460.056240] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [16460.064431] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16460.092148] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16460.096600] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16460.100046] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16460.102811] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16460.105133] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16460.113280] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16460.116611] RAX: 0000000000003123 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16460.119418] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16460.122079] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16460.130694] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16460.133893] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16460.141428] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16460.146588] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16460.149549] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16460.153427] Call Trace: [16460.154777] [16460.156779] ? show_regs.cold.9+0x22/0x2f [16460.159655] ? watchdog_timer_fn.cold.6+0x62/0x10a [16460.162431] ? __hrtimer_run_queues+0x1c2/0x4d0 [16460.166509] ? watchdog+0x40/0x40 [16460.167631] ? hrtimer_interrupt+0x128/0x2c0 [16460.171637] ? smp_apic_timer_interrupt+0xa6/0x270 [16460.179016] ? apic_timer_interrupt+0xf/0x20 [16460.183881] [16460.186968] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16460.193341] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16460.197248] ? newidle_balance.isra.93+0x207/0x580 [16460.202928] do_raw_spin_lock+0x177/0x190 [16460.206746] _raw_spin_lock+0x21/0x30 [16460.210568] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16460.216027] process_one_work+0x2c8/0x700 [16460.221301] worker_thread+0x296/0x6e0 [16460.224070] ? rescuer_thread+0x570/0x570 [16460.228915] kthread+0x1d1/0x200 [16460.230855] ? set_kthread_struct+0x70/0x70 [16460.233038] ret_from_fork+0x1f/0x30 [16465.888489] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13828s! [16465.897426] Showing busy workqueues and worker pools: [16465.901976] workqueue events: flags=0x0 [16465.905746] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16465.909049] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16465.919133] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16465.928930] pending: cpu_latency_work [ptlrpc] [16465.934013] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16465.940231] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16465.954543] workqueue events_long: flags=0x0 [16465.956771] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16465.967023] pending: lu_objects_flush [obdclass] [16465.972774] workqueue events_unbound: flags=0x2 [16465.978412] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16465.987804] pending: flush_memcg_stats_dwork, idle_cull_fn [16465.997801] workqueue events_power_efficient: flags=0x80 [16466.002545] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16466.011272] pending: check_lifetime, do_cache_clean [16466.016279] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16466.022913] pending: neigh_periodic_work, neigh_periodic_work [16466.028139] workqueue mm_percpu_wq: flags=0x8 [16466.030626] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16466.037959] pending: vmstat_update, lru_add_drain_per_cpu [16466.043193] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16466.049782] pending: vmstat_update, lru_add_drain_per_cpu [16466.056693] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16466.065627] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16466.069573] workqueue xprtiod: flags=0xa [16466.072277] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16466.077015] pending: xprt_autoclose, xs_error_handle [16466.082650] workqueue ipv6_addrconf: flags=0x40008 [16466.084681] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16466.090616] pending: addrconf_verify_work [16466.095798] workqueue knbd0-recv: flags=0x1a [16466.101465] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16466.105363] in-flight: 651:recv_work [16466.110431] workqueue ptlrpc_pinger: flags=0x2 [16466.114569] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16466.118971] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16466.126489] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13829s workers=4 idle: 15979 31824 8 [16466.132228] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16456s workers=2 idle: 654 [16476.058055] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16476.060839] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16476.099277] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16476.116590] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16476.132221] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16476.134011] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16476.158442] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16476.162984] RAX: 000000000000730d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16476.166732] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16476.173438] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16476.180496] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16476.184516] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16476.192092] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16476.196267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16476.202307] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16476.209207] Call Trace: [16476.219467] [16476.226100] ? show_regs.cold.9+0x22/0x2f [16476.233050] ? watchdog_timer_fn.cold.6+0x62/0x10a [16476.234569] ? __hrtimer_run_queues+0x1c2/0x4d0 [16476.243222] ? watchdog+0x40/0x40 [16476.244302] ? hrtimer_interrupt+0x128/0x2c0 [16476.248636] ? smp_apic_timer_interrupt+0xa6/0x270 [16476.251336] ? apic_timer_interrupt+0xf/0x20 [16476.253531] [16476.254160] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16476.255797] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16476.262635] do_raw_spin_lock+0x177/0x190 [16476.264560] _raw_spin_lock+0x21/0x30 [16476.266124] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16476.269430] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16476.272119] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16476.275608] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16476.278402] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16476.280422] ? woken_wake_function+0x30/0x30 [16476.281809] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16476.284214] kthread+0x1d1/0x200 [16476.285334] ? set_kthread_struct+0x70/0x70 [16476.286725] ret_from_fork+0x1f/0x30 [16484.056850] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [16484.060286] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [16484.068762] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16484.068849] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16484.068856] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16484.068859] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16484.068872] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16484.068963] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16484.068967] RAX: 0000000000007721 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16484.068970] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16484.068972] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16484.068974] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16484.068976] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16484.068979] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16484.068982] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16484.068984] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16484.068992] Call Trace: [16484.068997] [16484.069004] ? show_regs.cold.9+0x22/0x2f [16484.069015] ? watchdog_timer_fn.cold.6+0x62/0x10a [16484.069023] ? __hrtimer_run_queues+0x1c2/0x4d0 [16484.069027] ? watchdog+0x40/0x40 [16484.069031] ? hrtimer_interrupt+0x128/0x2c0 [16484.069035] ? smp_apic_timer_interrupt+0xa6/0x270 [16484.069044] ? apic_timer_interrupt+0xf/0x20 [16484.069050] [16484.069052] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16484.069059] do_raw_spin_lock+0x177/0x190 [16484.069064] _raw_spin_lock+0x21/0x30 [16484.069068] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16484.093043] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16484.093136] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16484.093143] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16484.093147] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16484.093161] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16484.093166] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16484.093171] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16484.093174] RDX: 00000000000028bd RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16484.093176] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16484.093179] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16484.093182] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16484.093186] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16484.093190] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16484.093192] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16484.093208] Call Trace: [16484.093218] [16484.093224] ? show_regs.cold.9+0x22/0x2f [16484.093234] ? watchdog_timer_fn.cold.6+0x62/0x10a [16484.093239] ? __hrtimer_run_queues+0x1c2/0x4d0 [16484.093243] ? watchdog+0x40/0x40 [16484.093248] ? hrtimer_interrupt+0x128/0x2c0 [16484.093252] ? smp_apic_timer_interrupt+0xa6/0x270 [16484.093259] ? apic_timer_interrupt+0xf/0x20 [16484.093264] [16484.093266] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16484.093271] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16484.093275] do_raw_spin_lock+0x177/0x190 [16484.093286] _raw_spin_lock+0x21/0x30 [16484.093293] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16484.094186] obd_mod_free+0x46/0x140 [ptlrpc] [16484.094656] mdc_commit_open+0x91/0xb0 [mdc] [16484.094693] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16484.095082] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16484.096993] after_reply+0xe2c/0x1780 [ptlrpc] [16484.098360] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16484.099033] ? __next_timer_interrupt+0x160/0x160 [16484.099043] ptlrpcd+0xa82/0xe40 [ptlrpc] [16484.099244] ? do_wait_intr+0xf0/0xf0 [16484.099252] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16484.134690] ? do_raw_spin_unlock+0x75/0x190 [16484.134711] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16484.143915] kthread+0x1d1/0x200 [16484.143943] ? set_kthread_struct+0x70/0x70 [16484.143948] ret_from_fork+0x1f/0x30 [16484.522718] ? do_wait_intr+0xf0/0xf0 [16484.524252] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16484.526653] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16484.528920] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16484.531310] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16484.533303] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16484.535896] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16484.537546] mdc_intent_lock+0x353/0x7e0 [mdc] [16484.539197] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16484.541435] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16484.551082] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16484.557286] lmv_intent_open+0x338/0x1150 [lmv] [16484.561376] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16484.564676] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16484.568255] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16484.573018] ll_intent_lock+0xc5/0xaa0 [lustre] [16484.574893] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16484.577760] ll_lookup_it+0xe0e/0x2270 [lustre] [16484.583290] ? do_raw_spin_unlock+0x75/0x190 [16484.589768] ? _raw_spin_unlock+0x12/0x30 [16484.598353] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16484.601732] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16484.609466] ll_atomic_open+0x36a/0xd70 [lustre] [16484.613036] lookup_open+0x356/0xb40 [16484.620394] do_last+0x313/0xfc0 [16484.626277] ? nd_jump_root+0xe5/0x160 [16484.633345] ? path_init+0x437/0x520 [16484.639903] path_openat+0xf7/0x500 [16484.646017] ? do_raw_spin_unlock+0x75/0x190 [16484.650725] do_filp_open+0x99/0x140 [16484.654412] ? getname_flags+0x6e/0x330 [16484.659841] ? __check_object_size+0xff/0x256 [16484.664972] ? do_raw_spin_unlock+0x75/0x190 [16484.667395] ? _raw_spin_unlock+0x12/0x30 [16484.670298] do_sys_openat2+0x2b4/0x410 [16484.672539] do_sys_open+0x73/0xa0 [16484.674412] __x64_sys_openat+0x24/0x30 [16484.678605] do_syscall_64+0xc1/0x440 [16484.681373] entry_SYSCALL_64_after_hwframe+0x49/0xae [16484.687149] RIP: 0033:0x7f53425a5332 [16484.692113] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16484.704310] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16484.710895] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16484.714005] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16484.719521] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16484.724569] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16484.732692] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16492.052088] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [16492.061907] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16492.106411] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16492.120050] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16492.136842] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16492.142210] RIP: 0010:native_safe_halt+0x2e/0x50 [16492.144516] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16492.170523] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16492.179367] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16492.185000] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [16492.191324] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16492.202689] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16492.208277] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16492.220255] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16492.227112] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16492.230708] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16492.235084] Call Trace: [16492.236373] [16492.238749] ? show_regs.cold.9+0x22/0x2f [16492.241853] ? watchdog_timer_fn.cold.6+0x62/0x10a [16492.243797] ? __hrtimer_run_queues+0x1c2/0x4d0 [16492.247205] ? watchdog+0x40/0x40 [16492.249405] ? hrtimer_interrupt+0x128/0x2c0 [16492.253170] ? smp_apic_timer_interrupt+0xa6/0x270 [16492.256108] ? apic_timer_interrupt+0xf/0x20 [16492.260172] [16492.261274] ? native_safe_halt+0x2e/0x50 [16492.266300] kvm_wait+0xcd/0xe0 [16492.268296] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16492.273414] ? newidle_balance.isra.93+0x207/0x580 [16492.281906] do_raw_spin_lock+0x177/0x190 [16492.283980] _raw_spin_lock+0x21/0x30 [16492.287332] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16492.290238] process_one_work+0x2c8/0x700 [16492.292350] worker_thread+0x296/0x6e0 [16492.295937] ? rescuer_thread+0x570/0x570 [16492.298344] kthread+0x1d1/0x200 [16492.300602] ? set_kthread_struct+0x70/0x70 [16492.310733] ret_from_fork+0x1f/0x30 [16496.609829] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13859s! [16496.621110] Showing busy workqueues and worker pools: [16496.629008] workqueue events: flags=0x0 [16496.632875] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16496.639289] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16496.647360] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16496.652840] pending: cpu_latency_work [ptlrpc] [16496.657395] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16496.659530] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16496.675251] workqueue events_long: flags=0x0 [16496.678461] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16496.681347] pending: lu_objects_flush [obdclass] [16496.685294] workqueue events_unbound: flags=0x2 [16496.688398] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16496.697203] pending: flush_memcg_stats_dwork, idle_cull_fn [16496.702238] workqueue events_power_efficient: flags=0x80 [16496.707270] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16496.714144] pending: check_lifetime, do_cache_clean [16496.715865] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16496.723863] pending: neigh_periodic_work, neigh_periodic_work [16496.731933] workqueue mm_percpu_wq: flags=0x8 [16496.736514] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16496.748031] pending: vmstat_update, lru_add_drain_per_cpu [16496.750084] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16496.752184] pending: vmstat_update, lru_add_drain_per_cpu [16496.764691] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16496.769633] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16496.773215] workqueue xprtiod: flags=0xa [16496.775219] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16496.782141] pending: xprt_autoclose, xs_error_handle [16496.788250] workqueue ipv6_addrconf: flags=0x40008 [16496.792712] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16496.799263] pending: addrconf_verify_work [16496.803692] workqueue knbd0-recv: flags=0x1a [16496.806482] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16496.809547] in-flight: 651:recv_work [16496.813756] workqueue ptlrpc_pinger: flags=0x2 [16496.816266] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16496.819358] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16496.829640] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13859s workers=4 idle: 15979 31824 8 [16496.837225] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16487s workers=2 idle: 654 [16504.058892] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16504.063320] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16504.095789] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16504.103887] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16504.109986] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16504.114832] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16504.123805] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16504.127716] RAX: 0000000000006686 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16504.140079] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16504.146250] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16504.150555] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16504.158086] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16504.164892] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16504.178418] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16504.183975] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16504.190663] Call Trace: [16504.191398] [16504.192063] ? show_regs.cold.9+0x22/0x2f [16504.195410] ? watchdog_timer_fn.cold.6+0x62/0x10a [16504.198969] ? __hrtimer_run_queues+0x1c2/0x4d0 [16504.201431] ? watchdog+0x40/0x40 [16504.202355] ? hrtimer_interrupt+0x128/0x2c0 [16504.204583] ? smp_apic_timer_interrupt+0xa6/0x270 [16504.207936] ? apic_timer_interrupt+0xf/0x20 [16504.211949] [16504.213783] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16504.219798] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16504.232951] do_raw_spin_lock+0x177/0x190 [16504.237212] _raw_spin_lock+0x21/0x30 [16504.239059] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16504.255019] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16504.263850] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16504.269289] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16504.273104] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16504.278324] ? woken_wake_function+0x30/0x30 [16504.280352] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16504.283762] kthread+0x1d1/0x200 [16504.287370] ? set_kthread_struct+0x70/0x70 [16504.290468] ret_from_fork+0x1f/0x30 [16512.055695] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [16512.057431] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [16512.067119] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash [16512.070950] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16512.071033] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16512.071040] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16512.071043] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16512.071058] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16512.071062] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16512.071066] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16512.071069] RDX: 0000000000005e2d RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16512.071072] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16512.071073] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16512.071076] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16512.071079] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16512.071082] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16512.071095] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16512.071106] Call Trace: [16512.071112] [16512.071118] ? show_regs.cold.9+0x22/0x2f [16512.071128] ? watchdog_timer_fn.cold.6+0x62/0x10a [16512.071134] ? __hrtimer_run_queues+0x1c2/0x4d0 [16512.071199] ? watchdog+0x40/0x40 [16512.071204] ? hrtimer_interrupt+0x128/0x2c0 [16512.071208] ? smp_apic_timer_interrupt+0xa6/0x270 [16512.071215] ? apic_timer_interrupt+0xf/0x20 [16512.071224] [16512.071227] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16512.071231] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16512.071236] do_raw_spin_lock+0x177/0x190 [16512.071241] _raw_spin_lock+0x21/0x30 [16512.071246] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16512.071515] obd_mod_free+0x46/0x140 [ptlrpc] [16512.145891] dm_log dm_mod [16512.177437] mdc_commit_open+0x91/0xb0 [mdc] [16512.183427] sha512_ssse3 sha512_generic [16512.183442] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16512.183450] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16512.183452] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16512.183467] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16512.183471] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16512.183475] RAX: 0000000000006c8d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16512.183477] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16512.183479] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16512.183481] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16512.183484] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16512.183486] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16512.183489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16512.183491] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16512.183500] Call Trace: [16512.183504] [16512.183510] ? show_regs.cold.9+0x22/0x2f [16512.183520] ? watchdog_timer_fn.cold.6+0x62/0x10a [16512.183526] ? __hrtimer_run_queues+0x1c2/0x4d0 [16512.183531] ? watchdog+0x40/0x40 [16512.183536] ? hrtimer_interrupt+0x128/0x2c0 [16512.183540] ? smp_apic_timer_interrupt+0xa6/0x270 [16512.183548] ? apic_timer_interrupt+0xf/0x20 [16512.183553] [16512.183556] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16512.183564] do_raw_spin_lock+0x177/0x190 [16512.183568] _raw_spin_lock+0x21/0x30 [16512.183573] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16512.187570] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16512.193442] ? do_raw_spin_unlock+0x75/0x190 [16512.193473] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16512.196640] ? do_wait_intr+0xf0/0xf0 [16512.196652] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16512.197582] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16512.211953] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16512.217076] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16512.219900] after_reply+0xe2c/0x1780 [ptlrpc] [16512.225895] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16512.228903] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16512.234847] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16512.237960] ? __next_timer_interrupt+0x160/0x160 [16512.237978] ptlrpcd+0xa82/0xe40 [ptlrpc] [16512.238237] ? do_wait_intr+0xf0/0xf0 [16512.238245] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16512.238472] kthread+0x1d1/0x200 [16512.238479] ? set_kthread_struct+0x70/0x70 [16512.238483] ret_from_fork+0x1f/0x30 [16512.453328] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16512.454956] mdc_intent_lock+0x353/0x7e0 [mdc] [16512.457686] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16512.459876] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16512.462935] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16512.465253] lmv_intent_open+0x338/0x1150 [lmv] [16512.467519] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16512.469896] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16512.472693] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16512.474845] ll_intent_lock+0xc5/0xaa0 [lustre] [16512.476176] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16512.478577] ll_lookup_it+0xe0e/0x2270 [lustre] [16512.480657] ? do_raw_spin_unlock+0x75/0x190 [16512.487576] ? _raw_spin_unlock+0x12/0x30 [16512.488879] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16512.490726] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16512.493768] ll_atomic_open+0x36a/0xd70 [lustre] [16512.496581] lookup_open+0x356/0xb40 [16512.498208] do_last+0x313/0xfc0 [16512.499822] ? nd_jump_root+0xe5/0x160 [16512.501231] ? path_init+0x437/0x520 [16512.502188] path_openat+0xf7/0x500 [16512.503206] ? do_raw_spin_unlock+0x75/0x190 [16512.506137] do_filp_open+0x99/0x140 [16512.508115] ? getname_flags+0x6e/0x330 [16512.510348] ? __check_object_size+0xff/0x256 [16512.511673] ? do_raw_spin_unlock+0x75/0x190 [16512.513031] ? _raw_spin_unlock+0x12/0x30 [16512.515125] do_sys_openat2+0x2b4/0x410 [16512.516441] do_sys_open+0x73/0xa0 [16512.517526] __x64_sys_openat+0x24/0x30 [16512.518624] do_syscall_64+0xc1/0x440 [16512.519754] entry_SYSCALL_64_after_hwframe+0x49/0xae [16512.521284] RIP: 0033:0x7f53425a5332 [16512.522641] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16512.529990] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16512.535384] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16512.537742] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16512.540156] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16512.543651] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16512.545519] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16524.053202] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [16524.060054] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16524.083313] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16524.094300] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16524.104550] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16524.110017] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [16524.113733] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [16524.127925] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16524.136593] RAX: 0000000000003255 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16524.141817] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16524.147321] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16524.165228] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16524.172569] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16524.183987] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16524.192118] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16524.202288] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16524.215939] Call Trace: [16524.218242] [16524.220620] ? show_regs.cold.9+0x22/0x2f [16524.223238] ? watchdog_timer_fn.cold.6+0x62/0x10a [16524.227476] ? __hrtimer_run_queues+0x1c2/0x4d0 [16524.230697] ? watchdog+0x40/0x40 [16524.233340] ? hrtimer_interrupt+0x128/0x2c0 [16524.236346] ? smp_apic_timer_interrupt+0xa6/0x270 [16524.242830] ? apic_timer_interrupt+0xf/0x20 [16524.246357] [16524.248502] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [16524.254916] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16524.258359] ? newidle_balance.isra.93+0x207/0x580 [16524.263261] do_raw_spin_lock+0x177/0x190 [16524.266695] _raw_spin_lock+0x21/0x30 [16524.277338] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16524.285030] process_one_work+0x2c8/0x700 [16524.288255] worker_thread+0x296/0x6e0 [16524.291542] ? rescuer_thread+0x570/0x570 [16524.294611] kthread+0x1d1/0x200 [16524.296544] ? set_kthread_struct+0x70/0x70 [16524.298554] ret_from_fork+0x1f/0x30 [16527.328043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13890s! [16527.331865] Showing busy workqueues and worker pools: [16527.337978] workqueue events: flags=0x0 [16527.344897] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16527.349287] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16527.356449] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16527.360587] pending: cpu_latency_work [ptlrpc] [16527.363929] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16527.368256] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16527.372303] workqueue events_long: flags=0x0 [16527.373669] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16527.376964] pending: lu_objects_flush [obdclass] [16527.379833] workqueue events_unbound: flags=0x2 [16527.382264] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16527.385426] pending: flush_memcg_stats_dwork, idle_cull_fn [16527.388916] workqueue events_power_efficient: flags=0x80 [16527.391918] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16527.395834] pending: check_lifetime, do_cache_clean [16527.398594] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16527.401470] pending: neigh_periodic_work, neigh_periodic_work [16527.404546] workqueue mm_percpu_wq: flags=0x8 [16527.406974] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16527.410893] pending: vmstat_update, lru_add_drain_per_cpu [16527.413683] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16527.416992] pending: vmstat_update, lru_add_drain_per_cpu [16527.419981] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16527.423507] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16527.426682] workqueue xprtiod: flags=0xa [16527.428660] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16527.432381] pending: xprt_autoclose, xs_error_handle [16527.435353] workqueue ipv6_addrconf: flags=0x40008 [16527.438361] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16527.442684] pending: addrconf_verify_work [16527.446464] workqueue knbd0-recv: flags=0x1a [16527.448571] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16527.451984] in-flight: 651:recv_work [16527.454404] workqueue ptlrpc_pinger: flags=0x2 [16527.456681] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16527.459943] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16527.464321] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13890s workers=4 idle: 15979 31824 8 [16527.470855] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16517s workers=2 idle: 654 [16532.058153] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16532.062630] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16532.086719] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16532.091921] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16532.096402] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16532.099308] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16532.109450] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16532.114194] RAX: 0000000000003c84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16532.118308] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16532.121766] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16532.125551] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16532.129357] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16532.132823] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16532.137516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16532.140259] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16532.145049] Call Trace: [16532.146085] [16532.147254] ? show_regs.cold.9+0x22/0x2f [16532.149650] ? watchdog_timer_fn.cold.6+0x62/0x10a [16532.152181] ? __hrtimer_run_queues+0x1c2/0x4d0 [16532.154534] ? watchdog+0x40/0x40 [16532.156211] ? hrtimer_interrupt+0x128/0x2c0 [16532.158820] ? smp_apic_timer_interrupt+0xa6/0x270 [16532.161549] ? apic_timer_interrupt+0xf/0x20 [16532.163847] [16532.165144] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16532.167807] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16532.170484] do_raw_spin_lock+0x177/0x190 [16532.172181] _raw_spin_lock+0x21/0x30 [16532.173741] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16532.177445] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16532.180237] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16532.183273] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16532.186352] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16532.189690] ? woken_wake_function+0x30/0x30 [16532.192357] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16532.196177] kthread+0x1d1/0x200 [16532.197749] ? set_kthread_struct+0x70/0x70 [16532.199828] ret_from_fork+0x1f/0x30 [16540.060143] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [16540.060952] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [16540.067528] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16540.067606] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16540.067614] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16540.067616] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16540.067632] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16540.067637] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16540.067642] RAX: 0000000000007984 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16540.067875] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16540.067878] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16540.067880] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16540.067882] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16540.067886] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16540.067889] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16540.067891] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16540.067899] Call Trace: [16540.067906] [16540.067914] ? show_regs.cold.9+0x22/0x2f [16540.067924] ? watchdog_timer_fn.cold.6+0x62/0x10a [16540.067931] ? __hrtimer_run_queues+0x1c2/0x4d0 [16540.067935] ? watchdog+0x40/0x40 [16540.067939] ? hrtimer_interrupt+0x128/0x2c0 [16540.067944] ? smp_apic_timer_interrupt+0xa6/0x270 [16540.067952] ? apic_timer_interrupt+0xf/0x20 [16540.067956] [16540.067958] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16540.067965] do_raw_spin_lock+0x177/0x190 [16540.067970] _raw_spin_lock+0x21/0x30 [16540.067974] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16540.068333] ? do_raw_spin_unlock+0x75/0x190 [16540.068343] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16540.073323] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16540.073399] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16540.073407] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16540.073409] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16540.073423] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16540.073427] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16540.073431] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16540.073434] RDX: 0000000000002265 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16540.073436] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16540.073438] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16540.073441] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16540.073444] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16540.073447] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16540.073451] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16540.073459] Call Trace: [16540.073477] [16540.073483] ? show_regs.cold.9+0x22/0x2f [16540.073493] ? watchdog_timer_fn.cold.6+0x62/0x10a [16540.073499] ? __hrtimer_run_queues+0x1c2/0x4d0 [16540.073504] ? watchdog+0x40/0x40 [16540.073509] ? hrtimer_interrupt+0x128/0x2c0 [16540.073513] ? smp_apic_timer_interrupt+0xa6/0x270 [16540.073523] ? apic_timer_interrupt+0xf/0x20 [16540.073528] [16540.073530] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16540.106008] ? do_wait_intr+0xf0/0xf0 [16540.106032] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16540.106318] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16540.106784] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16540.107076] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16540.107214] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16540.107263] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16540.107295] mdc_intent_lock+0x353/0x7e0 [mdc] [16540.107323] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16540.107489] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16540.107917] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16540.107949] lmv_intent_open+0x338/0x1150 [lmv] [16540.107985] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16540.108253] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16540.108336] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16540.108359] ll_intent_lock+0xc5/0xaa0 [lustre] [16540.108590] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16540.108752] ll_lookup_it+0xe0e/0x2270 [lustre] [16540.108970] ? do_raw_spin_unlock+0x75/0x190 [16540.108980] ? _raw_spin_unlock+0x12/0x30 [16540.108988] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16540.109132] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16540.109153] ll_atomic_open+0x36a/0xd70 [lustre] [16540.109239] lookup_open+0x356/0xb40 [16540.109248] do_last+0x313/0xfc0 [16540.109254] ? nd_jump_root+0xe5/0x160 [16540.109258] ? path_init+0x437/0x520 [16540.109263] path_openat+0xf7/0x500 [16540.109268] ? do_raw_spin_unlock+0x75/0x190 [16540.109273] do_filp_open+0x99/0x140 [16540.109279] ? getname_flags+0x6e/0x330 [16540.109283] ? __check_object_size+0xff/0x256 [16540.109290] ? do_raw_spin_unlock+0x75/0x190 [16540.109294] ? _raw_spin_unlock+0x12/0x30 [16540.109298] do_sys_openat2+0x2b4/0x410 [16540.109302] do_sys_open+0x73/0xa0 [16540.109306] __x64_sys_openat+0x24/0x30 [16540.109309] do_syscall_64+0xc1/0x440 [16540.109315] entry_SYSCALL_64_after_hwframe+0x49/0xae [16540.109321] RIP: 0033:0x7f53425a5332 [16540.109329] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16540.109333] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16540.109338] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16540.109340] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16540.109343] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16540.109344] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16540.109346] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16540.410624] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16540.413125] do_raw_spin_lock+0x177/0x190 [16540.414728] _raw_spin_lock+0x21/0x30 [16540.416555] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16540.419029] obd_mod_free+0x46/0x140 [ptlrpc] [16540.421268] mdc_commit_open+0x91/0xb0 [mdc] [16540.423094] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16540.425428] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16540.427797] after_reply+0xe2c/0x1780 [ptlrpc] [16540.429968] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16540.432296] ? __next_timer_interrupt+0x160/0x160 [16540.433573] ptlrpcd+0xa82/0xe40 [ptlrpc] [16540.435079] ? do_wait_intr+0xf0/0xf0 [16540.436169] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16540.438394] kthread+0x1d1/0x200 [16540.439745] ? set_kthread_struct+0x70/0x70 [16540.441672] ret_from_fork+0x1f/0x30 [16552.055264] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [16552.065264] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16552.111090] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16552.118896] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16552.126901] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16552.130993] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16552.135713] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16552.156099] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16552.164285] RAX: 0000000000007c90 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16552.172085] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16552.178398] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16552.188916] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16552.195311] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16552.201870] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16552.208324] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16552.213110] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16552.217417] Call Trace: [16552.218322] [16552.218996] ? show_regs.cold.9+0x22/0x2f [16552.220024] ? watchdog_timer_fn.cold.6+0x62/0x10a [16552.223579] ? __hrtimer_run_queues+0x1c2/0x4d0 [16552.228842] ? watchdog+0x40/0x40 [16552.231395] ? hrtimer_interrupt+0x128/0x2c0 [16552.233945] ? smp_apic_timer_interrupt+0xa6/0x270 [16552.241272] ? apic_timer_interrupt+0xf/0x20 [16552.245814] [16552.247748] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16552.250482] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16552.253472] ? newidle_balance.isra.93+0x207/0x580 [16552.263317] do_raw_spin_lock+0x177/0x190 [16552.267315] _raw_spin_lock+0x21/0x30 [16552.268485] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16552.272460] process_one_work+0x2c8/0x700 [16552.281441] worker_thread+0x296/0x6e0 [16552.282983] ? rescuer_thread+0x570/0x570 [16552.287857] kthread+0x1d1/0x200 [16552.292477] ? set_kthread_struct+0x70/0x70 [16552.298135] ret_from_fork+0x1f/0x30 [16554.786792] rcu: INFO: rcu_sched self-detected stall on CPU [16554.794281] rcu: 2-....: (8052717 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [16554.803642] (t=14460360 jiffies g=274185 q=67693) [16554.819317] rcu: rcu_sched kthread starved for 13918726 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [16554.830091] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [16554.844881] rcu: RCU grace-period kthread stack dump: [16554.851728] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [16554.871370] Call Trace: [16554.872251] __schedule+0x351/0xcb0 [16554.878437] schedule+0xc0/0x180 [16554.883952] schedule_timeout+0xb4/0x190 [16554.889282] ? do_raw_spin_unlock+0x75/0x190 [16554.893905] ? __next_timer_interrupt+0x160/0x160 [16554.901259] ? prepare_to_swait_event+0xf3/0x140 [16554.905686] rcu_gp_kthread+0x8dd/0x15b0 [16554.907923] ? force_qs_rnp+0x2c0/0x2c0 [16554.914757] kthread+0x1d1/0x200 [16554.917959] ? set_kthread_struct+0x70/0x70 [16554.920289] ret_from_fork+0x1f/0x30 [16554.922859] rcu: Stack dump where RCU GP kthread last ran: [16554.928532] NMI backtrace for cpu 2 [16554.933372] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16554.939749] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16554.947299] Call Trace: [16554.953571] [16554.954438] ? dump_stack+0xbb/0x10e [16554.959143] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16554.962087] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16554.964255] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16554.973647] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16554.988766] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16554.997273] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [16555.006017] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [16555.010921] ? update_process_times+0x6d/0xc0 [16555.014532] ? tick_sched_handle.isra.18+0x27/0x90 [16555.020106] ? tick_sched_timer+0x53/0xb0 [16555.023769] ? __hrtimer_run_queues+0x1c2/0x4d0 [16555.027863] ? tick_sched_handle.isra.18+0x90/0x90 [16555.030495] ? hrtimer_interrupt+0x128/0x2c0 [16555.034875] ? smp_apic_timer_interrupt+0xa6/0x270 [16555.037793] ? apic_timer_interrupt+0xf/0x20 [16555.040897] [16555.043033] ? native_safe_halt+0x2e/0x50 [16555.045263] ? kvm_wait+0xcd/0xe0 [16555.049845] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16555.055897] ? do_raw_spin_lock+0x177/0x190 [16555.059723] ? _raw_spin_lock+0x21/0x30 [16555.061837] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16555.065337] ? obd_mod_free+0x46/0x140 [ptlrpc] [16555.068716] ? mdc_commit_open+0x91/0xb0 [mdc] [16555.070711] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16555.075861] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16555.080822] ? after_reply+0xe2c/0x1780 [ptlrpc] [16555.082923] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16555.085062] ? __next_timer_interrupt+0x160/0x160 [16555.092631] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16555.094005] ? do_wait_intr+0xf0/0xf0 [16555.094916] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16555.099739] ? kthread+0x1d1/0x200 [16555.101886] ? set_kthread_struct+0x70/0x70 [16555.104389] ? ret_from_fork+0x1f/0x30 [16555.107999] Sending NMI from CPU 2 to CPUs 1: [16555.112117] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [16555.116063] NMI backtrace for cpu 2 [16555.119812] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16555.133413] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16555.137866] Call Trace: [16555.139201] [16555.140242] ? dump_stack+0xbb/0x10e [16555.143483] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16555.149056] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16555.155956] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16555.162934] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16555.170961] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16555.173867] ? rcu_dump_cpu_stacks+0x11b/0x171 [16555.175854] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [16555.178090] ? update_process_times+0x6d/0xc0 [16555.182618] ? tick_sched_handle.isra.18+0x27/0x90 [16555.184921] ? tick_sched_timer+0x53/0xb0 [16555.186886] ? __hrtimer_run_queues+0x1c2/0x4d0 [16555.189334] ? tick_sched_handle.isra.18+0x90/0x90 [16555.191175] ? hrtimer_interrupt+0x128/0x2c0 [16555.194472] ? smp_apic_timer_interrupt+0xa6/0x270 [16555.198676] ? apic_timer_interrupt+0xf/0x20 [16555.202947] [16555.204532] ? native_safe_halt+0x2e/0x50 [16555.206521] ? kvm_wait+0xcd/0xe0 [16555.207973] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16555.212279] ? do_raw_spin_lock+0x177/0x190 [16555.214293] ? _raw_spin_lock+0x21/0x30 [16555.216502] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16555.218807] ? obd_mod_free+0x46/0x140 [ptlrpc] [16555.221649] ? mdc_commit_open+0x91/0xb0 [mdc] [16555.224239] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16555.230047] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16555.233851] ? after_reply+0xe2c/0x1780 [ptlrpc] [16555.238528] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16555.244670] ? __next_timer_interrupt+0x160/0x160 [16555.246149] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16555.253084] ? do_wait_intr+0xf0/0xf0 [16555.254507] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16555.258106] ? kthread+0x1d1/0x200 [16555.260629] ? set_kthread_struct+0x70/0x70 [16555.262816] ? ret_from_fork+0x1f/0x30 [16557.539387] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13920s! [16557.543913] Showing busy workqueues and worker pools: [16557.547164] workqueue events: flags=0x0 [16557.548993] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16557.551135] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16557.555018] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16557.557147] pending: cpu_latency_work [ptlrpc] [16557.564809] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16557.573242] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16557.580102] workqueue events_long: flags=0x0 [16557.581837] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16557.584461] pending: lu_objects_flush [obdclass] [16557.586673] workqueue events_unbound: flags=0x2 [16557.588320] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16557.590945] pending: flush_memcg_stats_dwork, idle_cull_fn [16557.593032] workqueue events_power_efficient: flags=0x80 [16557.596063] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16557.598285] pending: check_lifetime, do_cache_clean [16557.600080] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16557.602057] pending: neigh_periodic_work, neigh_periodic_work [16557.607011] workqueue mm_percpu_wq: flags=0x8 [16557.609336] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16557.612240] pending: vmstat_update, lru_add_drain_per_cpu [16557.616065] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16557.620192] pending: vmstat_update, lru_add_drain_per_cpu [16557.625468] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16557.630428] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16557.634265] workqueue xprtiod: flags=0xa [16557.638133] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16557.644376] pending: xprt_autoclose, xs_error_handle [16557.646443] workqueue ipv6_addrconf: flags=0x40008 [16557.651016] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16557.653666] pending: addrconf_verify_work [16557.660955] workqueue knbd0-recv: flags=0x1a [16557.662355] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16557.672214] in-flight: 651:recv_work [16557.673790] workqueue ptlrpc_pinger: flags=0x2 [16557.675571] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16557.678481] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16557.681923] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13920s workers=4 idle: 15979 31824 8 [16557.685251] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16548s workers=2 idle: 654 [16560.059545] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16560.068179] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16560.112697] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16560.122866] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16560.132111] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16560.143318] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16560.177955] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16560.192955] RAX: 0000000000006a88 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16560.209422] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16560.220960] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16560.231130] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16560.243829] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16560.251788] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16560.264932] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16560.269595] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16560.282341] Call Trace: [16560.283169] [16560.290225] ? show_regs.cold.9+0x22/0x2f [16560.292364] ? watchdog_timer_fn.cold.6+0x62/0x10a [16560.301632] ? __hrtimer_run_queues+0x1c2/0x4d0 [16560.305243] ? watchdog+0x40/0x40 [16560.315205] ? hrtimer_interrupt+0x128/0x2c0 [16560.323005] ? smp_apic_timer_interrupt+0xa6/0x270 [16560.331895] ? apic_timer_interrupt+0xf/0x20 [16560.337883] [16560.339984] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16560.343886] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16560.350293] do_raw_spin_lock+0x177/0x190 [16560.356146] _raw_spin_lock+0x21/0x30 [16560.363361] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16560.371110] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16560.386195] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16560.395106] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16560.404352] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16560.412598] ? woken_wake_function+0x30/0x30 [16560.414016] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16560.423812] kthread+0x1d1/0x200 [16560.428834] ? set_kthread_struct+0x70/0x70 [16560.431944] ret_from_fork+0x1f/0x30 [16580.055010] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [16580.057027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [16580.071100] Modules linked in: lustre(O) [16580.075557] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16580.075632] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16580.075639] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16580.075642] RIP: 0010:native_safe_halt+0x2e/0x50 [16580.075656] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16580.075660] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16580.075664] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16580.075667] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16580.075669] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16580.075672] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16580.075674] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16580.075677] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16580.075680] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16580.075682] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16580.075691] Call Trace: [16580.075696] [16580.075702] ? show_regs.cold.9+0x22/0x2f [16580.075712] ? watchdog_timer_fn.cold.6+0x62/0x10a [16580.075719] ? __hrtimer_run_queues+0x1c2/0x4d0 [16580.075723] ? watchdog+0x40/0x40 [16580.075727] ? hrtimer_interrupt+0x128/0x2c0 [16580.075730] ? smp_apic_timer_interrupt+0xa6/0x270 [16580.075736] ? apic_timer_interrupt+0xf/0x20 [16580.075740] [16580.075743] ? native_safe_halt+0x2e/0x50 [16580.075747] kvm_wait+0xcd/0xe0 [16580.075754] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16580.075761] do_raw_spin_lock+0x177/0x190 [16580.075765] _raw_spin_lock+0x21/0x30 [16580.075771] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16580.076040] obd_mod_free+0x46/0x140 [ptlrpc] [16580.076619] mdc_commit_open+0x91/0xb0 [mdc] [16580.076654] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16580.076849] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16580.077041] after_reply+0xe2c/0x1780 [ptlrpc] [16580.079096] osp(O) ofd(O) [16580.123260] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16580.136476] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16580.136544] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16580.136552] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16580.136555] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16580.136569] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16580.136573] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16580.136577] RAX: 000000000000302b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16580.136580] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16580.136583] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16580.136585] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16580.136588] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16580.136591] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16580.136594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16580.136597] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16580.136606] Call Trace: [16580.136612] [16580.136617] ? show_regs.cold.9+0x22/0x2f [16580.136626] ? watchdog_timer_fn.cold.6+0x62/0x10a [16580.136633] ? __hrtimer_run_queues+0x1c2/0x4d0 [16580.136638] ? watchdog+0x40/0x40 [16580.136642] ? hrtimer_interrupt+0x128/0x2c0 [16580.136647] ? smp_apic_timer_interrupt+0xa6/0x270 [16580.136655] ? apic_timer_interrupt+0xf/0x20 [16580.136660] [16580.136663] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16580.136670] do_raw_spin_lock+0x177/0x190 [16580.136675] _raw_spin_lock+0x21/0x30 [16580.136680] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16580.136948] ? do_raw_spin_unlock+0x75/0x190 [16580.136954] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16580.142929] ? __next_timer_interrupt+0x160/0x160 [16580.142951] ptlrpcd+0xa82/0xe40 [ptlrpc] [16580.145497] ? do_wait_intr+0xf0/0xf0 [16580.145511] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16580.145738] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16580.145972] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16580.146208] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16580.146327] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16580.146368] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16580.146402] mdc_intent_lock+0x353/0x7e0 [mdc] [16580.146430] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16580.146516] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16580.146756] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16580.146787] lmv_intent_open+0x338/0x1150 [lmv] [16580.146821] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16580.146906] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16580.146985] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16580.147006] ll_intent_lock+0xc5/0xaa0 [lustre] [16580.150358] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16580.150447] ll_lookup_it+0xe0e/0x2270 [lustre] [16580.150541] ? do_raw_spin_unlock+0x75/0x190 [16580.150549] ? _raw_spin_unlock+0x12/0x30 [16580.150558] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16580.150587] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16580.150610] ll_atomic_open+0x36a/0xd70 [lustre] [16580.150706] lookup_open+0x356/0xb40 [16580.150715] do_last+0x313/0xfc0 [16580.150720] ? nd_jump_root+0xe5/0x160 [16580.150725] ? path_init+0x437/0x520 [16580.150729] path_openat+0xf7/0x500 [16580.150734] ? do_raw_spin_unlock+0x75/0x190 [16580.150739] do_filp_open+0x99/0x140 [16580.150745] ? getname_flags+0x6e/0x330 [16580.150750] ? __check_object_size+0xff/0x256 [16580.150757] ? do_raw_spin_unlock+0x75/0x190 [16580.150761] ? _raw_spin_unlock+0x12/0x30 [16580.150765] do_sys_openat2+0x2b4/0x410 [16580.150770] do_sys_open+0x73/0xa0 [16580.150775] __x64_sys_openat+0x24/0x30 [16580.150779] do_syscall_64+0xc1/0x440 [16580.150785] entry_SYSCALL_64_after_hwframe+0x49/0xae [16580.150791] RIP: 0033:0x7f53425a5332 [16580.150798] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16580.150802] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16580.150807] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16580.150810] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16580.150812] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16580.150814] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16580.150817] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16580.813055] ? do_wait_intr+0xf0/0xf0 [16580.816919] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16580.828969] kthread+0x1d1/0x200 [16580.831776] ? set_kthread_struct+0x70/0x70 [16580.837317] ret_from_fork+0x1f/0x30 [16584.055377] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [16584.062990] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16584.133570] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16584.156694] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16584.159565] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16584.174461] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16584.180915] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16584.202883] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16584.212933] RAX: 0000000000007a91 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16584.230526] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16584.244637] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16584.259325] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16584.268159] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16584.270700] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16584.277548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16584.286165] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16584.297992] Call Trace: [16584.298821] [16584.299515] ? show_regs.cold.9+0x22/0x2f [16584.300855] ? watchdog_timer_fn.cold.6+0x62/0x10a [16584.309097] ? __hrtimer_run_queues+0x1c2/0x4d0 [16584.313679] ? watchdog+0x40/0x40 [16584.318321] ? hrtimer_interrupt+0x128/0x2c0 [16584.320312] ? smp_apic_timer_interrupt+0xa6/0x270 [16584.329617] ? apic_timer_interrupt+0xf/0x20 [16584.342047] [16584.342961] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16584.347351] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16584.353218] ? newidle_balance.isra.93+0x207/0x580 [16584.354680] do_raw_spin_lock+0x177/0x190 [16584.360281] _raw_spin_lock+0x21/0x30 [16584.364468] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16584.369637] process_one_work+0x2c8/0x700 [16584.374020] worker_thread+0x296/0x6e0 [16584.378559] ? rescuer_thread+0x570/0x570 [16584.382722] kthread+0x1d1/0x200 [16584.386841] ? set_kthread_struct+0x70/0x70 [16584.394616] ret_from_fork+0x1f/0x30 [16587.745713] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13950s! [16587.754557] Showing busy workqueues and worker pools: [16587.759712] workqueue events: flags=0x0 [16587.762578] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16587.768903] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16587.780791] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16587.787643] pending: cpu_latency_work [ptlrpc] [16587.792193] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16587.798267] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16587.805036] workqueue events_long: flags=0x0 [16587.808888] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16587.814492] pending: lu_objects_flush [obdclass] [16587.818890] workqueue events_unbound: flags=0x2 [16587.822839] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16587.827493] pending: flush_memcg_stats_dwork, idle_cull_fn [16587.833435] workqueue events_power_efficient: flags=0x80 [16587.839184] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16587.842449] pending: check_lifetime, do_cache_clean [16587.847934] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16587.853438] pending: neigh_periodic_work, neigh_periodic_work [16587.857940] workqueue mm_percpu_wq: flags=0x8 [16587.861495] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16587.866056] pending: vmstat_update, lru_add_drain_per_cpu [16587.871879] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16587.877878] pending: vmstat_update, lru_add_drain_per_cpu [16587.882604] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16587.888227] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16587.892149] workqueue xprtiod: flags=0xa [16587.894914] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16587.899384] pending: xprt_autoclose, xs_error_handle [16587.901631] workqueue ipv6_addrconf: flags=0x40008 [16587.905397] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16587.908061] pending: addrconf_verify_work [16587.909647] workqueue knbd0-recv: flags=0x1a [16587.912710] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16587.915225] in-flight: 651:recv_work [16587.916529] workqueue ptlrpc_pinger: flags=0x2 [16587.919361] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16587.923739] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16587.929606] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13950s workers=4 idle: 15979 31824 8 [16587.936615] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16578s workers=2 idle: 654 [16588.058032] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16588.068326] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16588.116783] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16588.124181] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16588.127960] RIP: 0010:native_safe_halt+0x2e/0x50 [16588.130174] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16588.140470] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16588.149256] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16588.156168] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16588.164814] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16588.169533] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16588.173391] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16588.176259] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16588.183571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16588.189304] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16588.195277] Call Trace: [16588.196366] [16588.197385] ? show_regs.cold.9+0x22/0x2f [16588.201560] ? watchdog_timer_fn.cold.6+0x62/0x10a [16588.209980] ? __hrtimer_run_queues+0x1c2/0x4d0 [16588.215284] ? watchdog+0x40/0x40 [16588.216494] ? hrtimer_interrupt+0x128/0x2c0 [16588.219886] ? smp_apic_timer_interrupt+0xa6/0x270 [16588.223900] ? apic_timer_interrupt+0xf/0x20 [16588.227240] [16588.228128] ? native_safe_halt+0x2e/0x50 [16588.229930] kvm_wait+0xcd/0xe0 [16588.233365] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16588.238165] do_raw_spin_lock+0x177/0x190 [16588.243354] _raw_spin_lock+0x21/0x30 [16588.245346] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16588.251677] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16588.254662] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16588.256658] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16588.261399] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16588.264341] ? woken_wake_function+0x30/0x30 [16588.266281] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16588.269311] kthread+0x1d1/0x200 [16588.270746] ? set_kthread_struct+0x70/0x70 [16588.275283] ret_from_fork+0x1f/0x30 [16608.056563] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [16608.057067] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [16608.059228] Modules linked in: lustre(O) [16608.064553] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16608.064666] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16608.064672] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16608.064676] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16608.064693] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16608.064698] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16608.064702] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16608.064704] RDX: 000000000000431f RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16608.064707] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16608.064709] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16608.064712] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16608.064715] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16608.064718] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16608.064721] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16608.064729] Call Trace: [16608.064735] [16608.064740] ? show_regs.cold.9+0x22/0x2f [16608.064750] ? watchdog_timer_fn.cold.6+0x62/0x10a [16608.064756] ? __hrtimer_run_queues+0x1c2/0x4d0 [16608.064759] ? watchdog+0x40/0x40 [16608.064763] ? hrtimer_interrupt+0x128/0x2c0 [16608.064767] ? smp_apic_timer_interrupt+0xa6/0x270 [16608.064775] ? apic_timer_interrupt+0xf/0x20 [16608.064779] [16608.064782] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16608.064787] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16608.064791] do_raw_spin_lock+0x177/0x190 [16608.064796] _raw_spin_lock+0x21/0x30 [16608.064802] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16608.065143] obd_mod_free+0x46/0x140 [ptlrpc] [16608.065486] mdc_commit_open+0x91/0xb0 [mdc] [16608.065522] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16608.065790] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16608.070733] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr [16608.117556] after_reply+0xe2c/0x1780 [ptlrpc] [16608.118079] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16608.118343] ? __next_timer_interrupt+0x160/0x160 [16608.118354] ptlrpcd+0xa82/0xe40 [ptlrpc] [16608.118559] ? do_wait_intr+0xf0/0xf0 [16608.118568] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16608.119161] kthread+0x1d1/0x200 [16608.122283] ? set_kthread_struct+0x70/0x70 [16608.122300] ret_from_fork+0x1f/0x30 [16608.371495] squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16608.387694] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16608.397180] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16608.410176] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16608.418169] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16608.443776] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16608.454704] RAX: 0000000000002983 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16608.457262] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16608.474837] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16608.482576] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16608.491819] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16608.500411] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16608.504712] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16608.511909] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16608.526050] Call Trace: [16608.527120] [16608.528598] ? show_regs.cold.9+0x22/0x2f [16608.535566] ? watchdog_timer_fn.cold.6+0x62/0x10a [16608.539020] ? __hrtimer_run_queues+0x1c2/0x4d0 [16608.544367] ? watchdog+0x40/0x40 [16608.545648] ? hrtimer_interrupt+0x128/0x2c0 [16608.547177] ? smp_apic_timer_interrupt+0xa6/0x270 [16608.553596] ? apic_timer_interrupt+0xf/0x20 [16608.558538] [16608.559850] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16608.569574] do_raw_spin_lock+0x177/0x190 [16608.571927] _raw_spin_lock+0x21/0x30 [16608.573926] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16608.577842] ? do_raw_spin_unlock+0x75/0x190 [16608.581541] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16608.589387] ? do_wait_intr+0xf0/0xf0 [16608.593510] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16608.599276] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16608.606056] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16608.610538] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16608.616584] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16608.624675] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16608.628386] mdc_intent_lock+0x353/0x7e0 [mdc] [16608.633942] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16608.643577] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16608.646992] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16608.652105] lmv_intent_open+0x338/0x1150 [lmv] [16608.654869] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16608.659965] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16608.665934] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16608.671573] ll_intent_lock+0xc5/0xaa0 [lustre] [16608.676853] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16608.680445] ll_lookup_it+0xe0e/0x2270 [lustre] [16608.690477] ? do_raw_spin_unlock+0x75/0x190 [16608.692883] ? _raw_spin_unlock+0x12/0x30 [16608.700231] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16608.705895] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16608.715731] ll_atomic_open+0x36a/0xd70 [lustre] [16608.723210] lookup_open+0x356/0xb40 [16608.727113] do_last+0x313/0xfc0 [16608.731434] ? nd_jump_root+0xe5/0x160 [16608.732459] ? path_init+0x437/0x520 [16608.734799] path_openat+0xf7/0x500 [16608.737233] ? do_raw_spin_unlock+0x75/0x190 [16608.739765] do_filp_open+0x99/0x140 [16608.742194] ? getname_flags+0x6e/0x330 [16608.744600] ? __check_object_size+0xff/0x256 [16608.748911] ? do_raw_spin_unlock+0x75/0x190 [16608.752846] ? _raw_spin_unlock+0x12/0x30 [16608.757389] do_sys_openat2+0x2b4/0x410 [16608.758960] do_sys_open+0x73/0xa0 [16608.760512] __x64_sys_openat+0x24/0x30 [16608.764545] do_syscall_64+0xc1/0x440 [16608.766720] entry_SYSCALL_64_after_hwframe+0x49/0xae [16608.769212] RIP: 0033:0x7f53425a5332 [16608.771748] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16608.780244] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16608.782956] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16608.786459] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16608.789382] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16608.794133] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16608.799395] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16612.052112] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [16612.055289] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16612.096608] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16612.104929] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16612.109853] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16612.114323] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16612.116490] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16612.133275] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16612.137462] RAX: 0000000000006765 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16612.140339] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16612.144476] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16612.148477] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16612.153825] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16612.162626] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16612.171369] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16612.176508] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16612.181114] Call Trace: [16612.183860] [16612.184492] ? show_regs.cold.9+0x22/0x2f [16612.186739] ? watchdog_timer_fn.cold.6+0x62/0x10a [16612.189133] ? __hrtimer_run_queues+0x1c2/0x4d0 [16612.191286] ? watchdog+0x40/0x40 [16612.192837] ? hrtimer_interrupt+0x128/0x2c0 [16612.194781] ? smp_apic_timer_interrupt+0xa6/0x270 [16612.199783] ? apic_timer_interrupt+0xf/0x20 [16612.204110] [16612.206897] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16612.212201] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16612.217959] ? newidle_balance.isra.93+0x207/0x580 [16612.221078] do_raw_spin_lock+0x177/0x190 [16612.224313] _raw_spin_lock+0x21/0x30 [16612.227368] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16612.230096] process_one_work+0x2c8/0x700 [16612.231975] worker_thread+0x296/0x6e0 [16612.233508] ? rescuer_thread+0x570/0x570 [16612.235435] kthread+0x1d1/0x200 [16612.239404] ? set_kthread_struct+0x70/0x70 [16612.241085] ret_from_fork+0x1f/0x30 [16616.065457] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16616.071173] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16616.106942] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16616.115428] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16616.121980] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16616.128765] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16616.150019] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16616.156633] RAX: 0000000000007f8b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16616.161854] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16616.170188] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16616.176649] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16616.181974] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16616.186041] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16616.190162] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16616.193007] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16616.196675] Call Trace: [16616.197817] [16616.198788] ? show_regs.cold.9+0x22/0x2f [16616.200777] ? watchdog_timer_fn.cold.6+0x62/0x10a [16616.202887] ? __hrtimer_run_queues+0x1c2/0x4d0 [16616.205453] ? watchdog+0x40/0x40 [16616.206848] ? hrtimer_interrupt+0x128/0x2c0 [16616.210105] ? smp_apic_timer_interrupt+0xa6/0x270 [16616.214857] ? apic_timer_interrupt+0xf/0x20 [16616.216707] [16616.217405] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16616.222897] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16616.226725] do_raw_spin_lock+0x177/0x190 [16616.230751] _raw_spin_lock+0x21/0x30 [16616.233907] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16616.241448] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16616.246786] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16616.249572] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16616.254026] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16616.261672] ? woken_wake_function+0x30/0x30 [16616.265095] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16616.267377] kthread+0x1d1/0x200 [16616.269177] ? set_kthread_struct+0x70/0x70 [16616.275560] ret_from_fork+0x1f/0x30 [16617.957033] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 13980s! [16617.967610] Showing busy workqueues and worker pools: [16617.986027] workqueue events: flags=0x0 [16617.994815] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16618.005145] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16618.024597] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16618.030201] pending: cpu_latency_work [ptlrpc] [16618.033245] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16618.038036] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16618.051066] workqueue events_long: flags=0x0 [16618.054786] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16618.067686] pending: lu_objects_flush [obdclass] [16618.076183] workqueue events_unbound: flags=0x2 [16618.089079] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16618.103411] pending: flush_memcg_stats_dwork, idle_cull_fn [16618.107918] workqueue events_power_efficient: flags=0x80 [16618.117689] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16618.121038] pending: check_lifetime, do_cache_clean [16618.124516] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16618.129014] pending: neigh_periodic_work, neigh_periodic_work [16618.132675] workqueue mm_percpu_wq: flags=0x8 [16618.134712] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16618.142238] pending: vmstat_update, lru_add_drain_per_cpu [16618.146285] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16618.153712] pending: vmstat_update, lru_add_drain_per_cpu [16618.160630] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16618.168151] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16618.178914] workqueue xprtiod: flags=0xa [16618.184987] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16618.190411] pending: xprt_autoclose, xs_error_handle [16618.196676] workqueue ipv6_addrconf: flags=0x40008 [16618.200993] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16618.205792] pending: addrconf_verify_work [16618.209119] workqueue knbd0-recv: flags=0x1a [16618.213108] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16618.221973] in-flight: 651:recv_work [16618.229652] workqueue ptlrpc_pinger: flags=0x2 [16618.234942] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16618.241122] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16618.246749] pool 8: cpus=0-3 flags=0x4 nice=0 hung=13981s workers=4 idle: 15979 31824 8 [16618.256132] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16608s workers=2 idle: 654 [16636.056037] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [16636.057108] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [16636.064193] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16636.064473] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16636.064481] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16636.064485] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16636.064511] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16636.064514] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16636.064519] RAX: 000000000000208d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16636.064521] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16636.064523] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16636.064525] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16636.064528] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16636.064532] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16636.064535] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16636.064537] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16636.064546] Call Trace: [16636.064562] [16636.064569] ? show_regs.cold.9+0x22/0x2f [16636.064579] ? watchdog_timer_fn.cold.6+0x62/0x10a [16636.064585] ? __hrtimer_run_queues+0x1c2/0x4d0 [16636.064590] ? watchdog+0x40/0x40 [16636.064594] ? hrtimer_interrupt+0x128/0x2c0 [16636.064597] ? smp_apic_timer_interrupt+0xa6/0x270 [16636.064606] ? apic_timer_interrupt+0xf/0x20 [16636.064611] [16636.064614] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16636.064622] do_raw_spin_lock+0x177/0x190 [16636.064627] _raw_spin_lock+0x21/0x30 [16636.064632] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16636.064909] ? do_raw_spin_unlock+0x75/0x190 [16636.064915] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16636.069558] Modules linked in: lustre(O) [16636.099048] ? do_wait_intr+0xf0/0xf0 [16636.099073] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16636.099662] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16636.113509] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16636.113585] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16636.113593] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16636.113595] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16636.113609] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16636.113613] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16636.113618] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16636.113620] RDX: 0000000000001e19 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16636.113623] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16636.113625] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16636.113627] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16636.113631] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16636.113633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16636.113635] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16636.113644] Call Trace: [16636.113649] [16636.113655] ? show_regs.cold.9+0x22/0x2f [16636.113665] ? watchdog_timer_fn.cold.6+0x62/0x10a [16636.113671] ? __hrtimer_run_queues+0x1c2/0x4d0 [16636.113675] ? watchdog+0x40/0x40 [16636.113678] ? hrtimer_interrupt+0x128/0x2c0 [16636.113682] ? smp_apic_timer_interrupt+0xa6/0x270 [16636.113689] ? apic_timer_interrupt+0xf/0x20 [16636.113693] [16636.113696] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16636.113700] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16636.113705] do_raw_spin_lock+0x177/0x190 [16636.113709] _raw_spin_lock+0x21/0x30 [16636.113715] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16636.114028] obd_mod_free+0x46/0x140 [ptlrpc] [16636.114792] mdc_commit_open+0x91/0xb0 [mdc] [16636.114836] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16636.115318] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16636.115522] after_reply+0xe2c/0x1780 [ptlrpc] [16636.115845] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16636.116344] ? __next_timer_interrupt+0x160/0x160 [16636.116354] ptlrpcd+0xa82/0xe40 [ptlrpc] [16636.116556] ? do_wait_intr+0xf0/0xf0 [16636.116565] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16636.116830] kthread+0x1d1/0x200 [16636.116836] ? set_kthread_struct+0x70/0x70 [16636.116840] ret_from_fork+0x1f/0x30 [16636.415064] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16636.420371] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16636.422394] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16636.425031] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16636.427939] mdc_intent_lock+0x353/0x7e0 [mdc] [16636.430153] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16636.431523] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16636.435615] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16636.437986] lmv_intent_open+0x338/0x1150 [lmv] [16636.441054] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16636.442806] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16636.445039] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16636.447042] ll_intent_lock+0xc5/0xaa0 [lustre] [16636.451205] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16636.452943] ll_lookup_it+0xe0e/0x2270 [lustre] [16636.454857] ? do_raw_spin_unlock+0x75/0x190 [16636.456207] ? _raw_spin_unlock+0x12/0x30 [16636.458944] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16636.464510] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16636.469950] ll_atomic_open+0x36a/0xd70 [lustre] [16636.474436] lookup_open+0x356/0xb40 [16636.478071] do_last+0x313/0xfc0 [16636.479555] ? nd_jump_root+0xe5/0x160 [16636.480966] ? path_init+0x437/0x520 [16636.482558] path_openat+0xf7/0x500 [16636.484614] ? do_raw_spin_unlock+0x75/0x190 [16636.486220] do_filp_open+0x99/0x140 [16636.489766] ? getname_flags+0x6e/0x330 [16636.496540] ? __check_object_size+0xff/0x256 [16636.499940] ? do_raw_spin_unlock+0x75/0x190 [16636.504587] ? _raw_spin_unlock+0x12/0x30 [16636.509571] do_sys_openat2+0x2b4/0x410 [16636.513869] do_sys_open+0x73/0xa0 [16636.515942] __x64_sys_openat+0x24/0x30 [16636.519697] do_syscall_64+0xc1/0x440 [16636.523790] entry_SYSCALL_64_after_hwframe+0x49/0xae [16636.526313] RIP: 0033:0x7f53425a5332 [16636.528641] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16636.555743] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16636.570516] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16636.580298] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16636.588863] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16636.592922] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16636.599918] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16644.056474] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [16644.060286] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16644.069627] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16644.069700] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16644.069708] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16644.069716] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16644.073606] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16644.073684] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16644.073692] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16644.073695] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16644.073710] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16644.073714] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16644.073719] RAX: 0000000000003d04 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16644.073721] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16644.073723] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16644.073725] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16644.073728] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16644.073731] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16644.073734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16644.073737] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16644.073747] Call Trace: [16644.073753] [16644.073759] ? show_regs.cold.9+0x22/0x2f [16644.073770] ? watchdog_timer_fn.cold.6+0x62/0x10a [16644.073776] ? __hrtimer_run_queues+0x1c2/0x4d0 [16644.073781] ? watchdog+0x40/0x40 [16644.073785] ? hrtimer_interrupt+0x128/0x2c0 [16644.073790] ? smp_apic_timer_interrupt+0xa6/0x270 [16644.074197] ? apic_timer_interrupt+0xf/0x20 [16644.074202] [16644.074205] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16644.074210] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16644.074215] do_raw_spin_lock+0x177/0x190 [16644.074230] _raw_spin_lock+0x21/0x30 [16644.074235] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16644.074503] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16644.074682] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16644.075482] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16644.075689] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16644.076171] ? woken_wake_function+0x30/0x30 [16644.076181] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16644.103418] [16644.103428] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16644.103450] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16644.103454] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16644.103459] RAX: 0000000000007308 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16644.103462] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16644.103464] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16644.103466] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16644.103469] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16644.103472] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16644.103475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16644.103477] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16644.103487] Call Trace: [16644.103493] [16644.103500] ? show_regs.cold.9+0x22/0x2f [16644.103509] ? watchdog_timer_fn.cold.6+0x62/0x10a [16644.103515] ? __hrtimer_run_queues+0x1c2/0x4d0 [16644.103519] ? watchdog+0x40/0x40 [16644.103524] ? hrtimer_interrupt+0x128/0x2c0 [16644.103528] ? smp_apic_timer_interrupt+0xa6/0x270 [16644.103538] ? apic_timer_interrupt+0xf/0x20 [16644.103542] [16644.103545] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16644.103551] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16644.103554] ? newidle_balance.isra.93+0x207/0x580 [16644.103562] do_raw_spin_lock+0x177/0x190 [16644.103568] _raw_spin_lock+0x21/0x30 [16644.103573] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16644.103875] process_one_work+0x2c8/0x700 [16644.103883] worker_thread+0x296/0x6e0 [16644.103888] ? rescuer_thread+0x570/0x570 [16644.104290] kthread+0x1d1/0x200 [16644.104297] ? set_kthread_struct+0x70/0x70 [16644.104300] ret_from_fork+0x1f/0x30 [16644.416390] kthread+0x1d1/0x200 [16644.417725] ? set_kthread_struct+0x70/0x70 [16644.419691] ret_from_fork+0x1f/0x30 [16648.672148] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14011s! [16648.676438] Showing busy workqueues and worker pools: [16648.678534] workqueue events: flags=0x0 [16648.680376] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16648.686249] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16648.695210] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16648.704555] pending: cpu_latency_work [ptlrpc] [16648.711239] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16648.715842] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16648.720569] workqueue events_long: flags=0x0 [16648.722447] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16648.725458] pending: lu_objects_flush [obdclass] [16648.727397] workqueue events_unbound: flags=0x2 [16648.730805] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16648.737042] pending: flush_memcg_stats_dwork, idle_cull_fn [16648.739164] workqueue events_power_efficient: flags=0x80 [16648.750067] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16648.754217] pending: check_lifetime, do_cache_clean [16648.758101] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16648.762223] pending: neigh_periodic_work, neigh_periodic_work [16648.770053] workqueue mm_percpu_wq: flags=0x8 [16648.773095] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16648.784221] pending: vmstat_update, lru_add_drain_per_cpu [16648.792597] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16648.809363] pending: vmstat_update, lru_add_drain_per_cpu [16648.823364] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16648.827640] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16648.833369] workqueue xprtiod: flags=0xa [16648.840222] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16648.848599] pending: xprt_autoclose, xs_error_handle [16648.855693] workqueue ipv6_addrconf: flags=0x40008 [16648.864535] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16648.880449] pending: addrconf_verify_work [16648.882826] workqueue knbd0-recv: flags=0x1a [16648.887617] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16648.899030] in-flight: 651:recv_work [16648.904263] workqueue ptlrpc_pinger: flags=0x2 [16648.913056] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16648.920829] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16648.922967] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14011s workers=4 idle: 15979 31824 8 [16648.941172] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16639s workers=2 idle: 654 [16664.055024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [16664.057024] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [16664.059071] Modules linked in: [16664.065749] Modules linked in: [16664.065751] lustre(O) osp(O) ofd(O) [16664.067040] lustre(O) [16664.068580] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16664.068658] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16664.068665] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16664.068669] RIP: 0010:native_safe_halt+0x2e/0x50 [16664.068682] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16664.068686] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16664.068690] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16664.068693] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16664.068695] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16664.068697] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16664.068699] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16664.068703] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16664.068707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16664.068709] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16664.068719] Call Trace: [16664.068725] [16664.068731] ? show_regs.cold.9+0x22/0x2f [16664.068740] ? watchdog_timer_fn.cold.6+0x62/0x10a [16664.068747] ? __hrtimer_run_queues+0x1c2/0x4d0 [16664.068752] ? watchdog+0x40/0x40 [16664.068756] ? hrtimer_interrupt+0x128/0x2c0 [16664.068760] ? smp_apic_timer_interrupt+0xa6/0x270 [16664.068766] ? apic_timer_interrupt+0xf/0x20 [16664.068771] [16664.068774] ? native_safe_halt+0x2e/0x50 [16664.068778] kvm_wait+0xcd/0xe0 [16664.068785] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16664.068794] do_raw_spin_lock+0x177/0x190 [16664.068800] _raw_spin_lock+0x21/0x30 [16664.068805] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16664.069075] ? do_raw_spin_unlock+0x75/0x190 [16664.069083] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16664.069276] ? do_wait_intr+0xf0/0xf0 [16664.069286] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16664.069476] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16664.069620] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16664.070559] osp(O) [16664.071120] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16664.083623] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16664.083769] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16664.083778] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16664.083782] RIP: 0010:native_safe_halt+0x2e/0x50 [16664.083799] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16664.083804] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16664.083809] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16664.083811] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16664.083814] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16664.083815] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16664.083818] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16664.083821] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16664.083824] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16664.083827] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16664.083836] Call Trace: [16664.083841] [16664.083847] ? show_regs.cold.9+0x22/0x2f [16664.083858] ? watchdog_timer_fn.cold.6+0x62/0x10a [16664.083864] ? __hrtimer_run_queues+0x1c2/0x4d0 [16664.083868] ? watchdog+0x40/0x40 [16664.083872] ? hrtimer_interrupt+0x128/0x2c0 [16664.083875] ? smp_apic_timer_interrupt+0xa6/0x270 [16664.083882] ? apic_timer_interrupt+0xf/0x20 [16664.083886] [16664.083889] ? native_safe_halt+0x2e/0x50 [16664.083893] kvm_wait+0xcd/0xe0 [16664.083900] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16664.083908] do_raw_spin_lock+0x177/0x190 [16664.083912] _raw_spin_lock+0x21/0x30 [16664.083918] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16664.084176] obd_mod_free+0x46/0x140 [ptlrpc] [16664.084367] mdc_commit_open+0x91/0xb0 [mdc] [16664.084403] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16664.084593] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16664.088509] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16664.091153] after_reply+0xe2c/0x1780 [ptlrpc] [16664.092594] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16664.097410] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16664.099580] mdc_intent_lock+0x353/0x7e0 [mdc] [16664.099619] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16664.102209] ? __next_timer_interrupt+0x160/0x160 [16664.104095] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16664.107444] ptlrpcd+0xa82/0xe40 [ptlrpc] [16664.109392] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16664.111848] ? do_wait_intr+0xf0/0xf0 [16664.113915] lmv_intent_open+0x338/0x1150 [lmv] [16664.115657] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16664.117698] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16664.118610] kthread+0x1d1/0x200 [16664.119258] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16664.120476] ? set_kthread_struct+0x70/0x70 [16664.121705] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16664.121745] ll_intent_lock+0xc5/0xaa0 [lustre] [16664.123155] ret_from_fork+0x1f/0x30 [16664.124015] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16664.233255] ll_lookup_it+0xe0e/0x2270 [lustre] [16664.234617] ? do_raw_spin_unlock+0x75/0x190 [16664.235969] ? _raw_spin_unlock+0x12/0x30 [16664.237108] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16664.238800] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16664.240381] ll_atomic_open+0x36a/0xd70 [lustre] [16664.241918] lookup_open+0x356/0xb40 [16664.242913] do_last+0x313/0xfc0 [16664.243781] ? nd_jump_root+0xe5/0x160 [16664.244805] ? path_init+0x437/0x520 [16664.245780] path_openat+0xf7/0x500 [16664.246705] ? do_raw_spin_unlock+0x75/0x190 [16664.247891] do_filp_open+0x99/0x140 [16664.248888] ? getname_flags+0x6e/0x330 [16664.249913] ? __check_object_size+0xff/0x256 [16664.251141] ? do_raw_spin_unlock+0x75/0x190 [16664.252280] ? _raw_spin_unlock+0x12/0x30 [16664.253298] do_sys_openat2+0x2b4/0x410 [16664.254352] do_sys_open+0x73/0xa0 [16664.255247] __x64_sys_openat+0x24/0x30 [16664.256303] do_syscall_64+0xc1/0x440 [16664.257418] entry_SYSCALL_64_after_hwframe+0x49/0xae [16664.259076] RIP: 0033:0x7f53425a5332 [16664.260126] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16664.265056] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16664.267142] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16664.269133] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16664.271023] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16664.272961] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16664.274801] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16672.060047] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [16672.064473] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16672.118823] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16672.134246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16672.139107] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16672.144034] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16672.161074] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16672.167343] RAX: 0000000000001e88 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16672.174291] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16672.184397] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16672.197067] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16672.209398] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16672.224312] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16672.227158] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16672.247205] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16672.260134] Call Trace: [16672.268803] [16672.269569] ? show_regs.cold.9+0x22/0x2f [16672.270911] ? watchdog_timer_fn.cold.6+0x62/0x10a [16672.292952] ? __hrtimer_run_queues+0x1c2/0x4d0 [16672.303326] ? watchdog+0x40/0x40 [16672.305049] ? hrtimer_interrupt+0x128/0x2c0 [16672.313617] ? smp_apic_timer_interrupt+0xa6/0x270 [16672.321695] ? apic_timer_interrupt+0xf/0x20 [16672.330669] [16672.332358] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16672.339127] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16672.345577] do_raw_spin_lock+0x177/0x190 [16672.352188] _raw_spin_lock+0x21/0x30 [16672.357083] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16672.363167] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16672.369240] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16672.374124] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16672.379146] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16672.383308] ? woken_wake_function+0x30/0x30 [16672.387448] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16672.390482] kthread+0x1d1/0x200 [16672.392279] ? set_kthread_struct+0x70/0x70 [16672.396616] ret_from_fork+0x1f/0x30 [16676.055384] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [16676.073370] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16676.144457] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16676.155016] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16676.171982] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16676.179974] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16676.189962] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16676.216818] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16676.228188] RAX: 0000000000007818 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16676.236891] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16676.239359] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16676.250239] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16676.263368] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16676.272069] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16676.288502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16676.296845] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16676.302945] Call Trace: [16676.305030] [16676.306104] ? show_regs.cold.9+0x22/0x2f [16676.308186] ? watchdog_timer_fn.cold.6+0x62/0x10a [16676.318044] ? __hrtimer_run_queues+0x1c2/0x4d0 [16676.322162] ? watchdog+0x40/0x40 [16676.325702] ? hrtimer_interrupt+0x128/0x2c0 [16676.326865] ? smp_apic_timer_interrupt+0xa6/0x270 [16676.329019] ? apic_timer_interrupt+0xf/0x20 [16676.331819] [16676.332783] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16676.335338] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16676.338127] ? newidle_balance.isra.93+0x207/0x580 [16676.342316] do_raw_spin_lock+0x177/0x190 [16676.345760] _raw_spin_lock+0x21/0x30 [16676.347084] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16676.350394] process_one_work+0x2c8/0x700 [16676.355863] worker_thread+0x296/0x6e0 [16676.362996] ? rescuer_thread+0x570/0x570 [16676.369258] kthread+0x1d1/0x200 [16676.374077] ? set_kthread_struct+0x70/0x70 [16676.377767] ret_from_fork+0x1f/0x30 [16679.395982] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14042s! [16679.401882] Showing busy workqueues and worker pools: [16679.404889] workqueue events: flags=0x0 [16679.408010] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16679.411069] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16679.423266] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16679.430979] pending: cpu_latency_work [ptlrpc] [16679.436785] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16679.444059] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16679.450626] workqueue events_long: flags=0x0 [16679.453476] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16679.457207] pending: lu_objects_flush [obdclass] [16679.462234] workqueue events_unbound: flags=0x2 [16679.464491] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16679.467237] pending: flush_memcg_stats_dwork, idle_cull_fn [16679.469312] workqueue events_power_efficient: flags=0x80 [16679.477829] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16679.482587] pending: check_lifetime, do_cache_clean [16679.487969] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16679.493847] pending: neigh_periodic_work, neigh_periodic_work [16679.501573] workqueue mm_percpu_wq: flags=0x8 [16679.505177] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16679.511705] pending: vmstat_update, lru_add_drain_per_cpu [16679.513386] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16679.517512] pending: vmstat_update, lru_add_drain_per_cpu [16679.521215] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16679.525639] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16679.530060] workqueue xprtiod: flags=0xa [16679.532114] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16679.538334] pending: xprt_autoclose, xs_error_handle [16679.542372] workqueue ipv6_addrconf: flags=0x40008 [16679.545979] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16679.554297] pending: addrconf_verify_work [16679.562237] workqueue knbd0-recv: flags=0x1a [16679.565245] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16679.568658] in-flight: 651:recv_work [16679.570937] workqueue ptlrpc_pinger: flags=0x2 [16679.573345] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16679.577810] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16679.580543] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14042s workers=4 idle: 15979 31824 8 [16679.582994] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16670s workers=2 idle: 654 [16692.055586] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [16692.059160] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [16692.061776] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16692.062708] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16692.062716] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16692.062719] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16692.062733] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16692.062738] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16692.062743] RAX: 0000000000007f09 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16692.062745] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16692.062748] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16692.062750] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16692.062752] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16692.062756] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16692.062758] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16692.062760] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16692.062769] Call Trace: [16692.062775] [16692.063215] ? show_regs.cold.9+0x22/0x2f [16692.063227] ? watchdog_timer_fn.cold.6+0x62/0x10a [16692.063233] ? __hrtimer_run_queues+0x1c2/0x4d0 [16692.063238] ? watchdog+0x40/0x40 [16692.063242] ? hrtimer_interrupt+0x128/0x2c0 [16692.063247] ? smp_apic_timer_interrupt+0xa6/0x270 [16692.063255] ? apic_timer_interrupt+0xf/0x20 [16692.063260] [16692.063264] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16692.063271] do_raw_spin_lock+0x177/0x190 [16692.063276] _raw_spin_lock+0x21/0x30 [16692.063282] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16692.070890] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16692.070973] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16692.070981] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16692.070984] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16692.070999] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16692.071003] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16692.071008] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16692.071010] RDX: 0000000000007c33 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16692.071013] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16692.071016] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16692.071018] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16692.071021] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16692.071024] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16692.071026] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16692.071036] Call Trace: [16692.071041] [16692.071048] ? show_regs.cold.9+0x22/0x2f [16692.071057] ? watchdog_timer_fn.cold.6+0x62/0x10a [16692.071064] ? __hrtimer_run_queues+0x1c2/0x4d0 [16692.071068] ? watchdog+0x40/0x40 [16692.071073] ? hrtimer_interrupt+0x128/0x2c0 [16692.071077] ? smp_apic_timer_interrupt+0xa6/0x270 [16692.071087] ? apic_timer_interrupt+0xf/0x20 [16692.071092] [16692.071095] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16692.071100] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16692.071105] do_raw_spin_lock+0x177/0x190 [16692.071110] _raw_spin_lock+0x21/0x30 [16692.071116] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16692.127024] ? do_raw_spin_unlock+0x75/0x190 [16692.133669] obd_mod_free+0x46/0x140 [ptlrpc] [16692.141055] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16692.141417] ? do_wait_intr+0xf0/0xf0 [16692.141429] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16692.141725] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16692.142176] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16692.142562] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16692.142991] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16692.143099] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16692.143131] mdc_intent_lock+0x353/0x7e0 [mdc] [16692.143222] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16692.143320] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16692.146457] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16692.146502] lmv_intent_open+0x338/0x1150 [lmv] [16692.146539] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16692.153338] mdc_commit_open+0x91/0xb0 [mdc] [16692.153505] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16692.169989] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16692.170248] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16692.170293] ll_intent_lock+0xc5/0xaa0 [lustre] [16692.170397] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16692.170479] ll_lookup_it+0xe0e/0x2270 [lustre] [16692.170568] ? do_raw_spin_unlock+0x75/0x190 [16692.170576] ? _raw_spin_unlock+0x12/0x30 [16692.170583] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16692.170613] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16692.170635] ll_atomic_open+0x36a/0xd70 [lustre] [16692.170721] lookup_open+0x356/0xb40 [16692.170730] do_last+0x313/0xfc0 [16692.171039] ? nd_jump_root+0xe5/0x160 [16692.171046] ? path_init+0x437/0x520 [16692.171051] path_openat+0xf7/0x500 [16692.171057] ? do_raw_spin_unlock+0x75/0x190 [16692.171061] do_filp_open+0x99/0x140 [16692.171208] ? getname_flags+0x6e/0x330 [16692.176720] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16692.182195] ? __check_object_size+0xff/0x256 [16692.182228] ? do_raw_spin_unlock+0x75/0x190 [16692.182235] ? _raw_spin_unlock+0x12/0x30 [16692.182244] do_sys_openat2+0x2b4/0x410 [16692.182250] do_sys_open+0x73/0xa0 [16692.182256] __x64_sys_openat+0x24/0x30 [16692.182260] do_syscall_64+0xc1/0x440 [16692.182269] entry_SYSCALL_64_after_hwframe+0x49/0xae [16692.182277] RIP: 0033:0x7f53425a5332 [16692.182287] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16692.182292] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16692.182297] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16692.182301] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16692.182303] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16692.182305] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16692.182307] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16692.735937] after_reply+0xe2c/0x1780 [ptlrpc] [16692.739310] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16692.742871] ? __next_timer_interrupt+0x160/0x160 [16692.744529] ptlrpcd+0xa82/0xe40 [ptlrpc] [16692.748325] ? do_wait_intr+0xf0/0xf0 [16692.750367] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16692.759028] kthread+0x1d1/0x200 [16692.762458] ? set_kthread_struct+0x70/0x70 [16692.768211] ret_from_fork+0x1f/0x30 [16700.058024] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [16700.062200] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16700.088247] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16700.094151] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16700.097651] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16700.099801] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16700.105649] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16700.111176] RAX: 0000000000006152 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16700.117538] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16700.119938] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16700.122914] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16700.125397] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16700.127835] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16700.131696] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16700.133916] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16700.136618] Call Trace: [16700.137362] [16700.138140] ? show_regs.cold.9+0x22/0x2f [16700.140099] ? watchdog_timer_fn.cold.6+0x62/0x10a [16700.142245] ? __hrtimer_run_queues+0x1c2/0x4d0 [16700.144587] ? watchdog+0x40/0x40 [16700.145937] ? hrtimer_interrupt+0x128/0x2c0 [16700.147944] ? smp_apic_timer_interrupt+0xa6/0x270 [16700.150154] ? apic_timer_interrupt+0xf/0x20 [16700.152218] [16700.153064] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16700.155514] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16700.159076] do_raw_spin_lock+0x177/0x190 [16700.160673] _raw_spin_lock+0x21/0x30 [16700.162430] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16700.166153] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16700.168984] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16700.172248] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16700.175357] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16700.178067] ? woken_wake_function+0x30/0x30 [16700.180802] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16700.183952] kthread+0x1d1/0x200 [16700.185690] ? set_kthread_struct+0x70/0x70 [16700.187757] ret_from_fork+0x1f/0x30 [16704.052009] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [16704.063521] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16704.089213] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16704.093728] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16704.097870] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16704.100990] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16704.103384] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16704.111859] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16704.116411] RAX: 0000000000003412 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16704.119897] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16704.123442] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16704.129143] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16704.134637] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16704.136507] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16704.141406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16704.144511] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16704.147910] Call Trace: [16704.148668] [16704.151997] ? show_regs.cold.9+0x22/0x2f [16704.154493] ? watchdog_timer_fn.cold.6+0x62/0x10a [16704.156592] ? __hrtimer_run_queues+0x1c2/0x4d0 [16704.158154] ? watchdog+0x40/0x40 [16704.159165] ? hrtimer_interrupt+0x128/0x2c0 [16704.160504] ? smp_apic_timer_interrupt+0xa6/0x270 [16704.163512] ? apic_timer_interrupt+0xf/0x20 [16704.165213] [16704.166049] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16704.169537] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16704.171208] ? newidle_balance.isra.93+0x207/0x580 [16704.172997] do_raw_spin_lock+0x177/0x190 [16704.174413] _raw_spin_lock+0x21/0x30 [16704.175713] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16704.177754] process_one_work+0x2c8/0x700 [16704.179452] worker_thread+0x296/0x6e0 [16704.180611] ? rescuer_thread+0x570/0x570 [16704.183832] kthread+0x1d1/0x200 [16704.184784] ? set_kthread_struct+0x70/0x70 [16704.190256] ret_from_fork+0x1f/0x30 [16709.600152] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14072s! [16709.606046] Showing busy workqueues and worker pools: [16709.624573] workqueue events: flags=0x0 [16709.625928] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16709.629823] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16709.641109] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16709.647700] pending: cpu_latency_work [ptlrpc] [16709.651970] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16709.656360] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16709.662248] workqueue events_long: flags=0x0 [16709.666552] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16709.670260] pending: lu_objects_flush [obdclass] [16709.672787] workqueue events_unbound: flags=0x2 [16709.674896] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16709.678900] pending: flush_memcg_stats_dwork, idle_cull_fn [16709.684310] workqueue events_power_efficient: flags=0x80 [16709.687344] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16709.691456] pending: check_lifetime, do_cache_clean [16709.694248] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16709.697815] pending: neigh_periodic_work, neigh_periodic_work [16709.701454] workqueue mm_percpu_wq: flags=0x8 [16709.706728] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16709.712373] pending: vmstat_update, lru_add_drain_per_cpu [16709.716452] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16709.724013] pending: vmstat_update, lru_add_drain_per_cpu [16709.727392] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16709.733041] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16709.739483] workqueue xprtiod: flags=0xa [16709.744142] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16709.749389] pending: xprt_autoclose, xs_error_handle [16709.752237] workqueue ipv6_addrconf: flags=0x40008 [16709.755943] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16709.759504] pending: addrconf_verify_work [16709.762417] workqueue knbd0-recv: flags=0x1a [16709.766989] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16709.771936] in-flight: 651:recv_work [16709.773693] workqueue ptlrpc_pinger: flags=0x2 [16709.776013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16709.778861] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16709.781634] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14072s workers=4 idle: 15979 31824 8 [16709.784360] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16700s workers=2 idle: 654 [16720.056941] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [16720.063858] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [16720.064490] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16720.064560] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16720.064566] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16720.064569] RIP: 0010:native_safe_halt+0x2e/0x50 [16720.064584] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16720.064588] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16720.064593] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16720.064596] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16720.064598] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16720.064600] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16720.064603] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16720.064606] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16720.064609] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16720.064611] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16720.064620] Call Trace: [16720.064627] [16720.064633] ? show_regs.cold.9+0x22/0x2f [16720.064644] ? watchdog_timer_fn.cold.6+0x62/0x10a [16720.064651] ? __hrtimer_run_queues+0x1c2/0x4d0 [16720.064667] ? watchdog+0x40/0x40 [16720.064672] ? hrtimer_interrupt+0x128/0x2c0 [16720.064676] ? smp_apic_timer_interrupt+0xa6/0x270 [16720.064683] ? apic_timer_interrupt+0xf/0x20 [16720.064687] [16720.064691] ? native_safe_halt+0x2e/0x50 [16720.064695] kvm_wait+0xcd/0xe0 [16720.064703] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16720.064712] do_raw_spin_lock+0x177/0x190 [16720.064717] _raw_spin_lock+0x21/0x30 [16720.064722] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16720.065137] ? do_raw_spin_unlock+0x75/0x190 [16720.065144] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16720.065637] ? do_wait_intr+0xf0/0xf0 [16720.065647] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16720.065869] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16720.066141] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16720.066531] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16720.066651] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16720.066705] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16720.066736] mdc_intent_lock+0x353/0x7e0 [mdc] [16720.066774] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16720.066860] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16720.067139] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16720.067169] lmv_intent_open+0x338/0x1150 [lmv] [16720.067202] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16720.067287] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16720.067735] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16720.067760] ll_intent_lock+0xc5/0xaa0 [lustre] [16720.067845] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16720.067930] ll_lookup_it+0xe0e/0x2270 [lustre] [16720.068011] ? do_raw_spin_unlock+0x75/0x190 [16720.068018] ? _raw_spin_unlock+0x12/0x30 [16720.068149] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16720.068182] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16720.068203] ll_atomic_open+0x36a/0xd70 [lustre] [16720.068287] lookup_open+0x356/0xb40 [16720.068375] do_last+0x313/0xfc0 [16720.068380] ? nd_jump_root+0xe5/0x160 [16720.068385] ? path_init+0x437/0x520 [16720.068389] path_openat+0xf7/0x500 [16720.068394] ? do_raw_spin_unlock+0x75/0x190 [16720.068399] do_filp_open+0x99/0x140 [16720.068406] ? getname_flags+0x6e/0x330 [16720.068410] ? __check_object_size+0xff/0x256 [16720.068417] ? do_raw_spin_unlock+0x75/0x190 [16720.068422] ? _raw_spin_unlock+0x12/0x30 [16720.068426] do_sys_openat2+0x2b4/0x410 [16720.068431] do_sys_open+0x73/0xa0 [16720.068436] __x64_sys_openat+0x24/0x30 [16720.068439] do_syscall_64+0xc1/0x440 [16720.068447] entry_SYSCALL_64_after_hwframe+0x49/0xae [16720.068454] RIP: 0033:0x7f53425a5332 [16720.068462] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16720.068466] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16720.068471] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16720.068474] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16720.068476] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16720.068478] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16720.068481] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16720.425573] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16720.446730] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16720.451847] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16720.455438] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16720.458446] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16720.468760] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16720.474562] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16720.478082] RDX: 0000000000006902 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16720.480821] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16720.483519] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16720.486847] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16720.490629] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16720.496088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16720.500154] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16720.504345] Call Trace: [16720.505456] [16720.506558] ? show_regs.cold.9+0x22/0x2f [16720.508516] ? watchdog_timer_fn.cold.6+0x62/0x10a [16720.511159] ? __hrtimer_run_queues+0x1c2/0x4d0 [16720.513340] ? watchdog+0x40/0x40 [16720.515156] ? hrtimer_interrupt+0x128/0x2c0 [16720.517522] ? smp_apic_timer_interrupt+0xa6/0x270 [16720.520589] ? apic_timer_interrupt+0xf/0x20 [16720.522934] [16720.524191] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16720.528741] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16720.531507] do_raw_spin_lock+0x177/0x190 [16720.533810] _raw_spin_lock+0x21/0x30 [16720.536151] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16720.539056] obd_mod_free+0x46/0x140 [ptlrpc] [16720.542332] mdc_commit_open+0x91/0xb0 [mdc] [16720.544101] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16720.546110] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16720.549430] after_reply+0xe2c/0x1780 [ptlrpc] [16720.551473] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16720.554605] ? __next_timer_interrupt+0x160/0x160 [16720.556126] ptlrpcd+0xa82/0xe40 [ptlrpc] [16720.558382] ? do_wait_intr+0xf0/0xf0 [16720.559443] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16720.562182] kthread+0x1d1/0x200 [16720.563794] ? set_kthread_struct+0x70/0x70 [16720.567402] ret_from_fork+0x1f/0x30 [16728.058031] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [16728.060828] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16728.096192] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16728.105466] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16728.110044] RIP: 0010:native_safe_halt+0x2e/0x50 [16728.112176] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16728.139481] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [16728.144440] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16728.155636] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16728.161250] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16728.167130] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16728.171437] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16728.176015] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16728.180539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16728.183227] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16728.188099] Call Trace: [16728.190347] [16728.191351] ? show_regs.cold.9+0x22/0x2f [16728.193125] ? watchdog_timer_fn.cold.6+0x62/0x10a [16728.197057] ? __hrtimer_run_queues+0x1c2/0x4d0 [16728.199957] ? watchdog+0x40/0x40 [16728.201377] ? hrtimer_interrupt+0x128/0x2c0 [16728.204422] ? smp_apic_timer_interrupt+0xa6/0x270 [16728.206936] ? apic_timer_interrupt+0xf/0x20 [16728.210359] [16728.211352] ? native_safe_halt+0x2e/0x50 [16728.214419] kvm_wait+0xcd/0xe0 [16728.216084] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16728.219437] do_raw_spin_lock+0x177/0x190 [16728.221351] _raw_spin_lock+0x21/0x30 [16728.224234] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16728.228969] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16728.232435] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16728.235190] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16728.239165] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16728.242584] ? woken_wake_function+0x30/0x30 [16728.244082] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16728.253786] kthread+0x1d1/0x200 [16728.256595] ? set_kthread_struct+0x70/0x70 [16728.258457] ret_from_fork+0x1f/0x30 [16734.789145] rcu: INFO: rcu_sched self-detected stall on CPU [16734.792684] rcu: 2-....: (8166865 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [16734.796900] (t=14640363 jiffies g=274185 q=67742) [16734.799099] rcu: rcu_sched kthread starved for 14098729 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [16734.803941] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [16734.807316] rcu: RCU grace-period kthread stack dump: [16734.808784] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [16734.812483] Call Trace: [16734.813490] __schedule+0x351/0xcb0 [16734.815114] schedule+0xc0/0x180 [16734.816524] schedule_timeout+0xb4/0x190 [16734.818257] ? do_raw_spin_unlock+0x75/0x190 [16734.820091] ? __next_timer_interrupt+0x160/0x160 [16734.822193] ? prepare_to_swait_event+0xf3/0x140 [16734.824012] rcu_gp_kthread+0x8dd/0x15b0 [16734.825895] ? force_qs_rnp+0x2c0/0x2c0 [16734.827306] kthread+0x1d1/0x200 [16734.829087] ? set_kthread_struct+0x70/0x70 [16734.830766] ret_from_fork+0x1f/0x30 [16734.832259] rcu: Stack dump where RCU GP kthread last ran: [16734.834099] Sending NMI from CPU 2 to CPUs 1: [16734.835916] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [16734.838468] Sending NMI from CPU 2 to CPUs 1: [16734.839917] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [16734.842703] NMI backtrace for cpu 2 [16734.844375] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16734.848990] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16734.852651] Call Trace: [16734.853662] [16734.854319] ? dump_stack+0xbb/0x10e [16734.855778] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16734.857851] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16734.860457] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16734.862412] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16734.864428] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16734.866544] ? rcu_dump_cpu_stacks+0x11b/0x171 [16734.868828] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [16734.870953] ? update_process_times+0x6d/0xc0 [16734.872908] ? tick_sched_handle.isra.18+0x27/0x90 [16734.875373] ? tick_sched_timer+0x53/0xb0 [16734.876971] ? __hrtimer_run_queues+0x1c2/0x4d0 [16734.878830] ? tick_sched_handle.isra.18+0x90/0x90 [16734.880736] ? hrtimer_interrupt+0x128/0x2c0 [16734.882586] ? smp_apic_timer_interrupt+0xa6/0x270 [16734.884825] ? apic_timer_interrupt+0xf/0x20 [16734.886782] [16734.887673] ? native_safe_halt+0x2e/0x50 [16734.889928] ? kvm_wait+0xcd/0xe0 [16734.891547] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16734.894206] ? do_raw_spin_lock+0x177/0x190 [16734.896459] ? _raw_spin_lock+0x21/0x30 [16734.898225] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16734.900913] ? obd_mod_free+0x46/0x140 [ptlrpc] [16734.903371] ? mdc_commit_open+0x91/0xb0 [mdc] [16734.905467] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16734.908851] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16734.911697] ? after_reply+0xe2c/0x1780 [ptlrpc] [16734.915101] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16734.917841] ? __next_timer_interrupt+0x160/0x160 [16734.920382] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16734.922992] ? do_wait_intr+0xf0/0xf0 [16734.924475] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16734.926872] ? kthread+0x1d1/0x200 [16734.928066] ? set_kthread_struct+0x70/0x70 [16734.929783] ? ret_from_fork+0x1f/0x30 [16736.052030] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [16736.058246] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16736.108302] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16736.117750] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16736.125216] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16736.128638] RIP: 0010:native_safe_halt+0x2e/0x50 [16736.130461] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16736.143927] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16736.150613] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16736.155538] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [16736.158157] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16736.160925] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16736.163811] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16736.166785] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16736.170121] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16736.174659] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16736.182064] Call Trace: [16736.184071] [16736.185677] ? show_regs.cold.9+0x22/0x2f [16736.186947] ? watchdog_timer_fn.cold.6+0x62/0x10a [16736.190450] ? __hrtimer_run_queues+0x1c2/0x4d0 [16736.192857] ? watchdog+0x40/0x40 [16736.195229] ? hrtimer_interrupt+0x128/0x2c0 [16736.197661] ? smp_apic_timer_interrupt+0xa6/0x270 [16736.201611] ? apic_timer_interrupt+0xf/0x20 [16736.204589] [16736.205361] ? native_safe_halt+0x2e/0x50 [16736.206992] kvm_wait+0xcd/0xe0 [16736.209903] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16736.212811] ? newidle_balance.isra.93+0x207/0x580 [16736.214586] do_raw_spin_lock+0x177/0x190 [16736.216090] _raw_spin_lock+0x21/0x30 [16736.218034] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16736.221970] process_one_work+0x2c8/0x700 [16736.223856] worker_thread+0x296/0x6e0 [16736.225654] ? rescuer_thread+0x570/0x570 [16736.226961] kthread+0x1d1/0x200 [16736.228010] ? set_kthread_struct+0x70/0x70 [16736.230005] ret_from_fork+0x1f/0x30 [16739.809327] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14102s! [16739.817302] Showing busy workqueues and worker pools: [16739.820521] workqueue events: flags=0x0 [16739.823047] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16739.829199] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16739.836854] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16739.839797] pending: cpu_latency_work [ptlrpc] [16739.843737] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16739.849299] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16739.858620] workqueue events_long: flags=0x0 [16739.862722] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16739.870179] pending: lu_objects_flush [obdclass] [16739.875780] workqueue events_unbound: flags=0x2 [16739.881093] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16739.887765] pending: flush_memcg_stats_dwork, idle_cull_fn [16739.893572] workqueue events_power_efficient: flags=0x80 [16739.896702] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16739.900753] pending: check_lifetime, do_cache_clean [16739.905224] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16739.914263] pending: neigh_periodic_work, neigh_periodic_work [16739.917168] workqueue mm_percpu_wq: flags=0x8 [16739.919616] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16739.922955] pending: vmstat_update, lru_add_drain_per_cpu [16739.925885] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16739.932668] pending: vmstat_update, lru_add_drain_per_cpu [16739.935215] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16739.941323] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16739.949439] workqueue xprtiod: flags=0xa [16739.951578] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16739.957288] pending: xprt_autoclose, xs_error_handle [16739.959836] workqueue ipv6_addrconf: flags=0x40008 [16739.963489] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16739.971959] pending: addrconf_verify_work [16739.975682] workqueue knbd0-recv: flags=0x1a [16739.977709] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16739.983593] in-flight: 651:recv_work [16739.985701] workqueue ptlrpc_pinger: flags=0x2 [16739.990879] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16739.992922] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16740.000771] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14102s workers=4 idle: 15979 31824 8 [16740.014521] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16730s workers=2 idle: 654 [16756.061286] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [16756.068784] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16756.144825] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16756.173190] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16756.181019] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16756.187152] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16756.201987] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16756.208630] RAX: 0000000000007869 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16756.216923] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16756.223424] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16756.233970] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16756.240997] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16756.245432] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16756.252680] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16756.256139] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16756.263156] Call Trace: [16756.264604] [16756.268585] ? show_regs.cold.9+0x22/0x2f [16756.277550] ? watchdog_timer_fn.cold.6+0x62/0x10a [16756.281261] ? __hrtimer_run_queues+0x1c2/0x4d0 [16756.283354] ? watchdog+0x40/0x40 [16756.285165] ? hrtimer_interrupt+0x128/0x2c0 [16756.286334] ? smp_apic_timer_interrupt+0xa6/0x270 [16756.289852] ? apic_timer_interrupt+0xf/0x20 [16756.293758] [16756.294871] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16756.297417] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16756.300689] do_raw_spin_lock+0x177/0x190 [16756.302550] _raw_spin_lock+0x21/0x30 [16756.303815] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16756.306892] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16756.309188] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16756.311390] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16756.313367] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16756.315382] ? woken_wake_function+0x30/0x30 [16756.316434] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16756.318595] kthread+0x1d1/0x200 [16756.319811] ? set_kthread_struct+0x70/0x70 [16756.321552] ret_from_fork+0x1f/0x30 [16760.055416] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [16760.060932] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [16760.062013] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16760.062086] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16760.062094] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16760.062097] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16760.062117] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16760.062122] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16760.062129] RAX: 00000000000073fb RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16760.062132] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16760.062135] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16760.062137] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16760.062140] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16760.062144] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16760.062147] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16760.062154] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16760.062166] Call Trace: [16760.062181] [16760.062187] ? show_regs.cold.9+0x22/0x2f [16760.062197] ? watchdog_timer_fn.cold.6+0x62/0x10a [16760.062204] ? __hrtimer_run_queues+0x1c2/0x4d0 [16760.062208] ? watchdog+0x40/0x40 [16760.062213] ? hrtimer_interrupt+0x128/0x2c0 [16760.062217] ? smp_apic_timer_interrupt+0xa6/0x270 [16760.062227] ? apic_timer_interrupt+0xf/0x20 [16760.062232] [16760.062236] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16760.062243] do_raw_spin_lock+0x177/0x190 [16760.062248] _raw_spin_lock+0x21/0x30 [16760.062253] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16760.062644] ? do_raw_spin_unlock+0x75/0x190 [16760.062650] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16760.062848] ? do_wait_intr+0xf0/0xf0 [16760.062857] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16760.063225] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16760.063435] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16760.063658] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16760.063783] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16760.063829] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16760.063859] mdc_intent_lock+0x353/0x7e0 [mdc] [16760.063888] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16760.064109] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16760.064377] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16760.064406] lmv_intent_open+0x338/0x1150 [lmv] [16760.064436] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16760.064523] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16760.064604] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16760.064626] ll_intent_lock+0xc5/0xaa0 [lustre] [16760.064711] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16760.064794] ll_lookup_it+0xe0e/0x2270 [lustre] [16760.064876] ? do_raw_spin_unlock+0x75/0x190 [16760.064884] ? _raw_spin_unlock+0x12/0x30 [16760.064891] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16760.064917] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16760.064938] ll_atomic_open+0x36a/0xd70 [lustre] [16760.065234] lookup_open+0x356/0xb40 [16760.065245] do_last+0x313/0xfc0 [16760.065250] ? nd_jump_root+0xe5/0x160 [16760.065254] ? path_init+0x437/0x520 [16760.065260] path_openat+0xf7/0x500 [16760.065265] ? do_raw_spin_unlock+0x75/0x190 [16760.065270] do_filp_open+0x99/0x140 [16760.065276] ? getname_flags+0x6e/0x330 [16760.065280] ? __check_object_size+0xff/0x256 [16760.065290] ? do_raw_spin_unlock+0x75/0x190 [16760.065294] ? _raw_spin_unlock+0x12/0x30 [16760.065298] do_sys_openat2+0x2b4/0x410 [16760.065303] do_sys_open+0x73/0xa0 [16760.065307] __x64_sys_openat+0x24/0x30 [16760.065312] do_syscall_64+0xc1/0x440 [16760.065320] entry_SYSCALL_64_after_hwframe+0x49/0xae [16760.065327] RIP: 0033:0x7f53425a5332 [16760.065336] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16760.065340] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 [16760.068395] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16760.068475] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16760.068483] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16760.068486] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16760.068499] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16760.068504] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16760.068508] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16760.068511] RDX: 00000000000073c7 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16760.068513] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16760.068515] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16760.068517] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16760.068520] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16760.068524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16760.068526] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16760.068535] Call Trace: [16760.068541] [16760.068666] ? show_regs.cold.9+0x22/0x2f [16760.068679] ? watchdog_timer_fn.cold.6+0x62/0x10a [16760.068684] ? __hrtimer_run_queues+0x1c2/0x4d0 [16760.068688] ? watchdog+0x40/0x40 [16760.068692] ? hrtimer_interrupt+0x128/0x2c0 [16760.068696] ? smp_apic_timer_interrupt+0xa6/0x270 [16760.068709] ? apic_timer_interrupt+0xf/0x20 [16760.068715] [16760.068718] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16760.068723] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16760.068728] do_raw_spin_lock+0x177/0x190 [16760.068734] _raw_spin_lock+0x21/0x30 [16760.068740] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16760.096245] ORIG_RAX: 0000000000000101 [16760.096255] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16760.096258] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16760.096261] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16760.096264] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16760.096266] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16760.552229] obd_mod_free+0x46/0x140 [ptlrpc] [16760.560186] mdc_commit_open+0x91/0xb0 [mdc] [16760.561775] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16760.568528] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16760.579274] after_reply+0xe2c/0x1780 [ptlrpc] [16760.582931] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16760.588271] ? __next_timer_interrupt+0x160/0x160 [16760.592187] ptlrpcd+0xa82/0xe40 [ptlrpc] [16760.594466] ? do_wait_intr+0xf0/0xf0 [16760.596495] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16760.605367] kthread+0x1d1/0x200 [16760.609439] ? set_kthread_struct+0x70/0x70 [16760.615173] ret_from_fork+0x1f/0x30 [16764.052105] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [16764.055177] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16764.094953] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16764.105183] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16764.108618] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16764.117158] RIP: 0010:native_safe_halt+0x2e/0x50 [16764.119398] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16764.149724] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16764.155558] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16764.167689] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [16764.175343] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16764.183297] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16764.189720] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16764.196848] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16764.203284] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16764.206334] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16764.217416] Call Trace: [16764.219051] [16764.220515] ? show_regs.cold.9+0x22/0x2f [16764.223397] ? watchdog_timer_fn.cold.6+0x62/0x10a [16764.226433] ? __hrtimer_run_queues+0x1c2/0x4d0 [16764.230334] ? watchdog+0x40/0x40 [16764.231806] ? hrtimer_interrupt+0x128/0x2c0 [16764.233695] ? smp_apic_timer_interrupt+0xa6/0x270 [16764.235468] ? apic_timer_interrupt+0xf/0x20 [16764.237441] [16764.238189] ? native_safe_halt+0x2e/0x50 [16764.241839] kvm_wait+0xcd/0xe0 [16764.244003] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16764.246342] ? newidle_balance.isra.93+0x207/0x580 [16764.250194] do_raw_spin_lock+0x177/0x190 [16764.252298] _raw_spin_lock+0x21/0x30 [16764.254174] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16764.258242] process_one_work+0x2c8/0x700 [16764.262344] worker_thread+0x296/0x6e0 [16764.264318] ? rescuer_thread+0x570/0x570 [16764.265629] kthread+0x1d1/0x200 [16764.273046] ? set_kthread_struct+0x70/0x70 [16764.280243] ret_from_fork+0x1f/0x30 [16770.535408] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14133s! [16770.543801] Showing busy workqueues and worker pools: [16770.547412] workqueue events: flags=0x0 [16770.550144] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16770.554724] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16770.564656] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16770.571750] pending: cpu_latency_work [ptlrpc] [16770.574195] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16770.577168] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16770.584972] workqueue events_long: flags=0x0 [16770.589975] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16770.595566] pending: lu_objects_flush [obdclass] [16770.601495] workqueue events_unbound: flags=0x2 [16770.609220] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16770.612982] pending: flush_memcg_stats_dwork, idle_cull_fn [16770.615914] workqueue events_power_efficient: flags=0x80 [16770.623580] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16770.628372] pending: check_lifetime, do_cache_clean [16770.630357] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16770.634376] pending: neigh_periodic_work, neigh_periodic_work [16770.638984] workqueue mm_percpu_wq: flags=0x8 [16770.641415] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16770.644204] pending: vmstat_update, lru_add_drain_per_cpu [16770.646567] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16770.649403] pending: vmstat_update, lru_add_drain_per_cpu [16770.653073] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16770.658120] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16770.661494] workqueue xprtiod: flags=0xa [16770.663585] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16770.665748] pending: xprt_autoclose, xs_error_handle [16770.676590] workqueue ipv6_addrconf: flags=0x40008 [16770.680684] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16770.688026] pending: addrconf_verify_work [16770.690067] workqueue knbd0-recv: flags=0x1a [16770.693359] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16770.698244] in-flight: 651:recv_work [16770.701625] workqueue ptlrpc_pinger: flags=0x2 [16770.704537] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16770.709012] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16770.713084] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14133s workers=4 idle: 15979 31824 8 [16770.717732] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16761s workers=2 idle: 654 [16784.058009] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16784.063742] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16784.088346] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16784.093647] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16784.098217] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16784.101167] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16784.111015] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16784.115252] RAX: 00000000000067cd RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16784.119367] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16784.122850] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16784.126287] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16784.129378] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16784.133161] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16784.137505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16784.140229] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16784.144249] Call Trace: [16784.145601] [16784.146524] ? show_regs.cold.9+0x22/0x2f [16784.148588] ? watchdog_timer_fn.cold.6+0x62/0x10a [16784.151404] ? __hrtimer_run_queues+0x1c2/0x4d0 [16784.154882] ? watchdog+0x40/0x40 [16784.156772] ? hrtimer_interrupt+0x128/0x2c0 [16784.159363] ? smp_apic_timer_interrupt+0xa6/0x270 [16784.161830] ? apic_timer_interrupt+0xf/0x20 [16784.164450] [16784.165388] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16784.168179] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16784.170965] do_raw_spin_lock+0x177/0x190 [16784.172986] _raw_spin_lock+0x21/0x30 [16784.174202] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16784.178279] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16784.181720] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16784.185269] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16784.188489] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16784.191406] ? woken_wake_function+0x30/0x30 [16784.193574] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16784.196212] kthread+0x1d1/0x200 [16784.197594] ? set_kthread_struct+0x70/0x70 [16784.199708] ret_from_fork+0x1f/0x30 [16788.055946] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [16788.057364] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [16788.064175] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16788.064297] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16788.064313] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16788.064315] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16788.064329] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16788.064332] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16788.064336] RAX: 0000000000004b87 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16788.064338] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16788.064339] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16788.064341] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16788.064343] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16788.064345] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16788.064348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16788.064349] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16788.064356] Call Trace: [16788.064361] [16788.064365] ? show_regs.cold.9+0x22/0x2f [16788.064373] ? watchdog_timer_fn.cold.6+0x62/0x10a [16788.064379] ? __hrtimer_run_queues+0x1c2/0x4d0 [16788.064381] ? watchdog+0x40/0x40 [16788.064384] ? hrtimer_interrupt+0x128/0x2c0 [16788.064387] ? smp_apic_timer_interrupt+0xa6/0x270 [16788.064395] ? apic_timer_interrupt+0xf/0x20 [16788.064399] [16788.064401] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16788.064406] do_raw_spin_lock+0x177/0x190 [16788.064410] _raw_spin_lock+0x21/0x30 [16788.064413] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16788.065824] ? do_raw_spin_unlock+0x75/0x190 [16788.065832] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16788.066025] ? do_wait_intr+0xf0/0xf0 [16788.066034] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16788.066228] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16788.067594] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16788.068437] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16788.068756] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16788.068799] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16788.068831] mdc_intent_lock+0x353/0x7e0 [mdc] [16788.068858] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16788.068941] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16788.075365] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16788.075423] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16788.075429] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16788.075431] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16788.075443] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16788.075447] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16788.075451] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16788.075453] RDX: 0000000000004bdd RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16788.075456] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16788.075458] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16788.075609] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16788.075612] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16788.075615] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16788.075618] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16788.075625] Call Trace: [16788.075630] [16788.075635] ? show_regs.cold.9+0x22/0x2f [16788.075644] ? watchdog_timer_fn.cold.6+0x62/0x10a [16788.075650] ? __hrtimer_run_queues+0x1c2/0x4d0 [16788.075654] ? watchdog+0x40/0x40 [16788.075658] ? hrtimer_interrupt+0x128/0x2c0 [16788.075661] ? smp_apic_timer_interrupt+0xa6/0x270 [16788.075669] ? apic_timer_interrupt+0xf/0x20 [16788.075673] [16788.075680] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16788.075684] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16788.075689] do_raw_spin_lock+0x177/0x190 [16788.075693] _raw_spin_lock+0x21/0x30 [16788.075702] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16788.075957] obd_mod_free+0x46/0x140 [ptlrpc] [16788.076361] mdc_commit_open+0x91/0xb0 [mdc] [16788.076389] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16788.076698] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16788.077167] after_reply+0xe2c/0x1780 [ptlrpc] [16788.077293] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16788.079776] ? __next_timer_interrupt+0x160/0x160 [16788.079787] ptlrpcd+0xa82/0xe40 [ptlrpc] [16788.080003] ? do_wait_intr+0xf0/0xf0 [16788.080012] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16788.080225] kthread+0x1d1/0x200 [16788.080232] ? set_kthread_struct+0x70/0x70 [16788.080236] ret_from_fork+0x1f/0x30 [16788.590880] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16788.593078] lmv_intent_open+0x338/0x1150 [lmv] [16788.594817] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16788.597279] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16788.599706] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16788.601249] ll_intent_lock+0xc5/0xaa0 [lustre] [16788.603263] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16788.605247] ll_lookup_it+0xe0e/0x2270 [lustre] [16788.606828] ? do_raw_spin_unlock+0x75/0x190 [16788.608592] ? _raw_spin_unlock+0x12/0x30 [16788.610044] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16788.611768] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16788.614039] ll_atomic_open+0x36a/0xd70 [lustre] [16788.616121] lookup_open+0x356/0xb40 [16788.617754] do_last+0x313/0xfc0 [16788.619159] ? nd_jump_root+0xe5/0x160 [16788.620812] ? path_init+0x437/0x520 [16788.622222] path_openat+0xf7/0x500 [16788.623714] ? do_raw_spin_unlock+0x75/0x190 [16788.625431] do_filp_open+0x99/0x140 [16788.626869] ? getname_flags+0x6e/0x330 [16788.628380] ? __check_object_size+0xff/0x256 [16788.630004] ? do_raw_spin_unlock+0x75/0x190 [16788.632172] ? _raw_spin_unlock+0x12/0x30 [16788.633632] do_sys_openat2+0x2b4/0x410 [16788.635004] do_sys_open+0x73/0xa0 [16788.636199] __x64_sys_openat+0x24/0x30 [16788.638870] do_syscall_64+0xc1/0x440 [16788.639983] entry_SYSCALL_64_after_hwframe+0x49/0xae [16788.642679] RIP: 0033:0x7f53425a5332 [16788.644186] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16788.652149] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16788.654902] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16788.658347] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16788.661944] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16788.664968] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16788.668174] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16796.053244] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [16796.064478] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16796.107258] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16796.116699] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16796.122530] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16796.129544] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16796.135900] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16796.157180] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16796.164532] RAX: 0000000000007a10 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16796.171797] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16796.175476] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16796.185525] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16796.191657] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16796.198720] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16796.205405] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16796.208714] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16796.213916] Call Trace: [16796.217851] [16796.218806] ? show_regs.cold.9+0x22/0x2f [16796.220198] ? watchdog_timer_fn.cold.6+0x62/0x10a [16796.221912] ? __hrtimer_run_queues+0x1c2/0x4d0 [16796.227154] ? watchdog+0x40/0x40 [16796.229667] ? hrtimer_interrupt+0x128/0x2c0 [16796.235217] ? smp_apic_timer_interrupt+0xa6/0x270 [16796.237103] ? apic_timer_interrupt+0xf/0x20 [16796.241136] [16796.241917] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16796.248680] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16796.253979] ? newidle_balance.isra.93+0x207/0x580 [16796.255696] do_raw_spin_lock+0x177/0x190 [16796.257130] _raw_spin_lock+0x21/0x30 [16796.264506] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16796.267151] process_one_work+0x2c8/0x700 [16796.270928] worker_thread+0x296/0x6e0 [16796.275683] ? rescuer_thread+0x570/0x570 [16796.277135] kthread+0x1d1/0x200 [16796.281811] ? set_kthread_struct+0x70/0x70 [16796.283543] ret_from_fork+0x1f/0x30 [16800.735941] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14163s! [16800.741758] Showing busy workqueues and worker pools: [16800.746468] workqueue events: flags=0x0 [16800.748948] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16800.756748] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16800.765305] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16800.769777] pending: cpu_latency_work [ptlrpc] [16800.776974] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16800.782613] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16800.791132] workqueue events_long: flags=0x0 [16800.794171] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16800.797486] pending: lu_objects_flush [obdclass] [16800.800518] workqueue events_unbound: flags=0x2 [16800.803278] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16800.805809] pending: flush_memcg_stats_dwork, idle_cull_fn [16800.808086] workqueue events_power_efficient: flags=0x80 [16800.810661] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16800.817511] pending: check_lifetime, do_cache_clean [16800.820210] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16800.822553] pending: neigh_periodic_work, neigh_periodic_work [16800.826060] workqueue mm_percpu_wq: flags=0x8 [16800.828414] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16800.837904] pending: vmstat_update, lru_add_drain_per_cpu [16800.843168] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16800.847482] pending: vmstat_update, lru_add_drain_per_cpu [16800.849674] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16800.854659] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16800.864696] workqueue xprtiod: flags=0xa [16800.866212] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16800.876666] pending: xprt_autoclose, xs_error_handle [16800.883520] workqueue ipv6_addrconf: flags=0x40008 [16800.887689] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16800.894073] pending: addrconf_verify_work [16800.900220] workqueue knbd0-recv: flags=0x1a [16800.909196] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16800.915746] in-flight: 651:recv_work [16800.918547] workqueue ptlrpc_pinger: flags=0x2 [16800.923608] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16800.926568] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16800.928984] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14163s workers=4 idle: 15979 31824 8 [16800.932932] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16791s workers=2 idle: 654 [16812.059647] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16812.064348] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16812.087830] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16812.094466] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16812.096346] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16812.097708] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16812.111451] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16812.118862] RAX: 0000000000007e87 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16812.124224] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16812.130359] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16812.135678] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16812.138897] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16812.145713] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16812.153488] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16812.157031] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16812.163398] Call Trace: [16812.166508] [16812.167836] ? show_regs.cold.9+0x22/0x2f [16812.170214] ? watchdog_timer_fn.cold.6+0x62/0x10a [16812.174729] ? __hrtimer_run_queues+0x1c2/0x4d0 [16812.179014] ? watchdog+0x40/0x40 [16812.181679] ? hrtimer_interrupt+0x128/0x2c0 [16812.185658] ? smp_apic_timer_interrupt+0xa6/0x270 [16812.189361] ? apic_timer_interrupt+0xf/0x20 [16812.191945] [16812.193836] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16812.200215] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16812.203284] do_raw_spin_lock+0x177/0x190 [16812.205810] _raw_spin_lock+0x21/0x30 [16812.207987] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16812.214659] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16812.219359] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16812.223783] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16812.229314] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16812.234676] ? woken_wake_function+0x30/0x30 [16812.236520] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16812.241889] kthread+0x1d1/0x200 [16812.242931] ? set_kthread_struct+0x70/0x70 [16812.248387] ret_from_fork+0x1f/0x30 [16816.055369] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [16816.057101] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [16816.058856] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16816.058939] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16816.058947] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16816.058950] RIP: 0010:native_safe_halt+0x2e/0x50 [16816.058965] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16816.058969] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16816.058986] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16816.058989] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16816.058991] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16816.058993] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16816.058996] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16816.059000] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16816.059003] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16816.059005] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16816.059014] Call Trace: [16816.063854] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16816.063935] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16816.063944] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16816.063947] RIP: 0010:native_safe_halt+0x2e/0x50 [16816.063961] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16816.063966] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16816.063971] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16816.063987] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16816.063990] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16816.063993] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16816.063995] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16816.063999] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16816.064002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16816.064005] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16816.064013] Call Trace: [16816.064020] [16816.064027] ? show_regs.cold.9+0x22/0x2f [16816.064039] ? watchdog_timer_fn.cold.6+0x62/0x10a [16816.064048] ? __hrtimer_run_queues+0x1c2/0x4d0 [16816.064053] ? watchdog+0x40/0x40 [16816.064058] ? hrtimer_interrupt+0x128/0x2c0 [16816.064062] ? smp_apic_timer_interrupt+0xa6/0x270 [16816.064069] ? apic_timer_interrupt+0xf/0x20 [16816.064074] [16816.064076] ? native_safe_halt+0x2e/0x50 [16816.064080] kvm_wait+0xcd/0xe0 [16816.064089] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16816.064098] do_raw_spin_lock+0x177/0x190 [16816.064103] _raw_spin_lock+0x21/0x30 [16816.064108] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16816.121874] [16816.121889] ? show_regs.cold.9+0x22/0x2f [16816.121908] ? watchdog_timer_fn.cold.6+0x62/0x10a [16816.121915] ? __hrtimer_run_queues+0x1c2/0x4d0 [16816.121920] ? watchdog+0x40/0x40 [16816.121924] ? hrtimer_interrupt+0x128/0x2c0 [16816.121928] ? smp_apic_timer_interrupt+0xa6/0x270 [16816.121937] ? apic_timer_interrupt+0xf/0x20 [16816.121943] [16816.121946] ? native_safe_halt+0x2e/0x50 [16816.121950] kvm_wait+0xcd/0xe0 [16816.121958] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16816.121969] do_raw_spin_lock+0x177/0x190 [16816.121989] _raw_spin_lock+0x21/0x30 [16816.121995] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16816.122279] ? do_raw_spin_unlock+0x75/0x190 [16816.122286] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16816.122483] ? do_wait_intr+0xf0/0xf0 [16816.122492] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16816.122687] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16816.122905] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16816.150598] obd_mod_free+0x46/0x140 [ptlrpc] [16816.154603] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16816.154785] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16816.154827] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16816.154858] mdc_intent_lock+0x353/0x7e0 [mdc] [16816.154888] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16816.155056] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16816.157030] mdc_commit_open+0x91/0xb0 [mdc] [16816.165443] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16816.168251] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16816.172134] lmv_intent_open+0x338/0x1150 [lmv] [16816.176822] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16816.179491] after_reply+0xe2c/0x1780 [ptlrpc] [16816.179690] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16816.179893] ? __next_timer_interrupt+0x160/0x160 [16816.179902] ptlrpcd+0xa82/0xe40 [ptlrpc] [16816.180313] ? do_wait_intr+0xf0/0xf0 [16816.180324] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16816.180539] kthread+0x1d1/0x200 [16816.180545] ? set_kthread_struct+0x70/0x70 [16816.180549] ret_from_fork+0x1f/0x30 [16816.464051] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16816.467518] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16816.474136] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16816.476711] ll_intent_lock+0xc5/0xaa0 [lustre] [16816.479220] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16816.482601] ll_lookup_it+0xe0e/0x2270 [lustre] [16816.486201] ? do_raw_spin_unlock+0x75/0x190 [16816.488137] ? _raw_spin_unlock+0x12/0x30 [16816.490917] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16816.493719] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16816.496704] ll_atomic_open+0x36a/0xd70 [lustre] [16816.499717] lookup_open+0x356/0xb40 [16816.503136] do_last+0x313/0xfc0 [16816.504809] ? nd_jump_root+0xe5/0x160 [16816.506218] ? path_init+0x437/0x520 [16816.507391] path_openat+0xf7/0x500 [16816.509560] ? do_raw_spin_unlock+0x75/0x190 [16816.511945] do_filp_open+0x99/0x140 [16816.514769] ? getname_flags+0x6e/0x330 [16816.517488] ? __check_object_size+0xff/0x256 [16816.519789] ? do_raw_spin_unlock+0x75/0x190 [16816.523444] ? _raw_spin_unlock+0x12/0x30 [16816.525898] do_sys_openat2+0x2b4/0x410 [16816.527280] do_sys_open+0x73/0xa0 [16816.528771] __x64_sys_openat+0x24/0x30 [16816.529704] do_syscall_64+0xc1/0x440 [16816.530558] entry_SYSCALL_64_after_hwframe+0x49/0xae [16816.536802] RIP: 0033:0x7f53425a5332 [16816.541597] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16816.550792] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16816.553828] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16816.556998] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16816.561549] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16816.564226] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16816.568346] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16828.056011] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [16828.062667] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16828.117601] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16828.129629] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16828.140172] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16828.153253] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16828.161530] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16828.188232] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16828.195925] RAX: 0000000000007b2c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16828.200401] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16828.203786] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16828.206072] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16828.215196] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16828.223120] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16828.235376] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16828.238896] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16828.245293] Call Trace: [16828.246752] [16828.248376] ? show_regs.cold.9+0x22/0x2f [16828.252460] ? watchdog_timer_fn.cold.6+0x62/0x10a [16828.255178] ? __hrtimer_run_queues+0x1c2/0x4d0 [16828.256538] ? watchdog+0x40/0x40 [16828.257692] ? hrtimer_interrupt+0x128/0x2c0 [16828.269147] ? smp_apic_timer_interrupt+0xa6/0x270 [16828.274351] ? apic_timer_interrupt+0xf/0x20 [16828.279253] [16828.282380] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16828.286690] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16828.290469] ? newidle_balance.isra.93+0x207/0x580 [16828.293681] do_raw_spin_lock+0x177/0x190 [16828.295965] _raw_spin_lock+0x21/0x30 [16828.299339] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16828.301390] process_one_work+0x2c8/0x700 [16828.307949] worker_thread+0x296/0x6e0 [16828.311652] ? rescuer_thread+0x570/0x570 [16828.324017] kthread+0x1d1/0x200 [16828.325114] ? set_kthread_struct+0x70/0x70 [16828.326466] ret_from_fork+0x1f/0x30 [16831.463034] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14194s! [16831.466267] Showing busy workqueues and worker pools: [16831.468818] workqueue events: flags=0x0 [16831.470229] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16831.473208] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16831.477312] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16831.479537] pending: cpu_latency_work [ptlrpc] [16831.481133] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16831.483219] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16831.486593] workqueue events_long: flags=0x0 [16831.488108] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16831.490382] pending: lu_objects_flush [obdclass] [16831.492553] workqueue events_unbound: flags=0x2 [16831.494665] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16831.496654] pending: flush_memcg_stats_dwork, idle_cull_fn [16831.499097] workqueue events_power_efficient: flags=0x80 [16831.500969] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16831.503051] pending: check_lifetime, do_cache_clean [16831.504857] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16831.507143] pending: neigh_periodic_work, neigh_periodic_work [16831.510396] workqueue mm_percpu_wq: flags=0x8 [16831.512065] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16831.514402] pending: vmstat_update, lru_add_drain_per_cpu [16831.516487] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16831.519024] pending: vmstat_update, lru_add_drain_per_cpu [16831.520909] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16831.525154] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16831.528078] workqueue xprtiod: flags=0xa [16831.529636] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16831.532809] pending: xprt_autoclose, xs_error_handle [16831.536032] workqueue ipv6_addrconf: flags=0x40008 [16831.538050] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16831.540688] pending: addrconf_verify_work [16831.542260] workqueue knbd0-recv: flags=0x1a [16831.544197] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16831.546834] in-flight: 651:recv_work [16831.548350] workqueue ptlrpc_pinger: flags=0x2 [16831.549893] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16831.551937] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16831.554506] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14194s workers=4 idle: 15979 31824 8 [16831.557261] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16822s workers=2 idle: 654 [16840.061409] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16840.072517] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16840.136463] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16840.153252] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16840.167229] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16840.170426] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16840.202358] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16840.207287] RAX: 0000000000007388 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16840.210990] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16840.222159] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16840.234345] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16840.243738] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16840.251768] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16840.260409] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16840.270057] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16840.274199] Call Trace: [16840.277412] [16840.279688] ? show_regs.cold.9+0x22/0x2f [16840.290732] ? watchdog_timer_fn.cold.6+0x62/0x10a [16840.292153] ? __hrtimer_run_queues+0x1c2/0x4d0 [16840.293526] ? watchdog+0x40/0x40 [16840.299286] ? hrtimer_interrupt+0x128/0x2c0 [16840.307396] ? smp_apic_timer_interrupt+0xa6/0x270 [16840.309291] ? apic_timer_interrupt+0xf/0x20 [16840.314370] [16840.315406] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16840.320102] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16840.324171] do_raw_spin_lock+0x177/0x190 [16840.325607] _raw_spin_lock+0x21/0x30 [16840.328147] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16840.338150] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16840.341608] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16840.350441] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16840.356819] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16840.359317] ? woken_wake_function+0x30/0x30 [16840.366480] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16840.371487] kthread+0x1d1/0x200 [16840.373376] ? set_kthread_struct+0x70/0x70 [16840.381785] ret_from_fork+0x1f/0x30 [16844.056758] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [16844.057167] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [16844.066105] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16844.066182] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16844.066190] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16844.066193] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16844.066210] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16844.066214] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16844.066219] RAX: 0000000000005292 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16844.066222] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16844.066225] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16844.066227] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16844.066230] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16844.066233] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16844.066236] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16844.066239] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16844.066249] Call Trace: [16844.066255] [16844.066261] ? show_regs.cold.9+0x22/0x2f [16844.066271] ? watchdog_timer_fn.cold.6+0x62/0x10a [16844.066279] ? __hrtimer_run_queues+0x1c2/0x4d0 [16844.066284] ? watchdog+0x40/0x40 [16844.066289] ? hrtimer_interrupt+0x128/0x2c0 [16844.066293] ? smp_apic_timer_interrupt+0xa6/0x270 [16844.066302] ? apic_timer_interrupt+0xf/0x20 [16844.066308] [16844.066310] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16844.066318] do_raw_spin_lock+0x177/0x190 [16844.066323] _raw_spin_lock+0x21/0x30 [16844.066328] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16844.067746] ? do_raw_spin_unlock+0x75/0x190 [16844.067755] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16844.068242] ? do_wait_intr+0xf0/0xf0 [16844.068253] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16844.068467] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16844.069197] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16844.069445] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16844.070139] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16844.070183] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16844.070215] mdc_intent_lock+0x353/0x7e0 [mdc] [16844.070245] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16844.070340] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16844.070659] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16844.070693] lmv_intent_open+0x338/0x1150 [lmv] [16844.070722] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16844.070813] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16844.071038] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16844.071062] ll_intent_lock+0xc5/0xaa0 [lustre] [16844.071156] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16844.071248] ll_lookup_it+0xe0e/0x2270 [lustre] [16844.071333] ? do_raw_spin_unlock+0x75/0x190 [16844.071341] ? _raw_spin_unlock+0x12/0x30 [16844.071347] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16844.071373] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16844.071394] ll_atomic_open+0x36a/0xd70 [lustre] [16844.071476] lookup_open+0x356/0xb40 [16844.077368] Modules linked in: lustre(O) [16844.103335] do_last+0x313/0xfc0 [16844.103355] ? nd_jump_root+0xe5/0x160 [16844.103361] ? path_init+0x437/0x520 [16844.103367] path_openat+0xf7/0x500 [16844.103372] ? do_raw_spin_unlock+0x75/0x190 [16844.103380] do_filp_open+0x99/0x140 [16844.103386] ? getname_flags+0x6e/0x330 [16844.103391] ? __check_object_size+0xff/0x256 [16844.103398] ? do_raw_spin_unlock+0x75/0x190 [16844.103402] ? _raw_spin_unlock+0x12/0x30 [16844.103410] do_sys_openat2+0x2b4/0x410 [16844.103414] do_sys_open+0x73/0xa0 [16844.103419] __x64_sys_openat+0x24/0x30 [16844.103423] do_syscall_64+0xc1/0x440 [16844.103429] entry_SYSCALL_64_after_hwframe+0x49/0xae [16844.103435] RIP: 0033:0x7f53425a5332 [16844.103443] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16844.103447] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16844.103453] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16844.103456] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16844.103459] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16844.103462] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16844.103464] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16844.518343] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16844.544859] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16844.549938] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16844.554900] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16844.557123] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16844.565555] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16844.568951] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16844.579628] RDX: 00000000000077e8 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16844.590638] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16844.597300] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16844.606653] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16844.615167] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16844.623311] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16844.627073] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16844.631412] Call Trace: [16844.633465] [16844.634190] ? show_regs.cold.9+0x22/0x2f [16844.636220] ? watchdog_timer_fn.cold.6+0x62/0x10a [16844.642972] ? __hrtimer_run_queues+0x1c2/0x4d0 [16844.646396] ? watchdog+0x40/0x40 [16844.650706] ? hrtimer_interrupt+0x128/0x2c0 [16844.653536] ? smp_apic_timer_interrupt+0xa6/0x270 [16844.660684] ? apic_timer_interrupt+0xf/0x20 [16844.663244] [16844.664900] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16844.669974] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16844.672009] do_raw_spin_lock+0x177/0x190 [16844.673348] _raw_spin_lock+0x21/0x30 [16844.680870] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16844.691549] obd_mod_free+0x46/0x140 [ptlrpc] [16844.702402] mdc_commit_open+0x91/0xb0 [mdc] [16844.704504] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16844.707963] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16844.715076] after_reply+0xe2c/0x1780 [ptlrpc] [16844.725910] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16844.729261] ? __next_timer_interrupt+0x160/0x160 [16844.739234] ptlrpcd+0xa82/0xe40 [ptlrpc] [16844.744916] ? do_wait_intr+0xf0/0xf0 [16844.747972] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16844.750056] kthread+0x1d1/0x200 [16844.756963] ? set_kthread_struct+0x70/0x70 [16844.758629] ret_from_fork+0x1f/0x30 [16856.052033] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [16856.060707] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16856.108962] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16856.127799] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16856.136735] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16856.145891] RIP: 0010:native_safe_halt+0x2e/0x50 [16856.156383] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16856.172968] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16856.179328] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16856.185123] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16856.189501] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16856.198444] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16856.204234] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16856.206923] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16856.210761] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16856.216570] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16856.223754] Call Trace: [16856.224863] [16856.225896] ? show_regs.cold.9+0x22/0x2f [16856.230524] ? watchdog_timer_fn.cold.6+0x62/0x10a [16856.232067] ? __hrtimer_run_queues+0x1c2/0x4d0 [16856.236390] ? watchdog+0x40/0x40 [16856.238088] ? hrtimer_interrupt+0x128/0x2c0 [16856.241488] ? smp_apic_timer_interrupt+0xa6/0x270 [16856.243379] ? apic_timer_interrupt+0xf/0x20 [16856.245702] [16856.246287] ? native_safe_halt+0x2e/0x50 [16856.247845] kvm_wait+0xcd/0xe0 [16856.248990] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16856.251052] ? newidle_balance.isra.93+0x207/0x580 [16856.252745] do_raw_spin_lock+0x177/0x190 [16856.255120] _raw_spin_lock+0x21/0x30 [16856.256441] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16856.261261] process_one_work+0x2c8/0x700 [16856.262772] worker_thread+0x296/0x6e0 [16856.265502] ? rescuer_thread+0x570/0x570 [16856.266835] kthread+0x1d1/0x200 [16856.267851] ? set_kthread_struct+0x70/0x70 [16856.269913] ret_from_fork+0x1f/0x30 [16861.666293] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14224s! [16861.673312] Showing busy workqueues and worker pools: [16861.675415] workqueue events: flags=0x0 [16861.678749] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16861.682138] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16861.690990] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16861.698169] pending: cpu_latency_work [ptlrpc] [16861.705371] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16861.717330] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16861.731896] workqueue events_long: flags=0x0 [16861.736593] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16861.748562] pending: lu_objects_flush [obdclass] [16861.752677] workqueue events_unbound: flags=0x2 [16861.757150] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16861.760636] pending: flush_memcg_stats_dwork, idle_cull_fn [16861.769692] workqueue events_power_efficient: flags=0x80 [16861.773924] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16861.785134] pending: check_lifetime, do_cache_clean [16861.792945] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16861.801936] pending: neigh_periodic_work, neigh_periodic_work [16861.804097] workqueue mm_percpu_wq: flags=0x8 [16861.809288] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16861.812967] pending: vmstat_update, lru_add_drain_per_cpu [16861.816122] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16861.822078] pending: vmstat_update, lru_add_drain_per_cpu [16861.825936] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16861.834306] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16861.837046] workqueue xprtiod: flags=0xa [16861.838677] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16861.842090] pending: xprt_autoclose, xs_error_handle [16861.845869] workqueue ipv6_addrconf: flags=0x40008 [16861.857895] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16861.874093] pending: addrconf_verify_work [16861.878075] workqueue knbd0-recv: flags=0x1a [16861.881280] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16861.887028] in-flight: 651:recv_work [16861.890257] workqueue ptlrpc_pinger: flags=0x2 [16861.895674] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16861.899037] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16861.905830] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14224s workers=4 idle: 15979 31824 8 [16861.925333] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16852s workers=2 idle: 654 [16868.060718] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16868.066996] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16868.127343] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16868.138451] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16868.150025] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16868.156242] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16868.180416] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16868.189945] RAX: 0000000000007c0d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16868.207349] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16868.215566] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16868.228835] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16868.236977] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16868.239128] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16868.245277] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16868.256743] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16868.263981] Call Trace: [16868.265829] [16868.267093] ? show_regs.cold.9+0x22/0x2f [16868.270010] ? watchdog_timer_fn.cold.6+0x62/0x10a [16868.276550] ? __hrtimer_run_queues+0x1c2/0x4d0 [16868.281426] ? watchdog+0x40/0x40 [16868.285632] ? hrtimer_interrupt+0x128/0x2c0 [16868.289216] ? smp_apic_timer_interrupt+0xa6/0x270 [16868.294504] ? apic_timer_interrupt+0xf/0x20 [16868.300195] [16868.301296] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16868.307763] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16868.315539] do_raw_spin_lock+0x177/0x190 [16868.322223] _raw_spin_lock+0x21/0x30 [16868.325443] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16868.334898] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16868.340168] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16868.346936] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16868.350911] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16868.356817] ? woken_wake_function+0x30/0x30 [16868.359546] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16868.362960] kthread+0x1d1/0x200 [16868.364839] ? set_kthread_struct+0x70/0x70 [16868.367398] ret_from_fork+0x1f/0x30 [16872.057419] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [16872.066198] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [16872.069286] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16872.069368] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16872.069374] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16872.069377] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16872.069389] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16872.069393] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16872.069411] RAX: 0000000000000a1b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16872.069414] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16872.069416] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16872.069418] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16872.069420] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16872.069423] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16872.069426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16872.069429] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16872.069437] Call Trace: [16872.069443] [16872.069449] ? show_regs.cold.9+0x22/0x2f [16872.069460] ? watchdog_timer_fn.cold.6+0x62/0x10a [16872.069466] ? __hrtimer_run_queues+0x1c2/0x4d0 [16872.069470] ? watchdog+0x40/0x40 [16872.069474] ? hrtimer_interrupt+0x128/0x2c0 [16872.069478] ? smp_apic_timer_interrupt+0xa6/0x270 [16872.069486] ? apic_timer_interrupt+0xf/0x20 [16872.069491] [16872.069495] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16872.069502] do_raw_spin_lock+0x177/0x190 [16872.069507] _raw_spin_lock+0x21/0x30 [16872.069512] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16872.069775] ? do_raw_spin_unlock+0x75/0x190 [16872.069781] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16872.070382] ? do_wait_intr+0xf0/0xf0 [16872.070391] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16872.071572] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16872.089268] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16872.089900] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16872.089907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16872.089910] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16872.089925] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16872.089929] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16872.089933] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16872.089936] RDX: 0000000000006b0f RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16872.089939] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16872.089941] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16872.089944] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16872.089948] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16872.089951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16872.089954] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16872.089963] Call Trace: [16872.089969] [16872.089973] ? show_regs.cold.9+0x22/0x2f [16872.089982] ? watchdog_timer_fn.cold.6+0x62/0x10a [16872.089989] ? __hrtimer_run_queues+0x1c2/0x4d0 [16872.090237] ? watchdog+0x40/0x40 [16872.090242] ? hrtimer_interrupt+0x128/0x2c0 [16872.090247] ? smp_apic_timer_interrupt+0xa6/0x270 [16872.090254] ? apic_timer_interrupt+0xf/0x20 [16872.090259] [16872.090262] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16872.090266] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16872.090271] do_raw_spin_lock+0x177/0x190 [16872.090277] _raw_spin_lock+0x21/0x30 [16872.090282] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16872.090832] obd_mod_free+0x46/0x140 [ptlrpc] [16872.091176] mdc_commit_open+0x91/0xb0 [mdc] [16872.091212] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16872.095299] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16872.177946] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16872.178335] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16872.178575] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16872.178615] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16872.178646] mdc_intent_lock+0x353/0x7e0 [mdc] [16872.178675] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16872.179204] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16872.180551] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16872.180584] lmv_intent_open+0x338/0x1150 [lmv] [16872.180611] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16872.181162] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16872.188975] after_reply+0xe2c/0x1780 [ptlrpc] [16872.189251] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16872.189627] ? __next_timer_interrupt+0x160/0x160 [16872.189638] ptlrpcd+0xa82/0xe40 [ptlrpc] [16872.190296] ? do_wait_intr+0xf0/0xf0 [16872.190378] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16872.190589] kthread+0x1d1/0x200 [16872.190594] ? set_kthread_struct+0x70/0x70 [16872.190598] ret_from_fork+0x1f/0x30 [16872.984313] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16872.999343] ll_intent_lock+0xc5/0xaa0 [lustre] [16873.002086] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16873.003594] ll_lookup_it+0xe0e/0x2270 [lustre] [16873.011517] ? do_raw_spin_unlock+0x75/0x190 [16873.013504] ? _raw_spin_unlock+0x12/0x30 [16873.022383] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16873.036297] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16873.038511] ll_atomic_open+0x36a/0xd70 [lustre] [16873.049901] lookup_open+0x356/0xb40 [16873.060943] do_last+0x313/0xfc0 [16873.078330] ? nd_jump_root+0xe5/0x160 [16873.088650] ? path_init+0x437/0x520 [16873.089811] path_openat+0xf7/0x500 [16873.590450] ? do_raw_spin_unlock+0x75/0x190 [16873.603402] do_filp_open+0x99/0x140 [16873.606525] ? getname_flags+0x6e/0x330 [16873.618357] ? __check_object_size+0xff/0x256 [16873.626401] ? do_raw_spin_unlock+0x75/0x190 [16873.639402] ? _raw_spin_unlock+0x12/0x30 [16873.650637] do_sys_openat2+0x2b4/0x410 [16873.653587] do_sys_open+0x73/0xa0 [16873.657308] __x64_sys_openat+0x24/0x30 [16873.667439] do_syscall_64+0xc1/0x440 [16873.673392] entry_SYSCALL_64_after_hwframe+0x49/0xae [16873.688392] RIP: 0033:0x7f53425a5332 [16873.692356] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16873.721690] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16873.732570] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16873.747613] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16873.776272] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16873.790160] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16873.799197] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16888.053933] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [16888.058945] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16888.086772] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16888.094646] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16888.104151] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16888.107897] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16888.112648] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16888.132223] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16888.136014] RAX: 0000000000007308 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16888.142703] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16888.148118] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16888.151577] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16888.158284] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16888.164408] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16888.175193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16888.181902] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16888.193009] Call Trace: [16888.198629] [16888.202020] ? show_regs.cold.9+0x22/0x2f [16888.204673] ? watchdog_timer_fn.cold.6+0x62/0x10a [16888.208641] ? __hrtimer_run_queues+0x1c2/0x4d0 [16888.210929] ? watchdog+0x40/0x40 [16888.213231] ? hrtimer_interrupt+0x128/0x2c0 [16888.218176] ? smp_apic_timer_interrupt+0xa6/0x270 [16888.223849] ? apic_timer_interrupt+0xf/0x20 [16888.226146] [16888.229857] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16888.234576] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16888.242658] ? newidle_balance.isra.93+0x207/0x580 [16888.249136] do_raw_spin_lock+0x177/0x190 [16888.252199] _raw_spin_lock+0x21/0x30 [16888.258640] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16888.266378] process_one_work+0x2c8/0x700 [16888.271551] worker_thread+0x296/0x6e0 [16888.276690] ? rescuer_thread+0x570/0x570 [16888.282248] kthread+0x1d1/0x200 [16888.286953] ? set_kthread_struct+0x70/0x70 [16888.292427] ret_from_fork+0x1f/0x30 [16892.383214] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14255s! [16892.393123] Showing busy workqueues and worker pools: [16892.402318] workqueue events: flags=0x0 [16892.404089] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16892.408905] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16892.418115] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16892.425375] pending: cpu_latency_work [ptlrpc] [16892.435371] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16892.441086] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16892.447357] workqueue events_long: flags=0x0 [16892.451054] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16892.454447] pending: lu_objects_flush [obdclass] [16892.459957] workqueue events_unbound: flags=0x2 [16892.464025] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16892.471065] pending: flush_memcg_stats_dwork, idle_cull_fn [16892.475169] workqueue events_power_efficient: flags=0x80 [16892.477281] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16892.485572] pending: check_lifetime, do_cache_clean [16892.487427] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16892.491831] pending: neigh_periodic_work, neigh_periodic_work [16892.495181] workqueue mm_percpu_wq: flags=0x8 [16892.505448] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16892.513441] pending: vmstat_update, lru_add_drain_per_cpu [16892.516629] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16892.522458] pending: vmstat_update, lru_add_drain_per_cpu [16892.528546] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16892.531787] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16892.540828] workqueue xprtiod: flags=0xa [16892.544172] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16892.547299] pending: xprt_autoclose, xs_error_handle [16892.552228] workqueue ipv6_addrconf: flags=0x40008 [16892.556562] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16892.560056] pending: addrconf_verify_work [16892.566018] workqueue knbd0-recv: flags=0x1a [16892.567394] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16892.571306] in-flight: 651:recv_work [16892.573322] workqueue ptlrpc_pinger: flags=0x2 [16892.578013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16892.581257] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16892.587385] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14255s workers=4 idle: 15979 31824 8 [16892.595768] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16883s workers=2 idle: 654 [16896.058043] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16896.066511] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16896.099394] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16896.116940] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16896.119918] RIP: 0010:native_safe_halt+0x2e/0x50 [16896.122154] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16896.133245] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16896.137951] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16896.140763] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16896.144249] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16896.146273] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16896.148426] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16896.150547] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16896.162874] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16896.169665] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16896.175788] Call Trace: [16896.178817] [16896.179904] ? show_regs.cold.9+0x22/0x2f [16896.182153] ? watchdog_timer_fn.cold.6+0x62/0x10a [16896.185377] ? __hrtimer_run_queues+0x1c2/0x4d0 [16896.188024] ? watchdog+0x40/0x40 [16896.191191] ? hrtimer_interrupt+0x128/0x2c0 [16896.192606] ? smp_apic_timer_interrupt+0xa6/0x270 [16896.193845] ? apic_timer_interrupt+0xf/0x20 [16896.196811] [16896.197490] ? native_safe_halt+0x2e/0x50 [16896.199662] kvm_wait+0xcd/0xe0 [16896.202001] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16896.204893] do_raw_spin_lock+0x177/0x190 [16896.207712] _raw_spin_lock+0x21/0x30 [16896.209397] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16896.211908] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16896.217286] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16896.219837] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16896.223743] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16896.226283] ? woken_wake_function+0x30/0x30 [16896.227461] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16896.229539] kthread+0x1d1/0x200 [16896.230698] ? set_kthread_struct+0x70/0x70 [16896.232133] ret_from_fork+0x1f/0x30 [16900.059130] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [16900.061340] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [16900.071859] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16900.071940] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16900.071948] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16900.071951] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16900.071966] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16900.071971] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16900.071976] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16900.071978] RDX: 00000000000072f7 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16900.071980] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16900.071983] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16900.071986] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16900.071989] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16900.071993] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16900.071995] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16900.072003] Call Trace: [16900.072009] [16900.072015] ? show_regs.cold.9+0x22/0x2f [16900.072024] ? watchdog_timer_fn.cold.6+0x62/0x10a [16900.072031] ? __hrtimer_run_queues+0x1c2/0x4d0 [16900.072035] ? watchdog+0x40/0x40 [16900.072040] ? hrtimer_interrupt+0x128/0x2c0 [16900.072044] ? smp_apic_timer_interrupt+0xa6/0x270 [16900.072052] ? apic_timer_interrupt+0xf/0x20 [16900.072057] [16900.072060] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16900.072064] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16900.072069] do_raw_spin_lock+0x177/0x190 [16900.072073] _raw_spin_lock+0x21/0x30 [16900.072079] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16900.072347] obd_mod_free+0x46/0x140 [ptlrpc] [16900.072563] mdc_commit_open+0x91/0xb0 [mdc] [16900.072600] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16900.072799] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16900.072996] after_reply+0xe2c/0x1780 [ptlrpc] [16900.073199] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16900.073398] ? __next_timer_interrupt+0x160/0x160 [16900.073407] ptlrpcd+0xa82/0xe40 [ptlrpc] [16900.073616] ? do_wait_intr+0xf0/0xf0 [16900.073626] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16900.073822] kthread+0x1d1/0x200 [16900.073828] ? set_kthread_struct+0x70/0x70 [16900.073832] ret_from_fork+0x1f/0x30 [16900.333241] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16900.367023] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16900.373028] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16900.376800] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16900.379318] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16900.387205] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16900.390430] RAX: 0000000000003708 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16900.393426] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16900.396449] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16900.399449] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16900.402386] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16900.405362] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16900.408730] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16900.411009] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16900.415019] Call Trace: [16900.416439] [16900.417306] ? show_regs.cold.9+0x22/0x2f [16900.418814] ? watchdog_timer_fn.cold.6+0x62/0x10a [16900.421185] ? __hrtimer_run_queues+0x1c2/0x4d0 [16900.423373] ? watchdog+0x40/0x40 [16900.425164] ? hrtimer_interrupt+0x128/0x2c0 [16900.427828] ? smp_apic_timer_interrupt+0xa6/0x270 [16900.430094] ? apic_timer_interrupt+0xf/0x20 [16900.431816] [16900.433010] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16900.435614] do_raw_spin_lock+0x177/0x190 [16900.437182] _raw_spin_lock+0x21/0x30 [16900.438846] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16900.441732] ? do_raw_spin_unlock+0x75/0x190 [16900.443671] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16900.447150] ? do_wait_intr+0xf0/0xf0 [16900.449083] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16900.451682] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16900.453912] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16900.456438] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16900.458752] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16900.460987] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16900.463447] mdc_intent_lock+0x353/0x7e0 [mdc] [16900.465439] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16900.467405] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16900.471137] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16900.473424] lmv_intent_open+0x338/0x1150 [lmv] [16900.477334] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16900.481454] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16900.485124] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16900.491192] ll_intent_lock+0xc5/0xaa0 [lustre] [16900.494435] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16900.498139] ll_lookup_it+0xe0e/0x2270 [lustre] [16900.505309] ? do_raw_spin_unlock+0x75/0x190 [16900.512215] ? _raw_spin_unlock+0x12/0x30 [16900.515393] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16900.519116] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16900.523061] ll_atomic_open+0x36a/0xd70 [lustre] [16900.524885] lookup_open+0x356/0xb40 [16900.525946] do_last+0x313/0xfc0 [16900.527139] ? nd_jump_root+0xe5/0x160 [16900.537906] ? path_init+0x437/0x520 [16900.543051] path_openat+0xf7/0x500 [16900.545155] ? do_raw_spin_unlock+0x75/0x190 [16900.548633] do_filp_open+0x99/0x140 [16900.551143] ? getname_flags+0x6e/0x330 [16900.554866] ? __check_object_size+0xff/0x256 [16900.557952] ? do_raw_spin_unlock+0x75/0x190 [16900.559465] ? _raw_spin_unlock+0x12/0x30 [16900.561442] do_sys_openat2+0x2b4/0x410 [16900.563174] do_sys_open+0x73/0xa0 [16900.564835] __x64_sys_openat+0x24/0x30 [16900.566793] do_syscall_64+0xc1/0x440 [16900.570068] entry_SYSCALL_64_after_hwframe+0x49/0xae [16900.576469] RIP: 0033:0x7f53425a5332 [16900.582200] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16900.594042] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16900.596347] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16900.600270] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16900.604191] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16900.608854] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16900.613801] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16914.793889] rcu: INFO: rcu_sched self-detected stall on CPU [16914.796807] rcu: 2-....: (8265302 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [16914.803229] (t=14820367 jiffies g=274185 q=67810) [16914.806926] rcu: rcu_sched kthread starved for 14278733 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [16914.813233] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [16914.817956] rcu: RCU grace-period kthread stack dump: [16914.820190] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [16914.824203] Call Trace: [16914.825374] __schedule+0x351/0xcb0 [16914.827178] schedule+0xc0/0x180 [16914.829649] schedule_timeout+0xb4/0x190 [16914.831998] ? do_raw_spin_unlock+0x75/0x190 [16914.835684] ? __next_timer_interrupt+0x160/0x160 [16914.841174] ? prepare_to_swait_event+0xf3/0x140 [16914.844679] rcu_gp_kthread+0x8dd/0x15b0 [16914.849060] ? force_qs_rnp+0x2c0/0x2c0 [16914.854806] kthread+0x1d1/0x200 [16914.857226] ? set_kthread_struct+0x70/0x70 [16914.859188] ret_from_fork+0x1f/0x30 [16914.860618] rcu: Stack dump where RCU GP kthread last ran: [16914.864438] NMI backtrace for cpu 2 [16914.868544] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16914.874716] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16914.884322] Call Trace: [16914.885212] [16914.888613] ? dump_stack+0xbb/0x10e [16914.890212] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16914.891913] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16914.902178] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16914.906266] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16914.908453] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16914.911935] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [16914.918176] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [16914.925560] ? update_process_times+0x6d/0xc0 [16914.929030] ? tick_sched_handle.isra.18+0x27/0x90 [16914.934150] ? tick_sched_timer+0x53/0xb0 [16914.939671] ? __hrtimer_run_queues+0x1c2/0x4d0 [16914.942329] ? tick_sched_handle.isra.18+0x90/0x90 [16914.947707] ? hrtimer_interrupt+0x128/0x2c0 [16914.949714] ? smp_apic_timer_interrupt+0xa6/0x270 [16914.955450] ? apic_timer_interrupt+0xf/0x20 [16914.959608] [16914.960532] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16914.963214] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16914.965794] ? do_raw_spin_lock+0x177/0x190 [16914.971757] ? _raw_spin_lock+0x21/0x30 [16914.973443] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16914.978817] ? obd_mod_free+0x46/0x140 [ptlrpc] [16914.981796] ? mdc_commit_open+0x91/0xb0 [mdc] [16914.984026] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16914.986500] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16914.991387] ? after_reply+0xe2c/0x1780 [ptlrpc] [16914.996882] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16915.001157] ? __next_timer_interrupt+0x160/0x160 [16915.011753] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16915.013573] ? do_wait_intr+0xf0/0xf0 [16915.017227] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16915.020770] ? kthread+0x1d1/0x200 [16915.023539] ? set_kthread_struct+0x70/0x70 [16915.025259] ? ret_from_fork+0x1f/0x30 [16915.026758] Sending NMI from CPU 2 to CPUs 1: [16915.030764] NMI backtrace for cpu 1 [16915.030770] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16915.030773] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16915.030775] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16915.030782] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16915.030785] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [16915.030788] RAX: 000000000000132e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16915.030789] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16915.030790] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16915.030791] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16915.030793] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16915.030795] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16915.030796] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16915.030797] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16915.030799] Call Trace: [16915.030966] [16915.030967] ? show_regs.cold.9+0x22/0x2f [16915.030968] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [16915.030970] ? nmi_cpu_backtrace_handler+0x15/0x40 [16915.030971] ? nmi_handle+0xbd/0x210 [16915.030972] ? default_do_nmi+0x4b/0x1b0 [16915.030973] ? do_nmi+0x280/0x380 [16915.030975] ? end_repeat_nmi+0x16/0x1f [16915.030976] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16915.030977] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16915.030978] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16915.030979] [16915.030980] do_raw_spin_lock+0x177/0x190 [16915.030981] _raw_spin_lock+0x21/0x30 [16915.030983] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16915.030984] ? do_raw_spin_unlock+0x75/0x190 [16915.030985] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16915.030986] ? do_wait_intr+0xf0/0xf0 [16915.030987] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16915.030988] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16915.030989] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16915.030990] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16915.030991] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16915.030992] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16915.030993] mdc_intent_lock+0x353/0x7e0 [mdc] [16915.030994] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16915.030995] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16915.030996] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16915.030997] lmv_intent_open+0x338/0x1150 [lmv] [16915.030998] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16915.030999] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16915.031001] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16915.031002] ll_intent_lock+0xc5/0xaa0 [lustre] [16915.031003] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16915.031004] ll_lookup_it+0xe0e/0x2270 [lustre] [16915.031005] ? do_raw_spin_unlock+0x75/0x190 [16915.031006] ? _raw_spin_unlock+0x12/0x30 [16915.031007] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16915.031008] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16915.031009] ll_atomic_open+0x36a/0xd70 [lustre] [16915.031010] lookup_open+0x356/0xb40 [16915.031011] do_last+0x313/0xfc0 [16915.031012] ? nd_jump_root+0xe5/0x160 [16915.031013] ? path_init+0x437/0x520 [16915.031014] path_openat+0xf7/0x500 [16915.031015] ? do_raw_spin_unlock+0x75/0x190 [16915.031016] do_filp_open+0x99/0x140 [16915.031017] ? getname_flags+0x6e/0x330 [16915.031018] ? __check_object_size+0xff/0x256 [16915.031019] ? do_raw_spin_unlock+0x75/0x190 [16915.031020] ? _raw_spin_unlock+0x12/0x30 [16915.031021] do_sys_openat2+0x2b4/0x410 [16915.031022] do_sys_open+0x73/0xa0 [16915.031023] __x64_sys_openat+0x24/0x30 [16915.031025] do_syscall_64+0xc1/0x440 [16915.031027] entry_SYSCALL_64_after_hwframe+0x49/0xae [16915.031028] RIP: 0033:0x7f53425a5332 [16915.031029] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16915.031032] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16915.031035] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16915.031036] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16915.031038] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16915.031039] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16915.031040] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16915.033498] NMI backtrace for cpu 2 [16915.415178] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16915.425675] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16915.432860] Call Trace: [16915.433725] [16915.434316] ? dump_stack+0xbb/0x10e [16915.448231] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [16915.449831] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [16915.453907] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [16915.455520] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [16915.460078] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [16915.462395] ? rcu_dump_cpu_stacks+0x11b/0x171 [16915.464687] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [16915.467737] ? update_process_times+0x6d/0xc0 [16915.473122] ? tick_sched_handle.isra.18+0x27/0x90 [16915.474529] ? tick_sched_timer+0x53/0xb0 [16915.475942] ? __hrtimer_run_queues+0x1c2/0x4d0 [16915.483923] ? tick_sched_handle.isra.18+0x90/0x90 [16915.486206] ? hrtimer_interrupt+0x128/0x2c0 [16915.488199] ? smp_apic_timer_interrupt+0xa6/0x270 [16915.491663] ? apic_timer_interrupt+0xf/0x20 [16915.496978] [16915.501459] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16915.503149] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16915.505338] ? do_raw_spin_lock+0x177/0x190 [16915.510673] ? _raw_spin_lock+0x21/0x30 [16915.513898] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16915.521484] ? obd_mod_free+0x46/0x140 [ptlrpc] [16915.525002] ? mdc_commit_open+0x91/0xb0 [mdc] [16915.529254] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16915.532972] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16915.538989] ? after_reply+0xe2c/0x1780 [ptlrpc] [16915.546735] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16915.550669] ? __next_timer_interrupt+0x160/0x160 [16915.553151] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [16915.557066] ? do_wait_intr+0xf0/0xf0 [16915.559094] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16915.561354] ? kthread+0x1d1/0x200 [16915.562679] ? set_kthread_struct+0x70/0x70 [16915.565506] ? ret_from_fork+0x1f/0x30 [16920.052562] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [16920.057700] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16920.079300] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16920.089809] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16920.095072] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16920.101517] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16920.105823] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16920.116236] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16920.119702] RAX: 0000000000006c1c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16920.123077] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16920.127761] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16920.132358] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16920.138628] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16920.141750] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16920.144968] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16920.147575] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16920.151638] Call Trace: [16920.152494] [16920.154810] ? show_regs.cold.9+0x22/0x2f [16920.156312] ? watchdog_timer_fn.cold.6+0x62/0x10a [16920.157957] ? __hrtimer_run_queues+0x1c2/0x4d0 [16920.162447] ? watchdog+0x40/0x40 [16920.164756] ? hrtimer_interrupt+0x128/0x2c0 [16920.170714] ? smp_apic_timer_interrupt+0xa6/0x270 [16920.172784] ? apic_timer_interrupt+0xf/0x20 [16920.174777] [16920.176506] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16920.179114] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16920.181832] ? newidle_balance.isra.93+0x207/0x580 [16920.184813] do_raw_spin_lock+0x177/0x190 [16920.187855] _raw_spin_lock+0x21/0x30 [16920.191099] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16920.200785] process_one_work+0x2c8/0x700 [16920.203807] worker_thread+0x296/0x6e0 [16920.208443] ? rescuer_thread+0x570/0x570 [16920.212276] kthread+0x1d1/0x200 [16920.213684] ? set_kthread_struct+0x70/0x70 [16920.217675] ret_from_fork+0x1f/0x30 [16923.110027] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14286s! [16923.115549] Showing busy workqueues and worker pools: [16923.118965] workqueue events: flags=0x0 [16923.122025] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16923.126200] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16923.134539] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16923.136040] pending: cpu_latency_work [ptlrpc] [16923.144465] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16923.147650] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16923.152548] workqueue events_long: flags=0x0 [16923.155037] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16923.158086] pending: lu_objects_flush [obdclass] [16923.161312] workqueue events_unbound: flags=0x2 [16923.165542] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16923.172446] pending: flush_memcg_stats_dwork, idle_cull_fn [16923.176549] workqueue events_power_efficient: flags=0x80 [16923.178813] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16923.182220] pending: check_lifetime, do_cache_clean [16923.184533] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16923.187001] pending: neigh_periodic_work, neigh_periodic_work [16923.189432] workqueue mm_percpu_wq: flags=0x8 [16923.191284] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16923.194284] pending: vmstat_update, lru_add_drain_per_cpu [16923.196255] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16923.198886] pending: vmstat_update, lru_add_drain_per_cpu [16923.200283] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16923.202786] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16923.204864] workqueue xprtiod: flags=0xa [16923.206492] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16923.210461] pending: xprt_autoclose, xs_error_handle [16923.214172] workqueue ipv6_addrconf: flags=0x40008 [16923.223422] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16923.232337] pending: addrconf_verify_work [16923.237297] workqueue knbd0-recv: flags=0x1a [16923.241075] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16923.244612] in-flight: 651:recv_work [16923.247655] workqueue ptlrpc_pinger: flags=0x2 [16923.251285] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16923.254341] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16923.260659] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14286s workers=4 idle: 15979 31824 8 [16923.266519] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16913s workers=2 idle: 654 [16924.058096] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16924.066162] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16924.096635] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16924.102694] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16924.106747] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16924.109031] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16924.116957] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16924.120382] RAX: 0000000000004480 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16924.123819] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16924.126989] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16924.130499] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16924.133716] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16924.136878] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16924.140563] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16924.142986] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16924.145864] Call Trace: [16924.147093] [16924.147799] ? show_regs.cold.9+0x22/0x2f [16924.149561] ? watchdog_timer_fn.cold.6+0x62/0x10a [16924.151565] ? __hrtimer_run_queues+0x1c2/0x4d0 [16924.153323] ? watchdog+0x40/0x40 [16924.154691] ? hrtimer_interrupt+0x128/0x2c0 [16924.156471] ? smp_apic_timer_interrupt+0xa6/0x270 [16924.158375] ? apic_timer_interrupt+0xf/0x20 [16924.160434] [16924.161867] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16924.164429] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16924.166768] do_raw_spin_lock+0x177/0x190 [16924.168769] _raw_spin_lock+0x21/0x30 [16924.170190] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16924.173530] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16924.176927] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16924.179614] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16924.182708] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16924.185496] ? woken_wake_function+0x30/0x30 [16924.187400] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16924.190839] kthread+0x1d1/0x200 [16924.192477] ? set_kthread_struct+0x70/0x70 [16924.194193] ret_from_fork+0x1f/0x30 [16940.060197] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [16940.064658] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [16940.068550] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16940.068819] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16940.068826] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16940.068829] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16940.068843] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16940.068847] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16940.068851] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16940.068853] RDX: 00000000000047c3 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16940.068856] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16940.068858] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16940.068861] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16940.068864] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16940.068867] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16940.068869] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16940.068883] Call Trace: [16940.068890] [16940.068899] ? show_regs.cold.9+0x22/0x2f [16940.068908] ? watchdog_timer_fn.cold.6+0x62/0x10a [16940.068915] ? __hrtimer_run_queues+0x1c2/0x4d0 [16940.068919] ? watchdog+0x40/0x40 [16940.068923] ? hrtimer_interrupt+0x128/0x2c0 [16940.068927] ? smp_apic_timer_interrupt+0xa6/0x270 [16940.068935] ? apic_timer_interrupt+0xf/0x20 [16940.068939] [16940.068942] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16940.068951] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16940.068955] do_raw_spin_lock+0x177/0x190 [16940.068959] _raw_spin_lock+0x21/0x30 [16940.068964] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16940.069473] obd_mod_free+0x46/0x140 [ptlrpc] [16940.069909] mdc_commit_open+0x91/0xb0 [mdc] [16940.069956] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16940.091773] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16940.091995] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16940.092002] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16940.092005] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16940.092020] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16940.092026] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16940.092031] RAX: 0000000000007a15 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16940.092033] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16940.092036] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16940.092038] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16940.092041] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16940.092044] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16940.092047] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16940.092049] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16940.092059] Call Trace: [16940.092064] [16940.092069] ? show_regs.cold.9+0x22/0x2f [16940.092079] ? watchdog_timer_fn.cold.6+0x62/0x10a [16940.092085] ? __hrtimer_run_queues+0x1c2/0x4d0 [16940.092089] ? watchdog+0x40/0x40 [16940.092094] ? hrtimer_interrupt+0x128/0x2c0 [16940.092099] ? smp_apic_timer_interrupt+0xa6/0x270 [16940.092106] ? apic_timer_interrupt+0xf/0x20 [16940.092111] [16940.092114] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16940.092121] do_raw_spin_lock+0x177/0x190 [16940.092125] _raw_spin_lock+0x21/0x30 [16940.092130] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16940.092541] ? do_raw_spin_unlock+0x75/0x190 [16940.092548] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16940.092744] ? do_wait_intr+0xf0/0xf0 [16940.092752] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16940.129363] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16940.129864] after_reply+0xe2c/0x1780 [ptlrpc] [16940.130270] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16940.130555] ? __next_timer_interrupt+0x160/0x160 [16940.130566] ptlrpcd+0xa82/0xe40 [ptlrpc] [16940.131442] ? do_wait_intr+0xf0/0xf0 [16940.131453] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16940.132207] kthread+0x1d1/0x200 [16940.132215] ? set_kthread_struct+0x70/0x70 [16940.132220] ret_from_fork+0x1f/0x30 [16940.462468] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16940.465057] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16940.467617] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16940.469780] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16940.472693] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16940.474688] mdc_intent_lock+0x353/0x7e0 [mdc] [16940.477130] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16940.479600] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16940.482074] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16940.484090] lmv_intent_open+0x338/0x1150 [lmv] [16940.486003] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16940.487391] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16940.489622] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16940.491123] ll_intent_lock+0xc5/0xaa0 [lustre] [16940.493202] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16940.495405] ll_lookup_it+0xe0e/0x2270 [lustre] [16940.497547] ? do_raw_spin_unlock+0x75/0x190 [16940.499294] ? _raw_spin_unlock+0x12/0x30 [16940.501129] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16940.502876] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16940.504932] ll_atomic_open+0x36a/0xd70 [lustre] [16940.506964] lookup_open+0x356/0xb40 [16940.508347] do_last+0x313/0xfc0 [16940.509682] ? nd_jump_root+0xe5/0x160 [16940.511013] ? path_init+0x437/0x520 [16940.512184] path_openat+0xf7/0x500 [16940.513680] ? do_raw_spin_unlock+0x75/0x190 [16940.515095] do_filp_open+0x99/0x140 [16940.516317] ? getname_flags+0x6e/0x330 [16940.518009] ? __check_object_size+0xff/0x256 [16940.519616] ? do_raw_spin_unlock+0x75/0x190 [16940.521194] ? _raw_spin_unlock+0x12/0x30 [16940.522516] do_sys_openat2+0x2b4/0x410 [16940.524204] do_sys_open+0x73/0xa0 [16940.525337] __x64_sys_openat+0x24/0x30 [16940.526716] do_syscall_64+0xc1/0x440 [16940.528602] entry_SYSCALL_64_after_hwframe+0x49/0xae [16940.530185] RIP: 0033:0x7f53425a5332 [16940.531573] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16940.541583] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16940.545579] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16940.548283] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16940.550839] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16940.553668] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16940.556124] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16948.052071] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [16948.055887] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16948.071199] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16948.075308] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16948.078247] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16948.080451] RIP: 0010:native_safe_halt+0x2e/0x50 [16948.082621] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [16948.089966] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16948.093206] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16948.095766] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16948.098366] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16948.101069] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16948.103104] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16948.105501] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16948.108048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16948.109479] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16948.111828] Call Trace: [16948.112448] [16948.113058] ? show_regs.cold.9+0x22/0x2f [16948.114180] ? watchdog_timer_fn.cold.6+0x62/0x10a [16948.116010] ? __hrtimer_run_queues+0x1c2/0x4d0 [16948.117213] ? watchdog+0x40/0x40 [16948.118533] ? hrtimer_interrupt+0x128/0x2c0 [16948.119806] ? smp_apic_timer_interrupt+0xa6/0x270 [16948.121578] ? apic_timer_interrupt+0xf/0x20 [16948.122946] [16948.123460] ? native_safe_halt+0x2e/0x50 [16948.124779] kvm_wait+0xcd/0xe0 [16948.125962] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16948.127764] ? newidle_balance.isra.93+0x207/0x580 [16948.129617] do_raw_spin_lock+0x177/0x190 [16948.131957] _raw_spin_lock+0x21/0x30 [16948.133464] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16948.136601] process_one_work+0x2c8/0x700 [16948.139349] worker_thread+0x296/0x6e0 [16948.140807] ? rescuer_thread+0x570/0x570 [16948.142481] kthread+0x1d1/0x200 [16948.144150] ? set_kthread_struct+0x70/0x70 [16948.145777] ret_from_fork+0x1f/0x30 [16952.058141] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16952.062252] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16952.086673] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16952.092340] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16952.097685] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16952.100492] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16952.110437] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16952.117096] RAX: 0000000000005041 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16952.121062] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16952.125159] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16952.128684] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16952.132744] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16952.136499] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16952.140436] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16952.143571] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16952.147751] Call Trace: [16952.148814] [16952.149977] ? show_regs.cold.9+0x22/0x2f [16952.152317] ? watchdog_timer_fn.cold.6+0x62/0x10a [16952.154369] ? __hrtimer_run_queues+0x1c2/0x4d0 [16952.156785] ? watchdog+0x40/0x40 [16952.158785] ? hrtimer_interrupt+0x128/0x2c0 [16952.160455] ? smp_apic_timer_interrupt+0xa6/0x270 [16952.162098] ? apic_timer_interrupt+0xf/0x20 [16952.165220] [16952.166162] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16952.169083] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16952.171866] do_raw_spin_lock+0x177/0x190 [16952.173657] _raw_spin_lock+0x21/0x30 [16952.175578] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16952.179160] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16952.182205] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16952.186310] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16952.189283] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16952.192255] ? woken_wake_function+0x30/0x30 [16952.195362] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16952.198716] kthread+0x1d1/0x200 [16952.201111] ? set_kthread_struct+0x70/0x70 [16952.202627] ret_from_fork+0x1f/0x30 [16953.312046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14316s! [16953.316732] Showing busy workqueues and worker pools: [16953.319047] workqueue events: flags=0x0 [16953.321132] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16953.325425] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16953.330951] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16953.334154] pending: cpu_latency_work [ptlrpc] [16953.336234] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16953.339191] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16953.343194] workqueue events_long: flags=0x0 [16953.345456] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16953.347875] pending: lu_objects_flush [obdclass] [16953.350007] workqueue events_unbound: flags=0x2 [16953.351666] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16953.354285] pending: flush_memcg_stats_dwork, idle_cull_fn [16953.356846] workqueue events_power_efficient: flags=0x80 [16953.358841] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16953.361531] pending: check_lifetime, do_cache_clean [16953.363343] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16953.366074] pending: neigh_periodic_work, neigh_periodic_work [16953.368886] workqueue mm_percpu_wq: flags=0x8 [16953.370434] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16953.373493] pending: vmstat_update, lru_add_drain_per_cpu [16953.376643] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16953.381119] pending: vmstat_update, lru_add_drain_per_cpu [16953.385121] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16953.388273] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16953.390946] workqueue xprtiod: flags=0xa [16953.392526] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16953.395588] pending: xprt_autoclose, xs_error_handle [16953.398402] workqueue ipv6_addrconf: flags=0x40008 [16953.400405] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16953.403293] pending: addrconf_verify_work [16953.406301] workqueue knbd0-recv: flags=0x1a [16953.407968] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16953.410428] in-flight: 651:recv_work [16953.412062] workqueue ptlrpc_pinger: flags=0x2 [16953.414023] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16953.416221] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16953.418933] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14316s workers=4 idle: 15979 31824 8 [16953.422861] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16943s workers=2 idle: 654 [16968.055014] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [16968.057122] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [16968.058099] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16968.058170] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16968.058178] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16968.058181] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [16968.058194] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [16968.058198] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16968.058202] RAX: 00000000000008c0 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16968.058205] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16968.058207] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16968.058209] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16968.058211] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16968.058214] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16968.058216] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16968.058218] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16968.058227] Call Trace: [16968.058233] [16968.058239] ? show_regs.cold.9+0x22/0x2f [16968.058249] ? watchdog_timer_fn.cold.6+0x62/0x10a [16968.058255] ? __hrtimer_run_queues+0x1c2/0x4d0 [16968.058260] ? watchdog+0x40/0x40 [16968.058264] ? hrtimer_interrupt+0x128/0x2c0 [16968.058268] ? smp_apic_timer_interrupt+0xa6/0x270 [16968.058277] ? apic_timer_interrupt+0xf/0x20 [16968.058282] [16968.058285] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [16968.058291] do_raw_spin_lock+0x177/0x190 [16968.058295] _raw_spin_lock+0x21/0x30 [16968.058300] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16968.058586] ? do_raw_spin_unlock+0x75/0x190 [16968.058592] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16968.059999] ? do_wait_intr+0xf0/0xf0 [16968.060011] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16968.060217] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16968.061047] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16968.061265] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16968.075219] Modules linked in: lustre(O) [16968.100214] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16968.104944] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16968.105022] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16968.105030] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16968.105033] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16968.105180] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16968.105185] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16968.105190] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16968.105192] RDX: 00000000000019c8 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16968.105195] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16968.105197] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16968.105199] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16968.105203] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16968.105206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16968.105208] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16968.105217] Call Trace: [16968.105223] [16968.105229] ? show_regs.cold.9+0x22/0x2f [16968.105238] ? watchdog_timer_fn.cold.6+0x62/0x10a [16968.105245] ? __hrtimer_run_queues+0x1c2/0x4d0 [16968.105250] ? watchdog+0x40/0x40 [16968.105254] ? hrtimer_interrupt+0x128/0x2c0 [16968.105259] ? smp_apic_timer_interrupt+0xa6/0x270 [16968.105268] ? apic_timer_interrupt+0xf/0x20 [16968.105272] [16968.105275] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16968.105279] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16968.105284] do_raw_spin_lock+0x177/0x190 [16968.105288] _raw_spin_lock+0x21/0x30 [16968.105294] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16968.105636] obd_mod_free+0x46/0x140 [ptlrpc] [16968.105908] mdc_commit_open+0x91/0xb0 [mdc] [16968.105944] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16968.106214] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16968.106480] after_reply+0xe2c/0x1780 [ptlrpc] [16968.113701] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16968.116430] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16968.116809] ? __next_timer_interrupt+0x160/0x160 [16968.116819] ptlrpcd+0xa82/0xe40 [ptlrpc] [16968.117176] ? do_wait_intr+0xf0/0xf0 [16968.131708] mdc_intent_lock+0x353/0x7e0 [mdc] [16968.135294] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16968.141904] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16968.145890] kthread+0x1d1/0x200 [16968.150018] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16968.153849] ? set_kthread_struct+0x70/0x70 [16968.153867] ret_from_fork+0x1f/0x30 [16968.432708] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16968.436732] lmv_intent_open+0x338/0x1150 [lmv] [16968.438833] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16968.440891] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16968.442742] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16968.444520] ll_intent_lock+0xc5/0xaa0 [lustre] [16968.447732] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16968.449895] ll_lookup_it+0xe0e/0x2270 [lustre] [16968.451563] ? do_raw_spin_unlock+0x75/0x190 [16968.453253] ? _raw_spin_unlock+0x12/0x30 [16968.454816] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16968.456145] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16968.458969] ll_atomic_open+0x36a/0xd70 [lustre] [16968.460878] lookup_open+0x356/0xb40 [16968.461985] do_last+0x313/0xfc0 [16968.464410] ? nd_jump_root+0xe5/0x160 [16968.465897] ? path_init+0x437/0x520 [16968.467938] path_openat+0xf7/0x500 [16968.472301] ? do_raw_spin_unlock+0x75/0x190 [16968.475745] do_filp_open+0x99/0x140 [16968.480227] ? getname_flags+0x6e/0x330 [16968.483707] ? __check_object_size+0xff/0x256 [16968.487990] ? do_raw_spin_unlock+0x75/0x190 [16968.490743] ? _raw_spin_unlock+0x12/0x30 [16968.491987] do_sys_openat2+0x2b4/0x410 [16968.493743] do_sys_open+0x73/0xa0 [16968.497789] __x64_sys_openat+0x24/0x30 [16968.498949] do_syscall_64+0xc1/0x440 [16968.500081] entry_SYSCALL_64_after_hwframe+0x49/0xae [16968.503221] RIP: 0033:0x7f53425a5332 [16968.506314] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16968.515212] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16968.518950] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16968.521278] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16968.525078] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16968.527809] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16968.532258] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [16980.053218] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [16980.058097] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16980.058599] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [16980.099522] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16980.099537] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16980.099543] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [16980.100412] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16980.100426] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16980.100430] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16980.100435] RAX: 0000000000007dc9 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16980.100438] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [16980.100440] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [16980.100442] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [16980.100444] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [16980.100448] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [16980.100451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16980.100454] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [16980.100463] Call Trace: [16980.100469] [16980.100474] ? show_regs.cold.9+0x22/0x2f [16980.100485] ? watchdog_timer_fn.cold.6+0x62/0x10a [16980.100492] ? __hrtimer_run_queues+0x1c2/0x4d0 [16980.100496] ? watchdog+0x40/0x40 [16980.100501] ? hrtimer_interrupt+0x128/0x2c0 [16980.100505] ? smp_apic_timer_interrupt+0xa6/0x270 [16980.100512] ? apic_timer_interrupt+0xf/0x20 [16980.100517] [16980.100519] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16980.100524] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16980.100528] ? newidle_balance.isra.93+0x207/0x580 [16980.100534] do_raw_spin_lock+0x177/0x190 [16980.100539] _raw_spin_lock+0x21/0x30 [16980.100544] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [16980.100782] process_one_work+0x2c8/0x700 [16980.100790] worker_thread+0x296/0x6e0 [16980.100795] ? rescuer_thread+0x570/0x570 [16980.100798] kthread+0x1d1/0x200 [16980.100803] ? set_kthread_struct+0x70/0x70 [16980.100807] ret_from_fork+0x1f/0x30 [16980.273160] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16980.303933] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16980.311239] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16980.315368] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16980.319752] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16980.332322] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16980.340122] RAX: 0000000000007d84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16980.345534] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [16980.349279] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [16980.356206] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [16980.364361] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [16980.367822] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [16980.375117] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16980.378034] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [16980.383134] Call Trace: [16980.384323] [16980.385242] ? show_regs.cold.9+0x22/0x2f [16980.387233] ? watchdog_timer_fn.cold.6+0x62/0x10a [16980.389303] ? __hrtimer_run_queues+0x1c2/0x4d0 [16980.391758] ? watchdog+0x40/0x40 [16980.393328] ? hrtimer_interrupt+0x128/0x2c0 [16980.395725] ? smp_apic_timer_interrupt+0xa6/0x270 [16980.398263] ? apic_timer_interrupt+0xf/0x20 [16980.399834] [16980.400908] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16980.402985] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [16980.405358] do_raw_spin_lock+0x177/0x190 [16980.406970] _raw_spin_lock+0x21/0x30 [16980.408427] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [16980.412771] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [16980.416759] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [16980.420592] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [16980.423408] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [16980.426416] ? woken_wake_function+0x30/0x30 [16980.427692] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [16980.430132] kthread+0x1d1/0x200 [16980.431552] ? set_kthread_struct+0x70/0x70 [16980.433222] ret_from_fork+0x1f/0x30 [16983.528373] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14346s! [16983.548579] Showing busy workqueues and worker pools: [16983.560516] workqueue events: flags=0x0 [16983.563924] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [16983.575481] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [16983.584438] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16983.594524] pending: cpu_latency_work [ptlrpc] [16983.598328] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [16983.604796] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [16983.614363] workqueue events_long: flags=0x0 [16983.618630] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [16983.629525] pending: lu_objects_flush [obdclass] [16983.634313] workqueue events_unbound: flags=0x2 [16983.638592] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [16983.648392] pending: flush_memcg_stats_dwork, idle_cull_fn [16983.655701] workqueue events_power_efficient: flags=0x80 [16983.667227] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16983.671620] pending: check_lifetime, do_cache_clean [16983.678389] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16983.682673] pending: neigh_periodic_work, neigh_periodic_work [16983.687504] workqueue mm_percpu_wq: flags=0x8 [16983.691279] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16983.696134] pending: vmstat_update, lru_add_drain_per_cpu [16983.698762] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [16983.705338] pending: vmstat_update, lru_add_drain_per_cpu [16983.710792] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [16983.719207] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [16983.730132] workqueue xprtiod: flags=0xa [16983.736024] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [16983.743757] pending: xprt_autoclose, xs_error_handle [16983.746182] workqueue ipv6_addrconf: flags=0x40008 [16983.748179] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [16983.753505] pending: addrconf_verify_work [16983.757565] workqueue knbd0-recv: flags=0x1a [16983.762505] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [16983.765637] in-flight: 651:recv_work [16983.771430] workqueue ptlrpc_pinger: flags=0x2 [16983.773893] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [16983.780222] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [16983.784180] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14346s workers=4 idle: 15979 31824 8 [16983.791798] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=16974s workers=2 idle: 654 [16996.055306] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [16996.060903] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [16996.064203] Modules linked in: [16996.070587] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16996.070660] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16996.070668] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16996.070670] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [16996.070685] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [16996.070690] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [16996.070695] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [16996.070698] RDX: 0000000000007a6f RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [16996.070700] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [16996.070702] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [16996.070705] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [16996.070708] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [16996.070711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16996.070713] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [16996.070722] Call Trace: [16996.070727] [16996.070732] ? show_regs.cold.9+0x22/0x2f [16996.070742] ? watchdog_timer_fn.cold.6+0x62/0x10a [16996.070748] ? __hrtimer_run_queues+0x1c2/0x4d0 [16996.070753] ? watchdog+0x40/0x40 [16996.070756] ? hrtimer_interrupt+0x128/0x2c0 [16996.070760] ? smp_apic_timer_interrupt+0xa6/0x270 [16996.070767] ? apic_timer_interrupt+0xf/0x20 [16996.070772] [16996.070775] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [16996.070780] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [16996.070784] do_raw_spin_lock+0x177/0x190 [16996.070790] _raw_spin_lock+0x21/0x30 [16996.070796] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [16996.073082] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [16996.073158] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [16996.073165] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [16996.073168] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16996.073242] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [16996.073246] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [16996.073251] RAX: 0000000000003083 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [16996.073254] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [16996.073257] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [16996.092841] obd_mod_free+0x46/0x140 [ptlrpc] [16996.093126] mdc_commit_open+0x91/0xb0 [mdc] [16996.093180] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [16996.093371] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [16996.093569] after_reply+0xe2c/0x1780 [ptlrpc] [16996.093765] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [16996.094128] ? __next_timer_interrupt+0x160/0x160 [16996.094138] ptlrpcd+0xa82/0xe40 [ptlrpc] [16996.094764] ? do_wait_intr+0xf0/0xf0 [16996.094774] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [16996.095065] kthread+0x1d1/0x200 [16996.095071] ? set_kthread_struct+0x70/0x70 [16996.095076] ret_from_fork+0x1f/0x30 [16996.411692] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [16996.424701] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [16996.430178] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [16996.435398] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16996.437781] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [16996.441837] Call Trace: [16996.443186] [16996.445255] ? show_regs.cold.9+0x22/0x2f [16996.447352] ? watchdog_timer_fn.cold.6+0x62/0x10a [16996.449895] ? __hrtimer_run_queues+0x1c2/0x4d0 [16996.453720] ? watchdog+0x40/0x40 [16996.455906] ? hrtimer_interrupt+0x128/0x2c0 [16996.459137] ? smp_apic_timer_interrupt+0xa6/0x270 [16996.467694] ? apic_timer_interrupt+0xf/0x20 [16996.472495] [16996.476799] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [16996.484570] do_raw_spin_lock+0x177/0x190 [16996.488381] _raw_spin_lock+0x21/0x30 [16996.490661] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [16996.497273] ? do_raw_spin_unlock+0x75/0x190 [16996.501331] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [16996.503908] ? do_wait_intr+0xf0/0xf0 [16996.506891] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [16996.509690] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [16996.514162] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16996.517492] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16996.521253] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16996.525397] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [16996.533269] mdc_intent_lock+0x353/0x7e0 [mdc] [16996.540167] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16996.546758] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [16996.552320] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [16996.558191] lmv_intent_open+0x338/0x1150 [lmv] [16996.562299] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16996.564399] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16996.570661] lmv_intent_lock+0x1dd/0x4c0 [lmv] [16996.576698] ll_intent_lock+0xc5/0xaa0 [lustre] [16996.579608] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [16996.587317] ll_lookup_it+0xe0e/0x2270 [lustre] [16996.591783] ? do_raw_spin_unlock+0x75/0x190 [16996.595327] ? _raw_spin_unlock+0x12/0x30 [16996.598153] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [16996.603244] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [16996.609281] ll_atomic_open+0x36a/0xd70 [lustre] [16996.613967] lookup_open+0x356/0xb40 [16996.616065] do_last+0x313/0xfc0 [16996.620689] ? nd_jump_root+0xe5/0x160 [16996.623264] ? path_init+0x437/0x520 [16996.625565] path_openat+0xf7/0x500 [16996.629747] ? do_raw_spin_unlock+0x75/0x190 [16996.634318] do_filp_open+0x99/0x140 [16996.640154] ? getname_flags+0x6e/0x330 [16996.644648] ? __check_object_size+0xff/0x256 [16996.652989] ? do_raw_spin_unlock+0x75/0x190 [16996.662343] ? _raw_spin_unlock+0x12/0x30 [16996.667618] do_sys_openat2+0x2b4/0x410 [16996.668967] do_sys_open+0x73/0xa0 [16996.679337] __x64_sys_openat+0x24/0x30 [16996.684767] do_syscall_64+0xc1/0x440 [16996.691790] entry_SYSCALL_64_after_hwframe+0x49/0xae [16996.693667] RIP: 0033:0x7f53425a5332 [16996.699609] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [16996.723798] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [16996.727242] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [16996.739056] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [16996.745497] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [16996.754418] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [16996.759118] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17008.052021] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [17008.057387] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17008.058014] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [17008.084527] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17008.084543] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17008.084550] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17008.087693] Modules linked in: [17008.099016] [17008.099026] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17008.099044] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17008.099048] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17008.099054] RAX: 0000000000007215 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17008.099056] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17008.099059] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17008.099061] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17008.099064] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17008.099067] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17008.099070] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17008.099073] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17008.099084] Call Trace: [17008.102903] lustre(O) [17008.106319] [17008.108820] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17008.109109] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17008.109116] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17008.109124] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17008.109137] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17008.109142] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17008.109146] RAX: 000000000000566d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17008.109148] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17008.109150] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17008.109175] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17008.109178] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17008.109269] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17008.109307] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17008.109309] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17008.109320] Call Trace: [17008.109326] [17008.109331] ? show_regs.cold.9+0x22/0x2f [17008.109341] ? watchdog_timer_fn.cold.6+0x62/0x10a [17008.109347] ? __hrtimer_run_queues+0x1c2/0x4d0 [17008.109351] ? watchdog+0x40/0x40 [17008.109355] ? hrtimer_interrupt+0x128/0x2c0 [17008.109359] ? smp_apic_timer_interrupt+0xa6/0x270 [17008.109367] ? apic_timer_interrupt+0xf/0x20 [17008.109372] [17008.109375] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17008.109380] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17008.109387] do_raw_spin_lock+0x177/0x190 [17008.109392] _raw_spin_lock+0x21/0x30 [17008.109396] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17008.109693] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17008.109900] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17008.110141] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17008.110460] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17008.110692] ? woken_wake_function+0x30/0x30 [17008.110701] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17008.110978] kthread+0x1d1/0x200 [17008.110984] ? set_kthread_struct+0x70/0x70 [17008.110989] ret_from_fork+0x1f/0x30 [17008.267769] ? show_regs.cold.9+0x22/0x2f [17008.269520] ? watchdog_timer_fn.cold.6+0x62/0x10a [17008.271379] ? __hrtimer_run_queues+0x1c2/0x4d0 [17008.273341] ? watchdog+0x40/0x40 [17008.274835] ? hrtimer_interrupt+0x128/0x2c0 [17008.276675] ? smp_apic_timer_interrupt+0xa6/0x270 [17008.278816] ? apic_timer_interrupt+0xf/0x20 [17008.280104] [17008.281054] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17008.283059] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17008.285223] ? newidle_balance.isra.93+0x207/0x580 [17008.287075] do_raw_spin_lock+0x177/0x190 [17008.289162] _raw_spin_lock+0x21/0x30 [17008.290662] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17008.293009] process_one_work+0x2c8/0x700 [17008.294440] worker_thread+0x296/0x6e0 [17008.296240] ? rescuer_thread+0x570/0x570 [17008.297852] kthread+0x1d1/0x200 [17008.299220] ? set_kthread_struct+0x70/0x70 [17008.301140] ret_from_fork+0x1f/0x30 [17014.240080] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14377s! [17014.245601] Showing busy workqueues and worker pools: [17014.248132] workqueue events: flags=0x0 [17014.250121] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17014.253127] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17014.258209] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17014.261774] pending: cpu_latency_work [ptlrpc] [17014.264308] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17014.267697] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17014.272384] workqueue events_long: flags=0x0 [17014.273701] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17014.276254] pending: lu_objects_flush [obdclass] [17014.279269] workqueue events_unbound: flags=0x2 [17014.281742] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17014.285125] pending: flush_memcg_stats_dwork, idle_cull_fn [17014.288149] workqueue events_power_efficient: flags=0x80 [17014.292058] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17014.296324] pending: check_lifetime, do_cache_clean [17014.299142] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17014.302520] pending: neigh_periodic_work, neigh_periodic_work [17014.305535] workqueue mm_percpu_wq: flags=0x8 [17014.307452] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17014.311200] pending: vmstat_update, lru_add_drain_per_cpu [17014.314368] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17014.318114] pending: vmstat_update, lru_add_drain_per_cpu [17014.321009] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17014.324203] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17014.327568] workqueue xprtiod: flags=0xa [17014.329167] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17014.331073] pending: xprt_autoclose, xs_error_handle [17014.333308] workqueue ipv6_addrconf: flags=0x40008 [17014.335274] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17014.338607] pending: addrconf_verify_work [17014.340859] workqueue knbd0-recv: flags=0x1a [17014.342075] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17014.345499] in-flight: 651:recv_work [17014.347379] workqueue ptlrpc_pinger: flags=0x2 [17014.349351] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17014.352269] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17014.355209] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14377s workers=4 idle: 15979 31824 8 [17014.358393] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17004s workers=2 idle: 654 [17024.055083] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [17024.057019] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [17024.058137] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 [17024.061172] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17024.061266] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17024.061274] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17024.061281] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17024.061300] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17024.061308] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17024.061313] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17024.061316] RDX: 00000000000049b9 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17024.061318] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17024.061320] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17024.061323] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17024.061326] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17024.061329] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17024.076073] dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17024.076114] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17024.076122] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17024.076125] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17024.076139] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17024.076144] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17024.076149] RAX: 0000000000001a95 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17024.076151] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17024.076153] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17024.076155] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17024.076158] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17024.076161] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17024.076164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17024.076166] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17024.076175] Call Trace: [17024.076181] [17024.076187] ? show_regs.cold.9+0x22/0x2f [17024.076197] ? watchdog_timer_fn.cold.6+0x62/0x10a [17024.076204] ? __hrtimer_run_queues+0x1c2/0x4d0 [17024.076208] ? watchdog+0x40/0x40 [17024.076212] ? hrtimer_interrupt+0x128/0x2c0 [17024.076216] ? smp_apic_timer_interrupt+0xa6/0x270 [17024.076224] ? apic_timer_interrupt+0xf/0x20 [17024.076229] [17024.076231] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17024.076239] do_raw_spin_lock+0x177/0x190 [17024.076244] _raw_spin_lock+0x21/0x30 [17024.076248] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17024.110054] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17024.110070] Call Trace: [17024.110078] [17024.110085] ? show_regs.cold.9+0x22/0x2f [17024.110106] ? watchdog_timer_fn.cold.6+0x62/0x10a [17024.110113] ? __hrtimer_run_queues+0x1c2/0x4d0 [17024.110118] ? watchdog+0x40/0x40 [17024.116976] ? do_raw_spin_unlock+0x75/0x190 [17024.116995] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17024.132132] ? hrtimer_interrupt+0x128/0x2c0 [17024.132454] ? smp_apic_timer_interrupt+0xa6/0x270 [17024.132464] ? apic_timer_interrupt+0xf/0x20 [17024.132469] [17024.132474] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17024.132508] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17024.135847] ? do_wait_intr+0xf0/0xf0 [17024.135871] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17024.159895] do_raw_spin_lock+0x177/0x190 [17024.159919] _raw_spin_lock+0x21/0x30 [17024.159929] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17024.163883] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17024.164250] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17024.164618] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17024.164731] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17024.164768] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17024.164798] mdc_intent_lock+0x353/0x7e0 [mdc] [17024.171515] obd_mod_free+0x46/0x140 [ptlrpc] [17024.174509] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17024.174989] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17024.183584] mdc_commit_open+0x91/0xb0 [mdc] [17024.183639] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17024.183907] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17024.184210] after_reply+0xe2c/0x1780 [ptlrpc] [17024.184559] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17024.184773] ? __next_timer_interrupt+0x160/0x160 [17024.184784] ptlrpcd+0xa82/0xe40 [ptlrpc] [17024.186045] ? do_wait_intr+0xf0/0xf0 [17024.186057] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17024.186427] kthread+0x1d1/0x200 [17024.186434] ? set_kthread_struct+0x70/0x70 [17024.186438] ret_from_fork+0x1f/0x30 [17024.457590] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17024.460269] lmv_intent_open+0x338/0x1150 [lmv] [17024.462267] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17024.467239] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17024.470580] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17024.473460] ll_intent_lock+0xc5/0xaa0 [lustre] [17024.474940] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17024.476291] ll_lookup_it+0xe0e/0x2270 [lustre] [17024.477554] ? do_raw_spin_unlock+0x75/0x190 [17024.478701] ? _raw_spin_unlock+0x12/0x30 [17024.480035] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17024.488605] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17024.492477] ll_atomic_open+0x36a/0xd70 [lustre] [17024.495688] lookup_open+0x356/0xb40 [17024.497153] do_last+0x313/0xfc0 [17024.499464] ? nd_jump_root+0xe5/0x160 [17024.500954] ? path_init+0x437/0x520 [17024.502192] path_openat+0xf7/0x500 [17024.503176] ? do_raw_spin_unlock+0x75/0x190 [17024.505357] do_filp_open+0x99/0x140 [17024.508027] ? getname_flags+0x6e/0x330 [17024.509571] ? __check_object_size+0xff/0x256 [17024.511179] ? do_raw_spin_unlock+0x75/0x190 [17024.512809] ? _raw_spin_unlock+0x12/0x30 [17024.514002] do_sys_openat2+0x2b4/0x410 [17024.516515] do_sys_open+0x73/0xa0 [17024.518006] __x64_sys_openat+0x24/0x30 [17024.519608] do_syscall_64+0xc1/0x440 [17024.521098] entry_SYSCALL_64_after_hwframe+0x49/0xae [17024.523128] RIP: 0033:0x7f53425a5332 [17024.525184] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17024.541040] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17024.546401] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17024.549059] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17024.553509] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17024.557550] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17024.562895] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17036.058038] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [17036.065575] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17036.087953] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17036.092081] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17036.095864] RIP: 0010:native_safe_halt+0x2e/0x50 [17036.097646] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17036.105471] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17036.109379] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17036.111710] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17036.113572] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17036.115350] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17036.119903] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17036.122505] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17036.126957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17036.131601] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17036.136618] Call Trace: [17036.137433] [17036.138218] ? show_regs.cold.9+0x22/0x2f [17036.140088] ? watchdog_timer_fn.cold.6+0x62/0x10a [17036.141782] ? __hrtimer_run_queues+0x1c2/0x4d0 [17036.143725] ? watchdog+0x40/0x40 [17036.144810] ? hrtimer_interrupt+0x128/0x2c0 [17036.145809] ? smp_apic_timer_interrupt+0xa6/0x270 [17036.147407] ? apic_timer_interrupt+0xf/0x20 [17036.148495] [17036.149066] ? native_safe_halt+0x2e/0x50 [17036.150188] kvm_wait+0xcd/0xe0 [17036.150956] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17036.152605] do_raw_spin_lock+0x177/0x190 [17036.153730] _raw_spin_lock+0x21/0x30 [17036.156622] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17036.158685] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17036.160396] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17036.162330] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17036.164172] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17036.166420] ? woken_wake_function+0x30/0x30 [17036.167689] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17036.170077] kthread+0x1d1/0x200 [17036.171461] ? set_kthread_struct+0x70/0x70 [17036.172870] ret_from_fork+0x1f/0x30 [17040.052280] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [17040.065251] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17040.140442] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17040.165950] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17040.186277] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17040.203497] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17040.209789] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17040.238629] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17040.247338] RAX: 000000000000358b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17040.262360] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17040.275439] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17040.286137] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17040.288730] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17040.301491] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17040.312264] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17040.315384] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17040.319338] Call Trace: [17040.320490] [17040.324861] ? show_regs.cold.9+0x22/0x2f [17040.330623] ? watchdog_timer_fn.cold.6+0x62/0x10a [17040.332333] ? __hrtimer_run_queues+0x1c2/0x4d0 [17040.334520] ? watchdog+0x40/0x40 [17040.336363] ? hrtimer_interrupt+0x128/0x2c0 [17040.338324] ? smp_apic_timer_interrupt+0xa6/0x270 [17040.342366] ? apic_timer_interrupt+0xf/0x20 [17040.345097] [17040.345649] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17040.347028] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17040.354109] ? newidle_balance.isra.93+0x207/0x580 [17040.355342] do_raw_spin_lock+0x177/0x190 [17040.359498] _raw_spin_lock+0x21/0x30 [17040.363017] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17040.365542] process_one_work+0x2c8/0x700 [17040.366903] worker_thread+0x296/0x6e0 [17040.368614] ? rescuer_thread+0x570/0x570 [17040.370673] kthread+0x1d1/0x200 [17040.371873] ? set_kthread_struct+0x70/0x70 [17040.378089] ret_from_fork+0x1f/0x30 [17044.449108] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14407s! [17044.453811] Showing busy workqueues and worker pools: [17044.456652] workqueue events: flags=0x0 [17044.461758] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17044.464212] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17044.479231] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17044.496180] pending: cpu_latency_work [ptlrpc] [17044.498224] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17044.505299] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17044.510639] workqueue events_long: flags=0x0 [17044.527042] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17044.531652] pending: lu_objects_flush [obdclass] [17044.537460] workqueue events_unbound: flags=0x2 [17044.542313] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17044.547092] pending: flush_memcg_stats_dwork, idle_cull_fn [17044.556202] workqueue events_power_efficient: flags=0x80 [17044.565499] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17044.571330] pending: check_lifetime, do_cache_clean [17044.573020] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17044.588566] pending: neigh_periodic_work, neigh_periodic_work [17044.594390] workqueue mm_percpu_wq: flags=0x8 [17044.601365] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17044.603623] pending: vmstat_update, lru_add_drain_per_cpu [17044.625391] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17044.631122] pending: vmstat_update, lru_add_drain_per_cpu [17044.637443] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17044.646445] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17044.652826] workqueue xprtiod: flags=0xa [17044.654406] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17044.671724] pending: xprt_autoclose, xs_error_handle [17044.683433] workqueue ipv6_addrconf: flags=0x40008 [17044.685167] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17044.705822] pending: addrconf_verify_work [17044.707316] workqueue knbd0-recv: flags=0x1a [17044.716634] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17044.725193] in-flight: 651:recv_work [17044.730879] workqueue ptlrpc_pinger: flags=0x2 [17044.741812] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17044.755781] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17044.770960] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14407s workers=4 idle: 15979 31824 8 [17044.789427] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17035s workers=2 idle: 654 [17052.055049] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [17052.058709] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17052.060949] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [17052.088565] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17052.088581] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17052.088584] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17052.088598] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17052.088603] RSP: 0018:ffffbb5381503500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [17052.088609] RAX: 000000000000770a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17052.088611] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17052.088614] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17052.088616] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17052.088619] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17052.088622] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17052.088625] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17052.088628] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17052.088637] Call Trace: [17052.088643] [17052.088649] ? show_regs.cold.9+0x22/0x2f [17052.097253] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash [17052.104343] ? watchdog_timer_fn.cold.6+0x62/0x10a [17052.104362] ? __hrtimer_run_queues+0x1c2/0x4d0 [17052.104367] ? watchdog+0x40/0x40 [17052.104372] ? hrtimer_interrupt+0x128/0x2c0 [17052.104376] ? smp_apic_timer_interrupt+0xa6/0x270 [17052.104385] ? apic_timer_interrupt+0xf/0x20 [17052.104390] [17052.104395] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17052.104406] do_raw_spin_lock+0x177/0x190 [17052.104412] _raw_spin_lock+0x21/0x30 [17052.104483] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17052.104751] ? do_raw_spin_unlock+0x75/0x190 [17052.104758] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17052.113410] dm_log dm_mod sha512_ssse3 sha512_generic [17052.113558] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17052.113566] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17052.113570] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17052.113584] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17052.113589] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17052.113594] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17052.113597] RDX: 000000000000098b RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17052.113600] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17052.113602] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17052.113605] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17052.113609] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17052.113613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17052.113615] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17052.113624] Call Trace: [17052.113630] [17052.113635] ? show_regs.cold.9+0x22/0x2f [17052.113645] ? watchdog_timer_fn.cold.6+0x62/0x10a [17052.113652] ? __hrtimer_run_queues+0x1c2/0x4d0 [17052.113657] ? watchdog+0x40/0x40 [17052.113661] ? hrtimer_interrupt+0x128/0x2c0 [17052.113665] ? smp_apic_timer_interrupt+0xa6/0x270 [17052.113674] ? apic_timer_interrupt+0xf/0x20 [17052.113678] [17052.113681] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17052.113686] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17052.113690] do_raw_spin_lock+0x177/0x190 [17052.113695] _raw_spin_lock+0x21/0x30 [17052.113703] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17052.114975] obd_mod_free+0x46/0x140 [ptlrpc] [17052.115193] mdc_commit_open+0x91/0xb0 [mdc] [17052.115919] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17052.116138] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17052.116363] after_reply+0xe2c/0x1780 [ptlrpc] [17052.117972] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17052.118191] ? __next_timer_interrupt+0x160/0x160 [17052.118201] ptlrpcd+0xa82/0xe40 [ptlrpc] [17052.121299] ? do_wait_intr+0xf0/0xf0 [17052.121320] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17052.150407] ? do_wait_intr+0xf0/0xf0 [17052.161787] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17052.165800] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17052.170307] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17052.170950] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17052.171064] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17052.171105] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17052.171137] mdc_intent_lock+0x353/0x7e0 [mdc] [17052.171166] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17052.171251] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17052.171945] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17052.171979] lmv_intent_open+0x338/0x1150 [lmv] [17052.172014] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17052.172107] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17052.172193] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17052.172217] ll_intent_lock+0xc5/0xaa0 [lustre] [17052.172304] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17052.172389] ll_lookup_it+0xe0e/0x2270 [lustre] [17052.172940] ? do_raw_spin_unlock+0x75/0x190 [17052.172948] ? _raw_spin_unlock+0x12/0x30 [17052.172953] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17052.172977] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17052.172997] ll_atomic_open+0x36a/0xd70 [lustre] [17052.173081] lookup_open+0x356/0xb40 [17052.173092] do_last+0x313/0xfc0 [17052.173099] ? nd_jump_root+0xe5/0x160 [17052.173103] ? path_init+0x437/0x520 [17052.173107] path_openat+0xf7/0x500 [17052.173112] ? do_raw_spin_unlock+0x75/0x190 [17052.173116] do_filp_open+0x99/0x140 [17052.173122] ? getname_flags+0x6e/0x330 [17052.173127] ? __check_object_size+0xff/0x256 [17052.173143] ? do_raw_spin_unlock+0x75/0x190 [17052.173148] ? _raw_spin_unlock+0x12/0x30 [17052.173152] do_sys_openat2+0x2b4/0x410 [17052.173157] do_sys_open+0x73/0xa0 [17052.173161] __x64_sys_openat+0x24/0x30 [17052.173165] do_syscall_64+0xc1/0x440 [17052.173171] entry_SYSCALL_64_after_hwframe+0x49/0xae [17052.173177] RIP: 0033:0x7f53425a5332 [17052.173228] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17052.173233] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17052.173238] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17052.173241] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17052.173243] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17052.173245] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17052.173247] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17052.894955] kthread+0x1d1/0x200 [17052.899435] ? set_kthread_struct+0x70/0x70 [17052.902404] ret_from_fork+0x1f/0x30 [17064.058036] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [17064.061716] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17064.086277] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17064.100999] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17064.112547] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17064.119880] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17064.145691] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17064.153831] RAX: 0000000000006184 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17064.160146] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17064.169794] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17064.174178] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17064.183126] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17064.185648] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17064.189717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17064.194026] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17064.203521] Call Trace: [17064.207439] [17064.208148] ? show_regs.cold.9+0x22/0x2f [17064.213070] ? watchdog_timer_fn.cold.6+0x62/0x10a [17064.220945] ? __hrtimer_run_queues+0x1c2/0x4d0 [17064.232855] ? watchdog+0x40/0x40 [17064.236549] ? hrtimer_interrupt+0x128/0x2c0 [17064.245902] ? smp_apic_timer_interrupt+0xa6/0x270 [17064.247471] ? apic_timer_interrupt+0xf/0x20 [17064.257464] [17064.259328] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17064.271436] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17064.275532] do_raw_spin_lock+0x177/0x190 [17064.279868] _raw_spin_lock+0x21/0x30 [17064.281151] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17064.288118] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17064.292831] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17064.296159] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17064.301611] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17064.308031] ? woken_wake_function+0x30/0x30 [17064.311038] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17064.318084] kthread+0x1d1/0x200 [17064.321489] ? set_kthread_struct+0x70/0x70 [17064.327384] ret_from_fork+0x1f/0x30 [17072.054366] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [17072.059077] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17072.100293] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17072.106061] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17072.111706] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17072.114102] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17072.121413] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17072.128672] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17072.130905] RAX: 0000000000007f89 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17072.133512] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17072.136706] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17072.139866] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17072.142761] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17072.146103] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17072.149406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17072.153311] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17072.157898] Call Trace: [17072.160036] [17072.161108] ? show_regs.cold.9+0x22/0x2f [17072.162461] ? watchdog_timer_fn.cold.6+0x62/0x10a [17072.168827] ? __hrtimer_run_queues+0x1c2/0x4d0 [17072.172065] ? watchdog+0x40/0x40 [17072.174001] ? hrtimer_interrupt+0x128/0x2c0 [17072.177383] ? smp_apic_timer_interrupt+0xa6/0x270 [17072.184420] ? apic_timer_interrupt+0xf/0x20 [17072.188977] [17072.195295] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17072.197166] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17072.205168] ? newidle_balance.isra.93+0x207/0x580 [17072.208096] do_raw_spin_lock+0x177/0x190 [17072.210954] _raw_spin_lock+0x21/0x30 [17072.213083] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17072.215846] process_one_work+0x2c8/0x700 [17072.217678] worker_thread+0x296/0x6e0 [17072.224457] ? rescuer_thread+0x570/0x570 [17072.229955] kthread+0x1d1/0x200 [17072.233482] ? set_kthread_struct+0x70/0x70 [17072.235631] ret_from_fork+0x1f/0x30 [17075.167847] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14438s! [17075.176021] Showing busy workqueues and worker pools: [17075.178063] workqueue events: flags=0x0 [17075.183559] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17075.187967] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17075.204126] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17075.210813] pending: cpu_latency_work [ptlrpc] [17075.215522] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17075.220152] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17075.227165] workqueue events_long: flags=0x0 [17075.229681] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17075.233569] pending: lu_objects_flush [obdclass] [17075.236551] workqueue events_unbound: flags=0x2 [17075.238984] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17075.242003] pending: flush_memcg_stats_dwork, idle_cull_fn [17075.247172] workqueue events_power_efficient: flags=0x80 [17075.250154] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17075.253965] pending: check_lifetime, do_cache_clean [17075.257661] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17075.264155] pending: neigh_periodic_work, neigh_periodic_work [17075.270223] workqueue mm_percpu_wq: flags=0x8 [17075.275661] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17075.277677] pending: vmstat_update, lru_add_drain_per_cpu [17075.281785] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17075.291158] pending: vmstat_update, lru_add_drain_per_cpu [17075.294721] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17075.300937] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17075.304122] workqueue xprtiod: flags=0xa [17075.310840] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17075.320100] pending: xprt_autoclose, xs_error_handle [17075.324669] workqueue ipv6_addrconf: flags=0x40008 [17075.328166] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17075.334945] pending: addrconf_verify_work [17075.338988] workqueue knbd0-recv: flags=0x1a [17075.343959] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17075.350672] in-flight: 651:recv_work [17075.353224] workqueue ptlrpc_pinger: flags=0x2 [17075.355881] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17075.359799] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17075.361937] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14438s workers=4 idle: 15979 31824 8 [17075.369935] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17065s workers=2 idle: 654 [17080.059081] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [17080.059243] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [17080.067542] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17080.067622] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17080.067630] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17080.067632] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17080.067648] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17080.067652] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17080.067656] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17080.067659] RDX: 0000000000004603 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17080.067662] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17080.067664] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17080.067667] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17080.067670] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17080.067672] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17080.067674] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17080.067682] Call Trace: [17080.068440] [17080.075526] Modules linked in: [17080.099987] ? show_regs.cold.9+0x22/0x2f [17080.100015] ? watchdog_timer_fn.cold.6+0x62/0x10a [17080.100023] ? __hrtimer_run_queues+0x1c2/0x4d0 [17080.100028] ? watchdog+0x40/0x40 [17080.100033] ? hrtimer_interrupt+0x128/0x2c0 [17080.100037] ? smp_apic_timer_interrupt+0xa6/0x270 [17080.100046] ? apic_timer_interrupt+0xf/0x20 [17080.100051] [17080.100056] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17080.100062] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17080.100073] do_raw_spin_lock+0x177/0x190 [17080.100080] _raw_spin_lock+0x21/0x30 [17080.100087] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17080.121261] lustre(O) [17080.130847] obd_mod_free+0x46/0x140 [ptlrpc] [17080.132788] mdc_commit_open+0x91/0xb0 [mdc] [17080.132816] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17080.134233] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17080.134525] after_reply+0xe2c/0x1780 [ptlrpc] [17080.134721] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17080.145045] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17080.145120] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17080.145126] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17080.145128] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17080.146069] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17080.146074] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17080.146079] RAX: 000000000000380a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17080.146082] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17080.146084] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17080.146087] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17080.146089] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17080.146092] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17080.146094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17080.146096] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17080.146104] Call Trace: [17080.146110] [17080.146116] ? show_regs.cold.9+0x22/0x2f [17080.146126] ? watchdog_timer_fn.cold.6+0x62/0x10a [17080.146132] ? __hrtimer_run_queues+0x1c2/0x4d0 [17080.146136] ? watchdog+0x40/0x40 [17080.146140] ? hrtimer_interrupt+0x128/0x2c0 [17080.146144] ? smp_apic_timer_interrupt+0xa6/0x270 [17080.146160] ? apic_timer_interrupt+0xf/0x20 [17080.146165] [17080.146167] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17080.146174] do_raw_spin_lock+0x177/0x190 [17080.146179] _raw_spin_lock+0x21/0x30 [17080.146184] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17080.147547] ? do_raw_spin_unlock+0x75/0x190 [17080.147557] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17080.148458] ? do_wait_intr+0xf0/0xf0 [17080.148468] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17080.148672] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17080.150266] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17080.151588] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17080.151709] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17080.152031] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17080.152181] mdc_intent_lock+0x353/0x7e0 [mdc] [17080.152211] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17080.160685] ? __next_timer_interrupt+0x160/0x160 [17080.160708] ptlrpcd+0xa82/0xe40 [ptlrpc] [17080.161100] ? do_wait_intr+0xf0/0xf0 [17080.161110] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17080.161324] kthread+0x1d1/0x200 [17080.161333] ? set_kthread_struct+0x70/0x70 [17080.161337] ret_from_fork+0x1f/0x30 [17080.540335] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17080.544482] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17080.549987] lmv_intent_open+0x338/0x1150 [lmv] [17080.556299] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17080.564700] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17080.568809] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17080.570215] ll_intent_lock+0xc5/0xaa0 [lustre] [17080.572260] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17080.573862] ll_lookup_it+0xe0e/0x2270 [lustre] [17080.575429] ? do_raw_spin_unlock+0x75/0x190 [17080.576945] ? _raw_spin_unlock+0x12/0x30 [17080.578383] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17080.580189] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17080.582753] ll_atomic_open+0x36a/0xd70 [lustre] [17080.587993] lookup_open+0x356/0xb40 [17080.591870] do_last+0x313/0xfc0 [17080.593500] ? nd_jump_root+0xe5/0x160 [17080.598683] ? path_init+0x437/0x520 [17080.602941] path_openat+0xf7/0x500 [17080.604213] ? do_raw_spin_unlock+0x75/0x190 [17080.610258] do_filp_open+0x99/0x140 [17080.611225] ? getname_flags+0x6e/0x330 [17080.614547] ? __check_object_size+0xff/0x256 [17080.616526] ? do_raw_spin_unlock+0x75/0x190 [17080.620446] ? _raw_spin_unlock+0x12/0x30 [17080.626177] do_sys_openat2+0x2b4/0x410 [17080.633100] do_sys_open+0x73/0xa0 [17080.636021] __x64_sys_openat+0x24/0x30 [17080.638414] do_syscall_64+0xc1/0x440 [17080.639581] entry_SYSCALL_64_after_hwframe+0x49/0xae [17080.640899] RIP: 0033:0x7f53425a5332 [17080.642992] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17080.650226] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17080.655958] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17080.659791] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17080.661599] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17080.666142] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17080.673167] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17092.059696] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [17092.065819] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17092.105987] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17092.115322] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17092.124830] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17092.132031] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17092.147099] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [17092.159438] RAX: 0000000000007c7c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17092.165724] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17092.170278] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17092.173916] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17092.176637] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17092.182373] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17092.190041] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17092.193999] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17092.196839] Call Trace: [17092.197961] [17092.201977] ? show_regs.cold.9+0x22/0x2f [17092.203653] ? watchdog_timer_fn.cold.6+0x62/0x10a [17092.209069] ? __hrtimer_run_queues+0x1c2/0x4d0 [17092.214003] ? watchdog+0x40/0x40 [17092.215532] ? hrtimer_interrupt+0x128/0x2c0 [17092.219341] ? smp_apic_timer_interrupt+0xa6/0x270 [17092.220828] ? apic_timer_interrupt+0xf/0x20 [17092.223580] [17092.224760] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17092.226511] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17092.235868] do_raw_spin_lock+0x177/0x190 [17092.238067] _raw_spin_lock+0x21/0x30 [17092.241617] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17092.245899] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17092.255479] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17092.258780] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17092.262351] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17092.265047] ? woken_wake_function+0x30/0x30 [17092.266919] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17092.269833] kthread+0x1d1/0x200 [17092.271122] ? set_kthread_struct+0x70/0x70 [17092.272983] ret_from_fork+0x1f/0x30 [17094.796121] rcu: INFO: rcu_sched self-detected stall on CPU [17094.798545] rcu: 2-....: (8375962 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [17094.804571] (t=15000370 jiffies g=274185 q=67844) [17094.807571] rcu: rcu_sched kthread starved for 14458736 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [17094.813600] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [17094.818625] rcu: RCU grace-period kthread stack dump: [17094.821768] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [17094.828394] Call Trace: [17094.829858] __schedule+0x351/0xcb0 [17094.831549] schedule+0xc0/0x180 [17094.833061] schedule_timeout+0xb4/0x190 [17094.835625] ? do_raw_spin_unlock+0x75/0x190 [17094.837816] ? __next_timer_interrupt+0x160/0x160 [17094.842825] ? prepare_to_swait_event+0xf3/0x140 [17094.845208] rcu_gp_kthread+0x8dd/0x15b0 [17094.846913] ? force_qs_rnp+0x2c0/0x2c0 [17094.848817] kthread+0x1d1/0x200 [17094.850278] ? set_kthread_struct+0x70/0x70 [17094.853450] ret_from_fork+0x1f/0x30 [17094.854889] rcu: Stack dump where RCU GP kthread last ran: [17094.857303] NMI backtrace for cpu 2 [17094.858605] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17094.862532] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17094.866059] Call Trace: [17094.866986] [17094.867967] ? dump_stack+0xbb/0x10e [17094.869344] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [17094.871381] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [17094.872868] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [17094.874574] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [17094.876540] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [17094.877975] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [17094.879796] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [17094.881535] ? update_process_times+0x6d/0xc0 [17094.882857] ? tick_sched_handle.isra.18+0x27/0x90 [17094.884500] ? tick_sched_timer+0x53/0xb0 [17094.886382] ? __hrtimer_run_queues+0x1c2/0x4d0 [17094.888477] ? tick_sched_handle.isra.18+0x90/0x90 [17094.891240] ? hrtimer_interrupt+0x128/0x2c0 [17094.893002] ? smp_apic_timer_interrupt+0xa6/0x270 [17094.895630] ? apic_timer_interrupt+0xf/0x20 [17094.897403] [17094.898385] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17094.900875] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17094.903408] ? do_raw_spin_lock+0x177/0x190 [17094.904802] ? _raw_spin_lock+0x21/0x30 [17094.906432] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17094.908651] ? obd_mod_free+0x46/0x140 [ptlrpc] [17094.910883] ? mdc_commit_open+0x91/0xb0 [mdc] [17094.912606] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17094.915452] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17094.917532] ? after_reply+0xe2c/0x1780 [ptlrpc] [17094.920803] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17094.923792] ? __next_timer_interrupt+0x160/0x160 [17094.925869] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [17094.928376] ? do_wait_intr+0xf0/0xf0 [17094.930041] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17094.933034] ? kthread+0x1d1/0x200 [17094.934739] ? set_kthread_struct+0x70/0x70 [17094.936750] ? ret_from_fork+0x1f/0x30 [17094.938716] Sending NMI from CPU 2 to CPUs 1: [17094.942636] NMI backtrace for cpu 1 [17094.942641] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17094.942643] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17094.942645] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17094.942647] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17094.942743] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [17094.942746] RAX: 00000000000055d3 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17094.942747] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17094.942747] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17094.942748] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17094.942749] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17094.942750] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17094.942751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17094.942752] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17094.942753] Call Trace: [17094.942753] [17094.942754] ? show_regs.cold.9+0x22/0x2f [17094.942754] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [17094.942755] ? nmi_cpu_backtrace_handler+0x15/0x40 [17094.942755] ? nmi_handle+0xbd/0x210 [17094.942756] ? default_do_nmi+0x4b/0x1b0 [17094.942757] ? do_nmi+0x280/0x380 [17094.942757] ? end_repeat_nmi+0x16/0x1f [17094.942758] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17094.942759] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17094.942760] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17094.942760] [17094.942761] do_raw_spin_lock+0x177/0x190 [17094.942762] _raw_spin_lock+0x21/0x30 [17094.942762] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17094.942763] ? do_raw_spin_unlock+0x75/0x190 [17094.942763] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17094.942764] ? do_wait_intr+0xf0/0xf0 [17094.942765] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17094.942765] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17094.942766] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17094.942767] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17094.942768] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17094.942769] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17094.942769] mdc_intent_lock+0x353/0x7e0 [mdc] [17094.942770] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17094.942771] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17094.942772] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17094.942772] lmv_intent_open+0x338/0x1150 [lmv] [17094.942773] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17094.942773] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17094.942774] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17094.942775] ll_intent_lock+0xc5/0xaa0 [lustre] [17094.942775] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17094.942776] ll_lookup_it+0xe0e/0x2270 [lustre] [17094.942776] ? do_raw_spin_unlock+0x75/0x190 [17094.942777] ? _raw_spin_unlock+0x12/0x30 [17094.942777] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17094.942778] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17094.942779] ll_atomic_open+0x36a/0xd70 [lustre] [17094.942779] lookup_open+0x356/0xb40 [17094.942780] do_last+0x313/0xfc0 [17094.942780] ? nd_jump_root+0xe5/0x160 [17094.942781] ? path_init+0x437/0x520 [17094.942781] path_openat+0xf7/0x500 [17094.942782] ? do_raw_spin_unlock+0x75/0x190 [17094.942782] do_filp_open+0x99/0x140 [17094.942783] ? getname_flags+0x6e/0x330 [17094.942784] ? __check_object_size+0xff/0x256 [17094.942784] ? do_raw_spin_unlock+0x75/0x190 [17094.942785] ? _raw_spin_unlock+0x12/0x30 [17094.942785] do_sys_openat2+0x2b4/0x410 [17094.942786] do_sys_open+0x73/0xa0 [17094.942786] __x64_sys_openat+0x24/0x30 [17094.942787] do_syscall_64+0xc1/0x440 [17094.942788] entry_SYSCALL_64_after_hwframe+0x49/0xae [17094.942788] RIP: 0033:0x7f53425a5332 [17094.942789] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17094.942790] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17094.942791] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17094.942792] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17094.942793] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17094.942794] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17094.942794] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17094.942800] NMI backtrace for cpu 2 [17095.147917] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17095.153246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17095.157437] Call Trace: [17095.159866] [17095.164765] ? dump_stack+0xbb/0x10e [17095.171889] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [17095.175409] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [17095.180567] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [17095.181974] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [17095.185392] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [17095.192551] ? rcu_dump_cpu_stacks+0x11b/0x171 [17095.194130] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [17095.196297] ? update_process_times+0x6d/0xc0 [17095.198039] ? tick_sched_handle.isra.18+0x27/0x90 [17095.199883] ? tick_sched_timer+0x53/0xb0 [17095.201415] ? __hrtimer_run_queues+0x1c2/0x4d0 [17095.203196] ? tick_sched_handle.isra.18+0x90/0x90 [17095.204870] ? hrtimer_interrupt+0x128/0x2c0 [17095.206548] ? smp_apic_timer_interrupt+0xa6/0x270 [17095.212759] ? apic_timer_interrupt+0xf/0x20 [17095.213942] [17095.214646] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17095.220883] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17095.223591] ? do_raw_spin_lock+0x177/0x190 [17095.225886] ? _raw_spin_lock+0x21/0x30 [17095.227494] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17095.229650] ? obd_mod_free+0x46/0x140 [ptlrpc] [17095.232258] ? mdc_commit_open+0x91/0xb0 [mdc] [17095.233963] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17095.236895] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17095.239322] ? after_reply+0xe2c/0x1780 [ptlrpc] [17095.241631] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17095.244227] ? __next_timer_interrupt+0x160/0x160 [17095.246118] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [17095.248395] ? do_wait_intr+0xf0/0xf0 [17095.250065] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17095.252528] ? kthread+0x1d1/0x200 [17095.254071] ? set_kthread_struct+0x70/0x70 [17095.255731] ? ret_from_fork+0x1f/0x30 [17100.054037] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [17100.061970] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17100.083853] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17100.091654] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17100.100134] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17100.104480] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17100.106966] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17100.118349] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17100.122763] RAX: 0000000000005e84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17100.138386] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17100.149072] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17100.165743] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17100.172027] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17100.186898] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17100.189672] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17100.197342] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17100.207977] Call Trace: [17100.210886] [17100.216527] ? show_regs.cold.9+0x22/0x2f [17100.220886] ? watchdog_timer_fn.cold.6+0x62/0x10a [17100.227921] ? __hrtimer_run_queues+0x1c2/0x4d0 [17100.229620] ? watchdog+0x40/0x40 [17100.232375] ? hrtimer_interrupt+0x128/0x2c0 [17100.238555] ? smp_apic_timer_interrupt+0xa6/0x270 [17100.240192] ? apic_timer_interrupt+0xf/0x20 [17100.246189] [17100.247220] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17100.252872] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17100.259430] ? newidle_balance.isra.93+0x207/0x580 [17100.261096] do_raw_spin_lock+0x177/0x190 [17100.268696] _raw_spin_lock+0x21/0x30 [17100.274250] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17100.276864] process_one_work+0x2c8/0x700 [17100.281505] worker_thread+0x296/0x6e0 [17100.283384] ? rescuer_thread+0x570/0x570 [17100.287952] kthread+0x1d1/0x200 [17100.289039] ? set_kthread_struct+0x70/0x70 [17100.297558] ret_from_fork+0x1f/0x30 [17105.887351] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14468s! [17105.891573] Showing busy workqueues and worker pools: [17105.894208] workqueue events: flags=0x0 [17105.898115] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17105.902848] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17105.915993] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17105.918788] pending: cpu_latency_work [ptlrpc] [17105.922707] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17105.927561] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17105.932089] workqueue events_long: flags=0x0 [17105.935152] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17105.937459] pending: lu_objects_flush [obdclass] [17105.940561] workqueue events_unbound: flags=0x2 [17105.942240] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17105.945438] pending: flush_memcg_stats_dwork, idle_cull_fn [17105.948402] workqueue events_power_efficient: flags=0x80 [17105.950363] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17105.953388] pending: check_lifetime, do_cache_clean [17105.955967] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17105.960361] pending: neigh_periodic_work, neigh_periodic_work [17105.966777] workqueue mm_percpu_wq: flags=0x8 [17105.970073] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17105.972377] pending: vmstat_update, lru_add_drain_per_cpu [17105.976040] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17105.985413] pending: vmstat_update, lru_add_drain_per_cpu [17105.988336] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17105.993602] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17105.998348] workqueue xprtiod: flags=0xa [17106.002416] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17106.007915] pending: xprt_autoclose, xs_error_handle [17106.010582] workqueue ipv6_addrconf: flags=0x40008 [17106.017367] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17106.022236] pending: addrconf_verify_work [17106.029239] workqueue knbd0-recv: flags=0x1a [17106.034350] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17106.038873] in-flight: 651:recv_work [17106.040606] workqueue ptlrpc_pinger: flags=0x2 [17106.044648] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17106.046678] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17106.053594] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14468s workers=4 idle: 15979 31824 8 [17106.058279] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17096s workers=2 idle: 654 [17120.055647] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [17120.057055] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [17120.058042] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [17120.058049] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17120.058125] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17120.058131] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17120.058134] RIP: 0010:native_safe_halt+0x2e/0x50 [17120.058147] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17120.058151] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17120.058156] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17120.058158] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17120.058161] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17120.058163] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17120.058166] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17120.058169] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17120.058172] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17120.058175] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17120.058184] Call Trace: [17120.058188] [17120.058194] ? show_regs.cold.9+0x22/0x2f [17120.058206] ? watchdog_timer_fn.cold.6+0x62/0x10a [17120.058213] ? __hrtimer_run_queues+0x1c2/0x4d0 [17120.058218] ? watchdog+0x40/0x40 [17120.058222] ? hrtimer_interrupt+0x128/0x2c0 [17120.058226] ? smp_apic_timer_interrupt+0xa6/0x270 [17120.058234] ? apic_timer_interrupt+0xf/0x20 [17120.058238] [17120.058242] ? native_safe_halt+0x2e/0x50 [17120.058246] kvm_wait+0xcd/0xe0 [17120.058254] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17120.058264] do_raw_spin_lock+0x177/0x190 [17120.058269] _raw_spin_lock+0x21/0x30 [17120.058274] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17120.058430] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17120.058491] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17120.058496] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17120.058498] RIP: 0010:native_safe_halt+0x2e/0x50 [17120.058505] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17120.058508] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17120.058512] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17120.058514] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17120.058516] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17120.058518] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17120.058520] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17120.058523] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17120.058529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17120.058530] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17120.058742] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17120.058962] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17120.059180] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17120.059393] ? woken_wake_function+0x30/0x30 [17120.059402] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17120.059608] kthread+0x1d1/0x200 [17120.059615] ? set_kthread_struct+0x70/0x70 [17120.059619] ret_from_fork+0x1f/0x30 [17120.061880] Modules linked in: [17120.069241] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17120.069258] Call Trace: [17120.069265] [17120.069270] ? show_regs.cold.9+0x22/0x2f [17120.069283] ? watchdog_timer_fn.cold.6+0x62/0x10a [17120.069289] ? __hrtimer_run_queues+0x1c2/0x4d0 [17120.069293] ? watchdog+0x40/0x40 [17120.069296] ? hrtimer_interrupt+0x128/0x2c0 [17120.069300] ? smp_apic_timer_interrupt+0xa6/0x270 [17120.069308] ? apic_timer_interrupt+0xf/0x20 [17120.069312] [17120.069315] ? native_safe_halt+0x2e/0x50 [17120.069318] kvm_wait+0xcd/0xe0 [17120.069326] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17120.069335] do_raw_spin_lock+0x177/0x190 [17120.069339] _raw_spin_lock+0x21/0x30 [17120.069343] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17120.069738] ? do_raw_spin_unlock+0x75/0x190 [17120.069743] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17120.069928] ? do_wait_intr+0xf0/0xf0 [17120.069936] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17120.070283] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17120.100587] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix [17120.109049] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17120.109442] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17120.109555] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17120.109595] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17120.109624] mdc_intent_lock+0x353/0x7e0 [mdc] [17120.111018] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17120.116973] serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17120.116997] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17120.117010] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17120.117013] RIP: 0010:native_safe_halt+0x2e/0x50 [17120.117029] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17120.117035] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17120.117040] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17120.117043] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17120.117045] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17120.117047] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17120.117050] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17120.117054] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17120.117057] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17120.117059] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17120.117068] Call Trace: [17120.117075] [17120.117082] ? show_regs.cold.9+0x22/0x2f [17120.117094] ? watchdog_timer_fn.cold.6+0x62/0x10a [17120.117107] ? __hrtimer_run_queues+0x1c2/0x4d0 [17120.117112] ? watchdog+0x40/0x40 [17120.117116] ? hrtimer_interrupt+0x128/0x2c0 [17120.117121] ? smp_apic_timer_interrupt+0xa6/0x270 [17120.117129] ? apic_timer_interrupt+0xf/0x20 [17120.117134] [17120.117138] ? native_safe_halt+0x2e/0x50 [17120.117142] kvm_wait+0xcd/0xe0 [17120.117149] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17120.117156] do_raw_spin_lock+0x177/0x190 [17120.117161] _raw_spin_lock+0x21/0x30 [17120.117169] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17120.117450] obd_mod_free+0x46/0x140 [ptlrpc] [17120.117820] mdc_commit_open+0x91/0xb0 [mdc] [17120.117857] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17120.118070] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17120.118263] after_reply+0xe2c/0x1780 [ptlrpc] [17120.118462] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17120.131737] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17120.132084] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17120.132127] lmv_intent_open+0x338/0x1150 [lmv] [17120.132157] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17120.132283] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17120.132390] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17120.132417] ll_intent_lock+0xc5/0xaa0 [lustre] [17120.132503] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17120.132610] ll_lookup_it+0xe0e/0x2270 [lustre] [17120.139167] ? __next_timer_interrupt+0x160/0x160 [17120.139191] ptlrpcd+0xa82/0xe40 [ptlrpc] [17120.139517] ? do_wait_intr+0xf0/0xf0 [17120.139532] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17120.139753] kthread+0x1d1/0x200 [17120.139759] ? set_kthread_struct+0x70/0x70 [17120.139764] ret_from_fork+0x1f/0x30 [17120.540530] ? do_raw_spin_unlock+0x75/0x190 [17120.544340] ? _raw_spin_unlock+0x12/0x30 [17120.547960] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17120.551326] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17120.553354] ll_atomic_open+0x36a/0xd70 [lustre] [17120.554929] lookup_open+0x356/0xb40 [17120.556539] do_last+0x313/0xfc0 [17120.558613] ? nd_jump_root+0xe5/0x160 [17120.561613] ? path_init+0x437/0x520 [17120.563903] path_openat+0xf7/0x500 [17120.566436] ? do_raw_spin_unlock+0x75/0x190 [17120.570120] do_filp_open+0x99/0x140 [17120.572112] ? getname_flags+0x6e/0x330 [17120.573102] ? __check_object_size+0xff/0x256 [17120.574616] ? do_raw_spin_unlock+0x75/0x190 [17120.580912] ? _raw_spin_unlock+0x12/0x30 [17120.586350] do_sys_openat2+0x2b4/0x410 [17120.587997] do_sys_open+0x73/0xa0 [17120.589403] __x64_sys_openat+0x24/0x30 [17120.593400] do_syscall_64+0xc1/0x440 [17120.596482] entry_SYSCALL_64_after_hwframe+0x49/0xae [17120.599178] RIP: 0033:0x7f53425a5332 [17120.603832] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17120.616783] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17120.624622] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17120.631413] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17120.636460] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17120.639957] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17120.643413] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17132.058635] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [17132.069428] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17132.119704] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17132.135907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17132.142150] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17132.147435] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17132.154821] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17132.178582] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [17132.186828] RAX: 0000000000007b8c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17132.190770] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17132.196490] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17132.202839] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17132.215942] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17132.220532] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17132.228583] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17132.238340] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17132.255655] Call Trace: [17132.257138] [17132.258569] ? show_regs.cold.9+0x22/0x2f [17132.273711] ? watchdog_timer_fn.cold.6+0x62/0x10a [17132.275266] ? __hrtimer_run_queues+0x1c2/0x4d0 [17132.292684] ? watchdog+0x40/0x40 [17132.293864] ? hrtimer_interrupt+0x128/0x2c0 [17132.295310] ? smp_apic_timer_interrupt+0xa6/0x270 [17132.301087] ? apic_timer_interrupt+0xf/0x20 [17132.303333] [17132.304230] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17132.307132] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17132.318250] ? newidle_balance.isra.93+0x207/0x580 [17132.321610] do_raw_spin_lock+0x177/0x190 [17132.324658] _raw_spin_lock+0x21/0x30 [17132.328353] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17132.330434] process_one_work+0x2c8/0x700 [17132.333301] worker_thread+0x296/0x6e0 [17132.335528] ? rescuer_thread+0x570/0x570 [17132.344129] kthread+0x1d1/0x200 [17132.346510] ? set_kthread_struct+0x70/0x70 [17132.351648] ret_from_fork+0x1f/0x30 [17136.098403] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14499s! [17136.113311] Showing busy workqueues and worker pools: [17136.124320] workqueue events: flags=0x0 [17136.134885] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17136.142199] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17136.157446] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17136.165052] pending: cpu_latency_work [ptlrpc] [17136.167497] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17136.170575] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17136.175144] workqueue events_long: flags=0x0 [17136.176966] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17136.180015] pending: lu_objects_flush [obdclass] [17136.182403] workqueue events_unbound: flags=0x2 [17136.184583] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17136.187959] pending: flush_memcg_stats_dwork, idle_cull_fn [17136.190071] workqueue events_power_efficient: flags=0x80 [17136.192286] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17136.194878] pending: check_lifetime, do_cache_clean [17136.197146] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17136.203806] pending: neigh_periodic_work, neigh_periodic_work [17136.205584] workqueue mm_percpu_wq: flags=0x8 [17136.207852] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17136.210494] pending: vmstat_update, lru_add_drain_per_cpu [17136.212577] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17136.214957] pending: vmstat_update, lru_add_drain_per_cpu [17136.217001] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17136.219557] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17136.222011] workqueue xprtiod: flags=0xa [17136.223466] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17136.225655] pending: xprt_autoclose, xs_error_handle [17136.227663] workqueue ipv6_addrconf: flags=0x40008 [17136.229343] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17136.231823] pending: addrconf_verify_work [17136.233625] workqueue knbd0-recv: flags=0x1a [17136.235087] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17136.237607] in-flight: 651:recv_work [17136.239095] workqueue ptlrpc_pinger: flags=0x2 [17136.240817] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17136.243048] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17136.245275] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14499s workers=4 idle: 15979 31824 8 [17136.248140] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17126s workers=2 idle: 654 [17148.055025] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [17148.058063] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17148.060872] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [17148.066069] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [17148.066084] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17148.066153] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17148.066159] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17148.066161] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17148.066185] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17148.066190] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17148.066194] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17148.066196] RDX: 00000000000048e9 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17148.066199] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17148.066201] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17148.066203] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17148.066206] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17148.066209] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17148.066211] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17148.066220] Call Trace: [17148.066226] [17148.066231] ? show_regs.cold.9+0x22/0x2f [17148.066240] ? watchdog_timer_fn.cold.6+0x62/0x10a [17148.066247] ? __hrtimer_run_queues+0x1c2/0x4d0 [17148.066252] ? watchdog+0x40/0x40 [17148.066256] ? hrtimer_interrupt+0x128/0x2c0 [17148.066260] ? smp_apic_timer_interrupt+0xa6/0x270 [17148.066268] ? apic_timer_interrupt+0xf/0x20 [17148.066272] [17148.066276] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17148.066280] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17148.066285] do_raw_spin_lock+0x177/0x190 [17148.066289] _raw_spin_lock+0x21/0x30 [17148.066297] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17148.070760] obd_mod_free+0x46/0x140 [ptlrpc] [17148.074591] mdc_commit_open+0x91/0xb0 [mdc] [17148.074636] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17148.074889] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17148.077753] after_reply+0xe2c/0x1780 [ptlrpc] [17148.081942] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17148.082904] ? __next_timer_interrupt+0x160/0x160 [17148.082915] ptlrpcd+0xa82/0xe40 [ptlrpc] [17148.083146] ? do_wait_intr+0xf0/0xf0 [17148.083156] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17148.083371] kthread+0x1d1/0x200 [17148.083378] ? set_kthread_struct+0x70/0x70 [17148.083382] ret_from_fork+0x1f/0x30 [17148.105317] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17148.105332] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17148.105336] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17148.110182] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17148.110267] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17148.110274] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17148.110277] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17148.110291] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17148.110295] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17148.110300] RAX: 0000000000003e1f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17148.110303] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17148.110305] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17148.110307] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17148.110309] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17148.110312] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17148.110315] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17148.110317] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17148.110325] Call Trace: [17148.110330] [17148.110336] ? show_regs.cold.9+0x22/0x2f [17148.110346] ? watchdog_timer_fn.cold.6+0x62/0x10a [17148.110351] ? __hrtimer_run_queues+0x1c2/0x4d0 [17148.110355] ? watchdog+0x40/0x40 [17148.110360] ? hrtimer_interrupt+0x128/0x2c0 [17148.110364] ? smp_apic_timer_interrupt+0xa6/0x270 [17148.110372] ? apic_timer_interrupt+0xf/0x20 [17148.110377] [17148.110380] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17148.110385] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17148.110392] do_raw_spin_lock+0x177/0x190 [17148.110397] _raw_spin_lock+0x21/0x30 [17148.110402] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17148.110760] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17148.111049] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17148.111293] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17148.111660] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17148.111966] ? woken_wake_function+0x30/0x30 [17148.111976] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17148.112218] kthread+0x1d1/0x200 [17148.112224] ? set_kthread_struct+0x70/0x70 [17148.112228] ret_from_fork+0x1f/0x30 [17148.627276] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17148.642192] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17148.648473] RAX: 0000000000003719 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17148.656304] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17148.659926] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17148.667049] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17148.672749] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17148.684109] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17148.689416] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17148.692966] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17148.703962] Call Trace: [17148.708367] [17148.709033] ? show_regs.cold.9+0x22/0x2f [17148.712821] ? watchdog_timer_fn.cold.6+0x62/0x10a [17148.716929] ? __hrtimer_run_queues+0x1c2/0x4d0 [17148.719696] ? watchdog+0x40/0x40 [17148.721349] ? hrtimer_interrupt+0x128/0x2c0 [17148.725089] ? smp_apic_timer_interrupt+0xa6/0x270 [17148.727874] ? apic_timer_interrupt+0xf/0x20 [17148.730111] [17148.731408] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17148.734866] do_raw_spin_lock+0x177/0x190 [17148.737950] _raw_spin_lock+0x21/0x30 [17148.739661] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17148.742417] ? do_raw_spin_unlock+0x75/0x190 [17148.744560] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17148.747075] ? do_wait_intr+0xf0/0xf0 [17148.749128] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17148.751727] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17148.756467] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17148.759574] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17148.761972] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17148.768949] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17148.774391] mdc_intent_lock+0x353/0x7e0 [mdc] [17148.779431] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17148.783865] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17148.788029] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17148.791414] lmv_intent_open+0x338/0x1150 [lmv] [17148.794394] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17148.802019] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17148.810562] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17148.812208] ll_intent_lock+0xc5/0xaa0 [lustre] [17148.823442] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17148.836175] ll_lookup_it+0xe0e/0x2270 [lustre] [17148.837910] ? do_raw_spin_unlock+0x75/0x190 [17148.847359] ? _raw_spin_unlock+0x12/0x30 [17148.851077] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17148.860000] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17148.866950] ll_atomic_open+0x36a/0xd70 [lustre] [17148.869936] lookup_open+0x356/0xb40 [17148.876636] do_last+0x313/0xfc0 [17148.877668] ? nd_jump_root+0xe5/0x160 [17148.885803] ? path_init+0x437/0x520 [17148.887371] path_openat+0xf7/0x500 [17148.892486] ? do_raw_spin_unlock+0x75/0x190 [17148.897925] do_filp_open+0x99/0x140 [17148.901909] ? getname_flags+0x6e/0x330 [17148.905649] ? __check_object_size+0xff/0x256 [17148.908928] ? do_raw_spin_unlock+0x75/0x190 [17148.913409] ? _raw_spin_unlock+0x12/0x30 [17148.914701] do_sys_openat2+0x2b4/0x410 [17148.916990] do_sys_open+0x73/0xa0 [17148.922082] __x64_sys_openat+0x24/0x30 [17148.930392] do_syscall_64+0xc1/0x440 [17148.937485] entry_SYSCALL_64_after_hwframe+0x49/0xae [17148.946510] RIP: 0033:0x7f53425a5332 [17148.952973] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17148.974621] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17148.982000] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17148.993567] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17149.002478] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17149.019920] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17149.033081] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17164.053838] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [17164.063405] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17164.106153] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17164.111645] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17164.116287] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17164.126275] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17164.135839] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17164.147619] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17164.152389] RAX: 0000000000007d26 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17164.159721] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17164.166752] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17164.174115] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17164.178574] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17164.184212] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17164.192048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17164.201032] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17164.213384] Call Trace: [17164.214106] [17164.214545] ? show_regs.cold.9+0x22/0x2f [17164.220145] ? watchdog_timer_fn.cold.6+0x62/0x10a [17164.221446] ? __hrtimer_run_queues+0x1c2/0x4d0 [17164.222560] ? watchdog+0x40/0x40 [17164.223461] ? hrtimer_interrupt+0x128/0x2c0 [17164.233714] ? smp_apic_timer_interrupt+0xa6/0x270 [17164.235653] ? apic_timer_interrupt+0xf/0x20 [17164.237287] [17164.238364] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17164.242174] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17164.250234] ? newidle_balance.isra.93+0x207/0x580 [17164.252385] do_raw_spin_lock+0x177/0x190 [17164.256021] _raw_spin_lock+0x21/0x30 [17164.258385] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17164.262285] process_one_work+0x2c8/0x700 [17164.265216] worker_thread+0x296/0x6e0 [17164.271273] ? rescuer_thread+0x570/0x570 [17164.274964] kthread+0x1d1/0x200 [17164.276106] ? set_kthread_struct+0x70/0x70 [17164.281058] ret_from_fork+0x1f/0x30 [17166.304134] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14529s! [17166.307083] Showing busy workqueues and worker pools: [17166.316803] workqueue events: flags=0x0 [17166.324108] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17166.334460] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17166.343688] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17166.346435] pending: cpu_latency_work [ptlrpc] [17166.355701] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17166.359529] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17166.364769] workqueue events_long: flags=0x0 [17166.369797] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17166.373498] pending: lu_objects_flush [obdclass] [17166.375582] workqueue events_unbound: flags=0x2 [17166.383824] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17166.392963] pending: flush_memcg_stats_dwork, idle_cull_fn [17166.394829] workqueue events_power_efficient: flags=0x80 [17166.399431] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17166.403453] pending: check_lifetime, do_cache_clean [17166.405552] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17166.413321] pending: neigh_periodic_work, neigh_periodic_work [17166.416318] workqueue mm_percpu_wq: flags=0x8 [17166.422384] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17166.432170] pending: vmstat_update, lru_add_drain_per_cpu [17166.437330] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17166.442985] pending: vmstat_update, lru_add_drain_per_cpu [17166.451802] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17166.458139] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17166.466253] workqueue xprtiod: flags=0xa [17166.468821] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17166.476247] pending: xprt_autoclose, xs_error_handle [17166.480761] workqueue ipv6_addrconf: flags=0x40008 [17166.485567] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17166.488196] pending: addrconf_verify_work [17166.493462] workqueue knbd0-recv: flags=0x1a [17166.496801] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17166.503378] in-flight: 651:recv_work [17166.505802] workqueue ptlrpc_pinger: flags=0x2 [17166.515227] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17166.519068] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17166.527077] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14529s workers=4 idle: 15979 31824 8 [17166.535946] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17156s workers=2 idle: 654 [17176.055056] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [17176.057029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [17176.057761] Modules linked in: [17176.058030] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [17176.058036] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17176.058109] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17176.058114] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17176.058117] RIP: 0010:native_safe_halt+0x2e/0x50 [17176.058129] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17176.058133] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17176.058138] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17176.058140] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17176.058143] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17176.058145] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17176.058148] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17176.058152] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17176.058155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17176.058158] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17176.058167] Call Trace: [17176.058171] [17176.058176] ? show_regs.cold.9+0x22/0x2f [17176.058196] ? watchdog_timer_fn.cold.6+0x62/0x10a [17176.058203] ? __hrtimer_run_queues+0x1c2/0x4d0 [17176.058207] ? watchdog+0x40/0x40 [17176.058211] ? hrtimer_interrupt+0x128/0x2c0 [17176.058215] ? smp_apic_timer_interrupt+0xa6/0x270 [17176.058221] ? apic_timer_interrupt+0xf/0x20 [17176.058226] [17176.058229] ? native_safe_halt+0x2e/0x50 [17176.058233] kvm_wait+0xcd/0xe0 [17176.058239] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17176.058246] do_raw_spin_lock+0x177/0x190 [17176.058250] _raw_spin_lock+0x21/0x30 [17176.058254] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17176.058535] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17176.058739] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17176.058973] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17176.059193] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17176.059419] ? woken_wake_function+0x30/0x30 [17176.059428] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17176.059640] kthread+0x1d1/0x200 [17176.059645] ? set_kthread_struct+0x70/0x70 [17176.059649] ret_from_fork+0x1f/0x30 [17176.060457] Modules linked in: [17176.061764] lustre(O) [17176.074389] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17176.074461] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17176.074469] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17176.074472] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17176.094394] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) [17176.105772] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17176.105785] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17176.105792] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17176.105794] RDX: 0000000000001a09 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17176.105800] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17176.105803] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17176.105806] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17176.105809] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17176.105811] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17176.105813] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17176.105822] Call Trace: [17176.105828] [17176.105833] ? show_regs.cold.9+0x22/0x2f [17176.105846] ? watchdog_timer_fn.cold.6+0x62/0x10a [17176.105853] ? __hrtimer_run_queues+0x1c2/0x4d0 [17176.105858] ? watchdog+0x40/0x40 [17176.105862] ? hrtimer_interrupt+0x128/0x2c0 [17176.105866] ? smp_apic_timer_interrupt+0xa6/0x270 [17176.105874] ? apic_timer_interrupt+0xf/0x20 [17176.105878] [17176.105888] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17176.105896] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17176.105900] do_raw_spin_lock+0x177/0x190 [17176.105905] _raw_spin_lock+0x21/0x30 [17176.105912] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17176.113476] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17176.113519] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17176.113525] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17176.113528] RIP: 0010:native_safe_halt+0x2e/0x50 [17176.113539] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17176.113543] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17176.113547] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17176.113549] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17176.113552] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17176.113554] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17176.113557] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17176.113560] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17176.113563] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17176.113566] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17176.113573] Call Trace: [17176.113578] [17176.113584] ? show_regs.cold.9+0x22/0x2f [17176.113599] ? watchdog_timer_fn.cold.6+0x62/0x10a [17176.113605] ? __hrtimer_run_queues+0x1c2/0x4d0 [17176.113610] ? watchdog+0x40/0x40 [17176.113615] ? hrtimer_interrupt+0x128/0x2c0 [17176.113619] ? smp_apic_timer_interrupt+0xa6/0x270 [17176.113630] ? apic_timer_interrupt+0xf/0x20 [17176.113634] [17176.113638] ? native_safe_halt+0x2e/0x50 [17176.113642] kvm_wait+0xcd/0xe0 [17176.113650] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17176.113660] do_raw_spin_lock+0x177/0x190 [17176.113669] _raw_spin_lock+0x21/0x30 [17176.113738] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17176.114388] ? do_raw_spin_unlock+0x75/0x190 [17176.114394] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17176.114607] ? do_wait_intr+0xf0/0xf0 [17176.114621] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17176.114848] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17176.115205] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17176.115449] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17176.115588] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17176.115640] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17176.115740] mdc_intent_lock+0x353/0x7e0 [mdc] [17176.115767] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17176.115861] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17176.128766] obd_mod_free+0x46/0x140 [ptlrpc] [17176.132299] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17176.132351] lmv_intent_open+0x338/0x1150 [lmv] [17176.132385] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17176.132498] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17176.132585] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17176.132605] ll_intent_lock+0xc5/0xaa0 [lustre] [17176.132712] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17176.132795] ll_lookup_it+0xe0e/0x2270 [lustre] [17176.132877] ? do_raw_spin_unlock+0x75/0x190 [17176.132887] ? _raw_spin_unlock+0x12/0x30 [17176.132896] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17176.132925] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17176.132946] ll_atomic_open+0x36a/0xd70 [lustre] [17176.133133] lookup_open+0x356/0xb40 [17176.133142] do_last+0x313/0xfc0 [17176.133147] ? nd_jump_root+0xe5/0x160 [17176.133151] ? path_init+0x437/0x520 [17176.133160] path_openat+0xf7/0x500 [17176.133167] ? do_raw_spin_unlock+0x75/0x190 [17176.133171] do_filp_open+0x99/0x140 [17176.133177] ? getname_flags+0x6e/0x330 [17176.133181] ? __check_object_size+0xff/0x256 [17176.133188] ? do_raw_spin_unlock+0x75/0x190 [17176.133192] ? _raw_spin_unlock+0x12/0x30 [17176.133196] do_sys_openat2+0x2b4/0x410 [17176.133201] do_sys_open+0x73/0xa0 [17176.133204] __x64_sys_openat+0x24/0x30 [17176.133208] do_syscall_64+0xc1/0x440 [17176.133215] entry_SYSCALL_64_after_hwframe+0x49/0xae [17176.133221] RIP: 0033:0x7f53425a5332 [17176.133228] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17176.133232] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17176.133237] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17176.133239] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17176.133242] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17176.133244] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17176.133246] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17176.465923] mdc_commit_open+0x91/0xb0 [mdc] [17176.467306] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17176.469837] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17176.472199] after_reply+0xe2c/0x1780 [ptlrpc] [17176.474311] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17176.476751] ? __next_timer_interrupt+0x160/0x160 [17176.478904] ptlrpcd+0xa82/0xe40 [ptlrpc] [17176.480867] ? do_wait_intr+0xf0/0xf0 [17176.482279] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17176.484541] kthread+0x1d1/0x200 [17176.486408] ? set_kthread_struct+0x70/0x70 [17176.488234] ret_from_fork+0x1f/0x30 [17192.052029] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [17192.054623] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17192.080561] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17192.090003] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17192.093888] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17192.098247] RIP: 0010:native_safe_halt+0x2e/0x50 [17192.099995] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17192.107451] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17192.110104] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17192.113100] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17192.115526] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17192.118470] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17192.120553] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17192.122599] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17192.124674] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17192.127364] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17192.129602] Call Trace: [17192.130221] [17192.130772] ? show_regs.cold.9+0x22/0x2f [17192.132060] ? watchdog_timer_fn.cold.6+0x62/0x10a [17192.133626] ? __hrtimer_run_queues+0x1c2/0x4d0 [17192.135018] ? watchdog+0x40/0x40 [17192.136253] ? hrtimer_interrupt+0x128/0x2c0 [17192.137317] ? smp_apic_timer_interrupt+0xa6/0x270 [17192.138545] ? apic_timer_interrupt+0xf/0x20 [17192.139684] [17192.140476] ? native_safe_halt+0x2e/0x50 [17192.141623] kvm_wait+0xcd/0xe0 [17192.142669] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17192.144309] ? newidle_balance.isra.93+0x207/0x580 [17192.145794] do_raw_spin_lock+0x177/0x190 [17192.147221] _raw_spin_lock+0x21/0x30 [17192.148259] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17192.149911] process_one_work+0x2c8/0x700 [17192.151359] worker_thread+0x296/0x6e0 [17192.152781] ? rescuer_thread+0x570/0x570 [17192.154247] kthread+0x1d1/0x200 [17192.155511] ? set_kthread_struct+0x70/0x70 [17192.157136] ret_from_fork+0x1f/0x30 [17197.024081] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14559s! [17197.029244] Showing busy workqueues and worker pools: [17197.030935] workqueue events: flags=0x0 [17197.033136] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17197.035856] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17197.043657] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17197.047536] pending: cpu_latency_work [ptlrpc] [17197.051820] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17197.055293] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17197.067905] workqueue events_long: flags=0x0 [17197.069723] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17197.072980] pending: lu_objects_flush [obdclass] [17197.076790] workqueue events_unbound: flags=0x2 [17197.082136] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17197.085818] pending: flush_memcg_stats_dwork, idle_cull_fn [17197.088850] workqueue events_power_efficient: flags=0x80 [17197.093549] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17197.099040] pending: check_lifetime, do_cache_clean [17197.101576] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17197.104822] pending: neigh_periodic_work, neigh_periodic_work [17197.110178] workqueue mm_percpu_wq: flags=0x8 [17197.112831] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17197.115132] pending: vmstat_update, lru_add_drain_per_cpu [17197.118553] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17197.125166] pending: vmstat_update, lru_add_drain_per_cpu [17197.130971] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17197.137295] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17197.152022] workqueue xprtiod: flags=0xa [17197.153619] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17197.156896] pending: xprt_autoclose, xs_error_handle [17197.161747] workqueue ipv6_addrconf: flags=0x40008 [17197.163815] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17197.170927] pending: addrconf_verify_work [17197.173790] workqueue knbd0-recv: flags=0x1a [17197.175553] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17197.178066] in-flight: 651:recv_work [17197.180104] workqueue ptlrpc_pinger: flags=0x2 [17197.182048] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17197.185731] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17197.191426] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14560s workers=4 idle: 15979 31824 8 [17197.194703] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17187s workers=2 idle: 654 [17204.055753] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [17204.059091] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [17204.059579] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17204.059791] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17204.059797] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17204.059800] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17204.059816] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17204.059821] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17204.059924] RAX: 0000000000002f59 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17204.059926] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17204.059928] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17204.059930] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17204.059933] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17204.059936] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17204.059939] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17204.059942] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17204.059950] Call Trace: [17204.059957] [17204.059963] ? show_regs.cold.9+0x22/0x2f [17204.059974] ? watchdog_timer_fn.cold.6+0x62/0x10a [17204.059980] ? __hrtimer_run_queues+0x1c2/0x4d0 [17204.059984] ? watchdog+0x40/0x40 [17204.059989] ? hrtimer_interrupt+0x128/0x2c0 [17204.059993] ? smp_apic_timer_interrupt+0xa6/0x270 [17204.060001] ? apic_timer_interrupt+0xf/0x20 [17204.060005] [17204.060008] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17204.060016] do_raw_spin_lock+0x177/0x190 [17204.060021] _raw_spin_lock+0x21/0x30 [17204.060027] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17204.063075] ? do_raw_spin_unlock+0x75/0x190 [17204.063084] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17204.065806] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [17204.065817] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17204.065883] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17204.065888] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17204.065891] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17204.066438] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17204.066443] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17204.066448] RAX: 0000000000004a07 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17204.066451] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17204.066453] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17204.066455] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17204.066457] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17204.066460] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17204.066463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17204.066465] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17204.066473] Call Trace: [17204.066478] [17204.066484] ? show_regs.cold.9+0x22/0x2f [17204.066493] ? watchdog_timer_fn.cold.6+0x62/0x10a [17204.066501] ? __hrtimer_run_queues+0x1c2/0x4d0 [17204.066505] ? watchdog+0x40/0x40 [17204.066509] ? hrtimer_interrupt+0x128/0x2c0 [17204.066512] ? smp_apic_timer_interrupt+0xa6/0x270 [17204.066520] ? apic_timer_interrupt+0xf/0x20 [17204.066525] [17204.066527] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17204.066532] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17204.066539] do_raw_spin_lock+0x177/0x190 [17204.066544] _raw_spin_lock+0x21/0x30 [17204.066549] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17204.066798] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17204.068999] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17204.069247] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17204.069466] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17204.069680] ? woken_wake_function+0x30/0x30 [17204.069692] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17204.070756] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17204.070909] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17204.070915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17204.070918] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17204.070929] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17204.070933] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17204.070937] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17204.070939] RDX: 00000000000018c4 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17204.070942] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17204.070944] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17204.070946] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17204.070949] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17204.070952] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17204.070954] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17204.070961] Call Trace: [17204.070965] [17204.070969] ? show_regs.cold.9+0x22/0x2f [17204.070976] ? watchdog_timer_fn.cold.6+0x62/0x10a [17204.070982] ? __hrtimer_run_queues+0x1c2/0x4d0 [17204.070986] ? watchdog+0x40/0x40 [17204.070990] ? hrtimer_interrupt+0x128/0x2c0 [17204.070994] ? smp_apic_timer_interrupt+0xa6/0x270 [17204.071002] ? apic_timer_interrupt+0xf/0x20 [17204.071007] [17204.071010] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17204.071015] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17204.071024] do_raw_spin_lock+0x177/0x190 [17204.071028] _raw_spin_lock+0x21/0x30 [17204.071034] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17204.072273] kthread+0x1d1/0x200 [17204.072283] ? set_kthread_struct+0x70/0x70 [17204.072287] ret_from_fork+0x1f/0x30 [17204.104142] ? do_wait_intr+0xf0/0xf0 [17204.104165] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17204.104539] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17204.105169] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17204.105684] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17204.105815] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17204.105859] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17204.105890] mdc_intent_lock+0x353/0x7e0 [mdc] [17204.105922] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17204.106089] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17204.106340] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17204.106371] lmv_intent_open+0x338/0x1150 [lmv] [17204.106753] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17204.106843] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17204.106933] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17204.106956] ll_intent_lock+0xc5/0xaa0 [lustre] [17204.107046] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17204.107129] ll_lookup_it+0xe0e/0x2270 [lustre] [17204.107748] ? do_raw_spin_unlock+0x75/0x190 [17204.107756] ? _raw_spin_unlock+0x12/0x30 [17204.107764] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17204.107792] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17204.107814] ll_atomic_open+0x36a/0xd70 [lustre] [17204.107905] lookup_open+0x356/0xb40 [17204.107915] do_last+0x313/0xfc0 [17204.107920] ? nd_jump_root+0xe5/0x160 [17204.107924] ? path_init+0x437/0x520 [17204.107929] path_openat+0xf7/0x500 [17204.107934] ? do_raw_spin_unlock+0x75/0x190 [17204.107939] do_filp_open+0x99/0x140 [17204.107946] ? getname_flags+0x6e/0x330 [17204.107954] ? __check_object_size+0xff/0x256 [17204.107962] ? do_raw_spin_unlock+0x75/0x190 [17204.107967] ? _raw_spin_unlock+0x12/0x30 [17204.107972] do_sys_openat2+0x2b4/0x410 [17204.107977] do_sys_open+0x73/0xa0 [17204.107982] __x64_sys_openat+0x24/0x30 [17204.107985] do_syscall_64+0xc1/0x440 [17204.107992] entry_SYSCALL_64_after_hwframe+0x49/0xae [17204.107999] RIP: 0033:0x7f53425a5332 [17204.108007] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17204.108012] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17204.108017] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17204.108020] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17204.108022] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17204.108025] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17204.108027] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17204.641502] obd_mod_free+0x46/0x140 [ptlrpc] [17204.645517] mdc_commit_open+0x91/0xb0 [mdc] [17204.647977] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17204.651006] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17204.658291] after_reply+0xe2c/0x1780 [ptlrpc] [17204.659989] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17204.665025] ? __next_timer_interrupt+0x160/0x160 [17204.669985] ptlrpcd+0xa82/0xe40 [ptlrpc] [17204.672098] ? do_wait_intr+0xf0/0xf0 [17204.674039] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17204.678803] kthread+0x1d1/0x200 [17204.681087] ? set_kthread_struct+0x70/0x70 [17204.682559] ret_from_fork+0x1f/0x30 [17224.052602] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [17224.056466] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17224.087304] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17224.094968] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17224.098569] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17224.103105] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17224.107011] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17224.117200] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [17224.123097] RAX: 0000000000001c88 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17224.130343] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17224.133187] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17224.138508] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17224.143773] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17224.146378] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17224.155157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17224.158762] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17224.169146] Call Trace: [17224.170258] [17224.170824] ? show_regs.cold.9+0x22/0x2f [17224.175934] ? watchdog_timer_fn.cold.6+0x62/0x10a [17224.181415] ? __hrtimer_run_queues+0x1c2/0x4d0 [17224.183598] ? watchdog+0x40/0x40 [17224.190096] ? hrtimer_interrupt+0x128/0x2c0 [17224.191508] ? smp_apic_timer_interrupt+0xa6/0x270 [17224.197175] ? apic_timer_interrupt+0xf/0x20 [17224.201060] [17224.205031] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17224.208970] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17224.216283] ? newidle_balance.isra.93+0x207/0x580 [17224.222062] do_raw_spin_lock+0x177/0x190 [17224.224815] _raw_spin_lock+0x21/0x30 [17224.228275] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17224.234554] process_one_work+0x2c8/0x700 [17224.236524] worker_thread+0x296/0x6e0 [17224.238250] ? rescuer_thread+0x570/0x570 [17224.244257] kthread+0x1d1/0x200 [17224.245619] ? set_kthread_struct+0x70/0x70 [17224.247079] ret_from_fork+0x1f/0x30 [17227.232046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14590s! [17227.234861] Showing busy workqueues and worker pools: [17227.236570] workqueue events: flags=0x0 [17227.237991] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17227.240132] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17227.244305] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17227.246488] pending: cpu_latency_work [ptlrpc] [17227.248490] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17227.250998] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17227.254247] workqueue events_long: flags=0x0 [17227.255596] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17227.258053] pending: lu_objects_flush [obdclass] [17227.259819] workqueue events_unbound: flags=0x2 [17227.261298] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17227.264300] pending: flush_memcg_stats_dwork, idle_cull_fn [17227.266337] workqueue events_power_efficient: flags=0x80 [17227.268483] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17227.270911] pending: check_lifetime, do_cache_clean [17227.273290] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17227.275681] pending: neigh_periodic_work, neigh_periodic_work [17227.278725] workqueue mm_percpu_wq: flags=0x8 [17227.280733] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17227.283588] pending: vmstat_update, lru_add_drain_per_cpu [17227.285710] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17227.288207] pending: vmstat_update, lru_add_drain_per_cpu [17227.289957] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17227.292229] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17227.294279] workqueue xprtiod: flags=0xa [17227.295661] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17227.298250] pending: xprt_autoclose, xs_error_handle [17227.300166] workqueue ipv6_addrconf: flags=0x40008 [17227.302024] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17227.304725] pending: addrconf_verify_work [17227.306324] workqueue knbd0-recv: flags=0x1a [17227.308051] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17227.310719] in-flight: 651:recv_work [17227.312223] workqueue ptlrpc_pinger: flags=0x2 [17227.314081] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17227.316492] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17227.318590] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14590s workers=4 idle: 15979 31824 8 [17227.322058] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17217s workers=2 idle: 654 [17232.055341] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [17232.057123] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [17232.063810] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [17232.063828] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17232.063912] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17232.063920] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17232.063923] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17232.063938] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17232.063942] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17232.063947] RAX: 000000000000228e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17232.063949] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17232.063952] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17232.063954] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17232.063957] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17232.063960] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17232.063964] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17232.063966] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17232.063978] Call Trace: [17232.063983] [17232.063988] ? show_regs.cold.9+0x22/0x2f [17232.063997] ? watchdog_timer_fn.cold.6+0x62/0x10a [17232.064002] ? __hrtimer_run_queues+0x1c2/0x4d0 [17232.064006] ? watchdog+0x40/0x40 [17232.064011] ? hrtimer_interrupt+0x128/0x2c0 [17232.064015] ? smp_apic_timer_interrupt+0xa6/0x270 [17232.064022] ? apic_timer_interrupt+0xf/0x20 [17232.064026] [17232.064029] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17232.064034] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17232.064043] do_raw_spin_lock+0x177/0x190 [17232.064048] _raw_spin_lock+0x21/0x30 [17232.064052] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17232.064653] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17232.064853] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17232.065102] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17232.065173] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17232.065180] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17232.065183] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17232.065194] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17232.065197] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17232.065201] RAX: 0000000000006f08 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17232.065203] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17232.065206] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17232.065208] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17232.065211] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17232.065214] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17232.065217] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17232.065219] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17232.065227] Call Trace: [17232.065231] [17232.065234] ? show_regs.cold.9+0x22/0x2f [17232.065131] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17232.065242] ? watchdog_timer_fn.cold.6+0x62/0x10a [17232.065252] ? __hrtimer_run_queues+0x1c2/0x4d0 [17232.065256] ? watchdog+0x40/0x40 [17232.065260] ? hrtimer_interrupt+0x128/0x2c0 [17232.065264] ? smp_apic_timer_interrupt+0xa6/0x270 [17232.065271] ? apic_timer_interrupt+0xf/0x20 [17232.065276] [17232.065279] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17232.065286] do_raw_spin_lock+0x177/0x190 [17232.065290] _raw_spin_lock+0x21/0x30 [17232.065295] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17232.065674] ? do_raw_spin_unlock+0x75/0x190 [17232.065681] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17232.065900] ? do_wait_intr+0xf0/0xf0 [17232.065909] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17232.066121] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17232.066336] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17232.066880] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17232.067279] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17232.067320] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17232.067352] mdc_intent_lock+0x353/0x7e0 [mdc] [17232.067381] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17232.067468] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17232.068459] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17232.068632] lmv_intent_open+0x338/0x1150 [lmv] [17232.068665] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17232.068806] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17232.068890] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17232.068912] ll_intent_lock+0xc5/0xaa0 [lustre] [17232.068995] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17232.070135] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17232.070209] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17232.070216] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17232.070219] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17232.070232] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17232.070236] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17232.070241] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17232.070243] RDX: 00000000000068e1 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17232.070246] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17232.070248] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17232.070250] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17232.070253] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17232.070256] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17232.070259] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17232.070268] Call Trace: [17232.070273] [17232.070276] ? show_regs.cold.9+0x22/0x2f [17232.070285] ? watchdog_timer_fn.cold.6+0x62/0x10a [17232.070292] ? __hrtimer_run_queues+0x1c2/0x4d0 [17232.070296] ? watchdog+0x40/0x40 [17232.070300] ? hrtimer_interrupt+0x128/0x2c0 [17232.070304] ? smp_apic_timer_interrupt+0xa6/0x270 [17232.070311] ? apic_timer_interrupt+0xf/0x20 [17232.070316] [17232.070365] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17232.070372] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17232.070378] do_raw_spin_lock+0x177/0x190 [17232.070383] _raw_spin_lock+0x21/0x30 [17232.070389] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17232.070640] obd_mod_free+0x46/0x140 [ptlrpc] [17232.070887] ll_lookup_it+0xe0e/0x2270 [lustre] [17232.070994] ? do_raw_spin_unlock+0x75/0x190 [17232.071000] ? _raw_spin_unlock+0x12/0x30 [17232.071005] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17232.071034] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17232.071053] ll_atomic_open+0x36a/0xd70 [lustre] [17232.071318] lookup_open+0x356/0xb40 [17232.071327] do_last+0x313/0xfc0 [17232.071332] ? nd_jump_root+0xe5/0x160 [17232.071337] ? path_init+0x437/0x520 [17232.071342] path_openat+0xf7/0x500 [17232.071383] ? do_raw_spin_unlock+0x75/0x190 [17232.071388] do_filp_open+0x99/0x140 [17232.071395] ? getname_flags+0x6e/0x330 [17232.071404] ? __check_object_size+0xff/0x256 [17232.071412] ? do_raw_spin_unlock+0x75/0x190 [17232.071416] ? _raw_spin_unlock+0x12/0x30 [17232.071421] do_sys_openat2+0x2b4/0x410 [17232.071426] do_sys_open+0x73/0xa0 [17232.071431] __x64_sys_openat+0x24/0x30 [17232.071435] do_syscall_64+0xc1/0x440 [17232.071446] entry_SYSCALL_64_after_hwframe+0x49/0xae [17232.071452] RIP: 0033:0x7f53425a5332 [17232.071459] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17232.071463] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17232.071468] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17232.071470] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17232.071472] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17232.071474] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17232.071476] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17232.074172] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17232.107810] mdc_commit_open+0x91/0xb0 [mdc] [17232.107877] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17232.108208] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17232.108431] after_reply+0xe2c/0x1780 [ptlrpc] [17232.108638] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17232.108921] ? __next_timer_interrupt+0x160/0x160 [17232.108932] ptlrpcd+0xa82/0xe40 [ptlrpc] [17232.109141] ? do_wait_intr+0xf0/0xf0 [17232.109152] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17232.109427] kthread+0x1d1/0x200 [17232.109433] ? set_kthread_struct+0x70/0x70 [17232.109437] ret_from_fork+0x1f/0x30 [17232.732037] ? woken_wake_function+0x30/0x30 [17232.734082] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17232.737127] kthread+0x1d1/0x200 [17232.738761] ? set_kthread_struct+0x70/0x70 [17232.740639] ret_from_fork+0x1f/0x30 [17252.053330] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [17252.064734] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17252.112998] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17252.120578] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17252.127986] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17252.130320] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17252.135293] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17252.150780] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [17252.157127] RAX: 0000000000007789 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17252.160849] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17252.164506] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17252.169003] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17252.176042] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17252.185258] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17252.190089] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17252.197023] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17252.201403] Call Trace: [17252.202335] [17252.205183] ? show_regs.cold.9+0x22/0x2f [17252.207490] ? watchdog_timer_fn.cold.6+0x62/0x10a [17252.211584] ? __hrtimer_run_queues+0x1c2/0x4d0 [17252.214132] ? watchdog+0x40/0x40 [17252.219231] ? hrtimer_interrupt+0x128/0x2c0 [17252.222035] ? smp_apic_timer_interrupt+0xa6/0x270 [17252.224833] ? apic_timer_interrupt+0xf/0x20 [17252.229357] [17252.230167] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17252.232020] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17252.236716] ? newidle_balance.isra.93+0x207/0x580 [17252.239275] do_raw_spin_lock+0x177/0x190 [17252.241833] _raw_spin_lock+0x21/0x30 [17252.242924] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17252.248573] process_one_work+0x2c8/0x700 [17252.249939] worker_thread+0x296/0x6e0 [17252.253080] ? rescuer_thread+0x570/0x570 [17252.257497] kthread+0x1d1/0x200 [17252.260194] ? set_kthread_struct+0x70/0x70 [17252.261614] ret_from_fork+0x1f/0x30 [17257.447055] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14620s! [17257.452230] Showing busy workqueues and worker pools: [17257.454396] workqueue events: flags=0x0 [17257.456440] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17257.459885] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17257.466839] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17257.471976] pending: cpu_latency_work [ptlrpc] [17257.475882] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17257.480853] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17257.490092] workqueue events_long: flags=0x0 [17257.493790] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17257.498139] pending: lu_objects_flush [obdclass] [17257.502412] workqueue events_unbound: flags=0x2 [17257.504315] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17257.507725] pending: flush_memcg_stats_dwork, idle_cull_fn [17257.511494] workqueue events_power_efficient: flags=0x80 [17257.517430] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17257.520901] pending: check_lifetime, do_cache_clean [17257.524396] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17257.526718] pending: neigh_periodic_work, neigh_periodic_work [17257.531976] workqueue mm_percpu_wq: flags=0x8 [17257.534955] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17257.539116] pending: vmstat_update, lru_add_drain_per_cpu [17257.541955] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17257.547621] pending: vmstat_update, lru_add_drain_per_cpu [17257.550861] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17257.557407] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17257.561064] workqueue xprtiod: flags=0xa [17257.562973] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17257.567859] pending: xprt_autoclose, xs_error_handle [17257.571423] workqueue ipv6_addrconf: flags=0x40008 [17257.574561] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17257.580692] pending: addrconf_verify_work [17257.585479] workqueue knbd0-recv: flags=0x1a [17257.590213] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17257.595475] in-flight: 651:recv_work [17257.599818] workqueue ptlrpc_pinger: flags=0x2 [17257.601891] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17257.605062] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17257.609475] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14620s workers=4 idle: 15979 31824 8 [17257.614559] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17248s workers=2 idle: 654 [17260.055092] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [17260.057025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [17260.057547] Modules linked in: lustre(O) [17260.060676] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17260.060761] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17260.060769] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17260.060772] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17260.060787] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17260.060791] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17260.060796] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17260.060799] RDX: 0000000000001205 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17260.060801] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17260.060803] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17260.060817] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17260.060821] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17260.060823] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17260.060826] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17260.060833] Call Trace: [17260.060840] [17260.060846] ? show_regs.cold.9+0x22/0x2f [17260.060857] ? watchdog_timer_fn.cold.6+0x62/0x10a [17260.060864] ? __hrtimer_run_queues+0x1c2/0x4d0 [17260.060869] ? watchdog+0x40/0x40 [17260.060873] ? hrtimer_interrupt+0x128/0x2c0 [17260.060878] ? smp_apic_timer_interrupt+0xa6/0x270 [17260.060887] ? apic_timer_interrupt+0xf/0x20 [17260.060892] [17260.060895] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17260.060900] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17260.060906] do_raw_spin_lock+0x177/0x190 [17260.060911] _raw_spin_lock+0x21/0x30 [17260.060917] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17260.063489] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [17260.063501] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17260.063587] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17260.063593] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17260.063595] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17260.063609] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17260.063614] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17260.063619] RAX: 0000000000007b12 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17260.063622] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17260.063624] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17260.063626] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17260.063629] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17260.063632] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17260.063635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17260.063637] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17260.063645] Call Trace: [17260.063651] [17260.063660] ? show_regs.cold.9+0x22/0x2f [17260.063670] ? watchdog_timer_fn.cold.6+0x62/0x10a [17260.063677] ? __hrtimer_run_queues+0x1c2/0x4d0 [17260.063681] ? watchdog+0x40/0x40 [17260.063686] ? hrtimer_interrupt+0x128/0x2c0 [17260.063693] ? smp_apic_timer_interrupt+0xa6/0x270 [17260.063700] ? apic_timer_interrupt+0xf/0x20 [17260.063705] [17260.063708] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17260.063713] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17260.063719] do_raw_spin_lock+0x177/0x190 [17260.063726] _raw_spin_lock+0x21/0x30 [17260.064507] osp(O) ofd(O) lod(O) [17260.063731] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17260.066014] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17260.066233] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17260.068457] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17260.068703] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17260.068933] ? woken_wake_function+0x30/0x30 [17260.068943] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17260.069165] kthread+0x1d1/0x200 [17260.069171] ? set_kthread_struct+0x70/0x70 [17260.069181] ret_from_fork+0x1f/0x30 [17260.086859] obd_mod_free+0x46/0x140 [ptlrpc] [17260.087611] mdc_commit_open+0x91/0xb0 [mdc] [17260.087650] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17260.087851] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17260.088053] after_reply+0xe2c/0x1780 [ptlrpc] [17260.088400] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17260.088661] ? __next_timer_interrupt+0x160/0x160 [17260.088673] ptlrpcd+0xa82/0xe40 [ptlrpc] [17260.088883] ? do_wait_intr+0xf0/0xf0 [17260.088892] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17260.089108] kthread+0x1d1/0x200 [17260.089115] ? set_kthread_struct+0x70/0x70 [17260.089119] ret_from_fork+0x1f/0x30 [17260.351568] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17260.367559] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17260.375079] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17260.385171] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17260.392194] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17260.404152] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17260.406909] RAX: 000000000000043c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17260.409875] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17260.412624] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17260.415451] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17260.420964] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17260.428070] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17260.435099] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17260.443292] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17260.448565] Call Trace: [17260.449436] [17260.450169] ? show_regs.cold.9+0x22/0x2f [17260.454349] ? watchdog_timer_fn.cold.6+0x62/0x10a [17260.457761] ? __hrtimer_run_queues+0x1c2/0x4d0 [17260.461816] ? watchdog+0x40/0x40 [17260.463364] ? hrtimer_interrupt+0x128/0x2c0 [17260.467172] ? smp_apic_timer_interrupt+0xa6/0x270 [17260.472510] ? apic_timer_interrupt+0xf/0x20 [17260.477166] [17260.478028] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17260.483726] do_raw_spin_lock+0x177/0x190 [17260.486785] _raw_spin_lock+0x21/0x30 [17260.488479] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17260.490482] ? do_raw_spin_unlock+0x75/0x190 [17260.492494] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17260.494462] ? do_wait_intr+0xf0/0xf0 [17260.496311] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17260.500719] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17260.503100] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17260.507016] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17260.510338] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17260.514811] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17260.518672] mdc_intent_lock+0x353/0x7e0 [mdc] [17260.520549] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17260.525350] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17260.529305] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17260.532894] lmv_intent_open+0x338/0x1150 [lmv] [17260.537324] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17260.541174] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17260.545805] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17260.547935] ll_intent_lock+0xc5/0xaa0 [lustre] [17260.549621] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17260.552806] ll_lookup_it+0xe0e/0x2270 [lustre] [17260.554989] ? do_raw_spin_unlock+0x75/0x190 [17260.556994] ? _raw_spin_unlock+0x12/0x30 [17260.559345] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17260.561069] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17260.566441] ll_atomic_open+0x36a/0xd70 [lustre] [17260.572563] lookup_open+0x356/0xb40 [17260.573739] do_last+0x313/0xfc0 [17260.574854] ? nd_jump_root+0xe5/0x160 [17260.576903] ? path_init+0x437/0x520 [17260.578869] path_openat+0xf7/0x500 [17260.582745] ? do_raw_spin_unlock+0x75/0x190 [17260.584878] do_filp_open+0x99/0x140 [17260.586886] ? getname_flags+0x6e/0x330 [17260.589087] ? __check_object_size+0xff/0x256 [17260.595109] ? do_raw_spin_unlock+0x75/0x190 [17260.597053] ? _raw_spin_unlock+0x12/0x30 [17260.598561] do_sys_openat2+0x2b4/0x410 [17260.601812] do_sys_open+0x73/0xa0 [17260.603066] __x64_sys_openat+0x24/0x30 [17260.605455] do_syscall_64+0xc1/0x440 [17260.607343] entry_SYSCALL_64_after_hwframe+0x49/0xae [17260.610053] RIP: 0033:0x7f53425a5332 [17260.612897] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17260.623352] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17260.628928] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17260.633812] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17260.638596] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17260.642846] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17260.647558] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17274.799012] rcu: INFO: rcu_sched self-detected stall on CPU [17274.801226] rcu: 2-....: (8490917 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [17274.804348] (t=15180373 jiffies g=274185 q=67861) [17274.805887] rcu: rcu_sched kthread starved for 14638739 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [17274.809310] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [17274.812309] rcu: RCU grace-period kthread stack dump: [17274.814006] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [17274.816605] Call Trace: [17274.817369] __schedule+0x351/0xcb0 [17274.818783] schedule+0xc0/0x180 [17274.819799] schedule_timeout+0xb4/0x190 [17274.820923] ? do_raw_spin_unlock+0x75/0x190 [17274.822369] ? __next_timer_interrupt+0x160/0x160 [17274.823833] ? prepare_to_swait_event+0xf3/0x140 [17274.825513] rcu_gp_kthread+0x8dd/0x15b0 [17274.826836] ? force_qs_rnp+0x2c0/0x2c0 [17274.828339] kthread+0x1d1/0x200 [17274.829458] ? set_kthread_struct+0x70/0x70 [17274.830936] ret_from_fork+0x1f/0x30 [17274.832540] rcu: Stack dump where RCU GP kthread last ran: [17274.834424] Sending NMI from CPU 2 to CPUs 0: [17274.838620] NMI backtrace for cpu 0 [17274.838627] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17274.838629] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17274.838631] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17274.838634] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17274.838636] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17274.838639] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 [17274.838640] RAX: 0000000000007e10 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17274.838642] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17274.838643] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17274.838644] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17274.838646] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17274.838647] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17274.838649] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17274.838655] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17274.838657] Call Trace: [17274.838658] [17274.838659] ? show_regs.cold.9+0x22/0x2f [17274.838660] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [17274.838661] ? nmi_cpu_backtrace_handler+0x15/0x40 [17274.838663] ? nmi_handle+0xbd/0x210 [17274.838664] ? default_do_nmi+0x4b/0x1b0 [17274.838665] ? do_nmi+0x280/0x380 [17274.838666] ? end_repeat_nmi+0x16/0x1f [17274.838667] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17274.838669] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17274.838670] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17274.838671] [17274.838672] ? newidle_balance.isra.93+0x207/0x580 [17274.838673] do_raw_spin_lock+0x177/0x190 [17274.838675] _raw_spin_lock+0x21/0x30 [17274.838676] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17274.838677] process_one_work+0x2c8/0x700 [17274.838678] worker_thread+0x296/0x6e0 [17274.838679] ? rescuer_thread+0x570/0x570 [17274.838680] kthread+0x1d1/0x200 [17274.838682] ? set_kthread_struct+0x70/0x70 [17274.838683] ret_from_fork+0x1f/0x30 [17274.838823] Sending NMI from CPU 2 to CPUs 1: [17274.922015] NMI backtrace for cpu 1 [17274.922022] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17274.922024] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17274.922026] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17274.922028] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17274.922030] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [17274.922033] RAX: 0000000000006589 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17274.922035] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17274.922037] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17274.922038] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17274.922040] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17274.922041] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17274.922042] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17274.922044] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17274.922045] Call Trace: [17274.922046] [17274.922047] ? show_regs.cold.9+0x22/0x2f [17274.922048] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [17274.922049] ? nmi_cpu_backtrace_handler+0x15/0x40 [17274.922051] ? nmi_handle+0xbd/0x210 [17274.922052] ? default_do_nmi+0x4b/0x1b0 [17274.922052] ? do_nmi+0x280/0x380 [17274.922053] ? end_repeat_nmi+0x16/0x1f [17274.922054] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17274.922055] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17274.922056] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17274.922057] [17274.922058] do_raw_spin_lock+0x177/0x190 [17274.922059] _raw_spin_lock+0x21/0x30 [17274.922060] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17274.922061] ? do_raw_spin_unlock+0x75/0x190 [17274.922062] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17274.922063] ? do_wait_intr+0xf0/0xf0 [17274.922065] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17274.922066] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17274.922067] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17274.922068] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17274.922069] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17274.922070] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17274.922071] mdc_intent_lock+0x353/0x7e0 [mdc] [17274.922072] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17274.922073] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17274.922074] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17274.922075] lmv_intent_open+0x338/0x1150 [lmv] [17274.922076] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17274.922077] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17274.922078] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17274.922079] ll_intent_lock+0xc5/0xaa0 [lustre] [17274.922080] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17274.922081] ll_lookup_it+0xe0e/0x2270 [lustre] [17274.922082] ? do_raw_spin_unlock+0x75/0x190 [17274.922083] ? _raw_spin_unlock+0x12/0x30 [17274.922084] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17274.922085] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17274.922086] ll_atomic_open+0x36a/0xd70 [lustre] [17274.922087] lookup_open+0x356/0xb40 [17274.922088] do_last+0x313/0xfc0 [17274.922089] ? nd_jump_root+0xe5/0x160 [17274.922090] ? path_init+0x437/0x520 [17274.922092] path_openat+0xf7/0x500 [17274.922093] ? do_raw_spin_unlock+0x75/0x190 [17274.922094] do_filp_open+0x99/0x140 [17274.922096] ? getname_flags+0x6e/0x330 [17274.922097] ? __check_object_size+0xff/0x256 [17274.922098] ? do_raw_spin_unlock+0x75/0x190 [17274.922099] ? _raw_spin_unlock+0x12/0x30 [17274.922100] do_sys_openat2+0x2b4/0x410 [17274.922101] do_sys_open+0x73/0xa0 [17274.922102] __x64_sys_openat+0x24/0x30 [17274.922104] do_syscall_64+0xc1/0x440 [17274.922105] entry_SYSCALL_64_after_hwframe+0x49/0xae [17274.922106] RIP: 0033:0x7f53425a5332 [17274.922108] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17274.922110] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17274.922113] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17274.922114] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17274.922115] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17274.922117] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17274.922118] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17274.922161] NMI backtrace for cpu 2 [17275.069930] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17275.074305] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17275.077075] Call Trace: [17275.077735] [17275.078355] ? dump_stack+0xbb/0x10e [17275.079407] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [17275.080896] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [17275.082446] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [17275.083905] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [17275.086308] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [17275.087913] ? rcu_dump_cpu_stacks+0x11b/0x171 [17275.089443] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [17275.091145] ? update_process_times+0x6d/0xc0 [17275.092435] ? tick_sched_handle.isra.18+0x27/0x90 [17275.093893] ? tick_sched_timer+0x53/0xb0 [17275.095221] ? __hrtimer_run_queues+0x1c2/0x4d0 [17275.096666] ? tick_sched_handle.isra.18+0x90/0x90 [17275.098369] ? hrtimer_interrupt+0x128/0x2c0 [17275.099866] ? smp_apic_timer_interrupt+0xa6/0x270 [17275.101418] ? apic_timer_interrupt+0xf/0x20 [17275.102732] [17275.103387] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17275.105164] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17275.107130] ? do_raw_spin_lock+0x177/0x190 [17275.108432] ? _raw_spin_lock+0x21/0x30 [17275.109733] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17275.111747] ? obd_mod_free+0x46/0x140 [ptlrpc] [17275.113689] ? mdc_commit_open+0x91/0xb0 [mdc] [17275.114911] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17275.117186] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17275.119342] ? after_reply+0xe2c/0x1780 [ptlrpc] [17275.121146] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17275.123271] ? __next_timer_interrupt+0x160/0x160 [17275.124657] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [17275.126287] ? do_wait_intr+0xf0/0xf0 [17275.127378] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17275.129001] ? kthread+0x1d1/0x200 [17275.130138] ? set_kthread_struct+0x70/0x70 [17275.131400] ? ret_from_fork+0x1f/0x30 [17287.648296] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14650s! [17287.657514] Showing busy workqueues and worker pools: [17287.660571] workqueue events: flags=0x0 [17287.662643] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17287.668556] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17287.679191] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17287.688954] pending: cpu_latency_work [ptlrpc] [17287.693963] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17287.699875] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17287.711125] workqueue events_long: flags=0x0 [17287.713543] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17287.720121] pending: lu_objects_flush [obdclass] [17287.723217] workqueue events_unbound: flags=0x2 [17287.731031] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17287.734923] pending: flush_memcg_stats_dwork, idle_cull_fn [17287.741829] workqueue events_power_efficient: flags=0x80 [17287.746608] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17287.749591] pending: check_lifetime, do_cache_clean [17287.751444] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17287.755322] pending: neigh_periodic_work, neigh_periodic_work [17287.757866] workqueue mm_percpu_wq: flags=0x8 [17287.759925] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17287.762784] pending: vmstat_update, lru_add_drain_per_cpu [17287.766534] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17287.769405] pending: vmstat_update, lru_add_drain_per_cpu [17287.772301] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17287.778427] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17287.782988] workqueue xprtiod: flags=0xa [17287.785514] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17287.788841] pending: xprt_autoclose, xs_error_handle [17287.791849] workqueue ipv6_addrconf: flags=0x40008 [17287.794248] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17287.796965] pending: addrconf_verify_work [17287.798686] workqueue knbd0-recv: flags=0x1a [17287.802420] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17287.807396] in-flight: 651:recv_work [17287.809228] workqueue ptlrpc_pinger: flags=0x2 [17287.811225] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17287.814389] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17287.818842] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14650s workers=4 idle: 15979 31824 8 [17287.823762] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17278s workers=2 idle: 654 [17288.058024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [17288.062201] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17288.080184] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17288.084548] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17288.088107] RIP: 0010:native_safe_halt+0x2e/0x50 [17288.090364] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17288.098505] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17288.101240] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17288.104003] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17288.107059] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17288.114964] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17288.118457] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17288.124368] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17288.127342] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17288.129268] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17288.132177] Call Trace: [17288.133191] [17288.134410] ? show_regs.cold.9+0x22/0x2f [17288.135936] ? watchdog_timer_fn.cold.6+0x62/0x10a [17288.137761] ? __hrtimer_run_queues+0x1c2/0x4d0 [17288.140312] ? watchdog+0x40/0x40 [17288.141618] ? hrtimer_interrupt+0x128/0x2c0 [17288.143378] ? smp_apic_timer_interrupt+0xa6/0x270 [17288.145055] ? apic_timer_interrupt+0xf/0x20 [17288.146618] [17288.147543] ? native_safe_halt+0x2e/0x50 [17288.148888] kvm_wait+0xcd/0xe0 [17288.150406] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17288.152299] do_raw_spin_lock+0x177/0x190 [17288.153725] _raw_spin_lock+0x21/0x30 [17288.154967] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17288.157604] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17288.159479] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17288.162024] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17288.164168] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17288.166325] ? woken_wake_function+0x30/0x30 [17288.168006] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17288.170475] kthread+0x1d1/0x200 [17288.171701] ? set_kthread_struct+0x70/0x70 [17288.173088] ret_from_fork+0x1f/0x30 [17300.055026] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [17300.057599] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [17300.060365] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17300.060438] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17300.060445] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17300.060448] RIP: 0010:native_safe_halt+0x2e/0x50 [17300.060460] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17300.060465] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17300.060470] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17300.060472] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17300.060474] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17300.060476] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17300.060479] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17300.060482] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17300.060485] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17300.060488] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17300.060496] Call Trace: [17300.060518] [17300.060524] ? show_regs.cold.9+0x22/0x2f [17300.060534] ? watchdog_timer_fn.cold.6+0x62/0x10a [17300.060541] ? __hrtimer_run_queues+0x1c2/0x4d0 [17300.060545] ? watchdog+0x40/0x40 [17300.060549] ? hrtimer_interrupt+0x128/0x2c0 [17300.060553] ? smp_apic_timer_interrupt+0xa6/0x270 [17300.060559] ? apic_timer_interrupt+0xf/0x20 [17300.060564] [17300.060567] ? native_safe_halt+0x2e/0x50 [17300.060571] kvm_wait+0xcd/0xe0 [17300.060578] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17300.060597] do_raw_spin_lock+0x177/0x190 [17300.060603] _raw_spin_lock+0x21/0x30 [17300.060608] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17300.060882] ? do_raw_spin_unlock+0x75/0x190 [17300.060901] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17300.061104] ? do_wait_intr+0xf0/0xf0 [17300.061113] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17300.061314] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17300.061861] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17300.062136] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17300.062245] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17300.062282] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17300.062311] mdc_intent_lock+0x353/0x7e0 [mdc] [17300.062340] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17300.062422] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17300.066092] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17300.066169] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17300.066176] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17300.066179] RIP: 0010:native_safe_halt+0x2e/0x50 [17300.066193] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17300.066197] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17300.066201] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17300.066204] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17300.066206] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17300.066208] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17300.066211] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17300.066214] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17300.066217] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17300.066220] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17300.066227] Call Trace: [17300.066232] [17300.066237] ? show_regs.cold.9+0x22/0x2f [17300.066246] ? watchdog_timer_fn.cold.6+0x62/0x10a [17300.066252] ? __hrtimer_run_queues+0x1c2/0x4d0 [17300.066256] ? watchdog+0x40/0x40 [17300.066260] ? hrtimer_interrupt+0x128/0x2c0 [17300.066264] ? smp_apic_timer_interrupt+0xa6/0x270 [17300.066270] ? apic_timer_interrupt+0xf/0x20 [17300.066274] [17300.066277] ? native_safe_halt+0x2e/0x50 [17300.066281] kvm_wait+0xcd/0xe0 [17300.066287] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17300.066294] do_raw_spin_lock+0x177/0x190 [17300.066298] _raw_spin_lock+0x21/0x30 [17300.066303] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17300.066585] obd_mod_free+0x46/0x140 [ptlrpc] [17300.066776] mdc_commit_open+0x91/0xb0 [mdc] [17300.066814] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17300.067007] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17300.091357] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17300.091406] lmv_intent_open+0x338/0x1150 [lmv] [17300.091439] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17300.101813] after_reply+0xe2c/0x1780 [ptlrpc] [17300.102163] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17300.102361] ? __next_timer_interrupt+0x160/0x160 [17300.102370] ptlrpcd+0xa82/0xe40 [ptlrpc] [17300.110006] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17300.110132] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17300.110165] ll_intent_lock+0xc5/0xaa0 [lustre] [17300.110257] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17300.110348] ll_lookup_it+0xe0e/0x2270 [lustre] [17300.110439] ? do_raw_spin_unlock+0x75/0x190 [17300.110448] ? _raw_spin_unlock+0x12/0x30 [17300.110455] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17300.110482] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17300.114998] ? do_wait_intr+0xf0/0xf0 [17300.115019] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17300.125193] ll_atomic_open+0x36a/0xd70 [lustre] [17300.125320] lookup_open+0x356/0xb40 [17300.125332] do_last+0x313/0xfc0 [17300.125337] ? nd_jump_root+0xe5/0x160 [17300.125342] ? path_init+0x437/0x520 [17300.125347] path_openat+0xf7/0x500 [17300.125352] ? do_raw_spin_unlock+0x75/0x190 [17300.125359] do_filp_open+0x99/0x140 [17300.125366] ? getname_flags+0x6e/0x330 [17300.125370] ? __check_object_size+0xff/0x256 [17300.125378] ? do_raw_spin_unlock+0x75/0x190 [17300.125382] ? _raw_spin_unlock+0x12/0x30 [17300.125389] do_sys_openat2+0x2b4/0x410 [17300.125394] do_sys_open+0x73/0xa0 [17300.125398] __x64_sys_openat+0x24/0x30 [17300.125402] do_syscall_64+0xc1/0x440 [17300.125409] entry_SYSCALL_64_after_hwframe+0x49/0xae [17300.125415] RIP: 0033:0x7f53425a5332 [17300.125458] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17300.125463] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17300.125469] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17300.125472] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17300.125475] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17300.125477] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17300.125480] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17300.548419] kthread+0x1d1/0x200 [17300.549972] ? set_kthread_struct+0x70/0x70 [17300.551810] ret_from_fork+0x1f/0x30 [17312.054276] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [17312.059497] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17312.100074] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17312.109832] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17312.115759] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17312.123661] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17312.130013] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17312.144802] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [17312.152253] RAX: 0000000000006514 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17312.155450] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17312.161585] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17312.165570] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17312.168820] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17312.176191] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17312.186415] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17312.193708] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17312.196178] Call Trace: [17312.198122] [17312.199248] ? show_regs.cold.9+0x22/0x2f [17312.203638] ? watchdog_timer_fn.cold.6+0x62/0x10a [17312.206035] ? __hrtimer_run_queues+0x1c2/0x4d0 [17312.207992] ? watchdog+0x40/0x40 [17312.208962] ? hrtimer_interrupt+0x128/0x2c0 [17312.210815] ? smp_apic_timer_interrupt+0xa6/0x270 [17312.215896] ? apic_timer_interrupt+0xf/0x20 [17312.218073] [17312.220084] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17312.224269] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17312.227509] ? newidle_balance.isra.93+0x207/0x580 [17312.230282] do_raw_spin_lock+0x177/0x190 [17312.232681] _raw_spin_lock+0x21/0x30 [17312.234718] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17312.237470] process_one_work+0x2c8/0x700 [17312.239443] worker_thread+0x296/0x6e0 [17312.242073] ? rescuer_thread+0x570/0x570 [17312.244100] kthread+0x1d1/0x200 [17312.245752] ? set_kthread_struct+0x70/0x70 [17312.247990] ret_from_fork+0x1f/0x30 [17316.058503] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [17316.063054] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17316.106102] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17316.121579] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17316.124747] RIP: 0010:native_safe_halt+0x2e/0x50 [17316.126571] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17316.133368] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17316.136046] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17316.138868] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17316.141730] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17316.144573] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17316.147553] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17316.152103] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17316.155041] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17316.157335] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17316.159938] Call Trace: [17316.160916] [17316.161756] ? show_regs.cold.9+0x22/0x2f [17316.163189] ? watchdog_timer_fn.cold.6+0x62/0x10a [17316.165397] ? __hrtimer_run_queues+0x1c2/0x4d0 [17316.166835] ? watchdog+0x40/0x40 [17316.168107] ? hrtimer_interrupt+0x128/0x2c0 [17316.169973] ? smp_apic_timer_interrupt+0xa6/0x270 [17316.171679] ? apic_timer_interrupt+0xf/0x20 [17316.173342] [17316.174111] ? native_safe_halt+0x2e/0x50 [17316.175663] kvm_wait+0xcd/0xe0 [17316.176891] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17316.178891] do_raw_spin_lock+0x177/0x190 [17316.180484] _raw_spin_lock+0x21/0x30 [17316.182407] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17316.184940] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17316.187174] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17316.189526] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17316.191836] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17316.194008] ? woken_wake_function+0x30/0x30 [17316.195691] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17316.198618] kthread+0x1d1/0x200 [17316.199854] ? set_kthread_struct+0x70/0x70 [17316.201462] ret_from_fork+0x1f/0x30 [17317.856062] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14680s! [17317.860727] Showing busy workqueues and worker pools: [17317.863302] workqueue events: flags=0x0 [17317.865389] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17317.869175] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17317.876226] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17317.879715] pending: cpu_latency_work [ptlrpc] [17317.883072] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17317.886715] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17317.892085] workqueue events_long: flags=0x0 [17317.894988] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17317.899114] pending: lu_objects_flush [obdclass] [17317.904031] workqueue events_unbound: flags=0x2 [17317.905977] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17317.908705] pending: flush_memcg_stats_dwork, idle_cull_fn [17317.911893] workqueue events_power_efficient: flags=0x80 [17317.914905] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17317.918708] pending: check_lifetime, do_cache_clean [17317.921536] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17317.925861] pending: neigh_periodic_work, neigh_periodic_work [17317.928990] workqueue mm_percpu_wq: flags=0x8 [17317.931027] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17317.935737] pending: vmstat_update, lru_add_drain_per_cpu [17317.938750] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17317.942637] pending: vmstat_update, lru_add_drain_per_cpu [17317.946156] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17317.950165] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17317.953753] workqueue xprtiod: flags=0xa [17317.955719] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17317.958936] pending: xprt_autoclose, xs_error_handle [17317.962095] workqueue ipv6_addrconf: flags=0x40008 [17317.964686] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17317.967556] pending: addrconf_verify_work [17317.969137] workqueue knbd0-recv: flags=0x1a [17317.971290] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17317.975515] in-flight: 651:recv_work [17317.977767] workqueue ptlrpc_pinger: flags=0x2 [17317.980156] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17317.983173] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17317.985765] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14680s workers=4 idle: 15979 31824 8 [17317.989892] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17308s workers=2 idle: 654 [17328.055030] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [17328.057057] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [17328.062943] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17328.063022] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17328.063030] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17328.063034] RIP: 0010:native_safe_halt+0x2e/0x50 [17328.063047] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17328.063052] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17328.063057] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17328.063060] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17328.063063] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17328.063066] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17328.063068] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17328.063071] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17328.063074] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17328.063076] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17328.063084] Call Trace: [17328.063091] [17328.064025] ? show_regs.cold.9+0x22/0x2f [17328.066612] Modules linked in: lustre(O) [17328.091829] ? watchdog_timer_fn.cold.6+0x62/0x10a [17328.091848] ? __hrtimer_run_queues+0x1c2/0x4d0 [17328.091853] ? watchdog+0x40/0x40 [17328.091857] ? hrtimer_interrupt+0x128/0x2c0 [17328.091860] ? smp_apic_timer_interrupt+0xa6/0x270 [17328.091868] ? apic_timer_interrupt+0xf/0x20 [17328.091873] [17328.091877] ? native_safe_halt+0x2e/0x50 [17328.091882] kvm_wait+0xcd/0xe0 [17328.091892] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17328.091901] do_raw_spin_lock+0x177/0x190 [17328.091906] _raw_spin_lock+0x21/0x30 [17328.091911] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17328.092487] ? do_raw_spin_unlock+0x75/0x190 [17328.092493] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17328.092693] ? do_wait_intr+0xf0/0xf0 [17328.092702] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17328.092898] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17328.098539] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17328.098603] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17328.098610] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17328.098613] RIP: 0010:native_safe_halt+0x2e/0x50 [17328.098626] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17328.098629] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17328.098634] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17328.098635] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17328.098637] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17328.098640] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17328.098641] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17328.098644] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17328.098647] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17328.098649] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17328.105507] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17328.105802] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17328.105914] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17328.105953] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17328.105983] mdc_intent_lock+0x353/0x7e0 [mdc] [17328.106010] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17328.106237] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17328.106592] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17328.106623] lmv_intent_open+0x338/0x1150 [lmv] [17328.106654] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17328.106737] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17328.106819] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17328.106842] ll_intent_lock+0xc5/0xaa0 [lustre] [17328.106927] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17328.107008] ll_lookup_it+0xe0e/0x2270 [lustre] [17328.107235] ? do_raw_spin_unlock+0x75/0x190 [17328.107244] ? _raw_spin_unlock+0x12/0x30 [17328.107251] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17328.107278] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17328.107299] ll_atomic_open+0x36a/0xd70 [lustre] [17328.107393] lookup_open+0x356/0xb40 [17328.107521] do_last+0x313/0xfc0 [17328.107527] ? nd_jump_root+0xe5/0x160 [17328.107532] ? path_init+0x437/0x520 [17328.107536] path_openat+0xf7/0x500 [17328.107541] ? do_raw_spin_unlock+0x75/0x190 [17328.107546] do_filp_open+0x99/0x140 [17328.107552] ? getname_flags+0x6e/0x330 [17328.107556] ? __check_object_size+0xff/0x256 [17328.107562] ? do_raw_spin_unlock+0x75/0x190 [17328.107565] ? _raw_spin_unlock+0x12/0x30 [17328.107570] do_sys_openat2+0x2b4/0x410 [17328.107575] do_sys_open+0x73/0xa0 [17328.107579] __x64_sys_openat+0x24/0x30 [17328.107582] do_syscall_64+0xc1/0x440 [17328.107588] entry_SYSCALL_64_after_hwframe+0x49/0xae [17328.107594] RIP: 0033:0x7f53425a5332 [17328.107602] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17328.107606] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17328.107612] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17328.107615] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17328.107617] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17328.107619] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17328.107621] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17328.553347] Call Trace: [17328.554379] [17328.555114] ? show_regs.cold.9+0x22/0x2f [17328.556677] ? watchdog_timer_fn.cold.6+0x62/0x10a [17328.558116] ? __hrtimer_run_queues+0x1c2/0x4d0 [17328.559147] ? watchdog+0x40/0x40 [17328.560112] ? hrtimer_interrupt+0x128/0x2c0 [17328.561365] ? smp_apic_timer_interrupt+0xa6/0x270 [17328.562987] ? apic_timer_interrupt+0xf/0x20 [17328.564780] [17328.565435] ? native_safe_halt+0x2e/0x50 [17328.566444] kvm_wait+0xcd/0xe0 [17328.567613] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17328.569695] do_raw_spin_lock+0x177/0x190 [17328.571239] _raw_spin_lock+0x21/0x30 [17328.572577] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17328.574383] obd_mod_free+0x46/0x140 [ptlrpc] [17328.575725] mdc_commit_open+0x91/0xb0 [mdc] [17328.576806] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17328.578769] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17328.580450] after_reply+0xe2c/0x1780 [ptlrpc] [17328.582119] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17328.583620] ? __next_timer_interrupt+0x160/0x160 [17328.584790] ptlrpcd+0xa82/0xe40 [ptlrpc] [17328.586200] ? do_wait_intr+0xf0/0xf0 [17328.587161] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17328.588644] kthread+0x1d1/0x200 [17328.589475] ? set_kthread_struct+0x70/0x70 [17328.590455] ret_from_fork+0x1f/0x30 [17344.052139] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [17344.055127] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17344.058020] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [17344.077212] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17344.077227] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17344.077237] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17344.077535] RIP: 0010:native_safe_halt+0x2e/0x50 [17344.077546] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17344.077550] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17344.077554] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17344.077557] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [17344.077559] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17344.077562] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17344.077564] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17344.077567] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17344.077570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17344.077572] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17344.077580] Call Trace: [17344.077586] [17344.077591] ? show_regs.cold.9+0x22/0x2f [17344.077600] ? watchdog_timer_fn.cold.6+0x62/0x10a [17344.077607] ? __hrtimer_run_queues+0x1c2/0x4d0 [17344.077611] ? watchdog+0x40/0x40 [17344.077616] ? hrtimer_interrupt+0x128/0x2c0 [17344.077620] ? smp_apic_timer_interrupt+0xa6/0x270 [17344.077627] ? apic_timer_interrupt+0xf/0x20 [17344.077632] [17344.077635] ? native_safe_halt+0x2e/0x50 [17344.077640] kvm_wait+0xcd/0xe0 [17344.077646] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17344.077653] ? newidle_balance.isra.93+0x207/0x580 [17344.077659] do_raw_spin_lock+0x177/0x190 [17344.077663] _raw_spin_lock+0x21/0x30 [17344.077667] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17344.078363] process_one_work+0x2c8/0x700 [17344.078370] worker_thread+0x296/0x6e0 [17344.078375] ? rescuer_thread+0x570/0x570 [17344.078378] kthread+0x1d1/0x200 [17344.078383] ? set_kthread_struct+0x70/0x70 [17344.078386] ret_from_fork+0x1f/0x30 [17344.208684] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17344.244625] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17344.254742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17344.258988] RIP: 0010:native_safe_halt+0x2e/0x50 [17344.262099] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17344.272338] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17344.276123] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17344.277833] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17344.279556] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17344.283967] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17344.286713] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17344.289240] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17344.291934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17344.295132] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17344.296936] Call Trace: [17344.297511] [17344.298385] ? show_regs.cold.9+0x22/0x2f [17344.299502] ? watchdog_timer_fn.cold.6+0x62/0x10a [17344.301983] ? __hrtimer_run_queues+0x1c2/0x4d0 [17344.304389] ? watchdog+0x40/0x40 [17344.305248] ? hrtimer_interrupt+0x128/0x2c0 [17344.308784] ? smp_apic_timer_interrupt+0xa6/0x270 [17344.310367] ? apic_timer_interrupt+0xf/0x20 [17344.313678] [17344.314338] ? native_safe_halt+0x2e/0x50 [17344.316665] kvm_wait+0xcd/0xe0 [17344.317485] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17344.318823] do_raw_spin_lock+0x177/0x190 [17344.320038] _raw_spin_lock+0x21/0x30 [17344.321699] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17344.323612] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17344.326441] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17344.328256] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17344.331096] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17344.332603] ? woken_wake_function+0x30/0x30 [17344.333636] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17344.338852] kthread+0x1d1/0x200 [17344.339589] ? set_kthread_struct+0x70/0x70 [17344.340578] ret_from_fork+0x1f/0x30 [17348.064657] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14710s! [17348.072995] Showing busy workqueues and worker pools: [17348.074866] workqueue events: flags=0x0 [17348.075932] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17348.079598] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17348.085213] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17348.089480] pending: cpu_latency_work [ptlrpc] [17348.095886] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17348.100834] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17348.112068] workqueue events_long: flags=0x0 [17348.113560] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17348.121528] pending: lu_objects_flush [obdclass] [17348.124727] workqueue events_unbound: flags=0x2 [17348.127619] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17348.130434] pending: flush_memcg_stats_dwork, idle_cull_fn [17348.132535] workqueue events_power_efficient: flags=0x80 [17348.134552] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17348.137186] pending: check_lifetime, do_cache_clean [17348.141094] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17348.143514] pending: neigh_periodic_work, neigh_periodic_work [17348.146821] workqueue mm_percpu_wq: flags=0x8 [17348.148813] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17348.151872] pending: vmstat_update, lru_add_drain_per_cpu [17348.154394] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17348.157151] pending: vmstat_update, lru_add_drain_per_cpu [17348.159623] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17348.161773] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17348.164289] workqueue xprtiod: flags=0xa [17348.166241] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17348.168941] pending: xprt_autoclose, xs_error_handle [17348.171507] workqueue ipv6_addrconf: flags=0x40008 [17348.173529] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17348.176542] pending: addrconf_verify_work [17348.178588] workqueue knbd0-recv: flags=0x1a [17348.180463] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17348.183832] in-flight: 651:recv_work [17348.185638] workqueue ptlrpc_pinger: flags=0x2 [17348.187171] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17348.190570] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17348.192411] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14711s workers=4 idle: 15979 31824 8 [17348.195978] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17338s workers=2 idle: 654 [17356.058714] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [17356.059228] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [17356.062745] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17356.062895] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17356.062902] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17356.062905] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17356.062921] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17356.062925] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17356.062930] RAX: 000000000000098c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17356.062932] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17356.062935] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17356.062937] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17356.062939] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17356.062942] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17356.062944] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17356.062947] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17356.062956] Call Trace: [17356.062962] [17356.062967] ? show_regs.cold.9+0x22/0x2f [17356.062977] ? watchdog_timer_fn.cold.6+0x62/0x10a [17356.062983] ? __hrtimer_run_queues+0x1c2/0x4d0 [17356.062986] ? watchdog+0x40/0x40 [17356.062991] ? hrtimer_interrupt+0x128/0x2c0 [17356.062994] ? smp_apic_timer_interrupt+0xa6/0x270 [17356.063002] ? apic_timer_interrupt+0xf/0x20 [17356.063006] [17356.063009] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17356.063015] do_raw_spin_lock+0x177/0x190 [17356.063019] _raw_spin_lock+0x21/0x30 [17356.063024] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17356.063284] ? do_raw_spin_unlock+0x75/0x190 [17356.063290] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17356.063476] ? do_wait_intr+0xf0/0xf0 [17356.063483] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17356.065502] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17356.066092] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17356.070907] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17356.070988] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17356.070996] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17356.071000] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17356.071016] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17356.071020] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17356.071024] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17356.071027] RDX: 0000000000006a8c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17356.071030] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17356.071032] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17356.071035] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17356.071039] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17356.071041] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17356.071044] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17356.071054] Call Trace: [17356.071059] [17356.071065] ? show_regs.cold.9+0x22/0x2f [17356.071073] ? watchdog_timer_fn.cold.6+0x62/0x10a [17356.071080] ? __hrtimer_run_queues+0x1c2/0x4d0 [17356.071084] ? watchdog+0x40/0x40 [17356.071095] ? hrtimer_interrupt+0x128/0x2c0 [17356.071099] ? smp_apic_timer_interrupt+0xa6/0x270 [17356.071107] ? apic_timer_interrupt+0xf/0x20 [17356.071112] [17356.071115] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17356.071120] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17356.071125] do_raw_spin_lock+0x177/0x190 [17356.071129] _raw_spin_lock+0x21/0x30 [17356.071135] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17356.098273] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17356.098399] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17356.098439] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17356.098473] mdc_intent_lock+0x353/0x7e0 [mdc] [17356.098504] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17356.098807] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17356.099135] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17356.099167] lmv_intent_open+0x338/0x1150 [lmv] [17356.099199] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17356.099286] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17356.099367] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17356.099390] ll_intent_lock+0xc5/0xaa0 [lustre] [17356.099474] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17356.099926] ll_lookup_it+0xe0e/0x2270 [lustre] [17356.100008] ? do_raw_spin_unlock+0x75/0x190 [17356.100017] ? _raw_spin_unlock+0x12/0x30 [17356.100023] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17356.100052] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17356.100070] ll_atomic_open+0x36a/0xd70 [lustre] [17356.100154] lookup_open+0x356/0xb40 [17356.100165] do_last+0x313/0xfc0 [17356.100170] ? nd_jump_root+0xe5/0x160 [17356.100175] ? path_init+0x437/0x520 [17356.100179] path_openat+0xf7/0x500 [17356.100184] ? do_raw_spin_unlock+0x75/0x190 [17356.100189] do_filp_open+0x99/0x140 [17356.100195] ? getname_flags+0x6e/0x330 [17356.100199] ? __check_object_size+0xff/0x256 [17356.100207] ? do_raw_spin_unlock+0x75/0x190 [17356.100211] ? _raw_spin_unlock+0x12/0x30 [17356.100215] do_sys_openat2+0x2b4/0x410 [17356.100220] do_sys_open+0x73/0xa0 [17356.100224] __x64_sys_openat+0x24/0x30 [17356.100228] do_syscall_64+0xc1/0x440 [17356.100235] entry_SYSCALL_64_after_hwframe+0x49/0xae [17356.100241] RIP: 0033:0x7f53425a5332 [17356.100249] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17356.100253] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17356.100258] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17356.100261] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17356.100263] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17356.100265] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17356.100268] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17356.578898] obd_mod_free+0x46/0x140 [ptlrpc] [17356.583264] mdc_commit_open+0x91/0xb0 [mdc] [17356.587499] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17356.590507] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17356.593979] after_reply+0xe2c/0x1780 [ptlrpc] [17356.596005] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17356.599868] ? __next_timer_interrupt+0x160/0x160 [17356.604066] ptlrpcd+0xa82/0xe40 [ptlrpc] [17356.606401] ? do_wait_intr+0xf0/0xf0 [17356.609075] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17356.612084] kthread+0x1d1/0x200 [17356.614006] ? set_kthread_struct+0x70/0x70 [17356.616459] ret_from_fork+0x1f/0x30 [17372.061271] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [17372.066573] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17372.082144] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17372.086118] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17372.089727] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17372.091452] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17372.098448] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17372.101383] RAX: 000000000000530c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17372.104135] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17372.107072] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17372.109449] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17372.111753] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17372.114428] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17372.117777] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17372.119812] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17372.122493] Call Trace: [17372.123603] [17372.124221] ? show_regs.cold.9+0x22/0x2f [17372.125556] ? watchdog_timer_fn.cold.6+0x62/0x10a [17372.127148] ? __hrtimer_run_queues+0x1c2/0x4d0 [17372.129089] ? watchdog+0x40/0x40 [17372.130623] ? hrtimer_interrupt+0x128/0x2c0 [17372.133332] ? smp_apic_timer_interrupt+0xa6/0x270 [17372.135484] ? apic_timer_interrupt+0xf/0x20 [17372.137359] [17372.138206] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17372.140564] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17372.143010] do_raw_spin_lock+0x177/0x190 [17372.144742] _raw_spin_lock+0x21/0x30 [17372.146366] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17372.149285] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17372.151707] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17372.156747] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17372.159926] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17372.163088] ? woken_wake_function+0x30/0x30 [17372.165108] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17372.169099] kthread+0x1d1/0x200 [17372.170918] ? set_kthread_struct+0x70/0x70 [17372.173360] ret_from_fork+0x1f/0x30 [17376.052021] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [17376.054227] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17376.076760] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17376.082478] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17376.087239] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17376.093453] RIP: 0010:native_safe_halt+0x2e/0x50 [17376.096756] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17376.115763] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17376.125546] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17376.130754] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [17376.136134] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17376.140776] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17376.146632] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17376.150186] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17376.154840] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17376.157816] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17376.161855] Call Trace: [17376.163384] [17376.164370] ? show_regs.cold.9+0x22/0x2f [17376.165868] ? watchdog_timer_fn.cold.6+0x62/0x10a [17376.167763] ? __hrtimer_run_queues+0x1c2/0x4d0 [17376.169711] ? watchdog+0x40/0x40 [17376.171050] ? hrtimer_interrupt+0x128/0x2c0 [17376.172770] ? smp_apic_timer_interrupt+0xa6/0x270 [17376.175416] ? apic_timer_interrupt+0xf/0x20 [17376.177380] [17376.178124] ? native_safe_halt+0x2e/0x50 [17376.180860] kvm_wait+0xcd/0xe0 [17376.184346] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17376.186068] ? newidle_balance.isra.93+0x207/0x580 [17376.189694] do_raw_spin_lock+0x177/0x190 [17376.191377] _raw_spin_lock+0x21/0x30 [17376.192979] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17376.197385] process_one_work+0x2c8/0x700 [17376.199238] worker_thread+0x296/0x6e0 [17376.200767] ? rescuer_thread+0x570/0x570 [17376.202662] kthread+0x1d1/0x200 [17376.204257] ? set_kthread_struct+0x70/0x70 [17376.206201] ret_from_fork+0x1f/0x30 [17378.272048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14741s! [17378.276687] Showing busy workqueues and worker pools: [17378.279146] workqueue events: flags=0x0 [17378.286504] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17378.290245] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17378.300218] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17378.304689] pending: cpu_latency_work [ptlrpc] [17378.308131] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17378.313415] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17378.318157] workqueue events_long: flags=0x0 [17378.320443] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17378.323041] pending: lu_objects_flush [obdclass] [17378.325556] workqueue events_unbound: flags=0x2 [17378.327929] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17378.331073] pending: flush_memcg_stats_dwork, idle_cull_fn [17378.333944] workqueue events_power_efficient: flags=0x80 [17378.336498] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17378.341514] pending: check_lifetime, do_cache_clean [17378.344302] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17378.347739] pending: neigh_periodic_work, neigh_periodic_work [17378.351571] workqueue mm_percpu_wq: flags=0x8 [17378.353790] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17378.357100] pending: vmstat_update, lru_add_drain_per_cpu [17378.359573] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17378.361786] pending: vmstat_update, lru_add_drain_per_cpu [17378.364653] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17378.367846] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17378.371419] workqueue xprtiod: flags=0xa [17378.372898] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17378.375012] pending: xprt_autoclose, xs_error_handle [17378.377735] workqueue ipv6_addrconf: flags=0x40008 [17378.379655] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17378.382431] pending: addrconf_verify_work [17378.384808] workqueue knbd0-recv: flags=0x1a [17378.386451] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17378.388935] in-flight: 651:recv_work [17378.390562] workqueue ptlrpc_pinger: flags=0x2 [17378.391910] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17378.394934] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17378.397857] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14741s workers=4 idle: 15979 31824 8 [17378.401119] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17368s workers=2 idle: 654 [17384.056637] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [17384.057663] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [17384.065351] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17384.065434] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17384.065443] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17384.065446] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17384.065459] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17384.065464] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17384.065469] RAX: 0000000000006c83 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17384.065472] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17384.065474] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17384.065476] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17384.065479] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17384.065483] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17384.065486] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17384.065488] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17384.065497] Call Trace: [17384.065502] [17384.065508] ? show_regs.cold.9+0x22/0x2f [17384.065517] ? watchdog_timer_fn.cold.6+0x62/0x10a [17384.065523] ? __hrtimer_run_queues+0x1c2/0x4d0 [17384.065528] ? watchdog+0x40/0x40 [17384.065531] ? hrtimer_interrupt+0x128/0x2c0 [17384.065535] ? smp_apic_timer_interrupt+0xa6/0x270 [17384.065543] ? apic_timer_interrupt+0xf/0x20 [17384.065547] [17384.065550] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17384.065556] do_raw_spin_lock+0x177/0x190 [17384.065561] _raw_spin_lock+0x21/0x30 [17384.065567] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17384.065849] ? do_raw_spin_unlock+0x75/0x190 [17384.065855] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17384.070022] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17384.070284] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17384.070291] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17384.070294] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17384.070307] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17384.070311] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17384.070315] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17384.070318] RDX: 0000000000003c6c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17384.070320] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17384.070322] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17384.070324] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17384.070391] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17384.070393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17384.070396] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17384.070403] Call Trace: [17384.070409] [17384.070415] ? show_regs.cold.9+0x22/0x2f [17384.070425] ? watchdog_timer_fn.cold.6+0x62/0x10a [17384.070431] ? __hrtimer_run_queues+0x1c2/0x4d0 [17384.070434] ? watchdog+0x40/0x40 [17384.070438] ? hrtimer_interrupt+0x128/0x2c0 [17384.070441] ? smp_apic_timer_interrupt+0xa6/0x270 [17384.070449] ? apic_timer_interrupt+0xf/0x20 [17384.070453] [17384.070455] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17384.070459] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17384.070464] do_raw_spin_lock+0x177/0x190 [17384.070468] _raw_spin_lock+0x21/0x30 [17384.070473] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17384.070767] obd_mod_free+0x46/0x140 [ptlrpc] [17384.070958] mdc_commit_open+0x91/0xb0 [mdc] [17384.070996] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17384.117862] ? do_wait_intr+0xf0/0xf0 [17384.117884] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17384.118706] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17384.118930] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17384.119695] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17384.119808] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17384.119846] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17384.119871] mdc_intent_lock+0x353/0x7e0 [mdc] [17384.119894] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17384.119970] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17384.129828] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17384.134336] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17384.134401] lmv_intent_open+0x338/0x1150 [lmv] [17384.134435] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17384.134546] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17384.134636] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17384.134658] ll_intent_lock+0xc5/0xaa0 [lustre] [17384.134738] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17384.134828] ll_lookup_it+0xe0e/0x2270 [lustre] [17384.134915] ? do_raw_spin_unlock+0x75/0x190 [17384.134923] ? _raw_spin_unlock+0x12/0x30 [17384.134936] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17384.134969] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17384.134993] ll_atomic_open+0x36a/0xd70 [lustre] [17384.135094] lookup_open+0x356/0xb40 [17384.135103] do_last+0x313/0xfc0 [17384.135109] ? nd_jump_root+0xe5/0x160 [17384.135119] ? path_init+0x437/0x520 [17384.135124] path_openat+0xf7/0x500 [17384.135130] ? do_raw_spin_unlock+0x75/0x190 [17384.135135] do_filp_open+0x99/0x140 [17384.135141] ? getname_flags+0x6e/0x330 [17384.135146] ? __check_object_size+0xff/0x256 [17384.135154] ? do_raw_spin_unlock+0x75/0x190 [17384.135158] ? _raw_spin_unlock+0x12/0x30 [17384.135166] do_sys_openat2+0x2b4/0x410 [17384.135171] do_sys_open+0x73/0xa0 [17384.135175] __x64_sys_openat+0x24/0x30 [17384.135178] do_syscall_64+0xc1/0x440 [17384.135185] entry_SYSCALL_64_after_hwframe+0x49/0xae [17384.135194] RIP: 0033:0x7f53425a5332 [17384.135201] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17384.135206] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17384.135211] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17384.135213] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17384.135215] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17384.135217] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17384.135219] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17384.440497] after_reply+0xe2c/0x1780 [ptlrpc] [17384.444412] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17384.447377] ? __next_timer_interrupt+0x160/0x160 [17384.453528] ptlrpcd+0xa82/0xe40 [ptlrpc] [17384.457229] ? do_wait_intr+0xf0/0xf0 [17384.459222] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17384.466837] kthread+0x1d1/0x200 [17384.468530] ? set_kthread_struct+0x70/0x70 [17384.475540] ret_from_fork+0x1f/0x30 [17400.058717] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [17400.063495] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17400.095514] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17400.102551] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17400.105722] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17400.110472] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17400.123622] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17400.128720] RAX: 0000000000007f96 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17400.131317] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17400.141136] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17400.145246] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17400.150075] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17400.154201] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17400.162219] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17400.172016] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17400.177694] Call Trace: [17400.180596] [17400.181617] ? show_regs.cold.9+0x22/0x2f [17400.182891] ? watchdog_timer_fn.cold.6+0x62/0x10a [17400.185139] ? __hrtimer_run_queues+0x1c2/0x4d0 [17400.186520] ? watchdog+0x40/0x40 [17400.187621] ? hrtimer_interrupt+0x128/0x2c0 [17400.188909] ? smp_apic_timer_interrupt+0xa6/0x270 [17400.190469] ? apic_timer_interrupt+0xf/0x20 [17400.191739] [17400.193544] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17400.195658] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17400.198245] do_raw_spin_lock+0x177/0x190 [17400.199554] _raw_spin_lock+0x21/0x30 [17400.200695] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17400.203572] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17400.205859] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17400.211773] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17400.217545] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17400.219018] ? woken_wake_function+0x30/0x30 [17400.221905] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17400.224420] kthread+0x1d1/0x200 [17400.226423] ? set_kthread_struct+0x70/0x70 [17400.235987] ret_from_fork+0x1f/0x30 [17404.054541] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [17404.066669] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17404.113220] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17404.122226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17404.127266] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17404.132414] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17404.136621] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17404.152962] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17404.159570] RAX: 0000000000007c86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17404.172516] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17404.177776] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17404.182784] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17404.189668] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17404.197326] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17404.201250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17404.203802] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17404.207171] Call Trace: [17404.208351] [17404.211766] ? show_regs.cold.9+0x22/0x2f [17404.213700] ? watchdog_timer_fn.cold.6+0x62/0x10a [17404.216384] ? __hrtimer_run_queues+0x1c2/0x4d0 [17404.218339] ? watchdog+0x40/0x40 [17404.219935] ? hrtimer_interrupt+0x128/0x2c0 [17404.221673] ? smp_apic_timer_interrupt+0xa6/0x270 [17404.226346] ? apic_timer_interrupt+0xf/0x20 [17404.231959] [17404.233151] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17404.238791] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17404.241368] ? newidle_balance.isra.93+0x207/0x580 [17404.247005] do_raw_spin_lock+0x177/0x190 [17404.249129] _raw_spin_lock+0x21/0x30 [17404.250382] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17404.253100] process_one_work+0x2c8/0x700 [17404.254864] worker_thread+0x296/0x6e0 [17404.258313] ? rescuer_thread+0x570/0x570 [17404.260760] kthread+0x1d1/0x200 [17404.264861] ? set_kthread_struct+0x70/0x70 [17404.269564] ret_from_fork+0x1f/0x30 [17408.480088] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14771s! [17408.497058] Showing busy workqueues and worker pools: [17408.498904] workqueue events: flags=0x0 [17408.500118] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17408.512361] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17408.524322] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17408.534264] pending: cpu_latency_work [ptlrpc] [17408.539192] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17408.545767] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17408.556423] workqueue events_long: flags=0x0 [17408.561152] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17408.568423] pending: lu_objects_flush [obdclass] [17408.574145] workqueue events_unbound: flags=0x2 [17408.579387] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17408.587505] pending: flush_memcg_stats_dwork, idle_cull_fn [17408.599751] workqueue events_power_efficient: flags=0x80 [17408.605423] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17408.615366] pending: check_lifetime, do_cache_clean [17408.620436] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17408.624512] pending: neigh_periodic_work, neigh_periodic_work [17408.632277] workqueue mm_percpu_wq: flags=0x8 [17408.638219] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17408.647566] pending: vmstat_update, lru_add_drain_per_cpu [17408.654335] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17408.662983] pending: vmstat_update, lru_add_drain_per_cpu [17408.666234] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17408.679596] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17408.692251] workqueue xprtiod: flags=0xa [17408.694498] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17408.698530] pending: xprt_autoclose, xs_error_handle [17408.712182] workqueue ipv6_addrconf: flags=0x40008 [17408.717075] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17408.724174] pending: addrconf_verify_work [17408.727259] workqueue knbd0-recv: flags=0x1a [17408.731396] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17408.736747] in-flight: 651:recv_work [17408.740544] workqueue ptlrpc_pinger: flags=0x2 [17408.743920] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17408.752443] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17408.758288] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14771s workers=4 idle: 15979 31824 8 [17408.768640] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17399s workers=2 idle: 654 [17412.055595] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [17412.057014] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [17412.059279] Modules linked in: lustre(O) [17412.067845] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17412.067923] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17412.067931] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17412.067934] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17412.069884] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17412.069952] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17412.069959] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17412.069962] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17412.069975] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17412.069979] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17412.069983] RAX: 0000000000007d83 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17412.069986] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17412.069988] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17412.069990] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17412.069992] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17412.069995] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17412.069998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17412.070001] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17412.070010] Call Trace: [17412.070016] [17412.070023] ? show_regs.cold.9+0x22/0x2f [17412.070034] ? watchdog_timer_fn.cold.6+0x62/0x10a [17412.070041] ? __hrtimer_run_queues+0x1c2/0x4d0 [17412.070045] ? watchdog+0x40/0x40 [17412.070050] ? hrtimer_interrupt+0x128/0x2c0 [17412.070055] ? smp_apic_timer_interrupt+0xa6/0x270 [17412.070064] ? apic_timer_interrupt+0xf/0x20 [17412.070068] [17412.070071] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17412.070078] do_raw_spin_lock+0x177/0x190 [17412.070083] _raw_spin_lock+0x21/0x30 [17412.070087] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17412.070347] ? do_raw_spin_unlock+0x75/0x190 [17412.070353] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17412.071023] ? do_wait_intr+0xf0/0xf0 [17412.071032] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17412.071722] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17412.072150] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17412.072422] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17412.073044] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17412.073079] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17412.073106] mdc_intent_lock+0x353/0x7e0 [mdc] [17412.073133] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17412.073215] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17412.073438] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17412.073466] lmv_intent_open+0x338/0x1150 [lmv] [17412.073494] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17412.073702] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17412.073783] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17412.073872] ll_intent_lock+0xc5/0xaa0 [lustre] [17412.073954] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17412.074029] ll_lookup_it+0xe0e/0x2270 [lustre] [17412.074116] ? do_raw_spin_unlock+0x75/0x190 [17412.074124] ? _raw_spin_unlock+0x12/0x30 [17412.074131] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17412.074159] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17412.074179] ll_atomic_open+0x36a/0xd70 [lustre] [17412.074262] lookup_open+0x356/0xb40 [17412.074271] do_last+0x313/0xfc0 [17412.074276] ? nd_jump_root+0xe5/0x160 [17412.074280] ? path_init+0x437/0x520 [17412.074284] path_openat+0xf7/0x500 [17412.074290] ? do_raw_spin_unlock+0x75/0x190 [17412.074294] do_filp_open+0x99/0x140 [17412.074300] ? getname_flags+0x6e/0x330 [17412.074305] ? __check_object_size+0xff/0x256 [17412.074313] ? do_raw_spin_unlock+0x75/0x190 [17412.074317] ? _raw_spin_unlock+0x12/0x30 [17412.074321] do_sys_openat2+0x2b4/0x410 [17412.074326] do_sys_open+0x73/0xa0 [17412.074330] __x64_sys_openat+0x24/0x30 [17412.074334] do_syscall_64+0xc1/0x440 [17412.074341] entry_SYSCALL_64_after_hwframe+0x49/0xae [17412.074345] RIP: 0033:0x7f53425a5332 [17412.074351] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17412.074354] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17412.074357] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17412.074360] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17412.074362] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17412.074364] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17412.074367] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17412.579792] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17412.587689] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17412.591047] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17412.594435] RDX: 0000000000003621 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17412.597509] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17412.600697] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17412.603916] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17412.608031] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17412.614033] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17412.619251] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17412.623440] Call Trace: [17412.625385] [17412.626853] ? show_regs.cold.9+0x22/0x2f [17412.630884] ? watchdog_timer_fn.cold.6+0x62/0x10a [17412.636649] ? __hrtimer_run_queues+0x1c2/0x4d0 [17412.648411] ? watchdog+0x40/0x40 [17412.649554] ? hrtimer_interrupt+0x128/0x2c0 [17412.651048] ? smp_apic_timer_interrupt+0xa6/0x270 [17412.671922] ? apic_timer_interrupt+0xf/0x20 [17412.678661] [17412.682140] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17412.690544] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17412.702183] do_raw_spin_lock+0x177/0x190 [17412.709866] _raw_spin_lock+0x21/0x30 [17412.719981] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17412.728098] obd_mod_free+0x46/0x140 [ptlrpc] [17412.736878] mdc_commit_open+0x91/0xb0 [mdc] [17412.742819] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17412.760298] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17412.761917] after_reply+0xe2c/0x1780 [ptlrpc] [17412.771417] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17412.775526] ? __next_timer_interrupt+0x160/0x160 [17412.780241] ptlrpcd+0xa82/0xe40 [ptlrpc] [17412.793071] ? do_wait_intr+0xf0/0xf0 [17412.794285] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17412.803356] kthread+0x1d1/0x200 [17412.806260] ? set_kthread_struct+0x70/0x70 [17412.808430] ret_from_fork+0x1f/0x30 [17428.061444] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [17428.072129] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17428.133141] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17428.145488] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17428.158103] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17428.169134] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17428.188036] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17428.191896] RAX: 0000000000004185 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17428.197779] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17428.200193] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17428.212094] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17428.219833] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17428.224590] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17428.229559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17428.233059] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17428.239210] Call Trace: [17428.243290] [17428.252210] ? show_regs.cold.9+0x22/0x2f [17428.258165] ? watchdog_timer_fn.cold.6+0x62/0x10a [17428.263458] ? __hrtimer_run_queues+0x1c2/0x4d0 [17428.267047] ? watchdog+0x40/0x40 [17428.269831] ? hrtimer_interrupt+0x128/0x2c0 [17428.275639] ? smp_apic_timer_interrupt+0xa6/0x270 [17428.282147] ? apic_timer_interrupt+0xf/0x20 [17428.286991] [17428.287681] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17428.296647] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17428.298487] do_raw_spin_lock+0x177/0x190 [17428.303867] _raw_spin_lock+0x21/0x30 [17428.312037] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17428.320208] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17428.326774] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17428.333093] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17428.337212] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17428.342855] ? woken_wake_function+0x30/0x30 [17428.346716] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17428.354961] kthread+0x1d1/0x200 [17428.357178] ? set_kthread_struct+0x70/0x70 [17428.363945] ret_from_fork+0x1f/0x30 [17436.052030] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [17436.057666] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17436.081831] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17436.088714] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17436.092466] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17436.095102] RIP: 0010:native_safe_halt+0x2e/0x50 [17436.096485] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17436.102108] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17436.105424] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17436.108055] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [17436.110882] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17436.113724] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17436.116037] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17436.118648] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17436.122076] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17436.124795] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17436.128412] Call Trace: [17436.129504] [17436.130603] ? show_regs.cold.9+0x22/0x2f [17436.132385] ? watchdog_timer_fn.cold.6+0x62/0x10a [17436.134505] ? __hrtimer_run_queues+0x1c2/0x4d0 [17436.136303] ? watchdog+0x40/0x40 [17436.137748] ? hrtimer_interrupt+0x128/0x2c0 [17436.139604] ? smp_apic_timer_interrupt+0xa6/0x270 [17436.141749] ? apic_timer_interrupt+0xf/0x20 [17436.143903] [17436.144880] ? native_safe_halt+0x2e/0x50 [17436.146756] kvm_wait+0xcd/0xe0 [17436.148545] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17436.151034] ? newidle_balance.isra.93+0x207/0x580 [17436.153357] do_raw_spin_lock+0x177/0x190 [17436.156075] _raw_spin_lock+0x21/0x30 [17436.159108] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17436.161482] process_one_work+0x2c8/0x700 [17436.163435] worker_thread+0x296/0x6e0 [17436.164964] ? rescuer_thread+0x570/0x570 [17436.166866] kthread+0x1d1/0x200 [17436.168392] ? set_kthread_struct+0x70/0x70 [17436.170392] ret_from_fork+0x1f/0x30 [17439.201042] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14802s! [17439.203538] Showing busy workqueues and worker pools: [17439.205049] workqueue events: flags=0x0 [17439.206376] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17439.208360] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17439.211678] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17439.213588] pending: cpu_latency_work [ptlrpc] [17439.215500] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17439.217560] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17439.220438] workqueue events_long: flags=0x0 [17439.221732] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17439.223817] pending: lu_objects_flush [obdclass] [17439.225373] workqueue events_unbound: flags=0x2 [17439.226965] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17439.228914] pending: flush_memcg_stats_dwork, idle_cull_fn [17439.230591] workqueue events_power_efficient: flags=0x80 [17439.232334] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17439.234155] pending: check_lifetime, do_cache_clean [17439.236195] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17439.238890] pending: neigh_periodic_work, neigh_periodic_work [17439.240723] workqueue mm_percpu_wq: flags=0x8 [17439.241802] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17439.243952] pending: vmstat_update, lru_add_drain_per_cpu [17439.246114] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17439.248243] pending: vmstat_update, lru_add_drain_per_cpu [17439.250202] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17439.252564] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17439.254688] workqueue xprtiod: flags=0xa [17439.256257] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17439.258659] pending: xprt_autoclose, xs_error_handle [17439.260703] workqueue ipv6_addrconf: flags=0x40008 [17439.262246] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17439.264525] pending: addrconf_verify_work [17439.265966] workqueue knbd0-recv: flags=0x1a [17439.267790] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17439.270294] in-flight: 651:recv_work [17439.271472] workqueue ptlrpc_pinger: flags=0x2 [17439.273215] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17439.275532] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17439.277276] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14802s workers=4 idle: 15979 31824 8 [17439.280060] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17429s workers=2 idle: 654 [17440.055021] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [17440.057012] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [17440.057997] Modules linked in: lustre(O) [17440.061863] Modules linked in: [17440.061866] osp(O) ofd(O) [17440.063764] lustre(O) [17440.064892] lod(O) [17440.067829] osp(O) [17440.068748] mdt(O) [17440.069708] ofd(O) [17440.070640] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17440.070714] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17440.070720] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17440.070723] RIP: 0010:native_safe_halt+0x2e/0x50 [17440.070735] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17440.070739] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17440.070743] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17440.070745] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17440.070747] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17440.070749] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17440.070752] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17440.070755] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17440.070757] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17440.070759] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17440.070772] Call Trace: [17440.070778] [17440.070783] ? show_regs.cold.9+0x22/0x2f [17440.070793] ? watchdog_timer_fn.cold.6+0x62/0x10a [17440.070799] ? __hrtimer_run_queues+0x1c2/0x4d0 [17440.070803] ? watchdog+0x40/0x40 [17440.070807] ? hrtimer_interrupt+0x128/0x2c0 [17440.070811] ? smp_apic_timer_interrupt+0xa6/0x270 [17440.070818] ? apic_timer_interrupt+0xf/0x20 [17440.070822] [17440.070825] ? native_safe_halt+0x2e/0x50 [17440.070829] kvm_wait+0xcd/0xe0 [17440.070836] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17440.070845] do_raw_spin_lock+0x177/0x190 [17440.070849] _raw_spin_lock+0x21/0x30 [17440.070853] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17440.071173] ? do_raw_spin_unlock+0x75/0x190 [17440.071178] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17440.071392] ? do_wait_intr+0xf0/0xf0 [17440.071400] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17440.071768] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17440.071988] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17440.072230] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17440.072346] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17440.072386] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17440.072414] mdc_intent_lock+0x353/0x7e0 [mdc] [17440.072442] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17440.072524] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17440.072734] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17440.072761] lmv_intent_open+0x338/0x1150 [lmv] [17440.072789] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17440.072873] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17440.072957] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17440.072979] ll_intent_lock+0xc5/0xaa0 [lustre] [17440.073078] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17440.073163] ll_lookup_it+0xe0e/0x2270 [lustre] [17440.075827] lod(O) [17440.076712] ? do_raw_spin_unlock+0x75/0x190 [17440.121221] mdt(O) [17440.125636] ? _raw_spin_unlock+0x12/0x30 [17440.125656] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17440.125685] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17440.125709] ll_atomic_open+0x36a/0xd70 [lustre] [17440.125829] lookup_open+0x356/0xb40 [17440.125838] do_last+0x313/0xfc0 [17440.125844] ? nd_jump_root+0xe5/0x160 [17440.125848] ? path_init+0x437/0x520 [17440.125853] path_openat+0xf7/0x500 [17440.125858] ? do_raw_spin_unlock+0x75/0x190 [17440.125865] do_filp_open+0x99/0x140 [17440.125872] ? getname_flags+0x6e/0x330 [17440.125877] ? __check_object_size+0xff/0x256 [17440.125884] ? do_raw_spin_unlock+0x75/0x190 [17440.125888] ? _raw_spin_unlock+0x12/0x30 [17440.125893] do_sys_openat2+0x2b4/0x410 [17440.125898] do_sys_open+0x73/0xa0 [17440.125902] __x64_sys_openat+0x24/0x30 [17440.125906] do_syscall_64+0xc1/0x440 [17440.125914] entry_SYSCALL_64_after_hwframe+0x49/0xae [17440.125920] RIP: 0033:0x7f53425a5332 [17440.125928] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17440.125932] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17440.125937] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17440.125939] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17440.125941] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17440.125944] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17440.125946] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17440.445636] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17440.463860] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17440.469562] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17440.473662] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17440.476750] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17440.485697] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17440.488823] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17440.492923] RDX: 00000000000008c2 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17440.496172] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17440.498821] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17440.501737] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17440.508988] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17440.513492] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17440.516007] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17440.521379] Call Trace: [17440.522767] [17440.524268] ? show_regs.cold.9+0x22/0x2f [17440.527794] ? watchdog_timer_fn.cold.6+0x62/0x10a [17440.531701] ? __hrtimer_run_queues+0x1c2/0x4d0 [17440.534605] ? watchdog+0x40/0x40 [17440.536814] ? hrtimer_interrupt+0x128/0x2c0 [17440.540855] ? smp_apic_timer_interrupt+0xa6/0x270 [17440.546078] ? apic_timer_interrupt+0xf/0x20 [17440.549195] [17440.550057] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17440.553273] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17440.558344] do_raw_spin_lock+0x177/0x190 [17440.560590] _raw_spin_lock+0x21/0x30 [17440.563313] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17440.569777] obd_mod_free+0x46/0x140 [ptlrpc] [17440.575942] mdc_commit_open+0x91/0xb0 [mdc] [17440.578556] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17440.581458] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17440.583030] after_reply+0xe2c/0x1780 [ptlrpc] [17440.587041] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17440.590547] ? __next_timer_interrupt+0x160/0x160 [17440.593466] ptlrpcd+0xa82/0xe40 [ptlrpc] [17440.599982] ? do_wait_intr+0xf0/0xf0 [17440.606648] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17440.616111] kthread+0x1d1/0x200 [17440.620531] ? set_kthread_struct+0x70/0x70 [17440.628917] ret_from_fork+0x1f/0x30 [17454.804238] rcu: INFO: rcu_sched self-detected stall on CPU [17454.811386] rcu: 2-....: (8606206 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [17454.820522] (t=15360378 jiffies g=274185 q=67903) [17454.823422] rcu: rcu_sched kthread starved for 14818744 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [17454.828347] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [17454.835605] rcu: RCU grace-period kthread stack dump: [17454.839963] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [17454.849621] Call Trace: [17454.851552] __schedule+0x351/0xcb0 [17454.854995] schedule+0xc0/0x180 [17454.856025] schedule_timeout+0xb4/0x190 [17454.857550] ? do_raw_spin_unlock+0x75/0x190 [17454.863595] ? __next_timer_interrupt+0x160/0x160 [17454.870124] ? prepare_to_swait_event+0xf3/0x140 [17454.871368] rcu_gp_kthread+0x8dd/0x15b0 [17454.873238] ? force_qs_rnp+0x2c0/0x2c0 [17454.877562] kthread+0x1d1/0x200 [17454.878483] ? set_kthread_struct+0x70/0x70 [17454.884429] ret_from_fork+0x1f/0x30 [17454.887548] rcu: Stack dump where RCU GP kthread last ran: [17454.894099] Sending NMI from CPU 2 to CPUs 1: [17454.898437] NMI backtrace for cpu 1 [17454.898445] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17454.898447] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17454.898449] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17454.898450] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17454.898453] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [17454.898455] RAX: 000000000000697c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17454.898456] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17454.898458] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17454.898459] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17454.898461] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17454.898462] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17454.898464] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17454.898465] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17454.898466] Call Trace: [17454.898467] [17454.898468] ? show_regs.cold.9+0x22/0x2f [17454.898469] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [17454.898470] ? nmi_cpu_backtrace_handler+0x15/0x40 [17454.898471] ? nmi_handle+0xbd/0x210 [17454.898472] ? default_do_nmi+0x4b/0x1b0 [17454.898473] ? do_nmi+0x280/0x380 [17454.898474] ? end_repeat_nmi+0x16/0x1f [17454.898475] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17454.898477] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17454.898478] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17454.898479] [17454.898480] do_raw_spin_lock+0x177/0x190 [17454.898481] _raw_spin_lock+0x21/0x30 [17454.898482] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17454.898483] ? do_raw_spin_unlock+0x75/0x190 [17454.898484] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17454.898485] ? do_wait_intr+0xf0/0xf0 [17454.898486] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17454.898487] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17454.898488] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17454.898489] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17454.898490] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17454.898491] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17454.898492] mdc_intent_lock+0x353/0x7e0 [mdc] [17454.898493] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17454.898494] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17454.898495] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17454.898496] lmv_intent_open+0x338/0x1150 [lmv] [17454.898497] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17454.898498] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17454.898499] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17454.898500] ll_intent_lock+0xc5/0xaa0 [lustre] [17454.898501] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17454.898502] ll_lookup_it+0xe0e/0x2270 [lustre] [17454.898503] ? do_raw_spin_unlock+0x75/0x190 [17454.898504] ? _raw_spin_unlock+0x12/0x30 [17454.898505] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17454.898506] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17454.898507] ll_atomic_open+0x36a/0xd70 [lustre] [17454.898508] lookup_open+0x356/0xb40 [17454.898509] do_last+0x313/0xfc0 [17454.898510] ? nd_jump_root+0xe5/0x160 [17454.898510] ? path_init+0x437/0x520 [17454.898511] path_openat+0xf7/0x500 [17454.898512] ? do_raw_spin_unlock+0x75/0x190 [17454.898513] do_filp_open+0x99/0x140 [17454.898514] ? getname_flags+0x6e/0x330 [17454.898515] ? __check_object_size+0xff/0x256 [17454.898516] ? do_raw_spin_unlock+0x75/0x190 [17454.898517] ? _raw_spin_unlock+0x12/0x30 [17454.898518] do_sys_openat2+0x2b4/0x410 [17454.898519] do_sys_open+0x73/0xa0 [17454.898520] __x64_sys_openat+0x24/0x30 [17454.898521] do_syscall_64+0xc1/0x440 [17454.898522] entry_SYSCALL_64_after_hwframe+0x49/0xae [17454.898523] RIP: 0033:0x7f53425a5332 [17454.898524] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17454.898526] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17454.898529] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17454.898530] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17454.898532] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17454.898533] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17454.898535] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17454.901016] Sending NMI from CPU 2 to CPUs 1: [17455.323855] NMI backtrace for cpu 1 [17455.323864] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17455.323865] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17455.323867] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17455.323869] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17455.323871] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [17455.323875] RAX: 0000000000005b24 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17455.323876] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17455.323878] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17455.323879] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17455.323881] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17455.323882] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17455.323884] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17455.323886] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17455.323887] Call Trace: [17455.323888] [17455.323889] ? show_regs.cold.9+0x22/0x2f [17455.323890] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [17455.323891] ? nmi_cpu_backtrace_handler+0x15/0x40 [17455.323893] ? nmi_handle+0xbd/0x210 [17455.323894] ? default_do_nmi+0x4b/0x1b0 [17455.323895] ? do_nmi+0x280/0x380 [17455.323896] ? end_repeat_nmi+0x16/0x1f [17455.323897] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17455.323898] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17455.323899] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17455.323901] [17455.323902] do_raw_spin_lock+0x177/0x190 [17455.323903] _raw_spin_lock+0x21/0x30 [17455.323904] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17455.323905] ? do_raw_spin_unlock+0x75/0x190 [17455.323906] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17455.323907] ? do_wait_intr+0xf0/0xf0 [17455.323908] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17455.323910] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17455.323911] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17455.323912] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17455.323913] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17455.323914] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17455.323916] mdc_intent_lock+0x353/0x7e0 [mdc] [17455.323917] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17455.323918] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17455.323919] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17455.323920] lmv_intent_open+0x338/0x1150 [lmv] [17455.323921] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17455.323922] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17455.323924] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17455.323925] ll_intent_lock+0xc5/0xaa0 [lustre] [17455.323926] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17455.323927] ll_lookup_it+0xe0e/0x2270 [lustre] [17455.323928] ? do_raw_spin_unlock+0x75/0x190 [17455.323929] ? _raw_spin_unlock+0x12/0x30 [17455.323930] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17455.323932] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17455.323933] ll_atomic_open+0x36a/0xd70 [lustre] [17455.323934] lookup_open+0x356/0xb40 [17455.323935] do_last+0x313/0xfc0 [17455.323936] ? nd_jump_root+0xe5/0x160 [17455.323937] ? path_init+0x437/0x520 [17455.323938] path_openat+0xf7/0x500 [17455.323939] ? do_raw_spin_unlock+0x75/0x190 [17455.323941] do_filp_open+0x99/0x140 [17455.323942] ? getname_flags+0x6e/0x330 [17455.323943] ? __check_object_size+0xff/0x256 [17455.323944] ? do_raw_spin_unlock+0x75/0x190 [17455.323946] ? _raw_spin_unlock+0x12/0x30 [17455.323947] do_sys_openat2+0x2b4/0x410 [17455.323948] do_sys_open+0x73/0xa0 [17455.323949] __x64_sys_openat+0x24/0x30 [17455.323950] do_syscall_64+0xc1/0x440 [17455.323951] entry_SYSCALL_64_after_hwframe+0x49/0xae [17455.323952] RIP: 0033:0x7f53425a5332 [17455.323954] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17455.323956] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17455.323959] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17455.323960] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17455.323961] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17455.323963] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17455.323964] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17455.324622] NMI backtrace for cpu 2 [17455.709423] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17455.716589] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17455.722692] Call Trace: [17455.723566] [17455.727272] ? dump_stack+0xbb/0x10e [17455.728429] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [17455.737059] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [17455.739677] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [17455.745847] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [17455.756515] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [17455.766066] ? rcu_dump_cpu_stacks+0x11b/0x171 [17455.770476] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [17455.773069] ? update_process_times+0x6d/0xc0 [17455.780343] ? tick_sched_handle.isra.18+0x27/0x90 [17455.786415] ? tick_sched_timer+0x53/0xb0 [17455.789227] ? __hrtimer_run_queues+0x1c2/0x4d0 [17455.796242] ? tick_sched_handle.isra.18+0x90/0x90 [17455.802844] ? hrtimer_interrupt+0x128/0x2c0 [17455.805536] ? smp_apic_timer_interrupt+0xa6/0x270 [17455.808893] ? apic_timer_interrupt+0xf/0x20 [17455.811270] [17455.812497] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17455.815984] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17455.819229] ? do_raw_spin_lock+0x177/0x190 [17455.821764] ? _raw_spin_lock+0x21/0x30 [17455.824081] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17455.827174] ? obd_mod_free+0x46/0x140 [ptlrpc] [17455.830201] ? mdc_commit_open+0x91/0xb0 [mdc] [17455.832584] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17455.835901] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17455.839266] ? after_reply+0xe2c/0x1780 [ptlrpc] [17455.842259] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17455.845113] ? __next_timer_interrupt+0x160/0x160 [17455.847263] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [17455.849742] ? do_wait_intr+0xf0/0xf0 [17455.851363] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17455.853630] ? kthread+0x1d1/0x200 [17455.855243] ? set_kthread_struct+0x70/0x70 [17455.857312] ? ret_from_fork+0x1f/0x30 [17456.058789] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [17456.064851] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17456.101310] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17456.113289] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17456.120001] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17456.124151] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17456.136400] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17456.145221] RAX: 0000000000007c83 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17456.153064] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17456.157718] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17456.162274] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17456.167298] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17456.172337] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17456.177013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17456.181588] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17456.189655] Call Trace: [17456.191851] [17456.193013] ? show_regs.cold.9+0x22/0x2f [17456.196354] ? watchdog_timer_fn.cold.6+0x62/0x10a [17456.198249] ? __hrtimer_run_queues+0x1c2/0x4d0 [17456.199345] ? watchdog+0x40/0x40 [17456.201953] ? hrtimer_interrupt+0x128/0x2c0 [17456.205003] ? smp_apic_timer_interrupt+0xa6/0x270 [17456.207558] ? apic_timer_interrupt+0xf/0x20 [17456.213648] [17456.214753] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17456.221749] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17456.224636] do_raw_spin_lock+0x177/0x190 [17456.226787] _raw_spin_lock+0x21/0x30 [17456.229305] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17456.233278] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17456.237301] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17456.244564] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17456.248319] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17456.254254] ? woken_wake_function+0x30/0x30 [17456.258253] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17456.262987] kthread+0x1d1/0x200 [17456.265681] ? set_kthread_struct+0x70/0x70 [17456.268971] ret_from_fork+0x1f/0x30 [17464.056163] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [17464.063903] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17464.091181] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17464.097175] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17464.102336] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17464.106966] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17464.111176] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17464.121908] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17464.126331] RAX: 000000000000768a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17464.130947] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17464.135316] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17464.139132] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17464.143615] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17464.147795] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17464.152169] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17464.155841] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17464.162947] Call Trace: [17464.164778] [17464.166385] ? show_regs.cold.9+0x22/0x2f [17464.168888] ? watchdog_timer_fn.cold.6+0x62/0x10a [17464.171814] ? __hrtimer_run_queues+0x1c2/0x4d0 [17464.174190] ? watchdog+0x40/0x40 [17464.175992] ? hrtimer_interrupt+0x128/0x2c0 [17464.178226] ? smp_apic_timer_interrupt+0xa6/0x270 [17464.180740] ? apic_timer_interrupt+0xf/0x20 [17464.182615] [17464.183903] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17464.185746] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17464.188941] ? newidle_balance.isra.93+0x207/0x580 [17464.191917] do_raw_spin_lock+0x177/0x190 [17464.194038] _raw_spin_lock+0x21/0x30 [17464.195894] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17464.199191] process_one_work+0x2c8/0x700 [17464.201703] worker_thread+0x296/0x6e0 [17464.203769] ? rescuer_thread+0x570/0x570 [17464.206001] kthread+0x1d1/0x200 [17464.207855] ? set_kthread_struct+0x70/0x70 [17464.209974] ret_from_fork+0x1f/0x30 [17469.410910] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14832s! [17469.415185] Showing busy workqueues and worker pools: [17469.416939] workqueue events: flags=0x0 [17469.421177] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17469.432777] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17469.440299] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17469.444693] pending: cpu_latency_work [ptlrpc] [17469.451859] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17469.456247] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17469.462626] workqueue events_long: flags=0x0 [17469.476132] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17469.486843] pending: lu_objects_flush [obdclass] [17469.494898] workqueue events_unbound: flags=0x2 [17469.504397] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17469.520873] pending: flush_memcg_stats_dwork, idle_cull_fn [17469.531872] workqueue events_power_efficient: flags=0x80 [17469.536765] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17469.545260] pending: check_lifetime, do_cache_clean [17469.559850] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17469.566116] pending: neigh_periodic_work, neigh_periodic_work [17469.573776] workqueue mm_percpu_wq: flags=0x8 [17469.575241] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17469.587309] pending: vmstat_update, lru_add_drain_per_cpu [17469.591067] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17469.599995] pending: vmstat_update, lru_add_drain_per_cpu [17469.606981] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17469.621298] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17469.627899] workqueue xprtiod: flags=0xa [17469.635013] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17469.640947] pending: xprt_autoclose, xs_error_handle [17469.646361] workqueue ipv6_addrconf: flags=0x40008 [17469.648985] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17469.656602] pending: addrconf_verify_work [17469.658639] workqueue knbd0-recv: flags=0x1a [17469.663067] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17469.673035] in-flight: 651:recv_work [17469.676785] workqueue ptlrpc_pinger: flags=0x2 [17469.682611] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17469.691118] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17469.702401] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14832s workers=4 idle: 15979 31824 8 [17469.706074] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17460s workers=2 idle: 654 [17480.055075] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [17480.058375] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [17480.062433] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17480.062509] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17480.062516] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17480.062519] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17480.062533] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17480.062537] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17480.062542] RAX: 0000000000006393 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17480.062544] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17480.062546] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17480.062547] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17480.062550] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17480.062553] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17480.062555] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17480.062558] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17480.062567] Call Trace: [17480.062573] [17480.062578] ? show_regs.cold.9+0x22/0x2f [17480.062587] ? watchdog_timer_fn.cold.6+0x62/0x10a [17480.062593] ? __hrtimer_run_queues+0x1c2/0x4d0 [17480.062597] ? watchdog+0x40/0x40 [17480.062811] ? hrtimer_interrupt+0x128/0x2c0 [17480.062816] ? smp_apic_timer_interrupt+0xa6/0x270 [17480.062823] ? apic_timer_interrupt+0xf/0x20 [17480.062828] [17480.062831] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17480.062838] do_raw_spin_lock+0x177/0x190 [17480.062843] _raw_spin_lock+0x21/0x30 [17480.062847] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17480.063563] ? do_raw_spin_unlock+0x75/0x190 [17480.063570] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17480.065410] ? do_wait_intr+0xf0/0xf0 [17480.065420] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17480.066858] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17480.068098] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17480.068429] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17480.068561] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17480.071964] Modules linked in: lustre(O) [17480.135506] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17480.135567] mdc_intent_lock+0x353/0x7e0 [mdc] [17480.135896] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17480.136044] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17480.136589] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17480.136638] lmv_intent_open+0x338/0x1150 [lmv] [17480.136670] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17480.136918] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17480.137099] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17480.137122] ll_intent_lock+0xc5/0xaa0 [lustre] [17480.157822] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17480.157898] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17480.157907] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17480.157910] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17480.157925] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17480.157929] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17480.157933] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17480.157936] RDX: 000000000000724b RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17480.157938] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17480.157941] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17480.157943] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17480.157947] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17480.157950] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17480.157952] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17480.157962] Call Trace: [17480.157968] [17480.157974] ? show_regs.cold.9+0x22/0x2f [17480.157984] ? watchdog_timer_fn.cold.6+0x62/0x10a [17480.157990] ? __hrtimer_run_queues+0x1c2/0x4d0 [17480.157995] ? watchdog+0x40/0x40 [17480.157999] ? hrtimer_interrupt+0x128/0x2c0 [17480.158003] ? smp_apic_timer_interrupt+0xa6/0x270 [17480.158011] ? apic_timer_interrupt+0xf/0x20 [17480.158016] [17480.158019] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17480.158025] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17480.158032] do_raw_spin_lock+0x177/0x190 [17480.158038] _raw_spin_lock+0x21/0x30 [17480.158045] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17480.168439] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17480.168581] ll_lookup_it+0xe0e/0x2270 [lustre] [17480.168809] ? do_raw_spin_unlock+0x75/0x190 [17480.168817] ? _raw_spin_unlock+0x12/0x30 [17480.168824] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17480.168853] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17480.168874] ll_atomic_open+0x36a/0xd70 [lustre] [17480.169085] lookup_open+0x356/0xb40 [17480.169094] do_last+0x313/0xfc0 [17480.169099] ? nd_jump_root+0xe5/0x160 [17480.169104] ? path_init+0x437/0x520 [17480.169109] path_openat+0xf7/0x500 [17480.169114] ? do_raw_spin_unlock+0x75/0x190 [17480.169119] do_filp_open+0x99/0x140 [17480.169126] ? getname_flags+0x6e/0x330 [17480.169131] ? __check_object_size+0xff/0x256 [17480.169138] ? do_raw_spin_unlock+0x75/0x190 [17480.169142] ? _raw_spin_unlock+0x12/0x30 [17480.169148] do_sys_openat2+0x2b4/0x410 [17480.169152] do_sys_open+0x73/0xa0 [17480.169156] __x64_sys_openat+0x24/0x30 [17480.169160] do_syscall_64+0xc1/0x440 [17480.169166] entry_SYSCALL_64_after_hwframe+0x49/0xae [17480.169173] RIP: 0033:0x7f53425a5332 [17480.169308] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17480.169314] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17480.169320] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17480.169322] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17480.169325] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17480.169327] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17480.169330] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17480.792811] obd_mod_free+0x46/0x140 [ptlrpc] [17480.794761] mdc_commit_open+0x91/0xb0 [mdc] [17480.796205] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17480.798227] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17480.800451] after_reply+0xe2c/0x1780 [ptlrpc] [17480.802412] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17480.804779] ? __next_timer_interrupt+0x160/0x160 [17480.806685] ptlrpcd+0xa82/0xe40 [ptlrpc] [17480.808786] ? do_wait_intr+0xf0/0xf0 [17480.810318] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17480.813177] kthread+0x1d1/0x200 [17480.815687] ? set_kthread_struct+0x70/0x70 [17480.817085] ret_from_fork+0x1f/0x30 [17484.058111] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [17484.062451] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17484.113735] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17484.121189] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17484.127039] RIP: 0010:native_safe_halt+0x2e/0x50 [17484.129045] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17484.147927] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17484.152594] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17484.155876] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17484.165922] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17484.169700] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17484.177965] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17484.184239] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17484.191866] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17484.199275] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17484.206812] Call Trace: [17484.208963] [17484.211113] ? show_regs.cold.9+0x22/0x2f [17484.213091] ? watchdog_timer_fn.cold.6+0x62/0x10a [17484.214949] ? __hrtimer_run_queues+0x1c2/0x4d0 [17484.217277] ? watchdog+0x40/0x40 [17484.218094] ? hrtimer_interrupt+0x128/0x2c0 [17484.219092] ? smp_apic_timer_interrupt+0xa6/0x270 [17484.220242] ? apic_timer_interrupt+0xf/0x20 [17484.227946] [17484.228941] ? native_safe_halt+0x2e/0x50 [17484.238727] kvm_wait+0xcd/0xe0 [17484.239899] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17484.242863] do_raw_spin_lock+0x177/0x190 [17484.246506] _raw_spin_lock+0x21/0x30 [17484.247841] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17484.259438] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17484.265825] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17484.267861] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17484.283836] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17484.290287] ? woken_wake_function+0x30/0x30 [17484.291676] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17484.301870] kthread+0x1d1/0x200 [17484.303966] ? set_kthread_struct+0x70/0x70 [17484.309759] ret_from_fork+0x1f/0x30 [17496.052124] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [17496.062231] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17496.124341] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17496.138624] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17496.150337] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17496.157671] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17496.165667] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17496.178391] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17496.188955] RAX: 0000000000005bdc RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17496.195425] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17496.204044] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17496.210012] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17496.213383] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17496.217522] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17496.222610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17496.225671] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17496.230447] Call Trace: [17496.233515] [17496.235439] ? show_regs.cold.9+0x22/0x2f [17496.239216] ? watchdog_timer_fn.cold.6+0x62/0x10a [17496.243925] ? __hrtimer_run_queues+0x1c2/0x4d0 [17496.245626] ? watchdog+0x40/0x40 [17496.247202] ? hrtimer_interrupt+0x128/0x2c0 [17496.249668] ? smp_apic_timer_interrupt+0xa6/0x270 [17496.256577] ? apic_timer_interrupt+0xf/0x20 [17496.264132] [17496.267458] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17496.283311] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17496.297412] ? newidle_balance.isra.93+0x207/0x580 [17496.301314] do_raw_spin_lock+0x177/0x190 [17496.303572] _raw_spin_lock+0x21/0x30 [17496.304667] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17496.307066] process_one_work+0x2c8/0x700 [17496.308656] worker_thread+0x296/0x6e0 [17496.309982] ? rescuer_thread+0x570/0x570 [17496.311745] kthread+0x1d1/0x200 [17496.313047] ? set_kthread_struct+0x70/0x70 [17496.316648] ret_from_fork+0x1f/0x30 [17500.130518] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14863s! [17500.138255] Showing busy workqueues and worker pools: [17500.141555] workqueue events: flags=0x0 [17500.144160] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17500.148073] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17500.152887] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17500.154905] pending: cpu_latency_work [ptlrpc] [17500.156614] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17500.158582] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17500.161584] workqueue events_long: flags=0x0 [17500.162781] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17500.164867] pending: lu_objects_flush [obdclass] [17500.166522] workqueue events_unbound: flags=0x2 [17500.167952] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17500.169769] pending: flush_memcg_stats_dwork, idle_cull_fn [17500.171417] workqueue events_power_efficient: flags=0x80 [17500.172961] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17500.174838] pending: check_lifetime, do_cache_clean [17500.176397] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17500.178380] pending: neigh_periodic_work, neigh_periodic_work [17500.180250] workqueue mm_percpu_wq: flags=0x8 [17500.181571] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17500.183576] pending: vmstat_update, lru_add_drain_per_cpu [17500.185965] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17500.188279] pending: vmstat_update, lru_add_drain_per_cpu [17500.190251] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17500.192498] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17500.194556] workqueue xprtiod: flags=0xa [17500.195729] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17500.197754] pending: xprt_autoclose, xs_error_handle [17500.199337] workqueue ipv6_addrconf: flags=0x40008 [17500.201248] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17500.203990] pending: addrconf_verify_work [17500.205605] workqueue knbd0-recv: flags=0x1a [17500.206995] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17500.209359] in-flight: 651:recv_work [17500.210445] workqueue ptlrpc_pinger: flags=0x2 [17500.211660] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17500.213559] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17500.215925] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14863s workers=4 idle: 15979 31824 8 [17500.218482] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17490s workers=2 idle: 654 [17508.055554] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [17508.058754] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [17508.062827] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17508.062922] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17508.062929] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17508.062932] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17508.062946] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17508.062962] RSP: 0018:ffffbb5381503500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [17508.062967] RAX: 0000000000006807 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17508.062970] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17508.062972] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17508.062974] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17508.062977] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17508.062980] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17508.062983] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17508.062985] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17508.062993] Call Trace: [17508.062998] [17508.063004] ? show_regs.cold.9+0x22/0x2f [17508.063013] ? watchdog_timer_fn.cold.6+0x62/0x10a [17508.063018] ? __hrtimer_run_queues+0x1c2/0x4d0 [17508.063023] ? watchdog+0x40/0x40 [17508.063026] ? hrtimer_interrupt+0x128/0x2c0 [17508.063031] ? smp_apic_timer_interrupt+0xa6/0x270 [17508.063038] ? apic_timer_interrupt+0xf/0x20 [17508.063043] [17508.063046] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17508.063053] do_raw_spin_lock+0x177/0x190 [17508.063058] _raw_spin_lock+0x21/0x30 [17508.063063] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17508.063355] ? do_raw_spin_unlock+0x75/0x190 [17508.063361] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17508.073508] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17508.073745] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17508.073762] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17508.073765] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17508.073783] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17508.073787] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17508.073793] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17508.073795] RDX: 0000000000003d27 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17508.073798] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17508.073799] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17508.073802] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17508.073805] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17508.073807] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17508.073810] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17508.073970] Call Trace: [17508.073978] [17508.074095] ? show_regs.cold.9+0x22/0x2f [17508.074109] ? watchdog_timer_fn.cold.6+0x62/0x10a [17508.074117] ? __hrtimer_run_queues+0x1c2/0x4d0 [17508.074123] ? watchdog+0x40/0x40 [17508.074127] ? hrtimer_interrupt+0x128/0x2c0 [17508.074131] ? smp_apic_timer_interrupt+0xa6/0x270 [17508.074140] ? apic_timer_interrupt+0xf/0x20 [17508.074145] [17508.074149] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17508.074154] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17508.074160] do_raw_spin_lock+0x177/0x190 [17508.074165] _raw_spin_lock+0x21/0x30 [17508.074171] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17508.130153] ? do_wait_intr+0xf0/0xf0 [17508.136200] obd_mod_free+0x46/0x140 [ptlrpc] [17508.136546] mdc_commit_open+0x91/0xb0 [mdc] [17508.136573] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17508.136810] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17508.137122] after_reply+0xe2c/0x1780 [ptlrpc] [17508.137379] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17508.137573] ? __next_timer_interrupt+0x160/0x160 [17508.137582] ptlrpcd+0xa82/0xe40 [ptlrpc] [17508.146456] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17508.146744] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17508.147148] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17508.148058] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17508.153972] ? do_wait_intr+0xf0/0xf0 [17508.170478] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17508.174248] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17508.175192] kthread+0x1d1/0x200 [17508.175201] ? set_kthread_struct+0x70/0x70 [17508.175206] ret_from_fork+0x1f/0x30 [17508.673431] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17508.683326] mdc_intent_lock+0x353/0x7e0 [mdc] [17508.685469] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17508.690806] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17508.695572] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17508.706438] lmv_intent_open+0x338/0x1150 [lmv] [17508.708865] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17508.724350] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17508.730142] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17508.736353] ll_intent_lock+0xc5/0xaa0 [lustre] [17508.745264] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17508.749714] ll_lookup_it+0xe0e/0x2270 [lustre] [17508.752132] ? do_raw_spin_unlock+0x75/0x190 [17508.755357] ? _raw_spin_unlock+0x12/0x30 [17508.759299] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17508.763471] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17508.767824] ll_atomic_open+0x36a/0xd70 [lustre] [17508.771680] lookup_open+0x356/0xb40 [17508.773651] do_last+0x313/0xfc0 [17508.774825] ? nd_jump_root+0xe5/0x160 [17508.782590] ? path_init+0x437/0x520 [17508.786331] path_openat+0xf7/0x500 [17508.789415] ? do_raw_spin_unlock+0x75/0x190 [17508.793290] do_filp_open+0x99/0x140 [17508.794862] ? getname_flags+0x6e/0x330 [17508.797791] ? __check_object_size+0xff/0x256 [17508.808722] ? do_raw_spin_unlock+0x75/0x190 [17508.820521] ? _raw_spin_unlock+0x12/0x30 [17508.823343] do_sys_openat2+0x2b4/0x410 [17508.828589] do_sys_open+0x73/0xa0 [17508.834382] __x64_sys_openat+0x24/0x30 [17508.839418] do_syscall_64+0xc1/0x440 [17508.844789] entry_SYSCALL_64_after_hwframe+0x49/0xae [17508.849703] RIP: 0033:0x7f53425a5332 [17508.852827] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17508.880503] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17508.887789] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17508.892121] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17508.901782] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17508.909321] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17508.920821] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17512.062029] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [17512.065150] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17512.114983] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17512.126594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17512.133148] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17512.139318] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17512.170941] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17512.183742] RAX: 0000000000003f08 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17512.193398] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17512.202646] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17512.212462] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17512.216106] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17512.227702] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17512.239601] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17512.246695] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17512.255947] Call Trace: [17512.260781] [17512.262623] ? show_regs.cold.9+0x22/0x2f [17512.275246] ? watchdog_timer_fn.cold.6+0x62/0x10a [17512.286582] ? __hrtimer_run_queues+0x1c2/0x4d0 [17512.294953] ? watchdog+0x40/0x40 [17512.297673] ? hrtimer_interrupt+0x128/0x2c0 [17512.304767] ? smp_apic_timer_interrupt+0xa6/0x270 [17512.311810] ? apic_timer_interrupt+0xf/0x20 [17512.316158] [17512.320458] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17512.329273] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17512.337137] do_raw_spin_lock+0x177/0x190 [17512.343414] _raw_spin_lock+0x21/0x30 [17512.348844] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17512.358836] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17512.365657] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17512.376616] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17512.383602] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17512.385356] ? woken_wake_function+0x30/0x30 [17512.403448] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17512.417746] kthread+0x1d1/0x200 [17512.418674] ? set_kthread_struct+0x70/0x70 [17512.425589] ret_from_fork+0x1f/0x30 [17528.054290] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [17528.067477] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17528.128282] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17528.145650] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17528.159829] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17528.168549] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17528.171218] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17528.189308] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17528.193142] RAX: 0000000000006a07 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17528.196470] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17528.199333] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17528.202179] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17528.205422] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17528.208295] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17528.211302] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17528.213829] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17528.216682] Call Trace: [17528.217791] [17528.218812] ? show_regs.cold.9+0x22/0x2f [17528.220440] ? watchdog_timer_fn.cold.6+0x62/0x10a [17528.222474] ? __hrtimer_run_queues+0x1c2/0x4d0 [17528.224371] ? watchdog+0x40/0x40 [17528.226112] ? hrtimer_interrupt+0x128/0x2c0 [17528.227959] ? smp_apic_timer_interrupt+0xa6/0x270 [17528.230458] ? apic_timer_interrupt+0xf/0x20 [17528.232804] [17528.233760] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17528.236936] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17528.247197] ? newidle_balance.isra.93+0x207/0x580 [17528.253289] do_raw_spin_lock+0x177/0x190 [17528.254580] _raw_spin_lock+0x21/0x30 [17528.261421] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17528.269360] process_one_work+0x2c8/0x700 [17528.271013] worker_thread+0x296/0x6e0 [17528.273266] ? rescuer_thread+0x570/0x570 [17528.278418] kthread+0x1d1/0x200 [17528.281052] ? set_kthread_struct+0x70/0x70 [17528.286368] ret_from_fork+0x1f/0x30 [17530.342107] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14893s! [17530.347634] Showing busy workqueues and worker pools: [17530.351418] workqueue events: flags=0x0 [17530.354093] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17530.358355] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17530.367327] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17530.371382] pending: cpu_latency_work [ptlrpc] [17530.375427] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17530.378619] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17530.384945] workqueue events_long: flags=0x0 [17530.387426] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17530.392090] pending: lu_objects_flush [obdclass] [17530.395337] workqueue events_unbound: flags=0x2 [17530.399033] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17530.403251] pending: flush_memcg_stats_dwork, idle_cull_fn [17530.410245] workqueue events_power_efficient: flags=0x80 [17530.415122] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17530.419327] pending: check_lifetime, do_cache_clean [17530.422373] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17530.426693] pending: neigh_periodic_work, neigh_periodic_work [17530.431125] workqueue mm_percpu_wq: flags=0x8 [17530.434328] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17530.438259] pending: vmstat_update, lru_add_drain_per_cpu [17530.442240] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17530.446323] pending: vmstat_update, lru_add_drain_per_cpu [17530.450980] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17530.456026] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17530.462659] workqueue xprtiod: flags=0xa [17530.465653] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17530.470165] pending: xprt_autoclose, xs_error_handle [17530.473159] workqueue ipv6_addrconf: flags=0x40008 [17530.479146] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17530.483161] pending: addrconf_verify_work [17530.485118] workqueue knbd0-recv: flags=0x1a [17530.486794] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17530.489318] in-flight: 651:recv_work [17530.491619] workqueue ptlrpc_pinger: flags=0x2 [17530.493773] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17530.497662] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17530.502603] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14893s workers=4 idle: 15979 31824 8 [17530.506794] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17520s workers=2 idle: 654 [17536.055084] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [17536.059313] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17536.061852] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [17536.081380] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17536.081403] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17536.081407] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17536.081421] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17536.081425] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17536.081436] RAX: 0000000000002a97 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17536.081438] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17536.081441] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17536.081443] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17536.081445] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17536.081452] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17536.081455] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17536.081457] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17536.081465] Call Trace: [17536.081471] [17536.081477] ? show_regs.cold.9+0x22/0x2f [17536.081494] ? watchdog_timer_fn.cold.6+0x62/0x10a [17536.081505] ? __hrtimer_run_queues+0x1c2/0x4d0 [17536.081509] ? watchdog+0x40/0x40 [17536.081514] ? hrtimer_interrupt+0x128/0x2c0 [17536.081518] ? smp_apic_timer_interrupt+0xa6/0x270 [17536.081526] ? apic_timer_interrupt+0xf/0x20 [17536.081531] [17536.081538] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17536.081548] do_raw_spin_lock+0x177/0x190 [17536.081554] _raw_spin_lock+0x21/0x30 [17536.081563] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17536.081922] ? do_raw_spin_unlock+0x75/0x190 [17536.081926] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17536.082335] ? do_wait_intr+0xf0/0xf0 [17536.082345] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17536.082605] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17536.082813] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17536.083028] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17536.083331] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17536.083437] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17536.083468] mdc_intent_lock+0x353/0x7e0 [mdc] [17536.083503] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17536.083595] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17536.083759] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17536.083778] lmv_intent_open+0x338/0x1150 [lmv] [17536.083799] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17536.083856] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17536.083911] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17536.083927] ll_intent_lock+0xc5/0xaa0 [lustre] [17536.083994] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17536.084074] ll_lookup_it+0xe0e/0x2270 [lustre] [17536.084152] ? do_raw_spin_unlock+0x75/0x190 [17536.084160] ? _raw_spin_unlock+0x12/0x30 [17536.084168] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17536.084197] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17536.084217] ll_atomic_open+0x36a/0xd70 [lustre] [17536.084280] lookup_open+0x356/0xb40 [17536.084288] do_last+0x313/0xfc0 [17536.084292] ? nd_jump_root+0xe5/0x160 [17536.084295] ? path_init+0x437/0x520 [17536.084299] path_openat+0xf7/0x500 [17536.084302] ? do_raw_spin_unlock+0x75/0x190 [17536.084305] do_filp_open+0x99/0x140 [17536.084310] ? getname_flags+0x6e/0x330 [17536.084314] ? __check_object_size+0xff/0x256 [17536.084320] ? do_raw_spin_unlock+0x75/0x190 [17536.084322] ? _raw_spin_unlock+0x12/0x30 [17536.084325] do_sys_openat2+0x2b4/0x410 [17536.084329] do_sys_open+0x73/0xa0 [17536.084332] __x64_sys_openat+0x24/0x30 [17536.084334] do_syscall_64+0xc1/0x440 [17536.084340] entry_SYSCALL_64_after_hwframe+0x49/0xae [17536.095866] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17536.095957] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17536.095966] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17536.095968] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17536.095983] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17536.095987] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17536.095993] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17536.095997] RDX: 0000000000004163 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17536.109138] RIP: 0033:0x7f53425a5332 [17536.131537] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17536.131548] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17536.131552] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17536.131556] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17536.134756] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17536.134768] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17536.134775] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17536.134777] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17536.134779] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17536.134781] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17536.134783] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17536.571529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17536.573821] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17536.577065] Call Trace: [17536.578402] [17536.579544] ? show_regs.cold.9+0x22/0x2f [17536.581292] ? watchdog_timer_fn.cold.6+0x62/0x10a [17536.583051] ? __hrtimer_run_queues+0x1c2/0x4d0 [17536.584474] ? watchdog+0x40/0x40 [17536.585487] ? hrtimer_interrupt+0x128/0x2c0 [17536.590490] ? smp_apic_timer_interrupt+0xa6/0x270 [17536.597648] ? apic_timer_interrupt+0xf/0x20 [17536.601136] [17536.602347] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17536.610706] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17536.614814] do_raw_spin_lock+0x177/0x190 [17536.619348] _raw_spin_lock+0x21/0x30 [17536.624001] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17536.629876] obd_mod_free+0x46/0x140 [ptlrpc] [17536.637582] mdc_commit_open+0x91/0xb0 [mdc] [17536.638918] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17536.652785] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17536.659715] after_reply+0xe2c/0x1780 [ptlrpc] [17536.663664] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17536.669876] ? __next_timer_interrupt+0x160/0x160 [17536.672644] ptlrpcd+0xa82/0xe40 [ptlrpc] [17536.678988] ? do_wait_intr+0xf0/0xf0 [17536.680759] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17536.687876] kthread+0x1d1/0x200 [17536.694475] ? set_kthread_struct+0x70/0x70 [17536.698064] ret_from_fork+0x1f/0x30 [17540.064802] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [17540.074893] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17540.117541] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17540.123069] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17540.127807] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17540.130769] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17540.149172] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17540.155910] RAX: 0000000000004b82 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17540.165027] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17540.170868] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17540.173178] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17540.184984] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17540.188647] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17540.193231] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17540.197285] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17540.200713] Call Trace: [17540.202382] [17540.203617] ? show_regs.cold.9+0x22/0x2f [17540.211054] ? watchdog_timer_fn.cold.6+0x62/0x10a [17540.216558] ? __hrtimer_run_queues+0x1c2/0x4d0 [17540.222179] ? watchdog+0x40/0x40 [17540.224800] ? hrtimer_interrupt+0x128/0x2c0 [17540.230948] ? smp_apic_timer_interrupt+0xa6/0x270 [17540.235570] ? apic_timer_interrupt+0xf/0x20 [17540.238179] [17540.239606] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17540.243083] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17540.246686] do_raw_spin_lock+0x177/0x190 [17540.248958] _raw_spin_lock+0x21/0x30 [17540.250519] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17540.254654] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17540.258705] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17540.262059] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17540.264589] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17540.267290] ? woken_wake_function+0x30/0x30 [17540.270446] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17540.273469] kthread+0x1d1/0x200 [17540.274977] ? set_kthread_struct+0x70/0x70 [17540.278923] ret_from_fork+0x1f/0x30 [17556.054715] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [17556.061303] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17556.112127] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17556.121259] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17556.129358] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17556.134627] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17556.140161] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17556.157808] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17556.170981] RAX: 0000000000007d86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17556.176312] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17556.182889] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17556.186291] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17556.193349] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17556.198388] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17556.206859] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17556.208742] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17556.213284] Call Trace: [17556.215890] [17556.222259] ? show_regs.cold.9+0x22/0x2f [17556.223565] ? watchdog_timer_fn.cold.6+0x62/0x10a [17556.231090] ? __hrtimer_run_queues+0x1c2/0x4d0 [17556.232618] ? watchdog+0x40/0x40 [17556.233666] ? hrtimer_interrupt+0x128/0x2c0 [17556.241588] ? smp_apic_timer_interrupt+0xa6/0x270 [17556.245841] ? apic_timer_interrupt+0xf/0x20 [17556.249163] [17556.251206] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17556.257719] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17556.262171] ? newidle_balance.isra.93+0x207/0x580 [17556.266195] do_raw_spin_lock+0x177/0x190 [17556.268308] _raw_spin_lock+0x21/0x30 [17556.270741] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17556.275603] process_one_work+0x2c8/0x700 [17556.278323] worker_thread+0x296/0x6e0 [17556.282320] ? rescuer_thread+0x570/0x570 [17556.284492] kthread+0x1d1/0x200 [17556.288449] ? set_kthread_struct+0x70/0x70 [17556.290228] ret_from_fork+0x1f/0x30 [17560.553377] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14923s! [17560.563506] Showing busy workqueues and worker pools: [17560.565212] workqueue events: flags=0x0 [17560.571664] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17560.576885] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17560.587523] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17560.593693] pending: cpu_latency_work [ptlrpc] [17560.595331] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17560.598338] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17560.601319] workqueue events_long: flags=0x0 [17560.603854] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17560.607495] pending: lu_objects_flush [obdclass] [17560.610926] workqueue events_unbound: flags=0x2 [17560.615217] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17560.622254] pending: flush_memcg_stats_dwork, idle_cull_fn [17560.626620] workqueue events_power_efficient: flags=0x80 [17560.628739] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17560.630925] pending: check_lifetime, do_cache_clean [17560.632195] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17560.634707] pending: neigh_periodic_work, neigh_periodic_work [17560.639408] workqueue mm_percpu_wq: flags=0x8 [17560.644152] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17560.647543] pending: vmstat_update, lru_add_drain_per_cpu [17560.651515] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17560.656384] pending: vmstat_update, lru_add_drain_per_cpu [17560.659627] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17560.663329] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17560.665934] workqueue xprtiod: flags=0xa [17560.669446] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17560.673538] pending: xprt_autoclose, xs_error_handle [17560.681776] workqueue ipv6_addrconf: flags=0x40008 [17560.685532] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17560.691563] pending: addrconf_verify_work [17560.693220] workqueue knbd0-recv: flags=0x1a [17560.694871] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17560.702606] in-flight: 651:recv_work [17560.704555] workqueue ptlrpc_pinger: flags=0x2 [17560.708275] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17560.717868] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17560.724776] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14923s workers=4 idle: 15979 31824 8 [17560.729527] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17551s workers=2 idle: 654 [17564.055679] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [17564.057433] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [17564.059871] Modules linked in: lustre(O) [17564.063765] Modules linked in: [17564.063768] osp(O) ofd(O) [17564.065806] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log [17564.067537] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17564.067610] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17564.067617] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17564.067620] RIP: 0010:native_safe_halt+0x2e/0x50 [17564.067639] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17564.067643] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17564.067647] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17564.067650] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17564.067652] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17564.067654] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17564.067656] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17564.067659] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17564.067662] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17564.067664] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17564.067671] Call Trace: [17564.067676] [17564.067681] ? show_regs.cold.9+0x22/0x2f [17564.067690] ? watchdog_timer_fn.cold.6+0x62/0x10a [17564.067696] ? __hrtimer_run_queues+0x1c2/0x4d0 [17564.067700] ? watchdog+0x40/0x40 [17564.067705] ? hrtimer_interrupt+0x128/0x2c0 [17564.067710] ? smp_apic_timer_interrupt+0xa6/0x270 [17564.067716] ? apic_timer_interrupt+0xf/0x20 [17564.067721] [17564.067724] ? native_safe_halt+0x2e/0x50 [17564.067728] kvm_wait+0xcd/0xe0 [17564.067735] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17564.067744] do_raw_spin_lock+0x177/0x190 [17564.067749] _raw_spin_lock+0x21/0x30 [17564.067753] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17564.068256] ? do_raw_spin_unlock+0x75/0x190 [17564.068263] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17564.068475] ? do_wait_intr+0xf0/0xf0 [17564.068484] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17564.068856] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17564.069327] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17564.074061] dm_mod sha512_ssse3 sha512_generic [17564.074077] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17564.074084] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17564.074086] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17564.074100] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17564.074105] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17564.074108] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17564.074111] RDX: 00000000000032f8 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17564.074113] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17564.074115] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17564.074118] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17564.074121] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17564.074124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17564.074126] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17564.074135] Call Trace: [17564.074142] [17564.074147] ? show_regs.cold.9+0x22/0x2f [17564.074157] ? watchdog_timer_fn.cold.6+0x62/0x10a [17564.074234] ? __hrtimer_run_queues+0x1c2/0x4d0 [17564.074239] ? watchdog+0x40/0x40 [17564.074243] ? hrtimer_interrupt+0x128/0x2c0 [17564.074247] ? smp_apic_timer_interrupt+0xa6/0x270 [17564.074255] ? apic_timer_interrupt+0xf/0x20 [17564.074260] [17564.074264] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17564.074268] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17564.074273] do_raw_spin_lock+0x177/0x190 [17564.074278] _raw_spin_lock+0x21/0x30 [17564.074286] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17564.074569] obd_mod_free+0x46/0x140 [ptlrpc] [17564.074790] mdc_commit_open+0x91/0xb0 [mdc] [17564.074826] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17564.075237] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17564.075452] after_reply+0xe2c/0x1780 [ptlrpc] [17564.075651] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17564.075857] ? __next_timer_interrupt+0x160/0x160 [17564.075867] ptlrpcd+0xa82/0xe40 [ptlrpc] [17564.100844] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17564.100992] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17564.101046] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17564.101758] mdc_intent_lock+0x353/0x7e0 [mdc] [17564.101789] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17564.101944] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17564.102382] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17564.102415] lmv_intent_open+0x338/0x1150 [lmv] [17564.102450] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17564.102540] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17564.102627] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17564.102650] ll_intent_lock+0xc5/0xaa0 [lustre] [17564.102734] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17564.102816] ll_lookup_it+0xe0e/0x2270 [lustre] [17564.131182] ? do_wait_intr+0xf0/0xf0 [17564.131203] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17564.131474] kthread+0x1d1/0x200 [17564.131482] ? set_kthread_struct+0x70/0x70 [17564.131486] ret_from_fork+0x1f/0x30 [17564.508423] ? do_raw_spin_unlock+0x75/0x190 [17564.510687] ? _raw_spin_unlock+0x12/0x30 [17564.517119] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17564.519726] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17564.521458] ll_atomic_open+0x36a/0xd70 [lustre] [17564.525334] lookup_open+0x356/0xb40 [17564.526986] do_last+0x313/0xfc0 [17564.528856] ? nd_jump_root+0xe5/0x160 [17564.532085] ? path_init+0x437/0x520 [17564.533726] path_openat+0xf7/0x500 [17564.535126] ? do_raw_spin_unlock+0x75/0x190 [17564.537068] do_filp_open+0x99/0x140 [17564.539404] ? getname_flags+0x6e/0x330 [17564.542689] ? __check_object_size+0xff/0x256 [17564.544584] ? do_raw_spin_unlock+0x75/0x190 [17564.546730] ? _raw_spin_unlock+0x12/0x30 [17564.550555] do_sys_openat2+0x2b4/0x410 [17564.552810] do_sys_open+0x73/0xa0 [17564.554323] __x64_sys_openat+0x24/0x30 [17564.556399] do_syscall_64+0xc1/0x440 [17564.558864] entry_SYSCALL_64_after_hwframe+0x49/0xae [17564.560866] RIP: 0033:0x7f53425a5332 [17564.563081] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17564.570847] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17564.578315] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17564.581158] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17564.589296] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17564.593839] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17564.600862] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17568.058100] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [17568.062274] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17568.089413] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17568.106758] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17568.113203] RIP: 0010:native_safe_halt+0x2e/0x50 [17568.121178] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17568.133460] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17568.137147] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17568.145082] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17568.149012] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17568.153054] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17568.161446] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17568.173292] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17568.181300] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17568.187283] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17568.196182] Call Trace: [17568.198579] [17568.200213] ? show_regs.cold.9+0x22/0x2f [17568.205397] ? watchdog_timer_fn.cold.6+0x62/0x10a [17568.211770] ? __hrtimer_run_queues+0x1c2/0x4d0 [17568.219148] ? watchdog+0x40/0x40 [17568.224103] ? hrtimer_interrupt+0x128/0x2c0 [17568.231362] ? smp_apic_timer_interrupt+0xa6/0x270 [17568.235082] ? apic_timer_interrupt+0xf/0x20 [17568.239274] [17568.240068] ? native_safe_halt+0x2e/0x50 [17568.241614] kvm_wait+0xcd/0xe0 [17568.248311] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17568.253542] do_raw_spin_lock+0x177/0x190 [17568.261284] _raw_spin_lock+0x21/0x30 [17568.266774] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17568.278455] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17568.290264] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17568.299515] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17568.308649] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17568.317033] ? woken_wake_function+0x30/0x30 [17568.324912] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17568.329868] kthread+0x1d1/0x200 [17568.332562] ? set_kthread_struct+0x70/0x70 [17568.337388] ret_from_fork+0x1f/0x30 [17588.052086] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [17588.055224] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17588.076263] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17588.080858] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17588.084397] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17588.086357] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17588.089112] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17588.097707] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17588.100727] RAX: 0000000000007691 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17588.104139] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17588.107678] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17588.111064] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17588.115434] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17588.118534] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17588.121838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17588.124229] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17588.127094] Call Trace: [17588.128217] [17588.129147] ? show_regs.cold.9+0x22/0x2f [17588.130193] ? watchdog_timer_fn.cold.6+0x62/0x10a [17588.132746] ? __hrtimer_run_queues+0x1c2/0x4d0 [17588.134960] ? watchdog+0x40/0x40 [17588.137006] ? hrtimer_interrupt+0x128/0x2c0 [17588.138968] ? smp_apic_timer_interrupt+0xa6/0x270 [17588.148533] ? apic_timer_interrupt+0xf/0x20 [17588.154725] [17588.157282] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17588.165276] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17588.172569] ? newidle_balance.isra.93+0x207/0x580 [17588.176901] do_raw_spin_lock+0x177/0x190 [17588.180027] _raw_spin_lock+0x21/0x30 [17588.181404] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17588.188343] process_one_work+0x2c8/0x700 [17588.193520] worker_thread+0x296/0x6e0 [17588.196276] ? rescuer_thread+0x570/0x570 [17588.202823] kthread+0x1d1/0x200 [17588.204251] ? set_kthread_struct+0x70/0x70 [17588.210686] ret_from_fork+0x1f/0x30 [17590.760354] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14953s! [17590.771831] Showing busy workqueues and worker pools: [17590.792669] workqueue events: flags=0x0 [17590.803638] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17590.811349] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17590.818813] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17590.834087] pending: cpu_latency_work [ptlrpc] [17590.842092] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17590.859305] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17590.888885] workqueue events_long: flags=0x0 [17590.911173] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17590.922309] pending: lu_objects_flush [obdclass] [17590.945202] workqueue events_unbound: flags=0x2 [17590.957217] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17590.969261] pending: flush_memcg_stats_dwork, idle_cull_fn [17590.978304] workqueue events_power_efficient: flags=0x80 [17590.990263] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17591.001124] pending: check_lifetime, do_cache_clean [17591.015510] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17591.030001] pending: neigh_periodic_work, neigh_periodic_work [17591.031897] workqueue mm_percpu_wq: flags=0x8 [17591.041014] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17591.053067] pending: vmstat_update, lru_add_drain_per_cpu [17591.064268] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17591.078251] pending: vmstat_update, lru_add_drain_per_cpu [17591.089962] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17591.107378] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17591.129032] workqueue xprtiod: flags=0xa [17591.130291] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17591.147486] pending: xprt_autoclose, xs_error_handle [17591.158982] workqueue ipv6_addrconf: flags=0x40008 [17591.170490] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17591.181509] pending: addrconf_verify_work [17591.188928] workqueue knbd0-recv: flags=0x1a [17591.193888] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17591.203258] in-flight: 651:recv_work [17591.214549] workqueue ptlrpc_pinger: flags=0x2 [17591.221984] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17591.232621] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17591.248150] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14954s workers=4 idle: 15979 31824 8 [17591.264665] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17581s workers=2 idle: 654 [17592.055022] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [17592.057027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [17592.060950] Modules linked in: [17592.063230] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17592.063305] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17592.063312] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17592.063315] RIP: 0010:native_safe_halt+0x2e/0x50 [17592.063329] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17592.063333] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17592.063338] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17592.063340] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17592.063343] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17592.063345] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17592.063347] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17592.063350] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17592.063353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17592.063356] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17592.063365] Call Trace: [17592.063371] [17592.063378] ? show_regs.cold.9+0x22/0x2f [17592.063389] ? watchdog_timer_fn.cold.6+0x62/0x10a [17592.063458] ? __hrtimer_run_queues+0x1c2/0x4d0 [17592.063463] ? watchdog+0x40/0x40 [17592.063467] ? hrtimer_interrupt+0x128/0x2c0 [17592.063471] ? smp_apic_timer_interrupt+0xa6/0x270 [17592.063478] ? apic_timer_interrupt+0xf/0x20 [17592.063483] [17592.063486] ? native_safe_halt+0x2e/0x50 [17592.063490] kvm_wait+0xcd/0xe0 [17592.063498] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17592.063507] do_raw_spin_lock+0x177/0x190 [17592.063511] _raw_spin_lock+0x21/0x30 [17592.063518] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17592.063803] obd_mod_free+0x46/0x140 [ptlrpc] [17592.064002] mdc_commit_open+0x91/0xb0 [mdc] [17592.064041] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17592.064245] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17592.064456] after_reply+0xe2c/0x1780 [ptlrpc] [17592.064651] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17592.064864] ? __next_timer_interrupt+0x160/0x160 [17592.064874] ptlrpcd+0xa82/0xe40 [ptlrpc] [17592.065074] ? do_wait_intr+0xf0/0xf0 [17592.065082] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17592.065309] kthread+0x1d1/0x200 [17592.065315] ? set_kthread_struct+0x70/0x70 [17592.065318] ret_from_fork+0x1f/0x30 [17592.390962] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17592.449748] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17592.456495] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17592.460258] RIP: 0010:native_safe_halt+0x2e/0x50 [17592.462622] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17592.470993] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17592.480215] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17592.487901] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17592.493230] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17592.497081] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17592.500607] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17592.503761] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17592.509886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17592.513964] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17592.516745] Call Trace: [17592.518596] [17592.519301] ? show_regs.cold.9+0x22/0x2f [17592.523149] ? watchdog_timer_fn.cold.6+0x62/0x10a [17592.525684] ? __hrtimer_run_queues+0x1c2/0x4d0 [17592.527318] ? watchdog+0x40/0x40 [17592.536338] ? hrtimer_interrupt+0x128/0x2c0 [17592.538493] ? smp_apic_timer_interrupt+0xa6/0x270 [17592.542136] ? apic_timer_interrupt+0xf/0x20 [17592.545353] [17592.546076] ? native_safe_halt+0x2e/0x50 [17592.547213] kvm_wait+0xcd/0xe0 [17592.549191] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17592.557227] do_raw_spin_lock+0x177/0x190 [17592.559189] _raw_spin_lock+0x21/0x30 [17592.561628] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17592.569167] ? do_raw_spin_unlock+0x75/0x190 [17592.572096] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17592.576841] ? do_wait_intr+0xf0/0xf0 [17592.580975] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17592.587308] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17592.590700] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17592.598212] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17592.608111] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17592.617186] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17592.621152] mdc_intent_lock+0x353/0x7e0 [mdc] [17592.627689] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17592.632188] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17592.636050] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17592.641254] lmv_intent_open+0x338/0x1150 [lmv] [17592.645588] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17592.656546] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17592.665433] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17592.666792] ll_intent_lock+0xc5/0xaa0 [lustre] [17592.672827] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17592.677439] ll_lookup_it+0xe0e/0x2270 [lustre] [17592.679239] ? do_raw_spin_unlock+0x75/0x190 [17592.685153] ? _raw_spin_unlock+0x12/0x30 [17592.687824] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17592.694320] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17592.698844] ll_atomic_open+0x36a/0xd70 [lustre] [17592.702796] lookup_open+0x356/0xb40 [17592.705726] do_last+0x313/0xfc0 [17592.708196] ? nd_jump_root+0xe5/0x160 [17592.709850] ? path_init+0x437/0x520 [17592.712155] path_openat+0xf7/0x500 [17592.716953] ? do_raw_spin_unlock+0x75/0x190 [17592.721952] do_filp_open+0x99/0x140 [17592.726680] ? getname_flags+0x6e/0x330 [17592.728032] ? __check_object_size+0xff/0x256 [17592.737783] ? do_raw_spin_unlock+0x75/0x190 [17592.739194] ? _raw_spin_unlock+0x12/0x30 [17592.743244] do_sys_openat2+0x2b4/0x410 [17592.746268] do_sys_open+0x73/0xa0 [17592.750602] __x64_sys_openat+0x24/0x30 [17592.753881] do_syscall_64+0xc1/0x440 [17592.758687] entry_SYSCALL_64_after_hwframe+0x49/0xae [17592.763325] RIP: 0033:0x7f53425a5332 [17592.765476] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17592.780211] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17592.788210] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17592.797692] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17592.806799] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17592.810815] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17592.814137] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17596.058434] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [17596.070567] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17596.111986] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17596.124519] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17596.129751] RIP: 0010:native_safe_halt+0x2e/0x50 [17596.141300] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17596.154683] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17596.160375] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17596.163038] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17596.173147] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17596.177237] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17596.180474] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17596.183237] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17596.186461] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17596.188658] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17596.191682] Call Trace: [17596.192694] [17596.193562] ? show_regs.cold.9+0x22/0x2f [17596.195126] ? watchdog_timer_fn.cold.6+0x62/0x10a [17596.197073] ? __hrtimer_run_queues+0x1c2/0x4d0 [17596.198894] ? watchdog+0x40/0x40 [17596.200191] ? hrtimer_interrupt+0x128/0x2c0 [17596.201845] ? smp_apic_timer_interrupt+0xa6/0x270 [17596.208268] ? apic_timer_interrupt+0xf/0x20 [17596.211046] [17596.212024] ? native_safe_halt+0x2e/0x50 [17596.214910] kvm_wait+0xcd/0xe0 [17596.216455] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17596.220309] do_raw_spin_lock+0x177/0x190 [17596.221737] _raw_spin_lock+0x21/0x30 [17596.223209] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17596.231912] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17596.237895] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17596.240928] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17596.244979] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17596.249555] ? woken_wake_function+0x30/0x30 [17596.255104] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17596.261128] kthread+0x1d1/0x200 [17596.263677] ? set_kthread_struct+0x70/0x70 [17596.265201] ret_from_fork+0x1f/0x30 [17616.052007] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [17616.055048] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17616.108796] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17616.127883] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17616.132027] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17616.138063] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17616.141294] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17616.165365] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17616.169583] RAX: 00000000000014cc RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17616.173593] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17616.186350] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17616.190762] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17616.194497] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17616.198796] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17616.204244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17616.206828] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17616.209865] Call Trace: [17616.212525] [17616.213681] ? show_regs.cold.9+0x22/0x2f [17616.217417] ? watchdog_timer_fn.cold.6+0x62/0x10a [17616.221423] ? __hrtimer_run_queues+0x1c2/0x4d0 [17616.224726] ? watchdog+0x40/0x40 [17616.227330] ? hrtimer_interrupt+0x128/0x2c0 [17616.228825] ? smp_apic_timer_interrupt+0xa6/0x270 [17616.237082] ? apic_timer_interrupt+0xf/0x20 [17616.239046] [17616.239769] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17616.243661] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17616.246728] ? newidle_balance.isra.93+0x207/0x580 [17616.249919] do_raw_spin_lock+0x177/0x190 [17616.253243] _raw_spin_lock+0x21/0x30 [17616.254581] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17616.259578] process_one_work+0x2c8/0x700 [17616.260949] worker_thread+0x296/0x6e0 [17616.262873] ? rescuer_thread+0x570/0x570 [17616.264260] kthread+0x1d1/0x200 [17616.265447] ? set_kthread_struct+0x70/0x70 [17616.268160] ret_from_fork+0x1f/0x30 [17620.058667] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [17620.059017] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [17620.065371] Modules linked in: [17620.067530] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17620.067615] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17620.067622] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17620.067624] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17620.067638] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17620.067642] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17620.067647] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17620.067649] RDX: 0000000000007d01 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17620.067652] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17620.067654] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17620.067657] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17620.067660] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17620.067663] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17620.067665] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17620.067673] Call Trace: [17620.067678] [17620.067683] ? show_regs.cold.9+0x22/0x2f [17620.067692] ? watchdog_timer_fn.cold.6+0x62/0x10a [17620.067698] ? __hrtimer_run_queues+0x1c2/0x4d0 [17620.067702] ? watchdog+0x40/0x40 [17620.067706] ? hrtimer_interrupt+0x128/0x2c0 [17620.067710] ? smp_apic_timer_interrupt+0xa6/0x270 [17620.067718] ? apic_timer_interrupt+0xf/0x20 [17620.067722] [17620.067725] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17620.067729] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17620.067734] do_raw_spin_lock+0x177/0x190 [17620.067738] _raw_spin_lock+0x21/0x30 [17620.067744] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17620.068147] obd_mod_free+0x46/0x140 [ptlrpc] [17620.068408] mdc_commit_open+0x91/0xb0 [mdc] [17620.068449] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17620.068641] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17620.068897] after_reply+0xe2c/0x1780 [ptlrpc] [17620.070755] lustre(O) [17620.089720] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17620.090074] ? __next_timer_interrupt+0x160/0x160 [17620.090086] ptlrpcd+0xa82/0xe40 [ptlrpc] [17620.090362] ? do_wait_intr+0xf0/0xf0 [17620.090371] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17620.090556] kthread+0x1d1/0x200 [17620.090562] ? set_kthread_struct+0x70/0x70 [17620.090565] ret_from_fork+0x1f/0x30 [17620.221729] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17620.237849] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17620.241792] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17620.245061] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17620.247239] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17620.253821] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17620.256623] RAX: 0000000000007486 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17620.259261] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17620.261764] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17620.264417] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17620.267541] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17620.270184] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17620.273155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17620.275560] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17620.278222] Call Trace: [17620.279154] [17620.279869] ? show_regs.cold.9+0x22/0x2f [17620.281383] ? watchdog_timer_fn.cold.6+0x62/0x10a [17620.283139] ? __hrtimer_run_queues+0x1c2/0x4d0 [17620.284754] ? watchdog+0x40/0x40 [17620.286009] ? hrtimer_interrupt+0x128/0x2c0 [17620.288037] ? smp_apic_timer_interrupt+0xa6/0x270 [17620.290036] ? apic_timer_interrupt+0xf/0x20 [17620.291842] [17620.292638] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17620.294650] do_raw_spin_lock+0x177/0x190 [17620.296481] _raw_spin_lock+0x21/0x30 [17620.297823] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17620.300077] ? do_raw_spin_unlock+0x75/0x190 [17620.301665] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17620.303729] ? do_wait_intr+0xf0/0xf0 [17620.305724] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17620.308116] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17620.310338] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17620.312886] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17620.315077] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17620.317210] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17620.319066] mdc_intent_lock+0x353/0x7e0 [mdc] [17620.320961] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17620.323157] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17620.325851] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17620.328401] lmv_intent_open+0x338/0x1150 [lmv] [17620.330752] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17620.332598] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17620.338013] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17620.339981] ll_intent_lock+0xc5/0xaa0 [lustre] [17620.342104] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17620.344637] ll_lookup_it+0xe0e/0x2270 [lustre] [17620.346597] ? do_raw_spin_unlock+0x75/0x190 [17620.348663] ? _raw_spin_unlock+0x12/0x30 [17620.350493] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17620.352831] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17620.356270] ll_atomic_open+0x36a/0xd70 [lustre] [17620.357957] lookup_open+0x356/0xb40 [17620.359763] do_last+0x313/0xfc0 [17620.361079] ? nd_jump_root+0xe5/0x160 [17620.362488] ? path_init+0x437/0x520 [17620.363770] path_openat+0xf7/0x500 [17620.365152] ? do_raw_spin_unlock+0x75/0x190 [17620.366672] do_filp_open+0x99/0x140 [17620.370450] ? getname_flags+0x6e/0x330 [17620.371806] ? __check_object_size+0xff/0x256 [17620.375187] ? do_raw_spin_unlock+0x75/0x190 [17620.377177] ? _raw_spin_unlock+0x12/0x30 [17620.378820] do_sys_openat2+0x2b4/0x410 [17620.380748] do_sys_open+0x73/0xa0 [17620.382597] __x64_sys_openat+0x24/0x30 [17620.384240] do_syscall_64+0xc1/0x440 [17620.385788] entry_SYSCALL_64_after_hwframe+0x49/0xae [17620.387647] RIP: 0033:0x7f53425a5332 [17620.389831] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17620.403780] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17620.407245] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17620.410256] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17620.413287] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17620.418154] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17620.421781] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17621.472277] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 14984s! [17621.475556] Showing busy workqueues and worker pools: [17621.477535] workqueue events: flags=0x0 [17621.480978] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17621.487664] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17621.500150] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17621.506219] pending: cpu_latency_work [ptlrpc] [17621.509663] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17621.513811] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17621.525169] workqueue events_long: flags=0x0 [17621.528210] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17621.534075] pending: lu_objects_flush [obdclass] [17621.541319] workqueue events_unbound: flags=0x2 [17621.546015] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17621.551375] pending: flush_memcg_stats_dwork, idle_cull_fn [17621.557196] workqueue events_power_efficient: flags=0x80 [17621.561751] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17621.565457] pending: check_lifetime, do_cache_clean [17621.568760] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17621.575263] pending: neigh_periodic_work, neigh_periodic_work [17621.580209] workqueue mm_percpu_wq: flags=0x8 [17621.585701] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17621.595466] pending: vmstat_update, lru_add_drain_per_cpu [17621.600282] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17621.609646] pending: vmstat_update, lru_add_drain_per_cpu [17621.614430] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17621.622501] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17621.630166] workqueue xprtiod: flags=0xa [17621.632845] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17621.648494] pending: xprt_autoclose, xs_error_handle [17621.655792] workqueue ipv6_addrconf: flags=0x40008 [17621.657553] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17621.663127] pending: addrconf_verify_work [17621.665408] workqueue knbd0-recv: flags=0x1a [17621.667090] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17621.669247] in-flight: 651:recv_work [17621.670172] workqueue ptlrpc_pinger: flags=0x2 [17621.672761] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17621.675776] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17621.679434] pool 8: cpus=0-3 flags=0x4 nice=0 hung=14984s workers=4 idle: 15979 31824 8 [17621.684849] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17612s workers=2 idle: 654 [17624.059699] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [17624.066512] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17624.107651] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17624.115323] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17624.124810] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17624.128205] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17624.142388] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17624.146270] RAX: 0000000000000d0f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17624.149644] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17624.154204] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17624.157529] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17624.162179] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17624.165305] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17624.169786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17624.173683] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17624.179358] Call Trace: [17624.182164] [17624.183787] ? show_regs.cold.9+0x22/0x2f [17624.189019] ? watchdog_timer_fn.cold.6+0x62/0x10a [17624.194380] ? __hrtimer_run_queues+0x1c2/0x4d0 [17624.201232] ? watchdog+0x40/0x40 [17624.203701] ? hrtimer_interrupt+0x128/0x2c0 [17624.208436] ? smp_apic_timer_interrupt+0xa6/0x270 [17624.215418] ? apic_timer_interrupt+0xf/0x20 [17624.219400] [17624.222581] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17624.230177] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17624.237294] do_raw_spin_lock+0x177/0x190 [17624.238818] _raw_spin_lock+0x21/0x30 [17624.241844] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17624.247227] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17624.252704] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17624.260883] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17624.265296] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17624.270638] ? woken_wake_function+0x30/0x30 [17624.274413] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17624.278822] kthread+0x1d1/0x200 [17624.281819] ? set_kthread_struct+0x70/0x70 [17624.283909] ret_from_fork+0x1f/0x30 [17634.807056] rcu: INFO: rcu_sched self-detected stall on CPU [17634.809621] rcu: 2-....: (8710586 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [17634.813150] (t=15540381 jiffies g=274185 q=67965) [17634.814959] rcu: rcu_sched kthread starved for 14998747 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [17634.818313] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [17634.824698] rcu: RCU grace-period kthread stack dump: [17634.828405] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [17634.836342] Call Trace: [17634.838285] __schedule+0x351/0xcb0 [17634.840480] schedule+0xc0/0x180 [17634.842538] schedule_timeout+0xb4/0x190 [17634.846657] ? do_raw_spin_unlock+0x75/0x190 [17634.848358] ? __next_timer_interrupt+0x160/0x160 [17634.851627] ? prepare_to_swait_event+0xf3/0x140 [17634.854154] rcu_gp_kthread+0x8dd/0x15b0 [17634.856473] ? force_qs_rnp+0x2c0/0x2c0 [17634.857911] kthread+0x1d1/0x200 [17634.859214] ? set_kthread_struct+0x70/0x70 [17634.860714] ret_from_fork+0x1f/0x30 [17634.862337] rcu: Stack dump where RCU GP kthread last ran: [17634.864204] NMI backtrace for cpu 2 [17634.865850] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17634.870105] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17634.873609] Call Trace: [17634.874464] [17634.875361] ? dump_stack+0xbb/0x10e [17634.876738] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [17634.879071] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [17634.880918] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [17634.883160] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [17634.885341] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [17634.887433] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [17634.889617] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [17634.891481] ? update_process_times+0x6d/0xc0 [17634.893493] ? tick_sched_handle.isra.18+0x27/0x90 [17634.895146] ? tick_sched_timer+0x53/0xb0 [17634.897719] ? __hrtimer_run_queues+0x1c2/0x4d0 [17634.900174] ? tick_sched_handle.isra.18+0x90/0x90 [17634.902918] ? hrtimer_interrupt+0x128/0x2c0 [17634.904279] ? smp_apic_timer_interrupt+0xa6/0x270 [17634.905887] ? apic_timer_interrupt+0xf/0x20 [17634.907317] [17634.908079] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17634.910080] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17634.912604] ? do_raw_spin_lock+0x177/0x190 [17634.913921] ? _raw_spin_lock+0x21/0x30 [17634.915163] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17634.917168] ? obd_mod_free+0x46/0x140 [ptlrpc] [17634.919745] ? mdc_commit_open+0x91/0xb0 [mdc] [17634.921208] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17634.923472] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17634.925680] ? after_reply+0xe2c/0x1780 [ptlrpc] [17634.927806] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17634.930379] ? __next_timer_interrupt+0x160/0x160 [17634.932384] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [17634.934828] ? do_wait_intr+0xf0/0xf0 [17634.936198] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17634.938610] ? kthread+0x1d1/0x200 [17634.940476] ? set_kthread_struct+0x70/0x70 [17634.942377] ? ret_from_fork+0x1f/0x30 [17634.944212] Sending NMI from CPU 2 to CPUs 1: [17634.948573] NMI backtrace for cpu 1 [17634.948578] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17634.948580] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17634.948582] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17634.948584] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17634.948586] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [17634.948588] RAX: 0000000000002187 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17634.948589] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17634.948590] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17634.948592] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17634.948593] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17634.948594] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17634.948595] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17634.948596] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17634.948597] Call Trace: [17634.948598] [17634.948599] ? show_regs.cold.9+0x22/0x2f [17634.948600] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [17634.948601] ? nmi_cpu_backtrace_handler+0x15/0x40 [17634.948602] ? nmi_handle+0xbd/0x210 [17634.948603] ? default_do_nmi+0x4b/0x1b0 [17634.948604] ? do_nmi+0x280/0x380 [17634.948605] ? end_repeat_nmi+0x16/0x1f [17634.948606] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17634.948607] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17634.948608] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17634.948609] [17634.948610] do_raw_spin_lock+0x177/0x190 [17634.948611] _raw_spin_lock+0x21/0x30 [17634.948612] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17634.948613] ? do_raw_spin_unlock+0x75/0x190 [17634.948614] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17634.948615] ? do_wait_intr+0xf0/0xf0 [17634.948616] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17634.948617] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17634.948618] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17634.948619] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17634.948620] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17634.948621] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17634.948622] mdc_intent_lock+0x353/0x7e0 [mdc] [17634.948623] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17634.948624] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17634.948625] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17634.948626] lmv_intent_open+0x338/0x1150 [lmv] [17634.948627] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17634.948628] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17634.948629] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17634.948629] ll_intent_lock+0xc5/0xaa0 [lustre] [17634.948630] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17634.948631] ll_lookup_it+0xe0e/0x2270 [lustre] [17634.948632] ? do_raw_spin_unlock+0x75/0x190 [17634.948633] ? _raw_spin_unlock+0x12/0x30 [17634.948634] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17634.948635] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17634.948636] ll_atomic_open+0x36a/0xd70 [lustre] [17634.948637] lookup_open+0x356/0xb40 [17634.948638] do_last+0x313/0xfc0 [17634.948638] ? nd_jump_root+0xe5/0x160 [17634.948639] ? path_init+0x437/0x520 [17634.948640] path_openat+0xf7/0x500 [17634.948650] ? do_raw_spin_unlock+0x75/0x190 [17634.948651] do_filp_open+0x99/0x140 [17634.948652] ? getname_flags+0x6e/0x330 [17634.948653] ? __check_object_size+0xff/0x256 [17634.948654] ? do_raw_spin_unlock+0x75/0x190 [17634.948655] ? _raw_spin_unlock+0x12/0x30 [17634.948656] do_sys_openat2+0x2b4/0x410 [17634.948657] do_sys_open+0x73/0xa0 [17634.948657] __x64_sys_openat+0x24/0x30 [17634.948658] do_syscall_64+0xc1/0x440 [17634.948659] entry_SYSCALL_64_after_hwframe+0x49/0xae [17634.948660] RIP: 0033:0x7f53425a5332 [17634.948662] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17634.948663] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17634.948665] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17634.948667] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17634.948668] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17634.948669] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17634.948670] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17634.948811] NMI backtrace for cpu 2 [17635.122712] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17635.126955] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17635.130299] Call Trace: [17635.131199] [17635.131996] ? dump_stack+0xbb/0x10e [17635.133107] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [17635.134761] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [17635.137085] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [17635.140161] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [17635.142161] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [17635.144245] ? rcu_dump_cpu_stacks+0x11b/0x171 [17635.145954] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [17635.147493] ? update_process_times+0x6d/0xc0 [17635.149263] ? tick_sched_handle.isra.18+0x27/0x90 [17635.151199] ? tick_sched_timer+0x53/0xb0 [17635.152798] ? __hrtimer_run_queues+0x1c2/0x4d0 [17635.154603] ? tick_sched_handle.isra.18+0x90/0x90 [17635.156446] ? hrtimer_interrupt+0x128/0x2c0 [17635.158147] ? smp_apic_timer_interrupt+0xa6/0x270 [17635.160325] ? apic_timer_interrupt+0xf/0x20 [17635.162294] [17635.163211] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17635.165434] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17635.167634] ? do_raw_spin_lock+0x177/0x190 [17635.169782] ? _raw_spin_lock+0x21/0x30 [17635.170896] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17635.174624] ? obd_mod_free+0x46/0x140 [ptlrpc] [17635.177510] ? mdc_commit_open+0x91/0xb0 [mdc] [17635.179354] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17635.183424] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17635.187549] ? after_reply+0xe2c/0x1780 [ptlrpc] [17635.191279] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17635.193614] ? __next_timer_interrupt+0x160/0x160 [17635.196966] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [17635.199411] ? do_wait_intr+0xf0/0xf0 [17635.200790] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17635.203693] ? kthread+0x1d1/0x200 [17635.204955] ? set_kthread_struct+0x70/0x70 [17635.206204] ? ret_from_fork+0x1f/0x30 [17648.052069] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [17648.056548] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17648.093976] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17648.106746] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17648.115907] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17648.128949] RIP: 0010:native_safe_halt+0x2e/0x50 [17648.132230] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17648.161138] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17648.165921] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17648.176630] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17648.180921] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17648.187501] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17648.195625] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17648.199803] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17648.217197] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17648.229926] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17648.236521] Call Trace: [17648.241303] [17648.244949] ? show_regs.cold.9+0x22/0x2f [17648.249935] ? watchdog_timer_fn.cold.6+0x62/0x10a [17648.255645] ? __hrtimer_run_queues+0x1c2/0x4d0 [17648.257172] ? watchdog+0x40/0x40 [17648.258295] ? hrtimer_interrupt+0x128/0x2c0 [17648.261594] ? smp_apic_timer_interrupt+0xa6/0x270 [17648.263381] ? apic_timer_interrupt+0xf/0x20 [17648.266344] [17648.267073] ? native_safe_halt+0x2e/0x50 [17648.268109] kvm_wait+0xcd/0xe0 [17648.270484] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17648.272099] ? newidle_balance.isra.93+0x207/0x580 [17648.273367] do_raw_spin_lock+0x177/0x190 [17648.274887] _raw_spin_lock+0x21/0x30 [17648.276267] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17648.279739] process_one_work+0x2c8/0x700 [17648.280835] worker_thread+0x296/0x6e0 [17648.281716] ? rescuer_thread+0x570/0x570 [17648.282686] kthread+0x1d1/0x200 [17648.289574] ? set_kthread_struct+0x70/0x70 [17648.292766] ret_from_fork+0x1f/0x30 [17652.058141] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [17652.065042] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17652.122948] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17652.131326] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17652.136027] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17652.140489] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17652.149503] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17652.153059] RAX: 0000000000007509 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17652.156183] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17652.161237] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17652.165491] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17652.168354] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17652.172323] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17652.177833] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17652.180857] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17652.184777] Call Trace: [17652.186203] [17652.187794] ? show_regs.cold.9+0x22/0x2f [17652.190156] ? watchdog_timer_fn.cold.6+0x62/0x10a [17652.191333] BUG: workqueue lockup - pool [17652.194091] ? __hrtimer_run_queues+0x1c2/0x4d0 [17652.194108] ? watchdog+0x40/0x40 [17652.194115] ? hrtimer_interrupt+0x128/0x2c0 [17652.194120] ? smp_apic_timer_interrupt+0xa6/0x270 [17652.194128] ? apic_timer_interrupt+0xf/0x20 [17652.194132] [17652.194137] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17652.194143] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17652.194149] do_raw_spin_lock+0x177/0x190 [17652.194155] _raw_spin_lock+0x21/0x30 [17652.194160] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17652.194483] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17652.194888] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17652.195203] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17652.195568] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17652.196045] ? woken_wake_function+0x30/0x30 [17652.196055] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17652.196303] kthread+0x1d1/0x200 [17652.196309] ? set_kthread_struct+0x70/0x70 [17652.196313] ret_from_fork+0x1f/0x30 [17652.293275] cpus=0-3 flags=0x4 nice=0 stuck for 15015s! [17652.295018] Showing busy workqueues and worker pools: [17652.296654] workqueue events: flags=0x0 [17652.299126] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17652.302257] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17652.310476] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17652.314491] pending: cpu_latency_work [ptlrpc] [17652.317900] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17652.324031] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17652.332686] workqueue events_long: flags=0x0 [17652.336981] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17652.346234] pending: lu_objects_flush [obdclass] [17652.350267] workqueue events_unbound: flags=0x2 [17652.353227] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17652.357286] pending: flush_memcg_stats_dwork, idle_cull_fn [17652.364881] workqueue events_power_efficient: flags=0x80 [17652.373759] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17652.376919] pending: check_lifetime, do_cache_clean [17652.382613] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17652.385953] pending: neigh_periodic_work, neigh_periodic_work [17652.390928] workqueue mm_percpu_wq: flags=0x8 [17652.394530] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17652.402226] pending: vmstat_update, lru_add_drain_per_cpu [17652.408840] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17652.417839] pending: vmstat_update, lru_add_drain_per_cpu [17652.420850] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17652.425688] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17652.433021] workqueue xprtiod: flags=0xa [17652.442116] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17652.452020] pending: xprt_autoclose, xs_error_handle [17652.462088] workqueue ipv6_addrconf: flags=0x40008 [17652.468165] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17652.475299] pending: addrconf_verify_work [17652.478436] workqueue knbd0-recv: flags=0x1a [17652.483075] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17652.491870] in-flight: 651:recv_work [17652.499543] workqueue ptlrpc_pinger: flags=0x2 [17652.503050] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17652.510318] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17652.515121] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15015s workers=4 idle: 15979 31824 8 [17652.521246] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17642s workers=2 idle: 654 [17660.057174] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [17660.060733] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [17660.066285] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17660.066923] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17660.066929] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17660.066931] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17660.066954] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17660.066958] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17660.066962] RAX: 0000000000007a84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17660.066964] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17660.066966] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17660.066968] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17660.066970] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17660.066973] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17660.066976] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17660.066978] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17660.066985] Call Trace: [17660.067001] [17660.067006] ? show_regs.cold.9+0x22/0x2f [17660.067015] ? watchdog_timer_fn.cold.6+0x62/0x10a [17660.067021] ? __hrtimer_run_queues+0x1c2/0x4d0 [17660.067025] ? watchdog+0x40/0x40 [17660.067029] ? hrtimer_interrupt+0x128/0x2c0 [17660.067033] ? smp_apic_timer_interrupt+0xa6/0x270 [17660.067041] ? apic_timer_interrupt+0xf/0x20 [17660.067045] [17660.067049] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17660.067057] do_raw_spin_lock+0x177/0x190 [17660.067062] _raw_spin_lock+0x21/0x30 [17660.067067] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17660.067891] ? do_raw_spin_unlock+0x75/0x190 [17660.067897] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17660.076682] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17660.076754] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17660.076762] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17660.076765] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17660.076778] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17660.076782] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17660.076787] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17660.076790] RDX: 0000000000003eaa RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17660.076792] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17660.076794] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17660.076797] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17660.076800] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17660.076803] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17660.076805] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17660.076814] Call Trace: [17660.076819] [17660.076825] ? show_regs.cold.9+0x22/0x2f [17660.076835] ? watchdog_timer_fn.cold.6+0x62/0x10a [17660.076841] ? __hrtimer_run_queues+0x1c2/0x4d0 [17660.076845] ? watchdog+0x40/0x40 [17660.076849] ? hrtimer_interrupt+0x128/0x2c0 [17660.076853] ? smp_apic_timer_interrupt+0xa6/0x270 [17660.076860] ? apic_timer_interrupt+0xf/0x20 [17660.076865] [17660.076867] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17660.076871] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17660.076875] do_raw_spin_lock+0x177/0x190 [17660.076880] _raw_spin_lock+0x21/0x30 [17660.076886] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17660.077149] obd_mod_free+0x46/0x140 [ptlrpc] [17660.077362] mdc_commit_open+0x91/0xb0 [mdc] [17660.077399] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17660.077619] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17660.077815] after_reply+0xe2c/0x1780 [ptlrpc] [17660.078009] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17660.137178] ? do_wait_intr+0xf0/0xf0 [17660.137269] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17660.137525] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17660.138539] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17660.138932] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17660.139041] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17660.139076] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17660.139105] mdc_intent_lock+0x353/0x7e0 [mdc] [17660.139134] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17660.139943] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17660.140173] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17660.140534] lmv_intent_open+0x338/0x1150 [lmv] [17660.140563] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17660.140658] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17660.141357] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17660.141386] ll_intent_lock+0xc5/0xaa0 [lustre] [17660.141468] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17660.141550] ll_lookup_it+0xe0e/0x2270 [lustre] [17660.141632] ? do_raw_spin_unlock+0x75/0x190 [17660.141641] ? _raw_spin_unlock+0x12/0x30 [17660.141647] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17660.141971] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17660.141993] ll_atomic_open+0x36a/0xd70 [lustre] [17660.142351] lookup_open+0x356/0xb40 [17660.142360] do_last+0x313/0xfc0 [17660.142365] ? nd_jump_root+0xe5/0x160 [17660.142369] ? path_init+0x437/0x520 [17660.142374] path_openat+0xf7/0x500 [17660.142378] ? do_raw_spin_unlock+0x75/0x190 [17660.142383] do_filp_open+0x99/0x140 [17660.142389] ? getname_flags+0x6e/0x330 [17660.142393] ? __check_object_size+0xff/0x256 [17660.142400] ? do_raw_spin_unlock+0x75/0x190 [17660.142403] ? _raw_spin_unlock+0x12/0x30 [17660.142408] do_sys_openat2+0x2b4/0x410 [17660.142412] do_sys_open+0x73/0xa0 [17660.142416] __x64_sys_openat+0x24/0x30 [17660.142419] do_syscall_64+0xc1/0x440 [17660.142425] entry_SYSCALL_64_after_hwframe+0x49/0xae [17660.142430] RIP: 0033:0x7f53425a5332 [17660.142438] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17660.142442] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17660.142447] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17660.142450] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17660.142452] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17660.142454] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17660.142456] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17660.458651] ? __next_timer_interrupt+0x160/0x160 [17660.460766] ptlrpcd+0xa82/0xe40 [ptlrpc] [17660.462688] ? do_wait_intr+0xf0/0xf0 [17660.464700] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17660.467085] kthread+0x1d1/0x200 [17660.468407] ? set_kthread_struct+0x70/0x70 [17660.469864] ret_from_fork+0x1f/0x30 [17680.052013] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [17680.054732] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17680.061294] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [17680.075377] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17680.075392] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17680.075399] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17680.079251] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17680.079333] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17680.079341] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17680.079343] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [17680.079356] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [17680.079361] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17680.079366] RAX: 000000000000606d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17680.079368] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17680.079370] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17680.079373] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17680.079375] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17680.079379] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17680.079382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17680.079385] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17680.079393] Call Trace: [17680.079397] [17680.079403] ? show_regs.cold.9+0x22/0x2f [17680.079412] ? watchdog_timer_fn.cold.6+0x62/0x10a [17680.079418] ? __hrtimer_run_queues+0x1c2/0x4d0 [17680.079423] ? watchdog+0x40/0x40 [17680.079426] ? hrtimer_interrupt+0x128/0x2c0 [17680.079431] ? smp_apic_timer_interrupt+0xa6/0x270 [17680.079437] ? apic_timer_interrupt+0xf/0x20 [17680.079442] [17680.079444] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [17680.079449] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17680.079455] do_raw_spin_lock+0x177/0x190 [17680.079460] _raw_spin_lock+0x21/0x30 [17680.079478] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17680.079744] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17680.079962] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17680.080186] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17680.080406] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17680.080624] ? woken_wake_function+0x30/0x30 [17680.080633] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17680.080904] kthread+0x1d1/0x200 [17680.080910] ? set_kthread_struct+0x70/0x70 [17680.080914] ret_from_fork+0x1f/0x30 [17680.212330] [17680.212905] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17680.214689] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17680.224528] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17680.227714] RAX: 0000000000005ccc RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17680.230741] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17680.234086] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17680.237065] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17680.240964] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17680.244001] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17680.247699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17680.250134] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17680.253026] Call Trace: [17680.254235] [17680.255219] ? show_regs.cold.9+0x22/0x2f [17680.257130] ? watchdog_timer_fn.cold.6+0x62/0x10a [17680.259555] ? __hrtimer_run_queues+0x1c2/0x4d0 [17680.261794] ? watchdog+0x40/0x40 [17680.263419] ? hrtimer_interrupt+0x128/0x2c0 [17680.265679] ? smp_apic_timer_interrupt+0xa6/0x270 [17680.267634] ? apic_timer_interrupt+0xf/0x20 [17680.269775] [17680.270655] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17680.272302] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17680.275346] ? newidle_balance.isra.93+0x207/0x580 [17680.277391] do_raw_spin_lock+0x177/0x190 [17680.279041] _raw_spin_lock+0x21/0x30 [17680.280726] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17680.283323] process_one_work+0x2c8/0x700 [17680.285719] worker_thread+0x296/0x6e0 [17680.286543] ? rescuer_thread+0x570/0x570 [17680.287700] kthread+0x1d1/0x200 [17680.288665] ? set_kthread_struct+0x70/0x70 [17680.290233] ret_from_fork+0x1f/0x30 [17682.919552] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15045s! [17682.938074] Showing busy workqueues and worker pools: [17682.939923] workqueue events: flags=0x0 [17682.951055] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17682.957573] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17682.971542] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17682.976137] pending: cpu_latency_work [ptlrpc] [17682.989034] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17682.996527] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17683.001981] workqueue events_long: flags=0x0 [17683.003873] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17683.011603] pending: lu_objects_flush [obdclass] [17683.016227] workqueue events_unbound: flags=0x2 [17683.021552] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17683.025110] pending: flush_memcg_stats_dwork, idle_cull_fn [17683.029012] workqueue events_power_efficient: flags=0x80 [17683.038341] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17683.044342] pending: check_lifetime, do_cache_clean [17683.048416] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17683.054422] pending: neigh_periodic_work, neigh_periodic_work [17683.057260] workqueue mm_percpu_wq: flags=0x8 [17683.061080] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17683.065535] pending: vmstat_update, lru_add_drain_per_cpu [17683.072960] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17683.078422] pending: vmstat_update, lru_add_drain_per_cpu [17683.081198] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17683.089475] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17683.096156] workqueue xprtiod: flags=0xa [17683.097276] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17683.101965] pending: xprt_autoclose, xs_error_handle [17683.109634] workqueue ipv6_addrconf: flags=0x40008 [17683.117332] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17683.123121] pending: addrconf_verify_work [17683.129299] workqueue knbd0-recv: flags=0x1a [17683.135449] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17683.142044] in-flight: 651:recv_work [17683.146203] workqueue ptlrpc_pinger: flags=0x2 [17683.153627] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17683.163063] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17683.170149] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15046s workers=4 idle: 15979 31824 8 [17683.180578] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17673s workers=2 idle: 654 [17688.058978] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [17688.059109] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [17688.066978] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17688.067057] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17688.067065] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17688.067068] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17688.067082] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17688.067087] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17688.067091] RAX: 0000000000003e09 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17688.067094] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17688.067096] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17688.067098] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17688.067101] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17688.067104] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17688.067108] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17688.067110] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17688.067120] Call Trace: [17688.067126] [17688.067132] ? show_regs.cold.9+0x22/0x2f [17688.067142] ? watchdog_timer_fn.cold.6+0x62/0x10a [17688.067148] ? __hrtimer_run_queues+0x1c2/0x4d0 [17688.067152] ? watchdog+0x40/0x40 [17688.067156] ? hrtimer_interrupt+0x128/0x2c0 [17688.067160] ? smp_apic_timer_interrupt+0xa6/0x270 [17688.067168] ? apic_timer_interrupt+0xf/0x20 [17688.067172] [17688.067175] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17688.067181] do_raw_spin_lock+0x177/0x190 [17688.067186] _raw_spin_lock+0x21/0x30 [17688.067191] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17688.067450] ? do_raw_spin_unlock+0x75/0x190 [17688.067455] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17688.078059] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17688.078136] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17688.078143] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17688.078146] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17688.079799] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17688.079805] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17688.079811] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17688.079814] RDX: 0000000000003985 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17688.079816] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17688.079818] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17688.079821] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17688.079824] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17688.079827] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17688.079829] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17688.079838] Call Trace: [17688.079844] [17688.079851] ? show_regs.cold.9+0x22/0x2f [17688.079860] ? watchdog_timer_fn.cold.6+0x62/0x10a [17688.079867] ? __hrtimer_run_queues+0x1c2/0x4d0 [17688.079871] ? watchdog+0x40/0x40 [17688.079875] ? hrtimer_interrupt+0x128/0x2c0 [17688.079879] ? smp_apic_timer_interrupt+0xa6/0x270 [17688.079888] ? apic_timer_interrupt+0xf/0x20 [17688.079892] [17688.079894] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17688.079899] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17688.079903] do_raw_spin_lock+0x177/0x190 [17688.079907] _raw_spin_lock+0x21/0x30 [17688.079914] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17688.080193] obd_mod_free+0x46/0x140 [ptlrpc] [17688.080390] mdc_commit_open+0x91/0xb0 [mdc] [17688.081401] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17688.081674] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17688.118254] ? do_wait_intr+0xf0/0xf0 [17688.118277] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17688.118530] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17688.119602] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17688.139677] after_reply+0xe2c/0x1780 [ptlrpc] [17688.142878] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17688.143096] ? __next_timer_interrupt+0x160/0x160 [17688.143105] ptlrpcd+0xa82/0xe40 [ptlrpc] [17688.144016] ? do_wait_intr+0xf0/0xf0 [17688.144027] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17688.144241] kthread+0x1d1/0x200 [17688.144247] ? set_kthread_struct+0x70/0x70 [17688.144251] ret_from_fork+0x1f/0x30 [17688.578424] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17688.580990] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17688.584629] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17688.586373] mdc_intent_lock+0x353/0x7e0 [mdc] [17688.591439] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17688.593680] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17688.599072] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17688.601528] lmv_intent_open+0x338/0x1150 [lmv] [17688.605025] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17688.609331] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17688.612187] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17688.613964] ll_intent_lock+0xc5/0xaa0 [lustre] [17688.616906] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17688.621157] ll_lookup_it+0xe0e/0x2270 [lustre] [17688.623265] ? do_raw_spin_unlock+0x75/0x190 [17688.625934] ? _raw_spin_unlock+0x12/0x30 [17688.628781] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17688.630413] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17688.632148] ll_atomic_open+0x36a/0xd70 [lustre] [17688.633952] lookup_open+0x356/0xb40 [17688.635143] do_last+0x313/0xfc0 [17688.636358] ? nd_jump_root+0xe5/0x160 [17688.637898] ? path_init+0x437/0x520 [17688.639432] path_openat+0xf7/0x500 [17688.640814] ? do_raw_spin_unlock+0x75/0x190 [17688.642476] do_filp_open+0x99/0x140 [17688.643952] ? getname_flags+0x6e/0x330 [17688.645392] ? __check_object_size+0xff/0x256 [17688.648206] ? do_raw_spin_unlock+0x75/0x190 [17688.650650] ? _raw_spin_unlock+0x12/0x30 [17688.653359] do_sys_openat2+0x2b4/0x410 [17688.655489] do_sys_open+0x73/0xa0 [17688.656384] __x64_sys_openat+0x24/0x30 [17688.657865] do_syscall_64+0xc1/0x440 [17688.659340] entry_SYSCALL_64_after_hwframe+0x49/0xae [17688.667063] RIP: 0033:0x7f53425a5332 [17688.668677] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17688.683347] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17688.687138] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17688.689893] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17688.691597] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17688.694187] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17688.699073] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17708.053282] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [17708.058677] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [17708.058866] Modules linked in: lustre(O) [17708.062333] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) [17708.064558] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17708.064627] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17708.064634] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17708.064640] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17708.064904] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17708.064915] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17708.064919] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17708.064924] RAX: 0000000000003022 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17708.064926] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17708.064928] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17708.064930] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17708.064932] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17708.064935] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17708.064938] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17708.064940] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17708.064950] Call Trace: [17708.064955] [17708.064960] ? show_regs.cold.9+0x22/0x2f [17708.064969] ? watchdog_timer_fn.cold.6+0x62/0x10a [17708.064975] ? __hrtimer_run_queues+0x1c2/0x4d0 [17708.064979] ? watchdog+0x40/0x40 [17708.064983] ? hrtimer_interrupt+0x128/0x2c0 [17708.064987] ? smp_apic_timer_interrupt+0xa6/0x270 [17708.064995] ? apic_timer_interrupt+0xf/0x20 [17708.064999] [17708.065002] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17708.065006] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17708.065010] ? newidle_balance.isra.93+0x207/0x580 [17708.065015] do_raw_spin_lock+0x177/0x190 [17708.065019] _raw_spin_lock+0x21/0x30 [17708.065025] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17708.065245] process_one_work+0x2c8/0x700 [17708.065252] worker_thread+0x296/0x6e0 [17708.065257] ? rescuer_thread+0x570/0x570 [17708.065260] kthread+0x1d1/0x200 [17708.065266] ? set_kthread_struct+0x70/0x70 [17708.065270] ret_from_fork+0x1f/0x30 [17708.263890] lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17708.287781] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17708.294272] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17708.297736] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17708.305508] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17708.316858] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17708.321556] RAX: 0000000000006da6 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17708.326751] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17708.333907] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17708.338091] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17708.340927] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17708.344199] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17708.350884] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17708.355872] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17708.360791] Call Trace: [17708.361884] [17708.362434] ? show_regs.cold.9+0x22/0x2f [17708.363524] ? watchdog_timer_fn.cold.6+0x62/0x10a [17708.368747] ? __hrtimer_run_queues+0x1c2/0x4d0 [17708.373199] ? watchdog+0x40/0x40 [17708.374827] ? hrtimer_interrupt+0x128/0x2c0 [17708.376351] ? smp_apic_timer_interrupt+0xa6/0x270 [17708.378348] ? apic_timer_interrupt+0xf/0x20 [17708.380882] [17708.383638] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17708.386067] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17708.388310] do_raw_spin_lock+0x177/0x190 [17708.389678] _raw_spin_lock+0x21/0x30 [17708.390913] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17708.395928] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17708.398772] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17708.405104] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17708.409861] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17708.412094] ? woken_wake_function+0x30/0x30 [17708.414119] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17708.420735] kthread+0x1d1/0x200 [17708.422715] ? set_kthread_struct+0x70/0x70 [17708.426919] ret_from_fork+0x1f/0x30 [17713.632075] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15076s! [17713.636282] Showing busy workqueues and worker pools: [17713.638383] workqueue events: flags=0x0 [17713.640556] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17713.644184] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17713.649819] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17713.653940] pending: cpu_latency_work [ptlrpc] [17713.657285] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17713.660625] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17713.665509] workqueue events_long: flags=0x0 [17713.668477] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17713.672690] pending: lu_objects_flush [obdclass] [17713.675445] workqueue events_unbound: flags=0x2 [17713.678190] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17713.684682] pending: flush_memcg_stats_dwork, idle_cull_fn [17713.687915] workqueue events_power_efficient: flags=0x80 [17713.692101] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17713.695712] pending: check_lifetime, do_cache_clean [17713.697117] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17713.700344] pending: neigh_periodic_work, neigh_periodic_work [17713.702486] workqueue mm_percpu_wq: flags=0x8 [17713.704415] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17713.707138] pending: vmstat_update, lru_add_drain_per_cpu [17713.711157] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17713.714202] pending: vmstat_update, lru_add_drain_per_cpu [17713.716539] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17713.721541] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17713.726203] workqueue xprtiod: flags=0xa [17713.727223] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17713.729470] pending: xprt_autoclose, xs_error_handle [17713.731345] workqueue ipv6_addrconf: flags=0x40008 [17713.733225] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17713.735617] pending: addrconf_verify_work [17713.737438] workqueue knbd0-recv: flags=0x1a [17713.739099] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17713.742054] in-flight: 651:recv_work [17713.743393] workqueue ptlrpc_pinger: flags=0x2 [17713.749511] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17713.753960] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17713.757352] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15076s workers=4 idle: 15979 31824 8 [17713.763777] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17704s workers=2 idle: 654 [17716.056735] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [17716.062293] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [17716.063197] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17716.063819] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17716.063826] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17716.063830] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17716.063848] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17716.063853] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17716.063857] RAX: 0000000000007e3a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17716.063860] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17716.063863] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17716.063865] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17716.064505] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17716.064511] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17716.064514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17716.064517] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17716.064526] Call Trace: [17716.064532] [17716.064538] ? show_regs.cold.9+0x22/0x2f [17716.064549] ? watchdog_timer_fn.cold.6+0x62/0x10a [17716.064557] ? __hrtimer_run_queues+0x1c2/0x4d0 [17716.064561] ? watchdog+0x40/0x40 [17716.064566] ? hrtimer_interrupt+0x128/0x2c0 [17716.064570] ? smp_apic_timer_interrupt+0xa6/0x270 [17716.064578] ? apic_timer_interrupt+0xf/0x20 [17716.064582] [17716.064586] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17716.064593] do_raw_spin_lock+0x177/0x190 [17716.064599] _raw_spin_lock+0x21/0x30 [17716.064604] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17716.064877] ? do_raw_spin_unlock+0x75/0x190 [17716.064883] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17716.065381] ? do_wait_intr+0xf0/0xf0 [17716.065391] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17716.065595] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17716.065817] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17716.066094] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17716.066234] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17716.066667] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17716.066702] mdc_intent_lock+0x353/0x7e0 [mdc] [17716.066731] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17716.066827] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17716.067298] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17716.067332] lmv_intent_open+0x338/0x1150 [lmv] [17716.067364] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17716.067526] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17716.067606] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17716.067628] ll_intent_lock+0xc5/0xaa0 [lustre] [17716.067718] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17716.067802] ll_lookup_it+0xe0e/0x2270 [lustre] [17716.067894] ? do_raw_spin_unlock+0x75/0x190 [17716.067903] ? _raw_spin_unlock+0x12/0x30 [17716.067911] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17716.067944] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17716.067968] ll_atomic_open+0x36a/0xd70 [lustre] [17716.068383] lookup_open+0x356/0xb40 [17716.068393] do_last+0x313/0xfc0 [17716.068407] ? nd_jump_root+0xe5/0x160 [17716.068412] ? path_init+0x437/0x520 [17716.068417] path_openat+0xf7/0x500 [17716.068422] ? do_raw_spin_unlock+0x75/0x190 [17716.068427] do_filp_open+0x99/0x140 [17716.068439] ? getname_flags+0x6e/0x330 [17716.068445] ? __check_object_size+0xff/0x256 [17716.073739] Modules linked in: lustre(O) osp(O) ofd(O) [17716.102681] ? do_raw_spin_unlock+0x75/0x190 [17716.102701] ? _raw_spin_unlock+0x12/0x30 [17716.102709] do_sys_openat2+0x2b4/0x410 [17716.102717] do_sys_open+0x73/0xa0 [17716.102722] __x64_sys_openat+0x24/0x30 [17716.102725] do_syscall_64+0xc1/0x440 [17716.102732] entry_SYSCALL_64_after_hwframe+0x49/0xae [17716.102738] RIP: 0033:0x7f53425a5332 [17716.102747] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17716.102751] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17716.102757] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17716.102760] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17716.102762] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17716.102767] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17716.109149] lod(O) [17716.113555] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17716.117056] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17716.117139] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17716.117148] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17716.117152] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17716.424720] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17716.453260] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17716.463811] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17716.472829] RDX: 0000000000007d25 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17716.481693] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17716.495622] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17716.508975] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17716.518009] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17716.525979] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17716.539208] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17716.566203] Call Trace: [17716.572713] [17716.573198] ? show_regs.cold.9+0x22/0x2f [17716.574362] ? watchdog_timer_fn.cold.6+0x62/0x10a [17716.587664] ? __hrtimer_run_queues+0x1c2/0x4d0 [17716.593377] ? watchdog+0x40/0x40 [17716.600064] ? hrtimer_interrupt+0x128/0x2c0 [17716.602192] ? smp_apic_timer_interrupt+0xa6/0x270 [17716.616682] ? apic_timer_interrupt+0xf/0x20 [17716.626186] [17716.632074] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17716.639787] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17716.649168] do_raw_spin_lock+0x177/0x190 [17716.652169] _raw_spin_lock+0x21/0x30 [17716.659772] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17716.668755] obd_mod_free+0x46/0x140 [ptlrpc] [17716.676634] mdc_commit_open+0x91/0xb0 [mdc] [17716.681593] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17716.685132] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17716.688174] after_reply+0xe2c/0x1780 [ptlrpc] [17716.692062] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17716.697162] ? __next_timer_interrupt+0x160/0x160 [17716.701008] ptlrpcd+0xa82/0xe40 [ptlrpc] [17716.710773] ? do_wait_intr+0xf0/0xf0 [17716.715540] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17716.722021] kthread+0x1d1/0x200 [17716.727372] ? set_kthread_struct+0x70/0x70 [17716.730125] ret_from_fork+0x1f/0x30 [17736.058089] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [17736.061706] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17736.087588] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17736.095138] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17736.102166] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17736.105193] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17736.112985] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17736.120633] RAX: 000000000000640a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17736.123005] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17736.129357] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17736.135001] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17736.146777] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17736.159108] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17736.171201] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17736.173327] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17736.180260] Call Trace: [17736.187684] [17736.191489] ? show_regs.cold.9+0x22/0x2f [17736.192935] ? watchdog_timer_fn.cold.6+0x62/0x10a [17736.196295] ? __hrtimer_run_queues+0x1c2/0x4d0 [17736.201258] ? watchdog+0x40/0x40 [17736.206799] ? hrtimer_interrupt+0x128/0x2c0 [17736.212313] ? smp_apic_timer_interrupt+0xa6/0x270 [17736.217304] ? apic_timer_interrupt+0xf/0x20 [17736.218788] [17736.219546] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17736.227579] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17736.237644] do_raw_spin_lock+0x177/0x190 [17736.240021] _raw_spin_lock+0x21/0x30 [17736.242630] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17736.250555] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17736.253792] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17736.266636] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17736.272274] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17736.275319] ? woken_wake_function+0x30/0x30 [17736.278376] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17736.284263] kthread+0x1d1/0x200 [17736.287421] ? set_kthread_struct+0x70/0x70 [17736.294336] ret_from_fork+0x1f/0x30 [17740.052284] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [17740.067789] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17740.121768] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17740.132188] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17740.141582] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17740.148449] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17740.156272] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17740.169076] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17740.172383] RAX: 0000000000006a8a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17740.176386] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17740.180022] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17740.184312] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17740.188275] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17740.191493] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17740.200987] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17740.204639] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17740.212164] Call Trace: [17740.215422] [17740.218427] ? show_regs.cold.9+0x22/0x2f [17740.219857] ? watchdog_timer_fn.cold.6+0x62/0x10a [17740.226566] ? __hrtimer_run_queues+0x1c2/0x4d0 [17740.232190] ? watchdog+0x40/0x40 [17740.236939] ? hrtimer_interrupt+0x128/0x2c0 [17740.241976] ? smp_apic_timer_interrupt+0xa6/0x270 [17740.246009] ? apic_timer_interrupt+0xf/0x20 [17740.247689] [17740.249450] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17740.254005] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17740.257365] ? newidle_balance.isra.93+0x207/0x580 [17740.260029] do_raw_spin_lock+0x177/0x190 [17740.261660] _raw_spin_lock+0x21/0x30 [17740.265028] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17740.267988] process_one_work+0x2c8/0x700 [17740.270307] worker_thread+0x296/0x6e0 [17740.272337] ? rescuer_thread+0x570/0x570 [17740.274427] kthread+0x1d1/0x200 [17740.276265] ? set_kthread_struct+0x70/0x70 [17740.285286] ret_from_fork+0x1f/0x30 [17743.846194] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15106s! [17743.850708] Showing busy workqueues and worker pools: [17743.853013] workqueue events: flags=0x0 [17743.854898] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17743.859558] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17743.865572] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17743.869304] pending: cpu_latency_work [ptlrpc] [17743.872013] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17743.875438] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17743.880163] workqueue events_long: flags=0x0 [17743.882060] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17743.885208] pending: lu_objects_flush [obdclass] [17743.888049] workqueue events_unbound: flags=0x2 [17743.889899] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17743.892049] pending: flush_memcg_stats_dwork, idle_cull_fn [17743.895231] workqueue events_power_efficient: flags=0x80 [17743.898926] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17743.902325] pending: check_lifetime, do_cache_clean [17743.904630] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17743.907534] pending: neigh_periodic_work, neigh_periodic_work [17743.910358] workqueue mm_percpu_wq: flags=0x8 [17743.913171] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17743.916309] pending: vmstat_update, lru_add_drain_per_cpu [17743.919380] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17743.922547] pending: vmstat_update, lru_add_drain_per_cpu [17743.925364] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17743.928227] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17743.930606] workqueue xprtiod: flags=0xa [17743.932023] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17743.935281] pending: xprt_autoclose, xs_error_handle [17743.938243] workqueue ipv6_addrconf: flags=0x40008 [17743.939967] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17743.943582] pending: addrconf_verify_work [17743.945300] workqueue knbd0-recv: flags=0x1a [17743.947612] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17743.950239] in-flight: 651:recv_work [17743.952593] workqueue ptlrpc_pinger: flags=0x2 [17743.955161] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17743.959468] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17743.965290] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15106s workers=4 idle: 15979 31824 8 [17743.971418] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17734s workers=2 idle: 654 [17744.057434] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [17744.058667] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [17744.061059] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17744.061137] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17744.061145] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17744.061148] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17744.061162] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17744.061166] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17744.061171] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17744.061173] RDX: 0000000000006828 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17744.061175] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17744.061178] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17744.061180] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17744.061183] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17744.061185] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17744.061188] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17744.061197] Call Trace: [17744.061202] [17744.061208] ? show_regs.cold.9+0x22/0x2f [17744.061218] ? watchdog_timer_fn.cold.6+0x62/0x10a [17744.061226] ? __hrtimer_run_queues+0x1c2/0x4d0 [17744.061231] ? watchdog+0x40/0x40 [17744.061235] ? hrtimer_interrupt+0x128/0x2c0 [17744.061239] ? smp_apic_timer_interrupt+0xa6/0x270 [17744.061246] ? apic_timer_interrupt+0xf/0x20 [17744.061252] [17744.061255] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17744.061260] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17744.061264] do_raw_spin_lock+0x177/0x190 [17744.061269] _raw_spin_lock+0x21/0x30 [17744.061275] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17744.064933] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17744.065019] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17744.065028] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17744.065031] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17744.065046] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17744.065051] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17744.065056] RAX: 0000000000007099 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17744.065059] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17744.065061] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17744.065064] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17744.066149] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17744.066154] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17744.066157] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17744.066159] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17744.066167] Call Trace: [17744.066172] [17744.066177] ? show_regs.cold.9+0x22/0x2f [17744.066187] ? watchdog_timer_fn.cold.6+0x62/0x10a [17744.066193] ? __hrtimer_run_queues+0x1c2/0x4d0 [17744.066197] ? watchdog+0x40/0x40 [17744.066201] ? hrtimer_interrupt+0x128/0x2c0 [17744.066205] ? smp_apic_timer_interrupt+0xa6/0x270 [17744.066213] ? apic_timer_interrupt+0xf/0x20 [17744.066217] [17744.066220] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17744.066227] do_raw_spin_lock+0x177/0x190 [17744.066232] _raw_spin_lock+0x21/0x30 [17744.066236] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17744.066609] ? do_raw_spin_unlock+0x75/0x190 [17744.066615] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17744.066821] ? do_wait_intr+0xf0/0xf0 [17744.066831] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17744.067028] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17744.110636] obd_mod_free+0x46/0x140 [ptlrpc] [17744.111213] mdc_commit_open+0x91/0xb0 [mdc] [17744.111258] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17744.111451] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17744.111717] after_reply+0xe2c/0x1780 [ptlrpc] [17744.112059] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17744.112242] ? __next_timer_interrupt+0x160/0x160 [17744.112253] ptlrpcd+0xa82/0xe40 [ptlrpc] [17744.112450] ? do_wait_intr+0xf0/0xf0 [17744.131994] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17744.132295] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17744.132415] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17744.132460] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17744.132489] mdc_intent_lock+0x353/0x7e0 [mdc] [17744.132517] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17744.132603] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17744.132834] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17744.132864] lmv_intent_open+0x338/0x1150 [lmv] [17744.132896] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17744.132983] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17744.133063] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17744.133087] ll_intent_lock+0xc5/0xaa0 [lustre] [17744.133173] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17744.133260] ll_lookup_it+0xe0e/0x2270 [lustre] [17744.133343] ? do_raw_spin_unlock+0x75/0x190 [17744.133351] ? _raw_spin_unlock+0x12/0x30 [17744.133358] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17744.133388] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17744.133409] ll_atomic_open+0x36a/0xd70 [lustre] [17744.133497] lookup_open+0x356/0xb40 [17744.133507] do_last+0x313/0xfc0 [17744.133512] ? nd_jump_root+0xe5/0x160 [17744.133517] ? path_init+0x437/0x520 [17744.133522] path_openat+0xf7/0x500 [17744.133527] ? do_raw_spin_unlock+0x75/0x190 [17744.133532] do_filp_open+0x99/0x140 [17744.133539] ? getname_flags+0x6e/0x330 [17744.133544] ? __check_object_size+0xff/0x256 [17744.133553] ? do_raw_spin_unlock+0x75/0x190 [17744.133558] ? _raw_spin_unlock+0x12/0x30 [17744.133563] do_sys_openat2+0x2b4/0x410 [17744.133568] do_sys_open+0x73/0xa0 [17744.133572] __x64_sys_openat+0x24/0x30 [17744.133576] do_syscall_64+0xc1/0x440 [17744.133584] entry_SYSCALL_64_after_hwframe+0x49/0xae [17744.133592] RIP: 0033:0x7f53425a5332 [17744.133601] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17744.133606] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17744.133612] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17744.133615] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17744.133618] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17744.133620] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17744.133623] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17744.743034] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17744.749507] kthread+0x1d1/0x200 [17744.751659] ? set_kthread_struct+0x70/0x70 [17744.753950] ret_from_fork+0x1f/0x30 [17764.060003] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [17764.067296] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17764.099761] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17764.106170] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17764.110924] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17764.114986] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17764.125976] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17764.128993] RAX: 000000000000761a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17764.131242] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17764.134240] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17764.136662] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17764.139175] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17764.141715] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17764.144858] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17764.148059] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17764.150956] Call Trace: [17764.151717] [17764.152264] ? show_regs.cold.9+0x22/0x2f [17764.153947] ? watchdog_timer_fn.cold.6+0x62/0x10a [17764.157790] ? __hrtimer_run_queues+0x1c2/0x4d0 [17764.159648] ? watchdog+0x40/0x40 [17764.161637] ? hrtimer_interrupt+0x128/0x2c0 [17764.163720] ? smp_apic_timer_interrupt+0xa6/0x270 [17764.166769] ? apic_timer_interrupt+0xf/0x20 [17764.169121] [17764.170258] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17764.173962] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17764.177724] do_raw_spin_lock+0x177/0x190 [17764.179717] _raw_spin_lock+0x21/0x30 [17764.181285] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17764.185061] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17764.190192] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17764.192919] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17764.196456] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17764.199939] ? woken_wake_function+0x30/0x30 [17764.201864] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17764.204742] kthread+0x1d1/0x200 [17764.206079] ? set_kthread_struct+0x70/0x70 [17764.208410] ret_from_fork+0x1f/0x30 [17768.054277] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [17768.060514] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17768.102956] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17768.114696] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17768.124260] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17768.138784] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17768.146969] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17768.167292] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17768.171265] RAX: 0000000000007f85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17768.176166] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17768.183200] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17768.187013] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17768.191219] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17768.197081] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17768.203751] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17768.211583] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17768.218260] Call Trace: [17768.220878] [17768.224508] ? show_regs.cold.9+0x22/0x2f [17768.226008] ? watchdog_timer_fn.cold.6+0x62/0x10a [17768.233735] ? __hrtimer_run_queues+0x1c2/0x4d0 [17768.237795] ? watchdog+0x40/0x40 [17768.240350] ? hrtimer_interrupt+0x128/0x2c0 [17768.247926] ? smp_apic_timer_interrupt+0xa6/0x270 [17768.252777] ? apic_timer_interrupt+0xf/0x20 [17768.254545] [17768.262848] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17768.268635] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17768.271234] ? newidle_balance.isra.93+0x207/0x580 [17768.278663] do_raw_spin_lock+0x177/0x190 [17768.280801] _raw_spin_lock+0x21/0x30 [17768.290850] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17768.294995] process_one_work+0x2c8/0x700 [17768.297418] worker_thread+0x296/0x6e0 [17768.304891] ? rescuer_thread+0x570/0x570 [17768.307540] kthread+0x1d1/0x200 [17768.312553] ? set_kthread_struct+0x70/0x70 [17768.314815] ret_from_fork+0x1f/0x30 [17772.055022] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [17772.059399] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [17772.060282] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17772.060352] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17772.060359] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17772.060361] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17772.060375] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17772.060379] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17772.060384] RAX: 000000000000070c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17772.060386] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17772.060389] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17772.060391] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17772.060393] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17772.060397] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17772.060400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17772.060403] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17772.060412] Call Trace: [17772.060419] [17772.060424] ? show_regs.cold.9+0x22/0x2f [17772.060432] ? watchdog_timer_fn.cold.6+0x62/0x10a [17772.060438] ? __hrtimer_run_queues+0x1c2/0x4d0 [17772.060442] ? watchdog+0x40/0x40 [17772.060446] ? hrtimer_interrupt+0x128/0x2c0 [17772.060450] ? smp_apic_timer_interrupt+0xa6/0x270 [17772.060458] ? apic_timer_interrupt+0xf/0x20 [17772.060463] [17772.060465] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17772.060473] do_raw_spin_lock+0x177/0x190 [17772.060477] _raw_spin_lock+0x21/0x30 [17772.060482] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17772.060918] ? do_raw_spin_unlock+0x75/0x190 [17772.060924] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17772.061119] ? do_wait_intr+0xf0/0xf0 [17772.061129] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17772.061333] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17772.061552] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17772.061892] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17772.062001] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17772.062038] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17772.062068] mdc_intent_lock+0x353/0x7e0 [mdc] [17772.062094] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17772.062178] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17772.062660] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17772.062692] lmv_intent_open+0x338/0x1150 [lmv] [17772.062783] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17772.063074] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17772.063158] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17772.063183] ll_intent_lock+0xc5/0xaa0 [lustre] [17772.063401] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17772.063480] ll_lookup_it+0xe0e/0x2270 [lustre] [17772.063559] ? do_raw_spin_unlock+0x75/0x190 [17772.063566] ? _raw_spin_unlock+0x12/0x30 [17772.063574] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17772.065165] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17772.065190] ll_atomic_open+0x36a/0xd70 [lustre] [17772.065399] lookup_open+0x356/0xb40 [17772.065410] do_last+0x313/0xfc0 [17772.065415] ? nd_jump_root+0xe5/0x160 [17772.065419] ? path_init+0x437/0x520 [17772.065423] path_openat+0xf7/0x500 [17772.065428] ? do_raw_spin_unlock+0x75/0x190 [17772.065434] do_filp_open+0x99/0x140 [17772.065440] ? getname_flags+0x6e/0x330 [17772.065445] ? __check_object_size+0xff/0x256 [17772.065452] ? do_raw_spin_unlock+0x75/0x190 [17772.065455] ? _raw_spin_unlock+0x12/0x30 [17772.065461] do_sys_openat2+0x2b4/0x410 [17772.065465] do_sys_open+0x73/0xa0 [17772.065469] __x64_sys_openat+0x24/0x30 [17772.065472] do_syscall_64+0xc1/0x440 [17772.065479] entry_SYSCALL_64_after_hwframe+0x49/0xae [17772.065484] RIP: 0033:0x7f53425a5332 [17772.065498] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17772.065502] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17772.065507] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17772.065509] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17772.065511] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17772.065513] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17772.065515] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17772.913358] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17772.965705] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17772.973011] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17772.991394] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17772.999442] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17773.031120] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17773.035519] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17773.043620] RDX: 0000000000007084 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17773.051643] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17773.054594] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17773.063551] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17773.067584] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17773.073512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17773.077111] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17773.081064] Call Trace: [17773.087948] [17773.088516] ? show_regs.cold.9+0x22/0x2f [17773.090467] ? watchdog_timer_fn.cold.6+0x62/0x10a [17773.096883] ? __hrtimer_run_queues+0x1c2/0x4d0 [17773.098251] ? watchdog+0x40/0x40 [17773.117460] ? hrtimer_interrupt+0x128/0x2c0 [17773.119870] ? smp_apic_timer_interrupt+0xa6/0x270 [17773.124955] ? apic_timer_interrupt+0xf/0x20 [17773.132178] [17773.138687] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17773.143302] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17773.145125] do_raw_spin_lock+0x177/0x190 [17773.153807] _raw_spin_lock+0x21/0x30 [17773.155118] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17773.161978] obd_mod_free+0x46/0x140 [ptlrpc] [17773.167794] mdc_commit_open+0x91/0xb0 [mdc] [17773.170243] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17773.189141] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17773.212352] after_reply+0xe2c/0x1780 [ptlrpc] [17773.216164] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17773.220251] ? __next_timer_interrupt+0x160/0x160 [17773.226212] ptlrpcd+0xa82/0xe40 [ptlrpc] [17773.240161] ? do_wait_intr+0xf0/0xf0 [17773.249313] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17773.253327] kthread+0x1d1/0x200 [17773.260332] ? set_kthread_struct+0x70/0x70 [17773.265908] ret_from_fork+0x1f/0x30 [17774.055773] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15136s! [17774.068518] Showing busy workqueues and worker pools: [17774.074054] workqueue events: flags=0x0 [17774.077866] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17774.080653] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17774.088620] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17774.091656] pending: cpu_latency_work [ptlrpc] [17774.095028] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17774.098332] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17774.102586] workqueue events_long: flags=0x0 [17774.105639] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17774.108595] pending: lu_objects_flush [obdclass] [17774.111849] workqueue events_unbound: flags=0x2 [17774.117642] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17774.120760] pending: flush_memcg_stats_dwork, idle_cull_fn [17774.124744] workqueue events_power_efficient: flags=0x80 [17774.128143] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17774.137041] pending: check_lifetime, do_cache_clean [17774.145842] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17774.150056] pending: neigh_periodic_work, neigh_periodic_work [17774.156174] workqueue mm_percpu_wq: flags=0x8 [17774.161915] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17774.164134] pending: vmstat_update, lru_add_drain_per_cpu [17774.173032] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17774.181922] pending: vmstat_update, lru_add_drain_per_cpu [17774.186884] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17774.194728] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17774.212777] workqueue xprtiod: flags=0xa [17774.216671] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17774.223903] pending: xprt_autoclose, xs_error_handle [17774.227260] workqueue ipv6_addrconf: flags=0x40008 [17774.233955] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17774.236102] pending: addrconf_verify_work [17774.240742] workqueue knbd0-recv: flags=0x1a [17774.243409] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17774.247967] in-flight: 651:recv_work [17774.251290] workqueue ptlrpc_pinger: flags=0x2 [17774.254428] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17774.257532] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17774.261460] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15137s workers=4 idle: 15979 31824 8 [17774.270437] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17764s workers=2 idle: 654 [17792.058166] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [17792.061153] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17792.086302] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17792.091176] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17792.095415] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17792.098148] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17792.106674] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17792.110244] RAX: 0000000000007014 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17792.113564] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17792.118513] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17792.123267] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17792.126607] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17792.130099] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17792.134188] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17792.141510] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17792.145113] Call Trace: [17792.146758] [17792.147906] ? show_regs.cold.9+0x22/0x2f [17792.151465] ? watchdog_timer_fn.cold.6+0x62/0x10a [17792.155594] ? __hrtimer_run_queues+0x1c2/0x4d0 [17792.160277] ? watchdog+0x40/0x40 [17792.161811] ? hrtimer_interrupt+0x128/0x2c0 [17792.164380] ? smp_apic_timer_interrupt+0xa6/0x270 [17792.166513] ? apic_timer_interrupt+0xf/0x20 [17792.168722] [17792.169867] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17792.172954] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17792.175680] do_raw_spin_lock+0x177/0x190 [17792.178099] _raw_spin_lock+0x21/0x30 [17792.179823] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17792.184186] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17792.187344] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17792.190086] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17792.201866] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17792.209950] ? woken_wake_function+0x30/0x30 [17792.213059] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17792.219813] kthread+0x1d1/0x200 [17792.221554] ? set_kthread_struct+0x70/0x70 [17792.224426] ret_from_fork+0x1f/0x30 [17800.052090] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [17800.055034] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [17800.055649] Modules linked in: [17800.057061] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [17800.057069] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17800.057147] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17800.057153] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17800.057156] RIP: 0010:native_safe_halt+0x2e/0x50 [17800.057171] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17800.057175] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17800.057179] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17800.057181] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17800.057184] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17800.057186] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17800.057188] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17800.057190] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17800.057193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17800.057196] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17800.057204] Call Trace: [17800.057210] [17800.057215] ? show_regs.cold.9+0x22/0x2f [17800.057225] ? watchdog_timer_fn.cold.6+0x62/0x10a [17800.057232] ? __hrtimer_run_queues+0x1c2/0x4d0 [17800.057237] ? watchdog+0x40/0x40 [17800.057241] ? hrtimer_interrupt+0x128/0x2c0 [17800.057245] ? smp_apic_timer_interrupt+0xa6/0x270 [17800.057251] ? apic_timer_interrupt+0xf/0x20 [17800.057255] [17800.057258] ? native_safe_halt+0x2e/0x50 [17800.057262] kvm_wait+0xcd/0xe0 [17800.057268] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17800.057275] do_raw_spin_lock+0x177/0x190 [17800.057280] _raw_spin_lock+0x21/0x30 [17800.057285] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17800.057562] obd_mod_free+0x46/0x140 [ptlrpc] [17800.057784] mdc_commit_open+0x91/0xb0 [mdc] [17800.057822] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17800.058024] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17800.058218] after_reply+0xe2c/0x1780 [ptlrpc] [17800.058417] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17800.058609] ? __next_timer_interrupt+0x160/0x160 [17800.058618] ptlrpcd+0xa82/0xe40 [ptlrpc] [17800.058829] ? do_wait_intr+0xf0/0xf0 [17800.058837] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17800.059033] kthread+0x1d1/0x200 [17800.059039] ? set_kthread_struct+0x70/0x70 [17800.059043] ret_from_fork+0x1f/0x30 [17800.062999] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17800.063084] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17800.063090] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17800.063092] RIP: 0010:native_safe_halt+0x2e/0x50 [17800.064366] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17800.064436] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17800.064443] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17800.064450] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17800.064723] RIP: 0010:native_safe_halt+0x2e/0x50 [17800.064732] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17800.064736] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17800.064740] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17800.064743] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [17800.064745] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17800.064747] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17800.064749] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17800.064781] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17800.064785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17800.064787] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17800.064796] Call Trace: [17800.064801] [17800.064805] ? show_regs.cold.9+0x22/0x2f [17800.064815] ? watchdog_timer_fn.cold.6+0x62/0x10a [17800.064821] ? __hrtimer_run_queues+0x1c2/0x4d0 [17800.064825] ? watchdog+0x40/0x40 [17800.064829] ? hrtimer_interrupt+0x128/0x2c0 [17800.064833] ? smp_apic_timer_interrupt+0xa6/0x270 [17800.064839] ? apic_timer_interrupt+0xf/0x20 [17800.064844] [17800.064847] ? native_safe_halt+0x2e/0x50 [17800.064851] kvm_wait+0xcd/0xe0 [17800.064857] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17800.064864] ? newidle_balance.isra.93+0x207/0x580 [17800.071591] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17800.071604] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17800.071610] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17800.071613] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17800.071615] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17800.071617] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17800.071620] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17800.071625] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17800.090331] do_raw_spin_lock+0x177/0x190 [17800.090353] _raw_spin_lock+0x21/0x30 [17800.090360] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17800.090658] process_one_work+0x2c8/0x700 [17800.090666] worker_thread+0x296/0x6e0 [17800.090670] ? rescuer_thread+0x570/0x570 [17800.090675] kthread+0x1d1/0x200 [17800.090681] ? set_kthread_struct+0x70/0x70 [17800.090687] ret_from_fork+0x1f/0x30 [17800.587903] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17800.591291] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17800.595472] Call Trace: [17800.596891] [17800.597544] ? show_regs.cold.9+0x22/0x2f [17800.600041] ? watchdog_timer_fn.cold.6+0x62/0x10a [17800.601891] ? __hrtimer_run_queues+0x1c2/0x4d0 [17800.604255] ? watchdog+0x40/0x40 [17800.605888] ? hrtimer_interrupt+0x128/0x2c0 [17800.607634] ? smp_apic_timer_interrupt+0xa6/0x270 [17800.610151] ? apic_timer_interrupt+0xf/0x20 [17800.612330] [17800.613308] ? native_safe_halt+0x2e/0x50 [17800.617152] kvm_wait+0xcd/0xe0 [17800.619314] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17800.621550] do_raw_spin_lock+0x177/0x190 [17800.624067] _raw_spin_lock+0x21/0x30 [17800.625667] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17800.627924] ? do_raw_spin_unlock+0x75/0x190 [17800.630637] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17800.632137] ? do_wait_intr+0xf0/0xf0 [17800.633293] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17800.636015] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17800.638642] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17800.641897] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17800.644884] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17800.647545] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17800.649872] mdc_intent_lock+0x353/0x7e0 [mdc] [17800.652442] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17800.654932] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17800.658210] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17800.662178] lmv_intent_open+0x338/0x1150 [lmv] [17800.664247] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17800.667402] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17800.669556] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17800.671601] ll_intent_lock+0xc5/0xaa0 [lustre] [17800.674866] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17800.676967] ll_lookup_it+0xe0e/0x2270 [lustre] [17800.679454] ? do_raw_spin_unlock+0x75/0x190 [17800.681265] ? _raw_spin_unlock+0x12/0x30 [17800.683064] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17800.687181] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17800.690048] ll_atomic_open+0x36a/0xd70 [lustre] [17800.692350] lookup_open+0x356/0xb40 [17800.695318] do_last+0x313/0xfc0 [17800.696547] ? nd_jump_root+0xe5/0x160 [17800.698383] ? path_init+0x437/0x520 [17800.700417] path_openat+0xf7/0x500 [17800.701879] ? do_raw_spin_unlock+0x75/0x190 [17800.704092] do_filp_open+0x99/0x140 [17800.706039] ? getname_flags+0x6e/0x330 [17800.708409] ? __check_object_size+0xff/0x256 [17800.710637] ? do_raw_spin_unlock+0x75/0x190 [17800.712524] ? _raw_spin_unlock+0x12/0x30 [17800.714466] do_sys_openat2+0x2b4/0x410 [17800.716550] do_sys_open+0x73/0xa0 [17800.718234] __x64_sys_openat+0x24/0x30 [17800.720059] do_syscall_64+0xc1/0x440 [17800.722544] entry_SYSCALL_64_after_hwframe+0x49/0xae [17800.725033] RIP: 0033:0x7f53425a5332 [17800.726671] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17800.734989] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17800.738313] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17800.741529] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17800.744765] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17800.747257] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17800.750157] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17804.775493] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15167s! [17804.789021] Showing busy workqueues and worker pools: [17804.797251] workqueue events: flags=0x0 [17804.799155] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17804.802329] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17804.817772] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17804.824808] pending: cpu_latency_work [ptlrpc] [17804.832120] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17804.839066] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17804.845953] workqueue events_long: flags=0x0 [17804.850333] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17804.860545] pending: lu_objects_flush [obdclass] [17804.864939] workqueue events_unbound: flags=0x2 [17804.866321] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17804.872854] pending: flush_memcg_stats_dwork, idle_cull_fn [17804.877565] workqueue events_power_efficient: flags=0x80 [17804.882537] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17804.884425] pending: check_lifetime, do_cache_clean [17804.889234] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17804.895572] pending: neigh_periodic_work, neigh_periodic_work [17804.900428] workqueue mm_percpu_wq: flags=0x8 [17804.904060] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17804.905838] pending: vmstat_update, lru_add_drain_per_cpu [17804.915568] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17804.919616] pending: vmstat_update, lru_add_drain_per_cpu [17804.921887] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17804.925502] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17804.929066] workqueue xprtiod: flags=0xa [17804.935799] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17804.937387] pending: xprt_autoclose, xs_error_handle [17804.941607] workqueue ipv6_addrconf: flags=0x40008 [17804.944092] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17804.951309] pending: addrconf_verify_work [17804.953023] workqueue knbd0-recv: flags=0x1a [17804.955162] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17804.957511] in-flight: 651:recv_work [17804.959494] workqueue ptlrpc_pinger: flags=0x2 [17804.961618] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17804.966623] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17804.971441] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15167s workers=4 idle: 15979 31824 8 [17804.975288] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17795s workers=2 idle: 654 [17814.814760] rcu: INFO: rcu_sched self-detected stall on CPU [17814.828714] rcu: 2-....: (8808139 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [17814.842364] (t=15720388 jiffies g=274185 q=68000) [17814.844870] rcu: rcu_sched kthread starved for 15178754 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [17814.850374] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [17814.855164] rcu: RCU grace-period kthread stack dump: [17814.857597] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [17814.862956] Call Trace: [17814.864198] __schedule+0x351/0xcb0 [17814.865913] schedule+0xc0/0x180 [17814.867666] schedule_timeout+0xb4/0x190 [17814.869673] ? do_raw_spin_unlock+0x75/0x190 [17814.872072] ? __next_timer_interrupt+0x160/0x160 [17814.875085] ? prepare_to_swait_event+0xf3/0x140 [17814.877695] rcu_gp_kthread+0x8dd/0x15b0 [17814.879461] ? force_qs_rnp+0x2c0/0x2c0 [17814.881089] kthread+0x1d1/0x200 [17814.882845] ? set_kthread_struct+0x70/0x70 [17814.884884] ret_from_fork+0x1f/0x30 [17814.887000] rcu: Stack dump where RCU GP kthread last ran: [17814.889364] Sending NMI from CPU 2 to CPUs 3: [17814.892986] NMI backtrace for cpu 3 [17814.892993] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17814.892995] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17814.892997] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17814.892998] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17814.893001] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 [17814.893003] RAX: 0000000000005f84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17814.893004] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17814.893006] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17814.893008] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17814.893009] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17814.893010] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17814.893011] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17814.893012] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17814.893013] Call Trace: [17814.893014] [17814.893016] ? show_regs.cold.9+0x22/0x2f [17814.893016] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [17814.893017] ? nmi_cpu_backtrace_handler+0x15/0x40 [17814.893019] ? nmi_handle+0xbd/0x210 [17814.893020] ? default_do_nmi+0x4b/0x1b0 [17814.893021] ? do_nmi+0x280/0x380 [17814.893022] ? end_repeat_nmi+0x16/0x1f [17814.893023] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17814.893025] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17814.893026] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17814.893027] [17814.893029] do_raw_spin_lock+0x177/0x190 [17814.893030] _raw_spin_lock+0x21/0x30 [17814.893031] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17814.893032] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17814.893033] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17814.893034] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17814.893036] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17814.893037] ? woken_wake_function+0x30/0x30 [17814.893038] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17814.893039] kthread+0x1d1/0x200 [17814.893040] ? set_kthread_struct+0x70/0x70 [17814.893041] ret_from_fork+0x1f/0x30 [17814.895369] Sending NMI from CPU 2 to CPUs 1: [17815.015292] NMI backtrace for cpu 1 [17815.015299] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17815.015301] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17815.015303] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17815.015304] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17815.015306] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [17815.015309] RAX: 0000000000001f54 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17815.015310] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17815.015311] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17815.015383] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17815.015384] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17815.015386] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17815.015388] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17815.015389] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17815.015390] Call Trace: [17815.015391] [17815.015392] ? show_regs.cold.9+0x22/0x2f [17815.015393] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [17815.015394] ? nmi_cpu_backtrace_handler+0x15/0x40 [17815.015396] ? nmi_handle+0xbd/0x210 [17815.015397] ? default_do_nmi+0x4b/0x1b0 [17815.015397] ? do_nmi+0x280/0x380 [17815.015399] ? end_repeat_nmi+0x16/0x1f [17815.015400] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17815.015401] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17815.015402] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17815.015404] [17815.015404] do_raw_spin_lock+0x177/0x190 [17815.015406] _raw_spin_lock+0x21/0x30 [17815.015406] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17815.015407] ? do_raw_spin_unlock+0x75/0x190 [17815.015409] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17815.015409] ? do_wait_intr+0xf0/0xf0 [17815.015411] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17815.015411] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17815.015413] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17815.015414] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17815.015415] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17815.015416] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17815.015417] mdc_intent_lock+0x353/0x7e0 [mdc] [17815.015418] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17815.015419] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17815.015420] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17815.015422] lmv_intent_open+0x338/0x1150 [lmv] [17815.015423] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17815.015424] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17815.015425] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17815.015426] ll_intent_lock+0xc5/0xaa0 [lustre] [17815.015427] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17815.015428] ll_lookup_it+0xe0e/0x2270 [lustre] [17815.015429] ? do_raw_spin_unlock+0x75/0x190 [17815.015430] ? _raw_spin_unlock+0x12/0x30 [17815.015431] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17815.015432] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17815.015433] ll_atomic_open+0x36a/0xd70 [lustre] [17815.015434] lookup_open+0x356/0xb40 [17815.015435] do_last+0x313/0xfc0 [17815.015436] ? nd_jump_root+0xe5/0x160 [17815.015437] ? path_init+0x437/0x520 [17815.015438] path_openat+0xf7/0x500 [17815.015439] ? do_raw_spin_unlock+0x75/0x190 [17815.015440] do_filp_open+0x99/0x140 [17815.015442] ? getname_flags+0x6e/0x330 [17815.015442] ? __check_object_size+0xff/0x256 [17815.015443] ? do_raw_spin_unlock+0x75/0x190 [17815.015445] ? _raw_spin_unlock+0x12/0x30 [17815.015446] do_sys_openat2+0x2b4/0x410 [17815.015446] do_sys_open+0x73/0xa0 [17815.015447] __x64_sys_openat+0x24/0x30 [17815.015448] do_syscall_64+0xc1/0x440 [17815.015449] entry_SYSCALL_64_after_hwframe+0x49/0xae [17815.015450] RIP: 0033:0x7f53425a5332 [17815.015452] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17815.015454] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17815.015456] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17815.015457] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17815.015459] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17815.015460] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17815.015462] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17815.016598] NMI backtrace for cpu 2 [17815.313009] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17815.320806] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17815.326376] Call Trace: [17815.327219] [17815.328092] ? dump_stack+0xbb/0x10e [17815.331240] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [17815.335659] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [17815.342228] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [17815.346554] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [17815.348411] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [17815.353367] ? rcu_dump_cpu_stacks+0x11b/0x171 [17815.354648] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [17815.358573] ? update_process_times+0x6d/0xc0 [17815.360093] ? tick_sched_handle.isra.18+0x27/0x90 [17815.361256] ? tick_sched_timer+0x53/0xb0 [17815.364929] ? __hrtimer_run_queues+0x1c2/0x4d0 [17815.366906] ? tick_sched_handle.isra.18+0x90/0x90 [17815.371865] ? hrtimer_interrupt+0x128/0x2c0 [17815.374146] ? smp_apic_timer_interrupt+0xa6/0x270 [17815.376084] ? apic_timer_interrupt+0xf/0x20 [17815.381098] [17815.382220] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17815.385263] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17815.386579] ? do_raw_spin_lock+0x177/0x190 [17815.388551] ? _raw_spin_lock+0x21/0x30 [17815.389653] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17815.392401] ? obd_mod_free+0x46/0x140 [ptlrpc] [17815.396453] ? mdc_commit_open+0x91/0xb0 [mdc] [17815.402214] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17815.404846] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17815.410249] ? after_reply+0xe2c/0x1780 [ptlrpc] [17815.414102] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17815.416399] ? __next_timer_interrupt+0x160/0x160 [17815.418166] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [17815.420816] ? do_wait_intr+0xf0/0xf0 [17815.424269] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17815.426375] ? kthread+0x1d1/0x200 [17815.429246] ? set_kthread_struct+0x70/0x70 [17815.431956] ? ret_from_fork+0x1f/0x30 [17832.052081] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [17832.054963] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17832.074952] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17832.079835] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17832.084429] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17832.087547] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17832.090772] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17832.101069] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17832.104128] RAX: 0000000000006333 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17832.107218] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17832.110687] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17832.114127] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17832.117316] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17832.120051] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17832.123308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17832.125854] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17832.129056] Call Trace: [17832.129975] [17832.130921] ? show_regs.cold.9+0x22/0x2f [17832.132766] ? watchdog_timer_fn.cold.6+0x62/0x10a [17832.134899] ? __hrtimer_run_queues+0x1c2/0x4d0 [17832.137026] ? watchdog+0x40/0x40 [17832.138764] ? hrtimer_interrupt+0x128/0x2c0 [17832.141025] ? smp_apic_timer_interrupt+0xa6/0x270 [17832.143635] ? apic_timer_interrupt+0xf/0x20 [17832.145704] [17832.146649] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17832.149393] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17832.151765] ? newidle_balance.isra.93+0x207/0x580 [17832.153984] do_raw_spin_lock+0x177/0x190 [17832.156793] _raw_spin_lock+0x21/0x30 [17832.158465] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17832.161129] process_one_work+0x2c8/0x700 [17832.162552] worker_thread+0x296/0x6e0 [17832.164369] ? rescuer_thread+0x570/0x570 [17832.165962] kthread+0x1d1/0x200 [17832.167393] ? set_kthread_struct+0x70/0x70 [17832.169540] ret_from_fork+0x1f/0x30 [17835.494545] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15198s! [17835.500335] Showing busy workqueues and worker pools: [17835.509589] workqueue events: flags=0x0 [17835.514939] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17835.517968] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17835.530689] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17835.533789] pending: cpu_latency_work [ptlrpc] [17835.539824] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17835.546570] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17835.551528] workqueue events_long: flags=0x0 [17835.553755] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17835.560308] pending: lu_objects_flush [obdclass] [17835.562822] workqueue events_unbound: flags=0x2 [17835.568733] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17835.572925] pending: flush_memcg_stats_dwork, idle_cull_fn [17835.581182] workqueue events_power_efficient: flags=0x80 [17835.588915] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17835.594877] pending: check_lifetime, do_cache_clean [17835.600797] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17835.604915] pending: neigh_periodic_work, neigh_periodic_work [17835.612502] workqueue mm_percpu_wq: flags=0x8 [17835.615597] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17835.623259] pending: vmstat_update, lru_add_drain_per_cpu [17835.628646] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17835.634433] pending: vmstat_update, lru_add_drain_per_cpu [17835.636301] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17835.643898] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17835.649401] workqueue xprtiod: flags=0xa [17835.652386] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17835.658512] pending: xprt_autoclose, xs_error_handle [17835.664331] workqueue ipv6_addrconf: flags=0x40008 [17835.665949] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17835.672482] pending: addrconf_verify_work [17835.676940] workqueue knbd0-recv: flags=0x1a [17835.679184] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17835.683337] in-flight: 651:recv_work [17835.684675] workqueue ptlrpc_pinger: flags=0x2 [17835.688915] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17835.693368] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17835.697505] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15198s workers=4 idle: 15979 31824 8 [17835.701626] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17826s workers=2 idle: 654 [17840.055735] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [17840.058197] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [17840.060442] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17840.060528] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17840.060535] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17840.060538] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17840.060554] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17840.060558] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17840.060562] RAX: 0000000000005199 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17840.060564] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17840.060566] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17840.060569] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17840.060571] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17840.060577] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17840.060580] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17840.060582] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17840.060590] Call Trace: [17840.060657] [17840.060664] ? show_regs.cold.9+0x22/0x2f [17840.060673] ? watchdog_timer_fn.cold.6+0x62/0x10a [17840.060679] ? __hrtimer_run_queues+0x1c2/0x4d0 [17840.060684] ? watchdog+0x40/0x40 [17840.060689] ? hrtimer_interrupt+0x128/0x2c0 [17840.060693] ? smp_apic_timer_interrupt+0xa6/0x270 [17840.060702] ? apic_timer_interrupt+0xf/0x20 [17840.060706] [17840.060708] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17840.060713] do_raw_spin_lock+0x177/0x190 [17840.060717] _raw_spin_lock+0x21/0x30 [17840.060721] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17840.062286] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [17840.062292] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17840.062418] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17840.062423] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17840.062425] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17840.062436] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17840.062439] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17840.062444] RAX: 0000000000003872 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17840.062447] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17840.062449] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17840.062451] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17840.062454] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17840.062457] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17840.062460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17840.062463] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17840.062471] Call Trace: [17840.062474] [17840.062478] ? show_regs.cold.9+0x22/0x2f [17840.062486] ? watchdog_timer_fn.cold.6+0x62/0x10a [17840.062491] ? __hrtimer_run_queues+0x1c2/0x4d0 [17840.062495] ? watchdog+0x40/0x40 [17840.062500] ? hrtimer_interrupt+0x128/0x2c0 [17840.062504] ? smp_apic_timer_interrupt+0xa6/0x270 [17840.062511] ? apic_timer_interrupt+0xf/0x20 [17840.062516] [17840.062519] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17840.062524] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17840.062529] do_raw_spin_lock+0x177/0x190 [17840.062533] _raw_spin_lock+0x21/0x30 [17840.062537] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17840.063283] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17840.063521] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17840.063583] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17840.063588] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17840.063590] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17840.063599] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17840.063603] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17840.063607] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17840.063609] RDX: 0000000000000920 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17840.063611] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17840.063613] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17840.063616] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17840.063619] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17840.063622] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17840.063625] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17840.063633] Call Trace: [17840.063637] [17840.063640] ? show_regs.cold.9+0x22/0x2f [17840.063650] ? watchdog_timer_fn.cold.6+0x62/0x10a [17840.063656] ? __hrtimer_run_queues+0x1c2/0x4d0 [17840.063660] ? watchdog+0x40/0x40 [17840.063664] ? hrtimer_interrupt+0x128/0x2c0 [17840.063669] ? smp_apic_timer_interrupt+0xa6/0x270 [17840.063676] ? apic_timer_interrupt+0xf/0x20 [17840.063680] [17840.063683] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17840.063687] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17840.063693] do_raw_spin_lock+0x177/0x190 [17840.063698] _raw_spin_lock+0x21/0x30 [17840.063704] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17840.063951] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17840.064474] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17840.065050] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17840.066175] ? woken_wake_function+0x30/0x30 [17840.066186] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17840.066424] kthread+0x1d1/0x200 [17840.066430] ? set_kthread_struct+0x70/0x70 [17840.066434] ret_from_fork+0x1f/0x30 [17840.090385] ? do_raw_spin_unlock+0x75/0x190 [17840.090407] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17840.090699] ? do_wait_intr+0xf0/0xf0 [17840.090707] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17840.090911] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17840.091138] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17840.096511] obd_mod_free+0x46/0x140 [ptlrpc] [17840.101492] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17840.104448] mdc_commit_open+0x91/0xb0 [mdc] [17840.117073] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17840.124963] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17840.128906] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17840.128968] mdc_intent_lock+0x353/0x7e0 [mdc] [17840.129000] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17840.129405] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17840.129846] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17840.129880] lmv_intent_open+0x338/0x1150 [lmv] [17840.129912] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17840.129999] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17840.130162] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17840.130185] ll_intent_lock+0xc5/0xaa0 [lustre] [17840.130366] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17840.130456] ll_lookup_it+0xe0e/0x2270 [lustre] [17840.130542] ? do_raw_spin_unlock+0x75/0x190 [17840.130552] ? _raw_spin_unlock+0x12/0x30 [17840.130560] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17840.130586] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17840.130606] ll_atomic_open+0x36a/0xd70 [lustre] [17840.130849] lookup_open+0x356/0xb40 [17840.130859] do_last+0x313/0xfc0 [17840.130865] ? nd_jump_root+0xe5/0x160 [17840.130870] ? path_init+0x437/0x520 [17840.130875] path_openat+0xf7/0x500 [17840.130880] ? do_raw_spin_unlock+0x75/0x190 [17840.130885] do_filp_open+0x99/0x140 [17840.130892] ? getname_flags+0x6e/0x330 [17840.130897] ? __check_object_size+0xff/0x256 [17840.130905] ? do_raw_spin_unlock+0x75/0x190 [17840.130909] ? _raw_spin_unlock+0x12/0x30 [17840.130914] do_sys_openat2+0x2b4/0x410 [17840.130919] do_sys_open+0x73/0xa0 [17840.130923] __x64_sys_openat+0x24/0x30 [17840.130927] do_syscall_64+0xc1/0x440 [17840.130935] entry_SYSCALL_64_after_hwframe+0x49/0xae [17840.130942] RIP: 0033:0x7f53425a5332 [17840.130950] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17840.130955] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17840.130960] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17840.130963] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17840.130965] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17840.130968] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17840.130970] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17840.646434] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17840.649685] after_reply+0xe2c/0x1780 [ptlrpc] [17840.652317] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17840.655122] ? __next_timer_interrupt+0x160/0x160 [17840.657461] ptlrpcd+0xa82/0xe40 [ptlrpc] [17840.659904] ? do_wait_intr+0xf0/0xf0 [17840.662412] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17840.665527] kthread+0x1d1/0x200 [17840.667451] ? set_kthread_struct+0x70/0x70 [17840.669821] ret_from_fork+0x1f/0x30 [17860.052515] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [17860.056180] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17860.087781] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17860.092390] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17860.095835] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17860.098822] RIP: 0010:native_safe_halt+0x2e/0x50 [17860.100477] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [17860.107102] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17860.109694] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17860.112053] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17860.114520] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17860.117522] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17860.120397] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17860.123037] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17860.126287] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17860.128080] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17860.131108] Call Trace: [17860.132178] [17860.132765] ? show_regs.cold.9+0x22/0x2f [17860.134015] ? watchdog_timer_fn.cold.6+0x62/0x10a [17860.136055] ? __hrtimer_run_queues+0x1c2/0x4d0 [17860.138226] ? watchdog+0x40/0x40 [17860.139405] ? hrtimer_interrupt+0x128/0x2c0 [17860.140841] ? smp_apic_timer_interrupt+0xa6/0x270 [17860.142591] ? apic_timer_interrupt+0xf/0x20 [17860.144403] [17860.145350] ? native_safe_halt+0x2e/0x50 [17860.146752] kvm_wait+0xcd/0xe0 [17860.147973] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17860.149814] ? newidle_balance.isra.93+0x207/0x580 [17860.151821] do_raw_spin_lock+0x177/0x190 [17860.153244] _raw_spin_lock+0x21/0x30 [17860.154510] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17860.156554] process_one_work+0x2c8/0x700 [17860.157696] worker_thread+0x296/0x6e0 [17860.159043] ? rescuer_thread+0x570/0x570 [17860.160441] kthread+0x1d1/0x200 [17860.162222] ? set_kthread_struct+0x70/0x70 [17860.163521] ret_from_fork+0x1f/0x30 [17866.208210] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15229s! [17866.212638] Showing busy workqueues and worker pools: [17866.216401] workqueue events: flags=0x0 [17866.219017] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17866.222073] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17866.229737] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17866.232954] pending: cpu_latency_work [ptlrpc] [17866.237824] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17866.240524] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17866.244485] workqueue events_long: flags=0x0 [17866.254253] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17866.257923] pending: lu_objects_flush [obdclass] [17866.260554] workqueue events_unbound: flags=0x2 [17866.265826] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17866.275284] pending: flush_memcg_stats_dwork, idle_cull_fn [17866.280560] workqueue events_power_efficient: flags=0x80 [17866.284578] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17866.288699] pending: check_lifetime, do_cache_clean [17866.291410] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17866.300754] pending: neigh_periodic_work, neigh_periodic_work [17866.303278] workqueue mm_percpu_wq: flags=0x8 [17866.305247] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17866.308226] pending: vmstat_update, lru_add_drain_per_cpu [17866.311265] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17866.314487] pending: vmstat_update, lru_add_drain_per_cpu [17866.317246] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17866.326219] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17866.334425] workqueue xprtiod: flags=0xa [17866.339414] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17866.351370] pending: xprt_autoclose, xs_error_handle [17866.356915] workqueue ipv6_addrconf: flags=0x40008 [17866.359646] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17866.363068] pending: addrconf_verify_work [17866.364311] workqueue knbd0-recv: flags=0x1a [17866.365778] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17866.368126] in-flight: 651:recv_work [17866.369545] workqueue ptlrpc_pinger: flags=0x2 [17866.371146] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17866.373183] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17866.375828] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15229s workers=4 idle: 15979 31824 8 [17866.379116] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17856s workers=2 idle: 654 [17868.057409] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [17868.060483] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [17868.060525] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [17868.060532] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17868.060665] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17868.060670] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17868.060673] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17868.060687] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17868.060691] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17868.060695] RAX: 0000000000007994 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17868.060697] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17868.060699] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17868.060702] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17868.060705] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17868.060708] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17868.060711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17868.060713] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17868.060722] Call Trace: [17868.060726] [17868.060732] ? show_regs.cold.9+0x22/0x2f [17868.061485] ? watchdog_timer_fn.cold.6+0x62/0x10a [17868.061492] ? __hrtimer_run_queues+0x1c2/0x4d0 [17868.061497] ? watchdog+0x40/0x40 [17868.061501] ? hrtimer_interrupt+0x128/0x2c0 [17868.061505] ? smp_apic_timer_interrupt+0xa6/0x270 [17868.061513] ? apic_timer_interrupt+0xf/0x20 [17868.061518] [17868.061521] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17868.061527] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17868.061531] Modules linked in: [17868.061534] do_raw_spin_lock+0x177/0x190 [17868.061537] lustre(O) osp(O) [17868.061543] _raw_spin_lock+0x21/0x30 [17868.061547] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17868.061619] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17868.061625] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17868.061627] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17868.061554] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17868.061748] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17868.061754] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17868.061759] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17868.061761] RDX: 0000000000001b74 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17868.061764] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17868.061766] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17868.061769] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17868.061772] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17868.061775] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17868.061777] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17868.061783] Call Trace: [17868.061787] [17868.061790] ? show_regs.cold.9+0x22/0x2f [17868.061796] ? watchdog_timer_fn.cold.6+0x62/0x10a [17868.061800] ? __hrtimer_run_queues+0x1c2/0x4d0 [17868.061803] ? watchdog+0x40/0x40 [17868.061808] ? hrtimer_interrupt+0x128/0x2c0 [17868.061811] ? smp_apic_timer_interrupt+0xa6/0x270 [17868.061817] ? apic_timer_interrupt+0xf/0x20 [17868.061822] [17868.061825] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17868.061831] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17868.061835] do_raw_spin_lock+0x177/0x190 [17868.061841] _raw_spin_lock+0x21/0x30 [17868.061846] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17868.062491] obd_mod_free+0x46/0x140 [ptlrpc] [17868.062848] mdc_commit_open+0x91/0xb0 [mdc] [17868.062887] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17868.063159] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17868.061963] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17868.063675] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17868.063448] after_reply+0xe2c/0x1780 [ptlrpc] [17868.064603] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17868.065074] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17868.065284] ? woken_wake_function+0x30/0x30 [17868.065296] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17868.065959] kthread+0x1d1/0x200 [17868.065966] ? set_kthread_struct+0x70/0x70 [17868.065969] ret_from_fork+0x1f/0x30 [17868.067769] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17868.067842] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17868.067850] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17868.067853] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17868.067866] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17868.067870] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17868.067875] RAX: 0000000000005f3c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17868.067878] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17868.067881] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17868.067883] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17868.067886] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17868.067889] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17868.067892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17868.067895] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17868.068110] Call Trace: [17868.068117] [17868.068122] ? show_regs.cold.9+0x22/0x2f [17868.068133] ? watchdog_timer_fn.cold.6+0x62/0x10a [17868.068139] ? __hrtimer_run_queues+0x1c2/0x4d0 [17868.068144] ? watchdog+0x40/0x40 [17868.068149] ? hrtimer_interrupt+0x128/0x2c0 [17868.068154] ? smp_apic_timer_interrupt+0xa6/0x270 [17868.068162] ? apic_timer_interrupt+0xf/0x20 [17868.068166] [17868.068169] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17868.068175] do_raw_spin_lock+0x177/0x190 [17868.068180] _raw_spin_lock+0x21/0x30 [17868.068185] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17868.068533] ? do_raw_spin_unlock+0x75/0x190 [17868.068539] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17868.068750] ? do_wait_intr+0xf0/0xf0 [17868.068760] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17868.069034] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17868.069245] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17868.069534] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17868.069671] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17868.069708] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17868.069738] mdc_intent_lock+0x353/0x7e0 [mdc] [17868.069767] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17868.069851] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17868.074110] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17868.074470] ? __next_timer_interrupt+0x160/0x160 [17868.074483] ptlrpcd+0xa82/0xe40 [ptlrpc] [17868.074820] ? do_wait_intr+0xf0/0xf0 [17868.074830] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17868.075122] kthread+0x1d1/0x200 [17868.075128] ? set_kthread_struct+0x70/0x70 [17868.075132] ret_from_fork+0x1f/0x30 [17868.736671] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17868.738262] lmv_intent_open+0x338/0x1150 [lmv] [17868.739767] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17868.741709] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17868.744017] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17868.745466] ll_intent_lock+0xc5/0xaa0 [lustre] [17868.747154] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17868.748882] ll_lookup_it+0xe0e/0x2270 [lustre] [17868.750736] ? do_raw_spin_unlock+0x75/0x190 [17868.752153] ? _raw_spin_unlock+0x12/0x30 [17868.753459] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17868.756085] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17868.758465] ll_atomic_open+0x36a/0xd70 [lustre] [17868.760106] lookup_open+0x356/0xb40 [17868.764657] do_last+0x313/0xfc0 [17868.765594] ? nd_jump_root+0xe5/0x160 [17868.766594] ? path_init+0x437/0x520 [17868.768631] path_openat+0xf7/0x500 [17868.770557] ? do_raw_spin_unlock+0x75/0x190 [17868.771996] do_filp_open+0x99/0x140 [17868.774751] ? getname_flags+0x6e/0x330 [17868.776134] ? __check_object_size+0xff/0x256 [17868.793510] ? do_raw_spin_unlock+0x75/0x190 [17868.794937] ? _raw_spin_unlock+0x12/0x30 [17868.800815] do_sys_openat2+0x2b4/0x410 [17868.804982] do_sys_open+0x73/0xa0 [17868.807281] __x64_sys_openat+0x24/0x30 [17868.811047] do_syscall_64+0xc1/0x440 [17868.813417] entry_SYSCALL_64_after_hwframe+0x49/0xae [17868.820376] RIP: 0033:0x7f53425a5332 [17868.821622] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17868.833536] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17868.836050] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17868.843415] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17868.848040] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17868.851749] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17868.855425] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17892.052563] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [17892.066482] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17892.148146] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17892.161344] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17892.173426] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17892.182578] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17892.185662] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17892.197044] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17892.200862] RAX: 0000000000002a0c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17892.204062] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17892.207593] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17892.210686] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17892.213434] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17892.215978] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17892.218859] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17892.221796] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17892.225438] Call Trace: [17892.226644] [17892.231360] ? show_regs.cold.9+0x22/0x2f [17892.238632] ? watchdog_timer_fn.cold.6+0x62/0x10a [17892.240350] ? __hrtimer_run_queues+0x1c2/0x4d0 [17892.251255] ? watchdog+0x40/0x40 [17892.252470] ? hrtimer_interrupt+0x128/0x2c0 [17892.257198] ? smp_apic_timer_interrupt+0xa6/0x270 [17892.260754] ? apic_timer_interrupt+0xf/0x20 [17892.263757] [17892.264996] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17892.267960] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17892.271030] ? newidle_balance.isra.93+0x207/0x580 [17892.273510] do_raw_spin_lock+0x177/0x190 [17892.277692] _raw_spin_lock+0x21/0x30 [17892.280694] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17892.282773] process_one_work+0x2c8/0x700 [17892.295009] worker_thread+0x296/0x6e0 [17892.298811] ? rescuer_thread+0x570/0x570 [17892.302110] kthread+0x1d1/0x200 [17892.303964] ? set_kthread_struct+0x70/0x70 [17892.309884] ret_from_fork+0x1f/0x30 [17896.055165] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [17896.069549] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [17896.069600] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [17896.069609] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17896.069676] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17896.069681] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17896.069684] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17896.069698] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17896.069703] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [17896.069707] RAX: 0000000000004983 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17896.069710] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17896.069712] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17896.069714] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17896.069716] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17896.069720] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17896.069722] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17896.069725] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17896.069735] Call Trace: [17896.069741] [17896.069747] ? show_regs.cold.9+0x22/0x2f [17896.069758] ? watchdog_timer_fn.cold.6+0x62/0x10a [17896.069764] ? __hrtimer_run_queues+0x1c2/0x4d0 [17896.069768] ? watchdog+0x40/0x40 [17896.069772] ? hrtimer_interrupt+0x128/0x2c0 [17896.069776] ? smp_apic_timer_interrupt+0xa6/0x270 [17896.069795] ? apic_timer_interrupt+0xf/0x20 [17896.069800] [17896.069803] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17896.069808] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17896.069815] do_raw_spin_lock+0x177/0x190 [17896.069820] _raw_spin_lock+0x21/0x30 [17896.069825] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17896.070105] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17896.070311] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17896.071887] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17896.074642] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17896.074908] ? woken_wake_function+0x30/0x30 [17896.074918] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17896.075176] kthread+0x1d1/0x200 [17896.075182] ? set_kthread_struct+0x70/0x70 [17896.075186] ret_from_fork+0x1f/0x30 [17896.076469] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17896.076544] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17896.076553] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17896.076556] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17896.076570] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17896.076575] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17896.076580] RAX: 0000000000003c65 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17896.076583] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17896.076585] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17896.076587] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17896.076589] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17896.076593] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17896.076596] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17896.076598] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17896.076607] Call Trace: [17896.076612] [17896.076616] ? show_regs.cold.9+0x22/0x2f [17896.076626] ? watchdog_timer_fn.cold.6+0x62/0x10a [17896.076633] ? __hrtimer_run_queues+0x1c2/0x4d0 [17896.076637] ? watchdog+0x40/0x40 [17896.076642] ? hrtimer_interrupt+0x128/0x2c0 [17896.076646] ? smp_apic_timer_interrupt+0xa6/0x270 [17896.076653] ? apic_timer_interrupt+0xf/0x20 [17896.076658] [17896.076660] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17896.076667] do_raw_spin_lock+0x177/0x190 [17896.076672] _raw_spin_lock+0x21/0x30 [17896.076677] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17896.077540] ? do_raw_spin_unlock+0x75/0x190 [17896.077548] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17896.077766] ? do_wait_intr+0xf0/0xf0 [17896.077775] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17896.081678] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17896.082735] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17896.087686] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17896.088237] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17896.088308] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17896.088311] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17896.088326] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17896.088330] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17896.088335] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17896.088338] RDX: 00000000000024ac RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17896.088341] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17896.088343] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17896.088346] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17896.088350] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17896.088353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17896.088356] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17896.088364] Call Trace: [17896.088370] [17896.088376] ? show_regs.cold.9+0x22/0x2f [17896.088385] ? watchdog_timer_fn.cold.6+0x62/0x10a [17896.088452] ? __hrtimer_run_queues+0x1c2/0x4d0 [17896.088457] ? watchdog+0x40/0x40 [17896.088462] ? hrtimer_interrupt+0x128/0x2c0 [17896.088466] ? smp_apic_timer_interrupt+0xa6/0x270 [17896.088475] ? apic_timer_interrupt+0xf/0x20 [17896.088480] [17896.088483] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17896.088489] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17896.088494] do_raw_spin_lock+0x177/0x190 [17896.088499] _raw_spin_lock+0x21/0x30 [17896.088506] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17896.089625] obd_mod_free+0x46/0x140 [ptlrpc] [17896.090312] mdc_commit_open+0x91/0xb0 [mdc] [17896.090356] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17896.090763] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17896.091437] after_reply+0xe2c/0x1780 [ptlrpc] [17896.091636] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17896.102989] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17896.105372] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17896.105415] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17896.105448] mdc_intent_lock+0x353/0x7e0 [mdc] [17896.105479] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17896.105574] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17896.105878] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17896.105917] lmv_intent_open+0x338/0x1150 [lmv] [17896.105961] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17896.106057] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17896.106388] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17896.106411] ll_intent_lock+0xc5/0xaa0 [lustre] [17896.106492] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17896.106573] ll_lookup_it+0xe0e/0x2270 [lustre] [17896.106656] ? do_raw_spin_unlock+0x75/0x190 [17896.106664] ? _raw_spin_unlock+0x12/0x30 [17896.106670] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17896.106701] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17896.106720] ll_atomic_open+0x36a/0xd70 [lustre] [17896.106820] lookup_open+0x356/0xb40 [17896.107491] do_last+0x313/0xfc0 [17896.107499] ? nd_jump_root+0xe5/0x160 [17896.107504] ? path_init+0x437/0x520 [17896.107509] path_openat+0xf7/0x500 [17896.107515] ? do_raw_spin_unlock+0x75/0x190 [17896.107521] do_filp_open+0x99/0x140 [17896.107528] ? getname_flags+0x6e/0x330 [17896.107532] ? __check_object_size+0xff/0x256 [17896.107541] ? do_raw_spin_unlock+0x75/0x190 [17896.107545] ? _raw_spin_unlock+0x12/0x30 [17896.107550] do_sys_openat2+0x2b4/0x410 [17896.107555] do_sys_open+0x73/0xa0 [17896.107559] __x64_sys_openat+0x24/0x30 [17896.107563] do_syscall_64+0xc1/0x440 [17896.107571] entry_SYSCALL_64_after_hwframe+0x49/0xae [17896.107578] RIP: 0033:0x7f53425a5332 [17896.107586] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17896.107591] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17896.107597] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17896.107599] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17896.107602] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17896.107605] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17896.107607] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17897.181586] ? __next_timer_interrupt+0x160/0x160 [17897.192805] ptlrpcd+0xa82/0xe40 [ptlrpc] [17897.194406] ? do_wait_intr+0xf0/0xf0 [17897.195638] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17897.202850] kthread+0x1d1/0x200 [17897.203990] ? set_kthread_struct+0x70/0x70 [17897.210010] ret_from_fork+0x1f/0x30 [17897.214199] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15260s! [17897.219432] Showing busy workqueues and worker pools: [17897.222150] workqueue events: flags=0x0 [17897.223669] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17897.226212] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17897.230545] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17897.232910] pending: cpu_latency_work [ptlrpc] [17897.234920] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17897.237960] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17897.242029] workqueue events_long: flags=0x0 [17897.243656] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17897.246504] pending: lu_objects_flush [obdclass] [17897.248910] workqueue events_unbound: flags=0x2 [17897.250583] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17897.253159] pending: flush_memcg_stats_dwork, idle_cull_fn [17897.255492] workqueue events_power_efficient: flags=0x80 [17897.257671] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17897.260341] pending: check_lifetime, do_cache_clean [17897.270491] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17897.279769] pending: neigh_periodic_work, neigh_periodic_work [17897.284956] workqueue mm_percpu_wq: flags=0x8 [17897.287476] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17897.294626] pending: vmstat_update, lru_add_drain_per_cpu [17897.304700] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17897.315241] pending: vmstat_update, lru_add_drain_per_cpu [17897.320808] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17897.330421] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17897.336706] workqueue xprtiod: flags=0xa [17897.340841] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17897.344188] pending: xprt_autoclose, xs_error_handle [17897.348440] workqueue ipv6_addrconf: flags=0x40008 [17897.351387] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17897.359536] pending: addrconf_verify_work [17897.362109] workqueue knbd0-recv: flags=0x1a [17897.364683] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17897.367741] in-flight: 651:recv_work [17897.371870] workqueue ptlrpc_pinger: flags=0x2 [17897.375630] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17897.381031] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17897.386720] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15260s workers=4 idle: 15979 31824 8 [17897.393603] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17887s workers=2 idle: 654 [17924.054271] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [17924.055418] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [17924.059877] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [17924.059893] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17924.059968] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17924.059973] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17924.059976] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17924.059990] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17924.059994] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17924.059998] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17924.060000] RDX: 0000000000007e21 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17924.060002] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17924.060004] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17924.060006] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17924.060009] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17924.060012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17924.060014] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17924.060023] Call Trace: [17924.060029] [17924.060036] ? show_regs.cold.9+0x22/0x2f [17924.060045] ? watchdog_timer_fn.cold.6+0x62/0x10a [17924.060048] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [17924.060054] ? __hrtimer_run_queues+0x1c2/0x4d0 [17924.060058] Modules linked in: lustre(O) [17924.060063] ? watchdog+0x40/0x40 [17924.060068] osp(O) ofd(O) [17924.060072] ? hrtimer_interrupt+0x128/0x2c0 [17924.060077] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) [17924.060082] ? smp_apic_timer_interrupt+0xa6/0x270 [17924.060088] lfsck(O) obdecho(O) mgc(O) [17924.060095] ? apic_timer_interrupt+0xf/0x20 [17924.060098] mdc(O) lov(O) [17924.060105] [17924.060106] osc(O) lmv(O) ec(O) [17924.060112] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17924.060118] fid(O) fld(O) ptlrpc_gss(O) [17924.060124] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17924.060129] ptlrpc(O) obdclass(O) ksocklnd(O) [17924.060134] do_raw_spin_lock+0x177/0x190 [17924.060139] lnet(O) libcfs(O) [17924.060145] _raw_spin_lock+0x21/0x30 [17924.060147] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17924.060181] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17924.060185] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17924.060187] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17924.060194] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17924.060198] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17924.060202] RAX: 0000000000005c8f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17924.060204] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17924.060205] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17924.060208] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17924.060209] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17924.060212] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17924.060215] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17924.060217] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17924.060224] Call Trace: [17924.060227] [17924.060229] ? show_regs.cold.9+0x22/0x2f [17924.060236] ? watchdog_timer_fn.cold.6+0x62/0x10a [17924.060241] ? __hrtimer_run_queues+0x1c2/0x4d0 [17924.060244] ? watchdog+0x40/0x40 [17924.060248] ? hrtimer_interrupt+0x128/0x2c0 [17924.060253] ? smp_apic_timer_interrupt+0xa6/0x270 [17924.060258] ? apic_timer_interrupt+0xf/0x20 [17924.060263] [17924.060266] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17924.060271] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17924.060276] do_raw_spin_lock+0x177/0x190 [17924.060281] _raw_spin_lock+0x21/0x30 [17924.060158] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17924.060285] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17924.061152] obd_mod_free+0x46/0x140 [ptlrpc] [17924.061788] mdc_commit_open+0x91/0xb0 [mdc] [17924.061825] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17924.062022] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17924.062298] after_reply+0xe2c/0x1780 [ptlrpc] [17924.062491] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17924.062763] ? __next_timer_interrupt+0x160/0x160 [17924.062778] ptlrpcd+0xa82/0xe40 [ptlrpc] [17924.063011] ? do_wait_intr+0xf0/0xf0 [17924.063461] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17924.061250] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17924.063795] kthread+0x1d1/0x200 [17924.063801] ? set_kthread_struct+0x70/0x70 [17924.063805] ret_from_fork+0x1f/0x30 [17924.063882] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17924.064233] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17924.064448] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17924.064590] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17924.064657] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17924.064664] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17924.064857] ? woken_wake_function+0x30/0x30 [17924.064866] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17924.065308] kthread+0x1d1/0x200 [17924.065313] ? set_kthread_struct+0x70/0x70 [17924.065318] ret_from_fork+0x1f/0x30 [17924.064671] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17924.065800] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17924.065809] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17924.065813] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17924.065836] RAX: 0000000000006b06 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17924.065839] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17924.065841] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17924.065843] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17924.065846] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17924.065849] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17924.065852] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17924.065855] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17924.065864] Call Trace: [17924.065868] [17924.065871] ? show_regs.cold.9+0x22/0x2f [17924.065880] ? watchdog_timer_fn.cold.6+0x62/0x10a [17924.065886] ? __hrtimer_run_queues+0x1c2/0x4d0 [17924.065890] ? watchdog+0x40/0x40 [17924.065894] ? hrtimer_interrupt+0x128/0x2c0 [17924.065897] ? smp_apic_timer_interrupt+0xa6/0x270 [17924.065903] ? apic_timer_interrupt+0xf/0x20 [17924.065908] [17924.065910] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17924.065915] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17924.065918] ? newidle_balance.isra.93+0x207/0x580 [17924.065923] do_raw_spin_lock+0x177/0x190 [17924.065928] _raw_spin_lock+0x21/0x30 [17924.065932] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17924.066538] process_one_work+0x2c8/0x700 [17924.066549] worker_thread+0x296/0x6e0 [17924.066553] ? rescuer_thread+0x570/0x570 [17924.066557] kthread+0x1d1/0x200 [17924.066561] ? set_kthread_struct+0x70/0x70 [17924.066564] ret_from_fork+0x1f/0x30 [17925.121321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17925.173820] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17925.189479] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17925.204706] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17925.215833] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17925.248196] RSP: 0018:ffffbb5381503500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [17925.252974] RAX: 0000000000007009 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17925.257077] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17925.265229] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17925.273692] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17925.281136] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17925.292265] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17925.300729] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17925.305853] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17925.313456] Call Trace: [17925.316158] [17925.317842] ? show_regs.cold.9+0x22/0x2f [17925.320147] ? watchdog_timer_fn.cold.6+0x62/0x10a [17925.322643] ? __hrtimer_run_queues+0x1c2/0x4d0 [17925.325495] ? watchdog+0x40/0x40 [17925.327622] ? hrtimer_interrupt+0x128/0x2c0 [17925.330262] ? smp_apic_timer_interrupt+0xa6/0x270 [17925.333087] ? apic_timer_interrupt+0xf/0x20 [17925.335614] [17925.336908] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17925.339241] do_raw_spin_lock+0x177/0x190 [17925.340985] _raw_spin_lock+0x21/0x30 [17925.342323] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17925.344328] ? do_raw_spin_unlock+0x75/0x190 [17925.346509] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17925.351243] ? do_wait_intr+0xf0/0xf0 [17925.352621] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17925.354534] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17925.360231] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17925.365639] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17925.370998] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17925.372973] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17925.377961] mdc_intent_lock+0x353/0x7e0 [mdc] [17925.386886] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17925.391386] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17925.398102] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17925.408151] lmv_intent_open+0x338/0x1150 [lmv] [17925.410512] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17925.420231] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17925.428544] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17925.441086] ll_intent_lock+0xc5/0xaa0 [lustre] [17925.450521] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17925.465096] ll_lookup_it+0xe0e/0x2270 [lustre] [17925.467497] ? do_raw_spin_unlock+0x75/0x190 [17925.470407] ? _raw_spin_unlock+0x12/0x30 [17925.472256] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17925.485361] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17925.490035] ll_atomic_open+0x36a/0xd70 [lustre] [17925.497511] lookup_open+0x356/0xb40 [17925.504375] do_last+0x313/0xfc0 [17925.508415] ? nd_jump_root+0xe5/0x160 [17925.511276] ? path_init+0x437/0x520 [17925.512359] path_openat+0xf7/0x500 [17925.514919] ? do_raw_spin_unlock+0x75/0x190 [17925.519583] do_filp_open+0x99/0x140 [17925.523046] ? getname_flags+0x6e/0x330 [17925.526071] ? __check_object_size+0xff/0x256 [17925.529638] ? do_raw_spin_unlock+0x75/0x190 [17925.532494] ? _raw_spin_unlock+0x12/0x30 [17925.537393] do_sys_openat2+0x2b4/0x410 [17925.540146] do_sys_open+0x73/0xa0 [17925.542435] __x64_sys_openat+0x24/0x30 [17925.547800] do_syscall_64+0xc1/0x440 [17925.554706] entry_SYSCALL_64_after_hwframe+0x49/0xae [17925.563781] RIP: 0033:0x7f53425a5332 [17925.566280] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17925.579291] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17925.589607] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17925.591437] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17925.599797] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17925.606508] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17925.617170] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17927.657408] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15290s! [17927.667829] Showing busy workqueues and worker pools: [17927.680633] workqueue events: flags=0x0 [17927.687908] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17927.690289] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17927.720890] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17927.723166] pending: cpu_latency_work [ptlrpc] [17927.738202] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17927.748932] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17927.764722] workqueue events_long: flags=0x0 [17927.770362] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17927.778409] pending: lu_objects_flush [obdclass] [17927.780212] workqueue events_unbound: flags=0x2 [17927.792346] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17927.804999] pending: flush_memcg_stats_dwork, idle_cull_fn [17927.813496] workqueue events_power_efficient: flags=0x80 [17927.818340] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17927.824796] pending: check_lifetime, do_cache_clean [17927.829447] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17927.834464] pending: neigh_periodic_work, neigh_periodic_work [17927.842187] workqueue mm_percpu_wq: flags=0x8 [17927.843703] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17927.859291] pending: vmstat_update, lru_add_drain_per_cpu [17927.869655] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17927.871717] pending: vmstat_update, lru_add_drain_per_cpu [17927.884290] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17927.896243] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17927.898056] workqueue xprtiod: flags=0xa [17927.906035] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17927.915007] pending: xprt_autoclose, xs_error_handle [17927.923010] workqueue ipv6_addrconf: flags=0x40008 [17927.935259] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17927.956589] pending: addrconf_verify_work [17927.957941] workqueue knbd0-recv: flags=0x1a [17927.959280] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17927.973780] in-flight: 651:recv_work [17927.979807] workqueue ptlrpc_pinger: flags=0x2 [17927.985320] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17927.997476] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17928.009773] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15290s workers=4 idle: 15979 31824 8 [17928.016306] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17918s workers=2 idle: 654 [17952.052194] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [17952.055366] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17952.062615] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [17952.062681] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [17952.062688] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17952.062757] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17952.062762] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17952.062765] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17952.062784] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17952.062789] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17952.062793] RAX: 0000000000005a05 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17952.062795] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17952.062797] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17952.062799] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17952.062801] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17952.062804] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17952.062807] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17952.062809] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17952.062817] Call Trace: [17952.062822] [17952.062827] ? show_regs.cold.9+0x22/0x2f [17952.062836] ? watchdog_timer_fn.cold.6+0x62/0x10a [17952.062842] ? __hrtimer_run_queues+0x1c2/0x4d0 [17952.062847] ? watchdog+0x40/0x40 [17952.062851] ? hrtimer_interrupt+0x128/0x2c0 [17952.062854] ? smp_apic_timer_interrupt+0xa6/0x270 [17952.062863] ? apic_timer_interrupt+0xf/0x20 [17952.062868] [17952.062870] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17952.062877] do_raw_spin_lock+0x177/0x190 [17952.062881] _raw_spin_lock+0x21/0x30 [17952.064634] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [17952.064644] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17952.064712] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17952.064717] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17952.064719] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17952.064731] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17952.064734] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17952.064738] RAX: 0000000000006950 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17952.064741] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17952.064743] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17952.064745] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17952.064747] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17952.064750] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17952.064752] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17952.064754] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17952.064761] Call Trace: [17952.064766] [17952.064769] ? show_regs.cold.9+0x22/0x2f [17952.064778] ? watchdog_timer_fn.cold.6+0x62/0x10a [17952.064784] ? __hrtimer_run_queues+0x1c2/0x4d0 [17952.064788] ? watchdog+0x40/0x40 [17952.064792] ? hrtimer_interrupt+0x128/0x2c0 [17952.064796] ? smp_apic_timer_interrupt+0xa6/0x270 [17952.064804] ? apic_timer_interrupt+0xf/0x20 [17952.064809] [17952.064812] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17952.064817] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17952.064823] do_raw_spin_lock+0x177/0x190 [17952.064828] _raw_spin_lock+0x21/0x30 [17952.064832] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17952.065183] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17952.065386] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17952.065595] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17952.065808] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17952.062885] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17952.067150] ? do_raw_spin_unlock+0x75/0x190 [17952.067160] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17952.067401] ? do_wait_intr+0xf0/0xf0 [17952.067411] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17952.067759] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17952.067986] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17952.068425] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17952.068552] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17952.068590] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17952.068688] mdc_intent_lock+0x353/0x7e0 [mdc] [17952.068720] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17952.068803] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17952.069032] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17952.070205] ? woken_wake_function+0x30/0x30 [17952.070166] lmv_intent_open+0x338/0x1150 [lmv] [17952.070218] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17952.070600] kthread+0x1d1/0x200 [17952.070607] ? set_kthread_struct+0x70/0x70 [17952.070612] ret_from_fork+0x1f/0x30 [17952.070352] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17952.071361] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17952.072391] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17952.073295] ll_intent_lock+0xc5/0xaa0 [lustre] [17952.073771] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17952.075923] ll_lookup_it+0xe0e/0x2270 [lustre] [17952.076034] ? do_raw_spin_unlock+0x75/0x190 [17952.076043] ? _raw_spin_unlock+0x12/0x30 [17952.076051] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17952.076079] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17952.076112] ll_atomic_open+0x36a/0xd70 [lustre] [17952.076356] lookup_open+0x356/0xb40 [17952.076368] do_last+0x313/0xfc0 [17952.076373] ? nd_jump_root+0xe5/0x160 [17952.076378] ? path_init+0x437/0x520 [17952.076384] path_openat+0xf7/0x500 [17952.076390] ? do_raw_spin_unlock+0x75/0x190 [17952.076395] do_filp_open+0x99/0x140 [17952.076403] ? getname_flags+0x6e/0x330 [17952.076415] ? __check_object_size+0xff/0x256 [17952.076423] ? do_raw_spin_unlock+0x75/0x190 [17952.076427] ? _raw_spin_unlock+0x12/0x30 [17952.076432] do_sys_openat2+0x2b4/0x410 [17952.076438] do_sys_open+0x73/0xa0 [17952.076443] __x64_sys_openat+0x24/0x30 [17952.076447] do_syscall_64+0xc1/0x440 [17952.076455] entry_SYSCALL_64_after_hwframe+0x49/0xae [17952.076461] RIP: 0033:0x7f53425a5332 [17952.076470] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17952.076861] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17952.076869] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17952.076871] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17952.076873] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17952.076875] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17952.076878] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17952.094635] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17952.096039] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17952.096048] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17952.097051] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17952.097064] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17952.097068] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17952.097072] RAX: 0000000000003de8 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17952.097075] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17952.098516] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17952.109440] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17952.109543] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17952.109556] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17952.109559] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17952.109573] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17952.109577] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17952.109582] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17952.109584] RDX: 0000000000006238 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17952.109586] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17952.109588] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17952.109591] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17952.109594] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17952.109597] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17952.109599] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17952.109609] Call Trace: [17952.109615] [17952.109621] ? show_regs.cold.9+0x22/0x2f [17952.109632] ? watchdog_timer_fn.cold.6+0x62/0x10a [17952.109639] ? __hrtimer_run_queues+0x1c2/0x4d0 [17952.109644] ? watchdog+0x40/0x40 [17952.109649] ? hrtimer_interrupt+0x128/0x2c0 [17952.109653] ? smp_apic_timer_interrupt+0xa6/0x270 [17952.109662] ? apic_timer_interrupt+0xf/0x20 [17952.109667] [17952.109677] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17952.109682] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17952.109686] do_raw_spin_lock+0x177/0x190 [17952.109691] _raw_spin_lock+0x21/0x30 [17952.109700] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17952.110388] obd_mod_free+0x46/0x140 [ptlrpc] [17952.110700] mdc_commit_open+0x91/0xb0 [mdc] [17952.110752] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17952.115043] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17952.125387] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17952.125398] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17952.125402] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17952.125404] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17952.125406] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17952.125415] Call Trace: [17952.125423] [17952.125429] ? show_regs.cold.9+0x22/0x2f [17952.125442] ? watchdog_timer_fn.cold.6+0x62/0x10a [17952.125448] ? __hrtimer_run_queues+0x1c2/0x4d0 [17952.125453] ? watchdog+0x40/0x40 [17952.125457] ? hrtimer_interrupt+0x128/0x2c0 [17952.125461] ? smp_apic_timer_interrupt+0xa6/0x270 [17952.125470] ? apic_timer_interrupt+0xf/0x20 [17952.125474] [17952.125477] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17952.125483] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17952.125487] ? newidle_balance.isra.93+0x207/0x580 [17952.125495] do_raw_spin_lock+0x177/0x190 [17952.125500] _raw_spin_lock+0x21/0x30 [17952.125506] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17952.125786] process_one_work+0x2c8/0x700 [17952.125794] worker_thread+0x296/0x6e0 [17952.125798] ? rescuer_thread+0x570/0x570 [17952.125802] kthread+0x1d1/0x200 [17952.125806] ? set_kthread_struct+0x70/0x70 [17952.125810] ret_from_fork+0x1f/0x30 [17953.492270] after_reply+0xe2c/0x1780 [ptlrpc] [17953.494301] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17953.496668] ? __next_timer_interrupt+0x160/0x160 [17953.500467] ptlrpcd+0xa82/0xe40 [ptlrpc] [17953.502337] ? do_wait_intr+0xf0/0xf0 [17953.510792] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17953.515435] kthread+0x1d1/0x200 [17953.516898] ? set_kthread_struct+0x70/0x70 [17953.518756] ret_from_fork+0x1f/0x30 [17958.374576] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15321s! [17958.381857] Showing busy workqueues and worker pools: [17958.389048] workqueue events: flags=0x0 [17958.391819] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17958.396705] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17958.402281] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17958.406447] pending: cpu_latency_work [ptlrpc] [17958.407907] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17958.412582] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17958.422355] workqueue events_long: flags=0x0 [17958.423589] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17958.435019] pending: lu_objects_flush [obdclass] [17958.439197] workqueue events_unbound: flags=0x2 [17958.442219] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17958.449082] pending: flush_memcg_stats_dwork, idle_cull_fn [17958.453382] workqueue events_power_efficient: flags=0x80 [17958.455623] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17958.458889] pending: check_lifetime, do_cache_clean [17958.460807] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17958.462904] pending: neigh_periodic_work, neigh_periodic_work [17958.467778] workqueue mm_percpu_wq: flags=0x8 [17958.469638] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17958.473103] pending: vmstat_update, lru_add_drain_per_cpu [17958.476176] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17958.479330] pending: vmstat_update, lru_add_drain_per_cpu [17958.481919] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17958.490418] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17958.494938] workqueue xprtiod: flags=0xa [17958.498199] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17958.502319] pending: xprt_autoclose, xs_error_handle [17958.508602] workqueue ipv6_addrconf: flags=0x40008 [17958.514283] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17958.522125] pending: addrconf_verify_work [17958.533210] workqueue knbd0-recv: flags=0x1a [17958.534721] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17958.546962] in-flight: 651:recv_work [17958.552164] workqueue ptlrpc_pinger: flags=0x2 [17958.557935] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17958.566546] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17958.571347] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15321s workers=4 idle: 15979 31824 8 [17958.577263] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17949s workers=2 idle: 654 [17980.056278] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [17980.057021] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [17980.058094] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [17980.058101] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17980.058698] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17980.058730] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17980.058733] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17980.058747] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17980.058751] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [17980.058756] RAX: 0000000000004b85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17980.058758] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [17980.058760] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [17980.058762] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [17980.058764] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [17980.058768] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [17980.058770] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17980.058773] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [17980.058781] Call Trace: [17980.058787] [17980.058793] ? show_regs.cold.9+0x22/0x2f [17980.058802] ? watchdog_timer_fn.cold.6+0x62/0x10a [17980.058809] ? __hrtimer_run_queues+0x1c2/0x4d0 [17980.058813] ? watchdog+0x40/0x40 [17980.058817] ? hrtimer_interrupt+0x128/0x2c0 [17980.058822] ? smp_apic_timer_interrupt+0xa6/0x270 [17980.058831] ? apic_timer_interrupt+0xf/0x20 [17980.059450] [17980.059454] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17980.059460] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17980.059466] do_raw_spin_lock+0x177/0x190 [17980.059471] _raw_spin_lock+0x21/0x30 [17980.059477] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [17980.061594] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [17980.062918] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [17980.063176] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [17980.063476] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [17980.063776] ? woken_wake_function+0x30/0x30 [17980.063786] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [17980.064822] kthread+0x1d1/0x200 [17980.064829] ? set_kthread_struct+0x70/0x70 [17980.064832] ret_from_fork+0x1f/0x30 [17980.068870] Modules linked in: lustre(O) osp(O) [17980.072350] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17980.072631] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17980.072638] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17980.072641] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [17980.072655] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [17980.072659] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17980.072663] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [17980.072670] RDX: 0000000000002704 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [17980.072673] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [17980.072675] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [17980.072677] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [17980.072680] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [17980.072683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17980.072686] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [17980.072693] Call Trace: [17980.072699] [17980.072704] ? show_regs.cold.9+0x22/0x2f [17980.072721] ? watchdog_timer_fn.cold.6+0x62/0x10a [17980.072727] ? __hrtimer_run_queues+0x1c2/0x4d0 [17980.072732] ? watchdog+0x40/0x40 [17980.072736] ? hrtimer_interrupt+0x128/0x2c0 [17980.072740] ? smp_apic_timer_interrupt+0xa6/0x270 [17980.072748] ? apic_timer_interrupt+0xf/0x20 [17980.072753] [17980.072756] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17980.072760] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17980.072765] do_raw_spin_lock+0x177/0x190 [17980.072842] _raw_spin_lock+0x21/0x30 [17980.072850] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17980.073151] obd_mod_free+0x46/0x140 [ptlrpc] [17980.073377] mdc_commit_open+0x91/0xb0 [mdc] [17980.073427] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17980.073892] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17980.074123] after_reply+0xe2c/0x1780 [ptlrpc] [17980.074348] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17980.086007] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17980.086088] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17980.086096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17980.086099] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17980.086115] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17980.086119] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17980.086124] RAX: 000000000000770e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17980.086127] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17980.086130] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17980.086132] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17980.086134] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17980.086138] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17980.086141] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17980.086143] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17980.086152] Call Trace: [17980.086158] [17980.086165] ? show_regs.cold.9+0x22/0x2f [17980.086175] ? watchdog_timer_fn.cold.6+0x62/0x10a [17980.086180] ? __hrtimer_run_queues+0x1c2/0x4d0 [17980.086185] ? watchdog+0x40/0x40 [17980.086189] ? hrtimer_interrupt+0x128/0x2c0 [17980.086194] ? smp_apic_timer_interrupt+0xa6/0x270 [17980.086202] ? apic_timer_interrupt+0xf/0x20 [17980.086208] [17980.086211] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17980.086218] do_raw_spin_lock+0x177/0x190 [17980.086223] _raw_spin_lock+0x21/0x30 [17980.086229] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17980.087543] ? do_raw_spin_unlock+0x75/0x190 [17980.087551] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17980.088233] ? do_wait_intr+0xf0/0xf0 [17980.088243] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17980.089222] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17980.090368] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17980.136147] ? __next_timer_interrupt+0x160/0x160 [17980.136171] ptlrpcd+0xa82/0xe40 [ptlrpc] [17980.136440] ? do_wait_intr+0xf0/0xf0 [17980.136450] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17980.136674] kthread+0x1d1/0x200 [17980.136680] ? set_kthread_struct+0x70/0x70 [17980.136683] ret_from_fork+0x1f/0x30 [17980.745452] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17980.750521] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17980.756628] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17980.764897] mdc_intent_lock+0x353/0x7e0 [mdc] [17980.769866] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17980.777250] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17980.789431] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17980.795363] lmv_intent_open+0x338/0x1150 [lmv] [17980.797766] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17980.809555] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17980.812301] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17980.815876] ll_intent_lock+0xc5/0xaa0 [lustre] [17980.823008] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17980.827920] ll_lookup_it+0xe0e/0x2270 [lustre] [17980.831295] ? do_raw_spin_unlock+0x75/0x190 [17980.832770] ? _raw_spin_unlock+0x12/0x30 [17980.836321] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17980.838521] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17980.840705] ll_atomic_open+0x36a/0xd70 [lustre] [17980.842355] lookup_open+0x356/0xb40 [17980.843459] do_last+0x313/0xfc0 [17980.847875] ? nd_jump_root+0xe5/0x160 [17980.852166] ? path_init+0x437/0x520 [17980.856073] path_openat+0xf7/0x500 [17980.857376] ? do_raw_spin_unlock+0x75/0x190 [17980.859267] do_filp_open+0x99/0x140 [17980.861186] ? getname_flags+0x6e/0x330 [17980.862581] ? __check_object_size+0xff/0x256 [17980.864326] ? do_raw_spin_unlock+0x75/0x190 [17980.865860] ? _raw_spin_unlock+0x12/0x30 [17980.867238] do_sys_openat2+0x2b4/0x410 [17980.869017] do_sys_open+0x73/0xa0 [17980.870269] __x64_sys_openat+0x24/0x30 [17980.871574] do_syscall_64+0xc1/0x440 [17980.873374] entry_SYSCALL_64_after_hwframe+0x49/0xae [17980.875747] RIP: 0033:0x7f53425a5332 [17980.877237] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17980.896215] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17980.906444] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17980.911824] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17980.917195] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17980.923666] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17980.926529] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17984.055492] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [17984.062850] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [17984.100270] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17984.106340] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17984.111340] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17984.115749] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17984.120560] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17984.130255] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [17984.134481] RAX: 0000000000002a8f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17984.139504] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17984.143433] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17984.147433] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17984.151263] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17984.155096] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17984.158794] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17984.162623] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17984.167090] Call Trace: [17984.168613] [17984.169605] ? show_regs.cold.9+0x22/0x2f [17984.171797] ? watchdog_timer_fn.cold.6+0x62/0x10a [17984.174103] ? __hrtimer_run_queues+0x1c2/0x4d0 [17984.175963] ? watchdog+0x40/0x40 [17984.177363] ? hrtimer_interrupt+0x128/0x2c0 [17984.179468] ? smp_apic_timer_interrupt+0xa6/0x270 [17984.182104] ? apic_timer_interrupt+0xf/0x20 [17984.184130] [17984.185145] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17984.187824] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [17984.190349] ? newidle_balance.isra.93+0x207/0x580 [17984.192800] do_raw_spin_lock+0x177/0x190 [17984.194763] _raw_spin_lock+0x21/0x30 [17984.196484] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17984.199600] process_one_work+0x2c8/0x700 [17984.201617] worker_thread+0x296/0x6e0 [17984.203346] ? rescuer_thread+0x570/0x570 [17984.205317] kthread+0x1d1/0x200 [17984.206841] ? set_kthread_struct+0x70/0x70 [17984.208663] ret_from_fork+0x1f/0x30 [17989.091246] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15352s! [17989.103647] Showing busy workqueues and worker pools: [17989.111312] workqueue events: flags=0x0 [17989.121530] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [17989.137628] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [17989.144116] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17989.145920] pending: cpu_latency_work [ptlrpc] [17989.152082] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [17989.158167] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [17989.166108] workqueue events_long: flags=0x0 [17989.168590] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [17989.171652] pending: lu_objects_flush [obdclass] [17989.175619] workqueue events_unbound: flags=0x2 [17989.182765] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [17989.188158] pending: flush_memcg_stats_dwork, idle_cull_fn [17989.191972] workqueue events_power_efficient: flags=0x80 [17989.195314] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17989.203860] pending: check_lifetime, do_cache_clean [17989.213436] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17989.218574] pending: neigh_periodic_work, neigh_periodic_work [17989.235347] workqueue mm_percpu_wq: flags=0x8 [17989.237146] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17989.242935] pending: vmstat_update, lru_add_drain_per_cpu [17989.246296] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [17989.251187] pending: vmstat_update, lru_add_drain_per_cpu [17989.255705] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [17989.260006] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [17989.263713] workqueue xprtiod: flags=0xa [17989.268701] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [17989.274796] pending: xprt_autoclose, xs_error_handle [17989.276683] workqueue ipv6_addrconf: flags=0x40008 [17989.279425] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [17989.282372] pending: addrconf_verify_work [17989.284159] workqueue knbd0-recv: flags=0x1a [17989.290575] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [17989.295579] in-flight: 651:recv_work [17989.297067] workqueue ptlrpc_pinger: flags=0x2 [17989.299133] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [17989.303732] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [17989.307174] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15352s workers=4 idle: 15979 31824 8 [17989.314165] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=17979s workers=2 idle: 654 [17994.818804] rcu: INFO: rcu_sched self-detected stall on CPU [17994.824708] rcu: 2-....: (8906123 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [17994.855447] (t=15900392 jiffies g=274185 q=68034) [17994.863465] rcu: rcu_sched kthread starved for 15358758 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [17994.883668] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [17994.901433] rcu: RCU grace-period kthread stack dump: [17994.905800] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [17994.911826] Call Trace: [17994.913044] __schedule+0x351/0xcb0 [17994.915749] schedule+0xc0/0x180 [17994.920986] schedule_timeout+0xb4/0x190 [17994.922255] ? do_raw_spin_unlock+0x75/0x190 [17994.925956] ? __next_timer_interrupt+0x160/0x160 [17994.929379] ? prepare_to_swait_event+0xf3/0x140 [17994.938520] rcu_gp_kthread+0x8dd/0x15b0 [17994.939628] ? force_qs_rnp+0x2c0/0x2c0 [17994.940950] kthread+0x1d1/0x200 [17994.946980] ? set_kthread_struct+0x70/0x70 [17994.951951] ret_from_fork+0x1f/0x30 [17994.955390] rcu: Stack dump where RCU GP kthread last ran: [17994.957123] Sending NMI from CPU 2 to CPUs 0: [17994.964253] NMI backtrace for cpu 0 [17994.964259] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17994.964262] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17994.964263] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [17994.964267] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17994.964268] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17994.964271] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000212 [17994.964273] RAX: 0000000000003d93 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17994.964274] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [17994.964276] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [17994.964277] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [17994.964278] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [17994.964280] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [17994.964281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17994.964282] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [17994.964283] Call Trace: [17994.964284] [17994.964285] ? show_regs.cold.9+0x22/0x2f [17994.964286] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [17994.964287] ? nmi_cpu_backtrace_handler+0x15/0x40 [17994.964288] ? nmi_handle+0xbd/0x210 [17994.964289] ? default_do_nmi+0x4b/0x1b0 [17994.964290] ? do_nmi+0x280/0x380 [17994.964291] ? end_repeat_nmi+0x16/0x1f [17994.964292] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17994.964293] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17994.964294] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17994.964295] [17994.964296] ? newidle_balance.isra.93+0x207/0x580 [17994.964338] do_raw_spin_lock+0x177/0x190 [17994.964348] _raw_spin_lock+0x21/0x30 [17994.964349] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [17994.964350] process_one_work+0x2c8/0x700 [17994.964351] worker_thread+0x296/0x6e0 [17994.964352] ? rescuer_thread+0x570/0x570 [17994.964353] kthread+0x1d1/0x200 [17994.964354] ? set_kthread_struct+0x70/0x70 [17994.964355] ret_from_fork+0x1f/0x30 [17994.966380] Sending NMI from CPU 2 to CPUs 1: [17995.128910] NMI backtrace for cpu 1 [17995.128918] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17995.128920] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17995.128921] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17995.128923] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [17995.128925] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [17995.128928] RAX: 0000000000002c91 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [17995.128929] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [17995.128931] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [17995.128932] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [17995.128934] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [17995.128935] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [17995.128936] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [17995.128937] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [17995.128938] Call Trace: [17995.128939] [17995.128940] ? show_regs.cold.9+0x22/0x2f [17995.128941] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [17995.128942] ? nmi_cpu_backtrace_handler+0x15/0x40 [17995.128943] ? nmi_handle+0xbd/0x210 [17995.128944] ? default_do_nmi+0x4b/0x1b0 [17995.128945] ? do_nmi+0x280/0x380 [17995.128947] ? end_repeat_nmi+0x16/0x1f [17995.128948] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17995.128949] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17995.128951] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [17995.128952] [17995.128953] do_raw_spin_lock+0x177/0x190 [17995.128954] _raw_spin_lock+0x21/0x30 [17995.128955] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [17995.128956] ? do_raw_spin_unlock+0x75/0x190 [17995.128958] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [17995.128959] ? do_wait_intr+0xf0/0xf0 [17995.128960] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [17995.128961] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [17995.128963] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17995.128964] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17995.128965] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17995.128967] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [17995.128968] mdc_intent_lock+0x353/0x7e0 [mdc] [17995.128969] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17995.128971] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [17995.128972] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [17995.128973] lmv_intent_open+0x338/0x1150 [lmv] [17995.128975] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17995.128976] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17995.128978] lmv_intent_lock+0x1dd/0x4c0 [lmv] [17995.128979] ll_intent_lock+0xc5/0xaa0 [lustre] [17995.128980] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [17995.128982] ll_lookup_it+0xe0e/0x2270 [lustre] [17995.128983] ? do_raw_spin_unlock+0x75/0x190 [17995.128985] ? _raw_spin_unlock+0x12/0x30 [17995.128986] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [17995.128987] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [17995.128988] ll_atomic_open+0x36a/0xd70 [lustre] [17995.128989] lookup_open+0x356/0xb40 [17995.128990] do_last+0x313/0xfc0 [17995.128991] ? nd_jump_root+0xe5/0x160 [17995.128992] ? path_init+0x437/0x520 [17995.128993] path_openat+0xf7/0x500 [17995.128994] ? do_raw_spin_unlock+0x75/0x190 [17995.128995] do_filp_open+0x99/0x140 [17995.128996] ? getname_flags+0x6e/0x330 [17995.128997] ? __check_object_size+0xff/0x256 [17995.128998] ? do_raw_spin_unlock+0x75/0x190 [17995.128999] ? _raw_spin_unlock+0x12/0x30 [17995.129001] do_sys_openat2+0x2b4/0x410 [17995.129002] do_sys_open+0x73/0xa0 [17995.129003] __x64_sys_openat+0x24/0x30 [17995.129004] do_syscall_64+0xc1/0x440 [17995.129005] entry_SYSCALL_64_after_hwframe+0x49/0xae [17995.129007] RIP: 0033:0x7f53425a5332 [17995.129008] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [17995.129010] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [17995.129013] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [17995.129014] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [17995.129015] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [17995.129016] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [17995.129017] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [17995.129496] NMI backtrace for cpu 2 [17995.475978] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [17995.486957] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [17995.493862] Call Trace: [17995.495148] [17995.497992] ? dump_stack+0xbb/0x10e [17995.499927] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [17995.502388] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [17995.511651] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [17995.513936] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [17995.520213] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [17995.525169] ? rcu_dump_cpu_stacks+0x11b/0x171 [17995.530099] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [17995.536141] ? update_process_times+0x6d/0xc0 [17995.543433] ? tick_sched_handle.isra.18+0x27/0x90 [17995.546768] ? tick_sched_timer+0x53/0xb0 [17995.552732] ? __hrtimer_run_queues+0x1c2/0x4d0 [17995.559602] ? tick_sched_handle.isra.18+0x90/0x90 [17995.567383] ? hrtimer_interrupt+0x128/0x2c0 [17995.570924] ? smp_apic_timer_interrupt+0xa6/0x270 [17995.573870] ? apic_timer_interrupt+0xf/0x20 [17995.577579] [17995.578342] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [17995.582879] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [17995.591160] ? do_raw_spin_lock+0x177/0x190 [17995.597272] ? _raw_spin_lock+0x21/0x30 [17995.601419] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [17995.609702] ? obd_mod_free+0x46/0x140 [ptlrpc] [17995.614062] ? mdc_commit_open+0x91/0xb0 [mdc] [17995.617544] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [17995.621873] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [17995.626178] ? after_reply+0xe2c/0x1780 [ptlrpc] [17995.631795] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [17995.635310] ? __next_timer_interrupt+0x160/0x160 [17995.640200] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [17995.646082] ? do_wait_intr+0xf0/0xf0 [17995.649170] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [17995.652230] ? kthread+0x1d1/0x200 [17995.655120] ? set_kthread_struct+0x70/0x70 [17995.657975] ? ret_from_fork+0x1f/0x30 [18008.058148] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [18008.062385] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18008.091279] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18008.103573] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18008.117059] RIP: 0010:native_safe_halt+0x2e/0x50 [18008.121765] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18008.137058] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18008.144015] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18008.150186] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18008.152557] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18008.172588] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18008.179569] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18008.193137] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18008.202451] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18008.209498] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18008.217437] Call Trace: [18008.221197] [18008.224375] ? show_regs.cold.9+0x22/0x2f [18008.228424] ? watchdog_timer_fn.cold.6+0x62/0x10a [18008.232638] ? __hrtimer_run_queues+0x1c2/0x4d0 [18008.236832] ? watchdog+0x40/0x40 [18008.239661] ? hrtimer_interrupt+0x128/0x2c0 [18008.242625] ? smp_apic_timer_interrupt+0xa6/0x270 [18008.247594] ? apic_timer_interrupt+0xf/0x20 [18008.251677] [18008.254867] ? native_safe_halt+0x2e/0x50 [18008.260559] kvm_wait+0xcd/0xe0 [18008.261942] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18008.263258] do_raw_spin_lock+0x177/0x190 [18008.271687] _raw_spin_lock+0x21/0x30 [18008.276081] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18008.285591] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18008.293480] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18008.297870] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18008.303641] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18008.309912] ? woken_wake_function+0x30/0x30 [18008.311368] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18008.316818] kthread+0x1d1/0x200 [18008.317918] ? set_kthread_struct+0x70/0x70 [18008.319882] ret_from_fork+0x1f/0x30 [18019.808037] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15382s! [18019.810499] Showing busy workqueues and worker pools: [18019.811796] workqueue events: flags=0x0 [18019.815122] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18019.818665] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18019.823894] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18019.829879] pending: cpu_latency_work [ptlrpc] [18019.832012] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18019.838453] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18019.846832] workqueue events_long: flags=0x0 [18019.849457] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18019.855434] pending: lu_objects_flush [obdclass] [18019.860656] workqueue events_unbound: flags=0x2 [18019.863589] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18019.867622] pending: flush_memcg_stats_dwork, idle_cull_fn [18019.872080] workqueue events_power_efficient: flags=0x80 [18019.877710] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18019.883611] pending: check_lifetime, do_cache_clean [18019.887644] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18019.896407] pending: neigh_periodic_work, neigh_periodic_work [18019.903298] workqueue mm_percpu_wq: flags=0x8 [18019.906549] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18019.909860] pending: vmstat_update, lru_add_drain_per_cpu [18019.912854] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18019.917640] pending: vmstat_update, lru_add_drain_per_cpu [18019.921711] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18019.929557] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18019.937707] workqueue xprtiod: flags=0xa [18019.940989] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18019.948026] pending: xprt_autoclose, xs_error_handle [18019.950656] workqueue ipv6_addrconf: flags=0x40008 [18019.952678] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18019.957497] pending: addrconf_verify_work [18019.960042] workqueue knbd0-recv: flags=0x1a [18019.963670] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18019.969497] in-flight: 651:recv_work [18019.979071] workqueue ptlrpc_pinger: flags=0x2 [18019.984595] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18019.990645] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18019.997553] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15382s workers=4 idle: 15979 31824 8 [18020.002124] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18010s workers=2 idle: 654 [18020.055165] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18020.061367] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18020.067753] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18020.067853] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18020.067865] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18020.067868] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18020.067881] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18020.067886] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18020.067891] RAX: 0000000000005756 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18020.067894] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18020.067959] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18020.067963] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18020.067966] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18020.067969] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18020.067972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18020.067975] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18020.067986] Call Trace: [18020.067992] [18020.067998] ? show_regs.cold.9+0x22/0x2f [18020.068009] ? watchdog_timer_fn.cold.6+0x62/0x10a [18020.068016] ? __hrtimer_run_queues+0x1c2/0x4d0 [18020.068020] ? watchdog+0x40/0x40 [18020.068025] ? hrtimer_interrupt+0x128/0x2c0 [18020.068036] ? smp_apic_timer_interrupt+0xa6/0x270 [18020.068045] ? apic_timer_interrupt+0xf/0x20 [18020.068050] [18020.068053] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18020.068060] do_raw_spin_lock+0x177/0x190 [18020.068066] _raw_spin_lock+0x21/0x30 [18020.068075] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18020.068691] ? do_raw_spin_unlock+0x75/0x190 [18020.068697] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18020.069036] ? do_wait_intr+0xf0/0xf0 [18020.069045] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18020.069398] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18020.069737] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18020.070673] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18020.070786] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18020.070830] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18020.070859] mdc_intent_lock+0x353/0x7e0 [mdc] [18020.070887] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18020.071391] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18020.071827] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18020.071859] lmv_intent_open+0x338/0x1150 [lmv] [18020.071891] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18020.086002] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18020.086073] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18020.086081] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18020.086084] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18020.086098] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18020.086102] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18020.086106] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18020.086109] RDX: 0000000000003835 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18020.086111] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18020.086113] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18020.086115] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18020.086118] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18020.086121] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18020.086123] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18020.086131] Call Trace: [18020.086138] [18020.086143] ? show_regs.cold.9+0x22/0x2f [18020.086168] ? watchdog_timer_fn.cold.6+0x62/0x10a [18020.086174] ? __hrtimer_run_queues+0x1c2/0x4d0 [18020.086178] ? watchdog+0x40/0x40 [18020.086182] ? hrtimer_interrupt+0x128/0x2c0 [18020.086185] ? smp_apic_timer_interrupt+0xa6/0x270 [18020.086193] ? apic_timer_interrupt+0xf/0x20 [18020.086198] [18020.086200] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18020.086205] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18020.086209] do_raw_spin_lock+0x177/0x190 [18020.086214] _raw_spin_lock+0x21/0x30 [18020.086220] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18020.086485] obd_mod_free+0x46/0x140 [ptlrpc] [18020.086676] mdc_commit_open+0x91/0xb0 [mdc] [18020.086712] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18020.086915] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18020.087429] after_reply+0xe2c/0x1780 [ptlrpc] [18020.087633] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18020.087845] ? __next_timer_interrupt+0x160/0x160 [18020.087854] ptlrpcd+0xa82/0xe40 [ptlrpc] [18020.088066] ? do_wait_intr+0xf0/0xf0 [18020.090075] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18020.090357] kthread+0x1d1/0x200 [18020.090365] ? set_kthread_struct+0x70/0x70 [18020.090371] ret_from_fork+0x1f/0x30 [18020.605802] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18020.614836] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18020.618842] ll_intent_lock+0xc5/0xaa0 [lustre] [18020.627758] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18020.631435] ll_lookup_it+0xe0e/0x2270 [lustre] [18020.634251] ? do_raw_spin_unlock+0x75/0x190 [18020.636741] ? _raw_spin_unlock+0x12/0x30 [18020.640588] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18020.643710] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18020.646812] ll_atomic_open+0x36a/0xd70 [lustre] [18020.648514] lookup_open+0x356/0xb40 [18020.649664] do_last+0x313/0xfc0 [18020.651102] ? nd_jump_root+0xe5/0x160 [18020.653578] ? path_init+0x437/0x520 [18020.655422] path_openat+0xf7/0x500 [18020.657877] ? do_raw_spin_unlock+0x75/0x190 [18020.660754] do_filp_open+0x99/0x140 [18020.662862] ? getname_flags+0x6e/0x330 [18020.665030] ? __check_object_size+0xff/0x256 [18020.667254] ? do_raw_spin_unlock+0x75/0x190 [18020.669874] ? _raw_spin_unlock+0x12/0x30 [18020.672782] do_sys_openat2+0x2b4/0x410 [18020.675402] do_sys_open+0x73/0xa0 [18020.677353] __x64_sys_openat+0x24/0x30 [18020.679229] do_syscall_64+0xc1/0x440 [18020.681515] entry_SYSCALL_64_after_hwframe+0x49/0xae [18020.695407] RIP: 0033:0x7f53425a5332 [18020.699749] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18020.726233] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18020.730741] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18020.734877] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18020.738847] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18020.742971] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18020.747604] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18036.060802] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [18036.070291] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18036.103326] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18036.108481] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18036.113324] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18036.116232] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18036.128520] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18036.131521] RAX: 0000000000007e86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18036.135767] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18036.138754] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18036.142249] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18036.150896] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18036.156895] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18036.165064] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18036.168349] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18036.173958] Call Trace: [18036.175174] [18036.177192] ? show_regs.cold.9+0x22/0x2f [18036.180258] ? watchdog_timer_fn.cold.6+0x62/0x10a [18036.182881] ? __hrtimer_run_queues+0x1c2/0x4d0 [18036.190598] ? watchdog+0x40/0x40 [18036.191673] ? hrtimer_interrupt+0x128/0x2c0 [18036.194191] ? smp_apic_timer_interrupt+0xa6/0x270 [18036.201303] ? apic_timer_interrupt+0xf/0x20 [18036.203459] [18036.204460] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18036.207206] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18036.213236] do_raw_spin_lock+0x177/0x190 [18036.215805] _raw_spin_lock+0x21/0x30 [18036.217596] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18036.222155] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18036.226957] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18036.233210] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18036.239202] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18036.245865] ? woken_wake_function+0x30/0x30 [18036.248825] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18036.254944] kthread+0x1d1/0x200 [18036.258220] ? set_kthread_struct+0x70/0x70 [18036.260280] ret_from_fork+0x1f/0x30 [18044.052531] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [18044.057863] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18044.129715] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18044.151539] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18044.162349] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18044.172467] RIP: 0010:native_safe_halt+0x2e/0x50 [18044.183843] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18044.206673] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18044.212012] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18044.219200] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [18044.229816] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18044.240741] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18044.245536] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18044.254052] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18044.266088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18044.272416] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18044.282741] Call Trace: [18044.283917] [18044.285200] ? show_regs.cold.9+0x22/0x2f [18044.293743] ? watchdog_timer_fn.cold.6+0x62/0x10a [18044.298178] ? __hrtimer_run_queues+0x1c2/0x4d0 [18044.304799] ? watchdog+0x40/0x40 [18044.305978] ? hrtimer_interrupt+0x128/0x2c0 [18044.312831] ? smp_apic_timer_interrupt+0xa6/0x270 [18044.314438] ? apic_timer_interrupt+0xf/0x20 [18044.326608] [18044.327335] ? native_safe_halt+0x2e/0x50 [18044.328612] kvm_wait+0xcd/0xe0 [18044.334411] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18044.341091] ? newidle_balance.isra.93+0x207/0x580 [18044.347793] do_raw_spin_lock+0x177/0x190 [18044.352596] _raw_spin_lock+0x21/0x30 [18044.354661] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18044.364002] process_one_work+0x2c8/0x700 [18044.370412] worker_thread+0x296/0x6e0 [18044.378367] ? rescuer_thread+0x570/0x570 [18044.381309] kthread+0x1d1/0x200 [18044.382914] ? set_kthread_struct+0x70/0x70 [18044.390437] ret_from_fork+0x1f/0x30 [18048.055021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18048.057022] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18048.058076] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18048.058149] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18048.058155] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18048.058159] RIP: 0010:native_safe_halt+0x2e/0x50 [18048.058171] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18048.058176] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18048.058180] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18048.058183] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18048.058185] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18048.063598] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18048.064517] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18048.065009] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18048.065012] RIP: 0010:native_safe_halt+0x2e/0x50 [18048.065032] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18048.065040] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18048.065045] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18048.065048] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18048.065050] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18048.065052] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18048.065055] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18048.065058] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18048.065061] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18048.065063] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18048.065075] Call Trace: [18048.065080] [18048.065086] ? show_regs.cold.9+0x22/0x2f [18048.065101] ? watchdog_timer_fn.cold.6+0x62/0x10a [18048.065109] ? __hrtimer_run_queues+0x1c2/0x4d0 [18048.065113] ? watchdog+0x40/0x40 [18048.065117] ? hrtimer_interrupt+0x128/0x2c0 [18048.065121] ? smp_apic_timer_interrupt+0xa6/0x270 [18048.065128] ? apic_timer_interrupt+0xf/0x20 [18048.065133] [18048.065136] ? native_safe_halt+0x2e/0x50 [18048.065139] kvm_wait+0xcd/0xe0 [18048.065147] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18048.065158] do_raw_spin_lock+0x177/0x190 [18048.065164] _raw_spin_lock+0x21/0x30 [18048.065171] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18048.066053] obd_mod_free+0x46/0x140 [ptlrpc] [18048.066367] mdc_commit_open+0x91/0xb0 [mdc] [18048.066406] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18048.066612] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18048.066929] after_reply+0xe2c/0x1780 [ptlrpc] [18048.109892] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18048.109904] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18048.109908] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18048.109911] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18048.109913] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18048.109922] Call Trace: [18048.109930] [18048.109936] ? show_regs.cold.9+0x22/0x2f [18048.109949] ? watchdog_timer_fn.cold.6+0x62/0x10a [18048.109961] ? __hrtimer_run_queues+0x1c2/0x4d0 [18048.109966] ? watchdog+0x40/0x40 [18048.109970] ? hrtimer_interrupt+0x128/0x2c0 [18048.109975] ? smp_apic_timer_interrupt+0xa6/0x270 [18048.109983] ? apic_timer_interrupt+0xf/0x20 [18048.109988] [18048.109991] ? native_safe_halt+0x2e/0x50 [18048.109995] kvm_wait+0xcd/0xe0 [18048.110003] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18048.110013] do_raw_spin_lock+0x177/0x190 [18048.110023] _raw_spin_lock+0x21/0x30 [18048.110027] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18048.110302] ? do_raw_spin_unlock+0x75/0x190 [18048.110308] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18048.122178] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18048.129002] ? do_wait_intr+0xf0/0xf0 [18048.129026] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18048.129424] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18048.129677] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18048.129920] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18048.130042] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18048.130095] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18048.130125] mdc_intent_lock+0x353/0x7e0 [mdc] [18048.130155] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18048.136234] ? __next_timer_interrupt+0x160/0x160 [18048.148228] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18048.158373] ptlrpcd+0xa82/0xe40 [ptlrpc] [18048.158864] ? do_wait_intr+0xf0/0xf0 [18048.158877] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18048.159099] kthread+0x1d1/0x200 [18048.159106] ? set_kthread_struct+0x70/0x70 [18048.159111] ret_from_fork+0x1f/0x30 [18048.560658] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18048.563791] lmv_intent_open+0x338/0x1150 [lmv] [18048.565736] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18048.568281] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18048.571300] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18048.573938] ll_intent_lock+0xc5/0xaa0 [lustre] [18048.576224] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18048.579589] ll_lookup_it+0xe0e/0x2270 [lustre] [18048.582988] ? do_raw_spin_unlock+0x75/0x190 [18048.585721] ? _raw_spin_unlock+0x12/0x30 [18048.587786] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18048.590774] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18048.594281] ll_atomic_open+0x36a/0xd70 [lustre] [18048.597459] lookup_open+0x356/0xb40 [18048.599037] do_last+0x313/0xfc0 [18048.601599] ? nd_jump_root+0xe5/0x160 [18048.604684] ? path_init+0x437/0x520 [18048.607879] path_openat+0xf7/0x500 [18048.612934] ? do_raw_spin_unlock+0x75/0x190 [18048.616765] do_filp_open+0x99/0x140 [18048.618042] ? getname_flags+0x6e/0x330 [18048.621113] ? __check_object_size+0xff/0x256 [18048.624732] ? do_raw_spin_unlock+0x75/0x190 [18048.630502] ? _raw_spin_unlock+0x12/0x30 [18048.631849] do_sys_openat2+0x2b4/0x410 [18048.635517] do_sys_open+0x73/0xa0 [18048.636843] __x64_sys_openat+0x24/0x30 [18048.638645] do_syscall_64+0xc1/0x440 [18048.644525] entry_SYSCALL_64_after_hwframe+0x49/0xae [18048.646954] RIP: 0033:0x7f53425a5332 [18048.649632] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18048.662872] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18048.666941] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18048.671497] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18048.674559] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18048.677760] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18048.681252] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18050.021897] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15412s! [18050.038182] Showing busy workqueues and worker pools: [18050.050116] workqueue events: flags=0x0 [18050.058207] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18050.073872] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18050.087776] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18050.096273] pending: cpu_latency_work [ptlrpc] [18050.106490] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18050.115723] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18050.129016] workqueue events_long: flags=0x0 [18050.133015] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18050.145557] pending: lu_objects_flush [obdclass] [18050.150177] workqueue events_unbound: flags=0x2 [18050.157366] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18050.170958] pending: flush_memcg_stats_dwork, idle_cull_fn [18050.175854] workqueue events_power_efficient: flags=0x80 [18050.180097] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18050.184715] pending: check_lifetime, do_cache_clean [18050.188959] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18050.196063] pending: neigh_periodic_work, neigh_periodic_work [18050.202688] workqueue mm_percpu_wq: flags=0x8 [18050.206703] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18050.211276] pending: vmstat_update, lru_add_drain_per_cpu [18050.214772] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18050.216898] pending: vmstat_update, lru_add_drain_per_cpu [18050.222722] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18050.229095] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18050.233263] workqueue xprtiod: flags=0xa [18050.235204] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18050.240873] pending: xprt_autoclose, xs_error_handle [18050.246048] workqueue ipv6_addrconf: flags=0x40008 [18050.249879] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18050.257791] pending: addrconf_verify_work [18050.264436] workqueue knbd0-recv: flags=0x1a [18050.265994] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18050.279952] in-flight: 651:recv_work [18050.282060] workqueue ptlrpc_pinger: flags=0x2 [18050.285311] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18050.290754] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18050.294127] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15413s workers=4 idle: 15979 31824 8 [18050.298251] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18040s workers=2 idle: 654 [18064.058628] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [18064.068023] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18064.104517] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18064.111658] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18064.121717] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18064.130581] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18064.143823] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18064.148706] RAX: 0000000000003152 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18064.155549] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18064.163711] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18064.172413] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18064.178077] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18064.180933] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18064.185734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18064.192092] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18064.202443] Call Trace: [18064.205157] [18064.205914] ? show_regs.cold.9+0x22/0x2f [18064.209062] ? watchdog_timer_fn.cold.6+0x62/0x10a [18064.214468] ? __hrtimer_run_queues+0x1c2/0x4d0 [18064.218783] ? watchdog+0x40/0x40 [18064.222848] ? hrtimer_interrupt+0x128/0x2c0 [18064.225620] ? smp_apic_timer_interrupt+0xa6/0x270 [18064.228068] ? apic_timer_interrupt+0xf/0x20 [18064.230433] [18064.232869] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18064.235073] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18064.243661] do_raw_spin_lock+0x177/0x190 [18064.249626] _raw_spin_lock+0x21/0x30 [18064.251701] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18064.275836] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18064.277613] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18064.288381] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18064.296525] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18064.303885] ? woken_wake_function+0x30/0x30 [18064.306955] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18064.318987] kthread+0x1d1/0x200 [18064.320197] ? set_kthread_struct+0x70/0x70 [18064.321649] ret_from_fork+0x1f/0x30 [18076.052763] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [18076.056279] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18076.057276] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18076.057285] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18076.057445] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18076.057451] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18076.057454] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18076.057467] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18076.057471] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18076.057476] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18076.057478] RDX: 00000000000073fb RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18076.057480] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18076.057483] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18076.057485] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18076.057492] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18076.057495] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18076.057498] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18076.057506] Call Trace: [18076.057511] [18076.057517] ? show_regs.cold.9+0x22/0x2f [18076.057526] ? watchdog_timer_fn.cold.6+0x62/0x10a [18076.057537] ? __hrtimer_run_queues+0x1c2/0x4d0 [18076.057542] ? watchdog+0x40/0x40 [18076.057546] ? hrtimer_interrupt+0x128/0x2c0 [18076.057550] ? smp_apic_timer_interrupt+0xa6/0x270 [18076.057558] ? apic_timer_interrupt+0xf/0x20 [18076.057563] [18076.057566] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18076.057571] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18076.057575] do_raw_spin_lock+0x177/0x190 [18076.057580] _raw_spin_lock+0x21/0x30 [18076.057585] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18076.057857] obd_mod_free+0x46/0x140 [ptlrpc] [18076.058083] mdc_commit_open+0x91/0xb0 [mdc] [18076.058124] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18076.059193] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18076.060160] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18076.060230] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18076.060236] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18076.060243] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18076.059503] after_reply+0xe2c/0x1780 [ptlrpc] [18076.061087] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18076.061287] ? __next_timer_interrupt+0x160/0x160 [18076.061298] ptlrpcd+0xa82/0xe40 [ptlrpc] [18076.062248] ? do_wait_intr+0xf0/0xf0 [18076.062257] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18076.063553] kthread+0x1d1/0x200 [18076.063559] ? set_kthread_struct+0x70/0x70 [18076.063564] ret_from_fork+0x1f/0x30 [18076.067425] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) [18076.077317] [18076.077327] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18076.077412] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18076.077417] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18076.077422] RAX: 0000000000007e1d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18076.077424] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [18076.077426] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18076.077428] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18076.077431] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18076.077433] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18076.077436] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18076.077438] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18076.077447] Call Trace: [18076.077452] [18076.077458] ? show_regs.cold.9+0x22/0x2f [18076.077558] ? watchdog_timer_fn.cold.6+0x62/0x10a [18076.077564] ? __hrtimer_run_queues+0x1c2/0x4d0 [18076.077569] ? watchdog+0x40/0x40 [18076.077579] ? hrtimer_interrupt+0x128/0x2c0 [18076.077583] ? smp_apic_timer_interrupt+0xa6/0x270 [18076.077591] ? apic_timer_interrupt+0xf/0x20 [18076.077596] [18076.077600] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18076.077605] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18076.077609] ? newidle_balance.isra.93+0x207/0x580 [18076.077617] do_raw_spin_lock+0x177/0x190 [18076.077622] _raw_spin_lock+0x21/0x30 [18076.077628] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18076.077986] process_one_work+0x2c8/0x700 [18076.077995] worker_thread+0x296/0x6e0 [18076.078000] ? rescuer_thread+0x570/0x570 [18076.078004] kthread+0x1d1/0x200 [18076.078008] ? set_kthread_struct+0x70/0x70 [18076.078012] ret_from_fork+0x1f/0x30 [18076.637973] fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18076.666562] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18076.677249] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18076.690059] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18076.694637] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18076.714377] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18076.727110] RAX: 0000000000007f84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18076.731568] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18076.742176] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18076.746828] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18076.750634] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18076.760737] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18076.767089] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18076.769016] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18076.777278] Call Trace: [18076.779055] [18076.779852] ? show_regs.cold.9+0x22/0x2f [18076.781822] ? watchdog_timer_fn.cold.6+0x62/0x10a [18076.789226] ? __hrtimer_run_queues+0x1c2/0x4d0 [18076.792893] ? watchdog+0x40/0x40 [18076.794248] ? hrtimer_interrupt+0x128/0x2c0 [18076.797120] ? smp_apic_timer_interrupt+0xa6/0x270 [18076.800271] ? apic_timer_interrupt+0xf/0x20 [18076.804111] [18076.807555] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18076.810769] do_raw_spin_lock+0x177/0x190 [18076.812786] _raw_spin_lock+0x21/0x30 [18076.818155] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18076.824057] ? do_raw_spin_unlock+0x75/0x190 [18076.828623] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18076.836265] ? do_wait_intr+0xf0/0xf0 [18076.837659] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18076.841833] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18076.845269] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18076.850843] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18076.857449] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18076.861533] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18076.867238] mdc_intent_lock+0x353/0x7e0 [mdc] [18076.870456] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18076.875277] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18076.879245] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18076.883765] lmv_intent_open+0x338/0x1150 [lmv] [18076.886537] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18076.892437] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18076.895540] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18076.898265] ll_intent_lock+0xc5/0xaa0 [lustre] [18076.901344] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18076.908503] ll_lookup_it+0xe0e/0x2270 [lustre] [18076.915246] ? do_raw_spin_unlock+0x75/0x190 [18076.917373] ? _raw_spin_unlock+0x12/0x30 [18076.919420] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18076.921430] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18076.924381] ll_atomic_open+0x36a/0xd70 [lustre] [18076.926629] lookup_open+0x356/0xb40 [18076.930368] do_last+0x313/0xfc0 [18076.935446] ? nd_jump_root+0xe5/0x160 [18076.939034] ? path_init+0x437/0x520 [18076.941169] path_openat+0xf7/0x500 [18076.942937] ? do_raw_spin_unlock+0x75/0x190 [18076.946330] do_filp_open+0x99/0x140 [18076.952452] ? getname_flags+0x6e/0x330 [18076.955832] ? __check_object_size+0xff/0x256 [18076.961658] ? do_raw_spin_unlock+0x75/0x190 [18076.963211] ? _raw_spin_unlock+0x12/0x30 [18076.967462] do_sys_openat2+0x2b4/0x410 [18076.975207] do_sys_open+0x73/0xa0 [18076.976794] __x64_sys_openat+0x24/0x30 [18076.980729] do_syscall_64+0xc1/0x440 [18076.982871] entry_SYSCALL_64_after_hwframe+0x49/0xae [18076.986404] RIP: 0033:0x7f53425a5332 [18076.990815] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18077.005640] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18077.009112] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18077.016200] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18077.020785] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18077.025816] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18077.030356] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18080.740851] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15443s! [18080.748238] Showing busy workqueues and worker pools: [18080.754752] workqueue events: flags=0x0 [18080.759556] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18080.762755] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18080.775994] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18080.782273] pending: cpu_latency_work [ptlrpc] [18080.786060] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18080.789914] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18080.800485] workqueue events_long: flags=0x0 [18080.802477] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18080.809119] pending: lu_objects_flush [obdclass] [18080.819331] workqueue events_unbound: flags=0x2 [18080.821448] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18080.827401] pending: flush_memcg_stats_dwork, idle_cull_fn [18080.833313] workqueue events_power_efficient: flags=0x80 [18080.839296] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18080.846463] pending: check_lifetime, do_cache_clean [18080.851168] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18080.857725] pending: neigh_periodic_work, neigh_periodic_work [18080.863852] workqueue mm_percpu_wq: flags=0x8 [18080.867142] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18080.875016] pending: vmstat_update, lru_add_drain_per_cpu [18080.877388] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18080.885471] pending: vmstat_update, lru_add_drain_per_cpu [18080.890394] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18080.899952] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18080.906654] workqueue xprtiod: flags=0xa [18080.909025] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18080.915380] pending: xprt_autoclose, xs_error_handle [18080.921371] workqueue ipv6_addrconf: flags=0x40008 [18080.927365] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18080.933933] pending: addrconf_verify_work [18080.935498] workqueue knbd0-recv: flags=0x1a [18080.941209] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18080.945374] in-flight: 651:recv_work [18080.949676] workqueue ptlrpc_pinger: flags=0x2 [18080.951254] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18080.957570] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18080.965574] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15443s workers=4 idle: 15979 31824 8 [18080.973134] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18071s workers=2 idle: 654 [18092.059181] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [18092.066990] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18092.105095] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18092.113398] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18092.121112] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18092.126507] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18092.147785] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18092.157206] RAX: 0000000000007e89 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18092.168418] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18092.178964] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18092.186567] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18092.194028] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18092.199351] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18092.202560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18092.204872] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18092.209138] Call Trace: [18092.209835] [18092.210624] ? show_regs.cold.9+0x22/0x2f [18092.212563] ? watchdog_timer_fn.cold.6+0x62/0x10a [18092.217741] ? __hrtimer_run_queues+0x1c2/0x4d0 [18092.221305] ? watchdog+0x40/0x40 [18092.223365] ? hrtimer_interrupt+0x128/0x2c0 [18092.225732] ? smp_apic_timer_interrupt+0xa6/0x270 [18092.231017] ? apic_timer_interrupt+0xf/0x20 [18092.237154] [18092.240097] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18092.246927] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18092.250629] do_raw_spin_lock+0x177/0x190 [18092.253016] _raw_spin_lock+0x21/0x30 [18092.256356] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18092.261379] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18092.264225] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18092.268959] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18092.273267] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18092.278947] ? woken_wake_function+0x30/0x30 [18092.280715] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18092.283982] kthread+0x1d1/0x200 [18092.285836] ? set_kthread_struct+0x70/0x70 [18092.287931] ret_from_fork+0x1f/0x30 [18104.055926] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18104.058539] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18104.061705] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18104.061977] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18104.061984] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18104.061987] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18104.062001] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18104.062005] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18104.062010] RAX: 0000000000002b96 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18104.062012] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18104.062014] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18104.062017] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18104.062019] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18104.062023] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18104.062026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18104.062028] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18104.062037] Call Trace: [18104.062042] [18104.062048] ? show_regs.cold.9+0x22/0x2f [18104.062056] ? watchdog_timer_fn.cold.6+0x62/0x10a [18104.062064] ? __hrtimer_run_queues+0x1c2/0x4d0 [18104.062068] ? watchdog+0x40/0x40 [18104.062072] ? hrtimer_interrupt+0x128/0x2c0 [18104.062077] ? smp_apic_timer_interrupt+0xa6/0x270 [18104.062087] ? apic_timer_interrupt+0xf/0x20 [18104.062091] [18104.062095] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18104.062103] do_raw_spin_lock+0x177/0x190 [18104.062108] _raw_spin_lock+0x21/0x30 [18104.062113] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18104.062687] ? do_raw_spin_unlock+0x75/0x190 [18104.062697] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18104.062916] ? do_wait_intr+0xf0/0xf0 [18104.062924] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18104.063133] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18104.063492] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18104.063841] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18104.064014] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18104.064050] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18104.064079] mdc_intent_lock+0x353/0x7e0 [mdc] [18104.064158] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18104.064251] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18104.065106] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18104.065139] lmv_intent_open+0x338/0x1150 [lmv] [18104.065178] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18104.082478] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) [18104.116755] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18104.116957] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18104.116990] ll_intent_lock+0xc5/0xaa0 [lustre] [18104.117077] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18104.117159] ll_lookup_it+0xe0e/0x2270 [lustre] [18104.117240] ? do_raw_spin_unlock+0x75/0x190 [18104.117248] ? _raw_spin_unlock+0x12/0x30 [18104.117255] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18104.117283] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18104.117303] ll_atomic_open+0x36a/0xd70 [lustre] [18104.117387] lookup_open+0x356/0xb40 [18104.117398] do_last+0x313/0xfc0 [18104.117404] ? nd_jump_root+0xe5/0x160 [18104.117409] ? path_init+0x437/0x520 [18104.117414] path_openat+0xf7/0x500 [18104.117419] ? do_raw_spin_unlock+0x75/0x190 [18104.117424] do_filp_open+0x99/0x140 [18104.120715] ? getname_flags+0x6e/0x330 [18104.120720] ? __check_object_size+0xff/0x256 [18104.120727] ? do_raw_spin_unlock+0x75/0x190 [18104.120733] ? _raw_spin_unlock+0x12/0x30 [18104.120739] do_sys_openat2+0x2b4/0x410 [18104.120743] do_sys_open+0x73/0xa0 [18104.120747] __x64_sys_openat+0x24/0x30 [18104.120750] do_syscall_64+0xc1/0x440 [18104.120757] entry_SYSCALL_64_after_hwframe+0x49/0xae [18104.120763] RIP: 0033:0x7f53425a5332 [18104.120780] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18104.120785] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18104.120790] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18104.120792] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18104.120795] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18104.120798] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18104.120800] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18104.404202] ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18104.419924] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18104.426968] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18104.430831] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18104.433443] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18104.447088] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18104.450569] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18104.458193] RDX: 0000000000002c3c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18104.465989] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18104.472343] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18104.479064] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18104.486048] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18104.493366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18104.498280] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18104.504037] Call Trace: [18104.506174] [18104.508604] ? show_regs.cold.9+0x22/0x2f [18104.511323] ? watchdog_timer_fn.cold.6+0x62/0x10a [18104.516120] ? __hrtimer_run_queues+0x1c2/0x4d0 [18104.520115] ? watchdog+0x40/0x40 [18104.522173] ? hrtimer_interrupt+0x128/0x2c0 [18104.527805] ? smp_apic_timer_interrupt+0xa6/0x270 [18104.532740] ? apic_timer_interrupt+0xf/0x20 [18104.535125] [18104.536590] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18104.539503] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18104.545145] do_raw_spin_lock+0x177/0x190 [18104.548270] _raw_spin_lock+0x21/0x30 [18104.549660] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18104.553392] obd_mod_free+0x46/0x140 [ptlrpc] [18104.557106] mdc_commit_open+0x91/0xb0 [mdc] [18104.559762] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18104.568391] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18104.572725] after_reply+0xe2c/0x1780 [ptlrpc] [18104.578121] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18104.583294] ? __next_timer_interrupt+0x160/0x160 [18104.589181] ptlrpcd+0xa82/0xe40 [ptlrpc] [18104.594605] ? do_wait_intr+0xf0/0xf0 [18104.597895] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18104.602343] kthread+0x1d1/0x200 [18104.605890] ? set_kthread_struct+0x70/0x70 [18104.609010] ret_from_fork+0x1f/0x30 [18108.053278] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [18108.060657] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18108.091305] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18108.099260] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18108.105679] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18108.111490] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18108.114430] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18108.128894] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18108.138975] RAX: 0000000000007f0e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18108.146464] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [18108.152604] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18108.160966] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18108.169691] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18108.174933] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18108.191245] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18108.197775] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18108.206929] Call Trace: [18108.208311] [18108.209323] ? show_regs.cold.9+0x22/0x2f [18108.218667] ? watchdog_timer_fn.cold.6+0x62/0x10a [18108.220762] ? __hrtimer_run_queues+0x1c2/0x4d0 [18108.227143] ? watchdog+0x40/0x40 [18108.228909] ? hrtimer_interrupt+0x128/0x2c0 [18108.231730] ? smp_apic_timer_interrupt+0xa6/0x270 [18108.235050] ? apic_timer_interrupt+0xf/0x20 [18108.238057] [18108.238947] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18108.241860] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18108.244510] ? newidle_balance.isra.93+0x207/0x580 [18108.246947] do_raw_spin_lock+0x177/0x190 [18108.249056] _raw_spin_lock+0x21/0x30 [18108.250867] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18108.254657] process_one_work+0x2c8/0x700 [18108.256834] worker_thread+0x296/0x6e0 [18108.259862] ? rescuer_thread+0x570/0x570 [18108.266103] kthread+0x1d1/0x200 [18108.267920] ? set_kthread_struct+0x70/0x70 [18108.271570] ret_from_fork+0x1f/0x30 [18111.456025] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15474s! [18111.459945] Showing busy workqueues and worker pools: [18111.462275] workqueue events: flags=0x0 [18111.463974] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18111.467553] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18111.472654] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18111.475869] pending: cpu_latency_work [ptlrpc] [18111.480082] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18111.484221] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18111.490127] workqueue events_long: flags=0x0 [18111.492719] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18111.496242] pending: lu_objects_flush [obdclass] [18111.499575] workqueue events_unbound: flags=0x2 [18111.501821] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18111.504937] pending: flush_memcg_stats_dwork, idle_cull_fn [18111.507854] workqueue events_power_efficient: flags=0x80 [18111.511019] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18111.514912] pending: check_lifetime, do_cache_clean [18111.519362] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18111.522971] pending: neigh_periodic_work, neigh_periodic_work [18111.526691] workqueue mm_percpu_wq: flags=0x8 [18111.528986] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18111.532767] pending: vmstat_update, lru_add_drain_per_cpu [18111.535825] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18111.538296] pending: vmstat_update, lru_add_drain_per_cpu [18111.541858] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18111.545022] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18111.547923] workqueue xprtiod: flags=0xa [18111.550072] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18111.552753] pending: xprt_autoclose, xs_error_handle [18111.555942] workqueue ipv6_addrconf: flags=0x40008 [18111.558893] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18111.562815] pending: addrconf_verify_work [18111.565643] workqueue knbd0-recv: flags=0x1a [18111.568277] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18111.577029] in-flight: 651:recv_work [18111.580391] workqueue ptlrpc_pinger: flags=0x2 [18111.584758] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18111.590101] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18111.593022] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15474s workers=4 idle: 15979 31824 8 [18111.596814] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18102s workers=2 idle: 654 [18120.059357] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [18120.069045] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18120.119368] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18120.127905] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18120.136028] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18120.143022] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18120.166912] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18120.174544] RAX: 0000000000005385 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18120.182094] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18120.189966] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18120.200695] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18120.215741] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18120.219792] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18120.224837] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18120.228669] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18120.233691] Call Trace: [18120.234696] [18120.235784] ? show_regs.cold.9+0x22/0x2f [18120.239480] ? watchdog_timer_fn.cold.6+0x62/0x10a [18120.254849] ? __hrtimer_run_queues+0x1c2/0x4d0 [18120.256436] ? watchdog+0x40/0x40 [18120.262950] ? hrtimer_interrupt+0x128/0x2c0 [18120.264498] ? smp_apic_timer_interrupt+0xa6/0x270 [18120.276925] ? apic_timer_interrupt+0xf/0x20 [18120.280193] [18120.281672] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18120.288785] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18120.296639] do_raw_spin_lock+0x177/0x190 [18120.300605] _raw_spin_lock+0x21/0x30 [18120.303896] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18120.312094] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18120.315132] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18120.321592] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18120.328352] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18120.331392] ? woken_wake_function+0x30/0x30 [18120.333972] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18120.336268] kthread+0x1d1/0x200 [18120.337498] ? set_kthread_struct+0x70/0x70 [18120.338894] ret_from_fork+0x1f/0x30 [18132.059870] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18132.060740] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18132.078099] Modules linked in: lustre(O) osp(O) [18132.081267] Modules linked in: [18132.081270] ofd(O) lod(O) [18132.083523] lustre(O) [18132.094254] mdt(O) mdd(O) mgs(O) [18132.098661] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18132.098744] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18132.098750] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18132.098752] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18132.098765] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18132.098769] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18132.098773] RAX: 0000000000001c45 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18132.098776] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18132.098778] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18132.098781] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18132.098783] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18132.098786] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18132.098790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18132.098792] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18132.098800] Call Trace: [18132.098805] [18132.098811] ? show_regs.cold.9+0x22/0x2f [18132.098820] ? watchdog_timer_fn.cold.6+0x62/0x10a [18132.098826] ? __hrtimer_run_queues+0x1c2/0x4d0 [18132.098831] ? watchdog+0x40/0x40 [18132.098835] ? hrtimer_interrupt+0x128/0x2c0 [18132.098840] ? smp_apic_timer_interrupt+0xa6/0x270 [18132.098850] ? apic_timer_interrupt+0xf/0x20 [18132.098855] [18132.098858] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18132.098865] do_raw_spin_lock+0x177/0x190 [18132.098869] _raw_spin_lock+0x21/0x30 [18132.098875] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18132.099217] ? do_raw_spin_unlock+0x75/0x190 [18132.099223] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18132.099426] ? do_wait_intr+0xf0/0xf0 [18132.099435] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18132.099713] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18132.099930] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18132.100201] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18132.100315] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18132.100352] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18132.100381] mdc_intent_lock+0x353/0x7e0 [mdc] [18132.100406] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18132.103069] lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18132.103140] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18132.103147] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18132.103151] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18132.103164] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18132.103168] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18132.103173] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18132.103176] RDX: 0000000000006626 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18132.103178] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18132.103180] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18132.103186] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18132.103189] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18132.103191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18132.103194] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18132.103201] Call Trace: [18132.103205] [18132.103211] ? show_regs.cold.9+0x22/0x2f [18132.103219] ? watchdog_timer_fn.cold.6+0x62/0x10a [18132.103225] ? __hrtimer_run_queues+0x1c2/0x4d0 [18132.103229] ? watchdog+0x40/0x40 [18132.103236] ? hrtimer_interrupt+0x128/0x2c0 [18132.103241] ? smp_apic_timer_interrupt+0xa6/0x270 [18132.103248] ? apic_timer_interrupt+0xf/0x20 [18132.103252] [18132.103255] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18132.103259] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18132.103264] do_raw_spin_lock+0x177/0x190 [18132.103268] _raw_spin_lock+0x21/0x30 [18132.103273] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18132.103951] obd_mod_free+0x46/0x140 [ptlrpc] [18132.104190] mdc_commit_open+0x91/0xb0 [mdc] [18132.104225] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18132.104428] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18132.104708] after_reply+0xe2c/0x1780 [ptlrpc] [18132.105022] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18132.105217] ? __next_timer_interrupt+0x160/0x160 [18132.105226] ptlrpcd+0xa82/0xe40 [ptlrpc] [18132.105415] ? do_wait_intr+0xf0/0xf0 [18132.105423] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18132.105614] kthread+0x1d1/0x200 [18132.105693] ? set_kthread_struct+0x70/0x70 [18132.105724] ret_from_fork+0x1f/0x30 [18132.536500] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18132.540121] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18132.546231] lmv_intent_open+0x338/0x1150 [lmv] [18132.550359] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18132.554136] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18132.560743] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18132.567064] ll_intent_lock+0xc5/0xaa0 [lustre] [18132.572490] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18132.575283] ll_lookup_it+0xe0e/0x2270 [lustre] [18132.580571] ? do_raw_spin_unlock+0x75/0x190 [18132.584849] ? _raw_spin_unlock+0x12/0x30 [18132.590194] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18132.596558] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18132.600604] ll_atomic_open+0x36a/0xd70 [lustre] [18132.603095] lookup_open+0x356/0xb40 [18132.605086] do_last+0x313/0xfc0 [18132.606603] ? nd_jump_root+0xe5/0x160 [18132.610848] ? path_init+0x437/0x520 [18132.613046] path_openat+0xf7/0x500 [18132.617505] ? do_raw_spin_unlock+0x75/0x190 [18132.623238] do_filp_open+0x99/0x140 [18132.625214] ? getname_flags+0x6e/0x330 [18132.630195] ? __check_object_size+0xff/0x256 [18132.632452] ? do_raw_spin_unlock+0x75/0x190 [18132.634331] ? _raw_spin_unlock+0x12/0x30 [18132.639427] do_sys_openat2+0x2b4/0x410 [18132.641404] do_sys_open+0x73/0xa0 [18132.643165] __x64_sys_openat+0x24/0x30 [18132.646212] do_syscall_64+0xc1/0x440 [18132.648106] entry_SYSCALL_64_after_hwframe+0x49/0xae [18132.650211] RIP: 0033:0x7f53425a5332 [18132.651766] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18132.664312] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18132.668863] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18132.673515] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18132.684170] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18132.693811] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18132.697074] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18136.052149] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [18136.056169] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18136.075631] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18136.080263] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18136.084531] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18136.087674] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18136.090239] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18136.098464] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18136.101363] RAX: 000000000000378e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18136.104626] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [18136.107312] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18136.110718] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18136.113590] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18136.116700] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18136.120349] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18136.122941] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18136.125858] Call Trace: [18136.127036] [18136.127805] ? show_regs.cold.9+0x22/0x2f [18136.130426] ? watchdog_timer_fn.cold.6+0x62/0x10a [18136.132641] ? __hrtimer_run_queues+0x1c2/0x4d0 [18136.134851] ? watchdog+0x40/0x40 [18136.136509] ? hrtimer_interrupt+0x128/0x2c0 [18136.138678] ? smp_apic_timer_interrupt+0xa6/0x270 [18136.140901] ? apic_timer_interrupt+0xf/0x20 [18136.142944] [18136.143954] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18136.146522] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18136.148518] ? newidle_balance.isra.93+0x207/0x580 [18136.150656] do_raw_spin_lock+0x177/0x190 [18136.152509] _raw_spin_lock+0x21/0x30 [18136.154250] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18136.157195] process_one_work+0x2c8/0x700 [18136.158843] worker_thread+0x296/0x6e0 [18136.160812] ? rescuer_thread+0x570/0x570 [18136.162839] kthread+0x1d1/0x200 [18136.164377] ? set_kthread_struct+0x70/0x70 [18136.166488] ret_from_fork+0x1f/0x30 [18141.669126] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15504s! [18141.676655] Showing busy workqueues and worker pools: [18141.682855] workqueue events: flags=0x0 [18141.685172] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18141.688738] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18141.698654] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18141.702704] pending: cpu_latency_work [ptlrpc] [18141.705993] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18141.712845] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18141.728278] workqueue events_long: flags=0x0 [18141.731377] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18141.742913] pending: lu_objects_flush [obdclass] [18141.747268] workqueue events_unbound: flags=0x2 [18141.751561] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18141.755886] pending: flush_memcg_stats_dwork, idle_cull_fn [18141.766623] workqueue events_power_efficient: flags=0x80 [18141.773196] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18141.779779] pending: check_lifetime, do_cache_clean [18141.782500] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18141.787820] pending: neigh_periodic_work, neigh_periodic_work [18141.792089] workqueue mm_percpu_wq: flags=0x8 [18141.793691] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18141.797403] pending: vmstat_update, lru_add_drain_per_cpu [18141.803726] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18141.809669] pending: vmstat_update, lru_add_drain_per_cpu [18141.815763] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18141.818948] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18141.822273] workqueue xprtiod: flags=0xa [18141.824798] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18141.833265] pending: xprt_autoclose, xs_error_handle [18141.836172] workqueue ipv6_addrconf: flags=0x40008 [18141.840332] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18141.845437] pending: addrconf_verify_work [18141.850938] workqueue knbd0-recv: flags=0x1a [18141.854655] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18141.856966] in-flight: 651:recv_work [18141.862936] workqueue ptlrpc_pinger: flags=0x2 [18141.869870] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18141.875009] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18141.885343] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15504s workers=4 idle: 15979 31824 8 [18141.888629] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18132s workers=2 idle: 654 [18148.058073] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [18148.064242] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18148.123567] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18148.132277] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18148.137399] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18148.140569] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18148.150243] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18148.153204] RAX: 0000000000003f77 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18148.156323] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18148.160099] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18148.162912] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18148.165836] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18148.168877] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18148.172828] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18148.174868] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18148.178201] Call Trace: [18148.179291] [18148.180136] ? show_regs.cold.9+0x22/0x2f [18148.181878] ? watchdog_timer_fn.cold.6+0x62/0x10a [18148.183616] ? __hrtimer_run_queues+0x1c2/0x4d0 [18148.185403] ? watchdog+0x40/0x40 [18148.186618] ? hrtimer_interrupt+0x128/0x2c0 [18148.188345] ? smp_apic_timer_interrupt+0xa6/0x270 [18148.190561] ? apic_timer_interrupt+0xf/0x20 [18148.191976] [18148.192971] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18148.195235] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18148.197719] do_raw_spin_lock+0x177/0x190 [18148.199293] _raw_spin_lock+0x21/0x30 [18148.201064] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18148.203881] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18148.205986] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18148.209226] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18148.212176] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18148.214433] ? woken_wake_function+0x30/0x30 [18148.216312] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18148.218488] kthread+0x1d1/0x200 [18148.220114] ? set_kthread_struct+0x70/0x70 [18148.221595] ret_from_fork+0x1f/0x30 [18160.057690] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18160.060642] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [18160.061252] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18160.063293] mgc(O) [18160.070230] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18160.070414] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18160.070422] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18160.070426] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18160.070440] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18160.070444] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18160.070843] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18160.070845] RDX: 000000000000195f RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18160.070847] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18160.070848] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18160.070850] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18160.070852] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18160.070854] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18160.070856] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18160.070862] Call Trace: [18160.070866] [18160.070871] ? show_regs.cold.9+0x22/0x2f [18160.070879] ? watchdog_timer_fn.cold.6+0x62/0x10a [18160.070883] ? __hrtimer_run_queues+0x1c2/0x4d0 [18160.070886] ? watchdog+0x40/0x40 [18160.070889] ? hrtimer_interrupt+0x128/0x2c0 [18160.070891] ? smp_apic_timer_interrupt+0xa6/0x270 [18160.070898] ? apic_timer_interrupt+0xf/0x20 [18160.070902] [18160.070904] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18160.070907] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18160.070910] do_raw_spin_lock+0x177/0x190 [18160.070913] _raw_spin_lock+0x21/0x30 [18160.070919] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18160.071311] obd_mod_free+0x46/0x140 [ptlrpc] [18160.077322] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18160.077384] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18160.077392] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18160.077394] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18160.077407] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18160.077411] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18160.077415] RAX: 0000000000004963 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18160.077418] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18160.077420] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18160.077422] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18160.077424] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18160.077427] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18160.077430] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18160.077432] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18160.077441] Call Trace: [18160.077447] [18160.079379] ? show_regs.cold.9+0x22/0x2f [18160.079401] ? watchdog_timer_fn.cold.6+0x62/0x10a [18160.079409] ? __hrtimer_run_queues+0x1c2/0x4d0 [18160.079414] ? watchdog+0x40/0x40 [18160.079419] ? hrtimer_interrupt+0x128/0x2c0 [18160.079424] ? smp_apic_timer_interrupt+0xa6/0x270 [18160.079432] ? apic_timer_interrupt+0xf/0x20 [18160.079437] [18160.079441] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18160.079467] do_raw_spin_lock+0x177/0x190 [18160.079474] _raw_spin_lock+0x21/0x30 [18160.079481] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18160.079750] ? do_raw_spin_unlock+0x75/0x190 [18160.079757] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18160.079951] ? do_wait_intr+0xf0/0xf0 [18160.079959] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18160.080430] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18160.080712] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18160.080926] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18160.081192] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18160.081230] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18160.081261] mdc_intent_lock+0x353/0x7e0 [mdc] [18160.081291] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18160.081382] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18160.081627] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18160.081656] lmv_intent_open+0x338/0x1150 [lmv] [18160.081684] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18160.084881] mdc_commit_open+0x91/0xb0 [mdc] [18160.084935] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18160.085507] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18160.085831] after_reply+0xe2c/0x1780 [ptlrpc] [18160.086026] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18160.117194] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18160.117320] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18160.117355] ll_intent_lock+0xc5/0xaa0 [lustre] [18160.117438] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18160.117537] ll_lookup_it+0xe0e/0x2270 [lustre] [18160.117697] ? do_raw_spin_unlock+0x75/0x190 [18160.117706] ? _raw_spin_unlock+0x12/0x30 [18160.117712] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18160.117739] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18160.117759] ll_atomic_open+0x36a/0xd70 [lustre] [18160.117855] lookup_open+0x356/0xb40 [18160.117865] do_last+0x313/0xfc0 [18160.117870] ? nd_jump_root+0xe5/0x160 [18160.117875] ? path_init+0x437/0x520 [18160.117879] path_openat+0xf7/0x500 [18160.117884] ? do_raw_spin_unlock+0x75/0x190 [18160.117889] do_filp_open+0x99/0x140 [18160.117897] ? getname_flags+0x6e/0x330 [18160.117902] ? __check_object_size+0xff/0x256 [18160.117909] ? do_raw_spin_unlock+0x75/0x190 [18160.117912] ? _raw_spin_unlock+0x12/0x30 [18160.117917] do_sys_openat2+0x2b4/0x410 [18160.117922] do_sys_open+0x73/0xa0 [18160.117926] __x64_sys_openat+0x24/0x30 [18160.117930] do_syscall_64+0xc1/0x440 [18160.117936] entry_SYSCALL_64_after_hwframe+0x49/0xae [18160.117943] RIP: 0033:0x7f53425a5332 [18160.117998] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18160.118003] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18160.118009] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18160.118012] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18160.118014] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18160.118016] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18160.118018] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18160.735396] ? __next_timer_interrupt+0x160/0x160 [18160.737088] ptlrpcd+0xa82/0xe40 [ptlrpc] [18160.741171] ? do_wait_intr+0xf0/0xf0 [18160.743900] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18160.750227] kthread+0x1d1/0x200 [18160.751863] ? set_kthread_struct+0x70/0x70 [18160.753773] ret_from_fork+0x1f/0x30 [18168.052393] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [18168.059913] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18168.104447] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18168.109078] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18168.112569] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18168.115392] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18168.117863] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18168.131494] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18168.134737] RAX: 0000000000007083 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18168.139081] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [18168.141749] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18168.144555] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18168.148110] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18168.150742] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18168.156097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18168.158606] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18168.165436] Call Trace: [18168.166872] [18168.168094] ? show_regs.cold.9+0x22/0x2f [18168.170016] ? watchdog_timer_fn.cold.6+0x62/0x10a [18168.172275] ? __hrtimer_run_queues+0x1c2/0x4d0 [18168.174209] ? watchdog+0x40/0x40 [18168.175676] ? hrtimer_interrupt+0x128/0x2c0 [18168.177365] ? smp_apic_timer_interrupt+0xa6/0x270 [18168.179275] ? apic_timer_interrupt+0xf/0x20 [18168.180813] [18168.187454] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18168.189758] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18168.192512] ? newidle_balance.isra.93+0x207/0x580 [18168.195224] do_raw_spin_lock+0x177/0x190 [18168.197708] _raw_spin_lock+0x21/0x30 [18168.199209] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18168.201715] process_one_work+0x2c8/0x700 [18168.203749] worker_thread+0x296/0x6e0 [18168.205496] ? rescuer_thread+0x570/0x570 [18168.207128] kthread+0x1d1/0x200 [18168.208537] ? set_kthread_struct+0x70/0x70 [18168.210247] ret_from_fork+0x1f/0x30 [18172.385387] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15535s! [18172.406081] Showing busy workqueues and worker pools: [18172.408937] workqueue events: flags=0x0 [18172.412363] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18172.415111] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18172.424055] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18172.432940] pending: cpu_latency_work [ptlrpc] [18172.437830] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18172.445937] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18172.454795] workqueue events_long: flags=0x0 [18172.457907] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18172.463610] pending: lu_objects_flush [obdclass] [18172.466733] workqueue events_unbound: flags=0x2 [18172.469111] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18172.479856] pending: flush_memcg_stats_dwork, idle_cull_fn [18172.487368] workqueue events_power_efficient: flags=0x80 [18172.492024] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18172.499449] pending: check_lifetime, do_cache_clean [18172.503939] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18172.507667] pending: neigh_periodic_work, neigh_periodic_work [18172.516779] workqueue mm_percpu_wq: flags=0x8 [18172.520209] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18172.527127] pending: vmstat_update, lru_add_drain_per_cpu [18172.531593] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18172.535151] pending: vmstat_update, lru_add_drain_per_cpu [18172.538063] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18172.544543] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18172.548133] workqueue xprtiod: flags=0xa [18172.550748] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18172.557427] pending: xprt_autoclose, xs_error_handle [18172.565516] workqueue ipv6_addrconf: flags=0x40008 [18172.571783] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18172.577068] pending: addrconf_verify_work [18172.583406] workqueue knbd0-recv: flags=0x1a [18172.584851] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18172.594162] in-flight: 651:recv_work [18172.600171] workqueue ptlrpc_pinger: flags=0x2 [18172.606398] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18172.618634] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18172.626806] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15535s workers=4 idle: 15979 31824 8 [18172.640087] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18163s workers=2 idle: 654 [18174.821121] rcu: INFO: rcu_sched self-detected stall on CPU [18174.823748] rcu: 2-....: (9005309 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [18174.831266] (t=16080395 jiffies g=274185 q=68051) [18174.834764] rcu: rcu_sched kthread starved for 15538761 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [18174.843640] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [18174.849688] rcu: RCU grace-period kthread stack dump: [18174.851865] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [18174.857990] Call Trace: [18174.858820] __schedule+0x351/0xcb0 [18174.867318] schedule+0xc0/0x180 [18174.868859] schedule_timeout+0xb4/0x190 [18174.872348] ? do_raw_spin_unlock+0x75/0x190 [18174.878114] ? __next_timer_interrupt+0x160/0x160 [18174.879844] ? prepare_to_swait_event+0xf3/0x140 [18174.884584] rcu_gp_kthread+0x8dd/0x15b0 [18174.887821] ? force_qs_rnp+0x2c0/0x2c0 [18174.890296] kthread+0x1d1/0x200 [18174.891449] ? set_kthread_struct+0x70/0x70 [18174.896566] ret_from_fork+0x1f/0x30 [18174.898783] rcu: Stack dump where RCU GP kthread last ran: [18174.901506] Sending NMI from CPU 2 to CPUs 3: [18174.906856] NMI backtrace for cpu 3 [18174.906866] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18174.906868] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18174.906870] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18174.906872] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18174.906874] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 [18174.906877] RAX: 00000000000056fe RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18174.906879] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18174.906881] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18174.906882] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18174.906884] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18174.906885] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18174.906887] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18174.906888] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18174.906890] Call Trace: [18174.906891] [18174.906892] ? show_regs.cold.9+0x22/0x2f [18174.906893] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [18174.906894] ? nmi_cpu_backtrace_handler+0x15/0x40 [18174.906936] ? nmi_handle+0xbd/0x210 [18174.906937] ? default_do_nmi+0x4b/0x1b0 [18174.906938] ? do_nmi+0x280/0x380 [18174.906939] ? end_repeat_nmi+0x16/0x1f [18174.906940] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18174.906942] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18174.906943] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18174.906945] [18174.906945] do_raw_spin_lock+0x177/0x190 [18174.906946] _raw_spin_lock+0x21/0x30 [18174.906948] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18174.906949] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18174.906950] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18174.906951] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18174.906953] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18174.906954] ? woken_wake_function+0x30/0x30 [18174.906955] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18174.906957] kthread+0x1d1/0x200 [18174.906958] ? set_kthread_struct+0x70/0x70 [18174.906959] ret_from_fork+0x1f/0x30 [18174.909152] Sending NMI from CPU 2 to CPUs 1: [18175.096297] NMI backtrace for cpu 1 [18175.096303] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18175.096306] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18175.096307] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18175.096309] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18175.096311] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [18175.096313] RAX: 000000000000248d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18175.096315] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18175.096316] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18175.096317] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18175.096319] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18175.096320] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18175.096321] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18175.096322] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18175.096323] Call Trace: [18175.096324] [18175.096325] ? show_regs.cold.9+0x22/0x2f [18175.096326] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [18175.096327] ? nmi_cpu_backtrace_handler+0x15/0x40 [18175.096328] ? nmi_handle+0xbd/0x210 [18175.096329] ? default_do_nmi+0x4b/0x1b0 [18175.096330] ? do_nmi+0x280/0x380 [18175.096331] ? end_repeat_nmi+0x16/0x1f [18175.096332] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18175.096333] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18175.096334] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18175.096335] [18175.096336] do_raw_spin_lock+0x177/0x190 [18175.096337] _raw_spin_lock+0x21/0x30 [18175.096338] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18175.096339] ? do_raw_spin_unlock+0x75/0x190 [18175.096340] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18175.096342] ? do_wait_intr+0xf0/0xf0 [18175.096343] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18175.096344] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18175.096345] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18175.096346] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18175.096347] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18175.096348] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18175.096349] mdc_intent_lock+0x353/0x7e0 [mdc] [18175.096350] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18175.096351] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18175.096352] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18175.096353] lmv_intent_open+0x338/0x1150 [lmv] [18175.096354] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18175.096356] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18175.096356] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18175.096358] ll_intent_lock+0xc5/0xaa0 [lustre] [18175.096359] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18175.096360] ll_lookup_it+0xe0e/0x2270 [lustre] [18175.096361] ? do_raw_spin_unlock+0x75/0x190 [18175.096362] ? _raw_spin_unlock+0x12/0x30 [18175.096363] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18175.096364] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18175.096365] ll_atomic_open+0x36a/0xd70 [lustre] [18175.096366] lookup_open+0x356/0xb40 [18175.096368] do_last+0x313/0xfc0 [18175.096369] ? nd_jump_root+0xe5/0x160 [18175.096369] ? path_init+0x437/0x520 [18175.096370] path_openat+0xf7/0x500 [18175.096371] ? do_raw_spin_unlock+0x75/0x190 [18175.096372] do_filp_open+0x99/0x140 [18175.096373] ? getname_flags+0x6e/0x330 [18175.096375] ? __check_object_size+0xff/0x256 [18175.096376] ? do_raw_spin_unlock+0x75/0x190 [18175.096377] ? _raw_spin_unlock+0x12/0x30 [18175.096378] do_sys_openat2+0x2b4/0x410 [18175.096379] do_sys_open+0x73/0xa0 [18175.096381] __x64_sys_openat+0x24/0x30 [18175.096382] do_syscall_64+0xc1/0x440 [18175.096383] entry_SYSCALL_64_after_hwframe+0x49/0xae [18175.096385] RIP: 0033:0x7f53425a5332 [18175.096386] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18175.096389] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18175.096391] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18175.096392] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18175.096394] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18175.096395] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18175.096396] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18175.117176] NMI backtrace for cpu 2 [18175.260629] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18175.265691] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18175.269944] Call Trace: [18175.271312] [18175.272111] ? dump_stack+0xbb/0x10e [18175.273644] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [18175.276051] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [18175.278131] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [18175.281027] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [18175.283448] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [18175.284831] ? rcu_dump_cpu_stacks+0x11b/0x171 [18175.294638] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [18175.296342] ? update_process_times+0x6d/0xc0 [18175.298111] ? tick_sched_handle.isra.18+0x27/0x90 [18175.304031] ? tick_sched_timer+0x53/0xb0 [18175.307650] ? __hrtimer_run_queues+0x1c2/0x4d0 [18175.309112] ? tick_sched_handle.isra.18+0x90/0x90 [18175.310922] ? hrtimer_interrupt+0x128/0x2c0 [18175.314654] ? smp_apic_timer_interrupt+0xa6/0x270 [18175.320842] ? apic_timer_interrupt+0xf/0x20 [18175.322634] [18175.323363] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18175.329178] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18175.331777] ? do_raw_spin_lock+0x177/0x190 [18175.334176] ? _raw_spin_lock+0x21/0x30 [18175.339457] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18175.343338] ? obd_mod_free+0x46/0x140 [ptlrpc] [18175.346049] ? mdc_commit_open+0x91/0xb0 [mdc] [18175.352337] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18175.358488] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18175.362177] ? after_reply+0xe2c/0x1780 [ptlrpc] [18175.363994] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18175.367243] ? __next_timer_interrupt+0x160/0x160 [18175.371772] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [18175.377922] ? do_wait_intr+0xf0/0xf0 [18175.380361] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18175.383912] ? kthread+0x1d1/0x200 [18175.385519] ? set_kthread_struct+0x70/0x70 [18175.389371] ? ret_from_fork+0x1f/0x30 [18200.052050] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [18200.054641] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18200.059227] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [18200.060432] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [18200.060441] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18200.060510] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18200.060515] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18200.060518] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18200.060532] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18200.060536] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18200.060541] RAX: 0000000000007d85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18200.060544] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18200.060547] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18200.060549] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18200.060552] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18200.060555] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18200.060558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18200.060560] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18200.060570] Call Trace: [18200.060576] [18200.060582] ? show_regs.cold.9+0x22/0x2f [18200.060591] ? watchdog_timer_fn.cold.6+0x62/0x10a [18200.060598] ? __hrtimer_run_queues+0x1c2/0x4d0 [18200.060603] ? watchdog+0x40/0x40 [18200.060607] ? hrtimer_interrupt+0x128/0x2c0 [18200.060611] ? smp_apic_timer_interrupt+0xa6/0x270 [18200.060620] ? apic_timer_interrupt+0xf/0x20 [18200.060625] [18200.060628] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18200.060633] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18200.060640] do_raw_spin_lock+0x177/0x190 [18200.060645] _raw_spin_lock+0x21/0x30 [18200.060650] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18200.063033] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18200.063181] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18200.063326] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18200.063490] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18200.063633] ? woken_wake_function+0x30/0x30 [18200.063641] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18200.063823] kthread+0x1d1/0x200 [18200.063829] ? set_kthread_struct+0x70/0x70 [18200.063833] ret_from_fork+0x1f/0x30 [18200.066068] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [18200.066081] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18200.066157] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18200.066163] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18200.066166] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18200.066187] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18200.066193] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18200.066198] RAX: 000000000000640e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18200.066200] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18200.066203] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18200.066205] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18200.066207] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18200.066216] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18200.066219] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18200.066221] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18200.066230] Call Trace: [18200.066235] [18200.066240] ? show_regs.cold.9+0x22/0x2f [18200.066251] ? watchdog_timer_fn.cold.6+0x62/0x10a [18200.066258] ? __hrtimer_run_queues+0x1c2/0x4d0 [18200.066262] ? watchdog+0x40/0x40 [18200.066267] ? hrtimer_interrupt+0x128/0x2c0 [18200.066271] ? smp_apic_timer_interrupt+0xa6/0x270 [18200.066280] ? apic_timer_interrupt+0xf/0x20 [18200.066285] [18200.066288] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18200.066295] do_raw_spin_lock+0x177/0x190 [18200.066301] _raw_spin_lock+0x21/0x30 [18200.066306] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18200.066988] ? do_raw_spin_unlock+0x75/0x190 [18200.066995] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18200.067209] ? do_wait_intr+0xf0/0xf0 [18200.067219] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18200.067962] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18200.069049] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18200.069994] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18200.070002] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18200.070009] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18200.069265] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18200.071947] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18200.071986] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18200.072016] mdc_intent_lock+0x353/0x7e0 [mdc] [18200.072045] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18200.072872] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18200.072942] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18200.072949] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18200.072953] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18200.072966] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18200.072971] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18200.072975] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18200.072978] RDX: 0000000000005251 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18200.072980] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18200.072982] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18200.072985] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18200.072988] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18200.072991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18200.072993] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18200.073003] Call Trace: [18200.073008] [18200.073012] ? show_regs.cold.9+0x22/0x2f [18200.073021] ? watchdog_timer_fn.cold.6+0x62/0x10a [18200.073027] ? __hrtimer_run_queues+0x1c2/0x4d0 [18200.073031] ? watchdog+0x40/0x40 [18200.073035] ? hrtimer_interrupt+0x128/0x2c0 [18200.073039] ? smp_apic_timer_interrupt+0xa6/0x270 [18200.073047] ? apic_timer_interrupt+0xf/0x20 [18200.073051] [18200.073054] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18200.073066] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18200.073070] do_raw_spin_lock+0x177/0x190 [18200.073075] _raw_spin_lock+0x21/0x30 [18200.073080] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18200.073372] obd_mod_free+0x46/0x140 [ptlrpc] [18200.073585] mdc_commit_open+0x91/0xb0 [mdc] [18200.073625] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18200.072132] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18200.074119] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18200.074313] after_reply+0xe2c/0x1780 [ptlrpc] [18200.074520] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18200.075963] ? __next_timer_interrupt+0x160/0x160 [18200.075974] ptlrpcd+0xa82/0xe40 [ptlrpc] [18200.076177] ? do_wait_intr+0xf0/0xf0 [18200.076186] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18200.076388] kthread+0x1d1/0x200 [18200.076393] ? set_kthread_struct+0x70/0x70 [18200.076396] ret_from_fork+0x1f/0x30 [18200.077806] [18200.077812] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18200.077824] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18200.077828] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18200.077833] RAX: 0000000000004836 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18200.077835] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [18200.077837] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18200.077839] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18200.077842] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18200.077845] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18200.077848] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18200.077850] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18200.077857] Call Trace: [18200.077860] [18200.077863] ? show_regs.cold.9+0x22/0x2f [18200.077870] ? watchdog_timer_fn.cold.6+0x62/0x10a [18200.077876] ? __hrtimer_run_queues+0x1c2/0x4d0 [18200.077880] ? watchdog+0x40/0x40 [18200.077884] ? hrtimer_interrupt+0x128/0x2c0 [18200.077888] ? smp_apic_timer_interrupt+0xa6/0x270 [18200.077895] ? apic_timer_interrupt+0xf/0x20 [18200.077899] [18200.077902] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18200.077907] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18200.077912] ? newidle_balance.isra.93+0x207/0x580 [18200.077917] do_raw_spin_lock+0x177/0x190 [18200.077921] _raw_spin_lock+0x21/0x30 [18200.077926] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18200.078190] process_one_work+0x2c8/0x700 [18200.078197] worker_thread+0x296/0x6e0 [18200.078202] ? rescuer_thread+0x570/0x570 [18200.078209] kthread+0x1d1/0x200 [18200.078213] ? set_kthread_struct+0x70/0x70 [18200.078218] ret_from_fork+0x1f/0x30 [18200.459572] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18200.461328] lmv_intent_open+0x338/0x1150 [lmv] [18200.462634] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18200.464356] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18200.466122] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18200.467431] ll_intent_lock+0xc5/0xaa0 [lustre] [18200.469083] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18200.471081] ll_lookup_it+0xe0e/0x2270 [lustre] [18200.472533] ? do_raw_spin_unlock+0x75/0x190 [18200.473857] ? _raw_spin_unlock+0x12/0x30 [18200.475158] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18200.476712] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18200.478437] ll_atomic_open+0x36a/0xd70 [lustre] [18200.480121] lookup_open+0x356/0xb40 [18200.481210] do_last+0x313/0xfc0 [18200.482224] ? nd_jump_root+0xe5/0x160 [18200.483395] ? path_init+0x437/0x520 [18200.484464] path_openat+0xf7/0x500 [18200.485481] ? do_raw_spin_unlock+0x75/0x190 [18200.486867] do_filp_open+0x99/0x140 [18200.488308] ? getname_flags+0x6e/0x330 [18200.489480] ? __check_object_size+0xff/0x256 [18200.490867] ? do_raw_spin_unlock+0x75/0x190 [18200.492259] ? _raw_spin_unlock+0x12/0x30 [18200.493499] do_sys_openat2+0x2b4/0x410 [18200.494641] do_sys_open+0x73/0xa0 [18200.496162] __x64_sys_openat+0x24/0x30 [18200.497279] do_syscall_64+0xc1/0x440 [18200.498372] entry_SYSCALL_64_after_hwframe+0x49/0xae [18200.499993] RIP: 0033:0x7f53425a5332 [18200.500962] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18200.506284] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18200.508517] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18200.510818] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18200.513086] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18200.515247] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18200.517457] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18203.104296] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15566s! [18203.117969] Showing busy workqueues and worker pools: [18203.123034] workqueue events: flags=0x0 [18203.125340] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18203.130653] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18203.136900] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18203.140503] pending: cpu_latency_work [ptlrpc] [18203.143213] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18203.146651] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18203.153947] workqueue events_long: flags=0x0 [18203.156766] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18203.160686] pending: lu_objects_flush [obdclass] [18203.164274] workqueue events_unbound: flags=0x2 [18203.167195] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18203.170643] pending: flush_memcg_stats_dwork, idle_cull_fn [18203.173975] workqueue events_power_efficient: flags=0x80 [18203.176487] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18203.180676] pending: check_lifetime, do_cache_clean [18203.183461] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18203.187272] pending: neigh_periodic_work, neigh_periodic_work [18203.191050] workqueue mm_percpu_wq: flags=0x8 [18203.194333] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18203.198647] pending: vmstat_update, lru_add_drain_per_cpu [18203.202502] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18203.206920] pending: vmstat_update, lru_add_drain_per_cpu [18203.210493] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18203.215188] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18203.219460] workqueue xprtiod: flags=0xa [18203.221618] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18203.225228] pending: xprt_autoclose, xs_error_handle [18203.228482] workqueue ipv6_addrconf: flags=0x40008 [18203.233018] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18203.236557] pending: addrconf_verify_work [18203.238234] workqueue knbd0-recv: flags=0x1a [18203.240881] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18203.244847] in-flight: 651:recv_work [18203.247029] workqueue ptlrpc_pinger: flags=0x2 [18203.249512] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18203.253016] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18203.256902] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15566s workers=4 idle: 15979 31824 8 [18203.261637] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18193s workers=2 idle: 654 [18228.052034] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [18228.055944] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [18228.056811] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18228.056954] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18228.056961] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18228.057023] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [18228.057030] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) [18228.056968] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18228.057236] RIP: 0010:native_safe_halt+0x2e/0x50 [18228.057245] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18228.057249] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18228.057254] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18228.057256] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [18228.057268] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18228.057272] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18228.057274] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18228.057278] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18228.057281] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18228.057283] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18228.057292] Call Trace: [18228.057298] [18228.057303] ? show_regs.cold.9+0x22/0x2f [18228.057312] ? watchdog_timer_fn.cold.6+0x62/0x10a [18228.057319] ? __hrtimer_run_queues+0x1c2/0x4d0 [18228.057323] ? watchdog+0x40/0x40 [18228.057328] ? hrtimer_interrupt+0x128/0x2c0 [18228.057333] ? smp_apic_timer_interrupt+0xa6/0x270 [18228.057341] ? apic_timer_interrupt+0xf/0x20 [18228.057828] [18228.057835] ? native_safe_halt+0x2e/0x50 [18228.057841] kvm_wait+0xcd/0xe0 [18228.057849] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18228.057927] ? newidle_balance.isra.93+0x207/0x580 [18228.057934] do_raw_spin_lock+0x177/0x190 [18228.057939] _raw_spin_lock+0x21/0x30 [18228.057944] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18228.058168] process_one_work+0x2c8/0x700 [18228.058175] worker_thread+0x296/0x6e0 [18228.058180] ? rescuer_thread+0x570/0x570 [18228.058184] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [18228.058185] kthread+0x1d1/0x200 [18228.058194] Modules linked in: [18228.058195] ? set_kthread_struct+0x70/0x70 [18228.058199] lustre(O) osp(O) [18228.058203] ret_from_fork+0x1f/0x30 [18228.058208] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18228.058293] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18228.058297] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18228.058298] RIP: 0010:native_safe_halt+0x2e/0x50 [18228.058305] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18228.058307] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18228.058311] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18228.058313] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18228.058314] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18228.058316] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18228.058317] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18228.058320] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18228.058323] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18228.058325] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18228.058332] Call Trace: [18228.058356] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18228.059080] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18228.059085] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18228.059087] RIP: 0010:native_safe_halt+0x2e/0x50 [18228.059094] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18228.059098] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18228.059102] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18228.059105] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18228.059107] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18228.059109] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18228.059111] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18228.059114] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18228.059117] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18228.059120] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18228.059126] Call Trace: [18228.059130] [18228.059134] ? show_regs.cold.9+0x22/0x2f [18228.059144] ? watchdog_timer_fn.cold.6+0x62/0x10a [18228.059149] ? __hrtimer_run_queues+0x1c2/0x4d0 [18228.059154] ? watchdog+0x40/0x40 [18228.059171] ? hrtimer_interrupt+0x128/0x2c0 [18228.059178] ? smp_apic_timer_interrupt+0xa6/0x270 [18228.059183] ? apic_timer_interrupt+0xf/0x20 [18228.059187] [18228.059190] ? native_safe_halt+0x2e/0x50 [18228.059194] kvm_wait+0xcd/0xe0 [18228.059200] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18228.059205] do_raw_spin_lock+0x177/0x190 [18228.059208] _raw_spin_lock+0x21/0x30 [18228.059213] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18228.059673] obd_mod_free+0x46/0x140 [ptlrpc] [18228.059875] mdc_commit_open+0x91/0xb0 [mdc] [18228.059909] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18228.060108] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18228.060334] after_reply+0xe2c/0x1780 [ptlrpc] [18228.060791] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18228.060987] ? __next_timer_interrupt+0x160/0x160 [18228.060997] ptlrpcd+0xa82/0xe40 [ptlrpc] [18228.061499] ? do_wait_intr+0xf0/0xf0 [18228.061511] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18228.061871] kthread+0x1d1/0x200 [18228.061877] ? set_kthread_struct+0x70/0x70 [18228.061881] ret_from_fork+0x1f/0x30 [18228.065737] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18228.065819] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18228.065826] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18228.065829] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18228.065844] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18228.065849] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [18228.104114] [18228.104127] ? show_regs.cold.9+0x22/0x2f [18228.104141] ? watchdog_timer_fn.cold.6+0x62/0x10a [18228.104148] ? __hrtimer_run_queues+0x1c2/0x4d0 [18228.104152] ? watchdog+0x40/0x40 [18228.104157] ? hrtimer_interrupt+0x128/0x2c0 [18228.104160] ? smp_apic_timer_interrupt+0xa6/0x270 [18228.104168] ? apic_timer_interrupt+0xf/0x20 [18228.104173] [18228.104177] ? native_safe_halt+0x2e/0x50 [18228.104181] kvm_wait+0xcd/0xe0 [18228.104189] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18228.104199] do_raw_spin_lock+0x177/0x190 [18228.104203] _raw_spin_lock+0x21/0x30 [18228.104208] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18228.104515] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18228.105121] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18228.105521] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18228.105816] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18228.106098] ? woken_wake_function+0x30/0x30 [18228.106108] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18228.106766] kthread+0x1d1/0x200 [18228.106773] ? set_kthread_struct+0x70/0x70 [18228.106777] ret_from_fork+0x1f/0x30 [18228.826399] ORIG_RAX: ffffffffffffff13 [18228.829475] RAX: 0000000000007181 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18228.835581] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18228.843017] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18228.847116] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18228.853882] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18228.863358] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18228.875538] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18228.881048] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18228.890560] Call Trace: [18228.895543] [18228.897010] ? show_regs.cold.9+0x22/0x2f [18228.901345] ? watchdog_timer_fn.cold.6+0x62/0x10a [18228.903056] ? __hrtimer_run_queues+0x1c2/0x4d0 [18228.906072] ? watchdog+0x40/0x40 [18228.911839] ? hrtimer_interrupt+0x128/0x2c0 [18228.916049] ? smp_apic_timer_interrupt+0xa6/0x270 [18228.921054] ? apic_timer_interrupt+0xf/0x20 [18228.923451] [18228.924365] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18228.928010] do_raw_spin_lock+0x177/0x190 [18228.930325] _raw_spin_lock+0x21/0x30 [18228.936049] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18228.946065] ? do_raw_spin_unlock+0x75/0x190 [18228.951241] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18228.954027] ? do_wait_intr+0xf0/0xf0 [18228.960457] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18228.966816] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18228.971389] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18228.975080] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18228.986296] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18228.996101] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18229.005794] mdc_intent_lock+0x353/0x7e0 [mdc] [18229.009474] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18229.019890] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18229.027425] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18229.035010] lmv_intent_open+0x338/0x1150 [lmv] [18229.037111] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18229.044077] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18229.060558] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18229.067027] ll_intent_lock+0xc5/0xaa0 [lustre] [18229.069929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18229.077557] ll_lookup_it+0xe0e/0x2270 [lustre] [18229.081716] ? do_raw_spin_unlock+0x75/0x190 [18229.085340] ? _raw_spin_unlock+0x12/0x30 [18229.088737] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18229.095096] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18229.101035] ll_atomic_open+0x36a/0xd70 [lustre] [18229.105899] lookup_open+0x356/0xb40 [18229.111087] do_last+0x313/0xfc0 [18229.112648] ? nd_jump_root+0xe5/0x160 [18229.118594] ? path_init+0x437/0x520 [18229.122096] path_openat+0xf7/0x500 [18229.133482] ? do_raw_spin_unlock+0x75/0x190 [18229.137034] do_filp_open+0x99/0x140 [18229.139955] ? getname_flags+0x6e/0x330 [18229.142523] ? __check_object_size+0xff/0x256 [18229.149547] ? do_raw_spin_unlock+0x75/0x190 [18229.164181] ? _raw_spin_unlock+0x12/0x30 [18229.169381] do_sys_openat2+0x2b4/0x410 [18229.172532] do_sys_open+0x73/0xa0 [18229.177500] __x64_sys_openat+0x24/0x30 [18229.181920] do_syscall_64+0xc1/0x440 [18229.185637] entry_SYSCALL_64_after_hwframe+0x49/0xae [18229.189933] RIP: 0033:0x7f53425a5332 [18229.195298] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18229.205650] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18229.210063] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18229.217379] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18229.221042] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18229.225887] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18229.229080] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18233.313344] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15596s! [18233.321389] Showing busy workqueues and worker pools: [18233.326472] workqueue events: flags=0x0 [18233.330582] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18233.336844] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18233.346671] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18233.352954] pending: cpu_latency_work [ptlrpc] [18233.359837] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18233.370039] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18233.378614] workqueue events_long: flags=0x0 [18233.380484] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18233.386564] pending: lu_objects_flush [obdclass] [18233.389106] workqueue events_unbound: flags=0x2 [18233.391450] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18233.394141] pending: flush_memcg_stats_dwork, idle_cull_fn [18233.396896] workqueue events_power_efficient: flags=0x80 [18233.402641] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18233.409352] pending: check_lifetime, do_cache_clean [18233.412144] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18233.414834] pending: neigh_periodic_work, neigh_periodic_work [18233.417508] workqueue mm_percpu_wq: flags=0x8 [18233.424245] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18233.430010] pending: vmstat_update, lru_add_drain_per_cpu [18233.435157] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18233.444771] pending: vmstat_update, lru_add_drain_per_cpu [18233.446881] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18233.454448] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18233.456103] workqueue xprtiod: flags=0xa [18233.457747] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18233.462368] pending: xprt_autoclose, xs_error_handle [18233.465124] workqueue ipv6_addrconf: flags=0x40008 [18233.467493] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18233.470464] pending: addrconf_verify_work [18233.473559] workqueue knbd0-recv: flags=0x1a [18233.476070] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18233.478281] in-flight: 651:recv_work [18233.483675] workqueue ptlrpc_pinger: flags=0x2 [18233.485653] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18233.488621] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18233.491535] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15596s workers=4 idle: 15979 31824 8 [18233.498939] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18223s workers=2 idle: 654 [18256.055108] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [18256.057032] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 [18256.057025] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [18256.057088] pcspkr [18256.057093] Modules linked in: [18256.058236] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [18256.058245] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18256.058327] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18256.058332] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18256.058335] RIP: 0010:native_safe_halt+0x2e/0x50 [18256.058348] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18256.058353] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18256.058357] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18256.058360] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18256.058362] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18256.058364] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18256.058367] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18256.058369] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18256.058372] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18256.058374] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18256.058384] Call Trace: [18256.058390] [18256.058395] ? show_regs.cold.9+0x22/0x2f [18256.058405] ? watchdog_timer_fn.cold.6+0x62/0x10a [18256.058720] ? __hrtimer_run_queues+0x1c2/0x4d0 [18256.058724] ? watchdog+0x40/0x40 [18256.058729] ? hrtimer_interrupt+0x128/0x2c0 [18256.058732] ? smp_apic_timer_interrupt+0xa6/0x270 [18256.058740] ? apic_timer_interrupt+0xf/0x20 [18256.058745] [18256.058748] ? native_safe_halt+0x2e/0x50 [18256.058753] kvm_wait+0xcd/0xe0 [18256.058761] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18256.058780] do_raw_spin_lock+0x177/0x190 [18256.058786] _raw_spin_lock+0x21/0x30 [18256.058791] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18256.059048] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18256.059821] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18256.060062] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18256.060924] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18256.061152] ? woken_wake_function+0x30/0x30 [18256.061768] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18256.061998] kthread+0x1d1/0x200 [18256.062005] ? set_kthread_struct+0x70/0x70 [18256.062009] ret_from_fork+0x1f/0x30 [18256.065874] squashfs [18256.068712] lustre(O) [18256.069303] ata_generic [18256.070702] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18256.070834] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18256.070841] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18256.070844] RIP: 0010:native_safe_halt+0x2e/0x50 [18256.070857] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18256.070861] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18256.070865] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18256.070867] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18256.070869] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18256.070872] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18256.070874] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18256.070877] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18256.070880] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18256.070883] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18256.070891] Call Trace: [18256.070896] [18256.070902] ? show_regs.cold.9+0x22/0x2f [18256.070911] ? watchdog_timer_fn.cold.6+0x62/0x10a [18256.070917] ? __hrtimer_run_queues+0x1c2/0x4d0 [18256.070922] ? watchdog+0x40/0x40 [18256.070926] ? hrtimer_interrupt+0x128/0x2c0 [18256.070930] ? smp_apic_timer_interrupt+0xa6/0x270 [18256.070937] ? apic_timer_interrupt+0xf/0x20 [18256.070942] [18256.070945] ? native_safe_halt+0x2e/0x50 [18256.070949] kvm_wait+0xcd/0xe0 [18256.070956] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18256.070963] do_raw_spin_lock+0x177/0x190 [18256.070968] _raw_spin_lock+0x21/0x30 [18256.070974] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18256.071333] obd_mod_free+0x46/0x140 [ptlrpc] [18256.071546] mdc_commit_open+0x91/0xb0 [mdc] [18256.071582] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18256.072184] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18256.072388] after_reply+0xe2c/0x1780 [ptlrpc] [18256.072589] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18256.072932] ? __next_timer_interrupt+0x160/0x160 [18256.072942] ptlrpcd+0xa82/0xe40 [ptlrpc] [18256.073357] ? do_wait_intr+0xf0/0xf0 [18256.073367] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18256.073576] kthread+0x1d1/0x200 [18256.073582] ? set_kthread_struct+0x70/0x70 [18256.073586] ret_from_fork+0x1f/0x30 [18256.389941] crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18256.404329] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18256.417237] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18256.421873] RIP: 0010:native_safe_halt+0x2e/0x50 [18256.423407] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18256.434886] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18256.442184] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18256.448352] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18256.453287] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18256.460241] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18256.471892] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18256.476332] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18256.481457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18256.487221] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18256.499136] Call Trace: [18256.499813] [18256.500375] ? show_regs.cold.9+0x22/0x2f [18256.501501] ? watchdog_timer_fn.cold.6+0x62/0x10a [18256.506937] ? __hrtimer_run_queues+0x1c2/0x4d0 [18256.508723] ? watchdog+0x40/0x40 [18256.510040] ? hrtimer_interrupt+0x128/0x2c0 [18256.511528] ? smp_apic_timer_interrupt+0xa6/0x270 [18256.515357] ? apic_timer_interrupt+0xf/0x20 [18256.516648] [18256.518234] ? native_safe_halt+0x2e/0x50 [18256.521846] kvm_wait+0xcd/0xe0 [18256.522770] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18256.526036] do_raw_spin_lock+0x177/0x190 [18256.529512] _raw_spin_lock+0x21/0x30 [18256.534275] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18256.536364] ? do_raw_spin_unlock+0x75/0x190 [18256.540483] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18256.542991] ? do_wait_intr+0xf0/0xf0 [18256.544477] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18256.547379] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18256.555632] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18256.557725] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18256.562083] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18256.565533] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18256.569664] mdc_intent_lock+0x353/0x7e0 [mdc] [18256.572382] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18256.575340] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18256.578062] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18256.581240] lmv_intent_open+0x338/0x1150 [lmv] [18256.584455] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18256.586435] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18256.588783] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18256.592113] ll_intent_lock+0xc5/0xaa0 [lustre] [18256.594210] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18256.600853] ll_lookup_it+0xe0e/0x2270 [lustre] [18256.602579] ? do_raw_spin_unlock+0x75/0x190 [18256.607258] ? _raw_spin_unlock+0x12/0x30 [18256.608497] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18256.610517] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18256.613209] ll_atomic_open+0x36a/0xd70 [lustre] [18256.616156] lookup_open+0x356/0xb40 [18256.617730] do_last+0x313/0xfc0 [18256.619239] ? nd_jump_root+0xe5/0x160 [18256.620772] ? path_init+0x437/0x520 [18256.622442] path_openat+0xf7/0x500 [18256.624175] ? do_raw_spin_unlock+0x75/0x190 [18256.626027] do_filp_open+0x99/0x140 [18256.627353] ? getname_flags+0x6e/0x330 [18256.635339] ? __check_object_size+0xff/0x256 [18256.640977] ? do_raw_spin_unlock+0x75/0x190 [18256.645755] ? _raw_spin_unlock+0x12/0x30 [18256.648754] do_sys_openat2+0x2b4/0x410 [18256.653424] do_sys_open+0x73/0xa0 [18256.657594] __x64_sys_openat+0x24/0x30 [18256.667457] do_syscall_64+0xc1/0x440 [18256.672402] entry_SYSCALL_64_after_hwframe+0x49/0xae [18256.681063] RIP: 0033:0x7f53425a5332 [18256.683416] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18256.705318] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18256.720553] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18256.736057] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18256.738319] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18256.744108] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18256.753459] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18260.052025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [18260.056645] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18260.092513] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18260.101344] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18260.122460] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18260.135779] RIP: 0010:native_safe_halt+0x2e/0x50 [18260.144009] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18260.157626] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18260.161895] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18260.165970] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [18260.171620] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18260.179511] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18260.186769] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18260.196425] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18260.206306] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18260.210672] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18260.212679] Call Trace: [18260.213560] [18260.218489] ? show_regs.cold.9+0x22/0x2f [18260.221584] ? watchdog_timer_fn.cold.6+0x62/0x10a [18260.225373] ? __hrtimer_run_queues+0x1c2/0x4d0 [18260.230045] ? watchdog+0x40/0x40 [18260.233269] ? hrtimer_interrupt+0x128/0x2c0 [18260.238901] ? smp_apic_timer_interrupt+0xa6/0x270 [18260.242829] ? apic_timer_interrupt+0xf/0x20 [18260.245813] [18260.246470] ? native_safe_halt+0x2e/0x50 [18260.248353] kvm_wait+0xcd/0xe0 [18260.252930] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18260.257462] ? newidle_balance.isra.93+0x207/0x580 [18260.262627] do_raw_spin_lock+0x177/0x190 [18260.266968] _raw_spin_lock+0x21/0x30 [18260.271096] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18260.275881] process_one_work+0x2c8/0x700 [18260.281483] worker_thread+0x296/0x6e0 [18260.285008] ? rescuer_thread+0x570/0x570 [18260.290042] kthread+0x1d1/0x200 [18260.293482] ? set_kthread_struct+0x70/0x70 [18260.297604] ret_from_fork+0x1f/0x30 [18263.520096] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15626s! [18263.528564] Showing busy workqueues and worker pools: [18263.536358] workqueue events: flags=0x0 [18263.541225] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18263.548033] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18263.553688] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18263.557765] pending: cpu_latency_work [ptlrpc] [18263.560848] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18263.564057] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18263.569115] workqueue events_long: flags=0x0 [18263.570967] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18263.574034] pending: lu_objects_flush [obdclass] [18263.576955] workqueue events_unbound: flags=0x2 [18263.578379] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18263.581566] pending: flush_memcg_stats_dwork, idle_cull_fn [18263.585096] workqueue events_power_efficient: flags=0x80 [18263.592120] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18263.597097] pending: check_lifetime, do_cache_clean [18263.599541] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18263.603013] pending: neigh_periodic_work, neigh_periodic_work [18263.606151] workqueue mm_percpu_wq: flags=0x8 [18263.611878] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18263.618317] pending: vmstat_update, lru_add_drain_per_cpu [18263.626770] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18263.634913] pending: vmstat_update, lru_add_drain_per_cpu [18263.640951] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18263.645836] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18263.651032] workqueue xprtiod: flags=0xa [18263.652523] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18263.655544] pending: xprt_autoclose, xs_error_handle [18263.658288] workqueue ipv6_addrconf: flags=0x40008 [18263.660820] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18263.664118] pending: addrconf_verify_work [18263.666421] workqueue knbd0-recv: flags=0x1a [18263.668122] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18263.670757] in-flight: 651:recv_work [18263.672467] workqueue ptlrpc_pinger: flags=0x2 [18263.674879] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18263.677348] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18263.680051] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15626s workers=4 idle: 15979 31824 8 [18263.683813] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18254s workers=2 idle: 654 [18284.060428] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18284.060988] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18284.065704] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [18284.065721] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18284.065790] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18284.065796] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18284.065799] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18284.065815] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18284.065819] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18284.065825] RAX: 0000000000007c89 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18284.065827] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18284.065829] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18284.065832] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18284.065835] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18284.065839] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18284.065842] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18284.065844] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18284.065852] Call Trace: [18284.065857] [18284.065862] ? show_regs.cold.9+0x22/0x2f [18284.065872] ? watchdog_timer_fn.cold.6+0x62/0x10a [18284.065878] ? __hrtimer_run_queues+0x1c2/0x4d0 [18284.065882] ? watchdog+0x40/0x40 [18284.065886] ? hrtimer_interrupt+0x128/0x2c0 [18284.065889] ? smp_apic_timer_interrupt+0xa6/0x270 [18284.065897] ? apic_timer_interrupt+0xf/0x20 [18284.065901] [18284.065903] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18284.065909] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18284.065914] do_raw_spin_lock+0x177/0x190 [18284.065919] _raw_spin_lock+0x21/0x30 [18284.065923] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18284.068119] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18284.068967] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18284.069041] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18284.069047] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18284.069050] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18284.069063] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18284.069068] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18284.069073] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18284.069075] RDX: 0000000000001a4c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18284.069078] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18284.069080] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18284.069083] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18284.069086] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18284.069089] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18284.069091] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18284.069100] Call Trace: [18284.069104] [18284.069108] ? show_regs.cold.9+0x22/0x2f [18284.069117] ? watchdog_timer_fn.cold.6+0x62/0x10a [18284.069123] ? __hrtimer_run_queues+0x1c2/0x4d0 [18284.069127] ? watchdog+0x40/0x40 [18284.069132] ? hrtimer_interrupt+0x128/0x2c0 [18284.069136] ? smp_apic_timer_interrupt+0xa6/0x270 [18284.069144] ? apic_timer_interrupt+0xf/0x20 [18284.069149] [18284.069152] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18284.069157] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18284.069162] do_raw_spin_lock+0x177/0x190 [18284.069166] _raw_spin_lock+0x21/0x30 [18284.069341] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18284.069172] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18284.070530] obd_mod_free+0x46/0x140 [ptlrpc] [18284.070724] mdc_commit_open+0x91/0xb0 [mdc] [18284.070758] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18284.071112] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18284.071324] after_reply+0xe2c/0x1780 [ptlrpc] [18284.071737] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18284.073782] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18284.077389] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18284.077455] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18284.077461] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18284.077464] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18284.077475] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18284.077479] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18284.077483] RAX: 0000000000007e16 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18284.077485] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18284.077487] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18284.077489] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18284.077491] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18284.077494] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18284.077496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18284.077499] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18284.077506] Call Trace: [18284.077511] [18284.077516] ? show_regs.cold.9+0x22/0x2f [18284.077525] ? watchdog_timer_fn.cold.6+0x62/0x10a [18284.077531] ? __hrtimer_run_queues+0x1c2/0x4d0 [18284.077535] ? watchdog+0x40/0x40 [18284.077538] ? hrtimer_interrupt+0x128/0x2c0 [18284.077542] ? smp_apic_timer_interrupt+0xa6/0x270 [18284.077549] ? apic_timer_interrupt+0xf/0x20 [18284.077553] [18284.077556] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18284.077561] do_raw_spin_lock+0x177/0x190 [18284.077565] _raw_spin_lock+0x21/0x30 [18284.077570] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18284.077807] ? do_raw_spin_unlock+0x75/0x190 [18284.077812] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18284.078003] ? do_wait_intr+0xf0/0xf0 [18284.078011] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18284.078119] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18284.078486] ? woken_wake_function+0x30/0x30 [18284.078504] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18284.082475] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18284.082718] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18284.082929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18284.083043] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18284.083079] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18284.083108] mdc_intent_lock+0x353/0x7e0 [mdc] [18284.083132] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18284.083369] kthread+0x1d1/0x200 [18284.083377] ? set_kthread_struct+0x70/0x70 [18284.083382] ret_from_fork+0x1f/0x30 [18284.083213] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18284.087697] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18284.087739] lmv_intent_open+0x338/0x1150 [lmv] [18284.087769] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18284.087887] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18284.087988] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18284.088014] ll_intent_lock+0xc5/0xaa0 [lustre] [18284.088099] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18284.088180] ll_lookup_it+0xe0e/0x2270 [lustre] [18284.088263] ? do_raw_spin_unlock+0x75/0x190 [18284.088273] ? _raw_spin_unlock+0x12/0x30 [18284.099824] ? __next_timer_interrupt+0x160/0x160 [18284.099846] ptlrpcd+0xa82/0xe40 [ptlrpc] [18284.100346] ? do_wait_intr+0xf0/0xf0 [18284.100356] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18284.101434] kthread+0x1d1/0x200 [18284.101445] ? set_kthread_struct+0x70/0x70 [18284.101449] ret_from_fork+0x1f/0x30 [18284.974820] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18284.979762] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18284.983126] ll_atomic_open+0x36a/0xd70 [lustre] [18284.985908] lookup_open+0x356/0xb40 [18284.988080] do_last+0x313/0xfc0 [18284.989874] ? nd_jump_root+0xe5/0x160 [18284.995410] ? path_init+0x437/0x520 [18284.997499] path_openat+0xf7/0x500 [18284.998751] ? do_raw_spin_unlock+0x75/0x190 [18285.000801] do_filp_open+0x99/0x140 [18285.001775] ? getname_flags+0x6e/0x330 [18285.004886] ? __check_object_size+0xff/0x256 [18285.006796] ? do_raw_spin_unlock+0x75/0x190 [18285.009896] ? _raw_spin_unlock+0x12/0x30 [18285.011113] do_sys_openat2+0x2b4/0x410 [18285.017154] do_sys_open+0x73/0xa0 [18285.020955] __x64_sys_openat+0x24/0x30 [18285.023665] do_syscall_64+0xc1/0x440 [18285.032506] entry_SYSCALL_64_after_hwframe+0x49/0xae [18285.039782] RIP: 0033:0x7f53425a5332 [18285.042616] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18285.053180] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18285.062717] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18285.066103] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18285.069650] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18285.072681] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18285.081409] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18288.053653] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [18288.057047] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18288.108798] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18288.127692] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18288.132641] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18288.137905] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18288.140802] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18288.151346] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18288.155054] RAX: 000000000000298e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18288.158094] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [18288.161396] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18288.164798] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18288.168317] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18288.171415] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18288.175069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18288.177759] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18288.181097] Call Trace: [18288.182711] [18288.183438] ? show_regs.cold.9+0x22/0x2f [18288.189615] ? watchdog_timer_fn.cold.6+0x62/0x10a [18288.194057] ? __hrtimer_run_queues+0x1c2/0x4d0 [18288.196281] ? watchdog+0x40/0x40 [18288.198116] ? hrtimer_interrupt+0x128/0x2c0 [18288.203434] ? smp_apic_timer_interrupt+0xa6/0x270 [18288.208545] ? apic_timer_interrupt+0xf/0x20 [18288.212155] [18288.214346] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18288.225902] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18288.238521] ? newidle_balance.isra.93+0x207/0x580 [18288.240572] do_raw_spin_lock+0x177/0x190 [18288.242114] _raw_spin_lock+0x21/0x30 [18288.255372] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18288.260928] process_one_work+0x2c8/0x700 [18288.263448] worker_thread+0x296/0x6e0 [18288.266145] ? rescuer_thread+0x570/0x570 [18288.271608] kthread+0x1d1/0x200 [18288.273438] ? set_kthread_struct+0x70/0x70 [18288.275814] ret_from_fork+0x1f/0x30 [18293.734095] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15656s! [18293.749241] Showing busy workqueues and worker pools: [18293.750837] workqueue events: flags=0x0 [18293.757378] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18293.768950] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18293.789369] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18293.798203] pending: cpu_latency_work [ptlrpc] [18293.800092] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18293.810984] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18293.827877] workqueue events_long: flags=0x0 [18293.838234] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18293.850408] pending: lu_objects_flush [obdclass] [18293.858622] workqueue events_unbound: flags=0x2 [18293.865186] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18293.866643] pending: flush_memcg_stats_dwork, idle_cull_fn [18293.874801] workqueue events_power_efficient: flags=0x80 [18293.879526] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18293.891392] pending: check_lifetime, do_cache_clean [18293.903960] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18293.915009] pending: neigh_periodic_work, neigh_periodic_work [18293.917233] workqueue mm_percpu_wq: flags=0x8 [18293.927445] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18293.935508] pending: vmstat_update, lru_add_drain_per_cpu [18293.944832] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18293.952103] pending: vmstat_update, lru_add_drain_per_cpu [18293.966030] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18293.977691] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18293.988892] workqueue xprtiod: flags=0xa [18293.990142] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18294.005423] pending: xprt_autoclose, xs_error_handle [18294.007399] workqueue ipv6_addrconf: flags=0x40008 [18294.013939] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18294.025978] pending: addrconf_verify_work [18294.027418] workqueue knbd0-recv: flags=0x1a [18294.049086] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18294.057883] in-flight: 651:recv_work [18294.059314] workqueue ptlrpc_pinger: flags=0x2 [18294.062327] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18294.066740] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18294.074939] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15656s workers=4 idle: 15979 31824 8 [18294.086288] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18284s workers=2 idle: 654 [18312.055736] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18312.058573] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18312.060270] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18312.061233] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [18312.061240] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18312.061310] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18312.061316] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18312.061318] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18312.061333] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18312.061337] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18312.061342] RAX: 000000000000690a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18312.061344] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18312.061346] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18312.061348] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18312.061351] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18312.061354] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18312.061357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18312.061359] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18312.061367] Call Trace: [18312.061373] [18312.061378] ? show_regs.cold.9+0x22/0x2f [18312.061388] ? watchdog_timer_fn.cold.6+0x62/0x10a [18312.061394] ? __hrtimer_run_queues+0x1c2/0x4d0 [18312.061398] ? watchdog+0x40/0x40 [18312.061403] ? hrtimer_interrupt+0x128/0x2c0 [18312.061408] ? smp_apic_timer_interrupt+0xa6/0x270 [18312.061416] ? apic_timer_interrupt+0xf/0x20 [18312.061421] [18312.061424] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18312.061429] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18312.061436] do_raw_spin_lock+0x177/0x190 [18312.061440] _raw_spin_lock+0x21/0x30 [18312.061445] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18312.062431] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18312.062740] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18312.063050] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18312.063262] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18312.063471] ? woken_wake_function+0x30/0x30 [18312.063479] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18312.064422] kthread+0x1d1/0x200 [18312.064429] ? set_kthread_struct+0x70/0x70 [18312.064433] ret_from_fork+0x1f/0x30 [18312.126780] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18312.126850] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18312.126853] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18312.126868] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18312.126873] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18312.126878] RAX: 0000000000005e2f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18312.126880] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18312.126882] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18312.126884] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18312.126886] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18312.126889] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18312.126892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18312.126895] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18312.126905] Call Trace: [18312.126910] [18312.126916] ? show_regs.cold.9+0x22/0x2f [18312.126925] ? watchdog_timer_fn.cold.6+0x62/0x10a [18312.126932] ? __hrtimer_run_queues+0x1c2/0x4d0 [18312.126935] ? watchdog+0x40/0x40 [18312.126940] ? hrtimer_interrupt+0x128/0x2c0 [18312.126944] ? smp_apic_timer_interrupt+0xa6/0x270 [18312.126953] ? apic_timer_interrupt+0xf/0x20 [18312.126958] [18312.126961] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18312.126969] do_raw_spin_lock+0x177/0x190 [18312.126974] _raw_spin_lock+0x21/0x30 [18312.126979] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18312.128298] ? do_raw_spin_unlock+0x75/0x190 [18312.128305] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18312.128642] ? do_wait_intr+0xf0/0xf0 [18312.128651] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18312.128977] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18312.129286] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18312.129639] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18312.129763] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18312.130277] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18312.130310] mdc_intent_lock+0x353/0x7e0 [mdc] [18312.130337] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18312.130439] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18312.136180] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18312.136261] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18312.136268] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18312.136271] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18312.136301] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18312.136306] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18312.136311] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18312.136313] RDX: 0000000000005e3e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18312.136316] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18312.136318] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18312.136321] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18312.136324] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18312.136327] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18312.136330] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18312.136338] Call Trace: [18312.136344] [18312.136349] ? show_regs.cold.9+0x22/0x2f [18312.136359] ? watchdog_timer_fn.cold.6+0x62/0x10a [18312.136365] ? __hrtimer_run_queues+0x1c2/0x4d0 [18312.136369] ? watchdog+0x40/0x40 [18312.136374] ? hrtimer_interrupt+0x128/0x2c0 [18312.136379] ? smp_apic_timer_interrupt+0xa6/0x270 [18312.136389] ? apic_timer_interrupt+0xf/0x20 [18312.136393] [18312.136397] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18312.136402] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18312.136407] do_raw_spin_lock+0x177/0x190 [18312.136418] _raw_spin_lock+0x21/0x30 [18312.136427] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18312.136989] obd_mod_free+0x46/0x140 [ptlrpc] [18312.137287] mdc_commit_open+0x91/0xb0 [mdc] [18312.137383] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18312.138023] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18312.138230] after_reply+0xe2c/0x1780 [ptlrpc] [18312.148225] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18312.148278] lmv_intent_open+0x338/0x1150 [lmv] [18312.148312] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18312.149445] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18312.149755] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18312.149783] ll_intent_lock+0xc5/0xaa0 [lustre] [18312.149939] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18312.150027] ll_lookup_it+0xe0e/0x2270 [lustre] [18312.150366] ? do_raw_spin_unlock+0x75/0x190 [18312.150375] ? _raw_spin_unlock+0x12/0x30 [18312.150446] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18312.150858] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18312.150881] ll_atomic_open+0x36a/0xd70 [lustre] [18312.150971] lookup_open+0x356/0xb40 [18312.150980] do_last+0x313/0xfc0 [18312.150985] ? nd_jump_root+0xe5/0x160 [18312.150989] ? path_init+0x437/0x520 [18312.150994] path_openat+0xf7/0x500 [18312.151000] ? do_raw_spin_unlock+0x75/0x190 [18312.151004] do_filp_open+0x99/0x140 [18312.151011] ? getname_flags+0x6e/0x330 [18312.151016] ? __check_object_size+0xff/0x256 [18312.151023] ? do_raw_spin_unlock+0x75/0x190 [18312.151027] ? _raw_spin_unlock+0x12/0x30 [18312.151031] do_sys_openat2+0x2b4/0x410 [18312.151036] do_sys_open+0x73/0xa0 [18312.151041] __x64_sys_openat+0x24/0x30 [18312.151045] do_syscall_64+0xc1/0x440 [18312.151051] entry_SYSCALL_64_after_hwframe+0x49/0xae [18312.151058] RIP: 0033:0x7f53425a5332 [18312.151066] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18312.151070] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18312.151076] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18312.210356] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18312.210702] ? __next_timer_interrupt+0x160/0x160 [18312.210714] ptlrpcd+0xa82/0xe40 [ptlrpc] [18312.229344] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18312.229356] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18312.229359] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18312.229361] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18312.878359] ? do_wait_intr+0xf0/0xf0 [18312.883932] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18312.891841] kthread+0x1d1/0x200 [18312.896465] ? set_kthread_struct+0x70/0x70 [18312.901547] ret_from_fork+0x1f/0x30 [18320.054853] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [18320.059007] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18320.113885] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18320.124154] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18320.129547] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18320.138473] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18320.151741] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18320.164259] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18320.166839] RAX: 0000000000007f84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18320.180483] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [18320.187559] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18320.192325] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18320.197333] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18320.203632] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18320.217323] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18320.220624] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18320.225247] Call Trace: [18320.227058] [18320.232106] ? show_regs.cold.9+0x22/0x2f [18320.239093] ? watchdog_timer_fn.cold.6+0x62/0x10a [18320.243073] ? __hrtimer_run_queues+0x1c2/0x4d0 [18320.245554] ? watchdog+0x40/0x40 [18320.247224] ? hrtimer_interrupt+0x128/0x2c0 [18320.251820] ? smp_apic_timer_interrupt+0xa6/0x270 [18320.258441] ? apic_timer_interrupt+0xf/0x20 [18320.261978] [18320.262784] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18320.267426] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18320.273258] ? newidle_balance.isra.93+0x207/0x580 [18320.279222] do_raw_spin_lock+0x177/0x190 [18320.283589] _raw_spin_lock+0x21/0x30 [18320.285707] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18320.291104] process_one_work+0x2c8/0x700 [18320.292417] worker_thread+0x296/0x6e0 [18320.306549] ? rescuer_thread+0x570/0x570 [18320.310268] kthread+0x1d1/0x200 [18320.314118] ? set_kthread_struct+0x70/0x70 [18320.319270] ret_from_fork+0x1f/0x30 [18324.451043] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15687s! [18324.459609] Showing busy workqueues and worker pools: [18324.464442] workqueue events: flags=0x0 [18324.474081] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18324.480012] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18324.490672] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18324.499053] pending: cpu_latency_work [ptlrpc] [18324.502111] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18324.505988] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18324.518619] workqueue events_long: flags=0x0 [18324.523427] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18324.530273] pending: lu_objects_flush [obdclass] [18324.534050] workqueue events_unbound: flags=0x2 [18324.536732] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18324.540192] pending: flush_memcg_stats_dwork, idle_cull_fn [18324.544839] workqueue events_power_efficient: flags=0x80 [18324.547323] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18324.549370] pending: check_lifetime, do_cache_clean [18324.550907] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18324.553219] pending: neigh_periodic_work, neigh_periodic_work [18324.555117] workqueue mm_percpu_wq: flags=0x8 [18324.557035] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18324.560767] pending: vmstat_update, lru_add_drain_per_cpu [18324.563264] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18324.566706] pending: vmstat_update, lru_add_drain_per_cpu [18324.573783] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18324.579125] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18324.582692] workqueue xprtiod: flags=0xa [18324.591148] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18324.600146] pending: xprt_autoclose, xs_error_handle [18324.608549] workqueue ipv6_addrconf: flags=0x40008 [18324.611530] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18324.617512] pending: addrconf_verify_work [18324.621246] workqueue knbd0-recv: flags=0x1a [18324.624174] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18324.635292] in-flight: 651:recv_work [18324.638379] workqueue ptlrpc_pinger: flags=0x2 [18324.640489] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18324.643537] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18324.649236] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15687s workers=4 idle: 15979 31824 8 [18324.655778] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18315s workers=2 idle: 654 [18340.057107] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18340.057758] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18340.058077] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [18340.058084] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18340.058150] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18340.058155] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18340.058158] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18340.058172] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18340.058176] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18340.058180] RAX: 0000000000007713 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18340.058183] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18340.058185] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18340.058188] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18340.058190] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18340.058193] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18340.058206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18340.058209] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18340.058216] Call Trace: [18340.058222] [18340.058228] ? show_regs.cold.9+0x22/0x2f [18340.058237] ? watchdog_timer_fn.cold.6+0x62/0x10a [18340.058243] ? __hrtimer_run_queues+0x1c2/0x4d0 [18340.058248] ? watchdog+0x40/0x40 [18340.058253] ? hrtimer_interrupt+0x128/0x2c0 [18340.058257] ? smp_apic_timer_interrupt+0xa6/0x270 [18340.058266] ? apic_timer_interrupt+0xf/0x20 [18340.058270] [18340.058273] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18340.058278] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18340.058284] do_raw_spin_lock+0x177/0x190 [18340.058289] _raw_spin_lock+0x21/0x30 [18340.058295] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18340.058568] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18340.058756] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18340.058962] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18340.059390] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18340.059598] ? woken_wake_function+0x30/0x30 [18340.059607] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18340.059810] kthread+0x1d1/0x200 [18340.059816] ? set_kthread_struct+0x70/0x70 [18340.059820] ret_from_fork+0x1f/0x30 [18340.061520] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18340.061601] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18340.061608] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18340.061611] RIP: 0010:native_safe_halt+0x2e/0x50 [18340.061634] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18340.061639] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18340.061643] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18340.061651] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18340.061653] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18340.061655] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18340.061658] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18340.061661] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18340.061665] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18340.061667] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18340.061674] Call Trace: [18340.061679] [18340.061684] ? show_regs.cold.9+0x22/0x2f [18340.061693] ? watchdog_timer_fn.cold.6+0x62/0x10a [18340.061700] ? __hrtimer_run_queues+0x1c2/0x4d0 [18340.061705] ? watchdog+0x40/0x40 [18340.061709] ? hrtimer_interrupt+0x128/0x2c0 [18340.061719] ? smp_apic_timer_interrupt+0xa6/0x270 [18340.061732] ? apic_timer_interrupt+0xf/0x20 [18340.061737] [18340.061740] ? native_safe_halt+0x2e/0x50 [18340.061745] kvm_wait+0xcd/0xe0 [18340.061753] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18340.061761] do_raw_spin_lock+0x177/0x190 [18340.061766] _raw_spin_lock+0x21/0x30 [18340.061771] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18340.062048] obd_mod_free+0x46/0x140 [ptlrpc] [18340.062271] mdc_commit_open+0x91/0xb0 [mdc] [18340.062321] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18340.062624] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18340.074958] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18340.075034] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18340.075041] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18340.075044] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18340.075058] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18340.075062] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18340.075067] RAX: 0000000000007e0c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18340.075069] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18340.075071] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18340.075073] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18340.075076] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18340.075079] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18340.075082] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18340.075084] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18340.075093] Call Trace: [18340.075099] [18340.075105] ? show_regs.cold.9+0x22/0x2f [18340.075115] ? watchdog_timer_fn.cold.6+0x62/0x10a [18340.075121] ? __hrtimer_run_queues+0x1c2/0x4d0 [18340.075125] ? watchdog+0x40/0x40 [18340.075129] ? hrtimer_interrupt+0x128/0x2c0 [18340.075208] ? smp_apic_timer_interrupt+0xa6/0x270 [18340.075216] ? apic_timer_interrupt+0xf/0x20 [18340.075220] [18340.075224] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18340.075231] do_raw_spin_lock+0x177/0x190 [18340.075236] _raw_spin_lock+0x21/0x30 [18340.075241] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18340.078859] after_reply+0xe2c/0x1780 [ptlrpc] [18340.079161] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18340.079566] ? __next_timer_interrupt+0x160/0x160 [18340.079576] ptlrpcd+0xa82/0xe40 [ptlrpc] [18340.079879] ? do_wait_intr+0xf0/0xf0 [18340.079893] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18340.080147] kthread+0x1d1/0x200 [18340.080153] ? set_kthread_struct+0x70/0x70 [18340.080157] ret_from_fork+0x1f/0x30 [18340.719978] ? do_raw_spin_unlock+0x75/0x190 [18340.726321] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18340.730394] ? do_wait_intr+0xf0/0xf0 [18340.733099] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18340.736226] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18340.743053] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18340.749593] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18340.755174] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18340.757740] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18340.763898] mdc_intent_lock+0x353/0x7e0 [mdc] [18340.765923] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18340.770775] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18340.779350] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18340.784213] lmv_intent_open+0x338/0x1150 [lmv] [18340.787049] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18340.790431] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18340.798214] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18340.805411] ll_intent_lock+0xc5/0xaa0 [lustre] [18340.809363] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18340.814345] ll_lookup_it+0xe0e/0x2270 [lustre] [18340.825722] ? do_raw_spin_unlock+0x75/0x190 [18340.827576] ? _raw_spin_unlock+0x12/0x30 [18340.834534] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18340.836570] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18340.844229] ll_atomic_open+0x36a/0xd70 [lustre] [18340.851903] lookup_open+0x356/0xb40 [18340.855258] do_last+0x313/0xfc0 [18340.859499] ? nd_jump_root+0xe5/0x160 [18340.865176] ? path_init+0x437/0x520 [18340.871409] path_openat+0xf7/0x500 [18340.877549] ? do_raw_spin_unlock+0x75/0x190 [18340.881259] do_filp_open+0x99/0x140 [18340.886500] ? getname_flags+0x6e/0x330 [18340.889577] ? __check_object_size+0xff/0x256 [18340.893532] ? do_raw_spin_unlock+0x75/0x190 [18340.897518] ? _raw_spin_unlock+0x12/0x30 [18340.900450] do_sys_openat2+0x2b4/0x410 [18340.903479] do_sys_open+0x73/0xa0 [18340.905975] __x64_sys_openat+0x24/0x30 [18340.908818] do_syscall_64+0xc1/0x440 [18340.911456] entry_SYSCALL_64_after_hwframe+0x49/0xae [18340.914486] RIP: 0033:0x7f53425a5332 [18340.916836] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18340.928943] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18340.933116] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18340.937362] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18340.941485] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18340.945239] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18340.948428] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18352.052184] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [18352.060701] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18352.105303] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18352.116407] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18352.124080] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18352.135080] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18352.140623] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18352.161877] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18352.165188] RAX: 00000000000070d6 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18352.178166] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [18352.185275] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18352.191769] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18352.195874] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18352.200387] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18352.211688] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18352.219341] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18352.228216] Call Trace: [18352.229696] [18352.231763] ? show_regs.cold.9+0x22/0x2f [18352.234820] ? watchdog_timer_fn.cold.6+0x62/0x10a [18352.236875] ? __hrtimer_run_queues+0x1c2/0x4d0 [18352.244623] ? watchdog+0x40/0x40 [18352.246883] ? hrtimer_interrupt+0x128/0x2c0 [18352.248739] ? smp_apic_timer_interrupt+0xa6/0x270 [18352.251038] ? apic_timer_interrupt+0xf/0x20 [18352.255058] [18352.255928] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18352.263329] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18352.265647] ? newidle_balance.isra.93+0x207/0x580 [18352.269780] do_raw_spin_lock+0x177/0x190 [18352.271882] _raw_spin_lock+0x21/0x30 [18352.273339] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18352.276692] process_one_work+0x2c8/0x700 [18352.278853] worker_thread+0x296/0x6e0 [18352.281465] ? rescuer_thread+0x570/0x570 [18352.289007] kthread+0x1d1/0x200 [18352.290809] ? set_kthread_struct+0x70/0x70 [18352.292389] ret_from_fork+0x1f/0x30 [18354.825561] rcu: INFO: rcu_sched self-detected stall on CPU [18354.831509] rcu: 2-....: (9097850 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [18354.837456] (t=16260399 jiffies g=274185 q=68089) [18354.844650] rcu: rcu_sched kthread starved for 15718765 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [18354.856726] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [18354.865624] rcu: RCU grace-period kthread stack dump: [18354.871377] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [18354.879642] Call Trace: [18354.882303] __schedule+0x351/0xcb0 [18354.885906] schedule+0xc0/0x180 [18354.888474] schedule_timeout+0xb4/0x190 [18354.893490] ? do_raw_spin_unlock+0x75/0x190 [18354.898614] ? __next_timer_interrupt+0x160/0x160 [18354.903707] ? prepare_to_swait_event+0xf3/0x140 [18354.909186] rcu_gp_kthread+0x8dd/0x15b0 [18354.914139] ? force_qs_rnp+0x2c0/0x2c0 [18354.918380] kthread+0x1d1/0x200 [18354.920672] ? set_kthread_struct+0x70/0x70 [18354.925684] ret_from_fork+0x1f/0x30 [18354.927140] rcu: Stack dump where RCU GP kthread last ran: [18354.934003] Sending NMI from CPU 2 to CPUs 3: [18354.944330] NMI backtrace for cpu 3 [18354.944339] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18354.944341] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18354.944343] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18354.944345] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18354.944347] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 [18354.944350] RAX: 0000000000007e9c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18354.944351] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18354.944353] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18354.944354] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18354.944356] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18354.944358] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18354.944360] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18354.944361] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18354.944362] Call Trace: [18354.944363] [18354.944364] ? show_regs.cold.9+0x22/0x2f [18354.944365] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [18354.944366] ? nmi_cpu_backtrace_handler+0x15/0x40 [18354.944367] ? nmi_handle+0xbd/0x210 [18354.944368] ? default_do_nmi+0x4b/0x1b0 [18354.944369] ? do_nmi+0x280/0x380 [18354.944370] ? end_repeat_nmi+0x16/0x1f [18354.944372] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18354.944373] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18354.944374] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18354.944380] [18354.944381] do_raw_spin_lock+0x177/0x190 [18354.944382] _raw_spin_lock+0x21/0x30 [18354.944383] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18354.944384] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18354.944386] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18354.944387] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18354.944389] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18354.944390] ? woken_wake_function+0x30/0x30 [18354.944391] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18354.944392] kthread+0x1d1/0x200 [18354.944393] ? set_kthread_struct+0x70/0x70 [18354.944395] ret_from_fork+0x1f/0x30 [18354.947356] Sending NMI from CPU 2 to CPUs 1: [18355.079831] NMI backtrace for cpu 1 [18355.080174] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18355.080177] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18355.080178] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18355.080180] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18355.080182] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [18355.080184] RAX: 0000000000001079 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18355.080186] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18355.080188] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18355.080189] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18355.080190] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18355.080192] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18355.080193] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18355.080194] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18355.080196] Call Trace: [18355.080197] [18355.080197] ? show_regs.cold.9+0x22/0x2f [18355.080198] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [18355.080199] ? nmi_cpu_backtrace_handler+0x15/0x40 [18355.080200] ? nmi_handle+0xbd/0x210 [18355.080201] ? default_do_nmi+0x4b/0x1b0 [18355.080202] ? do_nmi+0x280/0x380 [18355.080204] ? end_repeat_nmi+0x16/0x1f [18355.080205] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18355.080206] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18355.080207] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18355.080208] [18355.080209] do_raw_spin_lock+0x177/0x190 [18355.080211] _raw_spin_lock+0x21/0x30 [18355.080211] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18355.080213] ? do_raw_spin_unlock+0x75/0x190 [18355.080214] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18355.080220] ? do_wait_intr+0xf0/0xf0 [18355.080222] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18355.080223] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18355.080224] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18355.080225] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18355.080227] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18355.080228] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18355.080230] mdc_intent_lock+0x353/0x7e0 [mdc] [18355.080234] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18355.080235] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18355.080237] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18355.080237] lmv_intent_open+0x338/0x1150 [lmv] [18355.080239] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18355.080240] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18355.080241] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18355.080242] ll_intent_lock+0xc5/0xaa0 [lustre] [18355.080243] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18355.080244] ll_lookup_it+0xe0e/0x2270 [lustre] [18355.080246] ? do_raw_spin_unlock+0x75/0x190 [18355.080247] ? _raw_spin_unlock+0x12/0x30 [18355.080248] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18355.080249] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18355.080250] ll_atomic_open+0x36a/0xd70 [lustre] [18355.080251] lookup_open+0x356/0xb40 [18355.080252] do_last+0x313/0xfc0 [18355.080253] ? nd_jump_root+0xe5/0x160 [18355.080254] ? path_init+0x437/0x520 [18355.080255] path_openat+0xf7/0x500 [18355.080256] ? do_raw_spin_unlock+0x75/0x190 [18355.080257] do_filp_open+0x99/0x140 [18355.080259] ? getname_flags+0x6e/0x330 [18355.080260] ? __check_object_size+0xff/0x256 [18355.080261] ? do_raw_spin_unlock+0x75/0x190 [18355.080262] ? _raw_spin_unlock+0x12/0x30 [18355.080263] do_sys_openat2+0x2b4/0x410 [18355.080264] do_sys_open+0x73/0xa0 [18355.080265] __x64_sys_openat+0x24/0x30 [18355.080266] do_syscall_64+0xc1/0x440 [18355.080268] entry_SYSCALL_64_after_hwframe+0x49/0xae [18355.080269] RIP: 0033:0x7f53425a5332 [18355.080274] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18355.080277] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18355.080280] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18355.080282] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18355.080283] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18355.080285] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18355.080287] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18355.080739] NMI backtrace for cpu 2 [18355.294744] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18355.299825] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18355.303884] Call Trace: [18355.305142] [18355.306295] ? dump_stack+0xbb/0x10e [18355.308992] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [18355.310951] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [18355.312897] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [18355.315235] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [18355.318012] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [18355.321097] ? rcu_dump_cpu_stacks+0x11b/0x171 [18355.323581] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [18355.325818] ? update_process_times+0x6d/0xc0 [18355.328249] ? tick_sched_handle.isra.18+0x27/0x90 [18355.329993] ? tick_sched_timer+0x53/0xb0 [18355.331779] ? __hrtimer_run_queues+0x1c2/0x4d0 [18355.333653] ? tick_sched_handle.isra.18+0x90/0x90 [18355.338802] ? hrtimer_interrupt+0x128/0x2c0 [18355.342551] ? smp_apic_timer_interrupt+0xa6/0x270 [18355.345554] ? apic_timer_interrupt+0xf/0x20 [18355.348886] [18355.350291] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18355.352658] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18355.360134] ? do_raw_spin_lock+0x177/0x190 [18355.361712] ? _raw_spin_lock+0x21/0x30 [18355.364366] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18355.367183] ? obd_mod_free+0x46/0x140 [ptlrpc] [18355.369634] ? mdc_commit_open+0x91/0xb0 [mdc] [18355.373154] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18355.374767] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18355.377750] ? after_reply+0xe2c/0x1780 [ptlrpc] [18355.380224] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18355.382520] ? __next_timer_interrupt+0x160/0x160 [18355.386619] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [18355.388666] ? do_wait_intr+0xf0/0xf0 [18355.390790] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18355.393467] ? kthread+0x1d1/0x200 [18355.395211] ? set_kthread_struct+0x70/0x70 [18355.397378] ? ret_from_fork+0x1f/0x30 [18355.398850] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15718s! [18355.407899] Showing busy workqueues and worker pools: [18355.412877] workqueue events: flags=0x0 [18355.417567] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18355.420808] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18355.433739] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18355.441559] pending: cpu_latency_work [ptlrpc] [18355.444384] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18355.447816] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18355.456775] workqueue events_long: flags=0x0 [18355.458333] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18355.460529] pending: lu_objects_flush [obdclass] [18355.462134] workqueue events_unbound: flags=0x2 [18355.463715] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18355.472460] pending: flush_memcg_stats_dwork, idle_cull_fn [18355.475459] workqueue events_power_efficient: flags=0x80 [18355.480477] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18355.483685] pending: check_lifetime, do_cache_clean [18355.485877] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18355.487846] pending: neigh_periodic_work, neigh_periodic_work [18355.489832] workqueue mm_percpu_wq: flags=0x8 [18355.491676] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18355.495188] pending: vmstat_update, lru_add_drain_per_cpu [18355.497978] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18355.500847] pending: vmstat_update, lru_add_drain_per_cpu [18355.504846] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18355.510024] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18355.513875] workqueue xprtiod: flags=0xa [18355.515216] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18355.517020] pending: xprt_autoclose, xs_error_handle [18355.523839] workqueue ipv6_addrconf: flags=0x40008 [18355.528256] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18355.530616] pending: addrconf_verify_work [18355.534890] workqueue knbd0-recv: flags=0x1a [18355.538014] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18355.541887] in-flight: 651:recv_work [18355.545429] workqueue ptlrpc_pinger: flags=0x2 [18355.548014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18355.553714] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18355.557187] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15718s workers=4 idle: 15979 31824 8 [18355.562610] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18346s workers=2 idle: 654 [18380.052032] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [18380.054896] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18380.055018] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18380.057159] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18380.057168] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18380.057233] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18380.057238] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18380.057241] RIP: 0010:native_safe_halt+0x2e/0x50 [18380.057254] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18380.057257] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18380.057262] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18380.057264] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18380.057266] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18380.057268] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18380.057270] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18380.057273] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18380.057340] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18380.057342] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18380.057350] Call Trace: [18380.057355] [18380.057359] ? show_regs.cold.9+0x22/0x2f [18380.057369] ? watchdog_timer_fn.cold.6+0x62/0x10a [18380.057375] ? __hrtimer_run_queues+0x1c2/0x4d0 [18380.057378] ? watchdog+0x40/0x40 [18380.057382] ? hrtimer_interrupt+0x128/0x2c0 [18380.057386] ? smp_apic_timer_interrupt+0xa6/0x270 [18380.057392] ? apic_timer_interrupt+0xf/0x20 [18380.057396] [18380.057398] ? native_safe_halt+0x2e/0x50 [18380.057402] kvm_wait+0xcd/0xe0 [18380.057408] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18380.057415] do_raw_spin_lock+0x177/0x190 [18380.057419] _raw_spin_lock+0x21/0x30 [18380.057424] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18380.057698] obd_mod_free+0x46/0x140 [ptlrpc] [18380.057972] mdc_commit_open+0x91/0xb0 [mdc] [18380.058009] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18380.058282] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [18380.058289] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18380.058355] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18380.058361] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18380.058363] RIP: 0010:native_safe_halt+0x2e/0x50 [18380.058372] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18380.058376] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [18380.058380] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18380.058382] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18380.058384] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18380.058387] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18380.058389] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18380.058392] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18380.058395] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18380.058398] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18380.058406] Call Trace: [18380.058410] [18380.058413] ? show_regs.cold.9+0x22/0x2f [18380.058421] ? watchdog_timer_fn.cold.6+0x62/0x10a [18380.058427] ? __hrtimer_run_queues+0x1c2/0x4d0 [18380.058431] ? watchdog+0x40/0x40 [18380.058201] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18380.058437] ? hrtimer_interrupt+0x128/0x2c0 [18380.058444] ? smp_apic_timer_interrupt+0xa6/0x270 [18380.058449] ? apic_timer_interrupt+0xf/0x20 [18380.058453] [18380.058457] ? native_safe_halt+0x2e/0x50 [18380.058461] kvm_wait+0xcd/0xe0 [18380.058468] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18380.058476] do_raw_spin_lock+0x177/0x190 [18380.058480] _raw_spin_lock+0x21/0x30 [18380.058484] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18380.058760] after_reply+0xe2c/0x1780 [ptlrpc] [18380.059302] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18380.059876] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18380.060390] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18380.060877] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18380.061090] ? woken_wake_function+0x30/0x30 [18380.060113] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18380.061101] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18380.061661] ? __next_timer_interrupt+0x160/0x160 [18380.061670] ptlrpcd+0xa82/0xe40 [ptlrpc] [18380.062031] ? do_wait_intr+0xf0/0xf0 [18380.062040] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18380.062263] kthread+0x1d1/0x200 [18380.062269] ? set_kthread_struct+0x70/0x70 [18380.062273] ret_from_fork+0x1f/0x30 [18380.064003] kthread+0x1d1/0x200 [18380.064013] ? set_kthread_struct+0x70/0x70 [18380.064018] ret_from_fork+0x1f/0x30 [18380.116658] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18380.119239] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18380.122881] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18380.139094] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18380.154885] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18380.157445] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18380.161235] [18380.166589] RIP: 0010:native_safe_halt+0x2e/0x50 [18380.173702] RIP: 0010:native_safe_halt+0x2e/0x50 [18380.175965] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18380.175977] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18380.175984] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18380.175986] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18380.175988] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18380.175990] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18380.175992] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18380.175996] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18380.175998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18380.176001] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18380.176009] Call Trace: [18380.176014] [18380.176019] ? show_regs.cold.9+0x22/0x2f [18380.176031] ? watchdog_timer_fn.cold.6+0x62/0x10a [18380.176038] ? __hrtimer_run_queues+0x1c2/0x4d0 [18380.176043] ? watchdog+0x40/0x40 [18380.176046] ? hrtimer_interrupt+0x128/0x2c0 [18380.176050] ? smp_apic_timer_interrupt+0xa6/0x270 [18380.176059] ? apic_timer_interrupt+0xf/0x20 [18380.176064] [18380.176067] ? native_safe_halt+0x2e/0x50 [18380.176072] kvm_wait+0xcd/0xe0 [18380.176079] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18380.176090] do_raw_spin_lock+0x177/0x190 [18380.176095] _raw_spin_lock+0x21/0x30 [18380.176100] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18380.176456] ? do_raw_spin_unlock+0x75/0x190 [18380.176463] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18380.183007] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18380.185279] ? do_wait_intr+0xf0/0xf0 [18380.185301] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18380.188120] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000212 [18380.190394] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18380.194126] ORIG_RAX: ffffffffffffff13 [18380.196195] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18380.196547] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18380.196676] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18380.196713] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18380.196745] mdc_intent_lock+0x353/0x7e0 [mdc] [18380.196773] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18380.196858] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18380.197079] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18380.197109] lmv_intent_open+0x338/0x1150 [lmv] [18380.197140] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18380.197223] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18380.197307] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18380.197480] ll_intent_lock+0xc5/0xaa0 [lustre] [18380.197570] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18380.197666] ll_lookup_it+0xe0e/0x2270 [lustre] [18380.197752] ? do_raw_spin_unlock+0x75/0x190 [18380.197761] ? _raw_spin_unlock+0x12/0x30 [18380.197768] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18380.197797] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18380.197816] ll_atomic_open+0x36a/0xd70 [lustre] [18380.197899] lookup_open+0x356/0xb40 [18380.197907] do_last+0x313/0xfc0 [18380.197911] ? nd_jump_root+0xe5/0x160 [18380.197915] ? path_init+0x437/0x520 [18380.197920] path_openat+0xf7/0x500 [18380.197924] ? do_raw_spin_unlock+0x75/0x190 [18380.197929] do_filp_open+0x99/0x140 [18380.197934] ? getname_flags+0x6e/0x330 [18380.197986] ? __check_object_size+0xff/0x256 [18380.197993] ? do_raw_spin_unlock+0x75/0x190 [18380.197996] ? _raw_spin_unlock+0x12/0x30 [18380.198001] do_sys_openat2+0x2b4/0x410 [18380.198005] do_sys_open+0x73/0xa0 [18380.198009] __x64_sys_openat+0x24/0x30 [18380.198012] do_syscall_64+0xc1/0x440 [18380.198018] entry_SYSCALL_64_after_hwframe+0x49/0xae [18380.198024] RIP: 0033:0x7f53425a5332 [18380.198031] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18380.198035] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18380.198039] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18380.198042] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18380.198044] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18380.198046] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18380.198049] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18381.014977] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18381.020817] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [18381.027203] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18381.035431] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18381.044716] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18381.047097] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18381.064871] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18381.069108] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18381.075958] Call Trace: [18381.079086] [18381.080526] ? show_regs.cold.9+0x22/0x2f [18381.088785] ? watchdog_timer_fn.cold.6+0x62/0x10a [18381.091682] ? __hrtimer_run_queues+0x1c2/0x4d0 [18381.105149] ? watchdog+0x40/0x40 [18381.110840] ? hrtimer_interrupt+0x128/0x2c0 [18381.112025] ? smp_apic_timer_interrupt+0xa6/0x270 [18381.113789] ? apic_timer_interrupt+0xf/0x20 [18381.115093] [18381.115699] ? native_safe_halt+0x2e/0x50 [18381.117833] kvm_wait+0xcd/0xe0 [18381.119169] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18381.122057] ? newidle_balance.isra.93+0x207/0x580 [18381.124072] do_raw_spin_lock+0x177/0x190 [18381.130962] _raw_spin_lock+0x21/0x30 [18381.132936] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18381.137962] process_one_work+0x2c8/0x700 [18381.141609] worker_thread+0x296/0x6e0 [18381.142936] ? rescuer_thread+0x570/0x570 [18381.147523] kthread+0x1d1/0x200 [18381.151017] ? set_kthread_struct+0x70/0x70 [18381.154776] ret_from_fork+0x1f/0x30 [18385.898103] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15748s! [18385.906526] Showing busy workqueues and worker pools: [18385.912953] workqueue events: flags=0x0 [18385.923207] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18385.941320] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18385.952158] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18385.959131] pending: cpu_latency_work [ptlrpc] [18385.970022] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18385.978984] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18385.996759] workqueue events_long: flags=0x0 [18385.999846] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18386.005050] pending: lu_objects_flush [obdclass] [18386.008996] workqueue events_unbound: flags=0x2 [18386.012228] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18386.018121] pending: flush_memcg_stats_dwork, idle_cull_fn [18386.023196] workqueue events_power_efficient: flags=0x80 [18386.028662] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18386.034260] pending: check_lifetime, do_cache_clean [18386.038611] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18386.044314] pending: neigh_periodic_work, neigh_periodic_work [18386.052721] workqueue mm_percpu_wq: flags=0x8 [18386.057106] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18386.062717] pending: vmstat_update, lru_add_drain_per_cpu [18386.066901] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18386.071117] pending: vmstat_update, lru_add_drain_per_cpu [18386.075724] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18386.082888] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18386.094772] workqueue xprtiod: flags=0xa [18386.102474] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18386.110586] pending: xprt_autoclose, xs_error_handle [18386.114336] workqueue ipv6_addrconf: flags=0x40008 [18386.117810] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18386.122029] pending: addrconf_verify_work [18386.126017] workqueue knbd0-recv: flags=0x1a [18386.131020] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18386.142774] in-flight: 651:recv_work [18386.150957] workqueue ptlrpc_pinger: flags=0x2 [18386.157138] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18386.166632] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18386.173737] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15749s workers=4 idle: 15979 31824 8 [18386.185738] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18376s workers=2 idle: 654 [18408.057168] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18408.058927] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [18408.059100] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18408.059106] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18408.059181] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18408.059186] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18408.059188] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18408.059202] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18408.059206] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18408.059211] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18408.059213] RDX: 00000000000034af RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18408.059216] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18408.059218] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18408.059220] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18408.059223] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18408.059226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18408.059228] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18408.059236] Call Trace: [18408.059241] [18408.059246] ? show_regs.cold.9+0x22/0x2f [18408.059253] ? watchdog_timer_fn.cold.6+0x62/0x10a [18408.059259] ? __hrtimer_run_queues+0x1c2/0x4d0 [18408.059263] ? watchdog+0x40/0x40 [18408.059266] ? hrtimer_interrupt+0x128/0x2c0 [18408.059270] ? smp_apic_timer_interrupt+0xa6/0x270 [18408.059277] ? apic_timer_interrupt+0xf/0x20 [18408.059282] [18408.059284] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18408.059288] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18408.059292] do_raw_spin_lock+0x177/0x190 [18408.059296] _raw_spin_lock+0x21/0x30 [18408.059302] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18408.059565] obd_mod_free+0x46/0x140 [ptlrpc] [18408.060150] mdc_commit_open+0x91/0xb0 [mdc] [18408.060188] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18408.060381] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18408.060573] after_reply+0xe2c/0x1780 [ptlrpc] [18408.063668] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18408.063744] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18408.063752] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18408.063755] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18408.063768] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18408.063772] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18408.063776] RAX: 0000000000007e29 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18408.063778] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18408.063780] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18408.063782] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18408.063785] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18408.063788] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18408.063791] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18408.063793] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18408.063802] Call Trace: [18408.063807] [18408.063812] ? show_regs.cold.9+0x22/0x2f [18408.060762] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18408.064223] ? __next_timer_interrupt+0x160/0x160 [18408.064231] ptlrpcd+0xa82/0xe40 [ptlrpc] [18408.064435] ? do_wait_intr+0xf0/0xf0 [18408.064443] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18408.064639] kthread+0x1d1/0x200 [18408.064645] ? set_kthread_struct+0x70/0x70 [18408.064648] ret_from_fork+0x1f/0x30 [18408.064944] ? watchdog_timer_fn.cold.6+0x62/0x10a [18408.064954] ? __hrtimer_run_queues+0x1c2/0x4d0 [18408.064958] ? watchdog+0x40/0x40 [18408.064962] ? hrtimer_interrupt+0x128/0x2c0 [18408.064967] ? smp_apic_timer_interrupt+0xa6/0x270 [18408.064973] ? apic_timer_interrupt+0xf/0x20 [18408.064977] [18408.064981] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18408.064988] do_raw_spin_lock+0x177/0x190 [18408.071325] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18408.071402] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18408.071408] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18408.071411] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18408.071424] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18408.071428] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18408.071432] RAX: 0000000000007e84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18408.071435] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18408.071437] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18408.071439] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18408.071442] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18408.071445] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18408.071448] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18408.071450] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18408.071458] Call Trace: [18408.071462] [18408.071467] ? show_regs.cold.9+0x22/0x2f [18408.071475] ? watchdog_timer_fn.cold.6+0x62/0x10a [18408.071480] ? __hrtimer_run_queues+0x1c2/0x4d0 [18408.071484] ? watchdog+0x40/0x40 [18408.071488] ? hrtimer_interrupt+0x128/0x2c0 [18408.071492] ? smp_apic_timer_interrupt+0xa6/0x270 [18408.071500] ? apic_timer_interrupt+0xf/0x20 [18408.071587] [18408.071590] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18408.071595] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18408.071602] do_raw_spin_lock+0x177/0x190 [18408.071607] _raw_spin_lock+0x21/0x30 [18408.071612] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18408.072186] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18408.072401] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18408.072610] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18408.072816] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18408.074666] ? woken_wake_function+0x30/0x30 [18408.074676] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18408.079075] _raw_spin_lock+0x21/0x30 [18408.079261] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18408.121469] kthread+0x1d1/0x200 [18408.121490] ? set_kthread_struct+0x70/0x70 [18408.121494] ret_from_fork+0x1f/0x30 [18408.742625] ? do_raw_spin_unlock+0x75/0x190 [18408.756024] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18408.768654] ? do_wait_intr+0xf0/0xf0 [18408.770935] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18408.774235] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18408.777373] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18408.782018] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18408.785197] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18408.788603] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18408.793962] mdc_intent_lock+0x353/0x7e0 [mdc] [18408.799920] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18408.806457] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18408.812240] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18408.817927] lmv_intent_open+0x338/0x1150 [lmv] [18408.822191] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18408.826681] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18408.830914] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18408.835226] ll_intent_lock+0xc5/0xaa0 [lustre] [18408.837627] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18408.842269] ll_lookup_it+0xe0e/0x2270 [lustre] [18408.845071] ? do_raw_spin_unlock+0x75/0x190 [18408.847310] ? _raw_spin_unlock+0x12/0x30 [18408.849381] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18408.851783] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18408.854564] ll_atomic_open+0x36a/0xd70 [lustre] [18408.857827] lookup_open+0x356/0xb40 [18408.859641] do_last+0x313/0xfc0 [18408.861855] ? nd_jump_root+0xe5/0x160 [18408.863309] ? path_init+0x437/0x520 [18408.864660] path_openat+0xf7/0x500 [18408.866193] ? do_raw_spin_unlock+0x75/0x190 [18408.867916] do_filp_open+0x99/0x140 [18408.869315] ? getname_flags+0x6e/0x330 [18408.870800] ? __check_object_size+0xff/0x256 [18408.872715] ? do_raw_spin_unlock+0x75/0x190 [18408.874315] ? _raw_spin_unlock+0x12/0x30 [18408.876383] do_sys_openat2+0x2b4/0x410 [18408.877827] do_sys_open+0x73/0xa0 [18408.879754] __x64_sys_openat+0x24/0x30 [18408.881500] do_syscall_64+0xc1/0x440 [18408.883050] entry_SYSCALL_64_after_hwframe+0x49/0xae [18408.884923] RIP: 0033:0x7f53425a5332 [18408.886500] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18408.894915] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18408.897868] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18408.901392] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18408.904451] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18408.908261] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18408.911587] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18412.052076] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [18412.055292] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18412.081776] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18412.087791] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18412.097970] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18412.104699] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18412.111510] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18412.131016] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18412.133970] RAX: 00000000000071e5 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18412.140535] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [18412.146657] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18412.150317] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18412.167460] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18412.195247] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18412.202134] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18412.222102] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18412.232680] Call Trace: [18412.243974] [18412.250880] ? show_regs.cold.9+0x22/0x2f [18412.260050] ? watchdog_timer_fn.cold.6+0x62/0x10a [18412.275458] ? __hrtimer_run_queues+0x1c2/0x4d0 [18412.290014] ? watchdog+0x40/0x40 [18412.296419] ? hrtimer_interrupt+0x128/0x2c0 [18412.308776] ? smp_apic_timer_interrupt+0xa6/0x270 [18412.319695] ? apic_timer_interrupt+0xf/0x20 [18412.329387] [18412.329897] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18412.331216] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18412.344051] ? newidle_balance.isra.93+0x207/0x580 [18412.347896] do_raw_spin_lock+0x177/0x190 [18412.361376] _raw_spin_lock+0x21/0x30 [18412.362258] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18412.364737] process_one_work+0x2c8/0x700 [18412.368581] worker_thread+0x296/0x6e0 [18412.372825] ? rescuer_thread+0x570/0x570 [18412.379948] kthread+0x1d1/0x200 [18412.382494] ? set_kthread_struct+0x70/0x70 [18412.389709] ret_from_fork+0x1f/0x30 [18416.617447] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15779s! [18416.625453] Showing busy workqueues and worker pools: [18416.628761] workqueue events: flags=0x0 [18416.630196] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18416.639657] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18416.645423] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18416.649216] pending: cpu_latency_work [ptlrpc] [18416.651532] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18416.654727] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18416.659271] workqueue events_long: flags=0x0 [18416.661165] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18416.663957] pending: lu_objects_flush [obdclass] [18416.666044] workqueue events_unbound: flags=0x2 [18416.667772] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18416.670476] pending: flush_memcg_stats_dwork, idle_cull_fn [18416.672777] workqueue events_power_efficient: flags=0x80 [18416.674936] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18416.678006] pending: check_lifetime, do_cache_clean [18416.680249] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18416.682946] pending: neigh_periodic_work, neigh_periodic_work [18416.688501] workqueue mm_percpu_wq: flags=0x8 [18416.690817] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18416.701603] pending: vmstat_update, lru_add_drain_per_cpu [18416.705593] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18416.710771] pending: vmstat_update, lru_add_drain_per_cpu [18416.715647] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18416.719607] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18416.722515] workqueue xprtiod: flags=0xa [18416.724408] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18416.727069] pending: xprt_autoclose, xs_error_handle [18416.731838] workqueue ipv6_addrconf: flags=0x40008 [18416.733822] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18416.737650] pending: addrconf_verify_work [18416.744300] workqueue knbd0-recv: flags=0x1a [18416.749801] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18416.754066] in-flight: 651:recv_work [18416.758506] workqueue ptlrpc_pinger: flags=0x2 [18416.764501] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18416.768800] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18416.776921] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15779s workers=4 idle: 15979 31824 8 [18416.784796] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18407s workers=2 idle: 654 [18436.060037] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18436.065514] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [18436.066918] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18436.066992] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18436.066999] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18436.067003] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18436.067195] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18436.067200] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18436.067205] RAX: 0000000000005b10 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18436.067207] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18436.067210] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18436.067212] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18436.067215] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18436.067218] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18436.067221] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18436.067224] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18436.067232] Call Trace: [18436.067357] [18436.067364] ? show_regs.cold.9+0x22/0x2f [18436.067373] ? watchdog_timer_fn.cold.6+0x62/0x10a [18436.067379] ? __hrtimer_run_queues+0x1c2/0x4d0 [18436.067383] ? watchdog+0x40/0x40 [18436.067388] ? hrtimer_interrupt+0x128/0x2c0 [18436.067392] ? smp_apic_timer_interrupt+0xa6/0x270 [18436.067401] ? apic_timer_interrupt+0xf/0x20 [18436.067406] [18436.067408] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18436.067416] do_raw_spin_lock+0x177/0x190 [18436.067420] _raw_spin_lock+0x21/0x30 [18436.067536] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18436.067542] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul [18436.067425] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18436.068032] ? do_raw_spin_unlock+0x75/0x190 [18436.067586] crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18436.068066] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18436.068071] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18436.068073] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18436.068081] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18436.068084] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18436.068088] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18436.068090] RDX: 0000000000004ae9 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18436.068093] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18436.068095] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18436.068098] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18436.068100] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18436.068103] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18436.068105] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18436.068112] Call Trace: [18436.068115] [18436.068118] ? show_regs.cold.9+0x22/0x2f [18436.068125] ? watchdog_timer_fn.cold.6+0x62/0x10a [18436.068131] ? __hrtimer_run_queues+0x1c2/0x4d0 [18436.068134] ? watchdog+0x40/0x40 [18436.068139] ? hrtimer_interrupt+0x128/0x2c0 [18436.068039] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18436.068389] ? do_wait_intr+0xf0/0xf0 [18436.068143] ? smp_apic_timer_interrupt+0xa6/0x270 [18436.068477] ? apic_timer_interrupt+0xf/0x20 [18436.068482] [18436.068485] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18436.068491] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18436.068552] do_raw_spin_lock+0x177/0x190 [18436.068404] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18436.068559] _raw_spin_lock+0x21/0x30 [18436.068567] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18436.069006] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18436.069534] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18436.069473] obd_mod_free+0x46/0x140 [ptlrpc] [18436.070313] mdc_commit_open+0x91/0xb0 [mdc] [18436.070349] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18436.070602] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18436.070809] after_reply+0xe2c/0x1780 [ptlrpc] [18436.071290] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18436.071564] ? __next_timer_interrupt+0x160/0x160 [18436.071289] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18436.072002] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18436.072037] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18436.072067] mdc_intent_lock+0x353/0x7e0 [mdc] [18436.072181] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18436.071574] ptlrpcd+0xa82/0xe40 [ptlrpc] [18436.072277] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18436.072714] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18436.072951] ? do_wait_intr+0xf0/0xf0 [18436.073022] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18436.073690] kthread+0x1d1/0x200 [18436.073697] ? set_kthread_struct+0x70/0x70 [18436.073701] ret_from_fork+0x1f/0x30 [18436.073708] lmv_intent_open+0x338/0x1150 [lmv] [18436.073742] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18436.093090] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18436.093276] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18436.093284] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18436.093286] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18436.093300] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18436.093305] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [18436.093309] RAX: 0000000000006812 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18436.093312] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18436.093314] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18436.093316] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18436.093318] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18436.093321] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18436.093324] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18436.093327] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18436.093335] Call Trace: [18436.093340] [18436.093406] ? show_regs.cold.9+0x22/0x2f [18436.093417] ? watchdog_timer_fn.cold.6+0x62/0x10a [18436.093423] ? __hrtimer_run_queues+0x1c2/0x4d0 [18436.093427] ? watchdog+0x40/0x40 [18436.093431] ? hrtimer_interrupt+0x128/0x2c0 [18436.093435] ? smp_apic_timer_interrupt+0xa6/0x270 [18436.093681] ? apic_timer_interrupt+0xf/0x20 [18436.093689] [18436.093692] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18436.093698] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18436.093704] do_raw_spin_lock+0x177/0x190 [18436.093709] _raw_spin_lock+0x21/0x30 [18436.093714] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18436.094679] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18436.094937] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18436.095190] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18436.095765] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18436.096005] ? woken_wake_function+0x30/0x30 [18436.096014] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18436.096561] kthread+0x1d1/0x200 [18436.096567] ? set_kthread_struct+0x70/0x70 [18436.096571] ret_from_fork+0x1f/0x30 [18436.575538] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18436.577345] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18436.578531] ll_intent_lock+0xc5/0xaa0 [lustre] [18436.581271] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18436.582875] ll_lookup_it+0xe0e/0x2270 [lustre] [18436.584266] ? do_raw_spin_unlock+0x75/0x190 [18436.585405] ? _raw_spin_unlock+0x12/0x30 [18436.586466] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18436.588325] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18436.589899] ll_atomic_open+0x36a/0xd70 [lustre] [18436.591289] lookup_open+0x356/0xb40 [18436.592386] do_last+0x313/0xfc0 [18436.593201] ? nd_jump_root+0xe5/0x160 [18436.595068] ? path_init+0x437/0x520 [18436.596017] path_openat+0xf7/0x500 [18436.596933] ? do_raw_spin_unlock+0x75/0x190 [18436.598262] do_filp_open+0x99/0x140 [18436.599324] ? getname_flags+0x6e/0x330 [18436.600356] ? __check_object_size+0xff/0x256 [18436.601458] ? do_raw_spin_unlock+0x75/0x190 [18436.602604] ? _raw_spin_unlock+0x12/0x30 [18436.603814] do_sys_openat2+0x2b4/0x410 [18436.604847] do_sys_open+0x73/0xa0 [18436.605622] __x64_sys_openat+0x24/0x30 [18436.606832] do_syscall_64+0xc1/0x440 [18436.607836] entry_SYSCALL_64_after_hwframe+0x49/0xae [18436.609335] RIP: 0033:0x7f53425a5332 [18436.610337] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18436.615537] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18436.618347] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18436.621562] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18436.624066] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18436.626641] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18436.629396] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18444.052253] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [18444.067151] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18444.152652] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18444.178410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18444.190690] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18444.198689] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18444.206154] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18444.219138] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18444.226141] RAX: 0000000000006b6a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18444.242831] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [18444.252099] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18444.263432] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18444.275929] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18444.283175] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18444.289094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18444.293465] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18444.308051] Call Trace: [18444.309260] [18444.310163] ? show_regs.cold.9+0x22/0x2f [18444.318067] ? watchdog_timer_fn.cold.6+0x62/0x10a [18444.326193] ? __hrtimer_run_queues+0x1c2/0x4d0 [18444.330951] ? watchdog+0x40/0x40 [18444.335610] ? hrtimer_interrupt+0x128/0x2c0 [18444.340138] ? smp_apic_timer_interrupt+0xa6/0x270 [18444.345214] ? apic_timer_interrupt+0xf/0x20 [18444.350161] [18444.351288] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18444.366184] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18444.368245] ? newidle_balance.isra.93+0x207/0x580 [18444.381078] do_raw_spin_lock+0x177/0x190 [18444.393625] _raw_spin_lock+0x21/0x30 [18444.400027] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18444.406939] process_one_work+0x2c8/0x700 [18444.410078] worker_thread+0x296/0x6e0 [18444.412567] ? rescuer_thread+0x570/0x570 [18444.415492] kthread+0x1d1/0x200 [18444.417512] ? set_kthread_struct+0x70/0x70 [18444.426652] ret_from_fork+0x1f/0x30 [18446.816110] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15809s! [18446.818451] Showing busy workqueues and worker pools: [18446.820007] workqueue events: flags=0x0 [18446.821815] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18446.825403] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18446.830180] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18446.832717] pending: cpu_latency_work [ptlrpc] [18446.834806] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18446.837688] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18446.841318] workqueue events_long: flags=0x0 [18446.843297] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18446.846640] pending: lu_objects_flush [obdclass] [18446.848804] workqueue events_unbound: flags=0x2 [18446.850609] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18446.853286] pending: flush_memcg_stats_dwork, idle_cull_fn [18446.855396] workqueue events_power_efficient: flags=0x80 [18446.858013] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18446.860641] pending: check_lifetime, do_cache_clean [18446.863165] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18446.866213] pending: neigh_periodic_work, neigh_periodic_work [18446.868671] workqueue mm_percpu_wq: flags=0x8 [18446.870772] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18446.874111] pending: vmstat_update, lru_add_drain_per_cpu [18446.876758] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18446.879978] pending: vmstat_update, lru_add_drain_per_cpu [18446.882371] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18446.885476] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18446.888364] workqueue xprtiod: flags=0xa [18446.892708] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18446.896330] pending: xprt_autoclose, xs_error_handle [18446.898308] workqueue ipv6_addrconf: flags=0x40008 [18446.900304] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18446.903002] pending: addrconf_verify_work [18446.904968] workqueue knbd0-recv: flags=0x1a [18446.906514] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18446.909087] in-flight: 651:recv_work [18446.910423] workqueue ptlrpc_pinger: flags=0x2 [18446.912042] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18446.914001] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18446.917053] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15809s workers=4 idle: 15979 31824 8 [18446.919721] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18437s workers=2 idle: 654 [18464.055027] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18464.058032] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [18464.059985] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18464.059997] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18464.060211] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18464.060217] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18464.060220] RIP: 0010:native_safe_halt+0x2e/0x50 [18464.060249] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18464.060253] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18464.060258] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18464.060261] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18464.060263] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18464.060271] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18464.060273] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18464.060277] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18464.060280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18464.060282] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18464.060291] Call Trace: [18464.060306] [18464.060312] ? show_regs.cold.9+0x22/0x2f [18464.060323] ? watchdog_timer_fn.cold.6+0x62/0x10a [18464.060331] ? __hrtimer_run_queues+0x1c2/0x4d0 [18464.060335] ? watchdog+0x40/0x40 [18464.060339] ? hrtimer_interrupt+0x128/0x2c0 [18464.060344] ? smp_apic_timer_interrupt+0xa6/0x270 [18464.060350] ? apic_timer_interrupt+0xf/0x20 [18464.060355] [18464.060358] ? native_safe_halt+0x2e/0x50 [18464.060362] kvm_wait+0xcd/0xe0 [18464.060371] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18464.060384] do_raw_spin_lock+0x177/0x190 [18464.060389] _raw_spin_lock+0x21/0x30 [18464.060394] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18464.060775] obd_mod_free+0x46/0x140 [ptlrpc] [18464.061150] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18464.061210] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18464.061216] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18464.061256] mdc_commit_open+0x91/0xb0 [mdc] [18464.061218] RIP: 0010:native_safe_halt+0x2e/0x50 [18464.061307] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18464.061433] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18464.061439] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18464.061443] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18464.061446] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18464.061448] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18464.061455] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18464.061457] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18464.061461] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18464.061464] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18464.061466] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18464.061473] Call Trace: [18464.061476] [18464.061479] ? show_regs.cold.9+0x22/0x2f [18464.061488] ? watchdog_timer_fn.cold.6+0x62/0x10a [18464.061494] ? __hrtimer_run_queues+0x1c2/0x4d0 [18464.061498] ? watchdog+0x40/0x40 [18464.061502] ? hrtimer_interrupt+0x128/0x2c0 [18464.061518] ? smp_apic_timer_interrupt+0xa6/0x270 [18464.061526] ? apic_timer_interrupt+0xf/0x20 [18464.061530] [18464.061533] ? native_safe_halt+0x2e/0x50 [18464.061538] kvm_wait+0xcd/0xe0 [18464.061544] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18464.061552] do_raw_spin_lock+0x177/0x190 [18464.061556] _raw_spin_lock+0x21/0x30 [18464.061560] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18464.061934] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18464.062420] ? do_raw_spin_unlock+0x75/0x190 [18464.062427] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18464.062974] ? do_wait_intr+0xf0/0xf0 [18464.062984] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18464.063339] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18464.063630] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18464.064055] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18464.064267] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18464.064310] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18464.064340] mdc_intent_lock+0x353/0x7e0 [mdc] [18464.062824] after_reply+0xe2c/0x1780 [ptlrpc] [18464.064861] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18464.064369] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18464.065275] ? __next_timer_interrupt+0x160/0x160 [18464.065287] ptlrpcd+0xa82/0xe40 [ptlrpc] [18464.065554] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18464.066048] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18464.066092] lmv_intent_open+0x338/0x1150 [lmv] [18464.066467] ? do_wait_intr+0xf0/0xf0 [18464.066480] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18464.066716] kthread+0x1d1/0x200 [18464.066722] ? set_kthread_struct+0x70/0x70 [18464.066726] ret_from_fork+0x1f/0x30 [18464.066147] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18464.067195] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18464.067294] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18464.067327] ll_intent_lock+0xc5/0xaa0 [lustre] [18464.067451] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18464.067569] ll_lookup_it+0xe0e/0x2270 [lustre] [18464.067649] ? do_raw_spin_unlock+0x75/0x190 [18464.067655] ? _raw_spin_unlock+0x12/0x30 [18464.067662] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18464.067688] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18464.067718] ll_atomic_open+0x36a/0xd70 [lustre] [18464.067942] lookup_open+0x356/0xb40 [18464.067952] do_last+0x313/0xfc0 [18464.067958] ? nd_jump_root+0xe5/0x160 [18464.067962] ? path_init+0x437/0x520 [18464.067967] path_openat+0xf7/0x500 [18464.067972] ? do_raw_spin_unlock+0x75/0x190 [18464.067976] do_filp_open+0x99/0x140 [18464.067982] ? getname_flags+0x6e/0x330 [18464.067986] ? __check_object_size+0xff/0x256 [18464.067994] ? do_raw_spin_unlock+0x75/0x190 [18464.067999] ? _raw_spin_unlock+0x12/0x30 [18464.068006] do_sys_openat2+0x2b4/0x410 [18464.068010] do_sys_open+0x73/0xa0 [18464.068015] __x64_sys_openat+0x24/0x30 [18464.068019] do_syscall_64+0xc1/0x440 [18464.068025] entry_SYSCALL_64_after_hwframe+0x49/0xae [18464.068032] RIP: 0033:0x7f53425a5332 [18464.068080] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18464.068085] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18464.068091] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18464.068093] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18464.068095] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18464.068303] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18464.068305] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18464.671079] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18464.702735] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18464.710359] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18464.716564] RIP: 0010:native_safe_halt+0x2e/0x50 [18464.719410] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18464.732408] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18464.736531] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18464.741661] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18464.746747] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18464.752825] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18464.757656] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18464.762730] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18464.769070] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18464.772928] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18464.777071] Call Trace: [18464.778771] [18464.779605] ? show_regs.cold.9+0x22/0x2f [18464.782460] ? watchdog_timer_fn.cold.6+0x62/0x10a [18464.785574] ? __hrtimer_run_queues+0x1c2/0x4d0 [18464.788717] ? watchdog+0x40/0x40 [18464.790751] ? hrtimer_interrupt+0x128/0x2c0 [18464.793633] ? smp_apic_timer_interrupt+0xa6/0x270 [18464.798593] ? apic_timer_interrupt+0xf/0x20 [18464.802753] [18464.804768] ? native_safe_halt+0x2e/0x50 [18464.809970] kvm_wait+0xcd/0xe0 [18464.819701] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18464.828259] do_raw_spin_lock+0x177/0x190 [18464.830755] _raw_spin_lock+0x21/0x30 [18464.839229] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18464.845798] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18464.859298] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18464.873076] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18464.885573] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18464.894747] ? woken_wake_function+0x30/0x30 [18464.899387] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18464.909330] kthread+0x1d1/0x200 [18464.913463] ? set_kthread_struct+0x70/0x70 [18464.918316] ret_from_fork+0x1f/0x30 [18472.052309] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [18472.058275] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18472.122611] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18472.137825] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18472.158739] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18472.166967] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18472.177939] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18472.191679] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18472.195743] RAX: 0000000000007a78 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18472.201191] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [18472.207231] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18472.211262] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18472.214260] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18472.217670] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18472.222194] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18472.224590] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18472.227597] Call Trace: [18472.228430] [18472.229233] ? show_regs.cold.9+0x22/0x2f [18472.231171] ? watchdog_timer_fn.cold.6+0x62/0x10a [18472.233375] ? __hrtimer_run_queues+0x1c2/0x4d0 [18472.236417] ? watchdog+0x40/0x40 [18472.237564] ? hrtimer_interrupt+0x128/0x2c0 [18472.238675] ? smp_apic_timer_interrupt+0xa6/0x270 [18472.249406] ? apic_timer_interrupt+0xf/0x20 [18472.252170] [18472.253167] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18472.258217] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18472.263701] ? newidle_balance.isra.93+0x207/0x580 [18472.267910] do_raw_spin_lock+0x177/0x190 [18472.269356] _raw_spin_lock+0x21/0x30 [18472.273740] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18472.279134] process_one_work+0x2c8/0x700 [18472.281839] worker_thread+0x296/0x6e0 [18472.283788] ? rescuer_thread+0x570/0x570 [18472.286558] kthread+0x1d1/0x200 [18472.288940] ? set_kthread_struct+0x70/0x70 [18472.293665] ret_from_fork+0x1f/0x30 [18477.023133] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15839s! [18477.029111] Showing busy workqueues and worker pools: [18477.033392] workqueue events: flags=0x0 [18477.036756] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18477.041431] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18477.058071] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18477.070751] pending: cpu_latency_work [ptlrpc] [18477.076310] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18477.083586] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18477.090945] workqueue events_long: flags=0x0 [18477.093648] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18477.098145] pending: lu_objects_flush [obdclass] [18477.102119] workqueue events_unbound: flags=0x2 [18477.109492] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18477.119104] pending: flush_memcg_stats_dwork, idle_cull_fn [18477.130105] workqueue events_power_efficient: flags=0x80 [18477.135971] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18477.145833] pending: check_lifetime, do_cache_clean [18477.149480] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18477.157616] pending: neigh_periodic_work, neigh_periodic_work [18477.161552] workqueue mm_percpu_wq: flags=0x8 [18477.165321] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18477.172585] pending: vmstat_update, lru_add_drain_per_cpu [18477.182722] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18477.192553] pending: vmstat_update, lru_add_drain_per_cpu [18477.199670] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18477.208067] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18477.216590] workqueue xprtiod: flags=0xa [18477.221509] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18477.225726] pending: xprt_autoclose, xs_error_handle [18477.230031] workqueue ipv6_addrconf: flags=0x40008 [18477.232731] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18477.239559] pending: addrconf_verify_work [18477.243449] workqueue knbd0-recv: flags=0x1a [18477.246818] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18477.258490] in-flight: 651:recv_work [18477.264330] workqueue ptlrpc_pinger: flags=0x2 [18477.268677] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18477.275627] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18477.286324] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15840s workers=4 idle: 15979 31824 8 [18477.296787] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18467s workers=2 idle: 654 [18492.055022] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [18492.057084] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [18492.057193] Modules linked in: [18492.058025] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [18492.058033] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18492.058108] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18492.058114] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18492.058117] RIP: 0010:native_safe_halt+0x2e/0x50 [18492.058130] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18492.058134] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18492.058137] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18492.058140] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18492.058142] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18492.058144] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18492.058146] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18492.058149] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18492.058152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18492.058154] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18492.058163] Call Trace: [18492.058169] [18492.058175] ? show_regs.cold.9+0x22/0x2f [18492.058186] ? watchdog_timer_fn.cold.6+0x62/0x10a [18492.058193] ? __hrtimer_run_queues+0x1c2/0x4d0 [18492.058197] ? watchdog+0x40/0x40 [18492.058201] ? hrtimer_interrupt+0x128/0x2c0 [18492.058206] ? smp_apic_timer_interrupt+0xa6/0x270 [18492.058213] ? apic_timer_interrupt+0xf/0x20 [18492.058217] [18492.058220] ? native_safe_halt+0x2e/0x50 [18492.058223] kvm_wait+0xcd/0xe0 [18492.058231] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18492.058240] do_raw_spin_lock+0x177/0x190 [18492.058244] _raw_spin_lock+0x21/0x30 [18492.058248] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18492.058758] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18492.058972] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18492.059188] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18492.059414] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18492.059626] ? woken_wake_function+0x30/0x30 [18492.059635] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18492.059854] kthread+0x1d1/0x200 [18492.059860] ? set_kthread_struct+0x70/0x70 [18492.059863] ret_from_fork+0x1f/0x30 [18492.060521] Modules linked in: [18492.061591] lustre(O) [18492.064434] lustre(O) [18492.085383] osp(O) [18492.090354] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18492.090427] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18492.090434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18492.090437] RIP: 0010:native_safe_halt+0x2e/0x50 [18492.090519] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18492.090524] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18492.090529] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18492.090531] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18492.090534] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18492.090536] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18492.090539] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18492.090542] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18492.090545] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18492.090547] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18492.090556] Call Trace: [18492.090562] [18492.090569] ? show_regs.cold.9+0x22/0x2f [18492.090578] ? watchdog_timer_fn.cold.6+0x62/0x10a [18492.090585] ? __hrtimer_run_queues+0x1c2/0x4d0 [18492.090590] ? watchdog+0x40/0x40 [18492.090594] ? hrtimer_interrupt+0x128/0x2c0 [18492.090599] ? smp_apic_timer_interrupt+0xa6/0x270 [18492.090605] ? apic_timer_interrupt+0xf/0x20 [18492.090610] [18492.090612] ? native_safe_halt+0x2e/0x50 [18492.090617] kvm_wait+0xcd/0xe0 [18492.090624] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18492.090632] do_raw_spin_lock+0x177/0x190 [18492.090698] _raw_spin_lock+0x21/0x30 [18492.090706] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18492.091266] obd_mod_free+0x46/0x140 [ptlrpc] [18492.091532] mdc_commit_open+0x91/0xb0 [mdc] [18492.091570] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18492.091839] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18492.092109] after_reply+0xe2c/0x1780 [ptlrpc] [18492.092311] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18492.092580] ? __next_timer_interrupt+0x160/0x160 [18492.092590] ptlrpcd+0xa82/0xe40 [ptlrpc] [18492.092871] ? do_wait_intr+0xf0/0xf0 [18492.092880] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18492.093167] kthread+0x1d1/0x200 [18492.093173] ? set_kthread_struct+0x70/0x70 [18492.093177] ret_from_fork+0x1f/0x30 [18492.375233] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18492.390278] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18492.394070] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18492.397342] RIP: 0010:native_safe_halt+0x2e/0x50 [18492.399038] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18492.405418] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18492.408812] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18492.411118] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18492.413505] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18492.416601] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18492.419052] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18492.420960] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18492.426052] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18492.427926] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18492.433000] Call Trace: [18492.434254] [18492.435189] ? show_regs.cold.9+0x22/0x2f [18492.436502] ? watchdog_timer_fn.cold.6+0x62/0x10a [18492.438010] ? __hrtimer_run_queues+0x1c2/0x4d0 [18492.439499] ? watchdog+0x40/0x40 [18492.440553] ? hrtimer_interrupt+0x128/0x2c0 [18492.441936] ? smp_apic_timer_interrupt+0xa6/0x270 [18492.443475] ? apic_timer_interrupt+0xf/0x20 [18492.445188] [18492.445901] ? native_safe_halt+0x2e/0x50 [18492.447212] kvm_wait+0xcd/0xe0 [18492.448416] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18492.451060] do_raw_spin_lock+0x177/0x190 [18492.455768] _raw_spin_lock+0x21/0x30 [18492.457074] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18492.460567] ? do_raw_spin_unlock+0x75/0x190 [18492.462052] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18492.467742] ? do_wait_intr+0xf0/0xf0 [18492.469332] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18492.474800] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18492.477068] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18492.478920] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18492.484067] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18492.486296] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18492.488103] mdc_intent_lock+0x353/0x7e0 [mdc] [18492.489961] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18492.491924] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18492.494325] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18492.501140] lmv_intent_open+0x338/0x1150 [lmv] [18492.507286] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18492.514385] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18492.516687] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18492.518294] ll_intent_lock+0xc5/0xaa0 [lustre] [18492.523849] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18492.531521] ll_lookup_it+0xe0e/0x2270 [lustre] [18492.533078] ? do_raw_spin_unlock+0x75/0x190 [18492.536006] ? _raw_spin_unlock+0x12/0x30 [18492.547878] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18492.552797] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18492.556618] ll_atomic_open+0x36a/0xd70 [lustre] [18492.558879] lookup_open+0x356/0xb40 [18492.560568] do_last+0x313/0xfc0 [18492.562134] ? nd_jump_root+0xe5/0x160 [18492.567166] ? path_init+0x437/0x520 [18492.571004] path_openat+0xf7/0x500 [18492.576173] ? do_raw_spin_unlock+0x75/0x190 [18492.580762] do_filp_open+0x99/0x140 [18492.585082] ? getname_flags+0x6e/0x330 [18492.587749] ? __check_object_size+0xff/0x256 [18492.590794] ? do_raw_spin_unlock+0x75/0x190 [18492.593146] ? _raw_spin_unlock+0x12/0x30 [18492.595353] do_sys_openat2+0x2b4/0x410 [18492.597611] do_sys_open+0x73/0xa0 [18492.599011] __x64_sys_openat+0x24/0x30 [18492.601121] do_syscall_64+0xc1/0x440 [18492.605394] entry_SYSCALL_64_after_hwframe+0x49/0xae [18492.610781] RIP: 0033:0x7f53425a5332 [18492.612999] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18492.623919] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18492.630885] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18492.637574] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18492.641991] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18492.646838] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18492.650929] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18504.057503] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [18504.061512] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18504.108267] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18504.117123] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18504.122645] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18504.128874] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18504.132734] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18504.141246] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18504.144325] RAX: 0000000000007d23 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18504.147154] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [18504.150498] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18504.154075] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18504.162813] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18504.174908] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18504.183295] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18504.187698] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18504.191327] Call Trace: [18504.192610] [18504.193377] ? show_regs.cold.9+0x22/0x2f [18504.194755] ? watchdog_timer_fn.cold.6+0x62/0x10a [18504.199629] ? __hrtimer_run_queues+0x1c2/0x4d0 [18504.201961] ? watchdog+0x40/0x40 [18504.203587] ? hrtimer_interrupt+0x128/0x2c0 [18504.205126] ? smp_apic_timer_interrupt+0xa6/0x270 [18504.207240] ? apic_timer_interrupt+0xf/0x20 [18504.209112] [18504.210130] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18504.212322] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18504.217097] ? newidle_balance.isra.93+0x207/0x580 [18504.222088] do_raw_spin_lock+0x177/0x190 [18504.225972] _raw_spin_lock+0x21/0x30 [18504.228633] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18504.231573] process_one_work+0x2c8/0x700 [18504.233659] worker_thread+0x296/0x6e0 [18504.237526] ? rescuer_thread+0x570/0x570 [18504.239147] kthread+0x1d1/0x200 [18504.242382] ? set_kthread_struct+0x70/0x70 [18504.246454] ret_from_fork+0x1f/0x30 [18507.745240] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15870s! [18507.748538] Showing busy workqueues and worker pools: [18507.751250] workqueue events: flags=0x0 [18507.753665] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18507.757764] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18507.763990] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18507.767505] pending: cpu_latency_work [ptlrpc] [18507.770565] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18507.774215] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18507.779438] workqueue events_long: flags=0x0 [18507.781690] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18507.785569] pending: lu_objects_flush [obdclass] [18507.788052] workqueue events_unbound: flags=0x2 [18507.790316] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18507.792326] pending: flush_memcg_stats_dwork, idle_cull_fn [18507.795504] workqueue events_power_efficient: flags=0x80 [18507.798357] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18507.801808] pending: check_lifetime, do_cache_clean [18507.804390] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18507.807574] pending: neigh_periodic_work, neigh_periodic_work [18507.811236] workqueue mm_percpu_wq: flags=0x8 [18507.813755] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18507.818733] pending: vmstat_update, lru_add_drain_per_cpu [18507.823355] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18507.827316] pending: vmstat_update, lru_add_drain_per_cpu [18507.829863] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18507.833736] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18507.837150] workqueue xprtiod: flags=0xa [18507.839185] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18507.842533] pending: xprt_autoclose, xs_error_handle [18507.845326] workqueue ipv6_addrconf: flags=0x40008 [18507.847186] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18507.849883] pending: addrconf_verify_work [18507.852094] workqueue knbd0-recv: flags=0x1a [18507.854568] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18507.857777] in-flight: 651:recv_work [18507.859924] workqueue ptlrpc_pinger: flags=0x2 [18507.862217] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18507.865640] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18507.869041] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15870s workers=4 idle: 15979 31824 8 [18507.873311] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18498s workers=2 idle: 654 [18520.055228] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [18520.057011] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [18520.060136] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [18520.060155] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18520.060244] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18520.060250] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18520.060253] RIP: 0010:native_safe_halt+0x2e/0x50 [18520.060266] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18520.060270] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18520.060275] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18520.060278] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18520.060280] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18520.060282] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18520.060285] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18520.060288] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18520.060291] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18520.060294] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18520.060302] Call Trace: [18520.060307] [18520.060312] ? show_regs.cold.9+0x22/0x2f [18520.060322] ? watchdog_timer_fn.cold.6+0x62/0x10a [18520.060328] ? __hrtimer_run_queues+0x1c2/0x4d0 [18520.060332] ? watchdog+0x40/0x40 [18520.060336] ? hrtimer_interrupt+0x128/0x2c0 [18520.060340] ? smp_apic_timer_interrupt+0xa6/0x270 [18520.060347] ? apic_timer_interrupt+0xf/0x20 [18520.060351] [18520.060354] ? native_safe_halt+0x2e/0x50 [18520.060357] kvm_wait+0xcd/0xe0 [18520.060365] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18520.060374] do_raw_spin_lock+0x177/0x190 [18520.060377] _raw_spin_lock+0x21/0x30 [18520.060381] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18520.060717] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18520.060842] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18520.061104] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18520.061325] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18520.061537] ? woken_wake_function+0x30/0x30 [18520.061545] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18520.062988] kthread+0x1d1/0x200 [18520.062993] ? set_kthread_struct+0x70/0x70 [18520.062996] ret_from_fork+0x1f/0x30 [18520.063083] Modules linked in: lustre(O) [18520.066992] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18520.067078] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18520.067086] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18520.067089] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18520.067102] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18520.067107] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18520.067111] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18520.067114] RDX: 000000000000161f RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18520.067117] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18520.067119] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18520.067121] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18520.067125] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18520.067128] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18520.067131] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18520.067139] Call Trace: [18520.067162] [18520.067169] ? show_regs.cold.9+0x22/0x2f [18520.067180] ? watchdog_timer_fn.cold.6+0x62/0x10a [18520.067187] ? __hrtimer_run_queues+0x1c2/0x4d0 [18520.067194] ? watchdog+0x40/0x40 [18520.067200] ? hrtimer_interrupt+0x128/0x2c0 [18520.067204] ? smp_apic_timer_interrupt+0xa6/0x270 [18520.067212] ? apic_timer_interrupt+0xf/0x20 [18520.067216] [18520.067219] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18520.067224] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18520.067229] do_raw_spin_lock+0x177/0x190 [18520.067234] _raw_spin_lock+0x21/0x30 [18520.067240] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18520.067633] obd_mod_free+0x46/0x140 [ptlrpc] [18520.068001] mdc_commit_open+0x91/0xb0 [mdc] [18520.068040] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18520.068294] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18520.068552] after_reply+0xe2c/0x1780 [ptlrpc] [18520.068883] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18520.069084] ? __next_timer_interrupt+0x160/0x160 [18520.069094] ptlrpcd+0xa82/0xe40 [ptlrpc] [18520.069307] ? do_wait_intr+0xf0/0xf0 [18520.069490] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18520.069694] kthread+0x1d1/0x200 [18520.069700] ? set_kthread_struct+0x70/0x70 [18520.069704] ret_from_fork+0x1f/0x30 [18520.534203] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18520.561826] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18520.569739] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18520.577094] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18520.583932] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18520.611073] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18520.617013] RAX: 0000000000006d21 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18520.628606] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18520.634416] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18520.638542] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18520.643684] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18520.649088] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18520.653460] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18520.657130] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18520.660785] Call Trace: [18520.662253] [18520.663280] ? show_regs.cold.9+0x22/0x2f [18520.664985] ? watchdog_timer_fn.cold.6+0x62/0x10a [18520.668831] ? __hrtimer_run_queues+0x1c2/0x4d0 [18520.671324] ? watchdog+0x40/0x40 [18520.673331] ? hrtimer_interrupt+0x128/0x2c0 [18520.676130] ? smp_apic_timer_interrupt+0xa6/0x270 [18520.679620] ? apic_timer_interrupt+0xf/0x20 [18520.682133] [18520.683812] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18520.687283] do_raw_spin_lock+0x177/0x190 [18520.689815] _raw_spin_lock+0x21/0x30 [18520.693075] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18520.696772] ? do_raw_spin_unlock+0x75/0x190 [18520.698803] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18520.704902] ? do_wait_intr+0xf0/0xf0 [18520.706501] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18520.712287] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18520.715260] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18520.721022] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18520.724872] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18520.727654] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18520.729980] mdc_intent_lock+0x353/0x7e0 [mdc] [18520.735225] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18520.738779] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18520.741810] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18520.744786] lmv_intent_open+0x338/0x1150 [lmv] [18520.747880] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18520.750287] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18520.752915] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18520.754990] ll_intent_lock+0xc5/0xaa0 [lustre] [18520.757074] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18520.761203] ll_lookup_it+0xe0e/0x2270 [lustre] [18520.768648] ? do_raw_spin_unlock+0x75/0x190 [18520.769826] ? _raw_spin_unlock+0x12/0x30 [18520.771255] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18520.774865] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18520.777797] ll_atomic_open+0x36a/0xd70 [lustre] [18520.780383] lookup_open+0x356/0xb40 [18520.783483] do_last+0x313/0xfc0 [18520.788182] ? nd_jump_root+0xe5/0x160 [18520.789645] ? path_init+0x437/0x520 [18520.792028] path_openat+0xf7/0x500 [18520.795390] ? do_raw_spin_unlock+0x75/0x190 [18520.801320] do_filp_open+0x99/0x140 [18520.806694] ? getname_flags+0x6e/0x330 [18520.809649] ? __check_object_size+0xff/0x256 [18520.811863] ? do_raw_spin_unlock+0x75/0x190 [18520.816093] ? _raw_spin_unlock+0x12/0x30 [18520.818573] do_sys_openat2+0x2b4/0x410 [18520.819626] do_sys_open+0x73/0xa0 [18520.820509] __x64_sys_openat+0x24/0x30 [18520.822099] do_syscall_64+0xc1/0x440 [18520.823849] entry_SYSCALL_64_after_hwframe+0x49/0xae [18520.826545] RIP: 0033:0x7f53425a5332 [18520.828033] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18520.848206] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18520.854656] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18520.861406] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18520.863558] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18520.869776] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18520.874931] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18532.053782] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [18532.065341] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18532.120301] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18532.125713] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18532.132855] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18532.135935] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18532.143011] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18532.166658] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18532.170390] RAX: 00000000000059c8 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18532.176072] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [18532.181688] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18532.188007] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18532.195377] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18532.201467] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18532.215955] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18532.226777] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18532.238018] Call Trace: [18532.239470] [18532.242775] ? show_regs.cold.9+0x22/0x2f [18532.247319] ? watchdog_timer_fn.cold.6+0x62/0x10a [18532.251331] ? __hrtimer_run_queues+0x1c2/0x4d0 [18532.254377] ? watchdog+0x40/0x40 [18532.257246] ? hrtimer_interrupt+0x128/0x2c0 [18532.259786] ? smp_apic_timer_interrupt+0xa6/0x270 [18532.265156] ? apic_timer_interrupt+0xf/0x20 [18532.266733] [18532.270454] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18532.272443] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18532.276781] ? newidle_balance.isra.93+0x207/0x580 [18532.282411] do_raw_spin_lock+0x177/0x190 [18532.285526] _raw_spin_lock+0x21/0x30 [18532.287805] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18532.299271] process_one_work+0x2c8/0x700 [18532.302804] worker_thread+0x296/0x6e0 [18532.306513] ? rescuer_thread+0x570/0x570 [18532.308771] kthread+0x1d1/0x200 [18532.311720] ? set_kthread_struct+0x70/0x70 [18532.316784] ret_from_fork+0x1f/0x30 [18534.835878] rcu: INFO: rcu_sched self-detected stall on CPU [18534.839203] rcu: 2-....: (9204012 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [18534.843502] (t=16440409 jiffies g=274185 q=68157) [18534.848531] rcu: rcu_sched kthread starved for 15898775 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [18534.858867] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [18534.864321] rcu: RCU grace-period kthread stack dump: [18534.866212] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [18534.871191] Call Trace: [18534.872446] __schedule+0x351/0xcb0 [18534.874509] schedule+0xc0/0x180 [18534.875466] schedule_timeout+0xb4/0x190 [18534.876714] ? do_raw_spin_unlock+0x75/0x190 [18534.880406] ? __next_timer_interrupt+0x160/0x160 [18534.882332] ? prepare_to_swait_event+0xf3/0x140 [18534.885883] rcu_gp_kthread+0x8dd/0x15b0 [18534.887432] ? force_qs_rnp+0x2c0/0x2c0 [18534.891856] kthread+0x1d1/0x200 [18534.897219] ? set_kthread_struct+0x70/0x70 [18534.898272] ret_from_fork+0x1f/0x30 [18534.899222] rcu: Stack dump where RCU GP kthread last ran: [18534.905179] NMI backtrace for cpu 2 [18534.910802] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18534.922401] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18534.941340] Call Trace: [18534.943715] [18534.944320] ? dump_stack+0xbb/0x10e [18534.947399] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [18534.954040] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [18534.959281] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [18534.962814] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [18534.966958] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [18534.972430] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [18534.978380] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [18534.983247] ? update_process_times+0x6d/0xc0 [18534.985749] ? tick_sched_handle.isra.18+0x27/0x90 [18534.989649] ? tick_sched_timer+0x53/0xb0 [18534.992285] ? __hrtimer_run_queues+0x1c2/0x4d0 [18534.995417] ? tick_sched_handle.isra.18+0x90/0x90 [18534.998170] ? hrtimer_interrupt+0x128/0x2c0 [18534.999352] ? smp_apic_timer_interrupt+0xa6/0x270 [18535.003719] ? apic_timer_interrupt+0xf/0x20 [18535.007218] [18535.008195] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18535.013724] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18535.019580] ? do_raw_spin_lock+0x177/0x190 [18535.021452] ? _raw_spin_lock+0x21/0x30 [18535.024669] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18535.027714] ? obd_mod_free+0x46/0x140 [ptlrpc] [18535.030583] ? mdc_commit_open+0x91/0xb0 [mdc] [18535.032954] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18535.037719] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18535.044612] ? after_reply+0xe2c/0x1780 [ptlrpc] [18535.048553] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18535.054673] ? __next_timer_interrupt+0x160/0x160 [18535.058496] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [18535.062420] ? do_wait_intr+0xf0/0xf0 [18535.063806] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18535.070204] ? kthread+0x1d1/0x200 [18535.071197] ? set_kthread_struct+0x70/0x70 [18535.072327] ? ret_from_fork+0x1f/0x30 [18535.074199] Sending NMI from CPU 2 to CPUs 1: [18535.077493] NMI backtrace for cpu 1 [18535.077500] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18535.077502] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18535.077507] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18535.077509] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18535.077512] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [18535.077514] RAX: 0000000000003390 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18535.077516] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18535.077517] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18535.077519] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18535.077520] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18535.077522] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18535.077523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18535.077524] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18535.077525] Call Trace: [18535.077526] [18535.077527] ? show_regs.cold.9+0x22/0x2f [18535.077528] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [18535.077531] ? nmi_cpu_backtrace_handler+0x15/0x40 [18535.077532] ? nmi_handle+0xbd/0x210 [18535.077533] ? default_do_nmi+0x4b/0x1b0 [18535.077534] ? do_nmi+0x280/0x380 [18535.077535] ? end_repeat_nmi+0x16/0x1f [18535.077535] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18535.077536] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18535.077537] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18535.077538] [18535.077539] do_raw_spin_lock+0x177/0x190 [18535.077540] _raw_spin_lock+0x21/0x30 [18535.077541] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18535.077545] ? do_raw_spin_unlock+0x75/0x190 [18535.077545] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18535.077546] ? do_wait_intr+0xf0/0xf0 [18535.077548] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18535.077549] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18535.077550] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18535.077551] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18535.077552] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18535.077553] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18535.077554] mdc_intent_lock+0x353/0x7e0 [mdc] [18535.077555] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18535.077556] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18535.077557] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18535.077558] lmv_intent_open+0x338/0x1150 [lmv] [18535.077559] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18535.077560] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18535.077561] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18535.077562] ll_intent_lock+0xc5/0xaa0 [lustre] [18535.077563] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18535.077564] ll_lookup_it+0xe0e/0x2270 [lustre] [18535.077565] ? do_raw_spin_unlock+0x75/0x190 [18535.077565] ? _raw_spin_unlock+0x12/0x30 [18535.077566] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18535.077567] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18535.077568] ll_atomic_open+0x36a/0xd70 [lustre] [18535.077571] lookup_open+0x356/0xb40 [18535.077573] do_last+0x313/0xfc0 [18535.077574] ? nd_jump_root+0xe5/0x160 [18535.077575] ? path_init+0x437/0x520 [18535.077576] path_openat+0xf7/0x500 [18535.077577] ? do_raw_spin_unlock+0x75/0x190 [18535.077578] do_filp_open+0x99/0x140 [18535.077579] ? getname_flags+0x6e/0x330 [18535.077579] ? __check_object_size+0xff/0x256 [18535.077580] ? do_raw_spin_unlock+0x75/0x190 [18535.077581] ? _raw_spin_unlock+0x12/0x30 [18535.077582] do_sys_openat2+0x2b4/0x410 [18535.077583] do_sys_open+0x73/0xa0 [18535.077584] __x64_sys_openat+0x24/0x30 [18535.077587] do_syscall_64+0xc1/0x440 [18535.077588] entry_SYSCALL_64_after_hwframe+0x49/0xae [18535.077589] RIP: 0033:0x7f53425a5332 [18535.077590] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18535.077591] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18535.077593] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18535.077595] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18535.077596] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18535.077597] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18535.077598] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18535.077602] NMI backtrace for cpu 2 [18535.338940] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18535.343914] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18535.349731] Call Trace: [18535.350688] [18535.351282] ? dump_stack+0xbb/0x10e [18535.352300] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [18535.356435] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [18535.359282] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [18535.364618] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [18535.366279] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [18535.374049] ? rcu_dump_cpu_stacks+0x11b/0x171 [18535.378673] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [18535.381825] ? update_process_times+0x6d/0xc0 [18535.389342] ? tick_sched_handle.isra.18+0x27/0x90 [18535.390673] ? tick_sched_timer+0x53/0xb0 [18535.391778] ? __hrtimer_run_queues+0x1c2/0x4d0 [18535.406509] ? tick_sched_handle.isra.18+0x90/0x90 [18535.407721] ? hrtimer_interrupt+0x128/0x2c0 [18535.408733] ? smp_apic_timer_interrupt+0xa6/0x270 [18535.415220] ? apic_timer_interrupt+0xf/0x20 [18535.416770] [18535.417388] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18535.422869] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18535.424198] ? do_raw_spin_lock+0x177/0x190 [18535.431314] ? _raw_spin_lock+0x21/0x30 [18535.432423] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18535.439563] ? obd_mod_free+0x46/0x140 [ptlrpc] [18535.444717] ? mdc_commit_open+0x91/0xb0 [mdc] [18535.452715] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18535.454486] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18535.461109] ? after_reply+0xe2c/0x1780 [ptlrpc] [18535.463148] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18535.466679] ? __next_timer_interrupt+0x160/0x160 [18535.468270] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [18535.473643] ? do_wait_intr+0xf0/0xf0 [18535.474537] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18535.478233] ? kthread+0x1d1/0x200 [18535.480539] ? set_kthread_struct+0x70/0x70 [18535.486641] ? ret_from_fork+0x1f/0x30 [18537.953490] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15900s! [18537.957126] Showing busy workqueues and worker pools: [18537.960358] workqueue events: flags=0x0 [18537.961626] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18537.963456] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18537.968499] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18537.972326] pending: cpu_latency_work [ptlrpc] [18537.974346] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18537.977218] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18537.980827] workqueue events_long: flags=0x0 [18537.982220] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18537.986279] pending: lu_objects_flush [obdclass] [18537.990252] workqueue events_unbound: flags=0x2 [18537.992521] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18537.996531] pending: flush_memcg_stats_dwork, idle_cull_fn [18537.998526] workqueue events_power_efficient: flags=0x80 [18538.000789] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18538.003930] pending: check_lifetime, do_cache_clean [18538.006306] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18538.010111] pending: neigh_periodic_work, neigh_periodic_work [18538.015703] workqueue mm_percpu_wq: flags=0x8 [18538.019993] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18538.021989] pending: vmstat_update, lru_add_drain_per_cpu [18538.025320] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18538.030543] pending: vmstat_update, lru_add_drain_per_cpu [18538.034557] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18538.037853] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18538.041699] workqueue xprtiod: flags=0xa [18538.043669] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18538.046813] pending: xprt_autoclose, xs_error_handle [18538.049600] workqueue ipv6_addrconf: flags=0x40008 [18538.051970] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18538.055710] pending: addrconf_verify_work [18538.058385] workqueue knbd0-recv: flags=0x1a [18538.060610] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18538.065060] in-flight: 651:recv_work [18538.067088] workqueue ptlrpc_pinger: flags=0x2 [18538.069737] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18538.072590] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18538.076419] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15900s workers=4 idle: 15979 31824 8 [18538.078409] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18528s workers=2 idle: 654 [18548.059609] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [18548.062304] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18548.106464] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18548.117336] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18548.122797] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18548.128268] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18548.137123] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18548.142881] RAX: 0000000000007739 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18548.152644] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18548.156163] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18548.161154] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18548.166400] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18548.170621] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18548.175106] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18548.177091] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18548.184909] Call Trace: [18548.186748] [18548.187746] ? show_regs.cold.9+0x22/0x2f [18548.189911] ? watchdog_timer_fn.cold.6+0x62/0x10a [18548.191955] ? __hrtimer_run_queues+0x1c2/0x4d0 [18548.193985] ? watchdog+0x40/0x40 [18548.195526] ? hrtimer_interrupt+0x128/0x2c0 [18548.197327] ? smp_apic_timer_interrupt+0xa6/0x270 [18548.205816] ? apic_timer_interrupt+0xf/0x20 [18548.217179] [18548.218010] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18548.220956] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18548.225986] do_raw_spin_lock+0x177/0x190 [18548.230236] _raw_spin_lock+0x21/0x30 [18548.232513] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18548.240405] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18548.243918] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18548.247199] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18548.249966] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18548.252606] ? woken_wake_function+0x30/0x30 [18548.254570] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18548.262719] kthread+0x1d1/0x200 [18548.264105] ? set_kthread_struct+0x70/0x70 [18548.266112] ret_from_fork+0x1f/0x30 [18560.055022] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18560.057029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18560.059338] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) [18560.062330] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18560.062406] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18560.062414] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18560.062416] RIP: 0010:native_safe_halt+0x2e/0x50 [18560.062432] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18560.062436] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18560.062440] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18560.062442] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18560.062444] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18560.075025] obdclass(O) [18560.100232] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18560.100243] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18560.100246] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18560.100249] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18560.100252] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18560.100260] Call Trace: [18560.100266] [18560.100271] ? show_regs.cold.9+0x22/0x2f [18560.100283] ? watchdog_timer_fn.cold.6+0x62/0x10a [18560.100289] ? __hrtimer_run_queues+0x1c2/0x4d0 [18560.100294] ? watchdog+0x40/0x40 [18560.100298] ? hrtimer_interrupt+0x128/0x2c0 [18560.100302] ? smp_apic_timer_interrupt+0xa6/0x270 [18560.100311] ? apic_timer_interrupt+0xf/0x20 [18560.100316] [18560.100318] ? native_safe_halt+0x2e/0x50 [18560.100323] kvm_wait+0xcd/0xe0 [18560.100330] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18560.100337] do_raw_spin_lock+0x177/0x190 [18560.100341] _raw_spin_lock+0x21/0x30 [18560.100348] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18560.108945] ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18560.108992] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18560.108998] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18560.109001] RIP: 0010:native_safe_halt+0x2e/0x50 [18560.109013] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18560.109017] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [18560.109023] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18560.109027] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18560.109029] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18560.109032] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18560.109035] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18560.109038] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18560.109041] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18560.109043] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18560.109051] Call Trace: [18560.109058] [18560.117171] obd_mod_free+0x46/0x140 [ptlrpc] [18560.117464] mdc_commit_open+0x91/0xb0 [mdc] [18560.117505] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18560.117694] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18560.117889] after_reply+0xe2c/0x1780 [ptlrpc] [18560.121684] ? show_regs.cold.9+0x22/0x2f [18560.145332] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18560.145638] ? __next_timer_interrupt+0x160/0x160 [18560.145649] ptlrpcd+0xa82/0xe40 [ptlrpc] [18560.145853] ? do_wait_intr+0xf0/0xf0 [18560.145863] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18560.150459] ? watchdog_timer_fn.cold.6+0x62/0x10a [18560.150477] ? __hrtimer_run_queues+0x1c2/0x4d0 [18560.150481] ? watchdog+0x40/0x40 [18560.150486] ? hrtimer_interrupt+0x128/0x2c0 [18560.150490] ? smp_apic_timer_interrupt+0xa6/0x270 [18560.150498] ? apic_timer_interrupt+0xf/0x20 [18560.150502] [18560.150507] ? native_safe_halt+0x2e/0x50 [18560.150511] kvm_wait+0xcd/0xe0 [18560.150520] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18560.150529] do_raw_spin_lock+0x177/0x190 [18560.150533] _raw_spin_lock+0x21/0x30 [18560.150538] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18560.150791] ? do_raw_spin_unlock+0x75/0x190 [18560.150798] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18560.151001] ? do_wait_intr+0xf0/0xf0 [18560.151010] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18560.151220] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18560.151438] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18560.159257] kthread+0x1d1/0x200 [18560.159277] ? set_kthread_struct+0x70/0x70 [18560.159283] ret_from_fork+0x1f/0x30 [18560.383685] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18560.386099] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18560.388713] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18560.390962] mdc_intent_lock+0x353/0x7e0 [mdc] [18560.392892] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18560.395102] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18560.397699] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18560.400255] lmv_intent_open+0x338/0x1150 [lmv] [18560.403100] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18560.405978] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18560.408736] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18560.411042] ll_intent_lock+0xc5/0xaa0 [lustre] [18560.415861] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18560.418857] ll_lookup_it+0xe0e/0x2270 [lustre] [18560.421829] ? do_raw_spin_unlock+0x75/0x190 [18560.424416] ? _raw_spin_unlock+0x12/0x30 [18560.426769] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18560.429396] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18560.432105] ll_atomic_open+0x36a/0xd70 [lustre] [18560.434792] lookup_open+0x356/0xb40 [18560.436713] do_last+0x313/0xfc0 [18560.438376] ? nd_jump_root+0xe5/0x160 [18560.440175] ? path_init+0x437/0x520 [18560.441937] path_openat+0xf7/0x500 [18560.444038] ? do_raw_spin_unlock+0x75/0x190 [18560.446701] do_filp_open+0x99/0x140 [18560.448070] ? getname_flags+0x6e/0x330 [18560.449645] ? __check_object_size+0xff/0x256 [18560.451379] ? do_raw_spin_unlock+0x75/0x190 [18560.453627] ? _raw_spin_unlock+0x12/0x30 [18560.455747] do_sys_openat2+0x2b4/0x410 [18560.457750] do_sys_open+0x73/0xa0 [18560.459403] __x64_sys_openat+0x24/0x30 [18560.461356] do_syscall_64+0xc1/0x440 [18560.463142] entry_SYSCALL_64_after_hwframe+0x49/0xae [18560.465812] RIP: 0033:0x7f53425a5332 [18560.473833] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18560.492215] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18560.497149] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18560.501646] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18560.507973] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18560.519304] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18560.524719] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18564.052618] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [18564.055442] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18564.097502] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18564.106915] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18564.110370] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18564.115789] RIP: 0010:native_safe_halt+0x2e/0x50 [18564.122752] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18564.141538] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18564.144773] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18564.147034] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [18564.154638] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18564.158862] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18564.167035] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18564.178985] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18564.183230] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18564.186304] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18564.190353] Call Trace: [18564.191228] [18564.191926] ? show_regs.cold.9+0x22/0x2f [18564.194987] ? watchdog_timer_fn.cold.6+0x62/0x10a [18564.196700] ? __hrtimer_run_queues+0x1c2/0x4d0 [18564.202014] ? watchdog+0x40/0x40 [18564.205452] ? hrtimer_interrupt+0x128/0x2c0 [18564.209941] ? smp_apic_timer_interrupt+0xa6/0x270 [18564.212520] ? apic_timer_interrupt+0xf/0x20 [18564.214810] [18564.219236] ? native_safe_halt+0x2e/0x50 [18564.223653] kvm_wait+0xcd/0xe0 [18564.225368] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18564.230345] ? newidle_balance.isra.93+0x207/0x580 [18564.232541] do_raw_spin_lock+0x177/0x190 [18564.234199] _raw_spin_lock+0x21/0x30 [18564.235480] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18564.240920] process_one_work+0x2c8/0x700 [18564.242818] worker_thread+0x296/0x6e0 [18564.247295] ? rescuer_thread+0x570/0x570 [18564.249303] kthread+0x1d1/0x200 [18564.250776] ? set_kthread_struct+0x70/0x70 [18564.258810] ret_from_fork+0x1f/0x30 [18568.163627] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15931s! [18568.171980] Showing busy workqueues and worker pools: [18568.175867] workqueue events: flags=0x0 [18568.176957] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18568.184037] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18568.200878] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18568.205241] pending: cpu_latency_work [ptlrpc] [18568.207855] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18568.212617] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18568.223118] workqueue events_long: flags=0x0 [18568.224546] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18568.230848] pending: lu_objects_flush [obdclass] [18568.234865] workqueue events_unbound: flags=0x2 [18568.237369] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18568.246939] pending: flush_memcg_stats_dwork, idle_cull_fn [18568.250526] workqueue events_power_efficient: flags=0x80 [18568.253689] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18568.257958] pending: check_lifetime, do_cache_clean [18568.261481] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18568.271720] pending: neigh_periodic_work, neigh_periodic_work [18568.273812] workqueue mm_percpu_wq: flags=0x8 [18568.274949] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18568.276845] pending: vmstat_update, lru_add_drain_per_cpu [18568.279029] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18568.281694] pending: vmstat_update, lru_add_drain_per_cpu [18568.283882] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18568.286229] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18568.289019] workqueue xprtiod: flags=0xa [18568.292103] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18568.296762] pending: xprt_autoclose, xs_error_handle [18568.298604] workqueue ipv6_addrconf: flags=0x40008 [18568.301365] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18568.307595] pending: addrconf_verify_work [18568.312570] workqueue knbd0-recv: flags=0x1a [18568.317120] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18568.321911] in-flight: 651:recv_work [18568.323542] workqueue ptlrpc_pinger: flags=0x2 [18568.326393] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18568.331068] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18568.335966] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15931s workers=4 idle: 15979 31824 8 [18568.338520] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18558s workers=2 idle: 654 [18576.058037] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [18576.060828] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18576.108789] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18576.117104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18576.122513] RIP: 0010:native_safe_halt+0x2e/0x50 [18576.124150] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18576.135110] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18576.137501] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18576.139841] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18576.142936] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18576.147013] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18576.149645] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18576.151758] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18576.155011] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18576.159455] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18576.164089] Call Trace: [18576.166039] [18576.172417] ? show_regs.cold.9+0x22/0x2f [18576.174487] ? watchdog_timer_fn.cold.6+0x62/0x10a [18576.177369] ? __hrtimer_run_queues+0x1c2/0x4d0 [18576.181109] ? watchdog+0x40/0x40 [18576.182869] ? hrtimer_interrupt+0x128/0x2c0 [18576.193634] ? smp_apic_timer_interrupt+0xa6/0x270 [18576.199104] ? apic_timer_interrupt+0xf/0x20 [18576.207635] [18576.208851] ? native_safe_halt+0x2e/0x50 [18576.210669] kvm_wait+0xcd/0xe0 [18576.211515] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18576.221592] do_raw_spin_lock+0x177/0x190 [18576.226078] _raw_spin_lock+0x21/0x30 [18576.229044] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18576.231444] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18576.233146] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18576.235452] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18576.238638] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18576.240896] ? woken_wake_function+0x30/0x30 [18576.242673] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18576.245736] kthread+0x1d1/0x200 [18576.247324] ? set_kthread_struct+0x70/0x70 [18576.248936] ret_from_fork+0x1f/0x30 [18588.055051] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18588.057914] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18588.058949] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18588.099296] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18588.099310] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18588.099313] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18588.099327] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18588.099332] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18588.099338] RAX: 0000000000006483 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18588.099341] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18588.099343] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18588.099345] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18588.099348] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18588.099351] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18588.099354] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18588.099357] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18588.099367] Call Trace: [18588.099373] [18588.099379] ? show_regs.cold.9+0x22/0x2f [18588.099389] ? watchdog_timer_fn.cold.6+0x62/0x10a [18588.099395] ? __hrtimer_run_queues+0x1c2/0x4d0 [18588.099400] ? watchdog+0x40/0x40 [18588.099404] ? hrtimer_interrupt+0x128/0x2c0 [18588.099408] ? smp_apic_timer_interrupt+0xa6/0x270 [18588.099416] ? apic_timer_interrupt+0xf/0x20 [18588.099421] [18588.099424] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18588.099432] do_raw_spin_lock+0x177/0x190 [18588.099438] _raw_spin_lock+0x21/0x30 [18588.099443] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18588.099720] ? do_raw_spin_unlock+0x75/0x190 [18588.099726] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18588.099943] ? do_wait_intr+0xf0/0xf0 [18588.099953] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18588.100427] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18588.100637] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18588.106801] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic [18588.110543] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18588.114943] crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18588.114983] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18588.114990] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18588.114993] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18588.115007] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18588.115011] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18588.115017] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18588.115019] RDX: 0000000000004be9 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18588.115022] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18588.115024] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18588.115026] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18588.115029] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18588.115031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18588.115034] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18588.115041] Call Trace: [18588.115048] [18588.115054] ? show_regs.cold.9+0x22/0x2f [18588.115064] ? watchdog_timer_fn.cold.6+0x62/0x10a [18588.115069] ? __hrtimer_run_queues+0x1c2/0x4d0 [18588.115074] ? watchdog+0x40/0x40 [18588.115077] ? hrtimer_interrupt+0x128/0x2c0 [18588.115081] ? smp_apic_timer_interrupt+0xa6/0x270 [18588.115088] ? apic_timer_interrupt+0xf/0x20 [18588.115093] [18588.115096] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18588.115101] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18588.115105] do_raw_spin_lock+0x177/0x190 [18588.115110] _raw_spin_lock+0x21/0x30 [18588.115123] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18588.115527] obd_mod_free+0x46/0x140 [ptlrpc] [18588.115810] mdc_commit_open+0x91/0xb0 [mdc] [18588.115848] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18588.116085] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18588.116293] after_reply+0xe2c/0x1780 [ptlrpc] [18588.116500] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18588.116887] ? __next_timer_interrupt+0x160/0x160 [18588.116899] ptlrpcd+0xa82/0xe40 [ptlrpc] [18588.117173] ? do_wait_intr+0xf0/0xf0 [18588.117183] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18588.117492] kthread+0x1d1/0x200 [18588.117498] ? set_kthread_struct+0x70/0x70 [18588.117502] ret_from_fork+0x1f/0x30 [18588.455505] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18588.458507] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18588.460554] mdc_intent_lock+0x353/0x7e0 [mdc] [18588.462386] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18588.465136] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18588.468416] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18588.470590] lmv_intent_open+0x338/0x1150 [lmv] [18588.473252] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18588.476053] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18588.478931] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18588.481657] ll_intent_lock+0xc5/0xaa0 [lustre] [18588.486550] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18588.490129] ll_lookup_it+0xe0e/0x2270 [lustre] [18588.493923] ? do_raw_spin_unlock+0x75/0x190 [18588.496304] ? _raw_spin_unlock+0x12/0x30 [18588.499325] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18588.502487] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18588.507734] ll_atomic_open+0x36a/0xd70 [lustre] [18588.511633] lookup_open+0x356/0xb40 [18588.514362] do_last+0x313/0xfc0 [18588.515450] ? nd_jump_root+0xe5/0x160 [18588.519226] ? path_init+0x437/0x520 [18588.524367] path_openat+0xf7/0x500 [18588.526232] ? do_raw_spin_unlock+0x75/0x190 [18588.528250] do_filp_open+0x99/0x140 [18588.529644] ? getname_flags+0x6e/0x330 [18588.530901] ? __check_object_size+0xff/0x256 [18588.532129] ? do_raw_spin_unlock+0x75/0x190 [18588.534476] ? _raw_spin_unlock+0x12/0x30 [18588.540213] do_sys_openat2+0x2b4/0x410 [18588.543777] do_sys_open+0x73/0xa0 [18588.547537] __x64_sys_openat+0x24/0x30 [18588.551328] do_syscall_64+0xc1/0x440 [18588.557921] entry_SYSCALL_64_after_hwframe+0x49/0xae [18588.562134] RIP: 0033:0x7f53425a5332 [18588.563869] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18588.579489] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18588.591522] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18588.598634] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18588.607337] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18588.617239] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18588.627203] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18596.052027] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [18596.059300] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18596.099560] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18596.106736] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18596.111467] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18596.115244] RIP: 0010:native_safe_halt+0x2e/0x50 [18596.116882] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18596.140459] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18596.147788] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18596.150925] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [18596.160521] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18596.166226] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18596.177496] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18596.188993] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18596.192747] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18596.197908] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18596.200440] Call Trace: [18596.201414] [18596.202217] ? show_regs.cold.9+0x22/0x2f [18596.203784] ? watchdog_timer_fn.cold.6+0x62/0x10a [18596.205550] ? __hrtimer_run_queues+0x1c2/0x4d0 [18596.207184] ? watchdog+0x40/0x40 [18596.208416] ? hrtimer_interrupt+0x128/0x2c0 [18596.209994] ? smp_apic_timer_interrupt+0xa6/0x270 [18596.212012] ? apic_timer_interrupt+0xf/0x20 [18596.213528] [18596.214429] ? native_safe_halt+0x2e/0x50 [18596.216003] kvm_wait+0xcd/0xe0 [18596.216998] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18596.218730] ? newidle_balance.isra.93+0x207/0x580 [18596.220573] do_raw_spin_lock+0x177/0x190 [18596.222149] _raw_spin_lock+0x21/0x30 [18596.223384] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18596.225547] process_one_work+0x2c8/0x700 [18596.227227] worker_thread+0x296/0x6e0 [18596.228625] ? rescuer_thread+0x570/0x570 [18596.230072] kthread+0x1d1/0x200 [18596.231346] ? set_kthread_struct+0x70/0x70 [18596.235282] ret_from_fork+0x1f/0x30 [18598.367052] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15961s! [18598.369531] Showing busy workqueues and worker pools: [18598.371284] workqueue events: flags=0x0 [18598.372438] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18598.374423] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18598.377702] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18598.379505] pending: cpu_latency_work [ptlrpc] [18598.381266] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18598.383168] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18598.385897] workqueue events_long: flags=0x0 [18598.386903] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18598.389234] pending: lu_objects_flush [obdclass] [18598.394414] workqueue events_unbound: flags=0x2 [18598.396190] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18598.401664] pending: flush_memcg_stats_dwork, idle_cull_fn [18598.403772] workqueue events_power_efficient: flags=0x80 [18598.405838] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18598.411700] pending: check_lifetime, do_cache_clean [18598.413610] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18598.417416] pending: neigh_periodic_work, neigh_periodic_work [18598.420919] workqueue mm_percpu_wq: flags=0x8 [18598.422657] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18598.425138] pending: vmstat_update, lru_add_drain_per_cpu [18598.427378] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18598.431224] pending: vmstat_update, lru_add_drain_per_cpu [18598.437101] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18598.441946] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18598.444301] workqueue xprtiod: flags=0xa [18598.447295] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18598.450092] pending: xprt_autoclose, xs_error_handle [18598.458317] workqueue ipv6_addrconf: flags=0x40008 [18598.462191] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18598.466317] pending: addrconf_verify_work [18598.473889] workqueue knbd0-recv: flags=0x1a [18598.475630] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18598.482435] in-flight: 651:recv_work [18598.484739] workqueue ptlrpc_pinger: flags=0x2 [18598.490408] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18598.493523] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18598.501601] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15961s workers=4 idle: 15979 31824 8 [18598.509043] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18588s workers=2 idle: 654 [18604.058011] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [18604.060791] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18604.097163] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18604.102761] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18604.111518] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18604.115903] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18604.131628] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18604.138619] RAX: 000000000000322b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18604.143668] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18604.149064] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18604.153823] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18604.158791] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18604.163564] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18604.168979] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18604.171268] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18604.179866] Call Trace: [18604.181229] [18604.182204] ? show_regs.cold.9+0x22/0x2f [18604.185202] ? watchdog_timer_fn.cold.6+0x62/0x10a [18604.187160] ? __hrtimer_run_queues+0x1c2/0x4d0 [18604.189643] ? watchdog+0x40/0x40 [18604.190928] ? hrtimer_interrupt+0x128/0x2c0 [18604.194178] ? smp_apic_timer_interrupt+0xa6/0x270 [18604.196175] ? apic_timer_interrupt+0xf/0x20 [18604.198344] [18604.200734] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18604.205033] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18604.209280] do_raw_spin_lock+0x177/0x190 [18604.212487] _raw_spin_lock+0x21/0x30 [18604.213828] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18604.218155] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18604.220555] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18604.223619] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18604.229869] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18604.233243] ? woken_wake_function+0x30/0x30 [18604.234627] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18604.242829] kthread+0x1d1/0x200 [18604.250209] ? set_kthread_struct+0x70/0x70 [18604.254528] ret_from_fork+0x1f/0x30 [18616.058053] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18616.061452] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18616.066864] Modules linked in: [18616.081706] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18616.081784] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18616.081791] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18616.081794] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18616.081809] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18616.081813] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18616.081818] RAX: 0000000000000210 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18616.081821] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18616.081823] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18616.081826] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18616.081828] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18616.081832] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18616.081835] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18616.081837] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18616.081846] Call Trace: [18616.081852] [18616.081859] ? show_regs.cold.9+0x22/0x2f [18616.081870] ? watchdog_timer_fn.cold.6+0x62/0x10a [18616.081877] ? __hrtimer_run_queues+0x1c2/0x4d0 [18616.081881] ? watchdog+0x40/0x40 [18616.081887] ? hrtimer_interrupt+0x128/0x2c0 [18616.081891] ? smp_apic_timer_interrupt+0xa6/0x270 [18616.081900] ? apic_timer_interrupt+0xf/0x20 [18616.081905] [18616.081909] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18616.081916] do_raw_spin_lock+0x177/0x190 [18616.081922] _raw_spin_lock+0x21/0x30 [18616.081927] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18616.082244] ? do_raw_spin_unlock+0x75/0x190 [18616.082250] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18616.082480] ? do_wait_intr+0xf0/0xf0 [18616.082490] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18616.082833] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18616.083056] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18616.083279] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18616.083830] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18616.083870] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18616.083897] mdc_intent_lock+0x353/0x7e0 [mdc] [18616.083924] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18616.084007] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18616.138088] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18616.138170] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18616.138178] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18616.138180] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18616.138194] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18616.138198] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18616.138203] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18616.138206] RDX: 0000000000007565 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18616.138208] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18616.138210] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18616.138213] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18616.138216] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18616.138220] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18616.138222] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18616.138232] Call Trace: [18616.138238] [18616.138244] ? show_regs.cold.9+0x22/0x2f [18616.138256] ? watchdog_timer_fn.cold.6+0x62/0x10a [18616.138262] ? __hrtimer_run_queues+0x1c2/0x4d0 [18616.138266] ? watchdog+0x40/0x40 [18616.138271] ? hrtimer_interrupt+0x128/0x2c0 [18616.138275] ? smp_apic_timer_interrupt+0xa6/0x270 [18616.138284] ? apic_timer_interrupt+0xf/0x20 [18616.138289] [18616.138292] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18616.138297] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18616.138302] do_raw_spin_lock+0x177/0x190 [18616.138307] _raw_spin_lock+0x21/0x30 [18616.138314] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18616.138996] obd_mod_free+0x46/0x140 [ptlrpc] [18616.139198] mdc_commit_open+0x91/0xb0 [mdc] [18616.139241] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18616.139449] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18616.160737] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18616.160796] lmv_intent_open+0x338/0x1150 [lmv] [18616.160830] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18616.160947] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18616.161040] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18616.161064] ll_intent_lock+0xc5/0xaa0 [lustre] [18616.161155] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18616.161248] ll_lookup_it+0xe0e/0x2270 [lustre] [18616.161340] ? do_raw_spin_unlock+0x75/0x190 [18616.161349] ? _raw_spin_unlock+0x12/0x30 [18616.161356] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18616.161382] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18616.161403] ll_atomic_open+0x36a/0xd70 [lustre] [18616.161506] lookup_open+0x356/0xb40 [18616.161515] do_last+0x313/0xfc0 [18616.161520] ? nd_jump_root+0xe5/0x160 [18616.161525] ? path_init+0x437/0x520 [18616.161529] path_openat+0xf7/0x500 [18616.161534] ? do_raw_spin_unlock+0x75/0x190 [18616.161539] do_filp_open+0x99/0x140 [18616.161545] ? getname_flags+0x6e/0x330 [18616.161550] ? __check_object_size+0xff/0x256 [18616.161557] ? do_raw_spin_unlock+0x75/0x190 [18616.161561] ? _raw_spin_unlock+0x12/0x30 [18616.161565] do_sys_openat2+0x2b4/0x410 [18616.161569] do_sys_open+0x73/0xa0 [18616.161573] __x64_sys_openat+0x24/0x30 [18616.161576] do_syscall_64+0xc1/0x440 [18616.161583] entry_SYSCALL_64_after_hwframe+0x49/0xae [18616.161588] RIP: 0033:0x7f53425a5332 [18616.161596] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18616.161600] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18616.161604] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18616.161606] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18616.161609] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18616.161610] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18616.161612] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18616.598672] after_reply+0xe2c/0x1780 [ptlrpc] [18616.600998] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18616.603408] ? __next_timer_interrupt+0x160/0x160 [18616.605798] ptlrpcd+0xa82/0xe40 [ptlrpc] [18616.607589] ? do_wait_intr+0xf0/0xf0 [18616.609094] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18616.614438] kthread+0x1d1/0x200 [18616.615711] ? set_kthread_struct+0x70/0x70 [18616.619357] ret_from_fork+0x1f/0x30 [18624.052106] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [18624.054907] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18624.094749] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18624.106581] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18624.115650] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18624.126954] RIP: 0010:native_safe_halt+0x2e/0x50 [18624.134880] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18624.157510] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18624.167207] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18624.181615] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [18624.192074] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18624.199758] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18624.207402] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18624.226741] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18624.245249] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18624.249267] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18624.262779] Call Trace: [18624.266295] [18624.269188] ? show_regs.cold.9+0x22/0x2f [18624.274654] ? watchdog_timer_fn.cold.6+0x62/0x10a [18624.280687] ? __hrtimer_run_queues+0x1c2/0x4d0 [18624.286027] ? watchdog+0x40/0x40 [18624.291496] ? hrtimer_interrupt+0x128/0x2c0 [18624.297173] ? smp_apic_timer_interrupt+0xa6/0x270 [18624.305263] ? apic_timer_interrupt+0xf/0x20 [18624.312734] [18624.315875] ? native_safe_halt+0x2e/0x50 [18624.321861] kvm_wait+0xcd/0xe0 [18624.326767] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18624.336119] ? newidle_balance.isra.93+0x207/0x580 [18624.343426] do_raw_spin_lock+0x177/0x190 [18624.346761] _raw_spin_lock+0x21/0x30 [18624.353440] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18624.358872] process_one_work+0x2c8/0x700 [18624.365933] worker_thread+0x296/0x6e0 [18624.367499] ? rescuer_thread+0x570/0x570 [18624.373438] kthread+0x1d1/0x200 [18624.377761] ? set_kthread_struct+0x70/0x70 [18624.383656] ret_from_fork+0x1f/0x30 [18628.575891] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 15991s! [18628.580631] Showing busy workqueues and worker pools: [18628.585428] workqueue events: flags=0x0 [18628.593587] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18628.598454] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18628.609218] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18628.613555] pending: cpu_latency_work [ptlrpc] [18628.615710] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18628.624533] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18628.641565] workqueue events_long: flags=0x0 [18628.644089] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18628.651619] pending: lu_objects_flush [obdclass] [18628.654229] workqueue events_unbound: flags=0x2 [18628.655948] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18628.658825] pending: flush_memcg_stats_dwork, idle_cull_fn [18628.668776] workqueue events_power_efficient: flags=0x80 [18628.673636] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18628.677669] pending: check_lifetime, do_cache_clean [18628.680372] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18628.683822] pending: neigh_periodic_work, neigh_periodic_work [18628.687086] workqueue mm_percpu_wq: flags=0x8 [18628.688546] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18628.695050] pending: vmstat_update, lru_add_drain_per_cpu [18628.696935] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18628.711016] pending: vmstat_update, lru_add_drain_per_cpu [18628.724725] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18628.727030] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18628.741266] workqueue xprtiod: flags=0xa [18628.746850] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18628.748993] pending: xprt_autoclose, xs_error_handle [18628.753113] workqueue ipv6_addrconf: flags=0x40008 [18628.758785] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18628.762911] pending: addrconf_verify_work [18628.764636] workqueue knbd0-recv: flags=0x1a [18628.773600] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18628.786490] in-flight: 651:recv_work [18628.792077] workqueue ptlrpc_pinger: flags=0x2 [18628.796450] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18628.804865] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18628.809472] pool 8: cpus=0-3 flags=0x4 nice=0 hung=15991s workers=4 idle: 15979 31824 8 [18628.822500] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18619s workers=2 idle: 654 [18632.058135] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [18632.062962] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18632.113881] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18632.120800] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18632.125069] RIP: 0010:native_safe_halt+0x2e/0x50 [18632.127715] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18632.137393] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18632.142596] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18632.149779] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18632.158012] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18632.166536] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18632.173228] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18632.175641] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18632.180791] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18632.187239] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18632.194460] Call Trace: [18632.195388] [18632.196138] ? show_regs.cold.9+0x22/0x2f [18632.198664] ? watchdog_timer_fn.cold.6+0x62/0x10a [18632.201498] ? __hrtimer_run_queues+0x1c2/0x4d0 [18632.203627] ? watchdog+0x40/0x40 [18632.205174] ? hrtimer_interrupt+0x128/0x2c0 [18632.208331] ? smp_apic_timer_interrupt+0xa6/0x270 [18632.211323] ? apic_timer_interrupt+0xf/0x20 [18632.213994] [18632.217168] ? native_safe_halt+0x2e/0x50 [18632.219278] kvm_wait+0xcd/0xe0 [18632.221976] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18632.223971] do_raw_spin_lock+0x177/0x190 [18632.225360] _raw_spin_lock+0x21/0x30 [18632.226819] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18632.229118] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18632.231813] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18632.237976] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18632.241523] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18632.246280] ? woken_wake_function+0x30/0x30 [18632.250121] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18632.254458] kthread+0x1d1/0x200 [18632.255413] ? set_kthread_struct+0x70/0x70 [18632.257473] ret_from_fork+0x1f/0x30 [18644.059584] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18644.062995] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18644.071306] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18644.071501] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18644.071509] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18644.071512] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18644.071526] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18644.071531] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18644.071536] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18644.071538] RDX: 0000000000001032 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18644.071541] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18644.071543] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18644.071546] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18644.071549] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18644.071552] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18644.071554] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18644.071563] Call Trace: [18644.071568] [18644.071574] ? show_regs.cold.9+0x22/0x2f [18644.071584] ? watchdog_timer_fn.cold.6+0x62/0x10a [18644.071591] ? __hrtimer_run_queues+0x1c2/0x4d0 [18644.071594] ? watchdog+0x40/0x40 [18644.071599] ? hrtimer_interrupt+0x128/0x2c0 [18644.071604] ? smp_apic_timer_interrupt+0xa6/0x270 [18644.071612] ? apic_timer_interrupt+0xf/0x20 [18644.071618] [18644.071621] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18644.071626] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18644.071631] do_raw_spin_lock+0x177/0x190 [18644.071635] _raw_spin_lock+0x21/0x30 [18644.071641] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18644.082316] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel [18644.125121] obd_mod_free+0x46/0x140 [ptlrpc] [18644.125535] mdc_commit_open+0x91/0xb0 [mdc] [18644.125672] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18644.126066] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18644.126276] after_reply+0xe2c/0x1780 [ptlrpc] [18644.127282] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18644.130375] ? __next_timer_interrupt+0x160/0x160 [18644.130387] ptlrpcd+0xa82/0xe40 [ptlrpc] [18644.130652] ? do_wait_intr+0xf0/0xf0 [18644.130662] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18644.131191] kthread+0x1d1/0x200 [18644.131198] ? set_kthread_struct+0x70/0x70 [18644.131202] ret_from_fork+0x1f/0x30 [18644.406105] ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18644.418262] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18644.433554] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18644.442672] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18644.448191] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18644.471050] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18644.479997] RAX: 0000000000005982 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18644.488048] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18644.491145] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18644.494462] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18644.500960] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18644.509587] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18644.519591] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18644.521779] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18644.527988] Call Trace: [18644.529384] [18644.533303] ? show_regs.cold.9+0x22/0x2f [18644.537993] ? watchdog_timer_fn.cold.6+0x62/0x10a [18644.543838] ? __hrtimer_run_queues+0x1c2/0x4d0 [18644.546828] ? watchdog+0x40/0x40 [18644.549832] ? hrtimer_interrupt+0x128/0x2c0 [18644.552090] ? smp_apic_timer_interrupt+0xa6/0x270 [18644.556627] ? apic_timer_interrupt+0xf/0x20 [18644.558022] [18644.558692] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18644.568183] do_raw_spin_lock+0x177/0x190 [18644.573864] _raw_spin_lock+0x21/0x30 [18644.576038] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18644.584901] ? do_raw_spin_unlock+0x75/0x190 [18644.587653] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18644.596531] ? do_wait_intr+0xf0/0xf0 [18644.600476] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18644.602595] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18644.604345] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18644.606630] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18644.610529] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18644.612902] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18644.614549] mdc_intent_lock+0x353/0x7e0 [mdc] [18644.616352] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18644.618614] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18644.621036] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18644.623355] lmv_intent_open+0x338/0x1150 [lmv] [18644.626693] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18644.628588] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18644.632379] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18644.634785] ll_intent_lock+0xc5/0xaa0 [lustre] [18644.637691] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18644.639806] ll_lookup_it+0xe0e/0x2270 [lustre] [18644.641118] ? do_raw_spin_unlock+0x75/0x190 [18644.644388] ? _raw_spin_unlock+0x12/0x30 [18644.646526] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18644.648795] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18644.651184] ll_atomic_open+0x36a/0xd70 [lustre] [18644.654674] lookup_open+0x356/0xb40 [18644.658865] do_last+0x313/0xfc0 [18644.660477] ? nd_jump_root+0xe5/0x160 [18644.662282] ? path_init+0x437/0x520 [18644.663640] path_openat+0xf7/0x500 [18644.665285] ? do_raw_spin_unlock+0x75/0x190 [18644.666855] do_filp_open+0x99/0x140 [18644.668879] ? getname_flags+0x6e/0x330 [18644.672693] ? __check_object_size+0xff/0x256 [18644.675485] ? do_raw_spin_unlock+0x75/0x190 [18644.681634] ? _raw_spin_unlock+0x12/0x30 [18644.686170] do_sys_openat2+0x2b4/0x410 [18644.687556] do_sys_open+0x73/0xa0 [18644.691187] __x64_sys_openat+0x24/0x30 [18644.694120] do_syscall_64+0xc1/0x440 [18644.699351] entry_SYSCALL_64_after_hwframe+0x49/0xae [18644.701205] RIP: 0033:0x7f53425a5332 [18644.705134] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18644.721885] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18644.731206] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18644.739052] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18644.744512] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18644.747523] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18644.750532] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18656.052127] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [18656.055840] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18656.083652] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18656.092410] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18656.104137] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18656.108171] RIP: 0010:native_safe_halt+0x2e/0x50 [18656.111641] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18656.119051] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18656.121628] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18656.124930] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [18656.127830] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18656.130622] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18656.133592] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18656.136472] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18656.139705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18656.141936] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18656.145842] Call Trace: [18656.147094] [18656.148131] ? show_regs.cold.9+0x22/0x2f [18656.150092] ? watchdog_timer_fn.cold.6+0x62/0x10a [18656.152253] ? __hrtimer_run_queues+0x1c2/0x4d0 [18656.154707] ? watchdog+0x40/0x40 [18656.156005] ? hrtimer_interrupt+0x128/0x2c0 [18656.157495] ? smp_apic_timer_interrupt+0xa6/0x270 [18656.159598] ? apic_timer_interrupt+0xf/0x20 [18656.161482] [18656.162212] ? native_safe_halt+0x2e/0x50 [18656.163568] kvm_wait+0xcd/0xe0 [18656.165226] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18656.173866] ? newidle_balance.isra.93+0x207/0x580 [18656.182706] do_raw_spin_lock+0x177/0x190 [18656.185443] _raw_spin_lock+0x21/0x30 [18656.188182] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18656.193806] process_one_work+0x2c8/0x700 [18656.196241] worker_thread+0x296/0x6e0 [18656.198619] ? rescuer_thread+0x570/0x570 [18656.201110] kthread+0x1d1/0x200 [18656.206893] ? set_kthread_struct+0x70/0x70 [18656.209421] ret_from_fork+0x1f/0x30 [18659.296220] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16022s! [18659.299914] Showing busy workqueues and worker pools: [18659.301641] workqueue events: flags=0x0 [18659.302896] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18659.304902] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18659.308895] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18659.310884] pending: cpu_latency_work [ptlrpc] [18659.312442] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18659.314682] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18659.318475] workqueue events_long: flags=0x0 [18659.319453] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18659.321295] pending: lu_objects_flush [obdclass] [18659.322923] workqueue events_unbound: flags=0x2 [18659.324190] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18659.328040] pending: flush_memcg_stats_dwork, idle_cull_fn [18659.333572] workqueue events_power_efficient: flags=0x80 [18659.335451] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18659.339077] pending: check_lifetime, do_cache_clean [18659.341766] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18659.344477] pending: neigh_periodic_work, neigh_periodic_work [18659.347113] workqueue mm_percpu_wq: flags=0x8 [18659.348705] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18659.351644] pending: vmstat_update, lru_add_drain_per_cpu [18659.353738] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18659.358868] pending: vmstat_update, lru_add_drain_per_cpu [18659.360307] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18659.365088] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18659.371168] workqueue xprtiod: flags=0xa [18659.373092] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18659.376036] pending: xprt_autoclose, xs_error_handle [18659.379371] workqueue ipv6_addrconf: flags=0x40008 [18659.381696] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18659.384578] pending: addrconf_verify_work [18659.386975] workqueue knbd0-recv: flags=0x1a [18659.389770] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18659.391428] in-flight: 651:recv_work [18659.402536] workqueue ptlrpc_pinger: flags=0x2 [18659.403791] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18659.407934] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18659.413065] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16022s workers=4 idle: 15979 31824 8 [18659.420421] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18649s workers=2 idle: 654 [18660.058976] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [18660.062909] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18660.104027] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18660.111031] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18660.116030] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18660.119983] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18660.131748] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [18660.135565] RAX: 0000000000007805 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18660.142413] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18660.148228] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18660.153989] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18660.158017] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18660.167168] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18660.177109] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18660.181191] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18660.185212] Call Trace: [18660.186839] [18660.188231] ? show_regs.cold.9+0x22/0x2f [18660.193142] ? watchdog_timer_fn.cold.6+0x62/0x10a [18660.197843] ? __hrtimer_run_queues+0x1c2/0x4d0 [18660.200209] ? watchdog+0x40/0x40 [18660.204060] ? hrtimer_interrupt+0x128/0x2c0 [18660.207996] ? smp_apic_timer_interrupt+0xa6/0x270 [18660.210984] ? apic_timer_interrupt+0xf/0x20 [18660.214131] [18660.217547] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18660.220508] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18660.223013] do_raw_spin_lock+0x177/0x190 [18660.226014] _raw_spin_lock+0x21/0x30 [18660.227827] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18660.230979] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18660.233834] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18660.237752] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18660.240081] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18660.242014] ? woken_wake_function+0x30/0x30 [18660.243598] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18660.245286] kthread+0x1d1/0x200 [18660.246885] ? set_kthread_struct+0x70/0x70 [18660.248249] ret_from_fork+0x1f/0x30 [18672.057083] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18672.057956] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18672.061135] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18672.061221] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18672.061229] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18672.061231] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18672.061245] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18672.061249] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18672.061254] RAX: 0000000000003d89 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18672.061257] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18672.061259] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18672.061262] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18672.061266] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18672.061269] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18672.061272] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18672.061275] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18672.061284] Call Trace: [18672.061290] [18672.061294] ? show_regs.cold.9+0x22/0x2f [18672.061303] ? watchdog_timer_fn.cold.6+0x62/0x10a [18672.061309] ? __hrtimer_run_queues+0x1c2/0x4d0 [18672.061313] ? watchdog+0x40/0x40 [18672.061317] ? hrtimer_interrupt+0x128/0x2c0 [18672.061321] ? smp_apic_timer_interrupt+0xa6/0x270 [18672.061329] ? apic_timer_interrupt+0xf/0x20 [18672.061334] [18672.061338] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18672.061345] do_raw_spin_lock+0x177/0x190 [18672.061410] _raw_spin_lock+0x21/0x30 [18672.061416] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18672.061840] ? do_raw_spin_unlock+0x75/0x190 [18672.061846] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18672.062057] ? do_wait_intr+0xf0/0xf0 [18672.062065] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18672.062270] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18672.062501] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18672.063038] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18672.063146] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18672.063181] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18672.063208] mdc_intent_lock+0x353/0x7e0 [mdc] [18672.063234] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18672.075027] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18672.075098] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18672.075104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18672.075107] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18672.075119] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18672.075123] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18672.075127] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18672.075129] RDX: 0000000000006fd1 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18672.075131] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18672.075133] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18672.075135] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18672.075138] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18672.075140] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18672.075143] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18672.075150] Call Trace: [18672.075155] [18672.075160] ? show_regs.cold.9+0x22/0x2f [18672.075167] ? watchdog_timer_fn.cold.6+0x62/0x10a [18672.075172] ? __hrtimer_run_queues+0x1c2/0x4d0 [18672.075175] ? watchdog+0x40/0x40 [18672.075179] ? hrtimer_interrupt+0x128/0x2c0 [18672.075182] ? smp_apic_timer_interrupt+0xa6/0x270 [18672.075189] ? apic_timer_interrupt+0xf/0x20 [18672.075193] [18672.075195] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18672.075199] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18672.075203] do_raw_spin_lock+0x177/0x190 [18672.075207] _raw_spin_lock+0x21/0x30 [18672.075211] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18672.075482] obd_mod_free+0x46/0x140 [ptlrpc] [18672.076886] mdc_commit_open+0x91/0xb0 [mdc] [18672.076926] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18672.077823] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18672.078065] after_reply+0xe2c/0x1780 [ptlrpc] [18672.078258] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18672.078455] ? __next_timer_interrupt+0x160/0x160 [18672.078465] ptlrpcd+0xa82/0xe40 [ptlrpc] [18672.112365] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18672.119166] ? do_wait_intr+0xf0/0xf0 [18672.119189] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18672.119812] kthread+0x1d1/0x200 [18672.119820] ? set_kthread_struct+0x70/0x70 [18672.119899] ret_from_fork+0x1f/0x30 [18672.600831] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18672.606640] lmv_intent_open+0x338/0x1150 [lmv] [18672.615819] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18672.626471] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18672.642191] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18672.655098] ll_intent_lock+0xc5/0xaa0 [lustre] [18672.660798] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18672.669323] ll_lookup_it+0xe0e/0x2270 [lustre] [18672.675902] ? do_raw_spin_unlock+0x75/0x190 [18672.684024] ? _raw_spin_unlock+0x12/0x30 [18672.690295] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18672.699653] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18672.703608] ll_atomic_open+0x36a/0xd70 [lustre] [18672.712297] lookup_open+0x356/0xb40 [18672.719160] do_last+0x313/0xfc0 [18672.723117] ? nd_jump_root+0xe5/0x160 [18672.724465] ? path_init+0x437/0x520 [18672.730211] path_openat+0xf7/0x500 [18672.739473] ? do_raw_spin_unlock+0x75/0x190 [18672.744529] do_filp_open+0x99/0x140 [18672.745751] ? getname_flags+0x6e/0x330 [18672.760447] ? __check_object_size+0xff/0x256 [18672.765753] ? do_raw_spin_unlock+0x75/0x190 [18672.773281] ? _raw_spin_unlock+0x12/0x30 [18672.784404] do_sys_openat2+0x2b4/0x410 [18672.791402] do_sys_open+0x73/0xa0 [18672.796844] __x64_sys_openat+0x24/0x30 [18672.800715] do_syscall_64+0xc1/0x440 [18672.810955] entry_SYSCALL_64_after_hwframe+0x49/0xae [18672.819057] RIP: 0033:0x7f53425a5332 [18672.826849] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18672.850010] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18672.861083] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18672.870936] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18672.881417] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18672.890658] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18672.901449] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18684.052661] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [18684.055844] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18684.123047] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18684.153115] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18684.170648] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18684.189763] RIP: 0010:native_safe_halt+0x2e/0x50 [18684.208489] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18684.245626] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [18684.263958] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18684.277131] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [18684.284984] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18684.292863] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18684.308533] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18684.316988] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18684.327625] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18684.333795] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18684.343968] Call Trace: [18684.345712] [18684.346986] ? show_regs.cold.9+0x22/0x2f [18684.353162] ? watchdog_timer_fn.cold.6+0x62/0x10a [18684.358882] ? __hrtimer_run_queues+0x1c2/0x4d0 [18684.365848] ? watchdog+0x40/0x40 [18684.367162] ? hrtimer_interrupt+0x128/0x2c0 [18684.370657] ? smp_apic_timer_interrupt+0xa6/0x270 [18684.373532] ? apic_timer_interrupt+0xf/0x20 [18684.375574] [18684.376588] ? native_safe_halt+0x2e/0x50 [18684.379707] kvm_wait+0xcd/0xe0 [18684.386579] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18684.391399] ? newidle_balance.isra.93+0x207/0x580 [18684.398670] do_raw_spin_lock+0x177/0x190 [18684.400969] _raw_spin_lock+0x21/0x30 [18684.404253] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18684.409238] process_one_work+0x2c8/0x700 [18684.411773] worker_thread+0x296/0x6e0 [18684.414155] ? rescuer_thread+0x570/0x570 [18684.417441] kthread+0x1d1/0x200 [18684.418581] ? set_kthread_struct+0x70/0x70 [18684.420626] ret_from_fork+0x1f/0x30 [18688.058354] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [18688.062980] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18688.087165] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18688.091939] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18688.102473] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18688.106800] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18688.116942] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18688.120737] RAX: 0000000000004e17 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18688.124677] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18688.128842] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18688.131758] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18688.135142] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18688.137935] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18688.141180] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18688.143357] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18688.146002] Call Trace: [18688.146956] [18688.149052] ? show_regs.cold.9+0x22/0x2f [18688.150733] ? watchdog_timer_fn.cold.6+0x62/0x10a [18688.153313] ? __hrtimer_run_queues+0x1c2/0x4d0 [18688.156839] ? watchdog+0x40/0x40 [18688.158314] ? hrtimer_interrupt+0x128/0x2c0 [18688.160007] ? smp_apic_timer_interrupt+0xa6/0x270 [18688.164117] ? apic_timer_interrupt+0xf/0x20 [18688.167004] [18688.168839] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18688.171866] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18688.175873] do_raw_spin_lock+0x177/0x190 [18688.177891] _raw_spin_lock+0x21/0x30 [18688.179736] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18688.182175] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18688.186012] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18688.188864] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18688.191385] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18688.194197] ? woken_wake_function+0x30/0x30 [18688.195934] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18688.199801] kthread+0x1d1/0x200 [18688.202683] ? set_kthread_struct+0x70/0x70 [18688.204374] ret_from_fork+0x1f/0x30 [18689.506027] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16052s! [18689.511315] Showing busy workqueues and worker pools: [18689.513928] workqueue events: flags=0x0 [18689.515691] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18689.519094] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18689.525698] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18689.527870] pending: cpu_latency_work [ptlrpc] [18689.530811] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18689.533926] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18689.538098] workqueue events_long: flags=0x0 [18689.540132] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18689.543632] pending: lu_objects_flush [obdclass] [18689.546618] workqueue events_unbound: flags=0x2 [18689.548761] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18689.551787] pending: flush_memcg_stats_dwork, idle_cull_fn [18689.554926] workqueue events_power_efficient: flags=0x80 [18689.557997] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18689.561469] pending: check_lifetime, do_cache_clean [18689.568700] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18689.573957] pending: neigh_periodic_work, neigh_periodic_work [18689.577156] workqueue mm_percpu_wq: flags=0x8 [18689.579565] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18689.582050] pending: vmstat_update, lru_add_drain_per_cpu [18689.584517] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18689.587255] pending: vmstat_update, lru_add_drain_per_cpu [18689.589893] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18689.592775] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18689.595794] workqueue xprtiod: flags=0xa [18689.597780] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18689.600809] pending: xprt_autoclose, xs_error_handle [18689.603805] workqueue ipv6_addrconf: flags=0x40008 [18689.606393] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18689.609347] pending: addrconf_verify_work [18689.611495] workqueue knbd0-recv: flags=0x1a [18689.613779] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18689.617576] in-flight: 651:recv_work [18689.619316] workqueue ptlrpc_pinger: flags=0x2 [18689.621889] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18689.624932] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18689.630106] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16052s workers=4 idle: 15979 31824 8 [18689.634870] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18680s workers=2 idle: 654 [18700.055036] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18700.057030] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18700.057535] Modules linked in: [18700.059643] Modules linked in: [18700.059645] lustre(O) [18700.060817] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18700.060892] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18700.060899] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18700.060903] RIP: 0010:native_safe_halt+0x2e/0x50 [18700.063016] osp(O) [18700.064124] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18700.064134] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18700.064139] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18700.064142] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18700.064144] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18700.064146] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18700.064148] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18700.064151] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18700.064154] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18700.064156] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18700.064164] Call Trace: [18700.064170] [18700.064176] ? show_regs.cold.9+0x22/0x2f [18700.064189] ? watchdog_timer_fn.cold.6+0x62/0x10a [18700.064195] ? __hrtimer_run_queues+0x1c2/0x4d0 [18700.064200] ? watchdog+0x40/0x40 [18700.064204] ? hrtimer_interrupt+0x128/0x2c0 [18700.064209] ? smp_apic_timer_interrupt+0xa6/0x270 [18700.064216] ? apic_timer_interrupt+0xf/0x20 [18700.064221] [18700.064224] ? native_safe_halt+0x2e/0x50 [18700.064228] kvm_wait+0xcd/0xe0 [18700.064236] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18700.064244] do_raw_spin_lock+0x177/0x190 [18700.064249] _raw_spin_lock+0x21/0x30 [18700.064255] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18700.064739] obd_mod_free+0x46/0x140 [ptlrpc] [18700.098692] ofd(O) [18700.103523] mdc_commit_open+0x91/0xb0 [mdc] [18700.110680] lod(O) [18700.112992] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18700.113263] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18700.113478] after_reply+0xe2c/0x1780 [ptlrpc] [18700.113689] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18700.115129] mdt(O) [18700.122452] ? __next_timer_interrupt+0x160/0x160 [18700.122475] ptlrpcd+0xa82/0xe40 [ptlrpc] [18700.122765] ? do_wait_intr+0xf0/0xf0 [18700.122776] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18700.122981] kthread+0x1d1/0x200 [18700.122987] ? set_kthread_struct+0x70/0x70 [18700.122991] ret_from_fork+0x1f/0x30 [18700.296238] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18700.325056] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18700.335351] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18700.343429] RIP: 0010:native_safe_halt+0x2e/0x50 [18700.345286] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18700.357640] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18700.362834] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18700.371664] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18700.377346] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18700.381748] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18700.383928] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18700.388363] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18700.393026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18700.394744] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18700.396880] Call Trace: [18700.397715] [18700.398333] ? show_regs.cold.9+0x22/0x2f [18700.399622] ? watchdog_timer_fn.cold.6+0x62/0x10a [18700.400901] ? __hrtimer_run_queues+0x1c2/0x4d0 [18700.402434] ? watchdog+0x40/0x40 [18700.403765] ? hrtimer_interrupt+0x128/0x2c0 [18700.405660] ? smp_apic_timer_interrupt+0xa6/0x270 [18700.407642] ? apic_timer_interrupt+0xf/0x20 [18700.409402] [18700.410311] ? native_safe_halt+0x2e/0x50 [18700.411956] kvm_wait+0xcd/0xe0 [18700.413297] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18700.415346] do_raw_spin_lock+0x177/0x190 [18700.417058] _raw_spin_lock+0x21/0x30 [18700.418625] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18700.421409] ? do_raw_spin_unlock+0x75/0x190 [18700.423823] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18700.426409] ? do_wait_intr+0xf0/0xf0 [18700.427938] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18700.431114] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18700.433517] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18700.436196] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18700.438877] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18700.441697] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18700.443717] mdc_intent_lock+0x353/0x7e0 [mdc] [18700.446470] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18700.450313] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18700.454569] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18700.456473] lmv_intent_open+0x338/0x1150 [lmv] [18700.457799] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18700.460376] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18700.461908] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18700.465412] ll_intent_lock+0xc5/0xaa0 [lustre] [18700.467431] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18700.471459] ll_lookup_it+0xe0e/0x2270 [lustre] [18700.475500] ? do_raw_spin_unlock+0x75/0x190 [18700.480383] ? _raw_spin_unlock+0x12/0x30 [18700.482061] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18700.485266] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18700.487677] ll_atomic_open+0x36a/0xd70 [lustre] [18700.490388] lookup_open+0x356/0xb40 [18700.492185] do_last+0x313/0xfc0 [18700.493427] ? nd_jump_root+0xe5/0x160 [18700.494871] ? path_init+0x437/0x520 [18700.499273] path_openat+0xf7/0x500 [18700.500975] ? do_raw_spin_unlock+0x75/0x190 [18700.503190] do_filp_open+0x99/0x140 [18700.504462] ? getname_flags+0x6e/0x330 [18700.508128] ? __check_object_size+0xff/0x256 [18700.510646] ? do_raw_spin_unlock+0x75/0x190 [18700.512360] ? _raw_spin_unlock+0x12/0x30 [18700.515766] do_sys_openat2+0x2b4/0x410 [18700.520184] do_sys_open+0x73/0xa0 [18700.522246] __x64_sys_openat+0x24/0x30 [18700.523667] do_syscall_64+0xc1/0x440 [18700.525777] entry_SYSCALL_64_after_hwframe+0x49/0xae [18700.531657] RIP: 0033:0x7f53425a5332 [18700.533486] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18700.554385] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18700.558359] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18700.566884] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18700.574755] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18700.581590] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18700.585205] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18714.838064] rcu: INFO: rcu_sched self-detected stall on CPU [18714.839762] rcu: 2-....: (9314641 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [18714.843073] (t=16620412 jiffies g=274185 q=68179) [18714.844592] rcu: rcu_sched kthread starved for 16078778 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [18714.848200] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [18714.852188] rcu: RCU grace-period kthread stack dump: [18714.854725] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [18714.859593] Call Trace: [18714.861115] __schedule+0x351/0xcb0 [18714.862718] schedule+0xc0/0x180 [18714.864579] schedule_timeout+0xb4/0x190 [18714.866589] ? do_raw_spin_unlock+0x75/0x190 [18714.868180] ? __next_timer_interrupt+0x160/0x160 [18714.870259] ? prepare_to_swait_event+0xf3/0x140 [18714.872283] rcu_gp_kthread+0x8dd/0x15b0 [18714.874294] ? force_qs_rnp+0x2c0/0x2c0 [18714.876055] kthread+0x1d1/0x200 [18714.877594] ? set_kthread_struct+0x70/0x70 [18714.879534] ret_from_fork+0x1f/0x30 [18714.881471] rcu: Stack dump where RCU GP kthread last ran: [18714.885503] Sending NMI from CPU 2 to CPUs 3: [18714.894119] NMI backtrace for cpu 3 [18714.894128] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18714.894130] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18714.894132] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18714.894133] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18714.894135] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 [18714.894138] RAX: 0000000000002e08 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18714.894139] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18714.894140] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18714.894142] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18714.894143] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18714.894144] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18714.894145] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18714.894146] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18714.894147] Call Trace: [18714.894148] [18714.894149] ? show_regs.cold.9+0x22/0x2f [18714.894150] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [18714.894151] ? nmi_cpu_backtrace_handler+0x15/0x40 [18714.894152] ? nmi_handle+0xbd/0x210 [18714.894153] ? default_do_nmi+0x4b/0x1b0 [18714.894153] ? do_nmi+0x280/0x380 [18714.894154] ? end_repeat_nmi+0x16/0x1f [18714.894155] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18714.894156] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18714.894157] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18714.894158] [18714.894159] do_raw_spin_lock+0x177/0x190 [18714.894160] _raw_spin_lock+0x21/0x30 [18714.894161] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18714.894162] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18714.894164] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18714.894164] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18714.894166] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18714.894166] ? woken_wake_function+0x30/0x30 [18714.894168] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18714.894169] kthread+0x1d1/0x200 [18714.894169] ? set_kthread_struct+0x70/0x70 [18714.894171] ret_from_fork+0x1f/0x30 [18714.894910] Sending NMI from CPU 2 to CPUs 1: [18715.001486] NMI backtrace for cpu 1 [18715.001492] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18715.001494] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18715.001496] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18715.001498] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18715.001500] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [18715.001503] RAX: 00000000000037d9 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18715.001504] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18715.001506] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18715.001508] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18715.001509] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18715.001511] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18715.001512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18715.001514] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18715.001515] Call Trace: [18715.001516] [18715.001517] ? show_regs.cold.9+0x22/0x2f [18715.001518] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [18715.001519] ? nmi_cpu_backtrace_handler+0x15/0x40 [18715.001520] ? nmi_handle+0xbd/0x210 [18715.001521] ? default_do_nmi+0x4b/0x1b0 [18715.001522] ? do_nmi+0x280/0x380 [18715.001523] ? end_repeat_nmi+0x16/0x1f [18715.001524] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18715.001526] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18715.001527] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18715.001528] [18715.001529] do_raw_spin_lock+0x177/0x190 [18715.001530] _raw_spin_lock+0x21/0x30 [18715.001531] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18715.001532] ? do_raw_spin_unlock+0x75/0x190 [18715.001533] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18715.001534] ? do_wait_intr+0xf0/0xf0 [18715.001535] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18715.001536] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18715.001537] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18715.001539] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18715.001540] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18715.001541] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18715.001542] mdc_intent_lock+0x353/0x7e0 [mdc] [18715.001543] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18715.001544] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18715.001545] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18715.001547] lmv_intent_open+0x338/0x1150 [lmv] [18715.001548] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18715.001550] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18715.001551] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18715.001552] ll_intent_lock+0xc5/0xaa0 [lustre] [18715.001553] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18715.001555] ll_lookup_it+0xe0e/0x2270 [lustre] [18715.001556] ? do_raw_spin_unlock+0x75/0x190 [18715.001557] ? _raw_spin_unlock+0x12/0x30 [18715.001558] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18715.001559] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18715.001560] ll_atomic_open+0x36a/0xd70 [lustre] [18715.001561] lookup_open+0x356/0xb40 [18715.001562] do_last+0x313/0xfc0 [18715.001863] ? nd_jump_root+0xe5/0x160 [18715.001866] ? path_init+0x437/0x520 [18715.001867] path_openat+0xf7/0x500 [18715.001868] ? do_raw_spin_unlock+0x75/0x190 [18715.001869] do_filp_open+0x99/0x140 [18715.001870] ? getname_flags+0x6e/0x330 [18715.001872] ? __check_object_size+0xff/0x256 [18715.001873] ? do_raw_spin_unlock+0x75/0x190 [18715.001874] ? _raw_spin_unlock+0x12/0x30 [18715.001875] do_sys_openat2+0x2b4/0x410 [18715.001876] do_sys_open+0x73/0xa0 [18715.001877] __x64_sys_openat+0x24/0x30 [18715.001879] do_syscall_64+0xc1/0x440 [18715.001880] entry_SYSCALL_64_after_hwframe+0x49/0xae [18715.001881] RIP: 0033:0x7f53425a5332 [18715.001882] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18715.001884] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18715.001887] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18715.001888] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18715.001890] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18715.001891] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18715.001892] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18715.003076] NMI backtrace for cpu 2 [18715.191854] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18715.195015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18715.197372] Call Trace: [18715.197946] [18715.198446] ? dump_stack+0xbb/0x10e [18715.199379] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [18715.201186] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [18715.202599] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [18715.204527] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [18715.205881] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [18715.207297] ? rcu_dump_cpu_stacks+0x11b/0x171 [18715.208453] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [18715.209901] ? update_process_times+0x6d/0xc0 [18715.211113] ? tick_sched_handle.isra.18+0x27/0x90 [18715.212522] ? tick_sched_timer+0x53/0xb0 [18715.213657] ? __hrtimer_run_queues+0x1c2/0x4d0 [18715.215003] ? tick_sched_handle.isra.18+0x90/0x90 [18715.216373] ? hrtimer_interrupt+0x128/0x2c0 [18715.217642] ? smp_apic_timer_interrupt+0xa6/0x270 [18715.219263] ? apic_timer_interrupt+0xf/0x20 [18715.220674] [18715.221381] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18715.222852] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18715.224465] ? do_raw_spin_lock+0x177/0x190 [18715.225559] ? _raw_spin_lock+0x21/0x30 [18715.226595] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18715.227939] ? obd_mod_free+0x46/0x140 [ptlrpc] [18715.229467] ? mdc_commit_open+0x91/0xb0 [mdc] [18715.230688] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18715.232500] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18715.234308] ? after_reply+0xe2c/0x1780 [ptlrpc] [18715.236087] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18715.237653] ? __next_timer_interrupt+0x160/0x160 [18715.238939] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [18715.240357] ? do_wait_intr+0xf0/0xf0 [18715.241300] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18715.243075] ? kthread+0x1d1/0x200 [18715.244056] ? set_kthread_struct+0x70/0x70 [18715.245312] ? ret_from_fork+0x1f/0x30 [18716.052027] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [18716.056169] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18716.084924] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18716.090894] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18716.094709] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18716.096911] RIP: 0010:native_safe_halt+0x2e/0x50 [18716.098293] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18716.107163] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18716.111099] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18716.115482] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [18716.120104] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18716.123169] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18716.126077] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18716.129856] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18716.140096] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18716.144003] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18716.147261] Call Trace: [18716.148619] [18716.149495] ? show_regs.cold.9+0x22/0x2f [18716.153053] ? watchdog_timer_fn.cold.6+0x62/0x10a [18716.156172] ? __hrtimer_run_queues+0x1c2/0x4d0 [18716.158504] ? watchdog+0x40/0x40 [18716.159498] ? hrtimer_interrupt+0x128/0x2c0 [18716.161840] ? smp_apic_timer_interrupt+0xa6/0x270 [18716.164053] ? apic_timer_interrupt+0xf/0x20 [18716.167053] [18716.168225] ? native_safe_halt+0x2e/0x50 [18716.171241] kvm_wait+0xcd/0xe0 [18716.173038] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18716.180086] ? newidle_balance.isra.93+0x207/0x580 [18716.189695] do_raw_spin_lock+0x177/0x190 [18716.194766] _raw_spin_lock+0x21/0x30 [18716.199306] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18716.204635] process_one_work+0x2c8/0x700 [18716.206871] worker_thread+0x296/0x6e0 [18716.208661] ? rescuer_thread+0x570/0x570 [18716.210876] kthread+0x1d1/0x200 [18716.212886] ? set_kthread_struct+0x70/0x70 [18716.215067] ret_from_fork+0x1f/0x30 [18719.712108] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16082s! [18719.715091] Showing busy workqueues and worker pools: [18719.716384] workqueue events: flags=0x0 [18719.717327] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18719.720552] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18719.725414] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18719.728180] pending: cpu_latency_work [ptlrpc] [18719.731054] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18719.735158] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18719.739518] workqueue events_long: flags=0x0 [18719.741879] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18719.745296] pending: lu_objects_flush [obdclass] [18719.747693] workqueue events_unbound: flags=0x2 [18719.749582] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18719.752239] pending: flush_memcg_stats_dwork, idle_cull_fn [18719.754982] workqueue events_power_efficient: flags=0x80 [18719.757266] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18719.760899] pending: check_lifetime, do_cache_clean [18719.763023] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18719.766018] pending: neigh_periodic_work, neigh_periodic_work [18719.769070] workqueue mm_percpu_wq: flags=0x8 [18719.770896] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18719.773914] pending: vmstat_update, lru_add_drain_per_cpu [18719.776606] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18719.779986] pending: vmstat_update, lru_add_drain_per_cpu [18719.783128] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18719.785624] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18719.788945] workqueue xprtiod: flags=0xa [18719.791113] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18719.794950] pending: xprt_autoclose, xs_error_handle [18719.797327] workqueue ipv6_addrconf: flags=0x40008 [18719.799250] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18719.802184] pending: addrconf_verify_work [18719.803948] workqueue knbd0-recv: flags=0x1a [18719.805948] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18719.807917] in-flight: 651:recv_work [18719.809324] workqueue ptlrpc_pinger: flags=0x2 [18719.811027] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18719.813792] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18719.816431] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16082s workers=4 idle: 15979 31824 8 [18719.820995] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18710s workers=2 idle: 654 [18740.055030] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [18740.059100] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18740.060319] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [18740.060280] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [18740.060345] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18740.060412] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18740.060418] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18740.060421] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18740.060434] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18740.060439] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18740.060443] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18740.060445] RDX: 000000000000066c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18740.060448] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18740.060450] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18740.060452] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18740.060455] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18740.060459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18740.060461] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18740.060475] Call Trace: [18740.060481] [18740.060492] ? show_regs.cold.9+0x22/0x2f [18740.060502] ? watchdog_timer_fn.cold.6+0x62/0x10a [18740.060508] ? __hrtimer_run_queues+0x1c2/0x4d0 [18740.060513] ? watchdog+0x40/0x40 [18740.060518] ? hrtimer_interrupt+0x128/0x2c0 [18740.060523] ? smp_apic_timer_interrupt+0xa6/0x270 [18740.060531] ? apic_timer_interrupt+0xf/0x20 [18740.060542] [18740.060545] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18740.060551] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18740.060556] do_raw_spin_lock+0x177/0x190 [18740.060562] _raw_spin_lock+0x21/0x30 [18740.060569] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18740.061118] obd_mod_free+0x46/0x140 [ptlrpc] [18740.061410] mdc_commit_open+0x91/0xb0 [mdc] [18740.061454] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18740.061661] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18740.062080] after_reply+0xe2c/0x1780 [ptlrpc] [18740.062390] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18740.062610] ? __next_timer_interrupt+0x160/0x160 [18740.062621] ptlrpcd+0xa82/0xe40 [ptlrpc] [18740.062826] ? do_wait_intr+0xf0/0xf0 [18740.065372] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18740.065655] kthread+0x1d1/0x200 [18740.065664] ? set_kthread_struct+0x70/0x70 [18740.065668] ret_from_fork+0x1f/0x30 [18740.081651] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18740.081665] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18740.081673] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18740.081688] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18740.081692] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18740.081698] RAX: 0000000000001819 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18740.081700] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18740.081703] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18740.081705] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18740.081786] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18740.081789] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18740.081792] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18740.081795] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18740.081804] Call Trace: [18740.081810] [18740.081821] ? show_regs.cold.9+0x22/0x2f [18740.081831] ? watchdog_timer_fn.cold.6+0x62/0x10a [18740.081837] ? __hrtimer_run_queues+0x1c2/0x4d0 [18740.081841] ? watchdog+0x40/0x40 [18740.081846] ? hrtimer_interrupt+0x128/0x2c0 [18740.081850] ? smp_apic_timer_interrupt+0xa6/0x270 [18740.081857] ? apic_timer_interrupt+0xf/0x20 [18740.081862] [18740.081865] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18740.081873] do_raw_spin_lock+0x177/0x190 [18740.081878] _raw_spin_lock+0x21/0x30 [18740.081883] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18740.082181] ? do_raw_spin_unlock+0x75/0x190 [18740.082188] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18740.082532] ? do_wait_intr+0xf0/0xf0 [18740.082543] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18740.083045] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18740.083277] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18740.083698] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18740.083886] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18740.083926] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18740.083958] mdc_intent_lock+0x353/0x7e0 [mdc] [18740.083987] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18740.084080] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18740.084684] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18740.084986] lmv_intent_open+0x338/0x1150 [lmv] [18740.085018] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18740.085119] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18740.089024] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18740.089112] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18740.089119] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18740.089123] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18740.089137] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18740.089142] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18740.089378] RAX: 0000000000006b3b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18740.092915] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18740.092969] ll_intent_lock+0xc5/0xaa0 [lustre] [18740.093088] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18740.093219] ll_lookup_it+0xe0e/0x2270 [lustre] [18740.093398] ? do_raw_spin_unlock+0x75/0x190 [18740.093406] ? _raw_spin_unlock+0x12/0x30 [18740.093413] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18740.093454] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18740.093475] ll_atomic_open+0x36a/0xd70 [lustre] [18740.093585] lookup_open+0x356/0xb40 [18740.093595] do_last+0x313/0xfc0 [18740.093599] ? nd_jump_root+0xe5/0x160 [18740.093604] ? path_init+0x437/0x520 [18740.093608] path_openat+0xf7/0x500 [18740.093613] ? do_raw_spin_unlock+0x75/0x190 [18740.093618] do_filp_open+0x99/0x140 [18740.093624] ? getname_flags+0x6e/0x330 [18740.093628] ? __check_object_size+0xff/0x256 [18740.093635] ? do_raw_spin_unlock+0x75/0x190 [18740.093639] ? _raw_spin_unlock+0x12/0x30 [18740.093644] do_sys_openat2+0x2b4/0x410 [18740.093648] do_sys_open+0x73/0xa0 [18740.093653] __x64_sys_openat+0x24/0x30 [18740.093657] do_syscall_64+0xc1/0x440 [18740.093674] entry_SYSCALL_64_after_hwframe+0x49/0xae [18740.093681] RIP: 0033:0x7f53425a5332 [18740.093693] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18740.093696] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18740.093701] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18740.093703] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18740.093706] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18740.093708] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18740.093710] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18740.719325] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18740.725178] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18740.728750] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18740.733134] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18740.736806] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18740.740942] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18740.743909] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18740.747705] Call Trace: [18740.749155] [18740.750065] ? show_regs.cold.9+0x22/0x2f [18740.752983] ? watchdog_timer_fn.cold.6+0x62/0x10a [18740.754935] ? __hrtimer_run_queues+0x1c2/0x4d0 [18740.758858] ? watchdog+0x40/0x40 [18740.760257] ? hrtimer_interrupt+0x128/0x2c0 [18740.763169] ? smp_apic_timer_interrupt+0xa6/0x270 [18740.772813] ? apic_timer_interrupt+0xf/0x20 [18740.777182] [18740.778156] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18740.782839] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18740.787737] do_raw_spin_lock+0x177/0x190 [18740.790151] _raw_spin_lock+0x21/0x30 [18740.796183] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18740.802473] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18740.805205] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18740.807733] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18740.812528] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18740.818831] ? woken_wake_function+0x30/0x30 [18740.821988] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18740.825995] kthread+0x1d1/0x200 [18740.829176] ? set_kthread_struct+0x70/0x70 [18740.832222] ret_from_fork+0x1f/0x30 [18744.053709] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [18744.057505] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18744.106467] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18744.114340] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18744.123004] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18744.126213] RIP: 0010:native_safe_halt+0x2e/0x50 [18744.130282] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18744.139603] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18744.146538] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18744.153114] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [18744.158376] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18744.165034] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18744.170599] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18744.176779] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18744.184745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18744.189967] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18744.194371] Call Trace: [18744.195262] [18744.196046] ? show_regs.cold.9+0x22/0x2f [18744.207141] ? watchdog_timer_fn.cold.6+0x62/0x10a [18744.217440] ? __hrtimer_run_queues+0x1c2/0x4d0 [18744.220465] ? watchdog+0x40/0x40 [18744.228884] ? hrtimer_interrupt+0x128/0x2c0 [18744.230438] ? smp_apic_timer_interrupt+0xa6/0x270 [18744.243358] ? apic_timer_interrupt+0xf/0x20 [18744.249428] [18744.255750] ? native_safe_halt+0x2e/0x50 [18744.256995] kvm_wait+0xcd/0xe0 [18744.257761] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18744.260705] ? newidle_balance.isra.93+0x207/0x580 [18744.267000] do_raw_spin_lock+0x177/0x190 [18744.270378] _raw_spin_lock+0x21/0x30 [18744.275323] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18744.279941] process_one_work+0x2c8/0x700 [18744.282034] worker_thread+0x296/0x6e0 [18744.284683] ? rescuer_thread+0x570/0x570 [18744.286134] kthread+0x1d1/0x200 [18744.297720] ? set_kthread_struct+0x70/0x70 [18744.306397] ret_from_fork+0x1f/0x30 [18749.928405] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16112s! [18749.932959] Showing busy workqueues and worker pools: [18749.935557] workqueue events: flags=0x0 [18749.937555] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18749.944228] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18749.953592] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18749.960451] pending: cpu_latency_work [ptlrpc] [18749.965683] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18749.969372] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18749.980624] workqueue events_long: flags=0x0 [18749.984670] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18749.992531] pending: lu_objects_flush [obdclass] [18749.998815] workqueue events_unbound: flags=0x2 [18750.001654] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18750.013187] pending: flush_memcg_stats_dwork, idle_cull_fn [18750.016757] workqueue events_power_efficient: flags=0x80 [18750.019834] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18750.023243] pending: check_lifetime, do_cache_clean [18750.026012] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18750.029279] pending: neigh_periodic_work, neigh_periodic_work [18750.042338] workqueue mm_percpu_wq: flags=0x8 [18750.047627] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18750.052056] pending: vmstat_update, lru_add_drain_per_cpu [18750.058620] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18750.063736] pending: vmstat_update, lru_add_drain_per_cpu [18750.072729] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18750.080256] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18750.086275] workqueue xprtiod: flags=0xa [18750.091048] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18750.095660] pending: xprt_autoclose, xs_error_handle [18750.099612] workqueue ipv6_addrconf: flags=0x40008 [18750.102378] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18750.106934] pending: addrconf_verify_work [18750.111438] workqueue knbd0-recv: flags=0x1a [18750.113589] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18750.120526] in-flight: 651:recv_work [18750.123576] workqueue ptlrpc_pinger: flags=0x2 [18750.126939] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18750.131043] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18750.134025] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16113s workers=4 idle: 15979 31824 8 [18750.140703] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18740s workers=2 idle: 654 [18768.055029] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [18768.057023] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [18768.058031] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [18768.058040] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18768.058118] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18768.058124] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18768.058127] RIP: 0010:native_safe_halt+0x2e/0x50 [18768.058140] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18768.058144] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18768.058148] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18768.058151] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18768.058153] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18768.058155] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18768.058157] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18768.058161] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18768.058164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18768.058166] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18768.058176] Call Trace: [18768.058181] [18768.058186] ? show_regs.cold.9+0x22/0x2f [18768.058197] ? watchdog_timer_fn.cold.6+0x62/0x10a [18768.058203] ? __hrtimer_run_queues+0x1c2/0x4d0 [18768.058208] ? watchdog+0x40/0x40 [18768.058212] ? hrtimer_interrupt+0x128/0x2c0 [18768.058216] ? smp_apic_timer_interrupt+0xa6/0x270 [18768.058225] ? apic_timer_interrupt+0xf/0x20 [18768.058229] [18768.058232] ? native_safe_halt+0x2e/0x50 [18768.058236] kvm_wait+0xcd/0xe0 [18768.058243] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18768.058251] do_raw_spin_lock+0x177/0x190 [18768.058256] _raw_spin_lock+0x21/0x30 [18768.058261] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18768.058561] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18768.059017] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18768.059244] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18768.059478] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18768.059598] Modules linked in: lustre(O) [18768.059699] ? woken_wake_function+0x30/0x30 [18768.059708] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18768.060041] kthread+0x1d1/0x200 [18768.060047] ? set_kthread_struct+0x70/0x70 [18768.060051] ret_from_fork+0x1f/0x30 [18768.063155] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18768.063229] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18768.063236] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18768.063240] RIP: 0010:native_safe_halt+0x2e/0x50 [18768.063252] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18768.063257] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18768.063262] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18768.063264] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18768.063267] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18768.063269] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18768.063271] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18768.063274] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18768.063277] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18768.063280] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18768.063287] Call Trace: [18768.063292] [18768.063297] ? show_regs.cold.9+0x22/0x2f [18768.063307] ? watchdog_timer_fn.cold.6+0x62/0x10a [18768.063314] ? __hrtimer_run_queues+0x1c2/0x4d0 [18768.063319] ? watchdog+0x40/0x40 [18768.063323] ? hrtimer_interrupt+0x128/0x2c0 [18768.063328] ? smp_apic_timer_interrupt+0xa6/0x270 [18768.063335] ? apic_timer_interrupt+0xf/0x20 [18768.063340] [18768.063343] ? native_safe_halt+0x2e/0x50 [18768.063347] kvm_wait+0xcd/0xe0 [18768.063355] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18768.063363] do_raw_spin_lock+0x177/0x190 [18768.063367] _raw_spin_lock+0x21/0x30 [18768.063373] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18768.063643] obd_mod_free+0x46/0x140 [ptlrpc] [18768.063857] mdc_commit_open+0x91/0xb0 [mdc] [18768.063895] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18768.064085] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18768.064279] after_reply+0xe2c/0x1780 [ptlrpc] [18768.064404] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18768.064547] ? __next_timer_interrupt+0x160/0x160 [18768.064554] ptlrpcd+0xa82/0xe40 [ptlrpc] [18768.064705] ? do_wait_intr+0xf0/0xf0 [18768.064712] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18768.064876] kthread+0x1d1/0x200 [18768.064881] ? set_kthread_struct+0x70/0x70 [18768.064885] ret_from_fork+0x1f/0x30 [18768.393490] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18768.419718] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18768.424741] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18768.431142] RIP: 0010:native_safe_halt+0x2e/0x50 [18768.433219] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18768.440522] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18768.443238] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18768.448717] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18768.451301] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18768.454892] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18768.460450] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18768.467367] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18768.471849] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18768.477338] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18768.480995] Call Trace: [18768.481867] [18768.484506] ? show_regs.cold.9+0x22/0x2f [18768.486097] ? watchdog_timer_fn.cold.6+0x62/0x10a [18768.488148] ? __hrtimer_run_queues+0x1c2/0x4d0 [18768.489773] ? watchdog+0x40/0x40 [18768.491266] ? hrtimer_interrupt+0x128/0x2c0 [18768.492991] ? smp_apic_timer_interrupt+0xa6/0x270 [18768.495154] ? apic_timer_interrupt+0xf/0x20 [18768.496761] [18768.497551] ? native_safe_halt+0x2e/0x50 [18768.499226] kvm_wait+0xcd/0xe0 [18768.500425] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18768.502563] do_raw_spin_lock+0x177/0x190 [18768.504852] _raw_spin_lock+0x21/0x30 [18768.506891] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18768.509795] ? do_raw_spin_unlock+0x75/0x190 [18768.512129] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18768.514930] ? do_wait_intr+0xf0/0xf0 [18768.516577] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18768.519394] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18768.520910] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18768.523373] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18768.525658] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18768.527409] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18768.530190] mdc_intent_lock+0x353/0x7e0 [mdc] [18768.532037] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18768.534401] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18768.537663] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18768.541863] lmv_intent_open+0x338/0x1150 [lmv] [18768.544243] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18768.546443] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18768.549956] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18768.551920] ll_intent_lock+0xc5/0xaa0 [lustre] [18768.553970] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18768.556682] ll_lookup_it+0xe0e/0x2270 [lustre] [18768.558703] ? do_raw_spin_unlock+0x75/0x190 [18768.560544] ? _raw_spin_unlock+0x12/0x30 [18768.562692] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18768.565876] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18768.567944] ll_atomic_open+0x36a/0xd70 [lustre] [18768.573090] lookup_open+0x356/0xb40 [18768.574529] do_last+0x313/0xfc0 [18768.576259] ? nd_jump_root+0xe5/0x160 [18768.577792] ? path_init+0x437/0x520 [18768.579579] path_openat+0xf7/0x500 [18768.581032] ? do_raw_spin_unlock+0x75/0x190 [18768.583534] do_filp_open+0x99/0x140 [18768.584860] ? getname_flags+0x6e/0x330 [18768.586533] ? __check_object_size+0xff/0x256 [18768.588396] ? do_raw_spin_unlock+0x75/0x190 [18768.590666] ? _raw_spin_unlock+0x12/0x30 [18768.592462] do_sys_openat2+0x2b4/0x410 [18768.594152] do_sys_open+0x73/0xa0 [18768.595501] __x64_sys_openat+0x24/0x30 [18768.598393] do_syscall_64+0xc1/0x440 [18768.599879] entry_SYSCALL_64_after_hwframe+0x49/0xae [18768.603653] RIP: 0033:0x7f53425a5332 [18768.605246] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18768.614448] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18768.617177] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18768.620491] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18768.623681] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18768.628906] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18768.631636] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18776.064855] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [18776.074747] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18776.144904] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18776.155311] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18776.160349] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18776.164396] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18776.168251] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18776.176733] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18776.180342] RAX: 0000000000007e97 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18776.183658] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [18776.186999] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18776.190254] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18776.193574] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18776.197188] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18776.200587] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18776.203590] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18776.207757] Call Trace: [18776.208993] [18776.210045] ? show_regs.cold.9+0x22/0x2f [18776.212177] ? watchdog_timer_fn.cold.6+0x62/0x10a [18776.214653] ? __hrtimer_run_queues+0x1c2/0x4d0 [18776.216909] ? watchdog+0x40/0x40 [18776.218573] ? hrtimer_interrupt+0x128/0x2c0 [18776.220522] ? smp_apic_timer_interrupt+0xa6/0x270 [18776.223684] ? apic_timer_interrupt+0xf/0x20 [18776.225571] [18776.226783] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18776.229333] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18776.231979] ? newidle_balance.isra.93+0x207/0x580 [18776.234328] do_raw_spin_lock+0x177/0x190 [18776.236300] _raw_spin_lock+0x21/0x30 [18776.238146] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18776.241149] process_one_work+0x2c8/0x700 [18776.243216] worker_thread+0x296/0x6e0 [18776.244654] ? rescuer_thread+0x570/0x570 [18776.246642] kthread+0x1d1/0x200 [18776.248263] ? set_kthread_struct+0x70/0x70 [18776.250323] ret_from_fork+0x1f/0x30 [18780.639304] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16143s! [18780.655914] Showing busy workqueues and worker pools: [18780.664390] workqueue events: flags=0x0 [18780.671341] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18780.679212] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18780.687662] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18780.690176] pending: cpu_latency_work [ptlrpc] [18780.697030] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18780.705226] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18780.710422] workqueue events_long: flags=0x0 [18780.713088] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18780.715836] pending: lu_objects_flush [obdclass] [18780.718231] workqueue events_unbound: flags=0x2 [18780.720232] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18780.723144] pending: flush_memcg_stats_dwork, idle_cull_fn [18780.725826] workqueue events_power_efficient: flags=0x80 [18780.727991] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18780.730841] pending: check_lifetime, do_cache_clean [18780.732965] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18780.735382] pending: neigh_periodic_work, neigh_periodic_work [18780.737970] workqueue mm_percpu_wq: flags=0x8 [18780.739402] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18780.742195] pending: vmstat_update, lru_add_drain_per_cpu [18780.744212] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18780.746694] pending: vmstat_update, lru_add_drain_per_cpu [18780.749043] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18780.752428] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18780.756711] workqueue xprtiod: flags=0xa [18780.758337] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18780.761917] pending: xprt_autoclose, xs_error_handle [18780.764693] workqueue ipv6_addrconf: flags=0x40008 [18780.767153] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18780.770910] pending: addrconf_verify_work [18780.772459] workqueue knbd0-recv: flags=0x1a [18780.774673] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18780.778344] in-flight: 651:recv_work [18780.780153] workqueue ptlrpc_pinger: flags=0x2 [18780.783894] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18780.787846] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18780.790683] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16143s workers=4 idle: 15979 31824 8 [18780.793767] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18771s workers=2 idle: 654 [18796.055020] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [18796.058241] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [18796.062723] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [18796.062746] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18796.062817] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18796.062823] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18796.062826] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18796.062839] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18796.062843] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18796.062853] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18796.062855] RDX: 00000000000033b2 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18796.062858] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18796.062860] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18796.062863] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18796.062866] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18796.062869] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18796.062872] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18796.062880] Call Trace: [18796.062885] [18796.062891] ? show_regs.cold.9+0x22/0x2f [18796.062901] ? watchdog_timer_fn.cold.6+0x62/0x10a [18796.062908] ? __hrtimer_run_queues+0x1c2/0x4d0 [18796.062915] ? watchdog+0x40/0x40 [18796.062919] ? hrtimer_interrupt+0x128/0x2c0 [18796.062923] ? smp_apic_timer_interrupt+0xa6/0x270 [18796.062930] ? apic_timer_interrupt+0xf/0x20 [18796.062935] [18796.062938] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18796.062943] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18796.062948] do_raw_spin_lock+0x177/0x190 [18796.062953] _raw_spin_lock+0x21/0x30 [18796.062959] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18796.063334] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) [18796.063219] obd_mod_free+0x46/0x140 [ptlrpc] [18796.063347] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18796.063395] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18796.063401] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18796.063403] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18796.063411] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18796.063415] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18796.063420] RAX: 0000000000002249 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18796.063423] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18796.063426] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18796.063428] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18796.063431] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18796.063433] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18796.063437] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18796.063439] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18796.063446] Call Trace: [18796.063448] [18796.063452] ? show_regs.cold.9+0x22/0x2f [18796.063458] ? watchdog_timer_fn.cold.6+0x62/0x10a [18796.063463] ? __hrtimer_run_queues+0x1c2/0x4d0 [18796.063467] ? watchdog+0x40/0x40 [18796.063472] ? hrtimer_interrupt+0x128/0x2c0 [18796.063476] ? smp_apic_timer_interrupt+0xa6/0x270 [18796.063482] ? apic_timer_interrupt+0xf/0x20 [18796.063487] [18796.063490] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18796.063496] do_raw_spin_lock+0x177/0x190 [18796.063500] _raw_spin_lock+0x21/0x30 [18796.063505] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18796.064021] mdc_commit_open+0x91/0xb0 [mdc] [18796.064060] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18796.064257] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18796.064670] after_reply+0xe2c/0x1780 [ptlrpc] [18796.064880] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18796.065070] ? __next_timer_interrupt+0x160/0x160 [18796.065081] ptlrpcd+0xa82/0xe40 [ptlrpc] [18796.065291] ? do_wait_intr+0xf0/0xf0 [18796.065301] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18796.066650] kthread+0x1d1/0x200 [18796.066658] ? set_kthread_struct+0x70/0x70 [18796.066663] ret_from_fork+0x1f/0x30 [18796.069474] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18796.069557] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18796.069568] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18796.069571] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18796.069645] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18796.069650] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18796.069655] RAX: 000000000000544d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18796.069658] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18796.069661] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18796.069667] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18796.069670] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18796.069673] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18796.069675] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18796.069677] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18796.069685] Call Trace: [18796.069694] [18796.069699] ? show_regs.cold.9+0x22/0x2f [18796.069713] ? watchdog_timer_fn.cold.6+0x62/0x10a [18796.069719] ? __hrtimer_run_queues+0x1c2/0x4d0 [18796.069724] ? watchdog+0x40/0x40 [18796.069732] ? hrtimer_interrupt+0x128/0x2c0 [18796.069737] ? smp_apic_timer_interrupt+0xa6/0x270 [18796.069746] ? apic_timer_interrupt+0xf/0x20 [18796.069750] [18796.069754] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18796.069758] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18796.069765] do_raw_spin_lock+0x177/0x190 [18796.069770] _raw_spin_lock+0x21/0x30 [18796.069775] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18796.074871] ? do_raw_spin_unlock+0x75/0x190 [18796.074896] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18796.075191] ? do_wait_intr+0xf0/0xf0 [18796.075200] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18796.075668] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18796.075994] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18796.076498] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18796.076754] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18796.076798] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18796.076830] mdc_intent_lock+0x353/0x7e0 [mdc] [18796.107721] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18796.108002] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18796.108538] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18796.108818] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18796.115901] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18796.116763] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18796.121231] ? woken_wake_function+0x30/0x30 [18796.121251] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18796.122467] kthread+0x1d1/0x200 [18796.122475] ? set_kthread_struct+0x70/0x70 [18796.122480] ret_from_fork+0x1f/0x30 [18796.723751] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18796.736878] lmv_intent_open+0x338/0x1150 [lmv] [18796.740692] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18796.751936] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18796.762578] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18796.769045] ll_intent_lock+0xc5/0xaa0 [lustre] [18796.770857] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18796.777912] ll_lookup_it+0xe0e/0x2270 [lustre] [18796.785823] ? do_raw_spin_unlock+0x75/0x190 [18796.792868] ? _raw_spin_unlock+0x12/0x30 [18796.796982] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18796.799987] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18796.810635] ll_atomic_open+0x36a/0xd70 [lustre] [18796.814250] lookup_open+0x356/0xb40 [18796.815604] do_last+0x313/0xfc0 [18796.828189] ? nd_jump_root+0xe5/0x160 [18796.833240] ? path_init+0x437/0x520 [18796.838907] path_openat+0xf7/0x500 [18796.844825] ? do_raw_spin_unlock+0x75/0x190 [18796.849052] do_filp_open+0x99/0x140 [18796.851704] ? getname_flags+0x6e/0x330 [18796.858121] ? __check_object_size+0xff/0x256 [18796.859929] ? do_raw_spin_unlock+0x75/0x190 [18796.861695] ? _raw_spin_unlock+0x12/0x30 [18796.865646] do_sys_openat2+0x2b4/0x410 [18796.868157] do_sys_open+0x73/0xa0 [18796.872937] __x64_sys_openat+0x24/0x30 [18796.876874] do_syscall_64+0xc1/0x440 [18796.879981] entry_SYSCALL_64_after_hwframe+0x49/0xae [18796.884094] RIP: 0033:0x7f53425a5332 [18796.885864] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18796.897184] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18796.902129] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18796.905899] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18796.909753] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18796.913170] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18796.916289] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18808.060268] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [18808.064887] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18808.095886] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18808.102619] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18808.106671] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18808.114120] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18808.116994] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18808.126350] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18808.128731] RAX: 0000000000006786 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18808.134475] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [18808.139893] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18808.146100] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18808.149595] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18808.152850] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18808.157030] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18808.159585] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18808.166697] Call Trace: [18808.168053] [18808.169327] ? show_regs.cold.9+0x22/0x2f [18808.171276] ? watchdog_timer_fn.cold.6+0x62/0x10a [18808.173502] ? __hrtimer_run_queues+0x1c2/0x4d0 [18808.176908] ? watchdog+0x40/0x40 [18808.180406] ? hrtimer_interrupt+0x128/0x2c0 [18808.183160] ? smp_apic_timer_interrupt+0xa6/0x270 [18808.185836] ? apic_timer_interrupt+0xf/0x20 [18808.188815] [18808.189876] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18808.193617] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18808.198387] ? newidle_balance.isra.93+0x207/0x580 [18808.201278] do_raw_spin_lock+0x177/0x190 [18808.201885] _raw_spin_lock+0x21/0x30 [18808.202517] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18808.205015] process_one_work+0x2c8/0x700 [18808.207179] worker_thread+0x296/0x6e0 [18808.208723] ? rescuer_thread+0x570/0x570 [18808.210648] kthread+0x1d1/0x200 [18808.211794] ? set_kthread_struct+0x70/0x70 [18808.214695] ret_from_fork+0x1f/0x30 [18810.847326] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16173s! [18810.850397] Showing busy workqueues and worker pools: [18810.853272] workqueue events: flags=0x0 [18810.856281] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18810.861800] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18810.868869] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18810.872373] pending: cpu_latency_work [ptlrpc] [18810.878860] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18810.888282] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18810.898363] workqueue events_long: flags=0x0 [18810.907338] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18810.914468] pending: lu_objects_flush [obdclass] [18810.919210] workqueue events_unbound: flags=0x2 [18810.922138] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18810.925665] pending: flush_memcg_stats_dwork, idle_cull_fn [18810.928665] workqueue events_power_efficient: flags=0x80 [18810.931286] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18810.936561] pending: check_lifetime, do_cache_clean [18810.948576] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18810.950916] pending: neigh_periodic_work, neigh_periodic_work [18810.960105] workqueue mm_percpu_wq: flags=0x8 [18810.964561] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18810.972979] pending: vmstat_update, lru_add_drain_per_cpu [18810.979274] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18810.984179] pending: vmstat_update, lru_add_drain_per_cpu [18810.987852] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18810.995271] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18811.003399] workqueue xprtiod: flags=0xa [18811.017393] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18811.019022] pending: xprt_autoclose, xs_error_handle [18811.023009] workqueue ipv6_addrconf: flags=0x40008 [18811.027585] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18811.033765] pending: addrconf_verify_work [18811.039361] workqueue knbd0-recv: flags=0x1a [18811.043143] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18811.045309] in-flight: 651:recv_work [18811.047822] workqueue ptlrpc_pinger: flags=0x2 [18811.049348] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18811.052434] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18811.062838] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16173s workers=4 idle: 15979 31824 8 [18811.070356] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18801s workers=2 idle: 654 [18824.055488] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [18824.058633] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [18824.060353] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [18824.060363] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18824.060452] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18824.060459] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18824.060462] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18824.060475] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18824.060479] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18824.060484] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18824.060486] RDX: 0000000000005860 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18824.060488] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18824.060491] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18824.060493] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18824.060496] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18824.060498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18824.060500] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18824.060638] Call Trace: [18824.060645] [18824.060654] Modules linked in: [18824.060650] ? show_regs.cold.9+0x22/0x2f [18824.060658] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18824.060712] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18824.060716] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18824.060718] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18824.060726] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18824.060731] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18824.060735] RAX: 0000000000006e8d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18824.060737] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18824.060739] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18824.060742] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18824.060744] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18824.060747] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18824.060750] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18824.060753] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18824.060759] Call Trace: [18824.060762] [18824.060765] ? show_regs.cold.9+0x22/0x2f [18824.060772] ? watchdog_timer_fn.cold.6+0x62/0x10a [18824.060778] ? __hrtimer_run_queues+0x1c2/0x4d0 [18824.060782] ? watchdog+0x40/0x40 [18824.060786] ? hrtimer_interrupt+0x128/0x2c0 [18824.060790] ? smp_apic_timer_interrupt+0xa6/0x270 [18824.060799] ? apic_timer_interrupt+0xf/0x20 [18824.060803] [18824.060806] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18824.060812] do_raw_spin_lock+0x177/0x190 [18824.060816] _raw_spin_lock+0x21/0x30 [18824.064658] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18824.064733] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18824.064740] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18824.064743] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18824.064757] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18824.064761] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18824.064765] RAX: 0000000000007482 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18824.064767] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18824.064770] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18824.064772] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18824.064774] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18824.064778] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18824.064780] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18824.064783] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18824.064790] Call Trace: [18824.064795] [18824.064799] ? show_regs.cold.9+0x22/0x2f [18824.064808] ? watchdog_timer_fn.cold.6+0x62/0x10a [18824.064813] ? __hrtimer_run_queues+0x1c2/0x4d0 [18824.064817] ? watchdog+0x40/0x40 [18824.064821] ? hrtimer_interrupt+0x128/0x2c0 [18824.064825] ? smp_apic_timer_interrupt+0xa6/0x270 [18824.064917] ? apic_timer_interrupt+0xf/0x20 [18824.064921] [18824.064924] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18824.064929] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18824.064935] do_raw_spin_lock+0x177/0x190 [18824.064940] _raw_spin_lock+0x21/0x30 [18824.064945] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18824.065222] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18824.065416] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18824.065747] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18824.066021] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18824.066228] ? woken_wake_function+0x30/0x30 [18824.066237] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18824.066441] kthread+0x1d1/0x200 [18824.066446] ? set_kthread_struct+0x70/0x70 [18824.066450] ret_from_fork+0x1f/0x30 [18824.060822] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18824.067020] ? do_raw_spin_unlock+0x75/0x190 [18824.067028] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18824.068060] ? do_wait_intr+0xf0/0xf0 [18824.068071] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18824.068374] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18824.068716] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18824.068938] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18824.069051] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18824.069092] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18824.069123] mdc_intent_lock+0x353/0x7e0 [mdc] [18824.069150] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18824.070206] ? watchdog_timer_fn.cold.6+0x62/0x10a [18824.070218] ? __hrtimer_run_queues+0x1c2/0x4d0 [18824.070222] ? watchdog+0x40/0x40 [18824.070225] ? hrtimer_interrupt+0x128/0x2c0 [18824.070229] ? smp_apic_timer_interrupt+0xa6/0x270 [18824.070235] ? apic_timer_interrupt+0xf/0x20 [18824.070239] [18824.070243] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18824.070249] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18824.070253] do_raw_spin_lock+0x177/0x190 [18824.070257] _raw_spin_lock+0x21/0x30 [18824.070262] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18824.070482] obd_mod_free+0x46/0x140 [ptlrpc] [18824.071504] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18824.071853] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18824.071897] lmv_intent_open+0x338/0x1150 [lmv] [18824.071931] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18824.072040] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18824.072121] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18824.072144] ll_intent_lock+0xc5/0xaa0 [lustre] [18824.072248] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18824.072330] ll_lookup_it+0xe0e/0x2270 [lustre] [18824.072421] ? do_raw_spin_unlock+0x75/0x190 [18824.072429] ? _raw_spin_unlock+0x12/0x30 [18824.072437] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18824.072470] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18824.072488] ll_atomic_open+0x36a/0xd70 [lustre] [18824.072579] lookup_open+0x356/0xb40 [18824.072588] do_last+0x313/0xfc0 [18824.072593] ? nd_jump_root+0xe5/0x160 [18824.072597] ? path_init+0x437/0x520 [18824.072602] path_openat+0xf7/0x500 [18824.072606] ? do_raw_spin_unlock+0x75/0x190 [18824.072611] do_filp_open+0x99/0x140 [18824.072616] ? getname_flags+0x6e/0x330 [18824.072621] ? __check_object_size+0xff/0x256 [18824.072761] ? do_raw_spin_unlock+0x75/0x190 [18824.072765] ? _raw_spin_unlock+0x12/0x30 [18824.072770] do_sys_openat2+0x2b4/0x410 [18824.072775] do_sys_open+0x73/0xa0 [18824.072780] __x64_sys_openat+0x24/0x30 [18824.072784] do_syscall_64+0xc1/0x440 [18824.072791] entry_SYSCALL_64_after_hwframe+0x49/0xae [18824.072799] RIP: 0033:0x7f53425a5332 [18824.072807] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18824.072811] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18824.072816] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18824.072819] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18824.072821] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18824.072823] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18824.072825] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18824.622282] mdc_commit_open+0x91/0xb0 [mdc] [18824.623374] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18824.628261] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18824.632759] after_reply+0xe2c/0x1780 [ptlrpc] [18824.637472] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18824.639275] ? __next_timer_interrupt+0x160/0x160 [18824.649143] ptlrpcd+0xa82/0xe40 [ptlrpc] [18824.653803] ? do_wait_intr+0xf0/0xf0 [18824.656462] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18824.660439] kthread+0x1d1/0x200 [18824.662456] ? set_kthread_struct+0x70/0x70 [18824.663919] ret_from_fork+0x1f/0x30 [18836.052265] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [18836.058298] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18836.081543] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18836.102024] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18836.121394] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18836.127332] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18836.130770] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18836.152418] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18836.164318] RAX: 00000000000021d6 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18836.168488] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [18836.175974] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18836.186251] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18836.194285] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18836.202402] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18836.210524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18836.217767] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18836.224138] Call Trace: [18836.226548] [18836.227354] ? show_regs.cold.9+0x22/0x2f [18836.228804] ? watchdog_timer_fn.cold.6+0x62/0x10a [18836.232483] ? __hrtimer_run_queues+0x1c2/0x4d0 [18836.235871] ? watchdog+0x40/0x40 [18836.241150] ? hrtimer_interrupt+0x128/0x2c0 [18836.243368] ? smp_apic_timer_interrupt+0xa6/0x270 [18836.246887] ? apic_timer_interrupt+0xf/0x20 [18836.248958] [18836.249770] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18836.251914] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18836.253683] ? newidle_balance.isra.93+0x207/0x580 [18836.255161] do_raw_spin_lock+0x177/0x190 [18836.256913] _raw_spin_lock+0x21/0x30 [18836.258183] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18836.261235] process_one_work+0x2c8/0x700 [18836.263995] worker_thread+0x296/0x6e0 [18836.265396] ? rescuer_thread+0x570/0x570 [18836.266969] kthread+0x1d1/0x200 [18836.268303] ? set_kthread_struct+0x70/0x70 [18836.271019] ret_from_fork+0x1f/0x30 [18841.568314] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16204s! [18841.574972] Showing busy workqueues and worker pools: [18841.578397] workqueue events: flags=0x0 [18841.580092] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18841.587931] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18841.594164] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18841.597300] pending: cpu_latency_work [ptlrpc] [18841.601656] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18841.620809] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18841.628569] workqueue events_long: flags=0x0 [18841.631050] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18841.635062] pending: lu_objects_flush [obdclass] [18841.639439] workqueue events_unbound: flags=0x2 [18841.642587] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18841.651366] pending: flush_memcg_stats_dwork, idle_cull_fn [18841.656384] workqueue events_power_efficient: flags=0x80 [18841.661929] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18841.668143] pending: check_lifetime, do_cache_clean [18841.672009] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18841.678185] pending: neigh_periodic_work, neigh_periodic_work [18841.682875] workqueue mm_percpu_wq: flags=0x8 [18841.684311] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18841.688174] pending: vmstat_update, lru_add_drain_per_cpu [18841.693026] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18841.696197] pending: vmstat_update, lru_add_drain_per_cpu [18841.702334] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18841.709631] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18841.712901] workqueue xprtiod: flags=0xa [18841.714857] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18841.717963] pending: xprt_autoclose, xs_error_handle [18841.720632] workqueue ipv6_addrconf: flags=0x40008 [18841.722504] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18841.725574] pending: addrconf_verify_work [18841.729602] workqueue knbd0-recv: flags=0x1a [18841.735069] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18841.740093] in-flight: 651:recv_work [18841.745056] workqueue ptlrpc_pinger: flags=0x2 [18841.746636] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18841.752002] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18841.761920] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16204s workers=4 idle: 15979 31824 8 [18841.774351] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18832s workers=2 idle: 654 [18852.058020] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [18852.058786] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [18852.060597] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [18852.060609] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18852.060673] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18852.060678] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18852.060681] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18852.060695] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18852.060699] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18852.060703] RAX: 0000000000007c01 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18852.060706] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18852.060708] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18852.060710] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18852.060712] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18852.060714] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18852.063487] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18852.063493] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18852.063501] Call Trace: [18852.063508] [18852.063515] ? show_regs.cold.9+0x22/0x2f [18852.063530] ? watchdog_timer_fn.cold.6+0x62/0x10a [18852.063535] ? __hrtimer_run_queues+0x1c2/0x4d0 [18852.063540] ? watchdog+0x40/0x40 [18852.063545] ? hrtimer_interrupt+0x128/0x2c0 [18852.063549] ? smp_apic_timer_interrupt+0xa6/0x270 [18852.063557] ? apic_timer_interrupt+0xf/0x20 [18852.063561] [18852.063564] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18852.063574] do_raw_spin_lock+0x177/0x190 [18852.063579] _raw_spin_lock+0x21/0x30 [18852.063584] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18852.063838] ? do_raw_spin_unlock+0x75/0x190 [18852.063846] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18852.064047] ? do_wait_intr+0xf0/0xf0 [18852.064058] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18852.064250] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18852.064481] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18852.064688] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18852.064797] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18852.064836] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18852.064865] mdc_intent_lock+0x353/0x7e0 [mdc] [18852.064891] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18852.064974] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18852.065190] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18852.065219] lmv_intent_open+0x338/0x1150 [lmv] [18852.065250] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18852.065335] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18852.065438] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18852.065462] ll_intent_lock+0xc5/0xaa0 [lustre] [18852.065547] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18852.065631] ll_lookup_it+0xe0e/0x2270 [lustre] [18852.065713] ? do_raw_spin_unlock+0x75/0x190 [18852.065719] ? _raw_spin_unlock+0x12/0x30 [18852.065727] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18852.065751] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18852.065769] ll_atomic_open+0x36a/0xd70 [lustre] [18852.065854] lookup_open+0x356/0xb40 [18852.065864] do_last+0x313/0xfc0 [18852.065869] ? nd_jump_root+0xe5/0x160 [18852.065874] ? path_init+0x437/0x520 [18852.065878] path_openat+0xf7/0x500 [18852.065883] ? do_raw_spin_unlock+0x75/0x190 [18852.065888] do_filp_open+0x99/0x140 [18852.065894] ? getname_flags+0x6e/0x330 [18852.065899] ? __check_object_size+0xff/0x256 [18852.065907] ? do_raw_spin_unlock+0x75/0x190 [18852.065911] ? _raw_spin_unlock+0x12/0x30 [18852.065916] do_sys_openat2+0x2b4/0x410 [18852.065921] do_sys_open+0x73/0xa0 [18852.065926] __x64_sys_openat+0x24/0x30 [18852.065930] do_syscall_64+0xc1/0x440 [18852.065936] entry_SYSCALL_64_after_hwframe+0x49/0xae [18852.065942] RIP: 0033:0x7f53425a5332 [18852.065949] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18852.065953] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18852.065958] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18852.065961] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18852.065963] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18852.065966] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18852.065968] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18852.067518] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18852.067588] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18852.067596] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18852.067599] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18852.067612] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18852.067616] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18852.067620] RAX: 0000000000002286 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18852.067622] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18852.067624] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18852.067627] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18852.067630] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18852.067645] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18852.067648] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18852.067651] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18852.067658] Call Trace: [18852.067664] [18852.067669] ? show_regs.cold.9+0x22/0x2f [18852.067677] ? watchdog_timer_fn.cold.6+0x62/0x10a [18852.067683] ? __hrtimer_run_queues+0x1c2/0x4d0 [18852.067687] ? watchdog+0x40/0x40 [18852.067691] ? hrtimer_interrupt+0x128/0x2c0 [18852.067695] ? smp_apic_timer_interrupt+0xa6/0x270 [18852.067702] ? apic_timer_interrupt+0xf/0x20 [18852.067707] [18852.067709] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18852.067714] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18852.067720] do_raw_spin_lock+0x177/0x190 [18852.067726] _raw_spin_lock+0x21/0x30 [18852.067731] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18852.067998] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18852.068187] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18852.068860] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18852.069088] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18852.069307] ? woken_wake_function+0x30/0x30 [18852.069315] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18852.077302] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) [18852.088019] kthread+0x1d1/0x200 [18852.088038] ? set_kthread_struct+0x70/0x70 [18852.088043] ret_from_fork+0x1f/0x30 [18852.826681] ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18852.845893] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18852.854220] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18852.858663] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18852.860737] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18852.880158] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18852.885471] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18852.889089] RDX: 000000000000014b RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18852.893008] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18852.896435] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18852.899239] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18852.905342] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18852.908907] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18852.911216] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18852.913691] Call Trace: [18852.914913] [18852.915695] ? show_regs.cold.9+0x22/0x2f [18852.921159] ? watchdog_timer_fn.cold.6+0x62/0x10a [18852.925777] ? __hrtimer_run_queues+0x1c2/0x4d0 [18852.931418] ? watchdog+0x40/0x40 [18852.933092] ? hrtimer_interrupt+0x128/0x2c0 [18852.935947] ? smp_apic_timer_interrupt+0xa6/0x270 [18852.938027] ? apic_timer_interrupt+0xf/0x20 [18852.940051] [18852.941023] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18852.944088] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18852.946987] do_raw_spin_lock+0x177/0x190 [18852.949273] _raw_spin_lock+0x21/0x30 [18852.950937] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18852.958603] obd_mod_free+0x46/0x140 [ptlrpc] [18852.963466] mdc_commit_open+0x91/0xb0 [mdc] [18852.966012] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18852.972488] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18852.976629] after_reply+0xe2c/0x1780 [ptlrpc] [18852.979242] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18852.981950] ? __next_timer_interrupt+0x160/0x160 [18852.984307] ptlrpcd+0xa82/0xe40 [ptlrpc] [18852.986986] ? do_wait_intr+0xf0/0xf0 [18852.990635] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18852.993139] kthread+0x1d1/0x200 [18852.994707] ? set_kthread_struct+0x70/0x70 [18852.996647] ret_from_fork+0x1f/0x30 [18868.055673] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [18868.060715] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18868.109157] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18868.113620] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18868.122336] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18868.130772] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18868.133714] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18868.143703] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18868.148013] RAX: 0000000000007873 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18868.152830] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [18868.154613] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18868.158516] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18868.162515] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18868.170555] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18868.179067] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18868.181398] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18868.186934] Call Trace: [18868.189034] [18868.192778] ? show_regs.cold.9+0x22/0x2f [18868.193812] ? watchdog_timer_fn.cold.6+0x62/0x10a [18868.197639] ? __hrtimer_run_queues+0x1c2/0x4d0 [18868.200844] ? watchdog+0x40/0x40 [18868.201908] ? hrtimer_interrupt+0x128/0x2c0 [18868.205847] ? smp_apic_timer_interrupt+0xa6/0x270 [18868.209486] ? apic_timer_interrupt+0xf/0x20 [18868.211560] [18868.212138] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18868.215823] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18868.219811] ? newidle_balance.isra.93+0x207/0x580 [18868.221201] do_raw_spin_lock+0x177/0x190 [18868.225970] _raw_spin_lock+0x21/0x30 [18868.227986] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18868.232695] process_one_work+0x2c8/0x700 [18868.234183] worker_thread+0x296/0x6e0 [18868.237999] ? rescuer_thread+0x570/0x570 [18868.239423] kthread+0x1d1/0x200 [18868.240661] ? set_kthread_struct+0x70/0x70 [18868.243036] ret_from_fork+0x1f/0x30 [18872.288218] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16235s! [18872.291471] Showing busy workqueues and worker pools: [18872.294762] workqueue events: flags=0x0 [18872.298434] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18872.301194] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18872.307440] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18872.312651] pending: cpu_latency_work [ptlrpc] [18872.317719] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18872.324396] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18872.328946] workqueue events_long: flags=0x0 [18872.333099] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18872.336879] pending: lu_objects_flush [obdclass] [18872.338427] workqueue events_unbound: flags=0x2 [18872.342495] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18872.349434] pending: flush_memcg_stats_dwork, idle_cull_fn [18872.356826] workqueue events_power_efficient: flags=0x80 [18872.361449] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18872.364940] pending: check_lifetime, do_cache_clean [18872.367441] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18872.371602] pending: neigh_periodic_work, neigh_periodic_work [18872.376244] workqueue mm_percpu_wq: flags=0x8 [18872.382935] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18872.392541] pending: vmstat_update, lru_add_drain_per_cpu [18872.397489] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18872.400507] pending: vmstat_update, lru_add_drain_per_cpu [18872.402865] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18872.408482] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18872.413957] workqueue xprtiod: flags=0xa [18872.415925] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18872.422689] pending: xprt_autoclose, xs_error_handle [18872.428227] workqueue ipv6_addrconf: flags=0x40008 [18872.431337] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18872.433524] pending: addrconf_verify_work [18872.435148] workqueue knbd0-recv: flags=0x1a [18872.436728] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18872.441570] in-flight: 651:recv_work [18872.443815] workqueue ptlrpc_pinger: flags=0x2 [18872.445452] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18872.447877] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18872.450562] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16235s workers=4 idle: 15979 31824 8 [18872.453472] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18862s workers=2 idle: 654 [18880.055125] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18880.057032] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18880.059309] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [18880.059321] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18880.059471] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18880.059477] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18880.059479] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18880.059493] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18880.059498] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18880.059502] RAX: 0000000000007e93 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18880.059505] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18880.059507] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18880.059510] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18880.059512] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18880.059515] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18880.059518] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18880.059521] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18880.059530] Call Trace: [18880.059535] [18880.059542] ? show_regs.cold.9+0x22/0x2f [18880.059551] ? watchdog_timer_fn.cold.6+0x62/0x10a [18880.059558] ? __hrtimer_run_queues+0x1c2/0x4d0 [18880.059562] ? watchdog+0x40/0x40 [18880.059567] ? hrtimer_interrupt+0x128/0x2c0 [18880.059571] ? smp_apic_timer_interrupt+0xa6/0x270 [18880.059579] ? apic_timer_interrupt+0xf/0x20 [18880.059585] [18880.059588] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18880.059593] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18880.059600] do_raw_spin_lock+0x177/0x190 [18880.059605] _raw_spin_lock+0x21/0x30 [18880.059652] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18880.059710] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18880.059714] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18880.059716] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18880.059724] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18880.059727] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18880.059731] RAX: 0000000000005d01 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18880.059734] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18880.059736] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18880.059738] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18880.059740] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18880.059743] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18880.059745] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18880.059747] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18880.059754] Call Trace: [18880.059768] [18880.059771] ? show_regs.cold.9+0x22/0x2f [18880.059609] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18880.059778] ? watchdog_timer_fn.cold.6+0x62/0x10a [18880.059787] ? __hrtimer_run_queues+0x1c2/0x4d0 [18880.059791] ? watchdog+0x40/0x40 [18880.059795] ? hrtimer_interrupt+0x128/0x2c0 [18880.059799] ? smp_apic_timer_interrupt+0xa6/0x270 [18880.059805] ? apic_timer_interrupt+0xf/0x20 [18880.059809] [18880.059811] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18880.059817] do_raw_spin_lock+0x177/0x190 [18880.059821] _raw_spin_lock+0x21/0x30 [18880.059826] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18880.061264] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18880.061758] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18880.062454] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18880.063512] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18880.063584] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18880.063591] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18880.063593] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18880.063605] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18880.063608] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18880.063612] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18880.063615] RDX: 00000000000074c0 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18880.063617] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18880.063619] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18880.063621] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18880.063625] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18880.063627] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18880.063629] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18880.063637] Call Trace: [18880.063642] [18880.063645] ? show_regs.cold.9+0x22/0x2f [18880.063655] ? watchdog_timer_fn.cold.6+0x62/0x10a [18880.062759] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18880.063820] ? woken_wake_function+0x30/0x30 [18880.063828] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18880.065193] kthread+0x1d1/0x200 [18880.065201] ? set_kthread_struct+0x70/0x70 [18880.065206] ret_from_fork+0x1f/0x30 [18880.066778] ? do_raw_spin_unlock+0x75/0x190 [18880.066794] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18880.067165] ? do_wait_intr+0xf0/0xf0 [18880.067182] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18880.105936] ? __hrtimer_run_queues+0x1c2/0x4d0 [18880.105953] ? watchdog+0x40/0x40 [18880.105960] ? hrtimer_interrupt+0x128/0x2c0 [18880.105964] ? smp_apic_timer_interrupt+0xa6/0x270 [18880.105972] ? apic_timer_interrupt+0xf/0x20 [18880.105976] [18880.105981] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18880.105988] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18880.105993] do_raw_spin_lock+0x177/0x190 [18880.105999] _raw_spin_lock+0x21/0x30 [18880.106007] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18880.106570] obd_mod_free+0x46/0x140 [ptlrpc] [18880.106768] mdc_commit_open+0x91/0xb0 [mdc] [18880.106804] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18880.107017] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18880.108379] after_reply+0xe2c/0x1780 [ptlrpc] [18880.108964] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18880.109170] ? __next_timer_interrupt+0x160/0x160 [18880.109181] ptlrpcd+0xa82/0xe40 [ptlrpc] [18880.124056] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18880.124346] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18880.124559] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18880.124687] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18880.124726] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18880.124755] mdc_intent_lock+0x353/0x7e0 [mdc] [18880.124782] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18880.124877] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18880.125131] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18880.125163] lmv_intent_open+0x338/0x1150 [lmv] [18880.125189] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18880.125275] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18880.125358] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18880.125382] ll_intent_lock+0xc5/0xaa0 [lustre] [18880.125469] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18880.125556] ll_lookup_it+0xe0e/0x2270 [lustre] [18880.125652] ? do_raw_spin_unlock+0x75/0x190 [18880.125660] ? _raw_spin_unlock+0x12/0x30 [18880.125666] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18880.125695] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18880.125717] ll_atomic_open+0x36a/0xd70 [lustre] [18880.125804] lookup_open+0x356/0xb40 [18880.125813] do_last+0x313/0xfc0 [18880.125818] ? nd_jump_root+0xe5/0x160 [18880.125823] ? path_init+0x437/0x520 [18880.125827] path_openat+0xf7/0x500 [18880.125832] ? do_raw_spin_unlock+0x75/0x190 [18880.125837] do_filp_open+0x99/0x140 [18880.125843] ? getname_flags+0x6e/0x330 [18880.144512] ? do_wait_intr+0xf0/0xf0 [18880.144533] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18880.144818] kthread+0x1d1/0x200 [18880.144824] ? set_kthread_struct+0x70/0x70 [18880.144829] ret_from_fork+0x1f/0x30 [18880.836887] ? __check_object_size+0xff/0x256 [18880.842765] ? do_raw_spin_unlock+0x75/0x190 [18880.852336] ? _raw_spin_unlock+0x12/0x30 [18880.859439] do_sys_openat2+0x2b4/0x410 [18880.860503] do_sys_open+0x73/0xa0 [18880.861493] __x64_sys_openat+0x24/0x30 [18880.864147] do_syscall_64+0xc1/0x440 [18880.867868] entry_SYSCALL_64_after_hwframe+0x49/0xae [18880.872311] RIP: 0033:0x7f53425a5332 [18880.875294] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18880.883409] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18880.885721] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18880.889632] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18880.892981] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18880.900119] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18880.907326] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18894.849792] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: [18894.858617] rcu: 1-....: (9256527 ticks this GP) idle=732/1/0x4000000000000000 softirq=131051/131051 fqs=96983 [18894.868904] rcu: 2-....: (9431953 ticks this GP) idle=b22/1/0x4000000000000000 softirq=137717/137717 fqs=96983 [18894.874224] (detected by 3, t=16800448 jiffies, g=274185, q=68213) [18894.877200] Sending NMI from CPU 3 to CPUs 1: [18894.885285] NMI backtrace for cpu 1 [18894.885292] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18894.885294] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18894.885296] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18894.885297] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18894.885300] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [18894.885303] RAX: 000000000000036d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18894.885304] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18894.885305] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18894.885307] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18894.885308] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18894.885310] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18894.885311] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18894.885312] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18894.885314] Call Trace: [18894.885314] [18894.885315] ? show_regs.cold.9+0x22/0x2f [18894.885317] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [18894.885318] ? nmi_cpu_backtrace_handler+0x15/0x40 [18894.885319] ? nmi_handle+0xbd/0x210 [18894.885321] ? default_do_nmi+0x4b/0x1b0 [18894.885322] ? do_nmi+0x280/0x380 [18894.885323] ? end_repeat_nmi+0x16/0x1f [18894.885324] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18894.885325] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18894.885326] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18894.885328] [18894.885329] do_raw_spin_lock+0x177/0x190 [18894.885330] _raw_spin_lock+0x21/0x30 [18894.885331] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18894.885333] ? do_raw_spin_unlock+0x75/0x190 [18894.885334] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18894.885335] ? do_wait_intr+0xf0/0xf0 [18894.885337] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18894.885338] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18894.885339] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18894.885341] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18894.885342] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18894.885343] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18894.885344] mdc_intent_lock+0x353/0x7e0 [mdc] [18894.885346] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18894.885347] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18894.885349] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18894.885350] lmv_intent_open+0x338/0x1150 [lmv] [18894.885351] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18894.885352] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18894.885354] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18894.885354] ll_intent_lock+0xc5/0xaa0 [lustre] [18894.885355] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18894.885356] ll_lookup_it+0xe0e/0x2270 [lustre] [18894.885357] ? do_raw_spin_unlock+0x75/0x190 [18894.885358] ? _raw_spin_unlock+0x12/0x30 [18894.885359] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18894.885359] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18894.885360] ll_atomic_open+0x36a/0xd70 [lustre] [18894.885361] lookup_open+0x356/0xb40 [18894.885362] do_last+0x313/0xfc0 [18894.885362] ? nd_jump_root+0xe5/0x160 [18894.885363] ? path_init+0x437/0x520 [18894.885363] path_openat+0xf7/0x500 [18894.885364] ? do_raw_spin_unlock+0x75/0x190 [18894.885364] do_filp_open+0x99/0x140 [18894.885365] ? getname_flags+0x6e/0x330 [18894.885366] ? __check_object_size+0xff/0x256 [18894.885366] ? do_raw_spin_unlock+0x75/0x190 [18894.885367] ? _raw_spin_unlock+0x12/0x30 [18894.885367] do_sys_openat2+0x2b4/0x410 [18894.885368] do_sys_open+0x73/0xa0 [18894.885368] __x64_sys_openat+0x24/0x30 [18894.885369] do_syscall_64+0xc1/0x440 [18894.885370] entry_SYSCALL_64_after_hwframe+0x49/0xae [18894.885370] RIP: 0033:0x7f53425a5332 [18894.885371] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18894.885372] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18894.885374] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18894.885375] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18894.885376] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18894.885377] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18894.885377] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18894.888827] Sending NMI from CPU 3 to CPUs 2: [18895.143213] NMI backtrace for cpu 2 [18895.143221] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18895.143223] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18895.143224] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18895.143226] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18895.143228] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 [18895.143231] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18895.143232] RDX: 0000000000004372 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18895.143234] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18895.143235] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18895.143237] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18895.143239] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18895.143240] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18895.143241] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18895.143243] Call Trace: [18895.143243] [18895.143245] ? show_regs.cold.9+0x22/0x2f [18895.143246] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [18895.143247] ? nmi_cpu_backtrace_handler+0x15/0x40 [18895.143248] ? nmi_handle+0xbd/0x210 [18895.143249] ? default_do_nmi+0x4b/0x1b0 [18895.143250] ? do_nmi+0x280/0x380 [18895.143251] ? end_repeat_nmi+0x16/0x1f [18895.143252] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18895.143253] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18895.143254] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18895.143255] [18895.143256] do_raw_spin_lock+0x177/0x190 [18895.143257] _raw_spin_lock+0x21/0x30 [18895.143258] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18895.143259] obd_mod_free+0x46/0x140 [ptlrpc] [18895.143260] mdc_commit_open+0x91/0xb0 [mdc] [18895.143262] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18895.143263] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18895.143264] after_reply+0xe2c/0x1780 [ptlrpc] [18895.143265] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18895.143267] ? __next_timer_interrupt+0x160/0x160 [18895.143268] ptlrpcd+0xa82/0xe40 [ptlrpc] [18895.143269] ? do_wait_intr+0xf0/0xf0 [18895.143270] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18895.143271] kthread+0x1d1/0x200 [18895.143272] ? set_kthread_struct+0x70/0x70 [18895.143273] ret_from_fork+0x1f/0x30 [18895.145479] rcu: rcu_sched kthread starved for 16259075 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [18895.253445] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [18895.256717] rcu: RCU grace-period kthread stack dump: [18895.259197] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [18895.264563] Call Trace: [18895.265478] __schedule+0x351/0xcb0 [18895.267060] schedule+0xc0/0x180 [18895.268822] schedule_timeout+0xb4/0x190 [18895.269832] ? do_raw_spin_unlock+0x75/0x190 [18895.271101] ? __next_timer_interrupt+0x160/0x160 [18895.272954] ? prepare_to_swait_event+0xf3/0x140 [18895.274842] rcu_gp_kthread+0x8dd/0x15b0 [18895.276368] ? force_qs_rnp+0x2c0/0x2c0 [18895.277974] kthread+0x1d1/0x200 [18895.279218] ? set_kthread_struct+0x70/0x70 [18895.280903] ret_from_fork+0x1f/0x30 [18895.282287] rcu: Stack dump where RCU GP kthread last ran: [18895.284450] Sending NMI from CPU 3 to CPUs 2: [18895.289772] NMI backtrace for cpu 2 [18895.289779] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18895.289781] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18895.289783] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18895.289785] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18895.289787] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 [18895.289789] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18895.289791] RDX: 00000000000068f1 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18895.289792] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18895.289794] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18895.289795] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18895.289797] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18895.289798] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18895.289800] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18895.289801] Call Trace: [18895.289802] [18895.289803] ? show_regs.cold.9+0x22/0x2f [18895.289805] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [18895.289806] ? nmi_cpu_backtrace_handler+0x15/0x40 [18895.289807] ? nmi_handle+0xbd/0x210 [18895.289808] ? default_do_nmi+0x4b/0x1b0 [18895.289809] ? do_nmi+0x280/0x380 [18895.289810] ? end_repeat_nmi+0x16/0x1f [18895.289811] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18895.289812] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18895.289814] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18895.289815] [18895.289815] do_raw_spin_lock+0x177/0x190 [18895.289816] _raw_spin_lock+0x21/0x30 [18895.289817] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18895.289818] obd_mod_free+0x46/0x140 [ptlrpc] [18895.289820] mdc_commit_open+0x91/0xb0 [mdc] [18895.289821] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18895.289822] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18895.289823] after_reply+0xe2c/0x1780 [ptlrpc] [18895.289824] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18895.289825] ? __next_timer_interrupt+0x160/0x160 [18895.289826] ptlrpcd+0xa82/0xe40 [ptlrpc] [18895.289827] ? do_wait_intr+0xf0/0xf0 [18895.289828] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18895.289829] kthread+0x1d1/0x200 [18895.289830] ? set_kthread_struct+0x70/0x70 [18895.289831] ret_from_fork+0x1f/0x30 [18900.052034] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [18900.058876] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18900.097661] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18900.104953] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18900.118304] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18900.129101] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18900.136189] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18900.153929] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18900.159965] RAX: 000000000000266f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18900.166002] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [18900.170169] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18900.175674] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18900.180260] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18900.183737] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18900.187238] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18900.189725] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18900.193365] Call Trace: [18900.194149] [18900.194807] ? show_regs.cold.9+0x22/0x2f [18900.196255] ? watchdog_timer_fn.cold.6+0x62/0x10a [18900.197645] ? __hrtimer_run_queues+0x1c2/0x4d0 [18900.201617] ? watchdog+0x40/0x40 [18900.203158] ? hrtimer_interrupt+0x128/0x2c0 [18900.205264] ? smp_apic_timer_interrupt+0xa6/0x270 [18900.207560] ? apic_timer_interrupt+0xf/0x20 [18900.210156] [18900.211254] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18900.214101] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18900.221540] ? newidle_balance.isra.93+0x207/0x580 [18900.223544] do_raw_spin_lock+0x177/0x190 [18900.225888] _raw_spin_lock+0x21/0x30 [18900.227321] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18900.229910] process_one_work+0x2c8/0x700 [18900.231475] worker_thread+0x296/0x6e0 [18900.233204] ? rescuer_thread+0x570/0x570 [18900.235098] kthread+0x1d1/0x200 [18900.236329] ? set_kthread_struct+0x70/0x70 [18900.238162] ret_from_fork+0x1f/0x30 [18902.501028] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16265s! [18902.513770] Showing busy workqueues and worker pools: [18902.522210] workqueue events: flags=0x0 [18902.529547] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18902.535265] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18902.544874] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18902.548865] pending: cpu_latency_work [ptlrpc] [18902.551110] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18902.561091] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18902.571190] workqueue events_long: flags=0x0 [18902.574380] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18902.584213] pending: lu_objects_flush [obdclass] [18902.590050] workqueue events_unbound: flags=0x2 [18902.600272] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18902.603082] pending: flush_memcg_stats_dwork, idle_cull_fn [18902.611243] workqueue events_power_efficient: flags=0x80 [18902.614762] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18902.623020] pending: check_lifetime, do_cache_clean [18902.627313] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18902.631898] pending: neigh_periodic_work, neigh_periodic_work [18902.642045] workqueue mm_percpu_wq: flags=0x8 [18902.643445] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18902.650968] pending: vmstat_update, lru_add_drain_per_cpu [18902.653491] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18902.655983] pending: vmstat_update, lru_add_drain_per_cpu [18902.660995] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18902.665440] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18902.672210] workqueue xprtiod: flags=0xa [18902.674327] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18902.683109] pending: xprt_autoclose, xs_error_handle [18902.687065] workqueue ipv6_addrconf: flags=0x40008 [18902.695032] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18902.699562] pending: addrconf_verify_work [18902.703344] workqueue knbd0-recv: flags=0x1a [18902.709094] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18902.710989] in-flight: 651:recv_work [18902.713753] workqueue ptlrpc_pinger: flags=0x2 [18902.717279] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18902.720080] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18902.724588] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16265s workers=4 idle: 15979 31824 8 [18902.730378] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18893s workers=2 idle: 654 [18920.055010] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18920.057227] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18920.059465] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [18920.059477] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18920.059541] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18920.059547] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18920.059550] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18920.060079] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18920.060083] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18920.060087] RAX: 0000000000007d08 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18920.060089] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18920.060091] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18920.060093] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18920.060095] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18920.060098] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18920.060100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18920.060102] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18920.060110] Call Trace: [18920.060115] [18920.060120] ? show_regs.cold.9+0x22/0x2f [18920.060128] ? watchdog_timer_fn.cold.6+0x62/0x10a [18920.060133] ? __hrtimer_run_queues+0x1c2/0x4d0 [18920.060138] ? watchdog+0x40/0x40 [18920.060142] ? hrtimer_interrupt+0x128/0x2c0 [18920.060146] ? smp_apic_timer_interrupt+0xa6/0x270 [18920.060164] ? apic_timer_interrupt+0xf/0x20 [18920.060169] [18920.060172] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18920.060176] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18920.060183] do_raw_spin_lock+0x177/0x190 [18920.060187] _raw_spin_lock+0x21/0x30 [18920.060192] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18920.060699] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18920.060825] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18920.060830] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18920.060832] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18920.060840] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18920.060844] RSP: 0018:ffffbb5381503500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [18920.060848] RAX: 00000000000029a0 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18920.060851] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18920.060854] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18920.060856] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18920.060859] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18920.060862] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18920.060865] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18920.060868] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18920.060874] Call Trace: [18920.060876] [18920.060879] ? show_regs.cold.9+0x22/0x2f [18920.060885] ? watchdog_timer_fn.cold.6+0x62/0x10a [18920.060890] ? __hrtimer_run_queues+0x1c2/0x4d0 [18920.060893] ? watchdog+0x40/0x40 [18920.060898] ? hrtimer_interrupt+0x128/0x2c0 [18920.060902] ? smp_apic_timer_interrupt+0xa6/0x270 [18920.060907] ? apic_timer_interrupt+0xf/0x20 [18920.060912] [18920.060915] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18920.060921] do_raw_spin_lock+0x177/0x190 [18920.060925] _raw_spin_lock+0x21/0x30 [18920.060929] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18920.061407] ? do_raw_spin_unlock+0x75/0x190 [18920.061121] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18920.061850] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18920.062075] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18920.061412] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18920.062536] ? do_wait_intr+0xf0/0xf0 [18920.062545] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18920.062291] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18920.063122] ? woken_wake_function+0x30/0x30 [18920.063129] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18920.063477] kthread+0x1d1/0x200 [18920.063484] ? set_kthread_struct+0x70/0x70 [18920.063488] ret_from_fork+0x1f/0x30 [18920.068444] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18920.068517] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18920.068525] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18920.068529] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18920.068542] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18920.068546] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18920.068562] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18920.068564] RDX: 0000000000007ac5 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18920.068567] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18920.068569] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18920.068572] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18920.068575] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18920.068578] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18920.068580] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18920.068588] Call Trace: [18920.068594] [18920.068600] ? show_regs.cold.9+0x22/0x2f [18920.068610] ? watchdog_timer_fn.cold.6+0x62/0x10a [18920.068616] ? __hrtimer_run_queues+0x1c2/0x4d0 [18920.068620] ? watchdog+0x40/0x40 [18920.068624] ? hrtimer_interrupt+0x128/0x2c0 [18920.068628] ? smp_apic_timer_interrupt+0xa6/0x270 [18920.068636] ? apic_timer_interrupt+0xf/0x20 [18920.068643] [18920.074727] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18920.075013] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18920.077444] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18920.077557] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18920.077594] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18920.077624] mdc_intent_lock+0x353/0x7e0 [mdc] [18920.077653] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18920.077742] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18920.077968] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18920.077995] lmv_intent_open+0x338/0x1150 [lmv] [18920.078025] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18920.078322] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18920.078403] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18920.078427] ll_intent_lock+0xc5/0xaa0 [lustre] [18920.078521] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18920.078606] ll_lookup_it+0xe0e/0x2270 [lustre] [18920.078693] ? do_raw_spin_unlock+0x75/0x190 [18920.078700] ? _raw_spin_unlock+0x12/0x30 [18920.078707] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18920.142701] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18920.142724] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18920.142729] do_raw_spin_lock+0x177/0x190 [18920.142736] _raw_spin_lock+0x21/0x30 [18920.142745] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18920.161143] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18920.161202] ll_atomic_open+0x36a/0xd70 [lustre] [18920.162511] lookup_open+0x356/0xb40 [18920.162523] do_last+0x313/0xfc0 [18920.162529] ? nd_jump_root+0xe5/0x160 [18920.162534] ? path_init+0x437/0x520 [18920.162539] path_openat+0xf7/0x500 [18920.162544] ? do_raw_spin_unlock+0x75/0x190 [18920.162551] do_filp_open+0x99/0x140 [18920.162558] ? getname_flags+0x6e/0x330 [18920.162563] ? __check_object_size+0xff/0x256 [18920.162570] ? do_raw_spin_unlock+0x75/0x190 [18920.162574] ? _raw_spin_unlock+0x12/0x30 [18920.162581] do_sys_openat2+0x2b4/0x410 [18920.162586] do_sys_open+0x73/0xa0 [18920.162590] __x64_sys_openat+0x24/0x30 [18920.162593] do_syscall_64+0xc1/0x440 [18920.162600] entry_SYSCALL_64_after_hwframe+0x49/0xae [18920.162607] RIP: 0033:0x7f53425a5332 [18920.162614] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18920.162619] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18920.162624] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18920.162627] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18920.162629] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18920.162631] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18920.162634] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18920.972980] obd_mod_free+0x46/0x140 [ptlrpc] [18920.979645] mdc_commit_open+0x91/0xb0 [mdc] [18920.986396] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18920.990334] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18920.996935] after_reply+0xe2c/0x1780 [ptlrpc] [18921.001776] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18921.008798] ? __next_timer_interrupt+0x160/0x160 [18921.013436] ptlrpcd+0xa82/0xe40 [ptlrpc] [18921.016464] ? do_wait_intr+0xf0/0xf0 [18921.017772] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18921.022957] kthread+0x1d1/0x200 [18921.024266] ? set_kthread_struct+0x70/0x70 [18921.027986] ret_from_fork+0x1f/0x30 [18928.052019] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [18928.063556] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18928.103148] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18928.113197] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18928.121340] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18928.125923] RIP: 0010:__pv_queued_spin_lock_slowpath+0x320/0x6d0 [18928.131818] Code: 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 <48> 83 05 18 d4 dc 02 01 eb e0 48 83 05 06 d4 dc 02 01 48 83 05 be [18928.143859] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18928.149441] RAX: 0000000000001936 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18928.152958] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [18928.157982] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18928.162143] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18928.169241] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18928.175251] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18928.180512] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18928.183684] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18928.187702] Call Trace: [18928.189573] [18928.191216] ? show_regs.cold.9+0x22/0x2f [18928.193513] ? watchdog_timer_fn.cold.6+0x62/0x10a [18928.196911] ? __hrtimer_run_queues+0x1c2/0x4d0 [18928.201153] ? watchdog+0x40/0x40 [18928.203084] ? hrtimer_interrupt+0x128/0x2c0 [18928.208913] ? smp_apic_timer_interrupt+0xa6/0x270 [18928.215999] ? apic_timer_interrupt+0xf/0x20 [18928.220121] [18928.221588] ? __pv_queued_spin_lock_slowpath+0x320/0x6d0 [18928.224468] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18928.230384] ? newidle_balance.isra.93+0x207/0x580 [18928.233608] do_raw_spin_lock+0x177/0x190 [18928.237440] _raw_spin_lock+0x21/0x30 [18928.239632] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18928.243178] process_one_work+0x2c8/0x700 [18928.245646] worker_thread+0x296/0x6e0 [18928.247722] ? rescuer_thread+0x570/0x570 [18928.252522] kthread+0x1d1/0x200 [18928.255943] ? set_kthread_struct+0x70/0x70 [18928.258235] ret_from_fork+0x1f/0x30 [18933.218083] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16296s! [18933.225109] Showing busy workqueues and worker pools: [18933.227583] workqueue events: flags=0x0 [18933.229458] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18933.234986] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18933.241340] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18933.248943] pending: cpu_latency_work [ptlrpc] [18933.253251] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18933.262638] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18933.274075] workqueue events_long: flags=0x0 [18933.277429] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18933.279100] pending: lu_objects_flush [obdclass] [18933.282267] workqueue events_unbound: flags=0x2 [18933.285256] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18933.290448] pending: flush_memcg_stats_dwork, idle_cull_fn [18933.295060] workqueue events_power_efficient: flags=0x80 [18933.299197] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18933.301472] pending: check_lifetime, do_cache_clean [18933.305150] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18933.308533] pending: neigh_periodic_work, neigh_periodic_work [18933.312819] workqueue mm_percpu_wq: flags=0x8 [18933.314721] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18933.322869] pending: vmstat_update, lru_add_drain_per_cpu [18933.328163] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18933.333499] pending: vmstat_update, lru_add_drain_per_cpu [18933.335827] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18933.340536] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18933.343378] workqueue xprtiod: flags=0xa [18933.345041] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18933.347831] pending: xprt_autoclose, xs_error_handle [18933.351888] workqueue ipv6_addrconf: flags=0x40008 [18933.355310] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18933.358571] pending: addrconf_verify_work [18933.359901] workqueue knbd0-recv: flags=0x1a [18933.361595] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18933.364142] in-flight: 651:recv_work [18933.365759] workqueue ptlrpc_pinger: flags=0x2 [18933.367801] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18933.370470] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18933.373243] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16296s workers=4 idle: 15979 31824 8 [18933.377872] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18923s workers=2 idle: 654 [18948.055916] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18948.057711] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18948.058412] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [18948.058419] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18948.058494] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18948.058499] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18948.058502] RIP: 0010:native_safe_halt+0x2e/0x50 [18948.058514] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18948.058518] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18948.058523] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18948.058525] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18948.058528] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18948.058530] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18948.058532] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18948.058535] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18948.058538] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18948.058540] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18948.058548] Call Trace: [18948.058554] [18948.058559] ? show_regs.cold.9+0x22/0x2f [18948.058569] ? watchdog_timer_fn.cold.6+0x62/0x10a [18948.058576] ? __hrtimer_run_queues+0x1c2/0x4d0 [18948.058580] ? watchdog+0x40/0x40 [18948.058584] ? hrtimer_interrupt+0x128/0x2c0 [18948.058589] ? smp_apic_timer_interrupt+0xa6/0x270 [18948.058595] ? apic_timer_interrupt+0xf/0x20 [18948.058600] [18948.058603] ? native_safe_halt+0x2e/0x50 [18948.058608] kvm_wait+0xcd/0xe0 [18948.058615] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18948.058623] do_raw_spin_lock+0x177/0x190 [18948.058628] _raw_spin_lock+0x21/0x30 [18948.058632] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18948.059160] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18948.059356] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18948.059571] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18948.061249] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18948.061364] Modules linked in: lustre(O) [18948.061466] ? woken_wake_function+0x30/0x30 [18948.061476] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18948.061703] kthread+0x1d1/0x200 [18948.061709] ? set_kthread_struct+0x70/0x70 [18948.061713] ret_from_fork+0x1f/0x30 [18948.065539] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18948.065611] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18948.065619] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18948.065736] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [18948.065752] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [18948.065757] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18948.065762] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18948.065764] RDX: 000000000000402b RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18948.065767] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18948.065769] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18948.065771] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18948.065774] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18948.065777] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18948.065780] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18948.065789] Call Trace: [18948.065794] [18948.065800] ? show_regs.cold.9+0x22/0x2f [18948.065810] ? watchdog_timer_fn.cold.6+0x62/0x10a [18948.065816] ? __hrtimer_run_queues+0x1c2/0x4d0 [18948.065820] ? watchdog+0x40/0x40 [18948.065825] ? hrtimer_interrupt+0x128/0x2c0 [18948.065829] ? smp_apic_timer_interrupt+0xa6/0x270 [18948.065837] ? apic_timer_interrupt+0xf/0x20 [18948.065842] [18948.065845] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [18948.065850] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18948.065856] do_raw_spin_lock+0x177/0x190 [18948.065861] _raw_spin_lock+0x21/0x30 [18948.065867] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18948.072158] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18948.072231] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18948.072237] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18948.072240] RIP: 0010:native_safe_halt+0x2e/0x50 [18948.072253] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18948.072257] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18948.072261] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18948.072264] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18948.072265] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18948.072268] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18948.072270] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18948.072274] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18948.072277] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18948.072280] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18948.072289] Call Trace: [18948.072295] [18948.072305] ? show_regs.cold.9+0x22/0x2f [18948.072321] ? watchdog_timer_fn.cold.6+0x62/0x10a [18948.072327] ? __hrtimer_run_queues+0x1c2/0x4d0 [18948.072332] ? watchdog+0x40/0x40 [18948.072336] ? hrtimer_interrupt+0x128/0x2c0 [18948.072340] ? smp_apic_timer_interrupt+0xa6/0x270 [18948.072346] ? apic_timer_interrupt+0xf/0x20 [18948.072350] [18948.072352] ? native_safe_halt+0x2e/0x50 [18948.072356] kvm_wait+0xcd/0xe0 [18948.072363] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18948.072372] do_raw_spin_lock+0x177/0x190 [18948.072376] _raw_spin_lock+0x21/0x30 [18948.072380] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18948.072862] ? do_raw_spin_unlock+0x75/0x190 [18948.072868] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18948.073116] ? do_wait_intr+0xf0/0xf0 [18948.073122] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18948.073249] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18948.073578] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18948.073831] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18948.073932] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18948.073959] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18948.073979] mdc_intent_lock+0x353/0x7e0 [mdc] [18948.073996] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18948.076863] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18948.077253] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18948.077283] lmv_intent_open+0x338/0x1150 [lmv] [18948.077303] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18948.077725] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18948.077781] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18948.118021] obd_mod_free+0x46/0x140 [ptlrpc] [18948.118334] mdc_commit_open+0x91/0xb0 [mdc] [18948.118379] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18948.118628] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18948.118971] after_reply+0xe2c/0x1780 [ptlrpc] [18948.119231] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18948.119585] ? __next_timer_interrupt+0x160/0x160 [18948.119596] ptlrpcd+0xa82/0xe40 [ptlrpc] [18948.119912] ? do_wait_intr+0xf0/0xf0 [18948.119922] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18948.125298] ll_intent_lock+0xc5/0xaa0 [lustre] [18948.125441] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18948.125603] ll_lookup_it+0xe0e/0x2270 [lustre] [18948.125759] ? do_raw_spin_unlock+0x75/0x190 [18948.125765] ? _raw_spin_unlock+0x12/0x30 [18948.125770] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18948.125789] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18948.125803] ll_atomic_open+0x36a/0xd70 [lustre] [18948.131992] kthread+0x1d1/0x200 [18948.134038] lookup_open+0x356/0xb40 [18948.134052] do_last+0x313/0xfc0 [18948.134056] ? nd_jump_root+0xe5/0x160 [18948.134060] ? path_init+0x437/0x520 [18948.134063] path_openat+0xf7/0x500 [18948.134067] ? do_raw_spin_unlock+0x75/0x190 [18948.134071] do_filp_open+0x99/0x140 [18948.134075] ? getname_flags+0x6e/0x330 [18948.134078] ? __check_object_size+0xff/0x256 [18948.134082] ? do_raw_spin_unlock+0x75/0x190 [18948.134085] ? _raw_spin_unlock+0x12/0x30 [18948.134089] do_sys_openat2+0x2b4/0x410 [18948.134091] do_sys_open+0x73/0xa0 [18948.134094] __x64_sys_openat+0x24/0x30 [18948.134097] do_syscall_64+0xc1/0x440 [18948.134101] entry_SYSCALL_64_after_hwframe+0x49/0xae [18948.134106] RIP: 0033:0x7f53425a5332 [18948.134113] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18948.134117] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18948.134120] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18948.134122] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18948.134124] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18948.134125] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18948.134127] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18948.637024] ? set_kthread_struct+0x70/0x70 [18948.638250] ret_from_fork+0x1f/0x30 [18960.052017] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [18960.054829] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18960.083145] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18960.098850] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18960.102520] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18960.105094] RIP: 0010:native_safe_halt+0x2e/0x50 [18960.107015] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18960.113874] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18960.117087] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18960.120369] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [18960.122701] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18960.125216] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18960.127622] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18960.130124] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18960.133314] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18960.135583] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18960.139410] Call Trace: [18960.140585] [18960.141766] ? show_regs.cold.9+0x22/0x2f [18960.142919] ? watchdog_timer_fn.cold.6+0x62/0x10a [18960.144667] ? __hrtimer_run_queues+0x1c2/0x4d0 [18960.146367] ? watchdog+0x40/0x40 [18960.148086] ? hrtimer_interrupt+0x128/0x2c0 [18960.149601] ? smp_apic_timer_interrupt+0xa6/0x270 [18960.150929] ? apic_timer_interrupt+0xf/0x20 [18960.152814] [18960.153574] ? native_safe_halt+0x2e/0x50 [18960.155347] kvm_wait+0xcd/0xe0 [18960.156338] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18960.158419] ? newidle_balance.isra.93+0x207/0x580 [18960.160628] do_raw_spin_lock+0x177/0x190 [18960.161912] _raw_spin_lock+0x21/0x30 [18960.163760] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18960.166402] process_one_work+0x2c8/0x700 [18960.168123] worker_thread+0x296/0x6e0 [18960.169504] ? rescuer_thread+0x570/0x570 [18960.171390] kthread+0x1d1/0x200 [18960.172681] ? set_kthread_struct+0x70/0x70 [18960.173894] ret_from_fork+0x1f/0x30 [18963.423635] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16326s! [18963.439963] Showing busy workqueues and worker pools: [18963.441410] workqueue events: flags=0x0 [18963.442288] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18963.456080] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18963.468923] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18963.475168] pending: cpu_latency_work [ptlrpc] [18963.478203] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18963.484791] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18963.489346] workqueue events_long: flags=0x0 [18963.492921] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18963.495748] pending: lu_objects_flush [obdclass] [18963.497366] workqueue events_unbound: flags=0x2 [18963.498682] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18963.503291] pending: flush_memcg_stats_dwork, idle_cull_fn [18963.504977] workqueue events_power_efficient: flags=0x80 [18963.508353] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18963.512292] pending: check_lifetime, do_cache_clean [18963.513778] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18963.515902] pending: neigh_periodic_work, neigh_periodic_work [18963.517628] workqueue mm_percpu_wq: flags=0x8 [18963.519927] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18963.523052] pending: vmstat_update, lru_add_drain_per_cpu [18963.524760] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18963.526824] pending: vmstat_update, lru_add_drain_per_cpu [18963.531309] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18963.534689] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18963.537742] workqueue xprtiod: flags=0xa [18963.539378] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18963.541834] pending: xprt_autoclose, xs_error_handle [18963.543958] workqueue ipv6_addrconf: flags=0x40008 [18963.545984] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18963.549221] pending: addrconf_verify_work [18963.550768] workqueue knbd0-recv: flags=0x1a [18963.552805] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18963.555468] in-flight: 651:recv_work [18963.557212] workqueue ptlrpc_pinger: flags=0x2 [18963.558642] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18963.561050] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18963.563902] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16326s workers=4 idle: 15979 31824 8 [18963.567838] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18954s workers=2 idle: 654 [18976.055024] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [18976.057025] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [18976.057667] Modules linked in: lustre(O) [18976.058129] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [18976.058133] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18976.058253] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18976.058259] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18976.058262] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18976.058276] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18976.058281] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18976.058286] RAX: 0000000000005f2a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18976.058288] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [18976.058291] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [18976.058293] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [18976.058297] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [18976.058300] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [18976.058302] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18976.058304] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [18976.058312] Call Trace: [18976.058538] [18976.058569] ? show_regs.cold.9+0x22/0x2f [18976.058582] ? watchdog_timer_fn.cold.6+0x62/0x10a [18976.058587] ? __hrtimer_run_queues+0x1c2/0x4d0 [18976.058591] ? watchdog+0x40/0x40 [18976.058595] ? hrtimer_interrupt+0x128/0x2c0 [18976.058599] ? smp_apic_timer_interrupt+0xa6/0x270 [18976.058606] ? apic_timer_interrupt+0xf/0x20 [18976.058611] [18976.058614] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18976.058618] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18976.058624] do_raw_spin_lock+0x177/0x190 [18976.058628] _raw_spin_lock+0x21/0x30 [18976.058632] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [18976.058910] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [18976.059057] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [18976.059241] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [18976.059718] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [18976.059962] ? woken_wake_function+0x30/0x30 [18976.059969] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [18976.060282] kthread+0x1d1/0x200 [18976.060294] ? set_kthread_struct+0x70/0x70 [18976.060298] ret_from_fork+0x1f/0x30 [18976.064823] Modules linked in: [18976.066264] osp(O) ofd(O) lod(O) mdt(O) mdd(O) [18976.072668] lustre(O) [18976.114289] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18976.114363] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18976.114370] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18976.114372] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18976.114385] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18976.114388] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18976.114393] RAX: 0000000000006d2f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18976.114395] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [18976.114397] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [18976.114399] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [18976.114402] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [18976.114405] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [18976.114407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18976.114410] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [18976.114417] Call Trace: [18976.114422] [18976.114427] ? show_regs.cold.9+0x22/0x2f [18976.114436] ? watchdog_timer_fn.cold.6+0x62/0x10a [18976.114442] ? __hrtimer_run_queues+0x1c2/0x4d0 [18976.114445] ? watchdog+0x40/0x40 [18976.114449] ? hrtimer_interrupt+0x128/0x2c0 [18976.114454] ? smp_apic_timer_interrupt+0xa6/0x270 [18976.114462] ? apic_timer_interrupt+0xf/0x20 [18976.114466] [18976.114470] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18976.114477] do_raw_spin_lock+0x177/0x190 [18976.114482] _raw_spin_lock+0x21/0x30 [18976.114487] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [18976.114745] ? do_raw_spin_unlock+0x75/0x190 [18976.114751] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [18976.115230] ? do_wait_intr+0xf0/0xf0 [18976.115240] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [18976.122857] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18976.122931] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18976.122938] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18976.122942] RIP: 0010:native_safe_halt+0x2e/0x50 [18976.122957] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [18976.127665] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [18976.127970] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18976.128422] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18976.128507] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18976.128543] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [18976.128574] mdc_intent_lock+0x353/0x7e0 [mdc] [18976.128603] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18976.128686] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [18976.128905] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [18976.128935] lmv_intent_open+0x338/0x1150 [lmv] [18976.129708] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18976.129802] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18976.129886] lmv_intent_lock+0x1dd/0x4c0 [lmv] [18976.129913] ll_intent_lock+0xc5/0xaa0 [lustre] [18976.129999] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [18976.132277] ll_lookup_it+0xe0e/0x2270 [lustre] [18976.132367] ? do_raw_spin_unlock+0x75/0x190 [18976.132375] ? _raw_spin_unlock+0x12/0x30 [18976.132382] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [18976.132406] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [18976.132431] ll_atomic_open+0x36a/0xd70 [lustre] [18976.132554] lookup_open+0x356/0xb40 [18976.132562] do_last+0x313/0xfc0 [18976.132567] ? nd_jump_root+0xe5/0x160 [18976.132572] ? path_init+0x437/0x520 [18976.132577] path_openat+0xf7/0x500 [18976.132581] ? do_raw_spin_unlock+0x75/0x190 [18976.132586] do_filp_open+0x99/0x140 [18976.132593] ? getname_flags+0x6e/0x330 [18976.132597] ? __check_object_size+0xff/0x256 [18976.132604] ? do_raw_spin_unlock+0x75/0x190 [18976.132608] ? _raw_spin_unlock+0x12/0x30 [18976.132613] do_sys_openat2+0x2b4/0x410 [18976.132618] do_sys_open+0x73/0xa0 [18976.132622] __x64_sys_openat+0x24/0x30 [18976.132625] do_syscall_64+0xc1/0x440 [18976.132632] entry_SYSCALL_64_after_hwframe+0x49/0xae [18976.132638] RIP: 0033:0x7f53425a5332 [18976.132689] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [18976.132693] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [18976.132699] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [18976.132701] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [18976.132704] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [18976.132706] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [18976.132709] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [18976.390671] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [18976.395472] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [18976.398094] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [18976.400658] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [18976.403146] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [18976.405431] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [18976.407746] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [18976.412062] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18976.415282] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [18976.418252] Call Trace: [18976.421380] [18976.422045] ? show_regs.cold.9+0x22/0x2f [18976.424311] ? watchdog_timer_fn.cold.6+0x62/0x10a [18976.426418] ? __hrtimer_run_queues+0x1c2/0x4d0 [18976.428927] ? watchdog+0x40/0x40 [18976.429869] ? hrtimer_interrupt+0x128/0x2c0 [18976.431818] ? smp_apic_timer_interrupt+0xa6/0x270 [18976.433690] ? apic_timer_interrupt+0xf/0x20 [18976.435709] [18976.436860] ? native_safe_halt+0x2e/0x50 [18976.438457] kvm_wait+0xcd/0xe0 [18976.439804] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [18976.443060] do_raw_spin_lock+0x177/0x190 [18976.444674] _raw_spin_lock+0x21/0x30 [18976.445924] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [18976.449622] obd_mod_free+0x46/0x140 [ptlrpc] [18976.451451] mdc_commit_open+0x91/0xb0 [mdc] [18976.452993] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [18976.454684] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [18976.456580] after_reply+0xe2c/0x1780 [ptlrpc] [18976.458280] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [18976.460743] ? __next_timer_interrupt+0x160/0x160 [18976.463652] ptlrpcd+0xa82/0xe40 [ptlrpc] [18976.465582] ? do_wait_intr+0xf0/0xf0 [18976.466909] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [18976.468860] kthread+0x1d1/0x200 [18976.470036] ? set_kthread_struct+0x70/0x70 [18976.471782] ret_from_fork+0x1f/0x30 [18988.052117] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [18988.056659] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [18988.085226] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [18988.098516] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [18988.110093] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [18988.113781] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18988.120093] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [18988.128424] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [18988.136157] RAX: 0000000000004a8c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [18988.141402] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [18988.144942] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [18988.147850] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [18988.151781] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [18988.158287] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [18988.164388] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [18988.172201] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [18988.180638] Call Trace: [18988.183284] [18988.184053] ? show_regs.cold.9+0x22/0x2f [18988.192656] ? watchdog_timer_fn.cold.6+0x62/0x10a [18988.195237] ? __hrtimer_run_queues+0x1c2/0x4d0 [18988.200844] ? watchdog+0x40/0x40 [18988.203499] ? hrtimer_interrupt+0x128/0x2c0 [18988.204868] ? smp_apic_timer_interrupt+0xa6/0x270 [18988.213135] ? apic_timer_interrupt+0xf/0x20 [18988.215329] [18988.216323] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [18988.220904] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [18988.225229] ? newidle_balance.isra.93+0x207/0x580 [18988.228092] do_raw_spin_lock+0x177/0x190 [18988.229501] _raw_spin_lock+0x21/0x30 [18988.230896] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [18988.236902] process_one_work+0x2c8/0x700 [18988.241461] worker_thread+0x296/0x6e0 [18988.244834] ? rescuer_thread+0x570/0x570 [18988.246315] kthread+0x1d1/0x200 [18988.248256] ? set_kthread_struct+0x70/0x70 [18988.249477] ret_from_fork+0x1f/0x30 [18993.633106] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16356s! [18993.642296] Showing busy workqueues and worker pools: [18993.644850] workqueue events: flags=0x0 [18993.645856] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [18993.651086] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [18993.656125] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18993.663511] pending: cpu_latency_work [ptlrpc] [18993.671397] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [18993.675025] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [18993.678798] workqueue events_long: flags=0x0 [18993.679977] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [18993.686969] pending: lu_objects_flush [obdclass] [18993.689654] workqueue events_unbound: flags=0x2 [18993.691257] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [18993.696527] pending: flush_memcg_stats_dwork, idle_cull_fn [18993.699479] workqueue events_power_efficient: flags=0x80 [18993.701810] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18993.704975] pending: check_lifetime, do_cache_clean [18993.706856] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18993.713793] pending: neigh_periodic_work, neigh_periodic_work [18993.719671] workqueue mm_percpu_wq: flags=0x8 [18993.721850] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18993.724663] pending: vmstat_update, lru_add_drain_per_cpu [18993.727045] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [18993.729425] pending: vmstat_update, lru_add_drain_per_cpu [18993.733374] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [18993.737376] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [18993.740354] workqueue xprtiod: flags=0xa [18993.741822] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [18993.746809] pending: xprt_autoclose, xs_error_handle [18993.749039] workqueue ipv6_addrconf: flags=0x40008 [18993.750958] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [18993.753235] pending: addrconf_verify_work [18993.755679] workqueue knbd0-recv: flags=0x1a [18993.759306] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [18993.762423] in-flight: 651:recv_work [18993.766387] workqueue ptlrpc_pinger: flags=0x2 [18993.768043] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [18993.773059] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [18993.777080] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16356s workers=4 idle: 15979 31824 8 [18993.785241] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=18984s workers=2 idle: 654 [19004.055022] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [19004.057733] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [19004.058037] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19004.058045] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19004.058159] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19004.058165] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19004.058169] RIP: 0010:native_safe_halt+0x2e/0x50 [19004.058182] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19004.058186] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19004.058191] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19004.058194] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19004.058197] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19004.058199] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19004.058202] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19004.058206] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19004.058211] Modules linked in: [19004.058210] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19004.058213] lustre(O) [19004.058215] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19004.058222] Call Trace: [19004.058228] [19004.058232] ? show_regs.cold.9+0x22/0x2f [19004.058242] ? watchdog_timer_fn.cold.6+0x62/0x10a [19004.058248] ? __hrtimer_run_queues+0x1c2/0x4d0 [19004.058253] ? watchdog+0x40/0x40 [19004.058257] ? hrtimer_interrupt+0x128/0x2c0 [19004.058261] ? smp_apic_timer_interrupt+0xa6/0x270 [19004.058267] ? apic_timer_interrupt+0xf/0x20 [19004.058271] [19004.058274] ? native_safe_halt+0x2e/0x50 [19004.058277] kvm_wait+0xcd/0xe0 [19004.058284] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19004.058291] do_raw_spin_lock+0x177/0x190 [19004.058295] _raw_spin_lock+0x21/0x30 [19004.058299] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19004.058603] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19004.058810] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19004.059351] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19004.059594] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19004.059830] ? woken_wake_function+0x30/0x30 [19004.059840] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19004.060473] kthread+0x1d1/0x200 [19004.060480] ? set_kthread_struct+0x70/0x70 [19004.060484] ret_from_fork+0x1f/0x30 [19004.060877] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19004.060944] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19004.060951] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19004.060955] RIP: 0010:native_safe_halt+0x2e/0x50 [19004.060963] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19004.060967] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19004.060971] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19004.060973] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19004.060975] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19004.060977] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19004.060979] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19004.060982] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19004.060985] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19004.060986] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19004.060994] Call Trace: [19004.060998] [19004.061001] ? show_regs.cold.9+0x22/0x2f [19004.061010] ? watchdog_timer_fn.cold.6+0x62/0x10a [19004.061015] ? __hrtimer_run_queues+0x1c2/0x4d0 [19004.061020] ? watchdog+0x40/0x40 [19004.061025] ? hrtimer_interrupt+0x128/0x2c0 [19004.061030] ? smp_apic_timer_interrupt+0xa6/0x270 [19004.061051] ? apic_timer_interrupt+0xf/0x20 [19004.061056] [19004.061059] ? native_safe_halt+0x2e/0x50 [19004.061063] kvm_wait+0xcd/0xe0 [19004.061070] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19004.061077] do_raw_spin_lock+0x177/0x190 [19004.061082] _raw_spin_lock+0x21/0x30 [19004.061086] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19004.061302] obd_mod_free+0x46/0x140 [ptlrpc] [19004.061496] mdc_commit_open+0x91/0xb0 [mdc] [19004.061535] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19004.069364] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [19004.097188] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19004.097449] after_reply+0xe2c/0x1780 [ptlrpc] [19004.097648] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19004.097845] ? __next_timer_interrupt+0x160/0x160 [19004.097855] ptlrpcd+0xa82/0xe40 [ptlrpc] [19004.098057] ? do_wait_intr+0xf0/0xf0 [19004.098066] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19004.098265] kthread+0x1d1/0x200 [19004.098270] ? set_kthread_struct+0x70/0x70 [19004.098274] ret_from_fork+0x1f/0x30 [19004.380393] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19004.398024] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19004.402529] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19004.407606] RIP: 0010:native_safe_halt+0x2e/0x50 [19004.409502] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19004.422619] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19004.427834] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19004.431753] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19004.434730] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19004.437555] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19004.439844] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19004.441591] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19004.451373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19004.454528] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19004.457598] Call Trace: [19004.459485] [19004.460214] ? show_regs.cold.9+0x22/0x2f [19004.464607] ? watchdog_timer_fn.cold.6+0x62/0x10a [19004.467264] ? __hrtimer_run_queues+0x1c2/0x4d0 [19004.469828] ? watchdog+0x40/0x40 [19004.473171] ? hrtimer_interrupt+0x128/0x2c0 [19004.476365] ? smp_apic_timer_interrupt+0xa6/0x270 [19004.481517] ? apic_timer_interrupt+0xf/0x20 [19004.486202] [19004.487203] ? native_safe_halt+0x2e/0x50 [19004.489649] kvm_wait+0xcd/0xe0 [19004.491328] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19004.494631] do_raw_spin_lock+0x177/0x190 [19004.496385] _raw_spin_lock+0x21/0x30 [19004.497781] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19004.500236] ? do_raw_spin_unlock+0x75/0x190 [19004.502885] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19004.507158] ? do_wait_intr+0xf0/0xf0 [19004.509440] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19004.514612] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19004.519373] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19004.521773] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19004.526781] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19004.531338] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19004.535340] mdc_intent_lock+0x353/0x7e0 [mdc] [19004.536974] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19004.540337] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19004.542924] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19004.547583] lmv_intent_open+0x338/0x1150 [lmv] [19004.549528] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19004.554374] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19004.558228] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19004.561750] ll_intent_lock+0xc5/0xaa0 [lustre] [19004.565175] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19004.567401] ll_lookup_it+0xe0e/0x2270 [lustre] [19004.569310] ? do_raw_spin_unlock+0x75/0x190 [19004.570479] ? _raw_spin_unlock+0x12/0x30 [19004.571471] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19004.572805] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19004.578663] ll_atomic_open+0x36a/0xd70 [lustre] [19004.584509] lookup_open+0x356/0xb40 [19004.587360] do_last+0x313/0xfc0 [19004.589559] ? nd_jump_root+0xe5/0x160 [19004.593310] ? path_init+0x437/0x520 [19004.594998] path_openat+0xf7/0x500 [19004.596002] ? do_raw_spin_unlock+0x75/0x190 [19004.599367] do_filp_open+0x99/0x140 [19004.600434] ? getname_flags+0x6e/0x330 [19004.601389] ? __check_object_size+0xff/0x256 [19004.602562] ? do_raw_spin_unlock+0x75/0x190 [19004.604577] ? _raw_spin_unlock+0x12/0x30 [19004.606204] do_sys_openat2+0x2b4/0x410 [19004.607811] do_sys_open+0x73/0xa0 [19004.609634] __x64_sys_openat+0x24/0x30 [19004.611386] do_syscall_64+0xc1/0x440 [19004.612613] entry_SYSCALL_64_after_hwframe+0x49/0xae [19004.614421] RIP: 0033:0x7f53425a5332 [19004.615835] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19004.622585] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19004.625257] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19004.627456] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19004.629756] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19004.632266] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19004.634386] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19020.052099] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [19020.055076] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19020.080162] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19020.084503] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19020.087651] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19020.090132] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19020.092358] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19020.102380] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19020.105574] RAX: 0000000000000d85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19020.108664] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19020.110628] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19020.113205] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19020.115612] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19020.117775] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19020.119867] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19020.121641] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19020.123799] Call Trace: [19020.124796] [19020.125581] ? show_regs.cold.9+0x22/0x2f [19020.127098] ? watchdog_timer_fn.cold.6+0x62/0x10a [19020.129032] ? __hrtimer_run_queues+0x1c2/0x4d0 [19020.130610] ? watchdog+0x40/0x40 [19020.131535] ? hrtimer_interrupt+0x128/0x2c0 [19020.133229] ? smp_apic_timer_interrupt+0xa6/0x270 [19020.134749] ? apic_timer_interrupt+0xf/0x20 [19020.136036] [19020.136858] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19020.138608] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19020.140556] ? newidle_balance.isra.93+0x207/0x580 [19020.142128] do_raw_spin_lock+0x177/0x190 [19020.143375] _raw_spin_lock+0x21/0x30 [19020.144633] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19020.147129] process_one_work+0x2c8/0x700 [19020.148451] worker_thread+0x296/0x6e0 [19020.149416] ? rescuer_thread+0x570/0x570 [19020.150629] kthread+0x1d1/0x200 [19020.151644] ? set_kthread_struct+0x70/0x70 [19020.152818] ret_from_fork+0x1f/0x30 [19023.842052] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16386s! [19023.844802] Showing busy workqueues and worker pools: [19023.848415] workqueue events: flags=0x0 [19023.851431] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19023.856343] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19023.866012] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19023.869951] pending: cpu_latency_work [ptlrpc] [19023.874682] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19023.885955] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19023.898280] workqueue events_long: flags=0x0 [19023.902016] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19023.906215] pending: lu_objects_flush [obdclass] [19023.914487] workqueue events_unbound: flags=0x2 [19023.916079] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19023.923273] pending: flush_memcg_stats_dwork, idle_cull_fn [19023.929572] workqueue events_power_efficient: flags=0x80 [19023.933120] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19023.935806] pending: check_lifetime, do_cache_clean [19023.942481] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19023.946052] pending: neigh_periodic_work, neigh_periodic_work [19023.952637] workqueue mm_percpu_wq: flags=0x8 [19023.957848] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19023.961972] pending: vmstat_update, lru_add_drain_per_cpu [19023.966629] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19023.972658] pending: vmstat_update, lru_add_drain_per_cpu [19023.974396] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19023.980222] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19023.985234] workqueue xprtiod: flags=0xa [19023.989099] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19023.992762] pending: xprt_autoclose, xs_error_handle [19023.998942] workqueue ipv6_addrconf: flags=0x40008 [19024.000642] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19024.004969] pending: addrconf_verify_work [19024.006883] workqueue knbd0-recv: flags=0x1a [19024.013812] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19024.019169] in-flight: 651:recv_work [19024.021185] workqueue ptlrpc_pinger: flags=0x2 [19024.023515] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19024.027245] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19024.032170] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16386s workers=4 idle: 15979 31824 8 [19024.035172] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19014s workers=2 idle: 654 [19032.055030] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [19032.057110] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19032.057902] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [19032.058020] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19032.058027] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19032.058099] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19032.058105] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19032.058107] RIP: 0010:native_safe_halt+0x2e/0x50 [19032.058120] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19032.058124] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19032.058129] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19032.058132] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19032.058134] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19032.058137] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19032.058139] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19032.058143] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19032.058146] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19032.058148] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19032.058157] Call Trace: [19032.058163] [19032.058168] ? show_regs.cold.9+0x22/0x2f [19032.058178] ? watchdog_timer_fn.cold.6+0x62/0x10a [19032.058185] ? __hrtimer_run_queues+0x1c2/0x4d0 [19032.058201] ? watchdog+0x40/0x40 [19032.058205] ? hrtimer_interrupt+0x128/0x2c0 [19032.058210] ? smp_apic_timer_interrupt+0xa6/0x270 [19032.058218] ? apic_timer_interrupt+0xf/0x20 [19032.058223] [19032.058226] ? native_safe_halt+0x2e/0x50 [19032.058231] kvm_wait+0xcd/0xe0 [19032.058239] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19032.058248] do_raw_spin_lock+0x177/0x190 [19032.058253] _raw_spin_lock+0x21/0x30 [19032.058258] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19032.058580] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19032.058957] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19032.059206] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19032.059503] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19032.059812] ? woken_wake_function+0x30/0x30 [19032.059821] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19032.060040] kthread+0x1d1/0x200 [19032.060046] ? set_kthread_struct+0x70/0x70 [19032.060050] ret_from_fork+0x1f/0x30 [19032.093377] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19032.095365] Modules linked in: [19032.097120] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19032.132552] lustre(O) [19032.139883] RIP: 0010:native_safe_halt+0x2e/0x50 [19032.140020] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19032.140025] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19032.140032] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19032.140034] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19032.140037] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19032.140039] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19032.140041] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19032.140045] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19032.140047] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19032.140050] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19032.140058] Call Trace: [19032.140063] [19032.140068] ? show_regs.cold.9+0x22/0x2f [19032.140077] ? watchdog_timer_fn.cold.6+0x62/0x10a [19032.140085] ? __hrtimer_run_queues+0x1c2/0x4d0 [19032.140089] ? watchdog+0x40/0x40 [19032.140093] ? hrtimer_interrupt+0x128/0x2c0 [19032.140097] ? smp_apic_timer_interrupt+0xa6/0x270 [19032.140104] ? apic_timer_interrupt+0xf/0x20 [19032.140108] [19032.140111] ? native_safe_halt+0x2e/0x50 [19032.140115] kvm_wait+0xcd/0xe0 [19032.140122] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19032.140132] do_raw_spin_lock+0x177/0x190 [19032.140136] _raw_spin_lock+0x21/0x30 [19032.140140] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19032.140411] ? do_raw_spin_unlock+0x75/0x190 [19032.140417] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19032.142036] ? do_wait_intr+0xf0/0xf0 [19032.142050] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19032.142278] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19032.143263] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19032.143493] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19032.149427] osp(O) [19032.153076] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19032.153129] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19032.153158] mdc_intent_lock+0x353/0x7e0 [mdc] [19032.153185] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19032.153308] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19032.153606] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19032.153634] lmv_intent_open+0x338/0x1150 [lmv] [19032.160154] ofd(O) [19032.163217] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19032.166384] lod(O) [19032.170829] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19032.170962] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19032.170995] ll_intent_lock+0xc5/0xaa0 [lustre] [19032.171075] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19032.171154] ll_lookup_it+0xe0e/0x2270 [lustre] [19032.171246] ? do_raw_spin_unlock+0x75/0x190 [19032.171253] ? _raw_spin_unlock+0x12/0x30 [19032.181352] mdt(O) [19032.183886] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19032.183993] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19032.184019] ll_atomic_open+0x36a/0xd70 [lustre] [19032.184145] lookup_open+0x356/0xb40 [19032.184156] do_last+0x313/0xfc0 [19032.184162] ? nd_jump_root+0xe5/0x160 [19032.184166] ? path_init+0x437/0x520 [19032.184171] path_openat+0xf7/0x500 [19032.184175] ? do_raw_spin_unlock+0x75/0x190 [19032.184182] do_filp_open+0x99/0x140 [19032.184199] ? getname_flags+0x6e/0x330 [19032.184203] ? __check_object_size+0xff/0x256 [19032.184210] ? do_raw_spin_unlock+0x75/0x190 [19032.184214] ? _raw_spin_unlock+0x12/0x30 [19032.184222] do_sys_openat2+0x2b4/0x410 [19032.184227] do_sys_open+0x73/0xa0 [19032.184232] __x64_sys_openat+0x24/0x30 [19032.184236] do_syscall_64+0xc1/0x440 [19032.184243] entry_SYSCALL_64_after_hwframe+0x49/0xae [19032.184249] RIP: 0033:0x7f53425a5332 [19032.184256] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19032.184260] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19032.184266] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19032.184268] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19032.184270] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19032.184272] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19032.184275] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19032.630716] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19032.657469] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19032.676720] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19032.681016] RIP: 0010:native_safe_halt+0x2e/0x50 [19032.683557] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19032.690905] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19032.695805] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19032.705430] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19032.712587] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19032.718117] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19032.728023] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19032.731002] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19032.735671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19032.745313] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19032.749685] Call Trace: [19032.750550] [19032.751526] ? show_regs.cold.9+0x22/0x2f [19032.752853] ? watchdog_timer_fn.cold.6+0x62/0x10a [19032.754280] ? __hrtimer_run_queues+0x1c2/0x4d0 [19032.760359] ? watchdog+0x40/0x40 [19032.762555] ? hrtimer_interrupt+0x128/0x2c0 [19032.767255] ? smp_apic_timer_interrupt+0xa6/0x270 [19032.772206] ? apic_timer_interrupt+0xf/0x20 [19032.773859] [19032.774577] ? native_safe_halt+0x2e/0x50 [19032.779059] kvm_wait+0xcd/0xe0 [19032.780447] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19032.787520] do_raw_spin_lock+0x177/0x190 [19032.789584] _raw_spin_lock+0x21/0x30 [19032.791804] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19032.795550] obd_mod_free+0x46/0x140 [ptlrpc] [19032.799131] mdc_commit_open+0x91/0xb0 [mdc] [19032.800610] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19032.806028] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19032.811139] after_reply+0xe2c/0x1780 [ptlrpc] [19032.818030] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19032.825546] ? __next_timer_interrupt+0x160/0x160 [19032.832171] ptlrpcd+0xa82/0xe40 [ptlrpc] [19032.833566] ? do_wait_intr+0xf0/0xf0 [19032.834457] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19032.841090] kthread+0x1d1/0x200 [19032.842752] ? set_kthread_struct+0x70/0x70 [19032.845061] ret_from_fork+0x1f/0x30 [19048.055920] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [19048.065029] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19048.096067] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19048.106523] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19048.111962] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19048.115865] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19048.118476] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19048.127291] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19048.131145] RAX: 0000000000007f85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19048.134036] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19048.137747] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19048.141667] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19048.144587] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19048.148183] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19048.154087] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19048.155995] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19048.158143] Call Trace: [19048.158630] [19048.159247] ? show_regs.cold.9+0x22/0x2f [19048.160252] ? watchdog_timer_fn.cold.6+0x62/0x10a [19048.162037] ? __hrtimer_run_queues+0x1c2/0x4d0 [19048.163396] ? watchdog+0x40/0x40 [19048.164176] ? hrtimer_interrupt+0x128/0x2c0 [19048.165859] ? smp_apic_timer_interrupt+0xa6/0x270 [19048.167255] ? apic_timer_interrupt+0xf/0x20 [19048.170098] [19048.170577] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19048.172202] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19048.173560] ? newidle_balance.isra.93+0x207/0x580 [19048.174844] do_raw_spin_lock+0x177/0x190 [19048.176129] _raw_spin_lock+0x21/0x30 [19048.177153] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19048.178667] process_one_work+0x2c8/0x700 [19048.179757] worker_thread+0x296/0x6e0 [19048.180739] ? rescuer_thread+0x570/0x570 [19048.181763] kthread+0x1d1/0x200 [19048.183305] ? set_kthread_struct+0x70/0x70 [19048.184399] ret_from_fork+0x1f/0x30 [19054.053520] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16416s! [19054.061983] Showing busy workqueues and worker pools: [19054.065251] workqueue events: flags=0x0 [19054.067611] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19054.072266] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19054.079458] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19054.083499] pending: cpu_latency_work [ptlrpc] [19054.087056] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19054.091145] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19054.096548] workqueue events_long: flags=0x0 [19054.099696] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19054.108445] pending: lu_objects_flush [obdclass] [19054.112061] workqueue events_unbound: flags=0x2 [19054.114614] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19054.117927] pending: flush_memcg_stats_dwork, idle_cull_fn [19054.121474] workqueue events_power_efficient: flags=0x80 [19054.124119] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19054.126892] pending: check_lifetime, do_cache_clean [19054.130452] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19054.134085] pending: neigh_periodic_work, neigh_periodic_work [19054.137179] workqueue mm_percpu_wq: flags=0x8 [19054.139601] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19054.144108] pending: vmstat_update, lru_add_drain_per_cpu [19054.146873] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19054.152970] pending: vmstat_update, lru_add_drain_per_cpu [19054.157145] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19054.161755] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19054.164660] workqueue xprtiod: flags=0xa [19054.166059] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19054.168425] pending: xprt_autoclose, xs_error_handle [19054.173406] workqueue ipv6_addrconf: flags=0x40008 [19054.175432] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19054.178095] pending: addrconf_verify_work [19054.180064] workqueue knbd0-recv: flags=0x1a [19054.181896] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19054.184786] in-flight: 651:recv_work [19054.188057] workqueue ptlrpc_pinger: flags=0x2 [19054.190668] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19054.193713] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19054.196130] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16417s workers=4 idle: 15979 31824 8 [19054.201036] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19044s workers=2 idle: 654 [19060.055016] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [19060.058018] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [19060.058560] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19060.058684] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19060.058689] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19060.058692] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19060.062116] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19060.062208] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19060.062216] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19060.062219] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19060.062235] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19060.062240] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19060.062244] RAX: 0000000000003606 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19060.062250] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19060.062252] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19060.062255] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19060.062258] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19060.062261] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19060.062264] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19060.062270] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19060.062279] Call Trace: [19060.062284] [19060.062289] ? show_regs.cold.9+0x22/0x2f [19060.062298] ? watchdog_timer_fn.cold.6+0x62/0x10a [19060.062304] ? __hrtimer_run_queues+0x1c2/0x4d0 [19060.062308] ? watchdog+0x40/0x40 [19060.062313] ? hrtimer_interrupt+0x128/0x2c0 [19060.062317] ? smp_apic_timer_interrupt+0xa6/0x270 [19060.062326] ? apic_timer_interrupt+0xf/0x20 [19060.062330] [19060.062333] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19060.062411] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19060.062417] do_raw_spin_lock+0x177/0x190 [19060.062422] _raw_spin_lock+0x21/0x30 [19060.062426] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19060.062956] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [19060.062962] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19060.063033] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19060.063037] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19060.063040] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19060.063047] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19060.063051] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19060.063055] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19060.063058] RDX: 00000000000018af RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19060.063060] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19060.063062] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19060.063064] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19060.063067] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19060.063070] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19060.063072] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19060.063080] Call Trace: [19060.063083] [19060.063085] ? show_regs.cold.9+0x22/0x2f [19060.063092] ? watchdog_timer_fn.cold.6+0x62/0x10a [19060.063097] ? __hrtimer_run_queues+0x1c2/0x4d0 [19060.063100] ? watchdog+0x40/0x40 [19060.063104] ? hrtimer_interrupt+0x128/0x2c0 [19060.063108] ? smp_apic_timer_interrupt+0xa6/0x270 [19060.063115] ? apic_timer_interrupt+0xf/0x20 [19060.063119] [19060.063123] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19060.063127] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19060.063132] do_raw_spin_lock+0x177/0x190 [19060.063136] _raw_spin_lock+0x21/0x30 [19060.063141] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19060.063268] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19060.063978] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19060.064180] obd_mod_free+0x46/0x140 [ptlrpc] [19060.065241] mdc_commit_open+0x91/0xb0 [mdc] [19060.064984] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19060.065311] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19060.066524] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19060.067206] ? woken_wake_function+0x30/0x30 [19060.067216] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19060.067473] kthread+0x1d1/0x200 [19060.067480] ? set_kthread_struct+0x70/0x70 [19060.067484] ret_from_fork+0x1f/0x30 [19060.088469] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19060.097829] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19060.114536] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19060.114550] RAX: 000000000000413a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19060.114553] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19060.114555] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19060.114558] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19060.114561] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19060.114564] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19060.114567] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19060.114569] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19060.114791] Call Trace: [19060.117317] after_reply+0xe2c/0x1780 [ptlrpc] [19060.117603] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19060.118113] ? __next_timer_interrupt+0x160/0x160 [19060.118124] ptlrpcd+0xa82/0xe40 [ptlrpc] [19060.118327] ? do_wait_intr+0xf0/0xf0 [19060.118338] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19060.118543] kthread+0x1d1/0x200 [19060.118550] ? set_kthread_struct+0x70/0x70 [19060.118554] ret_from_fork+0x1f/0x30 [19060.563524] [19060.564245] ? show_regs.cold.9+0x22/0x2f [19060.566794] ? watchdog_timer_fn.cold.6+0x62/0x10a [19060.569734] ? __hrtimer_run_queues+0x1c2/0x4d0 [19060.571937] ? watchdog+0x40/0x40 [19060.576215] ? hrtimer_interrupt+0x128/0x2c0 [19060.581094] ? smp_apic_timer_interrupt+0xa6/0x270 [19060.588031] ? apic_timer_interrupt+0xf/0x20 [19060.592934] [19060.595400] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19060.600110] do_raw_spin_lock+0x177/0x190 [19060.602125] _raw_spin_lock+0x21/0x30 [19060.604636] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19060.610216] ? do_raw_spin_unlock+0x75/0x190 [19060.613636] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19060.619030] ? do_wait_intr+0xf0/0xf0 [19060.620609] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19060.623258] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19060.625861] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19060.630317] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19060.635943] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19060.641107] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19060.643371] mdc_intent_lock+0x353/0x7e0 [mdc] [19060.648404] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19060.651584] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19060.658166] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19060.660745] lmv_intent_open+0x338/0x1150 [lmv] [19060.664537] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19060.667113] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19060.668898] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19060.671369] ll_intent_lock+0xc5/0xaa0 [lustre] [19060.673188] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19060.679316] ll_lookup_it+0xe0e/0x2270 [lustre] [19060.687143] ? do_raw_spin_unlock+0x75/0x190 [19060.690224] ? _raw_spin_unlock+0x12/0x30 [19060.697494] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19060.702554] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19060.705599] ll_atomic_open+0x36a/0xd70 [lustre] [19060.707551] lookup_open+0x356/0xb40 [19060.708860] do_last+0x313/0xfc0 [19060.710176] ? nd_jump_root+0xe5/0x160 [19060.711979] ? path_init+0x437/0x520 [19060.713564] path_openat+0xf7/0x500 [19060.714896] ? do_raw_spin_unlock+0x75/0x190 [19060.716098] do_filp_open+0x99/0x140 [19060.717598] ? getname_flags+0x6e/0x330 [19060.719135] ? __check_object_size+0xff/0x256 [19060.720663] ? do_raw_spin_unlock+0x75/0x190 [19060.722198] ? _raw_spin_unlock+0x12/0x30 [19060.723561] do_sys_openat2+0x2b4/0x410 [19060.724993] do_sys_open+0x73/0xa0 [19060.726187] __x64_sys_openat+0x24/0x30 [19060.727459] do_syscall_64+0xc1/0x440 [19060.728652] entry_SYSCALL_64_after_hwframe+0x49/0xae [19060.730217] RIP: 0033:0x7f53425a5332 [19060.731165] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19060.739059] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19060.746796] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19060.754086] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19060.759252] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19060.762844] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19060.766204] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19074.852125] rcu: INFO: rcu_sched self-detected stall on CPU [19074.854811] rcu: 2-....: (9556668 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [19074.864283] (t=16980426 jiffies g=274185 q=68230) [19074.865712] rcu: rcu_sched kthread starved for 16438792 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [19074.869072] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [19074.871277] rcu: RCU grace-period kthread stack dump: [19074.880287] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [19074.898024] Call Trace: [19074.899807] __schedule+0x351/0xcb0 [19074.902285] schedule+0xc0/0x180 [19074.903418] schedule_timeout+0xb4/0x190 [19074.904357] ? do_raw_spin_unlock+0x75/0x190 [19074.913111] ? __next_timer_interrupt+0x160/0x160 [19074.919225] ? prepare_to_swait_event+0xf3/0x140 [19074.923647] rcu_gp_kthread+0x8dd/0x15b0 [19074.929390] ? force_qs_rnp+0x2c0/0x2c0 [19074.937220] kthread+0x1d1/0x200 [19074.945598] ? set_kthread_struct+0x70/0x70 [19074.950384] ret_from_fork+0x1f/0x30 [19074.951427] rcu: Stack dump where RCU GP kthread last ran: [19074.953164] Sending NMI from CPU 2 to CPUs 0: [19074.958702] NMI backtrace for cpu 0 [19074.958709] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19074.958712] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19074.958713] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19074.958716] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19074.958718] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19074.958721] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 [19074.958723] RAX: 0000000000000604 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19074.958724] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19074.958726] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19074.958728] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19074.958729] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19074.958731] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19074.958732] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19074.958734] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19074.958735] Call Trace: [19074.958736] [19074.958737] ? show_regs.cold.9+0x22/0x2f [19074.958738] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [19074.958739] ? nmi_cpu_backtrace_handler+0x15/0x40 [19074.958741] ? nmi_handle+0xbd/0x210 [19074.958742] ? default_do_nmi+0x4b/0x1b0 [19074.958743] ? do_nmi+0x280/0x380 [19074.958744] ? end_repeat_nmi+0x16/0x1f [19074.958745] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19074.958747] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19074.958748] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19074.958749] [19074.958750] ? newidle_balance.isra.93+0x207/0x580 [19074.958751] do_raw_spin_lock+0x177/0x190 [19074.958753] _raw_spin_lock+0x21/0x30 [19074.958754] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19074.958755] process_one_work+0x2c8/0x700 [19074.958757] worker_thread+0x296/0x6e0 [19074.958758] ? rescuer_thread+0x570/0x570 [19074.958759] kthread+0x1d1/0x200 [19074.958760] ? set_kthread_struct+0x70/0x70 [19074.958761] ret_from_fork+0x1f/0x30 [19074.960742] Sending NMI from CPU 2 to CPUs 1: [19075.169497] NMI backtrace for cpu 1 [19075.169667] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19075.169671] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19075.169673] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19075.169675] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19075.169677] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [19075.169679] RAX: 0000000000000da9 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19075.169681] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19075.169682] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19075.169684] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19075.169685] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19075.169687] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19075.169688] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19075.169690] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19075.169691] Call Trace: [19075.169692] [19075.169693] ? show_regs.cold.9+0x22/0x2f [19075.169694] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [19075.169695] ? nmi_cpu_backtrace_handler+0x15/0x40 [19075.169697] ? nmi_handle+0xbd/0x210 [19075.169697] ? default_do_nmi+0x4b/0x1b0 [19075.169698] ? do_nmi+0x280/0x380 [19075.169699] ? end_repeat_nmi+0x16/0x1f [19075.169701] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19075.169702] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19075.169703] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19075.169704] [19075.169705] do_raw_spin_lock+0x177/0x190 [19075.169706] _raw_spin_lock+0x21/0x30 [19075.169707] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19075.169709] ? do_raw_spin_unlock+0x75/0x190 [19075.169710] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19075.169711] ? do_wait_intr+0xf0/0xf0 [19075.169712] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19075.169714] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19075.169715] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19075.169716] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19075.169717] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19075.169719] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19075.169720] mdc_intent_lock+0x353/0x7e0 [mdc] [19075.169722] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19075.169723] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19075.169724] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19075.169726] lmv_intent_open+0x338/0x1150 [lmv] [19075.169727] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19075.169729] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19075.169730] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19075.169731] ll_intent_lock+0xc5/0xaa0 [lustre] [19075.169732] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19075.169733] ll_lookup_it+0xe0e/0x2270 [lustre] [19075.169734] ? do_raw_spin_unlock+0x75/0x190 [19075.169736] ? _raw_spin_unlock+0x12/0x30 [19075.169736] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19075.169738] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19075.169739] ll_atomic_open+0x36a/0xd70 [lustre] [19075.169740] lookup_open+0x356/0xb40 [19075.169741] do_last+0x313/0xfc0 [19075.169742] ? nd_jump_root+0xe5/0x160 [19075.169743] ? path_init+0x437/0x520 [19075.169744] path_openat+0xf7/0x500 [19075.169745] ? do_raw_spin_unlock+0x75/0x190 [19075.169746] do_filp_open+0x99/0x140 [19075.169747] ? getname_flags+0x6e/0x330 [19075.169748] ? __check_object_size+0xff/0x256 [19075.169749] ? do_raw_spin_unlock+0x75/0x190 [19075.169750] ? _raw_spin_unlock+0x12/0x30 [19075.169751] do_sys_openat2+0x2b4/0x410 [19075.169752] do_sys_open+0x73/0xa0 [19075.169753] __x64_sys_openat+0x24/0x30 [19075.169754] do_syscall_64+0xc1/0x440 [19075.169756] entry_SYSCALL_64_after_hwframe+0x49/0xae [19075.169757] RIP: 0033:0x7f53425a5332 [19075.169759] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19075.169760] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19075.169763] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19075.169764] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19075.169765] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19075.169767] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19075.169768] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19075.174133] NMI backtrace for cpu 2 [19075.544398] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19075.562143] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19075.572341] Call Trace: [19075.576165] [19075.577838] ? dump_stack+0xbb/0x10e [19075.579378] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [19075.588331] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [19075.593060] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [19075.597852] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [19075.603265] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [19075.612065] ? rcu_dump_cpu_stacks+0x11b/0x171 [19075.616727] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [19075.621966] ? update_process_times+0x6d/0xc0 [19075.624106] ? tick_sched_handle.isra.18+0x27/0x90 [19075.626047] ? tick_sched_timer+0x53/0xb0 [19075.632359] ? __hrtimer_run_queues+0x1c2/0x4d0 [19075.635082] ? tick_sched_handle.isra.18+0x90/0x90 [19075.639287] ? hrtimer_interrupt+0x128/0x2c0 [19075.644063] ? smp_apic_timer_interrupt+0xa6/0x270 [19075.652314] ? apic_timer_interrupt+0xf/0x20 [19075.659299] [19075.661326] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19075.668018] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19075.671103] ? do_raw_spin_lock+0x177/0x190 [19075.673491] ? _raw_spin_lock+0x21/0x30 [19075.677818] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19075.682217] ? obd_mod_free+0x46/0x140 [ptlrpc] [19075.686975] ? mdc_commit_open+0x91/0xb0 [mdc] [19075.688996] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19075.697065] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19075.707208] ? after_reply+0xe2c/0x1780 [ptlrpc] [19075.711115] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19075.718880] ? __next_timer_interrupt+0x160/0x160 [19075.725495] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [19075.727803] ? do_wait_intr+0xf0/0xf0 [19075.730550] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19075.737852] ? kthread+0x1d1/0x200 [19075.740788] ? set_kthread_struct+0x70/0x70 [19075.744340] ? ret_from_fork+0x1f/0x30 [19084.255103] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16447s! [19084.267079] Showing busy workqueues and worker pools: [19084.277082] workqueue events: flags=0x0 [19084.278443] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19084.291813] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19084.304089] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19084.321741] pending: cpu_latency_work [ptlrpc] [19084.327774] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19084.335912] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19084.349040] workqueue events_long: flags=0x0 [19084.354106] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19084.376107] pending: lu_objects_flush [obdclass] [19084.398384] workqueue events_unbound: flags=0x2 [19084.399906] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19084.408567] pending: flush_memcg_stats_dwork, idle_cull_fn [19084.423987] workqueue events_power_efficient: flags=0x80 [19084.431073] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19084.437696] pending: check_lifetime, do_cache_clean [19084.441788] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19084.447099] pending: neigh_periodic_work, neigh_periodic_work [19084.452245] workqueue mm_percpu_wq: flags=0x8 [19084.460500] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19084.476025] pending: vmstat_update, lru_add_drain_per_cpu [19084.487674] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19084.497638] pending: vmstat_update, lru_add_drain_per_cpu [19084.504150] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19084.506418] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19084.517860] workqueue xprtiod: flags=0xa [19084.523082] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19084.533555] pending: xprt_autoclose, xs_error_handle [19084.539413] workqueue ipv6_addrconf: flags=0x40008 [19084.541027] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19084.553986] pending: addrconf_verify_work [19084.562201] workqueue knbd0-recv: flags=0x1a [19084.563656] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19084.576517] in-flight: 651:recv_work [19084.578947] workqueue ptlrpc_pinger: flags=0x2 [19084.582925] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19084.593880] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19084.601302] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16447s workers=4 idle: 15979 31824 8 [19084.609102] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19075s workers=2 idle: 654 [19088.061994] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [19088.066110] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19088.110742] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19088.119415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19088.125023] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19088.127711] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19088.137579] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19088.142126] RAX: 0000000000006c82 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19088.144082] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19088.150506] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19088.154379] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19088.159852] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19088.165559] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19088.173041] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19088.180014] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19088.183480] Call Trace: [19088.190483] [19088.191926] ? show_regs.cold.9+0x22/0x2f [19088.193502] ? watchdog_timer_fn.cold.6+0x62/0x10a [19088.195802] ? __hrtimer_run_queues+0x1c2/0x4d0 [19088.197709] ? watchdog+0x40/0x40 [19088.200582] ? hrtimer_interrupt+0x128/0x2c0 [19088.202496] ? smp_apic_timer_interrupt+0xa6/0x270 [19088.205066] ? apic_timer_interrupt+0xf/0x20 [19088.206786] [19088.215496] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19088.227184] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19088.231652] do_raw_spin_lock+0x177/0x190 [19088.233941] _raw_spin_lock+0x21/0x30 [19088.241017] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19088.251171] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19088.252643] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19088.261590] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19088.267487] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19088.277594] ? woken_wake_function+0x30/0x30 [19088.279507] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19088.285747] kthread+0x1d1/0x200 [19088.295015] ? set_kthread_struct+0x70/0x70 [19088.297961] ret_from_fork+0x1f/0x30 [19100.055120] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [19100.057082] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [19100.059112] Modules linked in: [19100.062136] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19100.062211] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19100.062218] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19100.062221] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19100.062234] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19100.062238] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19100.062243] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19100.062245] RDX: 0000000000006a25 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19100.062248] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19100.062250] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19100.062253] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19100.062257] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19100.062260] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19100.062263] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19100.062271] Call Trace: [19100.062277] [19100.062283] ? show_regs.cold.9+0x22/0x2f [19100.062294] ? watchdog_timer_fn.cold.6+0x62/0x10a [19100.062301] ? __hrtimer_run_queues+0x1c2/0x4d0 [19100.062305] ? watchdog+0x40/0x40 [19100.062310] ? hrtimer_interrupt+0x128/0x2c0 [19100.062313] ? smp_apic_timer_interrupt+0xa6/0x270 [19100.062324] ? apic_timer_interrupt+0xf/0x20 [19100.062328] [19100.062331] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19100.062335] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19100.062340] do_raw_spin_lock+0x177/0x190 [19100.062345] _raw_spin_lock+0x21/0x30 [19100.062351] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19100.062780] obd_mod_free+0x46/0x140 [ptlrpc] [19100.063311] mdc_commit_open+0x91/0xb0 [mdc] [19100.063352] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19100.063556] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19100.064331] after_reply+0xe2c/0x1780 [ptlrpc] [19100.064538] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19100.064755] ? __next_timer_interrupt+0x160/0x160 [19100.064766] ptlrpcd+0xa82/0xe40 [ptlrpc] [19100.065187] ? do_wait_intr+0xf0/0xf0 [19100.065197] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19100.065407] kthread+0x1d1/0x200 [19100.065413] ? set_kthread_struct+0x70/0x70 [19100.065417] ret_from_fork+0x1f/0x30 [19100.376745] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19100.439298] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19100.457837] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19100.467197] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19100.476982] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19100.494234] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19100.497116] RAX: 000000000000508b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19100.502449] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19100.517309] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19100.519588] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19100.534494] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19100.543244] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19100.553866] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19100.557046] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19100.560376] Call Trace: [19100.561460] [19100.563371] ? show_regs.cold.9+0x22/0x2f [19100.566929] ? watchdog_timer_fn.cold.6+0x62/0x10a [19100.568659] ? __hrtimer_run_queues+0x1c2/0x4d0 [19100.571323] ? watchdog+0x40/0x40 [19100.572875] ? hrtimer_interrupt+0x128/0x2c0 [19100.574930] ? smp_apic_timer_interrupt+0xa6/0x270 [19100.580299] ? apic_timer_interrupt+0xf/0x20 [19100.582607] [19100.586330] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19100.595035] do_raw_spin_lock+0x177/0x190 [19100.604562] _raw_spin_lock+0x21/0x30 [19100.605771] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19100.611488] ? do_raw_spin_unlock+0x75/0x190 [19100.612969] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19100.627366] ? do_wait_intr+0xf0/0xf0 [19100.634410] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19100.641601] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19100.651026] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19100.657947] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19100.661339] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19100.664872] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19100.667195] mdc_intent_lock+0x353/0x7e0 [mdc] [19100.669490] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19100.672267] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19100.674916] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19100.677422] lmv_intent_open+0x338/0x1150 [lmv] [19100.679371] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19100.683870] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19100.687579] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19100.699968] ll_intent_lock+0xc5/0xaa0 [lustre] [19100.702635] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19100.706319] ll_lookup_it+0xe0e/0x2270 [lustre] [19100.707538] ? do_raw_spin_unlock+0x75/0x190 [19100.712245] ? _raw_spin_unlock+0x12/0x30 [19100.714181] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19100.716975] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19100.722370] ll_atomic_open+0x36a/0xd70 [lustre] [19100.724879] lookup_open+0x356/0xb40 [19100.729072] do_last+0x313/0xfc0 [19100.732905] ? nd_jump_root+0xe5/0x160 [19100.736544] ? path_init+0x437/0x520 [19100.738393] path_openat+0xf7/0x500 [19100.740478] ? do_raw_spin_unlock+0x75/0x190 [19100.745746] do_filp_open+0x99/0x140 [19100.749414] ? getname_flags+0x6e/0x330 [19100.753153] ? __check_object_size+0xff/0x256 [19100.759901] ? do_raw_spin_unlock+0x75/0x190 [19100.761418] ? _raw_spin_unlock+0x12/0x30 [19100.769411] do_sys_openat2+0x2b4/0x410 [19100.770630] do_sys_open+0x73/0xa0 [19100.771807] __x64_sys_openat+0x24/0x30 [19100.782257] do_syscall_64+0xc1/0x440 [19100.784138] entry_SYSCALL_64_after_hwframe+0x49/0xae [19100.786748] RIP: 0033:0x7f53425a5332 [19100.791779] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19100.809228] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19100.818845] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19100.825328] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19100.835337] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19100.844233] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19100.847826] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19112.052025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [19112.061306] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19112.097306] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19112.107265] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19112.114588] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19112.118651] RIP: 0010:native_safe_halt+0x2e/0x50 [19112.123305] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19112.140106] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19112.145187] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19112.148898] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [19112.155275] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19112.157105] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19112.172618] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19112.178654] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19112.185539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19112.189510] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19112.195302] Call Trace: [19112.196547] [19112.197271] ? show_regs.cold.9+0x22/0x2f [19112.199244] ? watchdog_timer_fn.cold.6+0x62/0x10a [19112.202117] ? __hrtimer_run_queues+0x1c2/0x4d0 [19112.204450] ? watchdog+0x40/0x40 [19112.209264] ? hrtimer_interrupt+0x128/0x2c0 [19112.210727] ? smp_apic_timer_interrupt+0xa6/0x270 [19112.215524] ? apic_timer_interrupt+0xf/0x20 [19112.217544] [19112.218420] ? native_safe_halt+0x2e/0x50 [19112.219684] kvm_wait+0xcd/0xe0 [19112.221212] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19112.222761] ? newidle_balance.isra.93+0x207/0x580 [19112.228595] do_raw_spin_lock+0x177/0x190 [19112.230278] _raw_spin_lock+0x21/0x30 [19112.231621] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19112.234462] process_one_work+0x2c8/0x700 [19112.236119] worker_thread+0x296/0x6e0 [19112.237455] ? rescuer_thread+0x570/0x570 [19112.239531] kthread+0x1d1/0x200 [19112.241634] ? set_kthread_struct+0x70/0x70 [19112.251268] ret_from_fork+0x1f/0x30 [19114.976040] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16477s! [19114.978960] Showing busy workqueues and worker pools: [19114.980308] workqueue events: flags=0x0 [19114.982396] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19114.984961] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19114.989070] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19114.991660] pending: cpu_latency_work [ptlrpc] [19114.994649] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19114.997666] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19115.001023] workqueue events_long: flags=0x0 [19115.002663] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19115.005018] pending: lu_objects_flush [obdclass] [19115.007018] workqueue events_unbound: flags=0x2 [19115.008426] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19115.010678] pending: flush_memcg_stats_dwork, idle_cull_fn [19115.012686] workqueue events_power_efficient: flags=0x80 [19115.015196] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19115.023248] pending: check_lifetime, do_cache_clean [19115.026421] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19115.033801] pending: neigh_periodic_work, neigh_periodic_work [19115.038282] workqueue mm_percpu_wq: flags=0x8 [19115.042094] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19115.048046] pending: vmstat_update, lru_add_drain_per_cpu [19115.050584] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19115.053443] pending: vmstat_update, lru_add_drain_per_cpu [19115.056054] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19115.058082] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19115.061117] workqueue xprtiod: flags=0xa [19115.063201] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19115.067957] pending: xprt_autoclose, xs_error_handle [19115.070290] workqueue ipv6_addrconf: flags=0x40008 [19115.072638] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19115.079117] pending: addrconf_verify_work [19115.082249] workqueue knbd0-recv: flags=0x1a [19115.083878] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19115.086278] in-flight: 651:recv_work [19115.089664] workqueue ptlrpc_pinger: flags=0x2 [19115.093042] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19115.101909] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19115.108542] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16478s workers=4 idle: 15979 31824 8 [19115.113259] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19105s workers=2 idle: 654 [19116.058244] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [19116.065692] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19116.102523] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19116.112210] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19116.120571] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19116.124050] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19116.139589] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19116.151812] RAX: 0000000000005843 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19116.163713] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19116.178351] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19116.180612] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19116.189618] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19116.194516] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19116.203027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19116.210068] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19116.223493] Call Trace: [19116.229784] [19116.231559] ? show_regs.cold.9+0x22/0x2f [19116.233601] ? watchdog_timer_fn.cold.6+0x62/0x10a [19116.238959] ? __hrtimer_run_queues+0x1c2/0x4d0 [19116.240459] ? watchdog+0x40/0x40 [19116.247344] ? hrtimer_interrupt+0x128/0x2c0 [19116.254525] ? smp_apic_timer_interrupt+0xa6/0x270 [19116.267864] ? apic_timer_interrupt+0xf/0x20 [19116.269136] [19116.269610] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19116.273921] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19116.276947] do_raw_spin_lock+0x177/0x190 [19116.279061] _raw_spin_lock+0x21/0x30 [19116.289455] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19116.295270] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19116.300405] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19116.309323] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19116.318452] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19116.325492] ? woken_wake_function+0x30/0x30 [19116.328973] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19116.344328] kthread+0x1d1/0x200 [19116.351904] ? set_kthread_struct+0x70/0x70 [19116.354925] ret_from_fork+0x1f/0x30 [19128.058411] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [19128.059086] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [19128.065419] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19128.065499] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19128.066957] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19128.066962] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19128.066976] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19128.066980] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19128.066984] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19128.066986] RDX: 0000000000002175 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19128.066988] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19128.066990] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19128.066992] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19128.066995] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19128.066998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19128.067000] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19128.067008] Call Trace: [19128.067013] [19128.067019] ? show_regs.cold.9+0x22/0x2f [19128.067027] ? watchdog_timer_fn.cold.6+0x62/0x10a [19128.067033] ? __hrtimer_run_queues+0x1c2/0x4d0 [19128.067038] ? watchdog+0x40/0x40 [19128.067042] ? hrtimer_interrupt+0x128/0x2c0 [19128.067045] ? smp_apic_timer_interrupt+0xa6/0x270 [19128.067053] ? apic_timer_interrupt+0xf/0x20 [19128.067058] [19128.067061] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19128.067066] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19128.067071] do_raw_spin_lock+0x177/0x190 [19128.067076] _raw_spin_lock+0x21/0x30 [19128.067083] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19128.072144] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19128.072218] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19128.072225] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19128.072228] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19128.072243] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19128.072247] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19128.072252] RAX: 0000000000007c0a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19128.072254] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19128.072257] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19128.072259] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19128.072261] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19128.072265] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19128.072268] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19128.072271] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19128.072281] Call Trace: [19128.072286] [19128.072292] ? show_regs.cold.9+0x22/0x2f [19128.072301] ? watchdog_timer_fn.cold.6+0x62/0x10a [19128.072306] ? __hrtimer_run_queues+0x1c2/0x4d0 [19128.072311] ? watchdog+0x40/0x40 [19128.072316] ? hrtimer_interrupt+0x128/0x2c0 [19128.072320] ? smp_apic_timer_interrupt+0xa6/0x270 [19128.072329] ? apic_timer_interrupt+0xf/0x20 [19128.072334] [19128.072337] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19128.072343] do_raw_spin_lock+0x177/0x190 [19128.072348] _raw_spin_lock+0x21/0x30 [19128.072353] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19128.072645] ? do_raw_spin_unlock+0x75/0x190 [19128.072652] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19128.076408] ? do_wait_intr+0xf0/0xf0 [19128.076422] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19128.076700] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19128.076921] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19128.110275] obd_mod_free+0x46/0x140 [ptlrpc] [19128.115456] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19128.115593] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19128.115632] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19128.115661] mdc_intent_lock+0x353/0x7e0 [mdc] [19128.115688] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19128.115773] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19128.122661] mdc_commit_open+0x91/0xb0 [mdc] [19128.125449] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19128.125503] lmv_intent_open+0x338/0x1150 [lmv] [19128.125605] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19128.136198] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19128.142928] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19128.143052] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19128.143084] ll_intent_lock+0xc5/0xaa0 [lustre] [19128.143168] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19128.143250] ll_lookup_it+0xe0e/0x2270 [lustre] [19128.143334] ? do_raw_spin_unlock+0x75/0x190 [19128.143342] ? _raw_spin_unlock+0x12/0x30 [19128.143350] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19128.143378] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19128.143399] ll_atomic_open+0x36a/0xd70 [lustre] [19128.143483] lookup_open+0x356/0xb40 [19128.143492] do_last+0x313/0xfc0 [19128.143496] ? nd_jump_root+0xe5/0x160 [19128.143502] ? path_init+0x437/0x520 [19128.143516] path_openat+0xf7/0x500 [19128.143522] ? do_raw_spin_unlock+0x75/0x190 [19128.143527] do_filp_open+0x99/0x140 [19128.143532] ? getname_flags+0x6e/0x330 [19128.143537] ? __check_object_size+0xff/0x256 [19128.143544] ? do_raw_spin_unlock+0x75/0x190 [19128.143547] ? _raw_spin_unlock+0x12/0x30 [19128.143552] do_sys_openat2+0x2b4/0x410 [19128.143557] do_sys_open+0x73/0xa0 [19128.143561] __x64_sys_openat+0x24/0x30 [19128.143565] do_syscall_64+0xc1/0x440 [19128.143571] entry_SYSCALL_64_after_hwframe+0x49/0xae [19128.143577] RIP: 0033:0x7f53425a5332 [19128.143585] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19128.143589] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19128.143595] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19128.143598] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19128.143600] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19128.143602] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19128.143604] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19128.678730] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19128.684741] after_reply+0xe2c/0x1780 [ptlrpc] [19128.689068] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19128.693947] ? __next_timer_interrupt+0x160/0x160 [19128.695738] ptlrpcd+0xa82/0xe40 [ptlrpc] [19128.699760] ? do_wait_intr+0xf0/0xf0 [19128.702196] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19128.707832] kthread+0x1d1/0x200 [19128.710153] ? set_kthread_struct+0x70/0x70 [19128.711333] ret_from_fork+0x1f/0x30 [19140.054117] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [19140.067618] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19140.098267] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19140.103256] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19140.107597] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19140.119900] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19140.125026] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19140.142927] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19140.146828] RAX: 0000000000004388 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19140.155614] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19140.165183] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19140.171888] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19140.175968] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19140.185639] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19140.193651] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19140.199131] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19140.205075] Call Trace: [19140.210262] [19140.214969] ? show_regs.cold.9+0x22/0x2f [19140.218640] ? watchdog_timer_fn.cold.6+0x62/0x10a [19140.227903] ? __hrtimer_run_queues+0x1c2/0x4d0 [19140.229394] ? watchdog+0x40/0x40 [19140.234712] ? hrtimer_interrupt+0x128/0x2c0 [19140.236225] ? smp_apic_timer_interrupt+0xa6/0x270 [19140.241230] ? apic_timer_interrupt+0xf/0x20 [19140.247820] [19140.248546] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19140.254279] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19140.258969] ? newidle_balance.isra.93+0x207/0x580 [19140.263330] do_raw_spin_lock+0x177/0x190 [19140.264741] _raw_spin_lock+0x21/0x30 [19140.274097] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19140.279675] process_one_work+0x2c8/0x700 [19140.281604] worker_thread+0x296/0x6e0 [19140.285877] ? rescuer_thread+0x570/0x570 [19140.288810] kthread+0x1d1/0x200 [19140.294692] ? set_kthread_struct+0x70/0x70 [19140.300464] ret_from_fork+0x1f/0x30 [19144.060082] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [19144.066202] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19144.108815] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19144.130632] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19144.139051] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19144.143506] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19144.162375] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19144.164918] RAX: 0000000000007f8b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19144.171536] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19144.175152] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19144.185363] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19144.188220] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19144.191336] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19144.194902] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19144.197286] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19144.200449] Call Trace: [19144.202024] [19144.203077] ? show_regs.cold.9+0x22/0x2f [19144.204725] ? watchdog_timer_fn.cold.6+0x62/0x10a [19144.213423] ? __hrtimer_run_queues+0x1c2/0x4d0 [19144.216058] ? watchdog+0x40/0x40 [19144.217553] ? hrtimer_interrupt+0x128/0x2c0 [19144.219638] ? smp_apic_timer_interrupt+0xa6/0x270 [19144.221607] ? apic_timer_interrupt+0xf/0x20 [19144.223588] [19144.224400] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19144.226816] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19144.229086] do_raw_spin_lock+0x177/0x190 [19144.231089] _raw_spin_lock+0x21/0x30 [19144.232646] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19144.235287] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19144.240338] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19144.246525] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19144.249753] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19144.258127] ? woken_wake_function+0x30/0x30 [19144.262395] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19144.265191] kthread+0x1d1/0x200 [19144.266372] ? set_kthread_struct+0x70/0x70 [19144.267867] ret_from_fork+0x1f/0x30 [19145.183048] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16508s! [19145.187310] Showing busy workqueues and worker pools: [19145.191358] workqueue events: flags=0x0 [19145.193161] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19145.196977] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19145.205098] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19145.209080] pending: cpu_latency_work [ptlrpc] [19145.213703] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19145.217653] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19145.221238] workqueue events_long: flags=0x0 [19145.222902] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19145.225844] pending: lu_objects_flush [obdclass] [19145.228378] workqueue events_unbound: flags=0x2 [19145.230683] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19145.233438] pending: flush_memcg_stats_dwork, idle_cull_fn [19145.236865] workqueue events_power_efficient: flags=0x80 [19145.238826] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19145.242423] pending: check_lifetime, do_cache_clean [19145.244849] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19145.248022] pending: neigh_periodic_work, neigh_periodic_work [19145.251267] workqueue mm_percpu_wq: flags=0x8 [19145.253088] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19145.256274] pending: vmstat_update, lru_add_drain_per_cpu [19145.259013] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19145.262230] pending: vmstat_update, lru_add_drain_per_cpu [19145.264866] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19145.268314] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19145.271095] workqueue xprtiod: flags=0xa [19145.272995] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19145.275740] pending: xprt_autoclose, xs_error_handle [19145.278385] workqueue ipv6_addrconf: flags=0x40008 [19145.281362] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19145.284054] pending: addrconf_verify_work [19145.286080] workqueue knbd0-recv: flags=0x1a [19145.287764] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19145.290104] in-flight: 651:recv_work [19145.291454] workqueue ptlrpc_pinger: flags=0x2 [19145.293251] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19145.295731] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19145.298582] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16508s workers=4 idle: 15979 31824 8 [19145.302275] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19135s workers=2 idle: 654 [19156.055244] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [19156.058243] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [19156.058282] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19156.058367] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19156.058376] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19156.058379] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19156.058421] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19156.058426] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19156.058431] RAX: 0000000000005630 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19156.058434] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19156.058436] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19156.058441] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19156.058444] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19156.058447] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19156.058450] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19156.058453] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19156.058460] Call Trace: [19156.058465] [19156.058471] ? show_regs.cold.9+0x22/0x2f [19156.058484] ? watchdog_timer_fn.cold.6+0x62/0x10a [19156.058494] ? __hrtimer_run_queues+0x1c2/0x4d0 [19156.058502] ? watchdog+0x40/0x40 [19156.058506] ? hrtimer_interrupt+0x128/0x2c0 [19156.058514] ? smp_apic_timer_interrupt+0xa6/0x270 [19156.058523] ? apic_timer_interrupt+0xf/0x20 [19156.058527] [19156.058530] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19156.058539] do_raw_spin_lock+0x177/0x190 [19156.058548] _raw_spin_lock+0x21/0x30 [19156.058554] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19156.058898] ? do_raw_spin_unlock+0x75/0x190 [19156.058910] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19156.059197] ? do_wait_intr+0xf0/0xf0 [19156.059212] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19156.059454] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19156.059686] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19156.059917] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19156.060077] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19156.060510] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19156.060557] mdc_intent_lock+0x353/0x7e0 [mdc] [19156.060597] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19156.060714] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19156.061002] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19156.061343] lmv_intent_open+0x338/0x1150 [lmv] [19156.061450] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19156.061610] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19156.061692] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19156.061714] ll_intent_lock+0xc5/0xaa0 [lustre] [19156.061799] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19156.061889] ll_lookup_it+0xe0e/0x2270 [lustre] [19156.061990] ? do_raw_spin_unlock+0x75/0x190 [19156.061999] ? _raw_spin_unlock+0x12/0x30 [19156.062005] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19156.062289] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19156.062329] ll_atomic_open+0x36a/0xd70 [lustre] [19156.062522] lookup_open+0x356/0xb40 [19156.062531] do_last+0x313/0xfc0 [19156.062537] ? nd_jump_root+0xe5/0x160 [19156.062545] ? path_init+0x437/0x520 [19156.062553] path_openat+0xf7/0x500 [19156.062558] ? do_raw_spin_unlock+0x75/0x190 [19156.062566] do_filp_open+0x99/0x140 [19156.072215] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19156.072293] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19156.072300] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19156.072303] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19156.072317] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19156.072321] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19156.072325] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19156.072327] RDX: 00000000000063f3 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19156.072330] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19156.072332] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19156.072334] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19156.072337] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19156.072340] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19156.072343] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19156.072352] Call Trace: [19156.072358] [19156.072364] ? show_regs.cold.9+0x22/0x2f [19156.072375] ? watchdog_timer_fn.cold.6+0x62/0x10a [19156.072381] ? __hrtimer_run_queues+0x1c2/0x4d0 [19156.072385] ? watchdog+0x40/0x40 [19156.072389] ? hrtimer_interrupt+0x128/0x2c0 [19156.072393] ? smp_apic_timer_interrupt+0xa6/0x270 [19156.072532] ? apic_timer_interrupt+0xf/0x20 [19156.072540] [19156.072543] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19156.072549] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19156.072554] do_raw_spin_lock+0x177/0x190 [19156.072559] _raw_spin_lock+0x21/0x30 [19156.072567] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19156.135244] ? getname_flags+0x6e/0x330 [19156.135264] ? __check_object_size+0xff/0x256 [19156.135272] ? do_raw_spin_unlock+0x75/0x190 [19156.135278] ? _raw_spin_unlock+0x12/0x30 [19156.135285] do_sys_openat2+0x2b4/0x410 [19156.135290] do_sys_open+0x73/0xa0 [19156.135293] __x64_sys_openat+0x24/0x30 [19156.135297] do_syscall_64+0xc1/0x440 [19156.135304] entry_SYSCALL_64_after_hwframe+0x49/0xae [19156.135311] RIP: 0033:0x7f53425a5332 [19156.135319] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19156.135323] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19156.135328] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19156.135330] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19156.135333] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19156.135335] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19156.135336] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19156.713583] obd_mod_free+0x46/0x140 [ptlrpc] [19156.722821] mdc_commit_open+0x91/0xb0 [mdc] [19156.732862] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19156.738954] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19156.746298] after_reply+0xe2c/0x1780 [ptlrpc] [19156.749616] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19156.753437] ? __next_timer_interrupt+0x160/0x160 [19156.755506] ptlrpcd+0xa82/0xe40 [ptlrpc] [19156.757745] ? do_wait_intr+0xf0/0xf0 [19156.759328] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19156.761695] kthread+0x1d1/0x200 [19156.763650] ? set_kthread_struct+0x70/0x70 [19156.765715] ret_from_fork+0x1f/0x30 [19172.052306] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [19172.057894] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr [19172.058111] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19172.058120] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19172.058192] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19172.058199] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19172.058202] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19172.058217] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19172.058222] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19172.058226] RAX: 0000000000004e8a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19172.058228] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19172.058231] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19172.058233] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19172.058236] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19172.058239] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19172.058453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19172.058456] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19172.058465] Call Trace: [19172.058471] [19172.058476] ? show_regs.cold.9+0x22/0x2f [19172.058486] ? watchdog_timer_fn.cold.6+0x62/0x10a [19172.058494] ? __hrtimer_run_queues+0x1c2/0x4d0 [19172.058499] ? watchdog+0x40/0x40 [19172.058504] ? hrtimer_interrupt+0x128/0x2c0 [19172.058509] ? smp_apic_timer_interrupt+0xa6/0x270 [19172.058520] ? apic_timer_interrupt+0xf/0x20 [19172.077319] squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19172.077350] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19172.077357] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19172.077364] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19172.082312] [19172.082325] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19172.082339] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19172.082347] do_raw_spin_lock+0x177/0x190 [19172.082355] _raw_spin_lock+0x21/0x30 [19172.082362] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19172.082936] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19172.083341] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19172.083759] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19172.084196] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19172.084428] ? woken_wake_function+0x30/0x30 [19172.084436] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19172.084767] kthread+0x1d1/0x200 [19172.084773] ? set_kthread_struct+0x70/0x70 [19172.084776] ret_from_fork+0x1f/0x30 [19172.250291] [19172.251259] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19172.254368] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19172.264795] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19172.269254] RAX: 0000000000001d15 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19172.272423] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19172.275342] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19172.277799] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19172.281646] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19172.286305] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19172.290720] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19172.293676] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19172.297600] Call Trace: [19172.299042] [19172.300400] ? show_regs.cold.9+0x22/0x2f [19172.302589] ? watchdog_timer_fn.cold.6+0x62/0x10a [19172.305365] ? __hrtimer_run_queues+0x1c2/0x4d0 [19172.307694] ? watchdog+0x40/0x40 [19172.309570] ? hrtimer_interrupt+0x128/0x2c0 [19172.311940] ? smp_apic_timer_interrupt+0xa6/0x270 [19172.314631] ? apic_timer_interrupt+0xf/0x20 [19172.317300] [19172.318405] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19172.321402] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19172.324656] ? newidle_balance.isra.93+0x207/0x580 [19172.327230] do_raw_spin_lock+0x177/0x190 [19172.328928] _raw_spin_lock+0x21/0x30 [19172.330237] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19172.333835] process_one_work+0x2c8/0x700 [19172.335671] worker_thread+0x296/0x6e0 [19172.338307] ? rescuer_thread+0x570/0x570 [19172.340164] kthread+0x1d1/0x200 [19172.342279] ? set_kthread_struct+0x70/0x70 [19172.344923] ret_from_fork+0x1f/0x30 [19175.392385] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16538s! [19175.403216] Showing busy workqueues and worker pools: [19175.407671] workqueue events: flags=0x0 [19175.411263] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19175.421201] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19175.439940] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19175.446231] pending: cpu_latency_work [ptlrpc] [19175.449458] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19175.453582] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19175.464128] workqueue events_long: flags=0x0 [19175.469654] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19175.475909] pending: lu_objects_flush [obdclass] [19175.478758] workqueue events_unbound: flags=0x2 [19175.488590] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19175.492584] pending: flush_memcg_stats_dwork, idle_cull_fn [19175.501505] workqueue events_power_efficient: flags=0x80 [19175.508611] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19175.513660] pending: check_lifetime, do_cache_clean [19175.515960] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19175.523182] pending: neigh_periodic_work, neigh_periodic_work [19175.525855] workqueue mm_percpu_wq: flags=0x8 [19175.528151] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19175.531784] pending: vmstat_update, lru_add_drain_per_cpu [19175.534011] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19175.538632] pending: vmstat_update, lru_add_drain_per_cpu [19175.541182] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19175.548153] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19175.553584] workqueue xprtiod: flags=0xa [19175.555345] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19175.557540] pending: xprt_autoclose, xs_error_handle [19175.559647] workqueue ipv6_addrconf: flags=0x40008 [19175.563576] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19175.566548] pending: addrconf_verify_work [19175.569110] workqueue knbd0-recv: flags=0x1a [19175.575573] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19175.577813] in-flight: 651:recv_work [19175.579806] workqueue ptlrpc_pinger: flags=0x2 [19175.585994] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19175.590016] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19175.594830] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16538s workers=4 idle: 15979 31824 8 [19175.601219] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19166s workers=2 idle: 654 [19184.055605] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [19184.058981] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [19184.066262] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19184.066431] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19184.066439] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19184.066442] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19184.066456] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19184.066460] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19184.066465] RAX: 0000000000007905 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19184.066467] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19184.066469] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19184.066471] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19184.066473] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19184.066477] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19184.066481] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19184.066483] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19184.066492] Call Trace: [19184.066498] [19184.066504] ? show_regs.cold.9+0x22/0x2f [19184.066512] ? watchdog_timer_fn.cold.6+0x62/0x10a [19184.066518] ? __hrtimer_run_queues+0x1c2/0x4d0 [19184.066523] ? watchdog+0x40/0x40 [19184.066627] ? hrtimer_interrupt+0x128/0x2c0 [19184.066631] ? smp_apic_timer_interrupt+0xa6/0x270 [19184.066639] ? apic_timer_interrupt+0xf/0x20 [19184.066643] [19184.066646] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19184.066653] do_raw_spin_lock+0x177/0x190 [19184.066658] _raw_spin_lock+0x21/0x30 [19184.066664] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19184.066926] ? do_raw_spin_unlock+0x75/0x190 [19184.066931] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19184.067267] ? do_wait_intr+0xf0/0xf0 [19184.067279] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19184.073964] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [19184.116050] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19184.123868] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19184.123918] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19184.123925] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19184.123928] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19184.123941] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19184.123945] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19184.123950] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19184.123953] RDX: 000000000000081d RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19184.123955] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19184.123958] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19184.123960] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19184.123963] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19184.123967] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19184.123969] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19184.123979] Call Trace: [19184.123986] [19184.123991] ? show_regs.cold.9+0x22/0x2f [19184.124001] ? watchdog_timer_fn.cold.6+0x62/0x10a [19184.124007] ? __hrtimer_run_queues+0x1c2/0x4d0 [19184.124011] ? watchdog+0x40/0x40 [19184.124015] ? hrtimer_interrupt+0x128/0x2c0 [19184.124019] ? smp_apic_timer_interrupt+0xa6/0x270 [19184.124028] ? apic_timer_interrupt+0xf/0x20 [19184.124034] [19184.124038] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19184.124042] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19184.124046] do_raw_spin_lock+0x177/0x190 [19184.124051] _raw_spin_lock+0x21/0x30 [19184.124057] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19184.138183] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19184.140195] obd_mod_free+0x46/0x140 [ptlrpc] [19184.140491] mdc_commit_open+0x91/0xb0 [mdc] [19184.140549] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19184.140855] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19184.141072] after_reply+0xe2c/0x1780 [ptlrpc] [19184.141305] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19184.141839] ? __next_timer_interrupt+0x160/0x160 [19184.141850] ptlrpcd+0xa82/0xe40 [ptlrpc] [19184.142097] ? do_wait_intr+0xf0/0xf0 [19184.142106] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19184.142305] kthread+0x1d1/0x200 [19184.142312] ? set_kthread_struct+0x70/0x70 [19184.142316] ret_from_fork+0x1f/0x30 [19184.311351] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19184.313457] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19184.315835] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19184.317484] mdc_intent_lock+0x353/0x7e0 [mdc] [19184.319195] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19184.321124] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19184.323474] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19184.326095] lmv_intent_open+0x338/0x1150 [lmv] [19184.328178] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19184.330290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19184.332254] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19184.334189] ll_intent_lock+0xc5/0xaa0 [lustre] [19184.335945] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19184.337786] ll_lookup_it+0xe0e/0x2270 [lustre] [19184.339490] ? do_raw_spin_unlock+0x75/0x190 [19184.340978] ? _raw_spin_unlock+0x12/0x30 [19184.342324] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19184.344103] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19184.346020] ll_atomic_open+0x36a/0xd70 [lustre] [19184.347978] lookup_open+0x356/0xb40 [19184.349225] do_last+0x313/0xfc0 [19184.350395] ? nd_jump_root+0xe5/0x160 [19184.351741] ? path_init+0x437/0x520 [19184.353051] path_openat+0xf7/0x500 [19184.354194] ? do_raw_spin_unlock+0x75/0x190 [19184.355961] do_filp_open+0x99/0x140 [19184.357268] ? getname_flags+0x6e/0x330 [19184.358613] ? __check_object_size+0xff/0x256 [19184.360227] ? do_raw_spin_unlock+0x75/0x190 [19184.362377] ? _raw_spin_unlock+0x12/0x30 [19184.363882] do_sys_openat2+0x2b4/0x410 [19184.365161] do_sys_open+0x73/0xa0 [19184.366354] __x64_sys_openat+0x24/0x30 [19184.367945] do_syscall_64+0xc1/0x440 [19184.369235] entry_SYSCALL_64_after_hwframe+0x49/0xae [19184.371187] RIP: 0033:0x7f53425a5332 [19184.372617] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19184.381262] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19184.384231] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19184.387783] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19184.390461] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19184.393138] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19184.396063] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19200.057468] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [19200.059031] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19200.066589] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19200.066730] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19200.066737] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19200.066743] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19200.076747] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19200.076827] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19200.076836] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19200.076839] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19200.076854] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19200.076858] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19200.076863] RAX: 0000000000001a16 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19200.076865] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19200.076868] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19200.076870] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19200.076872] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19200.076875] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19200.076878] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19200.076881] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19200.076889] Call Trace: [19200.076896] [19200.076902] ? show_regs.cold.9+0x22/0x2f [19200.076911] ? watchdog_timer_fn.cold.6+0x62/0x10a [19200.076917] ? __hrtimer_run_queues+0x1c2/0x4d0 [19200.076922] ? watchdog+0x40/0x40 [19200.076926] ? hrtimer_interrupt+0x128/0x2c0 [19200.076930] ? smp_apic_timer_interrupt+0xa6/0x270 [19200.076939] ? apic_timer_interrupt+0xf/0x20 [19200.076944] [19200.076946] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19200.076951] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19200.076957] do_raw_spin_lock+0x177/0x190 [19200.076962] _raw_spin_lock+0x21/0x30 [19200.076967] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19200.140316] [19200.140326] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19200.140344] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19200.140350] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 [19200.150610] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19200.158021] ORIG_RAX: ffffffffffffff13 [19200.158031] RAX: 000000000000708b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19200.158034] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19200.158036] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19200.158038] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19200.158040] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19200.158044] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19200.158047] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19200.158050] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19200.158058] Call Trace: [19200.158065] [19200.158070] ? show_regs.cold.9+0x22/0x2f [19200.158084] ? watchdog_timer_fn.cold.6+0x62/0x10a [19200.158091] ? __hrtimer_run_queues+0x1c2/0x4d0 [19200.158096] ? watchdog+0x40/0x40 [19200.158100] ? hrtimer_interrupt+0x128/0x2c0 [19200.158103] ? smp_apic_timer_interrupt+0xa6/0x270 [19200.158111] ? apic_timer_interrupt+0xf/0x20 [19200.158115] [19200.158119] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19200.158125] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19200.158129] ? newidle_balance.isra.93+0x207/0x580 [19200.158137] do_raw_spin_lock+0x177/0x190 [19200.158142] _raw_spin_lock+0x21/0x30 [19200.158147] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19200.164865] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19200.165314] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19200.165786] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19200.168776] ? woken_wake_function+0x30/0x30 [19200.168787] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19200.200200] process_one_work+0x2c8/0x700 [19200.200219] worker_thread+0x296/0x6e0 [19200.200224] ? rescuer_thread+0x570/0x570 [19200.200228] kthread+0x1d1/0x200 [19200.200232] ? set_kthread_struct+0x70/0x70 [19200.200236] ret_from_fork+0x1f/0x30 [19200.600636] kthread+0x1d1/0x200 [19200.604230] ? set_kthread_struct+0x70/0x70 [19200.610570] ret_from_fork+0x1f/0x30 [19206.117208] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16569s! [19206.126712] Showing busy workqueues and worker pools: [19206.135768] workqueue events: flags=0x0 [19206.138408] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19206.147554] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19206.161017] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19206.167210] pending: cpu_latency_work [ptlrpc] [19206.174936] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19206.181089] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19206.194623] workqueue events_long: flags=0x0 [19206.199261] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19206.214133] pending: lu_objects_flush [obdclass] [19206.221868] workqueue events_unbound: flags=0x2 [19206.230051] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19206.238278] pending: flush_memcg_stats_dwork, idle_cull_fn [19206.244944] workqueue events_power_efficient: flags=0x80 [19206.250147] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19206.262065] pending: check_lifetime, do_cache_clean [19206.269690] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19206.277384] pending: neigh_periodic_work, neigh_periodic_work [19206.285365] workqueue mm_percpu_wq: flags=0x8 [19206.289912] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19206.299065] pending: vmstat_update, lru_add_drain_per_cpu [19206.307189] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19206.315746] pending: vmstat_update, lru_add_drain_per_cpu [19206.320664] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19206.326014] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19206.330115] workqueue xprtiod: flags=0xa [19206.337609] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19206.341717] pending: xprt_autoclose, xs_error_handle [19206.345871] workqueue ipv6_addrconf: flags=0x40008 [19206.347645] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19206.355903] pending: addrconf_verify_work [19206.359644] workqueue knbd0-recv: flags=0x1a [19206.363537] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19206.368212] in-flight: 651:recv_work [19206.370117] workqueue ptlrpc_pinger: flags=0x2 [19206.375685] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19206.378675] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19206.382222] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16569s workers=4 idle: 15979 31824 8 [19206.390383] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19196s workers=2 idle: 654 [19212.055009] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [19212.062793] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [19212.071378] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19212.071467] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19212.071474] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19212.071477] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19212.071491] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19212.071495] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19212.071500] RAX: 00000000000021b2 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19212.071502] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19212.071504] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19212.071507] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19212.071509] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19212.071512] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19212.071515] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19212.071518] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19212.071527] Call Trace: [19212.071533] [19212.071539] ? show_regs.cold.9+0x22/0x2f [19212.071550] ? watchdog_timer_fn.cold.6+0x62/0x10a [19212.071556] ? __hrtimer_run_queues+0x1c2/0x4d0 [19212.071560] ? watchdog+0x40/0x40 [19212.071565] ? hrtimer_interrupt+0x128/0x2c0 [19212.071569] ? smp_apic_timer_interrupt+0xa6/0x270 [19212.071577] ? apic_timer_interrupt+0xf/0x20 [19212.071582] [19212.071585] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19212.071591] do_raw_spin_lock+0x177/0x190 [19212.071596] _raw_spin_lock+0x21/0x30 [19212.071602] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19212.084645] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19212.084722] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19212.084730] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19212.084733] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19212.084746] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19212.084750] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19212.084754] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19212.084757] RDX: 0000000000001a96 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19212.084759] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19212.084761] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19212.084763] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19212.084765] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19212.084768] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19212.084769] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19212.084778] Call Trace: [19212.084784] [19212.084790] ? show_regs.cold.9+0x22/0x2f [19212.084799] ? watchdog_timer_fn.cold.6+0x62/0x10a [19212.084805] ? __hrtimer_run_queues+0x1c2/0x4d0 [19212.084809] ? watchdog+0x40/0x40 [19212.084813] ? hrtimer_interrupt+0x128/0x2c0 [19212.084817] ? smp_apic_timer_interrupt+0xa6/0x270 [19212.084826] ? apic_timer_interrupt+0xf/0x20 [19212.084831] [19212.084833] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19212.084838] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19212.084843] do_raw_spin_lock+0x177/0x190 [19212.084847] _raw_spin_lock+0x21/0x30 [19212.084853] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19212.124429] ? do_raw_spin_unlock+0x75/0x190 [19212.124452] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19212.124715] ? do_wait_intr+0xf0/0xf0 [19212.124724] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19212.124915] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19212.125489] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19212.125697] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19212.125814] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19212.125854] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19212.125879] mdc_intent_lock+0x353/0x7e0 [mdc] [19212.125902] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19212.125983] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19212.129276] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19212.129318] lmv_intent_open+0x338/0x1150 [lmv] [19212.129350] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19212.129581] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19212.129664] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19212.143188] obd_mod_free+0x46/0x140 [ptlrpc] [19212.143468] mdc_commit_open+0x91/0xb0 [mdc] [19212.143505] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19212.143711] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19212.143911] after_reply+0xe2c/0x1780 [ptlrpc] [19212.146189] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19212.146435] ? __next_timer_interrupt+0x160/0x160 [19212.146446] ptlrpcd+0xa82/0xe40 [ptlrpc] [19212.146665] ? do_wait_intr+0xf0/0xf0 [19212.146676] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19212.146881] kthread+0x1d1/0x200 [19212.146887] ? set_kthread_struct+0x70/0x70 [19212.146890] ret_from_fork+0x1f/0x30 [19212.781581] ll_intent_lock+0xc5/0xaa0 [lustre] [19212.784971] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19212.794099] ll_lookup_it+0xe0e/0x2270 [lustre] [19212.799681] ? do_raw_spin_unlock+0x75/0x190 [19212.806227] ? _raw_spin_unlock+0x12/0x30 [19212.808289] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19212.817440] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19212.819119] ll_atomic_open+0x36a/0xd70 [lustre] [19212.826270] lookup_open+0x356/0xb40 [19212.828910] do_last+0x313/0xfc0 [19212.835221] ? nd_jump_root+0xe5/0x160 [19212.839931] ? path_init+0x437/0x520 [19212.843941] path_openat+0xf7/0x500 [19212.845696] ? do_raw_spin_unlock+0x75/0x190 [19212.856207] do_filp_open+0x99/0x140 [19212.861747] ? getname_flags+0x6e/0x330 [19212.870128] ? __check_object_size+0xff/0x256 [19212.884398] ? do_raw_spin_unlock+0x75/0x190 [19212.892569] ? _raw_spin_unlock+0x12/0x30 [19212.902559] do_sys_openat2+0x2b4/0x410 [19212.905358] do_sys_open+0x73/0xa0 [19212.907914] __x64_sys_openat+0x24/0x30 [19212.915492] do_syscall_64+0xc1/0x440 [19212.920610] entry_SYSCALL_64_after_hwframe+0x49/0xae [19212.924194] RIP: 0033:0x7f53425a5332 [19212.929899] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19212.952178] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19212.968738] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19212.978360] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19212.984035] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19213.000680] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19213.014722] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19228.062295] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19228.066419] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19228.110980] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19228.119795] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19228.122734] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19228.125419] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19228.132602] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19228.136231] RAX: 0000000000007be6 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19228.138952] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19228.141320] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19228.143677] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19228.149439] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19228.153737] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19228.158060] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19228.162015] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19228.164654] Call Trace: [19228.165516] [19228.166146] ? show_regs.cold.9+0x22/0x2f [19228.167500] ? watchdog_timer_fn.cold.6+0x62/0x10a [19228.169210] ? __hrtimer_run_queues+0x1c2/0x4d0 [19228.170745] ? watchdog+0x40/0x40 [19228.173498] ? hrtimer_interrupt+0x128/0x2c0 [19228.178530] ? smp_apic_timer_interrupt+0xa6/0x270 [19228.180120] ? apic_timer_interrupt+0xf/0x20 [19228.182483] [19228.183230] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19228.185288] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19228.187222] do_raw_spin_lock+0x177/0x190 [19228.188643] _raw_spin_lock+0x21/0x30 [19228.189942] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19228.192140] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19228.194721] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19228.196897] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19228.199045] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19228.201107] ? woken_wake_function+0x30/0x30 [19228.202467] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19228.204574] kthread+0x1d1/0x200 [19228.205683] ? set_kthread_struct+0x70/0x70 [19228.207090] ret_from_fork+0x1f/0x30 [19232.052128] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [19232.057219] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19232.100581] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19232.108703] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19232.117645] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19232.122021] RIP: 0010:native_safe_halt+0x2e/0x50 [19232.127572] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19232.157125] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19232.163010] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19232.170281] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19232.184349] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19232.194904] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19232.202889] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19232.207662] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19232.212280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19232.216580] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19232.224150] Call Trace: [19232.225468] [19232.226339] ? show_regs.cold.9+0x22/0x2f [19232.228648] ? watchdog_timer_fn.cold.6+0x62/0x10a [19232.231147] ? __hrtimer_run_queues+0x1c2/0x4d0 [19232.233265] ? watchdog+0x40/0x40 [19232.234668] ? hrtimer_interrupt+0x128/0x2c0 [19232.236593] ? smp_apic_timer_interrupt+0xa6/0x270 [19232.239018] ? apic_timer_interrupt+0xf/0x20 [19232.241280] [19232.242317] ? native_safe_halt+0x2e/0x50 [19232.244465] kvm_wait+0xcd/0xe0 [19232.246122] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19232.249251] ? newidle_balance.isra.93+0x207/0x580 [19232.251568] do_raw_spin_lock+0x177/0x190 [19232.253484] _raw_spin_lock+0x21/0x30 [19232.255294] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19232.258467] process_one_work+0x2c8/0x700 [19232.260592] worker_thread+0x296/0x6e0 [19232.263292] ? rescuer_thread+0x570/0x570 [19232.265186] kthread+0x1d1/0x200 [19232.266362] ? set_kthread_struct+0x70/0x70 [19232.268943] ret_from_fork+0x1f/0x30 [19236.832032] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16599s! [19236.838000] Showing busy workqueues and worker pools: [19236.840973] workqueue events: flags=0x0 [19236.844783] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19236.849291] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19236.857078] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19236.860861] pending: cpu_latency_work [ptlrpc] [19236.863438] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19236.867578] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19236.873715] workqueue events_long: flags=0x0 [19236.876065] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19236.879905] pending: lu_objects_flush [obdclass] [19236.883027] workqueue events_unbound: flags=0x2 [19236.885878] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19236.889328] pending: flush_memcg_stats_dwork, idle_cull_fn [19236.892826] workqueue events_power_efficient: flags=0x80 [19236.895672] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19236.899208] pending: check_lifetime, do_cache_clean [19236.902342] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19236.906097] pending: neigh_periodic_work, neigh_periodic_work [19236.909824] workqueue mm_percpu_wq: flags=0x8 [19236.912449] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19236.916876] pending: vmstat_update, lru_add_drain_per_cpu [19236.920314] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19236.924948] pending: vmstat_update, lru_add_drain_per_cpu [19236.929288] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19236.933350] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19236.938386] workqueue xprtiod: flags=0xa [19236.943648] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19236.946761] pending: xprt_autoclose, xs_error_handle [19236.949566] workqueue ipv6_addrconf: flags=0x40008 [19236.952355] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19236.956803] pending: addrconf_verify_work [19236.959541] workqueue knbd0-recv: flags=0x1a [19236.961501] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19236.967945] in-flight: 651:recv_work [19236.970679] workqueue ptlrpc_pinger: flags=0x2 [19236.973222] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19236.976117] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19236.980809] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16599s workers=4 idle: 15979 31824 8 [19236.985303] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19227s workers=2 idle: 654 [19240.056150] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [19240.057083] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [19240.063540] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19240.063633] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19240.063640] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19240.063643] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19240.063657] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19240.063661] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19240.063666] RAX: 000000000000335e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19240.063668] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19240.063671] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19240.063673] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19240.063675] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19240.063678] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19240.063681] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19240.063684] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19240.063692] Call Trace: [19240.063698] [19240.063704] ? show_regs.cold.9+0x22/0x2f [19240.063716] ? watchdog_timer_fn.cold.6+0x62/0x10a [19240.063723] ? __hrtimer_run_queues+0x1c2/0x4d0 [19240.063733] ? watchdog+0x40/0x40 [19240.069679] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19240.069754] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19240.069761] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19240.069765] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19240.069778] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19240.069783] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19240.069788] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19240.069790] RDX: 00000000000079fd RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19240.069792] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19240.069794] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19240.069796] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19240.069799] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19240.069802] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19240.069804] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19240.069812] Call Trace: [19240.069817] [19240.069823] ? show_regs.cold.9+0x22/0x2f [19240.069832] ? watchdog_timer_fn.cold.6+0x62/0x10a [19240.069838] ? __hrtimer_run_queues+0x1c2/0x4d0 [19240.069842] ? watchdog+0x40/0x40 [19240.069846] ? hrtimer_interrupt+0x128/0x2c0 [19240.069849] ? smp_apic_timer_interrupt+0xa6/0x270 [19240.069857] ? apic_timer_interrupt+0xf/0x20 [19240.069861] [19240.069864] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19240.069869] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19240.069873] do_raw_spin_lock+0x177/0x190 [19240.069877] _raw_spin_lock+0x21/0x30 [19240.069883] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19240.090140] ? hrtimer_interrupt+0x128/0x2c0 [19240.090160] ? smp_apic_timer_interrupt+0xa6/0x270 [19240.090169] ? apic_timer_interrupt+0xf/0x20 [19240.090174] [19240.090179] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19240.090190] do_raw_spin_lock+0x177/0x190 [19240.090196] _raw_spin_lock+0x21/0x30 [19240.090201] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19240.098018] obd_mod_free+0x46/0x140 [ptlrpc] [19240.101019] ? do_raw_spin_unlock+0x75/0x190 [19240.107838] mdc_commit_open+0x91/0xb0 [mdc] [19240.107890] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19240.108155] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19240.108349] after_reply+0xe2c/0x1780 [ptlrpc] [19240.122166] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19240.128328] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19240.131893] ? do_wait_intr+0xf0/0xf0 [19240.131915] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19240.140215] ? __next_timer_interrupt+0x160/0x160 [19240.140237] ptlrpcd+0xa82/0xe40 [ptlrpc] [19240.145493] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19240.151041] ? do_wait_intr+0xf0/0xf0 [19240.151064] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19240.151334] kthread+0x1d1/0x200 [19240.151340] ? set_kthread_struct+0x70/0x70 [19240.151344] ret_from_fork+0x1f/0x30 [19240.344881] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19240.347535] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19240.351910] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19240.358246] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19240.363819] mdc_intent_lock+0x353/0x7e0 [mdc] [19240.368852] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19240.378447] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19240.380244] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19240.384700] lmv_intent_open+0x338/0x1150 [lmv] [19240.394759] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19240.400864] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19240.404231] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19240.407884] ll_intent_lock+0xc5/0xaa0 [lustre] [19240.410829] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19240.413240] ll_lookup_it+0xe0e/0x2270 [lustre] [19240.417314] ? do_raw_spin_unlock+0x75/0x190 [19240.427082] ? _raw_spin_unlock+0x12/0x30 [19240.430817] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19240.437565] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19240.444876] ll_atomic_open+0x36a/0xd70 [lustre] [19240.448069] lookup_open+0x356/0xb40 [19240.454197] do_last+0x313/0xfc0 [19240.455561] ? nd_jump_root+0xe5/0x160 [19240.456921] ? path_init+0x437/0x520 [19240.459867] path_openat+0xf7/0x500 [19240.463987] ? do_raw_spin_unlock+0x75/0x190 [19240.471236] do_filp_open+0x99/0x140 [19240.473918] ? getname_flags+0x6e/0x330 [19240.476369] ? __check_object_size+0xff/0x256 [19240.479273] ? do_raw_spin_unlock+0x75/0x190 [19240.481426] ? _raw_spin_unlock+0x12/0x30 [19240.485711] do_sys_openat2+0x2b4/0x410 [19240.488034] do_sys_open+0x73/0xa0 [19240.491679] __x64_sys_openat+0x24/0x30 [19240.496516] do_syscall_64+0xc1/0x440 [19240.501023] entry_SYSCALL_64_after_hwframe+0x49/0xae [19240.503095] RIP: 0033:0x7f53425a5332 [19240.505608] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19240.520152] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19240.526811] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19240.532108] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19240.545161] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19240.552710] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19240.561983] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19254.855872] rcu: INFO: rcu_sched self-detected stall on CPU [19254.859835] rcu: 2-....: (9664839 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [19254.867156] (t=17160429 jiffies g=274185 q=68297) [19254.873740] rcu: rcu_sched kthread starved for 16618795 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [19254.879801] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [19254.884599] rcu: RCU grace-period kthread stack dump: [19254.888242] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [19254.893251] Call Trace: [19254.894590] __schedule+0x351/0xcb0 [19254.896738] schedule+0xc0/0x180 [19254.898118] schedule_timeout+0xb4/0x190 [19254.901186] ? do_raw_spin_unlock+0x75/0x190 [19254.902706] ? __next_timer_interrupt+0x160/0x160 [19254.905756] ? prepare_to_swait_event+0xf3/0x140 [19254.911029] rcu_gp_kthread+0x8dd/0x15b0 [19254.913858] ? force_qs_rnp+0x2c0/0x2c0 [19254.916868] kthread+0x1d1/0x200 [19254.920132] ? set_kthread_struct+0x70/0x70 [19254.922690] ret_from_fork+0x1f/0x30 [19254.924714] rcu: Stack dump where RCU GP kthread last ran: [19254.927984] NMI backtrace for cpu 2 [19254.929894] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19254.935599] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19254.939756] Call Trace: [19254.941449] [19254.942813] ? dump_stack+0xbb/0x10e [19254.945528] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [19254.947748] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [19254.949946] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [19254.952405] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [19254.954946] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [19254.957781] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [19254.961165] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [19254.963383] ? update_process_times+0x6d/0xc0 [19254.965737] ? tick_sched_handle.isra.18+0x27/0x90 [19254.968316] ? tick_sched_timer+0x53/0xb0 [19254.969402] ? __hrtimer_run_queues+0x1c2/0x4d0 [19254.972526] ? tick_sched_handle.isra.18+0x90/0x90 [19254.975005] ? hrtimer_interrupt+0x128/0x2c0 [19254.977673] ? smp_apic_timer_interrupt+0xa6/0x270 [19254.979883] ? apic_timer_interrupt+0xf/0x20 [19254.982635] [19254.984428] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19254.986966] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19254.989826] ? do_raw_spin_lock+0x177/0x190 [19254.991962] ? _raw_spin_lock+0x21/0x30 [19254.994450] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19254.999947] ? obd_mod_free+0x46/0x140 [ptlrpc] [19255.005831] ? mdc_commit_open+0x91/0xb0 [mdc] [19255.012737] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19255.021641] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19255.026715] ? after_reply+0xe2c/0x1780 [ptlrpc] [19255.034790] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19255.044862] ? __next_timer_interrupt+0x160/0x160 [19255.056021] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [19255.064264] ? do_wait_intr+0xf0/0xf0 [19255.065472] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19255.070040] ? kthread+0x1d1/0x200 [19255.072252] ? set_kthread_struct+0x70/0x70 [19255.074132] ? ret_from_fork+0x1f/0x30 [19255.077237] Sending NMI from CPU 2 to CPUs 1: [19255.081242] NMI backtrace for cpu 1 [19255.081249] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19255.081251] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19255.081252] RIP: 0010:native_write_msr+0xe/0x50 [19255.081254] Code: 05 c6 54 e6 02 01 8b 44 24 04 48 83 c4 08 c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 83 05 48 54 e6 02 01 89 f9 89 f0 0f 30 <48> 83 05 42 54 e6 02 01 0f 1f 44 00 00 48 83 05 45 54 e6 02 01 48 [19255.081257] RSP: 0018:ffffbb5380d24f38 EFLAGS: 00000006 [19255.081259] RAX: 00000000610fd2f4 RBX: 00000000000946ba RCX: 00000000000006e0 [19255.081261] RDX: 0000000000002a08 RSI: 00000000610fd2f4 RDI: 00000000000006e0 [19255.081262] RBP: 0000000000000000 R08: 000011832c24d7c0 R09: 000011832c24d7c0 [19255.081264] R10: 00000000000f4240 R11: 000011832c24d7c0 R12: ffff98a70209df40 [19255.081265] R13: 000011832c341a00 R14: ffff98a7020a4e40 R15: ffff98a7020a4fb8 [19255.081267] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19255.081268] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19255.081269] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19255.081270] Call Trace: [19255.081271] [19255.081272] ? show_regs.cold.9+0x22/0x2f [19255.081273] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [19255.081274] ? nmi_cpu_backtrace_handler+0x15/0x40 [19255.081275] ? nmi_handle+0xbd/0x210 [19255.081276] ? default_do_nmi+0x4b/0x1b0 [19255.081277] ? do_nmi+0x280/0x380 [19255.081278] ? end_repeat_nmi+0x16/0x1f [19255.081279] ? native_write_msr+0xe/0x50 [19255.081280] ? native_write_msr+0xe/0x50 [19255.081281] ? native_write_msr+0xe/0x50 [19255.081282] [19255.081282] [19255.081283] lapic_next_deadline+0x3a/0x50 [19255.081293] clockevents_program_event+0xaf/0x1a0 [19255.081294] tick_program_event+0x47/0xa0 [19255.081295] hrtimer_interrupt+0x169/0x2c0 [19255.081297] smp_apic_timer_interrupt+0xa6/0x270 [19255.081297] apic_timer_interrupt+0xf/0x20 [19255.081298] [19255.081299] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19255.081301] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19255.081304] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19255.081306] RAX: 0000000000002ccd RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19255.081308] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19255.081309] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19255.081310] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19255.081312] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19255.081313] do_raw_spin_lock+0x177/0x190 [19255.081314] _raw_spin_lock+0x21/0x30 [19255.081315] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19255.081316] ? do_raw_spin_unlock+0x75/0x190 [19255.081317] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19255.081318] ? do_wait_intr+0xf0/0xf0 [19255.081319] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19255.081320] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19255.081321] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19255.081322] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19255.081324] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19255.081325] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19255.081326] mdc_intent_lock+0x353/0x7e0 [mdc] [19255.081327] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19255.081328] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19255.081329] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19255.081330] lmv_intent_open+0x338/0x1150 [lmv] [19255.081331] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19255.081332] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19255.081334] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19255.081335] ll_intent_lock+0xc5/0xaa0 [lustre] [19255.081336] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19255.081337] ll_lookup_it+0xe0e/0x2270 [lustre] [19255.081338] ? do_raw_spin_unlock+0x75/0x190 [19255.081339] ? _raw_spin_unlock+0x12/0x30 [19255.081340] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19255.081341] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19255.081342] ll_atomic_open+0x36a/0xd70 [lustre] [19255.081344] lookup_open+0x356/0xb40 [19255.081345] do_last+0x313/0xfc0 [19255.081346] ? nd_jump_root+0xe5/0x160 [19255.081347] ? path_init+0x437/0x520 [19255.081348] path_openat+0xf7/0x500 [19255.081349] ? do_raw_spin_unlock+0x75/0x190 [19255.081350] do_filp_open+0x99/0x140 [19255.081351] ? getname_flags+0x6e/0x330 [19255.081352] ? __check_object_size+0xff/0x256 [19255.081354] ? do_raw_spin_unlock+0x75/0x190 [19255.081355] ? _raw_spin_unlock+0x12/0x30 [19255.081356] do_sys_openat2+0x2b4/0x410 [19255.081357] do_sys_open+0x73/0xa0 [19255.081358] __x64_sys_openat+0x24/0x30 [19255.081359] do_syscall_64+0xc1/0x440 [19255.081360] entry_SYSCALL_64_after_hwframe+0x49/0xae [19255.081362] RIP: 0033:0x7f53425a5332 [19255.081363] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19255.081365] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19255.081368] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19255.081369] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19255.081371] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19255.081372] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19255.081373] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19255.081609] NMI backtrace for cpu 2 [19255.546130] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19255.552807] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19255.556558] Call Trace: [19255.557989] [19255.561635] ? dump_stack+0xbb/0x10e [19255.563105] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [19255.565789] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [19255.571873] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [19255.573822] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [19255.575633] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [19255.577681] ? rcu_dump_cpu_stacks+0x11b/0x171 [19255.579346] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [19255.581551] ? update_process_times+0x6d/0xc0 [19255.583258] ? tick_sched_handle.isra.18+0x27/0x90 [19255.595483] ? tick_sched_timer+0x53/0xb0 [19255.596802] ? __hrtimer_run_queues+0x1c2/0x4d0 [19255.602482] ? tick_sched_handle.isra.18+0x90/0x90 [19255.604234] ? hrtimer_interrupt+0x128/0x2c0 [19255.609767] ? smp_apic_timer_interrupt+0xa6/0x270 [19255.612971] ? apic_timer_interrupt+0xf/0x20 [19255.617380] [19255.617977] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19255.623619] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19255.629964] ? do_raw_spin_lock+0x177/0x190 [19255.632587] ? _raw_spin_lock+0x21/0x30 [19255.633851] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19255.638953] ? obd_mod_free+0x46/0x140 [ptlrpc] [19255.644612] ? mdc_commit_open+0x91/0xb0 [mdc] [19255.646777] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19255.651806] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19255.655951] ? after_reply+0xe2c/0x1780 [ptlrpc] [19255.658915] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19255.662942] ? __next_timer_interrupt+0x160/0x160 [19255.666533] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [19255.668995] ? do_wait_intr+0xf0/0xf0 [19255.670799] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19255.673268] ? kthread+0x1d1/0x200 [19255.674840] ? set_kthread_struct+0x70/0x70 [19255.676703] ? ret_from_fork+0x1f/0x30 [19256.062034] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19256.070475] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19256.109080] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19256.114087] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19256.121251] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19256.125227] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19256.147987] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19256.156489] RAX: 0000000000005f4a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19256.161245] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19256.166143] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19256.174929] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19256.180470] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19256.184891] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19256.187553] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19256.194468] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19256.200683] Call Trace: [19256.202024] [19256.203519] ? show_regs.cold.9+0x22/0x2f [19256.205484] ? watchdog_timer_fn.cold.6+0x62/0x10a [19256.209247] ? __hrtimer_run_queues+0x1c2/0x4d0 [19256.215014] ? watchdog+0x40/0x40 [19256.218958] ? hrtimer_interrupt+0x128/0x2c0 [19256.221249] ? smp_apic_timer_interrupt+0xa6/0x270 [19256.222624] ? apic_timer_interrupt+0xf/0x20 [19256.224656] [19256.225674] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19256.227913] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19256.230577] do_raw_spin_lock+0x177/0x190 [19256.232058] _raw_spin_lock+0x21/0x30 [19256.233595] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19256.236223] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19256.238759] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19256.241467] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19256.246880] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19256.253105] ? woken_wake_function+0x30/0x30 [19256.255241] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19256.259788] kthread+0x1d1/0x200 [19256.261655] ? set_kthread_struct+0x70/0x70 [19256.263374] ret_from_fork+0x1f/0x30 [19264.052483] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [19264.055930] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19264.076532] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19264.080990] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19264.084302] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19264.086914] RIP: 0010:native_safe_halt+0x2e/0x50 [19264.088670] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19264.095413] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19264.097879] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19264.100959] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [19264.103892] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19264.106455] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19264.109660] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19264.111990] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19264.114482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19264.115988] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19264.118504] Call Trace: [19264.119399] [19264.119916] ? show_regs.cold.9+0x22/0x2f [19264.123223] ? watchdog_timer_fn.cold.6+0x62/0x10a [19264.125157] ? __hrtimer_run_queues+0x1c2/0x4d0 [19264.126995] ? watchdog+0x40/0x40 [19264.128478] ? hrtimer_interrupt+0x128/0x2c0 [19264.130407] ? smp_apic_timer_interrupt+0xa6/0x270 [19264.132522] ? apic_timer_interrupt+0xf/0x20 [19264.135227] [19264.136077] ? native_safe_halt+0x2e/0x50 [19264.137603] kvm_wait+0xcd/0xe0 [19264.138980] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19264.140491] ? newidle_balance.isra.93+0x207/0x580 [19264.142553] do_raw_spin_lock+0x177/0x190 [19264.144191] _raw_spin_lock+0x21/0x30 [19264.145653] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19264.147833] process_one_work+0x2c8/0x700 [19264.149469] worker_thread+0x296/0x6e0 [19264.150785] ? rescuer_thread+0x570/0x570 [19264.151995] kthread+0x1d1/0x200 [19264.153061] ? set_kthread_struct+0x70/0x70 [19264.154499] ret_from_fork+0x1f/0x30 [19267.042231] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16629s! [19267.046785] Showing busy workqueues and worker pools: [19267.052859] workqueue events: flags=0x0 [19267.053973] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19267.058596] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19267.068285] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19267.071065] pending: cpu_latency_work [ptlrpc] [19267.075862] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19267.078538] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19267.086388] workqueue events_long: flags=0x0 [19267.088826] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19267.098293] pending: lu_objects_flush [obdclass] [19267.104589] workqueue events_unbound: flags=0x2 [19267.109960] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19267.113937] pending: flush_memcg_stats_dwork, idle_cull_fn [19267.117039] workqueue events_power_efficient: flags=0x80 [19267.119886] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19267.125807] pending: check_lifetime, do_cache_clean [19267.130782] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19267.134060] pending: neigh_periodic_work, neigh_periodic_work [19267.136463] workqueue mm_percpu_wq: flags=0x8 [19267.137695] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19267.147961] pending: vmstat_update, lru_add_drain_per_cpu [19267.153428] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19267.155972] pending: vmstat_update, lru_add_drain_per_cpu [19267.161582] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19267.167319] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19267.178617] workqueue xprtiod: flags=0xa [19267.182628] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19267.186742] pending: xprt_autoclose, xs_error_handle [19267.193918] workqueue ipv6_addrconf: flags=0x40008 [19267.199924] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19267.206847] pending: addrconf_verify_work [19267.209982] workqueue knbd0-recv: flags=0x1a [19267.212540] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19267.216213] in-flight: 651:recv_work [19267.217650] workqueue ptlrpc_pinger: flags=0x2 [19267.219826] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19267.226743] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19267.230919] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16630s workers=4 idle: 15979 31824 8 [19267.244772] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19257s workers=2 idle: 654 [19280.055032] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [19280.057173] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [19280.058117] Modules linked in: lustre(O) [19280.063253] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) [19280.064552] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19280.064624] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19280.064630] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19280.064633] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19280.064646] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19280.064650] RSP: 0018:ffffbb5381503500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [19280.064655] RAX: 00000000000013f1 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19280.064657] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19280.064659] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19280.064661] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19280.064664] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19280.064667] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19280.064670] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19280.064672] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19280.064681] Call Trace: [19280.064687] [19280.064692] ? show_regs.cold.9+0x22/0x2f [19280.064701] ? watchdog_timer_fn.cold.6+0x62/0x10a [19280.064708] ? __hrtimer_run_queues+0x1c2/0x4d0 [19280.064712] ? watchdog+0x40/0x40 [19280.064716] ? hrtimer_interrupt+0x128/0x2c0 [19280.064720] ? smp_apic_timer_interrupt+0xa6/0x270 [19280.064729] ? apic_timer_interrupt+0xf/0x20 [19280.064734] [19280.064736] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19280.064744] do_raw_spin_lock+0x177/0x190 [19280.064749] _raw_spin_lock+0x21/0x30 [19280.064754] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19280.068952] mdd(O) mgs(O) lquota(O) [19280.085575] ? do_raw_spin_unlock+0x75/0x190 [19280.092209] lfsck(O) [19280.096293] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19280.099958] obdecho(O) [19280.107694] ? do_wait_intr+0xf0/0xf0 [19280.107721] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19280.111485] mgc(O) [19280.114245] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19280.118409] mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19280.118469] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19280.118476] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19280.118479] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19280.118494] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19280.118498] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19280.118502] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19280.118504] RDX: 0000000000007f0b RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19280.118507] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19280.118509] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19280.118512] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19280.118515] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19280.118518] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19280.120407] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19280.123662] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19280.128673] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19280.133473] Call Trace: [19280.133486] [19280.133491] ? show_regs.cold.9+0x22/0x2f [19280.133505] ? watchdog_timer_fn.cold.6+0x62/0x10a [19280.133513] ? __hrtimer_run_queues+0x1c2/0x4d0 [19280.133518] ? watchdog+0x40/0x40 [19280.135985] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19280.139814] ? hrtimer_interrupt+0x128/0x2c0 [19280.140759] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19280.140806] mdc_intent_lock+0x353/0x7e0 [mdc] [19280.141490] ? smp_apic_timer_interrupt+0xa6/0x270 [19280.143271] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19280.145934] ? apic_timer_interrupt+0xf/0x20 [19280.145956] [19280.145962] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19280.145969] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19280.145974] do_raw_spin_lock+0x177/0x190 [19280.145980] _raw_spin_lock+0x21/0x30 [19280.145987] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19280.146269] obd_mod_free+0x46/0x140 [ptlrpc] [19280.146469] mdc_commit_open+0x91/0xb0 [mdc] [19280.146508] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19280.148108] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19280.150120] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19280.151996] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19280.156406] after_reply+0xe2c/0x1780 [ptlrpc] [19280.158082] lmv_intent_open+0x338/0x1150 [lmv] [19280.158129] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19280.159245] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19280.163016] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19280.165080] ? __next_timer_interrupt+0x160/0x160 [19280.165100] ptlrpcd+0xa82/0xe40 [ptlrpc] [19280.165460] ? do_wait_intr+0xf0/0xf0 [19280.165470] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19280.165895] kthread+0x1d1/0x200 [19280.165901] ? set_kthread_struct+0x70/0x70 [19280.165905] ret_from_fork+0x1f/0x30 [19280.167653] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19280.332031] ll_intent_lock+0xc5/0xaa0 [lustre] [19280.335106] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19280.336406] ll_lookup_it+0xe0e/0x2270 [lustre] [19280.337641] ? do_raw_spin_unlock+0x75/0x190 [19280.340222] ? _raw_spin_unlock+0x12/0x30 [19280.342202] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19280.344785] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19280.346089] ll_atomic_open+0x36a/0xd70 [lustre] [19280.348069] lookup_open+0x356/0xb40 [19280.349574] do_last+0x313/0xfc0 [19280.350804] ? nd_jump_root+0xe5/0x160 [19280.352629] ? path_init+0x437/0x520 [19280.353829] path_openat+0xf7/0x500 [19280.356580] ? do_raw_spin_unlock+0x75/0x190 [19280.358401] do_filp_open+0x99/0x140 [19280.359772] ? getname_flags+0x6e/0x330 [19280.362689] ? __check_object_size+0xff/0x256 [19280.364084] ? do_raw_spin_unlock+0x75/0x190 [19280.365801] ? _raw_spin_unlock+0x12/0x30 [19280.367608] do_sys_openat2+0x2b4/0x410 [19280.368586] do_sys_open+0x73/0xa0 [19280.369837] __x64_sys_openat+0x24/0x30 [19280.371118] do_syscall_64+0xc1/0x440 [19280.372066] entry_SYSCALL_64_after_hwframe+0x49/0xae [19280.373272] RIP: 0033:0x7f53425a5332 [19280.375222] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19280.386298] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19280.393263] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19280.398233] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19280.400661] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19280.403199] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19280.409740] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19284.058074] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19284.062157] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19284.119838] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19284.139124] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19284.142104] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19284.145382] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19284.158230] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19284.170264] RAX: 0000000000000587 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19284.176539] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19284.183016] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19284.188750] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19284.193447] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19284.196636] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19284.199562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19284.201692] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19284.204332] Call Trace: [19284.205254] [19284.205967] ? show_regs.cold.9+0x22/0x2f [19284.207485] ? watchdog_timer_fn.cold.6+0x62/0x10a [19284.209045] ? __hrtimer_run_queues+0x1c2/0x4d0 [19284.210673] ? watchdog+0x40/0x40 [19284.211935] ? hrtimer_interrupt+0x128/0x2c0 [19284.213489] ? smp_apic_timer_interrupt+0xa6/0x270 [19284.215145] ? apic_timer_interrupt+0xf/0x20 [19284.216666] [19284.217478] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19284.219468] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19284.221356] do_raw_spin_lock+0x177/0x190 [19284.222652] _raw_spin_lock+0x21/0x30 [19284.223613] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19284.225632] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19284.227657] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19284.229576] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19284.231912] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19284.234191] ? woken_wake_function+0x30/0x30 [19284.235874] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19284.238268] kthread+0x1d1/0x200 [19284.239645] ? set_kthread_struct+0x70/0x70 [19284.241443] ret_from_fork+0x1f/0x30 [19292.052024] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [19292.055338] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19292.076673] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19292.081184] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19292.083817] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19292.086741] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19292.089239] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19292.098054] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19292.102677] RAX: 0000000000005806 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19292.106461] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19292.111614] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19292.115161] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19292.119204] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19292.122791] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19292.127474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19292.131201] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19292.135203] Call Trace: [19292.136159] [19292.136964] ? show_regs.cold.9+0x22/0x2f [19292.138597] ? watchdog_timer_fn.cold.6+0x62/0x10a [19292.140520] ? __hrtimer_run_queues+0x1c2/0x4d0 [19292.142508] ? watchdog+0x40/0x40 [19292.144104] ? hrtimer_interrupt+0x128/0x2c0 [19292.145856] ? smp_apic_timer_interrupt+0xa6/0x270 [19292.147624] ? apic_timer_interrupt+0xf/0x20 [19292.149591] [19292.150337] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19292.152648] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19292.154728] ? newidle_balance.isra.93+0x207/0x580 [19292.156722] do_raw_spin_lock+0x177/0x190 [19292.158163] _raw_spin_lock+0x21/0x30 [19292.159723] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19292.162279] process_one_work+0x2c8/0x700 [19292.164216] worker_thread+0x296/0x6e0 [19292.165846] ? rescuer_thread+0x570/0x570 [19292.167690] kthread+0x1d1/0x200 [19292.169417] ? set_kthread_struct+0x70/0x70 [19292.171117] ret_from_fork+0x1f/0x30 [19297.761282] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16660s! [19297.767767] Showing busy workqueues and worker pools: [19297.770996] workqueue events: flags=0x0 [19297.779771] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19297.785556] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19297.792513] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19297.796973] pending: cpu_latency_work [ptlrpc] [19297.800386] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19297.804908] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19297.809644] workqueue events_long: flags=0x0 [19297.811795] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19297.814865] pending: lu_objects_flush [obdclass] [19297.817204] workqueue events_unbound: flags=0x2 [19297.820082] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19297.823369] pending: flush_memcg_stats_dwork, idle_cull_fn [19297.825862] workqueue events_power_efficient: flags=0x80 [19297.827948] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19297.831090] pending: check_lifetime, do_cache_clean [19297.833601] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19297.837190] pending: neigh_periodic_work, neigh_periodic_work [19297.841282] workqueue mm_percpu_wq: flags=0x8 [19297.843408] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19297.846730] pending: vmstat_update, lru_add_drain_per_cpu [19297.849589] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19297.852953] pending: vmstat_update, lru_add_drain_per_cpu [19297.856406] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19297.859125] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19297.862273] workqueue xprtiod: flags=0xa [19297.864096] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19297.866940] pending: xprt_autoclose, xs_error_handle [19297.870081] workqueue ipv6_addrconf: flags=0x40008 [19297.872454] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19297.875364] pending: addrconf_verify_work [19297.877737] workqueue knbd0-recv: flags=0x1a [19297.879908] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19297.883526] in-flight: 651:recv_work [19297.886070] workqueue ptlrpc_pinger: flags=0x2 [19297.888884] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19297.891506] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19297.894867] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16660s workers=4 idle: 15979 31824 8 [19297.898847] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19288s workers=2 idle: 654 [19308.055039] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [19308.057169] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [19308.060927] Modules linked in: [19308.068373] Modules linked in: [19308.068376] lustre(O) [19308.069433] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19308.069505] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19308.069512] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19308.069515] RIP: 0010:native_safe_halt+0x2e/0x50 [19308.069531] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19308.073772] osp(O) [19308.075215] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19308.075227] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19308.075230] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19308.075232] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19308.075234] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19308.075237] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19308.075240] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19308.075243] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19308.075245] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19308.075254] Call Trace: [19308.075262] [19308.075444] ? show_regs.cold.9+0x22/0x2f [19308.075459] ? watchdog_timer_fn.cold.6+0x62/0x10a [19308.075465] ? __hrtimer_run_queues+0x1c2/0x4d0 [19308.075470] ? watchdog+0x40/0x40 [19308.075474] ? hrtimer_interrupt+0x128/0x2c0 [19308.075479] ? smp_apic_timer_interrupt+0xa6/0x270 [19308.075487] ? apic_timer_interrupt+0xf/0x20 [19308.075492] [19308.075495] ? native_safe_halt+0x2e/0x50 [19308.075500] kvm_wait+0xcd/0xe0 [19308.112342] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19308.112480] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19308.112489] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19308.112492] RIP: 0010:native_safe_halt+0x2e/0x50 [19308.112506] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19308.112511] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19308.112516] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19308.112519] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19308.112521] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19308.112524] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19308.112526] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19308.112530] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19308.112533] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19308.112535] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19308.113019] Call Trace: [19308.120041] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19308.120064] do_raw_spin_lock+0x177/0x190 [19308.120069] _raw_spin_lock+0x21/0x30 [19308.120078] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19308.120351] obd_mod_free+0x46/0x140 [ptlrpc] [19308.120583] mdc_commit_open+0x91/0xb0 [mdc] [19308.120622] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19308.120850] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19308.121066] after_reply+0xe2c/0x1780 [ptlrpc] [19308.121267] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19308.127851] [19308.127863] ? show_regs.cold.9+0x22/0x2f [19308.127878] ? watchdog_timer_fn.cold.6+0x62/0x10a [19308.127884] ? __hrtimer_run_queues+0x1c2/0x4d0 [19308.127889] ? watchdog+0x40/0x40 [19308.127893] ? hrtimer_interrupt+0x128/0x2c0 [19308.127897] ? smp_apic_timer_interrupt+0xa6/0x270 [19308.127906] ? apic_timer_interrupt+0xf/0x20 [19308.127910] [19308.127913] ? native_safe_halt+0x2e/0x50 [19308.127917] kvm_wait+0xcd/0xe0 [19308.127925] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19308.130507] ? __next_timer_interrupt+0x160/0x160 [19308.130526] ptlrpcd+0xa82/0xe40 [ptlrpc] [19308.131004] ? do_wait_intr+0xf0/0xf0 [19308.131015] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19308.131221] kthread+0x1d1/0x200 [19308.131227] ? set_kthread_struct+0x70/0x70 [19308.131232] ret_from_fork+0x1f/0x30 [19308.366199] do_raw_spin_lock+0x177/0x190 [19308.367802] _raw_spin_lock+0x21/0x30 [19308.369299] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19308.372717] ? do_raw_spin_unlock+0x75/0x190 [19308.379795] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19308.386294] ? do_wait_intr+0xf0/0xf0 [19308.392490] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19308.394938] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19308.397235] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19308.401313] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19308.404232] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19308.407015] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19308.409122] mdc_intent_lock+0x353/0x7e0 [mdc] [19308.410622] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19308.416783] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19308.420857] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19308.423232] lmv_intent_open+0x338/0x1150 [lmv] [19308.424779] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19308.428561] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19308.433303] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19308.435181] ll_intent_lock+0xc5/0xaa0 [lustre] [19308.437143] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19308.446113] ll_lookup_it+0xe0e/0x2270 [lustre] [19308.449968] ? do_raw_spin_unlock+0x75/0x190 [19308.453303] ? _raw_spin_unlock+0x12/0x30 [19308.456596] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19308.464584] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19308.471601] ll_atomic_open+0x36a/0xd70 [lustre] [19308.479586] lookup_open+0x356/0xb40 [19308.484830] do_last+0x313/0xfc0 [19308.488845] ? nd_jump_root+0xe5/0x160 [19308.495555] ? path_init+0x437/0x520 [19308.496817] path_openat+0xf7/0x500 [19308.507437] ? do_raw_spin_unlock+0x75/0x190 [19308.510117] do_filp_open+0x99/0x140 [19308.511861] ? getname_flags+0x6e/0x330 [19308.513848] ? __check_object_size+0xff/0x256 [19308.517060] ? do_raw_spin_unlock+0x75/0x190 [19308.519975] ? _raw_spin_unlock+0x12/0x30 [19308.521865] do_sys_openat2+0x2b4/0x410 [19308.525776] do_sys_open+0x73/0xa0 [19308.527572] __x64_sys_openat+0x24/0x30 [19308.530868] do_syscall_64+0xc1/0x440 [19308.535142] entry_SYSCALL_64_after_hwframe+0x49/0xae [19308.544903] RIP: 0033:0x7f53425a5332 [19308.547547] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19308.555461] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19308.558482] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19308.561671] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19308.569013] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19308.573336] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19308.577427] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19312.066301] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19312.078456] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19312.095454] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19312.099760] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19312.104371] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19312.112399] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19312.126236] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19312.131404] RAX: 0000000000007f10 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19312.137177] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19312.143608] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19312.151927] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19312.153894] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19312.168009] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19312.175171] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19312.184136] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19312.191863] Call Trace: [19312.194883] [19312.196857] ? show_regs.cold.9+0x22/0x2f [19312.198497] ? watchdog_timer_fn.cold.6+0x62/0x10a [19312.201103] ? __hrtimer_run_queues+0x1c2/0x4d0 [19312.203045] ? watchdog+0x40/0x40 [19312.204855] ? hrtimer_interrupt+0x128/0x2c0 [19312.210947] ? smp_apic_timer_interrupt+0xa6/0x270 [19312.213111] ? apic_timer_interrupt+0xf/0x20 [19312.214926] [19312.216699] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19312.218191] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19312.219744] do_raw_spin_lock+0x177/0x190 [19312.224189] _raw_spin_lock+0x21/0x30 [19312.226782] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19312.229863] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19312.234951] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19312.240198] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19312.244151] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19312.247382] ? woken_wake_function+0x30/0x30 [19312.249343] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19312.256224] kthread+0x1d1/0x200 [19312.268617] ? set_kthread_struct+0x70/0x70 [19312.270021] ret_from_fork+0x1f/0x30 [19324.053357] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [19324.063115] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19324.120950] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19324.130646] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19324.138708] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19324.143264] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19324.147512] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19324.159359] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19324.162345] RAX: 000000000000702e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19324.172178] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19324.177029] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19324.181463] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19324.192426] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19324.200161] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19324.203042] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19324.208468] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19324.212644] Call Trace: [19324.215301] [19324.216205] ? show_regs.cold.9+0x22/0x2f [19324.219468] ? watchdog_timer_fn.cold.6+0x62/0x10a [19324.223750] ? __hrtimer_run_queues+0x1c2/0x4d0 [19324.229842] ? watchdog+0x40/0x40 [19324.232242] ? hrtimer_interrupt+0x128/0x2c0 [19324.239331] ? smp_apic_timer_interrupt+0xa6/0x270 [19324.243509] ? apic_timer_interrupt+0xf/0x20 [19324.245569] [19324.246579] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19324.252362] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19324.254065] ? newidle_balance.isra.93+0x207/0x580 [19324.258581] do_raw_spin_lock+0x177/0x190 [19324.260616] _raw_spin_lock+0x21/0x30 [19324.266763] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19324.281848] process_one_work+0x2c8/0x700 [19324.286006] worker_thread+0x296/0x6e0 [19324.295130] ? rescuer_thread+0x570/0x570 [19324.300351] kthread+0x1d1/0x200 [19324.304758] ? set_kthread_struct+0x70/0x70 [19324.306897] ret_from_fork+0x1f/0x30 [19327.984179] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16690s! [19327.989159] Showing busy workqueues and worker pools: [19327.995193] workqueue events: flags=0x0 [19328.000385] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19328.004363] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19328.011318] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19328.019344] pending: cpu_latency_work [ptlrpc] [19328.024245] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19328.030949] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19328.055143] workqueue events_long: flags=0x0 [19328.056808] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19328.074535] pending: lu_objects_flush [obdclass] [19328.083042] workqueue events_unbound: flags=0x2 [19328.084640] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19328.089381] pending: flush_memcg_stats_dwork, idle_cull_fn [19328.099343] workqueue events_power_efficient: flags=0x80 [19328.105336] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19328.116065] pending: check_lifetime, do_cache_clean [19328.127272] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19328.134003] pending: neigh_periodic_work, neigh_periodic_work [19328.143614] workqueue mm_percpu_wq: flags=0x8 [19328.152350] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19328.165917] pending: vmstat_update, lru_add_drain_per_cpu [19328.170926] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19328.179239] pending: vmstat_update, lru_add_drain_per_cpu [19328.188427] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19328.197365] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19328.200500] workqueue xprtiod: flags=0xa [19328.217754] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19328.224935] pending: xprt_autoclose, xs_error_handle [19328.235470] workqueue ipv6_addrconf: flags=0x40008 [19328.251129] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19328.263061] pending: addrconf_verify_work [19328.264700] workqueue knbd0-recv: flags=0x1a [19328.275399] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19328.285554] in-flight: 651:recv_work [19328.289530] workqueue ptlrpc_pinger: flags=0x2 [19328.296698] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19328.308298] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19328.314100] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16691s workers=4 idle: 15979 31824 8 [19328.330742] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19318s workers=2 idle: 654 [19336.055025] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [19336.058257] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [19336.061071] Modules linked in: [19336.067152] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19336.067302] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19336.067310] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19336.067313] RIP: 0010:native_safe_halt+0x2e/0x50 [19336.067327] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19336.067331] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19336.067336] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19336.067339] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19336.067341] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19336.067344] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19336.067346] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19336.067350] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19336.067353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19336.067355] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19336.067363] Call Trace: [19336.067369] [19336.067374] ? show_regs.cold.9+0x22/0x2f [19336.067384] ? watchdog_timer_fn.cold.6+0x62/0x10a [19336.067391] ? __hrtimer_run_queues+0x1c2/0x4d0 [19336.067395] ? watchdog+0x40/0x40 [19336.067399] ? hrtimer_interrupt+0x128/0x2c0 [19336.067404] ? smp_apic_timer_interrupt+0xa6/0x270 [19336.067411] ? apic_timer_interrupt+0xf/0x20 [19336.067416] [19336.067418] ? native_safe_halt+0x2e/0x50 [19336.067423] kvm_wait+0xcd/0xe0 [19336.067430] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19336.067437] do_raw_spin_lock+0x177/0x190 [19336.067442] _raw_spin_lock+0x21/0x30 [19336.067447] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19336.068140] obd_mod_free+0x46/0x140 [ptlrpc] [19336.071450] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19336.071528] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19336.071535] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19336.071538] RIP: 0010:native_safe_halt+0x2e/0x50 [19336.071551] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19336.071555] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19336.071560] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19336.071563] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19336.071565] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19336.071567] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19336.071570] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19336.071573] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19336.071577] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19336.071579] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19336.071587] Call Trace: [19336.071592] [19336.071597] ? show_regs.cold.9+0x22/0x2f [19336.071606] ? watchdog_timer_fn.cold.6+0x62/0x10a [19336.071612] ? __hrtimer_run_queues+0x1c2/0x4d0 [19336.072144] ? watchdog+0x40/0x40 [19336.072149] ? hrtimer_interrupt+0x128/0x2c0 [19336.072153] ? smp_apic_timer_interrupt+0xa6/0x270 [19336.072161] ? apic_timer_interrupt+0xf/0x20 [19336.072166] [19336.072169] ? native_safe_halt+0x2e/0x50 [19336.072173] kvm_wait+0xcd/0xe0 [19336.072180] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19336.072188] do_raw_spin_lock+0x177/0x190 [19336.072192] _raw_spin_lock+0x21/0x30 [19336.072197] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19336.108001] mdc_commit_open+0x91/0xb0 [mdc] [19336.114937] ? do_raw_spin_unlock+0x75/0x190 [19336.114960] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19336.115229] ? do_wait_intr+0xf0/0xf0 [19336.115239] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19336.115455] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19336.115728] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19336.116171] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19336.116284] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19336.116334] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19336.116423] mdc_intent_lock+0x353/0x7e0 [mdc] [19336.116455] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19336.116544] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19336.116765] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19336.116813] lmv_intent_open+0x338/0x1150 [lmv] [19336.116852] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19336.117029] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19336.117127] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19336.117151] ll_intent_lock+0xc5/0xaa0 [lustre] [19336.117247] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19336.117332] ll_lookup_it+0xe0e/0x2270 [lustre] [19336.122076] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19336.122487] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19336.122704] after_reply+0xe2c/0x1780 [ptlrpc] [19336.123169] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19336.123417] ? __next_timer_interrupt+0x160/0x160 [19336.123428] ptlrpcd+0xa82/0xe40 [ptlrpc] [19336.123645] ? do_wait_intr+0xf0/0xf0 [19336.123655] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19336.124046] kthread+0x1d1/0x200 [19336.124053] ? set_kthread_struct+0x70/0x70 [19336.124058] ret_from_fork+0x1f/0x30 [19336.400111] ? do_raw_spin_unlock+0x75/0x190 [19336.402365] ? _raw_spin_unlock+0x12/0x30 [19336.404477] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19336.407015] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19336.408918] ll_atomic_open+0x36a/0xd70 [lustre] [19336.411507] lookup_open+0x356/0xb40 [19336.413310] do_last+0x313/0xfc0 [19336.414720] ? nd_jump_root+0xe5/0x160 [19336.416289] ? path_init+0x437/0x520 [19336.418148] path_openat+0xf7/0x500 [19336.419730] ? do_raw_spin_unlock+0x75/0x190 [19336.422098] do_filp_open+0x99/0x140 [19336.423688] ? getname_flags+0x6e/0x330 [19336.425694] ? __check_object_size+0xff/0x256 [19336.427673] ? do_raw_spin_unlock+0x75/0x190 [19336.429746] ? _raw_spin_unlock+0x12/0x30 [19336.431971] do_sys_openat2+0x2b4/0x410 [19336.433651] do_sys_open+0x73/0xa0 [19336.435403] __x64_sys_openat+0x24/0x30 [19336.437069] do_syscall_64+0xc1/0x440 [19336.438608] entry_SYSCALL_64_after_hwframe+0x49/0xae [19336.440716] RIP: 0033:0x7f53425a5332 [19336.442470] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19336.449731] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19336.452714] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19336.455828] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19336.458968] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19336.462125] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19336.465608] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19340.060106] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19340.063413] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19340.105403] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19340.114560] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19340.118473] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19340.123539] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19340.147423] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19340.153715] RAX: 0000000000006f8d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19340.171076] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19340.181390] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19340.188395] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19340.195312] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19340.201553] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19340.206759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19340.212286] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19340.219814] Call Trace: [19340.221367] [19340.223208] ? show_regs.cold.9+0x22/0x2f [19340.229506] ? watchdog_timer_fn.cold.6+0x62/0x10a [19340.234750] ? __hrtimer_run_queues+0x1c2/0x4d0 [19340.238422] ? watchdog+0x40/0x40 [19340.244002] ? hrtimer_interrupt+0x128/0x2c0 [19340.245906] ? smp_apic_timer_interrupt+0xa6/0x270 [19340.248263] ? apic_timer_interrupt+0xf/0x20 [19340.252448] [19340.259773] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19340.262031] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19340.270427] do_raw_spin_lock+0x177/0x190 [19340.278647] _raw_spin_lock+0x21/0x30 [19340.285444] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19340.291846] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19340.294898] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19340.301725] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19340.305004] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19340.310138] ? woken_wake_function+0x30/0x30 [19340.312307] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19340.314672] kthread+0x1d1/0x200 [19340.316218] ? set_kthread_struct+0x70/0x70 [19340.321437] ret_from_fork+0x1f/0x30 [19356.052115] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [19356.055477] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19356.078161] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19356.084633] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19356.087859] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19356.091099] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19356.093493] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19356.101885] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19356.121671] RAX: 0000000000002f27 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19356.141485] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19356.145060] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19356.147836] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19356.149911] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19356.152365] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19356.155056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19356.157043] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19356.159027] Call Trace: [19356.159790] [19356.160273] ? show_regs.cold.9+0x22/0x2f [19356.161588] ? watchdog_timer_fn.cold.6+0x62/0x10a [19356.162977] ? __hrtimer_run_queues+0x1c2/0x4d0 [19356.164365] ? watchdog+0x40/0x40 [19356.165340] ? hrtimer_interrupt+0x128/0x2c0 [19356.166993] ? smp_apic_timer_interrupt+0xa6/0x270 [19356.168421] ? apic_timer_interrupt+0xf/0x20 [19356.170150] [19356.170952] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19356.172991] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19356.174504] ? newidle_balance.isra.93+0x207/0x580 [19356.175988] do_raw_spin_lock+0x177/0x190 [19356.177197] _raw_spin_lock+0x21/0x30 [19356.178330] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19356.180267] process_one_work+0x2c8/0x700 [19356.181330] worker_thread+0x296/0x6e0 [19356.182470] ? rescuer_thread+0x570/0x570 [19356.183864] kthread+0x1d1/0x200 [19356.185143] ? set_kthread_struct+0x70/0x70 [19356.186725] ret_from_fork+0x1f/0x30 [19358.687046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16721s! [19358.690156] Showing busy workqueues and worker pools: [19358.691906] workqueue events: flags=0x0 [19358.693092] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19358.695165] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19358.698956] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19358.701025] pending: cpu_latency_work [ptlrpc] [19358.703111] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19358.708739] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19358.712564] workqueue events_long: flags=0x0 [19358.714226] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19358.716913] pending: lu_objects_flush [obdclass] [19358.718801] workqueue events_unbound: flags=0x2 [19358.720267] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19358.722488] pending: flush_memcg_stats_dwork, idle_cull_fn [19358.724125] workqueue events_power_efficient: flags=0x80 [19358.725898] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19358.728817] pending: check_lifetime, do_cache_clean [19358.730887] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19358.732992] pending: neigh_periodic_work, neigh_periodic_work [19358.735240] workqueue mm_percpu_wq: flags=0x8 [19358.737544] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19358.740830] pending: vmstat_update, lru_add_drain_per_cpu [19358.743065] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19358.745717] pending: vmstat_update, lru_add_drain_per_cpu [19358.747936] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19358.750374] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19358.752325] workqueue xprtiod: flags=0xa [19358.753818] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19358.755943] pending: xprt_autoclose, xs_error_handle [19358.757897] workqueue ipv6_addrconf: flags=0x40008 [19358.759351] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19358.761598] pending: addrconf_verify_work [19358.763138] workqueue knbd0-recv: flags=0x1a [19358.764459] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19358.766751] in-flight: 651:recv_work [19358.771587] workqueue ptlrpc_pinger: flags=0x2 [19358.774117] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19358.781244] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19358.789085] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16721s workers=4 idle: 15979 31824 8 [19358.802241] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19349s workers=2 idle: 654 [19364.057127] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [19364.060216] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19364.064425] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [19364.078609] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19364.078623] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19364.078626] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19364.078641] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19364.078646] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19364.078651] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19364.078654] RDX: 000000000000390f RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19364.078657] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19364.078659] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19364.078661] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19364.078664] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19364.078666] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19364.078668] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19364.078677] Call Trace: [19364.078683] [19364.078689] ? show_regs.cold.9+0x22/0x2f [19364.078698] ? watchdog_timer_fn.cold.6+0x62/0x10a [19364.078704] ? __hrtimer_run_queues+0x1c2/0x4d0 [19364.078708] ? watchdog+0x40/0x40 [19364.078712] ? hrtimer_interrupt+0x128/0x2c0 [19364.078715] ? smp_apic_timer_interrupt+0xa6/0x270 [19364.078724] ? apic_timer_interrupt+0xf/0x20 [19364.078729] [19364.078732] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19364.078736] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19364.078741] do_raw_spin_lock+0x177/0x190 [19364.078746] _raw_spin_lock+0x21/0x30 [19364.078753] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19364.079040] obd_mod_free+0x46/0x140 [ptlrpc] [19364.079567] mdc_commit_open+0x91/0xb0 [mdc] [19364.079610] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19364.079865] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19364.080067] after_reply+0xe2c/0x1780 [ptlrpc] [19364.084056] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19364.084174] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19364.084182] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19364.084185] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19364.084207] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19364.084216] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19364.084220] RAX: 0000000000004b83 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19364.084224] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19364.084226] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19364.084228] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19364.084230] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19364.084233] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19364.084235] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19364.084237] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19364.084251] Call Trace: [19364.084256] [19364.084262] ? show_regs.cold.9+0x22/0x2f [19364.084275] ? watchdog_timer_fn.cold.6+0x62/0x10a [19364.084285] ? __hrtimer_run_queues+0x1c2/0x4d0 [19364.084289] ? watchdog+0x40/0x40 [19364.084296] ? hrtimer_interrupt+0x128/0x2c0 [19364.084301] ? smp_apic_timer_interrupt+0xa6/0x270 [19364.084309] ? apic_timer_interrupt+0xf/0x20 [19364.084313] [19364.084316] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19364.084322] do_raw_spin_lock+0x177/0x190 [19364.084326] _raw_spin_lock+0x21/0x30 [19364.084331] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19364.084662] ? do_raw_spin_unlock+0x75/0x190 [19364.084669] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19364.084888] ? do_wait_intr+0xf0/0xf0 [19364.084896] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19364.085120] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19364.085708] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19364.085924] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19364.086126] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19364.086170] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19364.086241] mdc_intent_lock+0x353/0x7e0 [mdc] [19364.086281] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19364.086450] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19364.086718] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19364.086746] lmv_intent_open+0x338/0x1150 [lmv] [19364.086773] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19364.086866] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19364.086966] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19364.086989] ll_intent_lock+0xc5/0xaa0 [lustre] [19364.087084] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19364.087172] ll_lookup_it+0xe0e/0x2270 [lustre] [19364.087254] ? do_raw_spin_unlock+0x75/0x190 [19364.087263] ? _raw_spin_unlock+0x12/0x30 [19364.087269] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19364.087296] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19364.087314] ll_atomic_open+0x36a/0xd70 [lustre] [19364.087485] lookup_open+0x356/0xb40 [19364.087494] do_last+0x313/0xfc0 [19364.087499] ? nd_jump_root+0xe5/0x160 [19364.087504] ? path_init+0x437/0x520 [19364.087509] path_openat+0xf7/0x500 [19364.087514] ? do_raw_spin_unlock+0x75/0x190 [19364.087518] do_filp_open+0x99/0x140 [19364.087525] ? getname_flags+0x6e/0x330 [19364.087530] ? __check_object_size+0xff/0x256 [19364.087537] ? do_raw_spin_unlock+0x75/0x190 [19364.087541] ? _raw_spin_unlock+0x12/0x30 [19364.087546] do_sys_openat2+0x2b4/0x410 [19364.087551] do_sys_open+0x73/0xa0 [19364.087555] __x64_sys_openat+0x24/0x30 [19364.087559] do_syscall_64+0xc1/0x440 [19364.087566] entry_SYSCALL_64_after_hwframe+0x49/0xae [19364.087572] RIP: 0033:0x7f53425a5332 [19364.087579] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19364.087584] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19364.087589] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19364.087591] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19364.087593] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19364.087595] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19364.087597] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19364.678975] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19364.687120] ? __next_timer_interrupt+0x160/0x160 [19364.693113] ptlrpcd+0xa82/0xe40 [ptlrpc] [19364.699319] ? do_wait_intr+0xf0/0xf0 [19364.700271] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19364.704983] kthread+0x1d1/0x200 [19364.705891] ? set_kthread_struct+0x70/0x70 [19364.708255] ret_from_fork+0x1f/0x30 [19368.061275] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19368.063904] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19368.122965] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19368.133920] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19368.144678] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19368.146753] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19368.168519] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19368.170936] RAX: 0000000000007f8b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19368.174059] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19368.177229] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19368.181070] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19368.184468] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19368.191420] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19368.197759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19368.200622] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19368.203251] Call Trace: [19368.203865] [19368.204624] ? show_regs.cold.9+0x22/0x2f [19368.208407] ? watchdog_timer_fn.cold.6+0x62/0x10a [19368.210798] ? __hrtimer_run_queues+0x1c2/0x4d0 [19368.212322] ? watchdog+0x40/0x40 [19368.221502] ? hrtimer_interrupt+0x128/0x2c0 [19368.223638] ? smp_apic_timer_interrupt+0xa6/0x270 [19368.226918] ? apic_timer_interrupt+0xf/0x20 [19368.232373] [19368.233075] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19368.240441] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19368.242844] do_raw_spin_lock+0x177/0x190 [19368.244562] _raw_spin_lock+0x21/0x30 [19368.246721] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19368.249460] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19368.252472] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19368.254817] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19368.257165] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19368.259695] ? woken_wake_function+0x30/0x30 [19368.261297] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19368.263968] kthread+0x1d1/0x200 [19368.265834] ? set_kthread_struct+0x70/0x70 [19368.268584] ret_from_fork+0x1f/0x30 [19384.052590] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [19384.056979] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19384.088852] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19384.097948] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19384.106949] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19384.112167] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19384.113791] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19384.120302] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [19384.122758] RAX: 0000000000002409 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19384.124556] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19384.126384] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19384.127974] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19384.130724] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19384.134087] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19384.137707] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19384.140773] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19384.144741] Call Trace: [19384.146301] [19384.148580] ? show_regs.cold.9+0x22/0x2f [19384.150945] ? watchdog_timer_fn.cold.6+0x62/0x10a [19384.153218] ? __hrtimer_run_queues+0x1c2/0x4d0 [19384.158473] ? watchdog+0x40/0x40 [19384.160133] ? hrtimer_interrupt+0x128/0x2c0 [19384.164994] ? smp_apic_timer_interrupt+0xa6/0x270 [19384.167807] ? apic_timer_interrupt+0xf/0x20 [19384.170629] [19384.172304] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19384.175792] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19384.183230] ? newidle_balance.isra.93+0x207/0x580 [19384.184846] do_raw_spin_lock+0x177/0x190 [19384.187419] _raw_spin_lock+0x21/0x30 [19384.189436] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19384.192480] process_one_work+0x2c8/0x700 [19384.194874] worker_thread+0x296/0x6e0 [19384.199723] ? rescuer_thread+0x570/0x570 [19384.203127] kthread+0x1d1/0x200 [19384.206895] ? set_kthread_struct+0x70/0x70 [19384.211404] ret_from_fork+0x1f/0x30 [19388.897667] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16751s! [19388.902105] Showing busy workqueues and worker pools: [19388.906754] workqueue events: flags=0x0 [19388.909510] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19388.913409] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19388.919200] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19388.922398] pending: cpu_latency_work [ptlrpc] [19388.924593] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19388.927253] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19388.931174] workqueue events_long: flags=0x0 [19388.933037] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19388.935933] pending: lu_objects_flush [obdclass] [19388.938882] workqueue events_unbound: flags=0x2 [19388.940838] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19388.946889] pending: flush_memcg_stats_dwork, idle_cull_fn [19388.951010] workqueue events_power_efficient: flags=0x80 [19388.954563] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19388.959257] pending: check_lifetime, do_cache_clean [19388.961700] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19388.970805] pending: neigh_periodic_work, neigh_periodic_work [19388.976577] workqueue mm_percpu_wq: flags=0x8 [19388.980609] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19388.985033] pending: vmstat_update, lru_add_drain_per_cpu [19388.989639] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19388.998963] pending: vmstat_update, lru_add_drain_per_cpu [19389.000506] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19389.011458] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19389.016380] workqueue xprtiod: flags=0xa [19389.017787] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19389.021641] pending: xprt_autoclose, xs_error_handle [19389.027028] workqueue ipv6_addrconf: flags=0x40008 [19389.031218] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19389.039173] pending: addrconf_verify_work [19389.043927] workqueue knbd0-recv: flags=0x1a [19389.048314] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19389.054662] in-flight: 651:recv_work [19389.061204] workqueue ptlrpc_pinger: flags=0x2 [19389.064141] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19389.069145] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19389.077383] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16751s workers=4 idle: 15979 31824 8 [19389.085180] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19379s workers=2 idle: 654 [19392.055448] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [19392.057147] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [19392.060151] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19392.060226] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19392.060233] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19392.060235] RIP: 0010:__pv_queued_spin_lock_slowpath+0x320/0x6d0 [19392.060249] Code: 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 <48> 83 05 18 d4 dc 02 01 eb e0 48 83 05 06 d4 dc 02 01 48 83 05 be [19392.060253] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19392.060258] RAX: 0000000000005a37 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19392.060260] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19392.060263] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19392.060265] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19392.060268] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19392.060271] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19392.060635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19392.060637] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19392.060660] Call Trace: [19392.060666] [19392.060672] ? show_regs.cold.9+0x22/0x2f [19392.060681] ? watchdog_timer_fn.cold.6+0x62/0x10a [19392.060687] ? __hrtimer_run_queues+0x1c2/0x4d0 [19392.060691] ? watchdog+0x40/0x40 [19392.060696] ? hrtimer_interrupt+0x128/0x2c0 [19392.060700] ? smp_apic_timer_interrupt+0xa6/0x270 [19392.060708] ? apic_timer_interrupt+0xf/0x20 [19392.060712] [19392.060715] ? __pv_queued_spin_lock_slowpath+0x320/0x6d0 [19392.060721] do_raw_spin_lock+0x177/0x190 [19392.060726] _raw_spin_lock+0x21/0x30 [19392.060731] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19392.064968] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19392.065052] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19392.093707] ? do_raw_spin_unlock+0x75/0x190 [19392.093728] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19392.094222] ? do_wait_intr+0xf0/0xf0 [19392.094234] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19392.094797] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19392.095155] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19392.095392] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19392.095612] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19392.095650] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19392.095680] mdc_intent_lock+0x353/0x7e0 [mdc] [19392.095714] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19392.095976] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19392.097765] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19392.097800] lmv_intent_open+0x338/0x1150 [lmv] [19392.097833] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19392.098008] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19392.098165] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19392.098191] ll_intent_lock+0xc5/0xaa0 [lustre] [19392.098281] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19392.098364] ll_lookup_it+0xe0e/0x2270 [lustre] [19392.098523] ? do_raw_spin_unlock+0x75/0x190 [19392.098532] ? _raw_spin_unlock+0x12/0x30 [19392.098539] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19392.098571] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19392.098594] ll_atomic_open+0x36a/0xd70 [lustre] [19392.105373] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19392.109117] lookup_open+0x356/0xb40 [19392.109140] do_last+0x313/0xfc0 [19392.109145] ? nd_jump_root+0xe5/0x160 [19392.109150] ? path_init+0x437/0x520 [19392.109154] path_openat+0xf7/0x500 [19392.109159] ? do_raw_spin_unlock+0x75/0x190 [19392.109166] do_filp_open+0x99/0x140 [19392.109172] ? getname_flags+0x6e/0x330 [19392.109177] ? __check_object_size+0xff/0x256 [19392.109185] ? do_raw_spin_unlock+0x75/0x190 [19392.109188] ? _raw_spin_unlock+0x12/0x30 [19392.109195] do_sys_openat2+0x2b4/0x410 [19392.109199] do_sys_open+0x73/0xa0 [19392.109203] __x64_sys_openat+0x24/0x30 [19392.109207] do_syscall_64+0xc1/0x440 [19392.109213] entry_SYSCALL_64_after_hwframe+0x49/0xae [19392.109219] RIP: 0033:0x7f53425a5332 [19392.109228] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19392.109231] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19392.109237] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19392.109239] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19392.109241] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19392.109242] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19392.109244] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19392.425474] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19392.428301] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19392.438259] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19392.440418] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19392.444307] RDX: 000000000000163b RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19392.448298] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19392.452549] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19392.456241] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19392.459631] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19392.463317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19392.467769] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19392.475704] Call Trace: [19392.478662] [19392.480021] ? show_regs.cold.9+0x22/0x2f [19392.484150] ? watchdog_timer_fn.cold.6+0x62/0x10a [19392.486501] ? __hrtimer_run_queues+0x1c2/0x4d0 [19392.488198] ? watchdog+0x40/0x40 [19392.490054] ? hrtimer_interrupt+0x128/0x2c0 [19392.491558] ? smp_apic_timer_interrupt+0xa6/0x270 [19392.494018] ? apic_timer_interrupt+0xf/0x20 [19392.496039] [19392.497075] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19392.499444] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19392.502643] do_raw_spin_lock+0x177/0x190 [19392.504508] _raw_spin_lock+0x21/0x30 [19392.506434] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19392.509516] obd_mod_free+0x46/0x140 [ptlrpc] [19392.512274] mdc_commit_open+0x91/0xb0 [mdc] [19392.514735] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19392.517471] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19392.520767] after_reply+0xe2c/0x1780 [ptlrpc] [19392.522817] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19392.525152] ? __next_timer_interrupt+0x160/0x160 [19392.526672] ptlrpcd+0xa82/0xe40 [ptlrpc] [19392.528447] ? do_wait_intr+0xf0/0xf0 [19392.529970] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19392.532728] kthread+0x1d1/0x200 [19392.534566] ? set_kthread_struct+0x70/0x70 [19392.537482] ret_from_fork+0x1f/0x30 [19396.058757] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19396.061615] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19396.100665] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19396.111315] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19396.114246] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19396.121550] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19396.144709] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19396.150262] RAX: 0000000000004b87 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19396.154281] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19396.157505] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19396.160716] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19396.164047] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19396.167126] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19396.170779] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19396.173365] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19396.176447] Call Trace: [19396.177317] [19396.178021] ? show_regs.cold.9+0x22/0x2f [19396.180084] ? watchdog_timer_fn.cold.6+0x62/0x10a [19396.182084] ? __hrtimer_run_queues+0x1c2/0x4d0 [19396.188199] ? watchdog+0x40/0x40 [19396.189673] ? hrtimer_interrupt+0x128/0x2c0 [19396.194293] ? smp_apic_timer_interrupt+0xa6/0x270 [19396.197756] ? apic_timer_interrupt+0xf/0x20 [19396.205717] [19396.208044] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19396.213326] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19396.219774] do_raw_spin_lock+0x177/0x190 [19396.222987] _raw_spin_lock+0x21/0x30 [19396.226067] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19396.228357] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19396.247554] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19396.254467] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19396.268152] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19396.274992] ? woken_wake_function+0x30/0x30 [19396.281316] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19396.288966] kthread+0x1d1/0x200 [19396.292312] ? set_kthread_struct+0x70/0x70 [19396.299479] ret_from_fork+0x1f/0x30 [19416.052018] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [19416.054223] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19416.065848] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19416.069920] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19416.073257] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19416.075047] RIP: 0010:native_safe_halt+0x2e/0x50 [19416.076163] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19416.080779] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19416.082660] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19416.084244] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [19416.085882] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19416.087719] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19416.089411] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19416.091277] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19416.093319] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19416.094992] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19416.097035] Call Trace: [19416.097763] [19416.098283] ? show_regs.cold.9+0x22/0x2f [19416.099192] ? watchdog_timer_fn.cold.6+0x62/0x10a [19416.100240] ? __hrtimer_run_queues+0x1c2/0x4d0 [19416.101229] ? watchdog+0x40/0x40 [19416.102054] ? hrtimer_interrupt+0x128/0x2c0 [19416.103226] ? smp_apic_timer_interrupt+0xa6/0x270 [19416.104510] ? apic_timer_interrupt+0xf/0x20 [19416.105592] [19416.106154] ? native_safe_halt+0x2e/0x50 [19416.107173] kvm_wait+0xcd/0xe0 [19416.108098] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19416.109285] ? newidle_balance.isra.93+0x207/0x580 [19416.110482] do_raw_spin_lock+0x177/0x190 [19416.111402] _raw_spin_lock+0x21/0x30 [19416.112235] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19416.114247] process_one_work+0x2c8/0x700 [19416.115748] worker_thread+0x296/0x6e0 [19416.116643] ? rescuer_thread+0x570/0x570 [19416.117688] kthread+0x1d1/0x200 [19416.118452] ? set_kthread_struct+0x70/0x70 [19416.119404] ret_from_fork+0x1f/0x30 [19419.110136] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16782s! [19419.116689] Showing busy workqueues and worker pools: [19419.118702] workqueue events: flags=0x0 [19419.120357] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19419.123154] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19419.127901] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19419.130253] pending: cpu_latency_work [ptlrpc] [19419.132952] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19419.135256] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19419.139310] workqueue events_long: flags=0x0 [19419.140931] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19419.144089] pending: lu_objects_flush [obdclass] [19419.146537] workqueue events_unbound: flags=0x2 [19419.148439] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19419.151138] pending: flush_memcg_stats_dwork, idle_cull_fn [19419.154294] workqueue events_power_efficient: flags=0x80 [19419.156113] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19419.158518] pending: check_lifetime, do_cache_clean [19419.160658] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19419.164142] pending: neigh_periodic_work, neigh_periodic_work [19419.166203] workqueue mm_percpu_wq: flags=0x8 [19419.167769] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19419.170141] pending: vmstat_update, lru_add_drain_per_cpu [19419.173184] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19419.176229] pending: vmstat_update, lru_add_drain_per_cpu [19419.178680] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19419.181507] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19419.184221] workqueue xprtiod: flags=0xa [19419.185823] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19419.188385] pending: xprt_autoclose, xs_error_handle [19419.190685] workqueue ipv6_addrconf: flags=0x40008 [19419.192550] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19419.195727] pending: addrconf_verify_work [19419.197717] workqueue knbd0-recv: flags=0x1a [19419.199842] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19419.202568] in-flight: 651:recv_work [19419.204090] workqueue ptlrpc_pinger: flags=0x2 [19419.205777] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19419.209207] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19419.212921] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16782s workers=4 idle: 15979 31824 8 [19419.216675] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19409s workers=2 idle: 654 [19420.055026] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [19420.057046] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [19420.059221] Modules linked in: [19420.062744] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19420.062892] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19420.062900] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19420.062903] RIP: 0010:native_safe_halt+0x2e/0x50 [19420.062918] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19420.062923] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19420.062928] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19420.062931] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19420.062933] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19420.062935] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19420.062938] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19420.062942] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19420.062944] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19420.062947] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19420.062955] Call Trace: [19420.062961] [19420.062968] ? show_regs.cold.9+0x22/0x2f [19420.063048] ? watchdog_timer_fn.cold.6+0x62/0x10a [19420.063055] ? __hrtimer_run_queues+0x1c2/0x4d0 [19420.063059] ? watchdog+0x40/0x40 [19420.063063] ? hrtimer_interrupt+0x128/0x2c0 [19420.063067] ? smp_apic_timer_interrupt+0xa6/0x270 [19420.063074] ? apic_timer_interrupt+0xf/0x20 [19420.063079] [19420.063082] ? native_safe_halt+0x2e/0x50 [19420.063085] kvm_wait+0xcd/0xe0 [19420.063093] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19420.063100] do_raw_spin_lock+0x177/0x190 [19420.063105] _raw_spin_lock+0x21/0x30 [19420.063110] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19420.063526] obd_mod_free+0x46/0x140 [ptlrpc] [19420.063880] mdc_commit_open+0x91/0xb0 [mdc] [19420.063921] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19420.064264] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19420.064540] after_reply+0xe2c/0x1780 [ptlrpc] [19420.064882] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19420.065165] ? __next_timer_interrupt+0x160/0x160 [19420.065175] ptlrpcd+0xa82/0xe40 [ptlrpc] [19420.065529] ? do_wait_intr+0xf0/0xf0 [19420.065539] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19420.065892] kthread+0x1d1/0x200 [19420.065898] ? set_kthread_struct+0x70/0x70 [19420.065902] ret_from_fork+0x1f/0x30 [19420.176794] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19420.188650] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19420.191741] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19420.194957] RIP: 0010:native_safe_halt+0x2e/0x50 [19420.196762] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19420.203600] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19420.206496] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19420.208435] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19420.215324] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19420.217989] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19420.220872] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19420.223888] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19420.228124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19420.230749] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19420.233326] Call Trace: [19420.234528] [19420.235896] ? show_regs.cold.9+0x22/0x2f [19420.237372] ? watchdog_timer_fn.cold.6+0x62/0x10a [19420.238972] ? __hrtimer_run_queues+0x1c2/0x4d0 [19420.240092] ? watchdog+0x40/0x40 [19420.240863] ? hrtimer_interrupt+0x128/0x2c0 [19420.241845] ? smp_apic_timer_interrupt+0xa6/0x270 [19420.243277] ? apic_timer_interrupt+0xf/0x20 [19420.244328] [19420.244901] ? native_safe_halt+0x2e/0x50 [19420.245953] kvm_wait+0xcd/0xe0 [19420.246840] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19420.248262] do_raw_spin_lock+0x177/0x190 [19420.249356] _raw_spin_lock+0x21/0x30 [19420.250768] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19420.252594] ? do_raw_spin_unlock+0x75/0x190 [19420.253777] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19420.255268] ? do_wait_intr+0xf0/0xf0 [19420.256368] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19420.257995] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19420.259652] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19420.261361] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19420.262858] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19420.270165] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19420.271663] mdc_intent_lock+0x353/0x7e0 [mdc] [19420.272794] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19420.274764] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19420.289646] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19420.292306] lmv_intent_open+0x338/0x1150 [lmv] [19420.294717] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19420.297250] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19420.299595] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19420.301642] ll_intent_lock+0xc5/0xaa0 [lustre] [19420.304038] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19420.306557] ll_lookup_it+0xe0e/0x2270 [lustre] [19420.308600] ? do_raw_spin_unlock+0x75/0x190 [19420.310577] ? _raw_spin_unlock+0x12/0x30 [19420.312107] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19420.314860] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19420.316734] ll_atomic_open+0x36a/0xd70 [lustre] [19420.320124] lookup_open+0x356/0xb40 [19420.321922] do_last+0x313/0xfc0 [19420.324777] ? nd_jump_root+0xe5/0x160 [19420.325893] ? path_init+0x437/0x520 [19420.327503] path_openat+0xf7/0x500 [19420.329075] ? do_raw_spin_unlock+0x75/0x190 [19420.330672] do_filp_open+0x99/0x140 [19420.332237] ? getname_flags+0x6e/0x330 [19420.334457] ? __check_object_size+0xff/0x256 [19420.336124] ? do_raw_spin_unlock+0x75/0x190 [19420.338003] ? _raw_spin_unlock+0x12/0x30 [19420.339966] do_sys_openat2+0x2b4/0x410 [19420.341612] do_sys_open+0x73/0xa0 [19420.343071] __x64_sys_openat+0x24/0x30 [19420.344674] do_syscall_64+0xc1/0x440 [19420.346155] entry_SYSCALL_64_after_hwframe+0x49/0xae [19420.348353] RIP: 0033:0x7f53425a5332 [19420.350255] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19420.360863] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19420.364239] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19420.367597] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19420.370901] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19420.374846] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19420.377924] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19424.058019] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [19424.061681] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19424.082723] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19424.089756] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19424.093316] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19424.095650] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19424.104842] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19424.107766] RAX: 00000000000051bb RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19424.110910] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19424.114016] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19424.116934] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19424.121341] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19424.124558] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19424.128161] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19424.131886] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19424.135290] Call Trace: [19424.136377] [19424.136988] ? show_regs.cold.9+0x22/0x2f [19424.138817] ? watchdog_timer_fn.cold.6+0x62/0x10a [19424.140667] ? __hrtimer_run_queues+0x1c2/0x4d0 [19424.142275] ? watchdog+0x40/0x40 [19424.143485] ? hrtimer_interrupt+0x128/0x2c0 [19424.144835] ? smp_apic_timer_interrupt+0xa6/0x270 [19424.146686] ? apic_timer_interrupt+0xf/0x20 [19424.148339] [19424.148896] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19424.151807] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19424.153826] do_raw_spin_lock+0x177/0x190 [19424.155481] _raw_spin_lock+0x21/0x30 [19424.157015] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19424.159539] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19424.161936] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19424.164299] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19424.166892] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19424.169822] ? woken_wake_function+0x30/0x30 [19424.171848] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19424.174727] kthread+0x1d1/0x200 [19424.176250] ? set_kthread_struct+0x70/0x70 [19424.177870] ret_from_fork+0x1f/0x30 [19434.858623] rcu: INFO: rcu_sched self-detected stall on CPU [19434.861035] rcu: 2-....: (9770205 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [19434.866525] (t=17340432 jiffies g=274185 q=68382) [19434.868370] rcu: rcu_sched kthread starved for 16798798 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [19434.872625] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [19434.876519] rcu: RCU grace-period kthread stack dump: [19434.877812] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [19434.882894] Call Trace: [19434.884522] __schedule+0x351/0xcb0 [19434.886699] schedule+0xc0/0x180 [19434.888804] schedule_timeout+0xb4/0x190 [19434.894444] ? do_raw_spin_unlock+0x75/0x190 [19434.896132] ? __next_timer_interrupt+0x160/0x160 [19434.902214] ? prepare_to_swait_event+0xf3/0x140 [19434.903339] rcu_gp_kthread+0x8dd/0x15b0 [19434.904209] ? force_qs_rnp+0x2c0/0x2c0 [19434.910189] kthread+0x1d1/0x200 [19434.912671] ? set_kthread_struct+0x70/0x70 [19434.916368] ret_from_fork+0x1f/0x30 [19434.919463] rcu: Stack dump where RCU GP kthread last ran: [19434.922720] NMI backtrace for cpu 2 [19434.924957] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19434.932415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19434.938316] Call Trace: [19434.939471] [19434.940710] ? dump_stack+0xbb/0x10e [19434.942064] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [19434.945443] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [19434.947048] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [19434.950408] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [19434.954356] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [19434.958755] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [19434.961660] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [19434.964292] ? update_process_times+0x6d/0xc0 [19434.967242] ? tick_sched_handle.isra.18+0x27/0x90 [19434.969803] ? tick_sched_timer+0x53/0xb0 [19434.971269] ? __hrtimer_run_queues+0x1c2/0x4d0 [19434.972868] ? tick_sched_handle.isra.18+0x90/0x90 [19434.976071] ? hrtimer_interrupt+0x128/0x2c0 [19434.978404] ? smp_apic_timer_interrupt+0xa6/0x270 [19434.980793] ? apic_timer_interrupt+0xf/0x20 [19434.984036] [19434.984690] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19434.986753] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19434.990178] ? do_raw_spin_lock+0x177/0x190 [19434.991701] ? _raw_spin_lock+0x21/0x30 [19434.995004] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19434.997868] ? obd_mod_free+0x46/0x140 [ptlrpc] [19435.001550] ? mdc_commit_open+0x91/0xb0 [mdc] [19435.003116] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19435.009474] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19435.013237] ? after_reply+0xe2c/0x1780 [ptlrpc] [19435.015686] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19435.018005] ? __next_timer_interrupt+0x160/0x160 [19435.021048] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [19435.022741] ? do_wait_intr+0xf0/0xf0 [19435.034042] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19435.037980] ? kthread+0x1d1/0x200 [19435.039222] ? set_kthread_struct+0x70/0x70 [19435.040704] ? ret_from_fork+0x1f/0x30 [19435.044716] Sending NMI from CPU 2 to CPUs 1: [19435.055923] NMI backtrace for cpu 1 [19435.055932] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19435.055934] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19435.055936] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19435.055938] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19435.055940] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [19435.055943] RAX: 00000000000033f9 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19435.055944] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19435.055946] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19435.055947] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19435.055949] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19435.055950] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19435.055951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19435.055953] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19435.055954] Call Trace: [19435.055955] [19435.055955] ? show_regs.cold.9+0x22/0x2f [19435.055957] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [19435.055958] ? nmi_cpu_backtrace_handler+0x15/0x40 [19435.055959] ? nmi_handle+0xbd/0x210 [19435.055961] ? default_do_nmi+0x4b/0x1b0 [19435.055962] ? do_nmi+0x280/0x380 [19435.055963] ? end_repeat_nmi+0x16/0x1f [19435.055964] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19435.055965] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19435.055966] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19435.055967] [19435.055968] do_raw_spin_lock+0x177/0x190 [19435.055969] _raw_spin_lock+0x21/0x30 [19435.055970] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19435.055977] ? do_raw_spin_unlock+0x75/0x190 [19435.055978] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19435.055980] ? do_wait_intr+0xf0/0xf0 [19435.055981] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19435.055982] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19435.055983] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19435.055985] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19435.055986] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19435.055987] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19435.055989] mdc_intent_lock+0x353/0x7e0 [mdc] [19435.055990] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19435.055991] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19435.055992] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19435.055994] lmv_intent_open+0x338/0x1150 [lmv] [19435.055995] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19435.055996] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19435.055998] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19435.055999] ll_intent_lock+0xc5/0xaa0 [lustre] [19435.056000] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19435.056005] ll_lookup_it+0xe0e/0x2270 [lustre] [19435.056007] ? do_raw_spin_unlock+0x75/0x190 [19435.056008] ? _raw_spin_unlock+0x12/0x30 [19435.056009] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19435.056011] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19435.056012] ll_atomic_open+0x36a/0xd70 [lustre] [19435.056013] lookup_open+0x356/0xb40 [19435.056015] do_last+0x313/0xfc0 [19435.056016] ? nd_jump_root+0xe5/0x160 [19435.056018] ? path_init+0x437/0x520 [19435.056019] path_openat+0xf7/0x500 [19435.056020] ? do_raw_spin_unlock+0x75/0x190 [19435.056021] do_filp_open+0x99/0x140 [19435.056022] ? getname_flags+0x6e/0x330 [19435.056023] ? __check_object_size+0xff/0x256 [19435.056024] ? do_raw_spin_unlock+0x75/0x190 [19435.056025] ? _raw_spin_unlock+0x12/0x30 [19435.056026] do_sys_openat2+0x2b4/0x410 [19435.056027] do_sys_open+0x73/0xa0 [19435.056029] __x64_sys_openat+0x24/0x30 [19435.056029] do_syscall_64+0xc1/0x440 [19435.056030] entry_SYSCALL_64_after_hwframe+0x49/0xae [19435.056032] RIP: 0033:0x7f53425a5332 [19435.056033] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19435.056035] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19435.056038] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19435.056040] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19435.056041] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19435.056043] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19435.056044] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19435.056500] NMI backtrace for cpu 2 [19435.362801] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19435.367498] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19435.370758] Call Trace: [19435.373718] [19435.375523] ? dump_stack+0xbb/0x10e [19435.380018] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [19435.382290] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [19435.384286] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [19435.386394] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [19435.388434] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [19435.390367] ? rcu_dump_cpu_stacks+0x11b/0x171 [19435.392316] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [19435.394476] ? update_process_times+0x6d/0xc0 [19435.396408] ? tick_sched_handle.isra.18+0x27/0x90 [19435.397752] ? tick_sched_timer+0x53/0xb0 [19435.398808] ? __hrtimer_run_queues+0x1c2/0x4d0 [19435.400716] ? tick_sched_handle.isra.18+0x90/0x90 [19435.402805] ? hrtimer_interrupt+0x128/0x2c0 [19435.404723] ? smp_apic_timer_interrupt+0xa6/0x270 [19435.411030] ? apic_timer_interrupt+0xf/0x20 [19435.412513] [19435.413234] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19435.415993] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19435.422521] ? do_raw_spin_lock+0x177/0x190 [19435.424772] ? _raw_spin_lock+0x21/0x30 [19435.426500] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19435.431646] ? obd_mod_free+0x46/0x140 [ptlrpc] [19435.434862] ? mdc_commit_open+0x91/0xb0 [mdc] [19435.437704] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19435.441985] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19435.448958] ? after_reply+0xe2c/0x1780 [ptlrpc] [19435.452558] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19435.456978] ? __next_timer_interrupt+0x160/0x160 [19435.459083] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [19435.460778] ? do_wait_intr+0xf0/0xf0 [19435.463372] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19435.466062] ? kthread+0x1d1/0x200 [19435.467609] ? set_kthread_struct+0x70/0x70 [19435.469102] ? ret_from_fork+0x1f/0x30 [19444.052078] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [19444.056419] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19444.108911] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19444.115158] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19444.126126] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19444.130373] RIP: 0010:native_safe_halt+0x2e/0x50 [19444.132820] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19444.143543] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19444.148130] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19444.151444] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [19444.154957] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19444.158654] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19444.162304] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19444.165618] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19444.169459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19444.171817] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19444.175293] Call Trace: [19444.176419] [19444.177166] ? show_regs.cold.9+0x22/0x2f [19444.178922] ? watchdog_timer_fn.cold.6+0x62/0x10a [19444.181232] ? __hrtimer_run_queues+0x1c2/0x4d0 [19444.183451] ? watchdog+0x40/0x40 [19444.184733] ? hrtimer_interrupt+0x128/0x2c0 [19444.186358] ? smp_apic_timer_interrupt+0xa6/0x270 [19444.188185] ? apic_timer_interrupt+0xf/0x20 [19444.190444] [19444.191398] ? native_safe_halt+0x2e/0x50 [19444.193362] kvm_wait+0xcd/0xe0 [19444.194799] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19444.197451] ? newidle_balance.isra.93+0x207/0x580 [19444.199450] do_raw_spin_lock+0x177/0x190 [19444.201280] _raw_spin_lock+0x21/0x30 [19444.202725] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19444.205370] process_one_work+0x2c8/0x700 [19444.207252] worker_thread+0x296/0x6e0 [19444.209110] ? rescuer_thread+0x570/0x570 [19444.211159] kthread+0x1d1/0x200 [19444.212605] ? set_kthread_struct+0x70/0x70 [19444.214911] ret_from_fork+0x1f/0x30 [19449.313830] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16812s! [19449.319542] Showing busy workqueues and worker pools: [19449.323442] workqueue events: flags=0x0 [19449.324936] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19449.327712] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19449.332056] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19449.334793] pending: cpu_latency_work [ptlrpc] [19449.337129] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19449.339814] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19449.343707] workqueue events_long: flags=0x0 [19449.345333] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19449.348134] pending: lu_objects_flush [obdclass] [19449.350125] workqueue events_unbound: flags=0x2 [19449.356033] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19449.362349] pending: flush_memcg_stats_dwork, idle_cull_fn [19449.367221] workqueue events_power_efficient: flags=0x80 [19449.372330] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19449.378382] pending: check_lifetime, do_cache_clean [19449.384276] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19449.390259] pending: neigh_periodic_work, neigh_periodic_work [19449.396238] workqueue mm_percpu_wq: flags=0x8 [19449.400062] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19449.405175] pending: vmstat_update, lru_add_drain_per_cpu [19449.410301] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19449.415598] pending: vmstat_update, lru_add_drain_per_cpu [19449.419851] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19449.423254] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19449.428205] workqueue xprtiod: flags=0xa [19449.434271] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19449.436268] pending: xprt_autoclose, xs_error_handle [19449.438052] workqueue ipv6_addrconf: flags=0x40008 [19449.439555] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19449.441769] pending: addrconf_verify_work [19449.443404] workqueue knbd0-recv: flags=0x1a [19449.445978] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19449.448317] in-flight: 651:recv_work [19449.449665] workqueue ptlrpc_pinger: flags=0x2 [19449.451297] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19449.453214] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19449.455815] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16812s workers=4 idle: 15979 31824 8 [19449.458539] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19439s workers=2 idle: 654 [19452.058193] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [19452.061313] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19452.080416] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19452.084724] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19452.087832] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19452.090027] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19452.099740] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19452.103353] RAX: 0000000000006d05 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19452.106672] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19452.109683] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19452.117795] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19452.126544] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19452.130235] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19452.136181] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19452.138439] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19452.143232] Call Trace: [19452.144105] [19452.144922] ? show_regs.cold.9+0x22/0x2f [19452.146268] ? watchdog_timer_fn.cold.6+0x62/0x10a [19452.148116] ? __hrtimer_run_queues+0x1c2/0x4d0 [19452.150608] ? watchdog+0x40/0x40 [19452.151686] ? hrtimer_interrupt+0x128/0x2c0 [19452.154578] ? smp_apic_timer_interrupt+0xa6/0x270 [19452.156514] ? apic_timer_interrupt+0xf/0x20 [19452.158049] [19452.158611] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19452.160403] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19452.162162] do_raw_spin_lock+0x177/0x190 [19452.163565] _raw_spin_lock+0x21/0x30 [19452.164666] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19452.169255] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19452.171666] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19452.174510] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19452.177941] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19452.181215] ? woken_wake_function+0x30/0x30 [19452.182555] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19452.186380] kthread+0x1d1/0x200 [19452.187202] ? set_kthread_struct+0x70/0x70 [19452.192319] ret_from_fork+0x1f/0x30 [19460.055108] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [19460.057011] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [19460.058352] Modules linked in: lustre(O) [19460.062125] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19460.062206] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19460.062213] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19460.062215] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19460.062229] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19460.062233] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19460.062237] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19460.062240] RDX: 0000000000001fae RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19460.062242] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19460.062244] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19460.062246] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19460.062249] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19460.062252] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19460.062254] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19460.062261] Call Trace: [19460.062266] [19460.062271] ? show_regs.cold.9+0x22/0x2f [19460.062280] ? watchdog_timer_fn.cold.6+0x62/0x10a [19460.062292] ? __hrtimer_run_queues+0x1c2/0x4d0 [19460.062297] ? watchdog+0x40/0x40 [19460.062301] ? hrtimer_interrupt+0x128/0x2c0 [19460.062305] ? smp_apic_timer_interrupt+0xa6/0x270 [19460.062313] ? apic_timer_interrupt+0xf/0x20 [19460.062317] [19460.062321] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19460.062326] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19460.062331] do_raw_spin_lock+0x177/0x190 [19460.062335] _raw_spin_lock+0x21/0x30 [19460.065685] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19460.065810] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19460.065817] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19460.065820] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19460.065832] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19460.065837] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19460.065841] RAX: 0000000000007303 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19460.065844] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19460.065846] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19460.065848] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19460.065850] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19460.065853] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19460.065856] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19460.065859] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19460.065866] Call Trace: [19460.065870] [19460.066072] ? show_regs.cold.9+0x22/0x2f [19460.082316] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19460.086554] ? watchdog_timer_fn.cold.6+0x62/0x10a [19460.086572] ? __hrtimer_run_queues+0x1c2/0x4d0 [19460.086577] ? watchdog+0x40/0x40 [19460.086581] ? hrtimer_interrupt+0x128/0x2c0 [19460.086585] ? smp_apic_timer_interrupt+0xa6/0x270 [19460.086592] ? apic_timer_interrupt+0xf/0x20 [19460.086597] [19460.086601] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19460.086613] do_raw_spin_lock+0x177/0x190 [19460.086619] _raw_spin_lock+0x21/0x30 [19460.086624] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19460.090025] obd_mod_free+0x46/0x140 [ptlrpc] [19460.092449] ? do_raw_spin_unlock+0x75/0x190 [19460.103624] mdc_commit_open+0x91/0xb0 [mdc] [19460.107994] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19460.112572] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19460.113480] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19460.113727] after_reply+0xe2c/0x1780 [ptlrpc] [19460.114007] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19460.114212] ? __next_timer_interrupt+0x160/0x160 [19460.114222] ptlrpcd+0xa82/0xe40 [ptlrpc] [19460.114414] ? do_wait_intr+0xf0/0xf0 [19460.114421] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19460.114959] kthread+0x1d1/0x200 [19460.114969] ? set_kthread_struct+0x70/0x70 [19460.114973] ret_from_fork+0x1f/0x30 [19460.246510] ? do_wait_intr+0xf0/0xf0 [19460.247750] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19460.249598] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19460.251787] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19460.254404] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19460.257293] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19460.259532] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19460.261060] mdc_intent_lock+0x353/0x7e0 [mdc] [19460.262399] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19460.264962] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19460.267451] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19460.269134] lmv_intent_open+0x338/0x1150 [lmv] [19460.270605] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19460.272451] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19460.274458] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19460.275986] ll_intent_lock+0xc5/0xaa0 [lustre] [19460.278131] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19460.279863] ll_lookup_it+0xe0e/0x2270 [lustre] [19460.281575] ? do_raw_spin_unlock+0x75/0x190 [19460.285454] ? _raw_spin_unlock+0x12/0x30 [19460.286834] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19460.288402] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19460.290105] ll_atomic_open+0x36a/0xd70 [lustre] [19460.291465] lookup_open+0x356/0xb40 [19460.292578] do_last+0x313/0xfc0 [19460.293657] ? nd_jump_root+0xe5/0x160 [19460.294951] ? path_init+0x437/0x520 [19460.296187] path_openat+0xf7/0x500 [19460.297294] ? do_raw_spin_unlock+0x75/0x190 [19460.298621] do_filp_open+0x99/0x140 [19460.299826] ? getname_flags+0x6e/0x330 [19460.301024] ? __check_object_size+0xff/0x256 [19460.302436] ? do_raw_spin_unlock+0x75/0x190 [19460.303767] ? _raw_spin_unlock+0x12/0x30 [19460.305094] do_sys_openat2+0x2b4/0x410 [19460.306393] do_sys_open+0x73/0xa0 [19460.307542] __x64_sys_openat+0x24/0x30 [19460.308796] do_syscall_64+0xc1/0x440 [19460.309972] entry_SYSCALL_64_after_hwframe+0x49/0xae [19460.311480] RIP: 0033:0x7f53425a5332 [19460.312755] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19460.319199] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19460.322006] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19460.324648] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19460.327475] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19460.330029] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19460.332440] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19476.052819] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [19476.057505] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19476.077809] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19476.081351] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19476.086468] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19476.089260] RIP: 0010:native_safe_halt+0x2e/0x50 [19476.091958] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19476.107003] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19476.112631] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19476.117166] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [19476.120604] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19476.125369] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19476.129928] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19476.134902] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19476.138947] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19476.142439] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19476.145981] Call Trace: [19476.147233] [19476.148228] ? show_regs.cold.9+0x22/0x2f [19476.149613] ? watchdog_timer_fn.cold.6+0x62/0x10a [19476.151347] ? __hrtimer_run_queues+0x1c2/0x4d0 [19476.169492] ? watchdog+0x40/0x40 [19476.172923] ? hrtimer_interrupt+0x128/0x2c0 [19476.175321] ? smp_apic_timer_interrupt+0xa6/0x270 [19476.177738] ? apic_timer_interrupt+0xf/0x20 [19476.179332] [19476.181401] ? native_safe_halt+0x2e/0x50 [19476.189461] kvm_wait+0xcd/0xe0 [19476.195784] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19476.204136] ? newidle_balance.isra.93+0x207/0x580 [19476.211060] do_raw_spin_lock+0x177/0x190 [19476.214091] _raw_spin_lock+0x21/0x30 [19476.217019] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19476.228491] process_one_work+0x2c8/0x700 [19476.230150] worker_thread+0x296/0x6e0 [19476.232240] ? rescuer_thread+0x570/0x570 [19476.240987] kthread+0x1d1/0x200 [19476.247131] ? set_kthread_struct+0x70/0x70 [19476.256034] ret_from_fork+0x1f/0x30 [19479.520045] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16842s! [19479.523320] Showing busy workqueues and worker pools: [19479.525150] workqueue events: flags=0x0 [19479.526420] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19479.531493] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19479.535857] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19479.537836] pending: cpu_latency_work [ptlrpc] [19479.541965] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19479.544347] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19479.549227] workqueue events_long: flags=0x0 [19479.551321] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19479.553803] pending: lu_objects_flush [obdclass] [19479.555624] workqueue events_unbound: flags=0x2 [19479.557799] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19479.560297] pending: flush_memcg_stats_dwork, idle_cull_fn [19479.562726] workqueue events_power_efficient: flags=0x80 [19479.564711] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19479.567781] pending: check_lifetime, do_cache_clean [19479.569855] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19479.573016] pending: neigh_periodic_work, neigh_periodic_work [19479.575500] workqueue mm_percpu_wq: flags=0x8 [19479.576948] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19479.579528] pending: vmstat_update, lru_add_drain_per_cpu [19479.581183] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19479.583295] pending: vmstat_update, lru_add_drain_per_cpu [19479.585151] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19479.587743] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19479.589741] workqueue xprtiod: flags=0xa [19479.591351] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19479.593253] pending: xprt_autoclose, xs_error_handle [19479.594672] workqueue ipv6_addrconf: flags=0x40008 [19479.595963] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19479.597760] pending: addrconf_verify_work [19479.598949] workqueue knbd0-recv: flags=0x1a [19479.600289] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19479.602705] in-flight: 651:recv_work [19479.604042] workqueue ptlrpc_pinger: flags=0x2 [19479.605682] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19479.607817] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19479.609853] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16842s workers=4 idle: 15979 31824 8 [19479.612494] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19470s workers=2 idle: 654 [19480.058766] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [19480.068322] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19480.104837] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19480.108880] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19480.116206] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19480.120458] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19480.128933] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19480.131794] RAX: 0000000000007f90 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19480.134488] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19480.136946] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19480.139837] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19480.142148] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19480.144200] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19480.147407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19480.149514] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19480.151928] Call Trace: [19480.153234] [19480.153917] ? show_regs.cold.9+0x22/0x2f [19480.155280] ? watchdog_timer_fn.cold.6+0x62/0x10a [19480.156826] ? __hrtimer_run_queues+0x1c2/0x4d0 [19480.158167] ? watchdog+0x40/0x40 [19480.159752] ? hrtimer_interrupt+0x128/0x2c0 [19480.161875] ? smp_apic_timer_interrupt+0xa6/0x270 [19480.165097] ? apic_timer_interrupt+0xf/0x20 [19480.167233] [19480.167953] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19480.169985] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19480.173120] do_raw_spin_lock+0x177/0x190 [19480.175454] _raw_spin_lock+0x21/0x30 [19480.177329] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19480.181212] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19480.184794] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19480.187525] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19480.192081] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19480.193911] ? woken_wake_function+0x30/0x30 [19480.196872] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19480.199095] kthread+0x1d1/0x200 [19480.200316] ? set_kthread_struct+0x70/0x70 [19480.201992] ret_from_fork+0x1f/0x30 [19488.055150] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [19488.057026] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [19488.057838] Modules linked in: [19488.060710] Modules linked in: [19488.060737] lustre(O) [19488.061897] lustre(O) [19488.063011] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19488.063183] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19488.063189] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19488.063192] RIP: 0010:native_safe_halt+0x2e/0x50 [19488.063218] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19488.063224] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19488.063228] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19488.063230] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19488.063232] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19488.063234] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19488.063236] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19488.063238] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19488.063241] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19488.063243] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19488.063251] Call Trace: [19488.063257] [19488.063263] ? show_regs.cold.9+0x22/0x2f [19488.063272] ? watchdog_timer_fn.cold.6+0x62/0x10a [19488.063278] ? __hrtimer_run_queues+0x1c2/0x4d0 [19488.063283] ? watchdog+0x40/0x40 [19488.063287] ? hrtimer_interrupt+0x128/0x2c0 [19488.063291] ? smp_apic_timer_interrupt+0xa6/0x270 [19488.063296] ? apic_timer_interrupt+0xf/0x20 [19488.063301] [19488.063303] ? native_safe_halt+0x2e/0x50 [19488.063307] kvm_wait+0xcd/0xe0 [19488.063316] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19488.063325] do_raw_spin_lock+0x177/0x190 [19488.063330] _raw_spin_lock+0x21/0x30 [19488.063334] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19488.063638] ? do_raw_spin_unlock+0x75/0x190 [19488.063645] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19488.063843] ? do_wait_intr+0xf0/0xf0 [19488.063852] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19488.064045] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19488.064309] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19488.064534] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19488.064650] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19488.064689] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19488.064719] mdc_intent_lock+0x353/0x7e0 [mdc] [19488.064746] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19488.064827] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19488.065033] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19488.065060] lmv_intent_open+0x338/0x1150 [lmv] [19488.065093] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19488.065177] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19488.065352] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19488.065375] ll_intent_lock+0xc5/0xaa0 [lustre] [19488.065457] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19488.065536] ll_lookup_it+0xe0e/0x2270 [lustre] [19488.065615] ? do_raw_spin_unlock+0x75/0x190 [19488.065622] ? _raw_spin_unlock+0x12/0x30 [19488.065629] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19488.065656] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19488.065675] ll_atomic_open+0x36a/0xd70 [lustre] [19488.065757] lookup_open+0x356/0xb40 [19488.065766] do_last+0x313/0xfc0 [19488.065770] ? nd_jump_root+0xe5/0x160 [19488.065774] ? path_init+0x437/0x520 [19488.065779] path_openat+0xf7/0x500 [19488.065784] ? do_raw_spin_unlock+0x75/0x190 [19488.065789] do_filp_open+0x99/0x140 [19488.065795] ? getname_flags+0x6e/0x330 [19488.065800] ? __check_object_size+0xff/0x256 [19488.065807] ? do_raw_spin_unlock+0x75/0x190 [19488.065811] ? _raw_spin_unlock+0x12/0x30 [19488.065816] do_sys_openat2+0x2b4/0x410 [19488.065820] do_sys_open+0x73/0xa0 [19488.065824] __x64_sys_openat+0x24/0x30 [19488.065828] do_syscall_64+0xc1/0x440 [19488.065834] entry_SYSCALL_64_after_hwframe+0x49/0xae [19488.065840] RIP: 0033:0x7f53425a5332 [19488.065847] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19488.065851] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19488.065855] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19488.065858] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19488.065860] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19488.065862] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19488.065864] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19488.354509] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19488.371419] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19488.375747] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19488.379342] RIP: 0010:native_safe_halt+0x2e/0x50 [19488.381139] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19488.388426] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19488.391219] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19488.393796] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19488.396382] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19488.398960] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19488.401900] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19488.404540] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19488.407450] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19488.409693] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19488.412412] Call Trace: [19488.413465] [19488.414349] ? show_regs.cold.9+0x22/0x2f [19488.415789] ? watchdog_timer_fn.cold.6+0x62/0x10a [19488.417688] ? __hrtimer_run_queues+0x1c2/0x4d0 [19488.419306] ? watchdog+0x40/0x40 [19488.420551] ? hrtimer_interrupt+0x128/0x2c0 [19488.422612] ? smp_apic_timer_interrupt+0xa6/0x270 [19488.424822] ? apic_timer_interrupt+0xf/0x20 [19488.426467] [19488.427335] ? native_safe_halt+0x2e/0x50 [19488.428992] kvm_wait+0xcd/0xe0 [19488.430426] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19488.432864] do_raw_spin_lock+0x177/0x190 [19488.434733] _raw_spin_lock+0x21/0x30 [19488.436486] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19488.438788] obd_mod_free+0x46/0x140 [ptlrpc] [19488.440979] mdc_commit_open+0x91/0xb0 [mdc] [19488.442955] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19488.445557] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19488.448269] after_reply+0xe2c/0x1780 [ptlrpc] [19488.450478] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19488.453272] ? __next_timer_interrupt+0x160/0x160 [19488.455342] ptlrpcd+0xa82/0xe40 [ptlrpc] [19488.457495] ? do_wait_intr+0xf0/0xf0 [19488.459190] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19488.461868] kthread+0x1d1/0x200 [19488.463423] ? set_kthread_struct+0x70/0x70 [19488.465329] ret_from_fork+0x1f/0x30 [19504.052228] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [19504.054811] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19504.121231] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19504.130267] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19504.145570] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19504.149316] RIP: 0010:native_safe_halt+0x2e/0x50 [19504.155474] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19504.181530] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19504.187829] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19504.204790] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [19504.214769] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19504.222534] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19504.226616] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19504.236198] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19504.242453] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19504.253983] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19504.266131] Call Trace: [19504.269121] [19504.270912] ? show_regs.cold.9+0x22/0x2f [19504.275899] ? watchdog_timer_fn.cold.6+0x62/0x10a [19504.279797] ? __hrtimer_run_queues+0x1c2/0x4d0 [19504.282635] ? watchdog+0x40/0x40 [19504.284424] ? hrtimer_interrupt+0x128/0x2c0 [19504.285816] ? smp_apic_timer_interrupt+0xa6/0x270 [19504.298292] ? apic_timer_interrupt+0xf/0x20 [19504.299649] [19504.300396] ? native_safe_halt+0x2e/0x50 [19504.301800] kvm_wait+0xcd/0xe0 [19504.309679] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19504.313002] ? newidle_balance.isra.93+0x207/0x580 [19504.319608] do_raw_spin_lock+0x177/0x190 [19504.321959] _raw_spin_lock+0x21/0x30 [19504.323661] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19504.327290] process_one_work+0x2c8/0x700 [19504.328888] worker_thread+0x296/0x6e0 [19504.331887] ? rescuer_thread+0x570/0x570 [19504.336754] kthread+0x1d1/0x200 [19504.342100] ? set_kthread_struct+0x70/0x70 [19504.347744] ret_from_fork+0x1f/0x30 [19508.058022] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [19508.059989] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19508.072532] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19508.076317] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19508.078619] RIP: 0010:native_safe_halt+0x2e/0x50 [19508.080978] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19508.087697] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19508.091257] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19508.094041] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19508.096714] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19508.099576] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19508.102350] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19508.104712] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19508.107790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19508.110414] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19508.113121] Call Trace: [19508.113918] [19508.114708] ? show_regs.cold.9+0x22/0x2f [19508.116313] ? watchdog_timer_fn.cold.6+0x62/0x10a [19508.118395] ? __hrtimer_run_queues+0x1c2/0x4d0 [19508.119712] ? watchdog+0x40/0x40 [19508.121066] ? hrtimer_interrupt+0x128/0x2c0 [19508.122550] ? smp_apic_timer_interrupt+0xa6/0x270 [19508.124416] ? apic_timer_interrupt+0xf/0x20 [19508.126003] [19508.126711] ? native_safe_halt+0x2e/0x50 [19508.128423] kvm_wait+0xcd/0xe0 [19508.129627] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19508.131548] do_raw_spin_lock+0x177/0x190 [19508.133484] _raw_spin_lock+0x21/0x30 [19508.135067] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19508.137682] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19508.140007] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19508.142673] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19508.145073] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19508.147359] ? woken_wake_function+0x30/0x30 [19508.149146] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19508.151981] kthread+0x1d1/0x200 [19508.153329] ? set_kthread_struct+0x70/0x70 [19508.155036] ret_from_fork+0x1f/0x30 [19509.728068] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16872s! [19509.731108] Showing busy workqueues and worker pools: [19509.733721] workqueue events: flags=0x0 [19509.735129] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19509.743656] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19509.754754] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19509.757612] pending: cpu_latency_work [ptlrpc] [19509.759765] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19509.763451] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19509.768388] workqueue events_long: flags=0x0 [19509.771039] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19509.775707] pending: lu_objects_flush [obdclass] [19509.778455] workqueue events_unbound: flags=0x2 [19509.780982] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19509.784267] pending: flush_memcg_stats_dwork, idle_cull_fn [19509.790385] workqueue events_power_efficient: flags=0x80 [19509.793515] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19509.796567] pending: check_lifetime, do_cache_clean [19509.801770] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19509.806423] pending: neigh_periodic_work, neigh_periodic_work [19509.813351] workqueue mm_percpu_wq: flags=0x8 [19509.816347] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19509.819909] pending: vmstat_update, lru_add_drain_per_cpu [19509.824266] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19509.834394] pending: vmstat_update, lru_add_drain_per_cpu [19509.837326] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19509.840677] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19509.843538] workqueue xprtiod: flags=0xa [19509.846564] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19509.850005] pending: xprt_autoclose, xs_error_handle [19509.855492] workqueue ipv6_addrconf: flags=0x40008 [19509.857804] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19509.862154] pending: addrconf_verify_work [19509.866726] workqueue knbd0-recv: flags=0x1a [19509.869211] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19509.872563] in-flight: 651:recv_work [19509.874259] workqueue ptlrpc_pinger: flags=0x2 [19509.876030] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19509.878820] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19509.884142] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16872s workers=4 idle: 15979 31824 8 [19509.890803] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19500s workers=2 idle: 654 [19516.059440] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [19516.061629] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [19516.069483] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19516.069662] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19516.069669] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19516.069672] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19516.069686] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19516.069691] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19516.069696] RAX: 0000000000007809 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19516.069699] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19516.069701] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19516.069703] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19516.069706] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19516.069709] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19516.069712] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19516.069715] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19516.069732] Call Trace: [19516.069737] [19516.069743] ? show_regs.cold.9+0x22/0x2f [19516.069753] ? watchdog_timer_fn.cold.6+0x62/0x10a [19516.069760] ? __hrtimer_run_queues+0x1c2/0x4d0 [19516.069766] ? watchdog+0x40/0x40 [19516.069770] ? hrtimer_interrupt+0x128/0x2c0 [19516.069774] ? smp_apic_timer_interrupt+0xa6/0x270 [19516.069785] ? apic_timer_interrupt+0xf/0x20 [19516.069790] [19516.069793] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19516.069801] do_raw_spin_lock+0x177/0x190 [19516.069806] _raw_spin_lock+0x21/0x30 [19516.069811] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19516.070094] ? do_raw_spin_unlock+0x75/0x190 [19516.070100] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19516.070311] ? do_wait_intr+0xf0/0xf0 [19516.071843] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19516.077743] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19516.077834] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19516.077841] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19516.077844] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19516.077857] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19516.077861] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19516.077865] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19516.077868] RDX: 0000000000002527 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19516.077870] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19516.077872] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19516.077874] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19516.077877] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19516.077879] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19516.077881] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19516.077889] Call Trace: [19516.077893] [19516.077898] ? show_regs.cold.9+0x22/0x2f [19516.077905] ? watchdog_timer_fn.cold.6+0x62/0x10a [19516.077910] ? __hrtimer_run_queues+0x1c2/0x4d0 [19516.077913] ? watchdog+0x40/0x40 [19516.077917] ? hrtimer_interrupt+0x128/0x2c0 [19516.077920] ? smp_apic_timer_interrupt+0xa6/0x270 [19516.077928] ? apic_timer_interrupt+0xf/0x20 [19516.077940] [19516.077943] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19516.077947] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19516.077951] do_raw_spin_lock+0x177/0x190 [19516.077955] _raw_spin_lock+0x21/0x30 [19516.077960] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19516.078369] obd_mod_free+0x46/0x140 [ptlrpc] [19516.078574] mdc_commit_open+0x91/0xb0 [mdc] [19516.078613] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19516.117483] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19516.127164] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19516.127452] after_reply+0xe2c/0x1780 [ptlrpc] [19516.127690] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19516.127936] ? __next_timer_interrupt+0x160/0x160 [19516.127947] ptlrpcd+0xa82/0xe40 [ptlrpc] [19516.128170] ? do_wait_intr+0xf0/0xf0 [19516.128179] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19516.139307] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19516.139597] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19516.139713] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19516.139769] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19516.139801] mdc_intent_lock+0x353/0x7e0 [mdc] [19516.139831] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19516.139918] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19516.140141] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19516.140172] lmv_intent_open+0x338/0x1150 [lmv] [19516.140207] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19516.140293] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19516.140374] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19516.140397] ll_intent_lock+0xc5/0xaa0 [lustre] [19516.140479] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19516.147697] kthread+0x1d1/0x200 [19516.147727] ? set_kthread_struct+0x70/0x70 [19516.147733] ret_from_fork+0x1f/0x30 [19516.623155] ll_lookup_it+0xe0e/0x2270 [lustre] [19516.626285] ? do_raw_spin_unlock+0x75/0x190 [19516.628586] ? _raw_spin_unlock+0x12/0x30 [19516.630755] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19516.633420] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19516.636269] ll_atomic_open+0x36a/0xd70 [lustre] [19516.638950] lookup_open+0x356/0xb40 [19516.640607] do_last+0x313/0xfc0 [19516.642305] ? nd_jump_root+0xe5/0x160 [19516.644163] ? path_init+0x437/0x520 [19516.648041] path_openat+0xf7/0x500 [19516.649849] ? do_raw_spin_unlock+0x75/0x190 [19516.654622] do_filp_open+0x99/0x140 [19516.657724] ? getname_flags+0x6e/0x330 [19516.660158] ? __check_object_size+0xff/0x256 [19516.662628] ? do_raw_spin_unlock+0x75/0x190 [19516.665146] ? _raw_spin_unlock+0x12/0x30 [19516.667267] do_sys_openat2+0x2b4/0x410 [19516.669252] do_sys_open+0x73/0xa0 [19516.671092] __x64_sys_openat+0x24/0x30 [19516.673238] do_syscall_64+0xc1/0x440 [19516.675081] entry_SYSCALL_64_after_hwframe+0x49/0xae [19516.678029] RIP: 0033:0x7f53425a5332 [19516.679332] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19516.694120] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19516.698438] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19516.702248] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19516.705836] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19516.713429] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19516.717206] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19536.052736] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [19536.058521] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19536.060192] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19536.096643] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19536.096657] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19536.096664] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19536.102296] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19536.102384] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19536.102391] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19536.102394] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19536.102407] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19536.102411] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19536.102416] RAX: 0000000000000032 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19536.102419] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19536.102421] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19536.102423] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19536.102426] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19536.102429] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19536.102432] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19536.102434] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19536.102441] Call Trace: [19536.102446] [19536.102451] ? show_regs.cold.9+0x22/0x2f [19536.102459] ? watchdog_timer_fn.cold.6+0x62/0x10a [19536.102465] ? __hrtimer_run_queues+0x1c2/0x4d0 [19536.102468] ? watchdog+0x40/0x40 [19536.102473] ? hrtimer_interrupt+0x128/0x2c0 [19536.102477] ? smp_apic_timer_interrupt+0xa6/0x270 [19536.102485] ? apic_timer_interrupt+0xf/0x20 [19536.102489] [19536.102492] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19536.102497] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19536.102503] do_raw_spin_lock+0x177/0x190 [19536.102507] _raw_spin_lock+0x21/0x30 [19536.102511] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19536.111219] [19536.111227] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19536.114905] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19536.118398] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19536.118410] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 [19536.146661] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19536.151308] ORIG_RAX: ffffffffffffff13 [19536.154461] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19536.154897] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19536.155108] ? woken_wake_function+0x30/0x30 [19536.155118] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19536.157302] RAX: 0000000000000306 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19536.163932] kthread+0x1d1/0x200 [19536.163952] ? set_kthread_struct+0x70/0x70 [19536.163957] ret_from_fork+0x1f/0x30 [19536.331614] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19536.335281] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19536.342889] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19536.351334] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19536.358348] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19536.361743] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19536.367074] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19536.373769] Call Trace: [19536.376437] [19536.378238] ? show_regs.cold.9+0x22/0x2f [19536.383327] ? watchdog_timer_fn.cold.6+0x62/0x10a [19536.387921] ? __hrtimer_run_queues+0x1c2/0x4d0 [19536.390480] ? watchdog+0x40/0x40 [19536.393812] ? hrtimer_interrupt+0x128/0x2c0 [19536.395290] ? smp_apic_timer_interrupt+0xa6/0x270 [19536.398981] ? apic_timer_interrupt+0xf/0x20 [19536.401293] [19536.402659] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19536.406729] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19536.411116] ? newidle_balance.isra.93+0x207/0x580 [19536.414007] do_raw_spin_lock+0x177/0x190 [19536.416079] _raw_spin_lock+0x21/0x30 [19536.417850] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19536.422208] process_one_work+0x2c8/0x700 [19536.424696] worker_thread+0x296/0x6e0 [19536.425893] ? rescuer_thread+0x570/0x570 [19536.427892] kthread+0x1d1/0x200 [19536.428928] ? set_kthread_struct+0x70/0x70 [19536.431184] ret_from_fork+0x1f/0x30 [19539.943524] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16902s! [19539.947339] Showing busy workqueues and worker pools: [19539.954583] workqueue events: flags=0x0 [19539.955957] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19539.971690] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19539.982341] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19539.992508] pending: cpu_latency_work [ptlrpc] [19539.998293] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19540.000291] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19540.009787] workqueue events_long: flags=0x0 [19540.013321] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19540.015630] pending: lu_objects_flush [obdclass] [19540.026233] workqueue events_unbound: flags=0x2 [19540.033395] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19540.037909] pending: flush_memcg_stats_dwork, idle_cull_fn [19540.040888] workqueue events_power_efficient: flags=0x80 [19540.043751] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19540.046964] pending: check_lifetime, do_cache_clean [19540.050177] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19540.053641] pending: neigh_periodic_work, neigh_periodic_work [19540.056995] workqueue mm_percpu_wq: flags=0x8 [19540.064745] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19540.068261] pending: vmstat_update, lru_add_drain_per_cpu [19540.071006] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19540.078951] pending: vmstat_update, lru_add_drain_per_cpu [19540.087585] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19540.092432] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19540.096492] workqueue xprtiod: flags=0xa [19540.098341] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19540.100461] pending: xprt_autoclose, xs_error_handle [19540.115782] workqueue ipv6_addrconf: flags=0x40008 [19540.120636] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19540.126252] pending: addrconf_verify_work [19540.128292] workqueue knbd0-recv: flags=0x1a [19540.130984] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19540.135132] in-flight: 651:recv_work [19540.141275] workqueue ptlrpc_pinger: flags=0x2 [19540.142996] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19540.149593] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19540.164172] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16903s workers=4 idle: 15979 31824 8 [19540.166702] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19530s workers=2 idle: 654 [19544.056281] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [19544.059493] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [19544.061846] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19544.061917] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19544.061923] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19544.061926] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19544.061939] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19544.061943] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19544.061947] RAX: 00000000000017ea RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19544.061949] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19544.061951] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19544.061953] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19544.061956] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19544.061958] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19544.061961] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19544.061963] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19544.061970] Call Trace: [19544.061975] [19544.061980] ? show_regs.cold.9+0x22/0x2f [19544.061988] ? watchdog_timer_fn.cold.6+0x62/0x10a [19544.061995] ? __hrtimer_run_queues+0x1c2/0x4d0 [19544.061999] ? watchdog+0x40/0x40 [19544.062003] ? hrtimer_interrupt+0x128/0x2c0 [19544.062913] ? smp_apic_timer_interrupt+0xa6/0x270 [19544.062927] ? apic_timer_interrupt+0xf/0x20 [19544.062932] [19544.062936] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19544.062944] do_raw_spin_lock+0x177/0x190 [19544.062950] _raw_spin_lock+0x21/0x30 [19544.062955] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19544.063523] ? do_raw_spin_unlock+0x75/0x190 [19544.063530] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19544.063847] ? do_wait_intr+0xf0/0xf0 [19544.063857] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19544.064478] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19544.064702] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19544.064965] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19544.065162] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19544.065205] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19544.065248] mdc_intent_lock+0x353/0x7e0 [mdc] [19544.065277] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19544.065423] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19544.065641] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19544.065670] lmv_intent_open+0x338/0x1150 [lmv] [19544.065704] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19544.065789] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19544.067766] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19544.067799] ll_intent_lock+0xc5/0xaa0 [lustre] [19544.067892] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19544.067969] ll_lookup_it+0xe0e/0x2270 [lustre] [19544.068504] ? do_raw_spin_unlock+0x75/0x190 [19544.068513] ? _raw_spin_unlock+0x12/0x30 [19544.068521] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19544.068551] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19544.068570] ll_atomic_open+0x36a/0xd70 [lustre] [19544.068650] lookup_open+0x356/0xb40 [19544.068667] do_last+0x313/0xfc0 [19544.068672] ? nd_jump_root+0xe5/0x160 [19544.068676] ? path_init+0x437/0x520 [19544.068681] path_openat+0xf7/0x500 [19544.068686] ? do_raw_spin_unlock+0x75/0x190 [19544.068691] do_filp_open+0x99/0x140 [19544.068698] ? getname_flags+0x6e/0x330 [19544.068703] ? __check_object_size+0xff/0x256 [19544.068711] ? do_raw_spin_unlock+0x75/0x190 [19544.068716] ? _raw_spin_unlock+0x12/0x30 [19544.068720] do_sys_openat2+0x2b4/0x410 [19544.068725] do_sys_open+0x73/0xa0 [19544.068730] __x64_sys_openat+0x24/0x30 [19544.068734] do_syscall_64+0xc1/0x440 [19544.068742] entry_SYSCALL_64_after_hwframe+0x49/0xae [19544.068748] RIP: 0033:0x7f53425a5332 [19544.068756] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19544.068761] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19544.068766] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19544.068769] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19544.068771] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19544.068773] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19544.068776] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19544.422998] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19544.453894] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19544.462820] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19544.471813] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19544.481867] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19544.494414] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19544.503025] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19544.513667] RDX: 0000000000007d5b RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19544.524604] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19544.528678] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19544.532888] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19544.537847] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19544.544583] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19544.550991] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19544.560146] Call Trace: [19544.566743] [19544.568775] ? show_regs.cold.9+0x22/0x2f [19544.573058] ? watchdog_timer_fn.cold.6+0x62/0x10a [19544.577751] ? __hrtimer_run_queues+0x1c2/0x4d0 [19544.585650] ? watchdog+0x40/0x40 [19544.586941] ? hrtimer_interrupt+0x128/0x2c0 [19544.592606] ? smp_apic_timer_interrupt+0xa6/0x270 [19544.594632] ? apic_timer_interrupt+0xf/0x20 [19544.597150] [19544.598507] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19544.601785] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19544.610826] do_raw_spin_lock+0x177/0x190 [19544.623080] _raw_spin_lock+0x21/0x30 [19544.624342] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19544.626148] obd_mod_free+0x46/0x140 [ptlrpc] [19544.635654] mdc_commit_open+0x91/0xb0 [mdc] [19544.641731] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19544.645452] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19544.648759] after_reply+0xe2c/0x1780 [ptlrpc] [19544.651879] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19544.654593] ? __next_timer_interrupt+0x160/0x160 [19544.657288] ptlrpcd+0xa82/0xe40 [ptlrpc] [19544.659091] ? do_wait_intr+0xf0/0xf0 [19544.660895] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19544.663192] kthread+0x1d1/0x200 [19544.664611] ? set_kthread_struct+0x70/0x70 [19544.667206] ret_from_fork+0x1f/0x30 [19564.058199] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19564.061670] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19564.089066] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19564.093682] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19564.097699] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19564.100099] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19564.113828] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19564.117288] RAX: 0000000000007012 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19564.122903] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19564.126052] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19564.130024] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19564.132940] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19564.135999] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19564.140252] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19564.144217] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19564.148191] Call Trace: [19564.150634] [19564.151497] ? show_regs.cold.9+0x22/0x2f [19564.153439] ? watchdog_timer_fn.cold.6+0x62/0x10a [19564.158115] ? __hrtimer_run_queues+0x1c2/0x4d0 [19564.165051] ? watchdog+0x40/0x40 [19564.166130] ? hrtimer_interrupt+0x128/0x2c0 [19564.167254] ? smp_apic_timer_interrupt+0xa6/0x270 [19564.171050] ? apic_timer_interrupt+0xf/0x20 [19564.173037] [19564.176043] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19564.180687] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19564.187752] do_raw_spin_lock+0x177/0x190 [19564.190869] _raw_spin_lock+0x21/0x30 [19564.194862] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19564.202044] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19564.212388] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19564.219287] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19564.222190] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19564.226952] ? woken_wake_function+0x30/0x30 [19564.229817] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19564.236347] kthread+0x1d1/0x200 [19564.237497] ? set_kthread_struct+0x70/0x70 [19564.240247] ret_from_fork+0x1f/0x30 [19568.053068] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [19568.058998] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19568.078799] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19568.083728] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19568.088275] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19568.094526] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19568.097327] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19568.105631] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19568.109531] RAX: 0000000000007d0d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19568.111862] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19568.114139] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19568.116761] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19568.119528] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19568.122402] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19568.125382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19568.127597] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19568.130507] Call Trace: [19568.131488] [19568.132292] ? show_regs.cold.9+0x22/0x2f [19568.133840] ? watchdog_timer_fn.cold.6+0x62/0x10a [19568.135701] ? __hrtimer_run_queues+0x1c2/0x4d0 [19568.137471] ? watchdog+0x40/0x40 [19568.138779] ? hrtimer_interrupt+0x128/0x2c0 [19568.140428] ? smp_apic_timer_interrupt+0xa6/0x270 [19568.142356] ? apic_timer_interrupt+0xf/0x20 [19568.144081] [19568.144761] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19568.147025] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19568.149522] ? newidle_balance.isra.93+0x207/0x580 [19568.151326] do_raw_spin_lock+0x177/0x190 [19568.153061] _raw_spin_lock+0x21/0x30 [19568.154360] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19568.156364] process_one_work+0x2c8/0x700 [19568.157760] worker_thread+0x296/0x6e0 [19568.159270] ? rescuer_thread+0x570/0x570 [19568.160670] kthread+0x1d1/0x200 [19568.162111] ? set_kthread_struct+0x70/0x70 [19568.163399] ret_from_fork+0x1f/0x30 [19570.656639] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16933s! [19570.661567] Showing busy workqueues and worker pools: [19570.664150] workqueue events: flags=0x0 [19570.665723] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19570.668890] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19570.673326] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19570.676476] pending: cpu_latency_work [ptlrpc] [19570.679505] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19570.682775] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19570.687012] workqueue events_long: flags=0x0 [19570.688955] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19570.692192] pending: lu_objects_flush [obdclass] [19570.694617] workqueue events_unbound: flags=0x2 [19570.696958] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19570.699776] pending: flush_memcg_stats_dwork, idle_cull_fn [19570.702358] workqueue events_power_efficient: flags=0x80 [19570.704948] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19570.707901] pending: check_lifetime, do_cache_clean [19570.710447] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19570.714259] pending: neigh_periodic_work, neigh_periodic_work [19570.716682] workqueue mm_percpu_wq: flags=0x8 [19570.718441] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19570.721464] pending: vmstat_update, lru_add_drain_per_cpu [19570.724451] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19570.727565] pending: vmstat_update, lru_add_drain_per_cpu [19570.729296] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19570.731970] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19570.734919] workqueue xprtiod: flags=0xa [19570.736626] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19570.739554] pending: xprt_autoclose, xs_error_handle [19570.742050] workqueue ipv6_addrconf: flags=0x40008 [19570.745012] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19570.747997] pending: addrconf_verify_work [19570.750201] workqueue knbd0-recv: flags=0x1a [19570.752183] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19570.755076] in-flight: 651:recv_work [19570.756570] workqueue ptlrpc_pinger: flags=0x2 [19570.758706] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19570.761459] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19570.764352] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16933s workers=4 idle: 15979 31824 8 [19570.767708] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19561s workers=2 idle: 654 [19572.055012] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [19572.057519] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [19572.058979] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common [19572.064884] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19572.064969] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19572.064977] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19572.064980] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19572.064994] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19572.064999] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19572.065003] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19572.065006] RDX: 0000000000005909 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19572.065008] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19572.065011] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19572.065014] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19572.065017] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19572.065020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19572.065022] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19572.065031] Call Trace: [19572.065036] [19572.065043] ? show_regs.cold.9+0x22/0x2f [19572.065053] ? watchdog_timer_fn.cold.6+0x62/0x10a [19572.065059] ? __hrtimer_run_queues+0x1c2/0x4d0 [19572.065063] ? watchdog+0x40/0x40 [19572.065067] ? hrtimer_interrupt+0x128/0x2c0 [19572.065071] ? smp_apic_timer_interrupt+0xa6/0x270 [19572.065078] ? apic_timer_interrupt+0xf/0x20 [19572.065082] [19572.065086] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19572.065091] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19572.065096] do_raw_spin_lock+0x177/0x190 [19572.065100] _raw_spin_lock+0x21/0x30 [19572.065107] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19572.065378] obd_mod_free+0x46/0x140 [ptlrpc] [19572.065570] mdc_commit_open+0x91/0xb0 [mdc] [19572.065611] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19572.066155] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19572.066362] after_reply+0xe2c/0x1780 [ptlrpc] [19572.066560] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19572.066953] ? __next_timer_interrupt+0x160/0x160 [19572.066963] ptlrpcd+0xa82/0xe40 [ptlrpc] [19572.067177] ? do_wait_intr+0xf0/0xf0 [19572.067187] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19572.067428] kthread+0x1d1/0x200 [19572.067434] ? set_kthread_struct+0x70/0x70 [19572.067438] ret_from_fork+0x1f/0x30 [19572.199597] sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19572.207562] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19572.212024] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19572.215420] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19572.222991] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19572.234570] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19572.238394] RAX: 0000000000005e98 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19572.241453] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19572.244509] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19572.248347] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19572.251443] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19572.254551] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19572.259545] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19572.269488] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19572.271513] Call Trace: [19572.272843] [19572.273581] ? show_regs.cold.9+0x22/0x2f [19572.275016] ? watchdog_timer_fn.cold.6+0x62/0x10a [19572.277871] ? __hrtimer_run_queues+0x1c2/0x4d0 [19572.281607] ? watchdog+0x40/0x40 [19572.290348] ? hrtimer_interrupt+0x128/0x2c0 [19572.294653] ? smp_apic_timer_interrupt+0xa6/0x270 [19572.296696] ? apic_timer_interrupt+0xf/0x20 [19572.298037] [19572.298784] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19572.302225] do_raw_spin_lock+0x177/0x190 [19572.306513] _raw_spin_lock+0x21/0x30 [19572.310530] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19572.312190] ? do_raw_spin_unlock+0x75/0x190 [19572.318345] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19572.321249] ? do_wait_intr+0xf0/0xf0 [19572.326299] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19572.333226] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19572.335496] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19572.338454] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19572.340959] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19572.343458] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19572.345349] mdc_intent_lock+0x353/0x7e0 [mdc] [19572.348190] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19572.350284] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19572.353573] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19572.363256] lmv_intent_open+0x338/0x1150 [lmv] [19572.365921] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19572.368875] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19572.374887] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19572.384772] ll_intent_lock+0xc5/0xaa0 [lustre] [19572.387428] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19572.389466] ll_lookup_it+0xe0e/0x2270 [lustre] [19572.391925] ? do_raw_spin_unlock+0x75/0x190 [19572.393850] ? _raw_spin_unlock+0x12/0x30 [19572.395610] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19572.397928] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19572.400233] ll_atomic_open+0x36a/0xd70 [lustre] [19572.403637] lookup_open+0x356/0xb40 [19572.405693] do_last+0x313/0xfc0 [19572.407388] ? nd_jump_root+0xe5/0x160 [19572.409517] ? path_init+0x437/0x520 [19572.411469] path_openat+0xf7/0x500 [19572.413124] ? do_raw_spin_unlock+0x75/0x190 [19572.415393] do_filp_open+0x99/0x140 [19572.417500] ? getname_flags+0x6e/0x330 [19572.420182] ? __check_object_size+0xff/0x256 [19572.422376] ? do_raw_spin_unlock+0x75/0x190 [19572.424489] ? _raw_spin_unlock+0x12/0x30 [19572.426357] do_sys_openat2+0x2b4/0x410 [19572.428214] do_sys_open+0x73/0xa0 [19572.430091] __x64_sys_openat+0x24/0x30 [19572.431778] do_syscall_64+0xc1/0x440 [19572.433509] entry_SYSCALL_64_after_hwframe+0x49/0xae [19572.436298] RIP: 0033:0x7f53425a5332 [19572.438666] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19572.447934] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19572.451393] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19572.454987] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19572.458526] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19572.462138] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19572.465455] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19592.058020] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19592.064258] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19592.105864] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19592.125251] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19592.135288] RIP: 0010:native_safe_halt+0x2e/0x50 [19592.138998] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19592.153129] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19592.156996] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19592.163800] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19592.169933] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19592.177885] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19592.183546] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19592.192826] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19592.197676] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19592.199452] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19592.206878] Call Trace: [19592.208206] [19592.210304] ? show_regs.cold.9+0x22/0x2f [19592.213954] ? watchdog_timer_fn.cold.6+0x62/0x10a [19592.218074] ? __hrtimer_run_queues+0x1c2/0x4d0 [19592.221376] ? watchdog+0x40/0x40 [19592.224435] ? hrtimer_interrupt+0x128/0x2c0 [19592.227973] ? smp_apic_timer_interrupt+0xa6/0x270 [19592.231953] ? apic_timer_interrupt+0xf/0x20 [19592.233833] [19592.234623] ? native_safe_halt+0x2e/0x50 [19592.236264] kvm_wait+0xcd/0xe0 [19592.245954] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19592.249624] do_raw_spin_lock+0x177/0x190 [19592.252068] _raw_spin_lock+0x21/0x30 [19592.255845] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19592.262274] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19592.266719] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19592.270335] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19592.273795] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19592.277845] ? woken_wake_function+0x30/0x30 [19592.280100] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19592.285866] kthread+0x1d1/0x200 [19592.287863] ? set_kthread_struct+0x70/0x70 [19592.290702] ret_from_fork+0x1f/0x30 [19596.052005] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [19596.069104] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19596.104500] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19596.111451] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19596.118106] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19596.121004] RIP: 0010:native_safe_halt+0x2e/0x50 [19596.123185] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19596.132646] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19596.135896] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19596.139844] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [19596.142908] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19596.145895] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19596.149787] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19596.152880] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19596.156828] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19596.159300] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19596.164093] Call Trace: [19596.164945] [19596.165743] ? show_regs.cold.9+0x22/0x2f [19596.168482] ? watchdog_timer_fn.cold.6+0x62/0x10a [19596.169675] ? __hrtimer_run_queues+0x1c2/0x4d0 [19596.170952] ? watchdog+0x40/0x40 [19596.172371] ? hrtimer_interrupt+0x128/0x2c0 [19596.173886] ? smp_apic_timer_interrupt+0xa6/0x270 [19596.177090] ? apic_timer_interrupt+0xf/0x20 [19596.178831] [19596.179969] ? native_safe_halt+0x2e/0x50 [19596.181819] kvm_wait+0xcd/0xe0 [19596.182966] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19596.186568] ? newidle_balance.isra.93+0x207/0x580 [19596.188179] do_raw_spin_lock+0x177/0x190 [19596.189631] _raw_spin_lock+0x21/0x30 [19596.190653] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19596.192835] process_one_work+0x2c8/0x700 [19596.196818] worker_thread+0x296/0x6e0 [19596.198336] ? rescuer_thread+0x570/0x570 [19596.200979] kthread+0x1d1/0x200 [19596.203019] ? set_kthread_struct+0x70/0x70 [19596.204887] ret_from_fork+0x1f/0x30 [19600.055035] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [19600.057375] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [19600.060533] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19600.060607] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19600.060614] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19600.060617] RIP: 0010:native_safe_halt+0x2e/0x50 [19600.060631] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19600.060635] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19600.060640] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19600.060643] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19600.060645] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19600.060648] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19600.060650] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19600.060653] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19600.060656] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19600.060659] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19600.060667] Call Trace: [19600.060673] [19600.060679] ? show_regs.cold.9+0x22/0x2f [19600.060692] ? watchdog_timer_fn.cold.6+0x62/0x10a [19600.060699] ? __hrtimer_run_queues+0x1c2/0x4d0 [19600.060703] ? watchdog+0x40/0x40 [19600.060708] ? hrtimer_interrupt+0x128/0x2c0 [19600.060712] ? smp_apic_timer_interrupt+0xa6/0x270 [19600.060720] ? apic_timer_interrupt+0xf/0x20 [19600.060724] [19600.060727] ? native_safe_halt+0x2e/0x50 [19600.060731] kvm_wait+0xcd/0xe0 [19600.060738] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19600.060747] do_raw_spin_lock+0x177/0x190 [19600.060751] _raw_spin_lock+0x21/0x30 [19600.060755] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19600.061024] ? do_raw_spin_unlock+0x75/0x190 [19600.061031] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19600.069857] Modules linked in: [19600.093675] ? do_wait_intr+0xf0/0xf0 [19600.093698] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19600.093958] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19600.103324] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19600.103409] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19600.103417] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19600.103421] RIP: 0010:native_safe_halt+0x2e/0x50 [19600.103435] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19600.103440] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19600.103444] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19600.103447] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19600.103450] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19600.103452] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19600.103455] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19600.103458] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19600.103461] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19600.103464] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19600.103473] Call Trace: [19600.103478] [19600.103483] ? show_regs.cold.9+0x22/0x2f [19600.103493] ? watchdog_timer_fn.cold.6+0x62/0x10a [19600.103499] ? __hrtimer_run_queues+0x1c2/0x4d0 [19600.103503] ? watchdog+0x40/0x40 [19600.103508] ? hrtimer_interrupt+0x128/0x2c0 [19600.103512] ? smp_apic_timer_interrupt+0xa6/0x270 [19600.103519] ? apic_timer_interrupt+0xf/0x20 [19600.103524] [19600.103527] ? native_safe_halt+0x2e/0x50 [19600.103531] kvm_wait+0xcd/0xe0 [19600.103540] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19600.103549] do_raw_spin_lock+0x177/0x190 [19600.103554] _raw_spin_lock+0x21/0x30 [19600.103560] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19600.104713] obd_mod_free+0x46/0x140 [ptlrpc] [19600.104932] mdc_commit_open+0x91/0xb0 [mdc] [19600.104973] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19600.105168] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19600.105393] after_reply+0xe2c/0x1780 [ptlrpc] [19600.105650] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19600.105863] ? __next_timer_interrupt+0x160/0x160 [19600.105872] ptlrpcd+0xa82/0xe40 [ptlrpc] [19600.106082] ? do_wait_intr+0xf0/0xf0 [19600.106092] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19600.106590] kthread+0x1d1/0x200 [19600.106596] ? set_kthread_struct+0x70/0x70 [19600.106599] ret_from_fork+0x1f/0x30 [19600.439678] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19600.442161] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19600.445577] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19600.453204] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19600.454438] mdc_intent_lock+0x353/0x7e0 [mdc] [19600.464002] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19600.469121] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19600.473507] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19600.482637] lmv_intent_open+0x338/0x1150 [lmv] [19600.485872] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19600.489136] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19600.495057] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19600.498022] ll_intent_lock+0xc5/0xaa0 [lustre] [19600.501982] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19600.503907] ll_lookup_it+0xe0e/0x2270 [lustre] [19600.509817] ? do_raw_spin_unlock+0x75/0x190 [19600.514127] ? _raw_spin_unlock+0x12/0x30 [19600.515976] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19600.518134] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19600.520823] ll_atomic_open+0x36a/0xd70 [lustre] [19600.526271] lookup_open+0x356/0xb40 [19600.530800] do_last+0x313/0xfc0 [19600.531830] ? nd_jump_root+0xe5/0x160 [19600.533224] ? path_init+0x437/0x520 [19600.540347] path_openat+0xf7/0x500 [19600.542522] ? do_raw_spin_unlock+0x75/0x190 [19600.544227] do_filp_open+0x99/0x140 [19600.545786] ? getname_flags+0x6e/0x330 [19600.546996] ? __check_object_size+0xff/0x256 [19600.551752] ? do_raw_spin_unlock+0x75/0x190 [19600.553153] ? _raw_spin_unlock+0x12/0x30 [19600.556966] do_sys_openat2+0x2b4/0x410 [19600.559789] do_sys_open+0x73/0xa0 [19600.561047] __x64_sys_openat+0x24/0x30 [19600.563751] do_syscall_64+0xc1/0x440 [19600.565491] entry_SYSCALL_64_after_hwframe+0x49/0xae [19600.568548] RIP: 0033:0x7f53425a5332 [19600.570990] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19600.587635] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19600.593025] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19600.596018] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19600.601844] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19600.605752] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19600.608750] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19600.865391] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16963s! [19600.874725] Showing busy workqueues and worker pools: [19600.878653] workqueue events: flags=0x0 [19600.884989] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19600.892765] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19600.899483] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19600.902877] pending: cpu_latency_work [ptlrpc] [19600.905507] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19600.908441] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19600.912461] workqueue events_long: flags=0x0 [19600.914361] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19600.917463] pending: lu_objects_flush [obdclass] [19600.919626] workqueue events_unbound: flags=0x2 [19600.921732] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19600.924486] pending: flush_memcg_stats_dwork, idle_cull_fn [19600.926643] workqueue events_power_efficient: flags=0x80 [19600.928926] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19600.931847] pending: check_lifetime, do_cache_clean [19600.933828] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19600.935953] pending: neigh_periodic_work, neigh_periodic_work [19600.938397] workqueue mm_percpu_wq: flags=0x8 [19600.940354] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19600.943215] pending: vmstat_update, lru_add_drain_per_cpu [19600.945697] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19600.948681] pending: vmstat_update, lru_add_drain_per_cpu [19600.950742] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19600.952930] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19600.955404] workqueue xprtiod: flags=0xa [19600.957595] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19600.960593] pending: xprt_autoclose, xs_error_handle [19600.962953] workqueue ipv6_addrconf: flags=0x40008 [19600.964973] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19600.968425] pending: addrconf_verify_work [19600.969988] workqueue knbd0-recv: flags=0x1a [19600.972056] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19600.974923] in-flight: 651:recv_work [19600.976611] workqueue ptlrpc_pinger: flags=0x2 [19600.978790] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19600.981328] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19600.984526] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16963s workers=4 idle: 15979 31824 8 [19600.987908] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19591s workers=2 idle: 654 [19614.861243] rcu: INFO: rcu_sched self-detected stall on CPU [19614.863194] rcu: 2-....: (9885568 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [19614.867209] (t=17520435 jiffies g=274185 q=68404) [19614.869401] rcu: rcu_sched kthread starved for 16978801 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [19614.875450] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [19614.881392] rcu: RCU grace-period kthread stack dump: [19614.884984] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [19614.890048] Call Trace: [19614.890832] __schedule+0x351/0xcb0 [19614.892233] schedule+0xc0/0x180 [19614.893615] schedule_timeout+0xb4/0x190 [19614.895261] ? do_raw_spin_unlock+0x75/0x190 [19614.898425] ? __next_timer_interrupt+0x160/0x160 [19614.901425] ? prepare_to_swait_event+0xf3/0x140 [19614.904230] rcu_gp_kthread+0x8dd/0x15b0 [19614.905469] ? force_qs_rnp+0x2c0/0x2c0 [19614.906599] kthread+0x1d1/0x200 [19614.913660] ? set_kthread_struct+0x70/0x70 [19614.915395] ret_from_fork+0x1f/0x30 [19614.917585] rcu: Stack dump where RCU GP kthread last ran: [19614.920936] Sending NMI from CPU 2 to CPUs 0: [19614.923659] NMI backtrace for cpu 0 [19614.923665] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19614.923667] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19614.923668] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19614.923671] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19614.923673] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19614.923676] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 [19614.923678] RAX: 0000000000001389 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19614.923679] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19614.923681] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19614.923682] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19614.923684] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19614.923685] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19614.923687] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19614.923688] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19614.923689] Call Trace: [19614.923690] [19614.923691] ? show_regs.cold.9+0x22/0x2f [19614.923692] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [19614.923693] ? nmi_cpu_backtrace_handler+0x15/0x40 [19614.923694] ? nmi_handle+0xbd/0x210 [19614.923696] ? default_do_nmi+0x4b/0x1b0 [19614.923697] ? do_nmi+0x280/0x380 [19614.923698] ? end_repeat_nmi+0x16/0x1f [19614.923699] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19614.923701] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19614.923702] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19614.923703] [19614.923704] ? newidle_balance.isra.93+0x207/0x580 [19614.923706] do_raw_spin_lock+0x177/0x190 [19614.923707] _raw_spin_lock+0x21/0x30 [19614.923708] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19614.923710] process_one_work+0x2c8/0x700 [19614.923711] worker_thread+0x296/0x6e0 [19614.923711] ? rescuer_thread+0x570/0x570 [19614.923712] kthread+0x1d1/0x200 [19614.923713] ? set_kthread_struct+0x70/0x70 [19614.923714] ret_from_fork+0x1f/0x30 [19614.924121] Sending NMI from CPU 2 to CPUs 1: [19615.038386] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [19615.040966] NMI backtrace for cpu 2 [19615.043299] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19615.048216] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19615.052558] Call Trace: [19615.053792] [19615.054887] ? dump_stack+0xbb/0x10e [19615.056462] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [19615.059599] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [19615.062232] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [19615.065285] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [19615.067771] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [19615.071221] ? rcu_dump_cpu_stacks+0x11b/0x171 [19615.073960] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [19615.076470] ? update_process_times+0x6d/0xc0 [19615.078969] ? tick_sched_handle.isra.18+0x27/0x90 [19615.082121] ? tick_sched_timer+0x53/0xb0 [19615.084810] ? __hrtimer_run_queues+0x1c2/0x4d0 [19615.087813] ? tick_sched_handle.isra.18+0x90/0x90 [19615.090271] ? hrtimer_interrupt+0x128/0x2c0 [19615.094097] ? smp_apic_timer_interrupt+0xa6/0x270 [19615.096192] ? apic_timer_interrupt+0xf/0x20 [19615.098359] [19615.099433] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19615.101153] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19615.102284] ? do_raw_spin_lock+0x177/0x190 [19615.103773] ? _raw_spin_lock+0x21/0x30 [19615.105997] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19615.110177] ? obd_mod_free+0x46/0x140 [ptlrpc] [19615.112458] ? mdc_commit_open+0x91/0xb0 [mdc] [19615.114569] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19615.117360] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19615.120188] ? after_reply+0xe2c/0x1780 [ptlrpc] [19615.122195] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19615.124731] ? __next_timer_interrupt+0x160/0x160 [19615.126518] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [19615.128624] ? do_wait_intr+0xf0/0xf0 [19615.130819] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19615.134393] ? kthread+0x1d1/0x200 [19615.135866] ? set_kthread_struct+0x70/0x70 [19615.137787] ? ret_from_fork+0x1f/0x30 [19620.058013] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19620.061389] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19620.081150] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19620.085790] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19620.089569] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19620.092065] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19620.099671] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19620.102694] RAX: 0000000000003edd RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19620.105599] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19620.109380] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19620.112187] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19620.115126] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19620.117655] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19620.121313] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19620.123526] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19620.126301] Call Trace: [19620.127288] [19620.128219] ? show_regs.cold.9+0x22/0x2f [19620.130207] ? watchdog_timer_fn.cold.6+0x62/0x10a [19620.132119] ? __hrtimer_run_queues+0x1c2/0x4d0 [19620.133992] ? watchdog+0x40/0x40 [19620.135449] ? hrtimer_interrupt+0x128/0x2c0 [19620.136847] ? smp_apic_timer_interrupt+0xa6/0x270 [19620.138892] ? apic_timer_interrupt+0xf/0x20 [19620.140602] [19620.141551] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19620.143788] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19620.145994] do_raw_spin_lock+0x177/0x190 [19620.147593] _raw_spin_lock+0x21/0x30 [19620.149153] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19620.152005] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19620.153931] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19620.155824] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19620.157685] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19620.160630] ? woken_wake_function+0x30/0x30 [19620.162177] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19620.164579] kthread+0x1d1/0x200 [19620.165669] ? set_kthread_struct+0x70/0x70 [19620.167449] ret_from_fork+0x1f/0x30 [19631.073573] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 16993s! [19631.080588] Showing busy workqueues and worker pools: [19631.084777] workqueue events: flags=0x0 [19631.088205] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19631.094289] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19631.113166] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19631.116702] pending: cpu_latency_work [ptlrpc] [19631.128982] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19631.136523] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19631.144847] workqueue events_long: flags=0x0 [19631.148275] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19631.153078] pending: lu_objects_flush [obdclass] [19631.167080] workqueue events_unbound: flags=0x2 [19631.174283] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19631.199935] pending: flush_memcg_stats_dwork, idle_cull_fn [19631.207047] workqueue events_power_efficient: flags=0x80 [19631.213391] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19631.222737] pending: check_lifetime, do_cache_clean [19631.224496] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19631.234396] pending: neigh_periodic_work, neigh_periodic_work [19631.240517] workqueue mm_percpu_wq: flags=0x8 [19631.245502] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19631.253293] pending: vmstat_update, lru_add_drain_per_cpu [19631.260065] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19631.264938] pending: vmstat_update, lru_add_drain_per_cpu [19631.273380] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19631.276809] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19631.283546] workqueue xprtiod: flags=0xa [19631.288412] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19631.290615] pending: xprt_autoclose, xs_error_handle [19631.300498] workqueue ipv6_addrconf: flags=0x40008 [19631.304374] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19631.313150] pending: addrconf_verify_work [19631.316179] workqueue knbd0-recv: flags=0x1a [19631.324454] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19631.333130] in-flight: 651:recv_work [19631.337223] workqueue ptlrpc_pinger: flags=0x2 [19631.340334] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19631.352143] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19631.360684] pool 8: cpus=0-3 flags=0x4 nice=0 hung=16994s workers=4 idle: 15979 31824 8 [19631.372676] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19621s workers=2 idle: 654 [19640.055010] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [19640.057390] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [19640.062089] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19640.062166] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19640.062175] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19640.062179] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [19640.062192] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [19640.062197] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19640.062202] RAX: 000000000000357e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19640.062204] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19640.062206] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19640.062209] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19640.062211] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19640.062215] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19640.062218] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19640.062221] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19640.062231] Call Trace: [19640.062237] [19640.062243] ? show_regs.cold.9+0x22/0x2f [19640.062252] ? watchdog_timer_fn.cold.6+0x62/0x10a [19640.062265] ? __hrtimer_run_queues+0x1c2/0x4d0 [19640.062270] ? watchdog+0x40/0x40 [19640.062274] ? hrtimer_interrupt+0x128/0x2c0 [19640.062278] ? smp_apic_timer_interrupt+0xa6/0x270 [19640.062285] ? apic_timer_interrupt+0xf/0x20 [19640.062290] [19640.062293] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [19640.062300] do_raw_spin_lock+0x177/0x190 [19640.062305] _raw_spin_lock+0x21/0x30 [19640.062309] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19640.067286] Modules linked in: lustre(O) [19640.131761] ? do_raw_spin_unlock+0x75/0x190 [19640.131782] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19640.132346] ? do_wait_intr+0xf0/0xf0 [19640.132358] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19640.141239] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log [19640.146719] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19640.147189] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19640.150324] dm_mod sha512_ssse3 sha512_generic [19640.150337] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19640.150344] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19640.150347] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19640.150359] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19640.150363] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19640.150368] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19640.150371] RDX: 00000000000072ac RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19640.150374] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19640.150377] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19640.150380] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19640.150385] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19640.180971] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19640.184379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19640.184389] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19640.184398] Call Trace: [19640.184406] [19640.184411] ? show_regs.cold.9+0x22/0x2f [19640.184425] ? watchdog_timer_fn.cold.6+0x62/0x10a [19640.184432] ? __hrtimer_run_queues+0x1c2/0x4d0 [19640.184437] ? watchdog+0x40/0x40 [19640.184440] ? hrtimer_interrupt+0x128/0x2c0 [19640.184444] ? smp_apic_timer_interrupt+0xa6/0x270 [19640.184452] ? apic_timer_interrupt+0xf/0x20 [19640.184457] [19640.184460] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19640.184466] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19640.184471] do_raw_spin_lock+0x177/0x190 [19640.184476] _raw_spin_lock+0x21/0x30 [19640.184483] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19640.184840] obd_mod_free+0x46/0x140 [ptlrpc] [19640.185048] mdc_commit_open+0x91/0xb0 [mdc] [19640.185088] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19640.191320] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19640.191375] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19640.191408] mdc_intent_lock+0x353/0x7e0 [mdc] [19640.191438] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19640.192155] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19640.192448] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19640.192480] lmv_intent_open+0x338/0x1150 [lmv] [19640.192511] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19640.193198] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19640.193281] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19640.193306] ll_intent_lock+0xc5/0xaa0 [lustre] [19640.193389] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19640.193476] ll_lookup_it+0xe0e/0x2270 [lustre] [19640.194433] ? do_raw_spin_unlock+0x75/0x190 [19640.194441] ? _raw_spin_unlock+0x12/0x30 [19640.194448] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19640.194475] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19640.194496] ll_atomic_open+0x36a/0xd70 [lustre] [19640.194791] lookup_open+0x356/0xb40 [19640.194801] do_last+0x313/0xfc0 [19640.194806] ? nd_jump_root+0xe5/0x160 [19640.194811] ? path_init+0x437/0x520 [19640.194816] path_openat+0xf7/0x500 [19640.194822] ? do_raw_spin_unlock+0x75/0x190 [19640.194827] do_filp_open+0x99/0x140 [19640.194834] ? getname_flags+0x6e/0x330 [19640.194838] ? __check_object_size+0xff/0x256 [19640.194845] ? do_raw_spin_unlock+0x75/0x190 [19640.194849] ? _raw_spin_unlock+0x12/0x30 [19640.194854] do_sys_openat2+0x2b4/0x410 [19640.194859] do_sys_open+0x73/0xa0 [19640.194864] __x64_sys_openat+0x24/0x30 [19640.194868] do_syscall_64+0xc1/0x440 [19640.194874] entry_SYSCALL_64_after_hwframe+0x49/0xae [19640.194881] RIP: 0033:0x7f53425a5332 [19640.194888] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19640.194893] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19640.194899] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19640.194901] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19640.194904] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19640.194906] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19640.194908] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19640.456931] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19640.460090] after_reply+0xe2c/0x1780 [ptlrpc] [19640.462283] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19640.465403] ? __next_timer_interrupt+0x160/0x160 [19640.466974] ptlrpcd+0xa82/0xe40 [ptlrpc] [19640.469126] ? do_wait_intr+0xf0/0xf0 [19640.470878] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19640.473009] kthread+0x1d1/0x200 [19640.474199] ? set_kthread_struct+0x70/0x70 [19640.476005] ret_from_fork+0x1f/0x30 [19648.061129] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19648.069813] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19648.114653] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19648.126092] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19648.133375] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19648.137065] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19648.158948] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [19648.167841] RAX: 0000000000003961 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19648.170312] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19648.180298] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19648.186268] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19648.189993] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19648.194134] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19648.199768] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19648.205707] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19648.215975] Call Trace: [19648.218229] [19648.219671] ? show_regs.cold.9+0x22/0x2f [19648.221704] ? watchdog_timer_fn.cold.6+0x62/0x10a [19648.224154] ? __hrtimer_run_queues+0x1c2/0x4d0 [19648.226680] ? watchdog+0x40/0x40 [19648.228710] ? hrtimer_interrupt+0x128/0x2c0 [19648.230741] ? smp_apic_timer_interrupt+0xa6/0x270 [19648.233130] ? apic_timer_interrupt+0xf/0x20 [19648.235224] [19648.236253] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19648.238791] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19648.241416] do_raw_spin_lock+0x177/0x190 [19648.242872] _raw_spin_lock+0x21/0x30 [19648.246088] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19648.249938] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19648.253110] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19648.255579] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19648.257565] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19648.260346] ? woken_wake_function+0x30/0x30 [19648.262375] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19648.269582] kthread+0x1d1/0x200 [19648.274167] ? set_kthread_struct+0x70/0x70 [19648.276499] ret_from_fork+0x1f/0x30 [19656.052014] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [19656.054110] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19656.071999] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19656.076040] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19656.079346] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19656.085800] RIP: 0010:native_safe_halt+0x2e/0x50 [19656.087845] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19656.104511] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19656.110739] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19656.114302] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19656.119830] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19656.127356] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19656.130054] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19656.140413] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19656.148863] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19656.150987] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19656.154110] Call Trace: [19656.154962] [19656.155849] ? show_regs.cold.9+0x22/0x2f [19656.157149] ? watchdog_timer_fn.cold.6+0x62/0x10a [19656.158841] ? __hrtimer_run_queues+0x1c2/0x4d0 [19656.160949] ? watchdog+0x40/0x40 [19656.162439] ? hrtimer_interrupt+0x128/0x2c0 [19656.164429] ? smp_apic_timer_interrupt+0xa6/0x270 [19656.166112] ? apic_timer_interrupt+0xf/0x20 [19656.167763] [19656.168846] ? native_safe_halt+0x2e/0x50 [19656.170568] kvm_wait+0xcd/0xe0 [19656.171717] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19656.173936] ? newidle_balance.isra.93+0x207/0x580 [19656.176116] do_raw_spin_lock+0x177/0x190 [19656.180564] _raw_spin_lock+0x21/0x30 [19656.193356] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19656.202424] process_one_work+0x2c8/0x700 [19656.206634] worker_thread+0x296/0x6e0 [19656.210050] ? rescuer_thread+0x570/0x570 [19656.212427] kthread+0x1d1/0x200 [19656.214200] ? set_kthread_struct+0x70/0x70 [19656.218356] ret_from_fork+0x1f/0x30 [19661.802042] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17024s! [19661.805367] Showing busy workqueues and worker pools: [19661.807598] workqueue events: flags=0x0 [19661.809132] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19661.811567] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19661.816149] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19661.818891] pending: cpu_latency_work [ptlrpc] [19661.821064] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19661.823657] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19661.827437] workqueue events_long: flags=0x0 [19661.829071] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19661.831723] pending: lu_objects_flush [obdclass] [19661.833996] workqueue events_unbound: flags=0x2 [19661.835834] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19661.838606] pending: flush_memcg_stats_dwork, idle_cull_fn [19661.841085] workqueue events_power_efficient: flags=0x80 [19661.843353] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19661.846009] pending: check_lifetime, do_cache_clean [19661.847397] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19661.852442] pending: neigh_periodic_work, neigh_periodic_work [19661.854724] workqueue mm_percpu_wq: flags=0x8 [19661.856607] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19661.858990] pending: vmstat_update, lru_add_drain_per_cpu [19661.860924] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19661.865746] pending: vmstat_update, lru_add_drain_per_cpu [19661.868827] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19661.872993] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19661.874700] workqueue xprtiod: flags=0xa [19661.880015] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19661.883331] pending: xprt_autoclose, xs_error_handle [19661.885374] workqueue ipv6_addrconf: flags=0x40008 [19661.887037] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19661.889725] pending: addrconf_verify_work [19661.890952] workqueue knbd0-recv: flags=0x1a [19661.892854] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19661.895702] in-flight: 651:recv_work [19661.897341] workqueue ptlrpc_pinger: flags=0x2 [19661.898852] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19661.901444] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19661.903996] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17024s workers=4 idle: 15979 31824 8 [19661.907638] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19652s workers=2 idle: 654 [19668.055032] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [19668.057055] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [19668.057876] Modules linked in: [19668.060033] Modules linked in: [19668.060034] lustre(O) osp(O) [19668.062004] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19668.062555] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19668.062562] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19668.062565] RIP: 0010:native_safe_halt+0x2e/0x50 [19668.062578] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19668.062582] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19668.062586] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19668.062589] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19668.062590] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19668.062593] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19668.062595] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19668.062598] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19668.062601] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19668.062603] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19668.062610] Call Trace: [19668.062616] [19668.062621] ? show_regs.cold.9+0x22/0x2f [19668.062631] ? watchdog_timer_fn.cold.6+0x62/0x10a [19668.062637] ? __hrtimer_run_queues+0x1c2/0x4d0 [19668.062641] ? watchdog+0x40/0x40 [19668.062645] ? hrtimer_interrupt+0x128/0x2c0 [19668.062649] ? smp_apic_timer_interrupt+0xa6/0x270 [19668.062656] ? apic_timer_interrupt+0xf/0x20 [19668.062661] [19668.062664] ? native_safe_halt+0x2e/0x50 [19668.062668] kvm_wait+0xcd/0xe0 [19668.062676] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19668.062683] do_raw_spin_lock+0x177/0x190 [19668.062687] _raw_spin_lock+0x21/0x30 [19668.062692] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19668.062949] obd_mod_free+0x46/0x140 [ptlrpc] [19668.064219] mdc_commit_open+0x91/0xb0 [mdc] [19668.064255] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19668.064469] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19668.064755] after_reply+0xe2c/0x1780 [ptlrpc] [19668.064974] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19668.065190] ? __next_timer_interrupt+0x160/0x160 [19668.065201] ptlrpcd+0xa82/0xe40 [ptlrpc] [19668.065406] ? do_wait_intr+0xf0/0xf0 [19668.065416] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19668.066243] kthread+0x1d1/0x200 [19668.066250] ? set_kthread_struct+0x70/0x70 [19668.066253] ret_from_fork+0x1f/0x30 [19668.197539] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19668.215818] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19668.220879] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19668.224774] RIP: 0010:native_safe_halt+0x2e/0x50 [19668.225860] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19668.232031] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19668.236892] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19668.240561] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19668.242420] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19668.244703] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19668.246984] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19668.252502] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19668.263797] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19668.265749] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19668.270924] Call Trace: [19668.271502] [19668.272171] ? show_regs.cold.9+0x22/0x2f [19668.274703] ? watchdog_timer_fn.cold.6+0x62/0x10a [19668.275980] ? __hrtimer_run_queues+0x1c2/0x4d0 [19668.278281] ? watchdog+0x40/0x40 [19668.279507] ? hrtimer_interrupt+0x128/0x2c0 [19668.282953] ? smp_apic_timer_interrupt+0xa6/0x270 [19668.284017] ? apic_timer_interrupt+0xf/0x20 [19668.289659] [19668.290224] ? native_safe_halt+0x2e/0x50 [19668.292959] kvm_wait+0xcd/0xe0 [19668.294945] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19668.297218] do_raw_spin_lock+0x177/0x190 [19668.302899] _raw_spin_lock+0x21/0x30 [19668.305414] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19668.308485] ? do_raw_spin_unlock+0x75/0x190 [19668.311411] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19668.317572] ? do_wait_intr+0xf0/0xf0 [19668.319476] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19668.323627] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19668.325718] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19668.329872] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19668.332718] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19668.334856] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19668.336459] mdc_intent_lock+0x353/0x7e0 [mdc] [19668.337883] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19668.341476] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19668.343725] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19668.345012] lmv_intent_open+0x338/0x1150 [lmv] [19668.346784] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19668.348665] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19668.350825] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19668.351973] ll_intent_lock+0xc5/0xaa0 [lustre] [19668.353375] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19668.354787] ll_lookup_it+0xe0e/0x2270 [lustre] [19668.356176] ? do_raw_spin_unlock+0x75/0x190 [19668.357457] ? _raw_spin_unlock+0x12/0x30 [19668.358559] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19668.359777] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19668.360954] ll_atomic_open+0x36a/0xd70 [lustre] [19668.362198] lookup_open+0x356/0xb40 [19668.363025] do_last+0x313/0xfc0 [19668.363837] ? nd_jump_root+0xe5/0x160 [19668.364759] ? path_init+0x437/0x520 [19668.365646] path_openat+0xf7/0x500 [19668.366714] ? do_raw_spin_unlock+0x75/0x190 [19668.367739] do_filp_open+0x99/0x140 [19668.368676] ? getname_flags+0x6e/0x330 [19668.369594] ? __check_object_size+0xff/0x256 [19668.370645] ? do_raw_spin_unlock+0x75/0x190 [19668.371690] ? _raw_spin_unlock+0x12/0x30 [19668.372800] do_sys_openat2+0x2b4/0x410 [19668.373764] do_sys_open+0x73/0xa0 [19668.374609] __x64_sys_openat+0x24/0x30 [19668.375629] do_syscall_64+0xc1/0x440 [19668.376573] entry_SYSCALL_64_after_hwframe+0x49/0xae [19668.377822] RIP: 0033:0x7f53425a5332 [19668.378882] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19668.384035] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19668.385947] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19668.387768] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19668.389543] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19668.391534] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19668.393189] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19676.059073] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19676.067740] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19676.125335] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19676.134834] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19676.140947] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19676.148638] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19676.169834] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19676.174778] RAX: 0000000000005089 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19676.178736] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19676.182042] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19676.185660] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19676.191208] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19676.197143] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19676.201280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19676.206821] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19676.214481] Call Trace: [19676.216183] [19676.217210] ? show_regs.cold.9+0x22/0x2f [19676.222551] ? watchdog_timer_fn.cold.6+0x62/0x10a [19676.227152] ? __hrtimer_run_queues+0x1c2/0x4d0 [19676.234603] ? watchdog+0x40/0x40 [19676.235889] ? hrtimer_interrupt+0x128/0x2c0 [19676.237406] ? smp_apic_timer_interrupt+0xa6/0x270 [19676.251864] ? apic_timer_interrupt+0xf/0x20 [19676.258412] [19676.259149] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19676.260944] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19676.268172] do_raw_spin_lock+0x177/0x190 [19676.269535] _raw_spin_lock+0x21/0x30 [19676.282201] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19676.283891] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19676.304405] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19676.306437] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19676.319079] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19676.325236] ? woken_wake_function+0x30/0x30 [19676.326695] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19676.330550] kthread+0x1d1/0x200 [19676.332722] ? set_kthread_struct+0x70/0x70 [19676.334814] ret_from_fork+0x1f/0x30 [19688.053369] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [19688.068559] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19688.095769] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19688.102789] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19688.108419] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19688.112452] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19688.115361] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19688.122703] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19688.130273] RAX: 0000000000007f0a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19688.134600] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19688.137609] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19688.141138] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19688.144559] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19688.147788] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19688.151788] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19688.155089] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19688.158505] Call Trace: [19688.159797] [19688.160857] ? show_regs.cold.9+0x22/0x2f [19688.163079] ? watchdog_timer_fn.cold.6+0x62/0x10a [19688.165395] ? __hrtimer_run_queues+0x1c2/0x4d0 [19688.168083] ? watchdog+0x40/0x40 [19688.170888] ? hrtimer_interrupt+0x128/0x2c0 [19688.174079] ? smp_apic_timer_interrupt+0xa6/0x270 [19688.177007] ? apic_timer_interrupt+0xf/0x20 [19688.182374] [19688.184152] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19688.190279] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19688.193277] ? newidle_balance.isra.93+0x207/0x580 [19688.196653] do_raw_spin_lock+0x177/0x190 [19688.199573] _raw_spin_lock+0x21/0x30 [19688.201520] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19688.204185] process_one_work+0x2c8/0x700 [19688.206042] worker_thread+0x296/0x6e0 [19688.207911] ? rescuer_thread+0x570/0x570 [19688.209846] kthread+0x1d1/0x200 [19688.211726] ? set_kthread_struct+0x70/0x70 [19688.216203] ret_from_fork+0x1f/0x30 [19692.000032] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17054s! [19692.004721] Showing busy workqueues and worker pools: [19692.015874] workqueue events: flags=0x0 [19692.019604] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19692.027227] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19692.038023] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19692.047200] pending: cpu_latency_work [ptlrpc] [19692.054727] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19692.061588] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19692.069012] workqueue events_long: flags=0x0 [19692.074738] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19692.079103] pending: lu_objects_flush [obdclass] [19692.083977] workqueue events_unbound: flags=0x2 [19692.089973] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19692.095799] pending: flush_memcg_stats_dwork, idle_cull_fn [19692.101766] workqueue events_power_efficient: flags=0x80 [19692.106197] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19692.112987] pending: check_lifetime, do_cache_clean [19692.118934] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19692.121285] pending: neigh_periodic_work, neigh_periodic_work [19692.127302] workqueue mm_percpu_wq: flags=0x8 [19692.135557] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19692.145221] pending: vmstat_update, lru_add_drain_per_cpu [19692.152096] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19692.160296] pending: vmstat_update, lru_add_drain_per_cpu [19692.168186] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19692.173495] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19692.184164] workqueue xprtiod: flags=0xa [19692.193125] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19692.198783] pending: xprt_autoclose, xs_error_handle [19692.205056] workqueue ipv6_addrconf: flags=0x40008 [19692.210534] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19692.219273] pending: addrconf_verify_work [19692.226418] workqueue knbd0-recv: flags=0x1a [19692.231607] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19692.242222] in-flight: 651:recv_work [19692.247631] workqueue ptlrpc_pinger: flags=0x2 [19692.253924] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19692.259128] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19692.265111] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17055s workers=4 idle: 15979 31824 8 [19692.273101] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19682s workers=2 idle: 654 [19696.055011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [19696.057292] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [19696.058008] Modules linked in: lustre(O) osp(O) ofd(O) [19696.064665] Modules linked in: [19696.064669] lod(O) mdt(O) [19696.070763] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19696.070844] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19696.070854] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19696.070857] RIP: 0010:native_safe_halt+0x2e/0x50 [19696.070869] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19696.070875] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19696.070879] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19696.070882] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19696.070884] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19696.070886] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19696.070889] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19696.070891] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19696.070894] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19696.070897] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19696.072303] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19696.072442] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19696.072449] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19696.072452] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19696.072464] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19696.072468] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19696.072472] RAX: 0000000000001061 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19696.072475] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19696.072477] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19696.072480] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19696.072482] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19696.072485] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19696.072487] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19696.072490] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19696.072498] Call Trace: [19696.072502] [19696.072508] ? show_regs.cold.9+0x22/0x2f [19696.072517] ? watchdog_timer_fn.cold.6+0x62/0x10a [19696.072523] ? __hrtimer_run_queues+0x1c2/0x4d0 [19696.072527] ? watchdog+0x40/0x40 [19696.072531] ? hrtimer_interrupt+0x128/0x2c0 [19696.072535] ? smp_apic_timer_interrupt+0xa6/0x270 [19696.072542] ? apic_timer_interrupt+0xf/0x20 [19696.072546] [19696.072549] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19696.072555] do_raw_spin_lock+0x177/0x190 [19696.072560] _raw_spin_lock+0x21/0x30 [19696.072564] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19696.072889] ? do_raw_spin_unlock+0x75/0x190 [19696.072895] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19696.073107] ? do_wait_intr+0xf0/0xf0 [19696.073115] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19696.073878] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19696.076270] Call Trace: [19696.095152] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19696.109100] [19696.109116] ? show_regs.cold.9+0x22/0x2f [19696.109130] ? watchdog_timer_fn.cold.6+0x62/0x10a [19696.109137] ? __hrtimer_run_queues+0x1c2/0x4d0 [19696.109142] ? watchdog+0x40/0x40 [19696.109147] ? hrtimer_interrupt+0x128/0x2c0 [19696.109151] ? smp_apic_timer_interrupt+0xa6/0x270 [19696.109161] ? apic_timer_interrupt+0xf/0x20 [19696.109166] [19696.109169] ? native_safe_halt+0x2e/0x50 [19696.109173] kvm_wait+0xcd/0xe0 [19696.109182] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19696.109190] do_raw_spin_lock+0x177/0x190 [19696.109194] _raw_spin_lock+0x21/0x30 [19696.109201] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19696.115898] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19696.117748] obd_mod_free+0x46/0x140 [ptlrpc] [19696.118046] mdc_commit_open+0x91/0xb0 [mdc] [19696.118087] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19696.130896] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19696.133357] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19696.133665] after_reply+0xe2c/0x1780 [ptlrpc] [19696.133859] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19696.134064] ? __next_timer_interrupt+0x160/0x160 [19696.134075] ptlrpcd+0xa82/0xe40 [ptlrpc] [19696.134276] ? do_wait_intr+0xf0/0xf0 [19696.134285] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19696.134482] kthread+0x1d1/0x200 [19696.134489] ? set_kthread_struct+0x70/0x70 [19696.134493] ret_from_fork+0x1f/0x30 [19696.362505] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19696.364284] mdc_intent_lock+0x353/0x7e0 [mdc] [19696.366085] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19696.368322] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19696.370731] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19696.372859] lmv_intent_open+0x338/0x1150 [lmv] [19696.374655] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19696.376803] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19696.378971] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19696.380678] ll_intent_lock+0xc5/0xaa0 [lustre] [19696.382555] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19696.384817] ll_lookup_it+0xe0e/0x2270 [lustre] [19696.386997] ? do_raw_spin_unlock+0x75/0x190 [19696.388685] ? _raw_spin_unlock+0x12/0x30 [19696.390175] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19696.392125] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19696.394200] ll_atomic_open+0x36a/0xd70 [lustre] [19696.396090] lookup_open+0x356/0xb40 [19696.397629] do_last+0x313/0xfc0 [19696.399147] ? nd_jump_root+0xe5/0x160 [19696.400420] ? path_init+0x437/0x520 [19696.401394] path_openat+0xf7/0x500 [19696.402973] ? do_raw_spin_unlock+0x75/0x190 [19696.404465] do_filp_open+0x99/0x140 [19696.405965] ? getname_flags+0x6e/0x330 [19696.407323] ? __check_object_size+0xff/0x256 [19696.408958] ? do_raw_spin_unlock+0x75/0x190 [19696.410112] ? _raw_spin_unlock+0x12/0x30 [19696.411681] do_sys_openat2+0x2b4/0x410 [19696.412986] do_sys_open+0x73/0xa0 [19696.414436] __x64_sys_openat+0x24/0x30 [19696.415624] do_syscall_64+0xc1/0x440 [19696.417063] entry_SYSCALL_64_after_hwframe+0x49/0xae [19696.419202] RIP: 0033:0x7f53425a5332 [19696.420769] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19696.427932] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19696.430798] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19696.433584] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19696.435965] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19696.438443] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19696.441293] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19704.058012] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19704.062396] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19704.083442] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19704.088347] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19704.092516] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19704.095474] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19704.104582] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19704.107937] RAX: 0000000000005092 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19704.110700] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19704.114251] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19704.118368] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19704.122546] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19704.127545] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19704.133546] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19704.137478] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19704.139948] Call Trace: [19704.140938] [19704.142122] ? show_regs.cold.9+0x22/0x2f [19704.143823] ? watchdog_timer_fn.cold.6+0x62/0x10a [19704.145333] ? __hrtimer_run_queues+0x1c2/0x4d0 [19704.146839] ? watchdog+0x40/0x40 [19704.148149] ? hrtimer_interrupt+0x128/0x2c0 [19704.149585] ? smp_apic_timer_interrupt+0xa6/0x270 [19704.151119] ? apic_timer_interrupt+0xf/0x20 [19704.152583] [19704.153242] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19704.154965] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19704.156572] do_raw_spin_lock+0x177/0x190 [19704.157900] _raw_spin_lock+0x21/0x30 [19704.159131] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19704.161640] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19704.163282] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19704.165422] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19704.167731] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19704.169648] ? woken_wake_function+0x30/0x30 [19704.170865] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19704.172849] kthread+0x1d1/0x200 [19704.173865] ? set_kthread_struct+0x70/0x70 [19704.175046] ret_from_fork+0x1f/0x30 [19720.052821] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [19720.066788] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19720.122741] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19720.138108] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19720.145448] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19720.151705] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19720.157592] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19720.173714] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19720.177961] RAX: 0000000000007f87 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19720.180937] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19720.186435] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19720.194834] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19720.198897] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19720.204520] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19720.208733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19720.211517] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19720.213541] Call Trace: [19720.214593] [19720.215331] ? show_regs.cold.9+0x22/0x2f [19720.217490] ? watchdog_timer_fn.cold.6+0x62/0x10a [19720.219482] ? __hrtimer_run_queues+0x1c2/0x4d0 [19720.221578] ? watchdog+0x40/0x40 [19720.222995] ? hrtimer_interrupt+0x128/0x2c0 [19720.224791] ? smp_apic_timer_interrupt+0xa6/0x270 [19720.227088] ? apic_timer_interrupt+0xf/0x20 [19720.228530] [19720.229174] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19720.231251] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19720.232675] ? newidle_balance.isra.93+0x207/0x580 [19720.234889] do_raw_spin_lock+0x177/0x190 [19720.237284] _raw_spin_lock+0x21/0x30 [19720.239714] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19720.244471] process_one_work+0x2c8/0x700 [19720.246523] worker_thread+0x296/0x6e0 [19720.247947] ? rescuer_thread+0x570/0x570 [19720.250066] kthread+0x1d1/0x200 [19720.251270] ? set_kthread_struct+0x70/0x70 [19720.252306] ret_from_fork+0x1f/0x30 [19722.720209] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17085s! [19722.729798] Showing busy workqueues and worker pools: [19722.733184] workqueue events: flags=0x0 [19722.734920] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19722.741383] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19722.749555] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19722.753281] pending: cpu_latency_work [ptlrpc] [19722.758683] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19722.763626] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19722.771912] workqueue events_long: flags=0x0 [19722.776739] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19722.780271] pending: lu_objects_flush [obdclass] [19722.782995] workqueue events_unbound: flags=0x2 [19722.787662] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19722.791530] pending: flush_memcg_stats_dwork, idle_cull_fn [19722.794852] workqueue events_power_efficient: flags=0x80 [19722.800881] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19722.805957] pending: check_lifetime, do_cache_clean [19722.808558] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19722.813443] pending: neigh_periodic_work, neigh_periodic_work [19722.818935] workqueue mm_percpu_wq: flags=0x8 [19722.823747] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19722.832984] pending: vmstat_update, lru_add_drain_per_cpu [19722.835322] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19722.842680] pending: vmstat_update, lru_add_drain_per_cpu [19722.845697] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19722.849511] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19722.851656] workqueue xprtiod: flags=0xa [19722.857823] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19722.860812] pending: xprt_autoclose, xs_error_handle [19722.868053] workqueue ipv6_addrconf: flags=0x40008 [19722.874679] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19722.877588] pending: addrconf_verify_work [19722.883427] workqueue knbd0-recv: flags=0x1a [19722.885539] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19722.891510] in-flight: 651:recv_work [19722.892959] workqueue ptlrpc_pinger: flags=0x2 [19722.899852] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19722.908963] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19722.913163] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17085s workers=4 idle: 15979 31824 8 [19722.922486] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19713s workers=2 idle: 654 [19724.057066] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [19724.058175] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [19724.060650] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19724.060863] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19724.060872] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19724.060875] RIP: 0010:native_safe_halt+0x2e/0x50 [19724.060889] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19724.060894] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19724.060898] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19724.060900] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19724.060902] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19724.060905] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19724.060908] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19724.060911] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19724.060914] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19724.060917] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19724.060926] Call Trace: [19724.060932] [19724.060939] ? show_regs.cold.9+0x22/0x2f [19724.060951] ? watchdog_timer_fn.cold.6+0x62/0x10a [19724.060957] ? __hrtimer_run_queues+0x1c2/0x4d0 [19724.060962] ? watchdog+0x40/0x40 [19724.060967] ? hrtimer_interrupt+0x128/0x2c0 [19724.060972] ? smp_apic_timer_interrupt+0xa6/0x270 [19724.060979] ? apic_timer_interrupt+0xf/0x20 [19724.060984] [19724.060987] ? native_safe_halt+0x2e/0x50 [19724.060991] kvm_wait+0xcd/0xe0 [19724.060998] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19724.061005] do_raw_spin_lock+0x177/0x190 [19724.061009] _raw_spin_lock+0x21/0x30 [19724.061015] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19724.061466] obd_mod_free+0x46/0x140 [ptlrpc] [19724.061677] mdc_commit_open+0x91/0xb0 [mdc] [19724.061716] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19724.061918] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19724.062134] after_reply+0xe2c/0x1780 [ptlrpc] [19724.062336] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19724.062545] ? __next_timer_interrupt+0x160/0x160 [19724.062556] ptlrpcd+0xa82/0xe40 [ptlrpc] [19724.062769] ? do_wait_intr+0xf0/0xf0 [19724.062780] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19724.062993] kthread+0x1d1/0x200 [19724.062999] ? set_kthread_struct+0x70/0x70 [19724.063003] ret_from_fork+0x1f/0x30 [19724.243767] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19724.281349] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19724.285794] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19724.288863] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19724.291194] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19724.299229] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19724.301596] RAX: 0000000000003b41 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19724.304238] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19724.306437] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19724.309397] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19724.311816] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19724.314433] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19724.317636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19724.319926] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19724.322752] Call Trace: [19724.323820] [19724.324883] ? show_regs.cold.9+0x22/0x2f [19724.326684] ? watchdog_timer_fn.cold.6+0x62/0x10a [19724.328559] ? __hrtimer_run_queues+0x1c2/0x4d0 [19724.330273] ? watchdog+0x40/0x40 [19724.331694] ? hrtimer_interrupt+0x128/0x2c0 [19724.334008] ? smp_apic_timer_interrupt+0xa6/0x270 [19724.336467] ? apic_timer_interrupt+0xf/0x20 [19724.338179] [19724.338993] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19724.344904] do_raw_spin_lock+0x177/0x190 [19724.346333] _raw_spin_lock+0x21/0x30 [19724.347930] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19724.350975] ? do_raw_spin_unlock+0x75/0x190 [19724.352921] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19724.355207] ? do_wait_intr+0xf0/0xf0 [19724.356874] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19724.359044] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19724.361555] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19724.364068] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19724.366047] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19724.368822] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19724.370484] mdc_intent_lock+0x353/0x7e0 [mdc] [19724.372698] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19724.375039] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19724.381679] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19724.384216] lmv_intent_open+0x338/0x1150 [lmv] [19724.386843] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19724.389834] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19724.392734] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19724.395101] ll_intent_lock+0xc5/0xaa0 [lustre] [19724.397272] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19724.399609] ll_lookup_it+0xe0e/0x2270 [lustre] [19724.401295] ? do_raw_spin_unlock+0x75/0x190 [19724.406806] ? _raw_spin_unlock+0x12/0x30 [19724.408971] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19724.412644] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19724.414826] ll_atomic_open+0x36a/0xd70 [lustre] [19724.417425] lookup_open+0x356/0xb40 [19724.419476] do_last+0x313/0xfc0 [19724.421353] ? nd_jump_root+0xe5/0x160 [19724.423541] ? path_init+0x437/0x520 [19724.425425] path_openat+0xf7/0x500 [19724.426810] ? do_raw_spin_unlock+0x75/0x190 [19724.428689] do_filp_open+0x99/0x140 [19724.430205] ? getname_flags+0x6e/0x330 [19724.431967] ? __check_object_size+0xff/0x256 [19724.433805] ? do_raw_spin_unlock+0x75/0x190 [19724.435492] ? _raw_spin_unlock+0x12/0x30 [19724.436918] do_sys_openat2+0x2b4/0x410 [19724.438641] do_sys_open+0x73/0xa0 [19724.439862] __x64_sys_openat+0x24/0x30 [19724.441479] do_syscall_64+0xc1/0x440 [19724.442849] entry_SYSCALL_64_after_hwframe+0x49/0xae [19724.444951] RIP: 0033:0x7f53425a5332 [19724.446379] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19724.453248] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19724.458574] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19724.461310] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19724.469069] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19724.471582] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19724.473771] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19732.064383] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19732.071199] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19732.095483] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19732.100045] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19732.104523] RIP: 0010:native_safe_halt+0x2e/0x50 [19732.106456] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19732.115546] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19732.120510] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19732.123613] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19732.127175] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19732.130112] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19732.133178] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19732.136897] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19732.140373] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19732.142386] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19732.145477] Call Trace: [19732.146532] [19732.147548] ? show_regs.cold.9+0x22/0x2f [19732.149600] ? watchdog_timer_fn.cold.6+0x62/0x10a [19732.151777] ? __hrtimer_run_queues+0x1c2/0x4d0 [19732.153846] ? watchdog+0x40/0x40 [19732.155442] ? hrtimer_interrupt+0x128/0x2c0 [19732.157001] ? smp_apic_timer_interrupt+0xa6/0x270 [19732.158921] ? apic_timer_interrupt+0xf/0x20 [19732.160461] [19732.161400] ? native_safe_halt+0x2e/0x50 [19732.163119] kvm_wait+0xcd/0xe0 [19732.164358] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19732.167696] do_raw_spin_lock+0x177/0x190 [19732.170647] _raw_spin_lock+0x21/0x30 [19732.172149] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19732.175096] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19732.177326] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19732.179990] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19732.182129] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19732.184601] ? woken_wake_function+0x30/0x30 [19732.190686] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19732.194653] kthread+0x1d1/0x200 [19732.195808] ? set_kthread_struct+0x70/0x70 [19732.204803] ret_from_fork+0x1f/0x30 [19748.056453] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [19748.061478] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19748.116361] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19748.128888] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19748.134948] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19748.139187] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19748.141567] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19748.157707] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19748.165195] RAX: 0000000000007096 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19748.173756] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19748.180095] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19748.183263] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19748.188227] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19748.192868] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19748.199493] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19748.203198] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19748.209899] Call Trace: [19748.211175] [19748.212149] ? show_regs.cold.9+0x22/0x2f [19748.213643] ? watchdog_timer_fn.cold.6+0x62/0x10a [19748.217060] ? __hrtimer_run_queues+0x1c2/0x4d0 [19748.220509] ? watchdog+0x40/0x40 [19748.222476] ? hrtimer_interrupt+0x128/0x2c0 [19748.224121] ? smp_apic_timer_interrupt+0xa6/0x270 [19748.226028] ? apic_timer_interrupt+0xf/0x20 [19748.227352] [19748.228328] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19748.230523] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19748.232911] ? newidle_balance.isra.93+0x207/0x580 [19748.234793] do_raw_spin_lock+0x177/0x190 [19748.236604] _raw_spin_lock+0x21/0x30 [19748.239177] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19748.241477] process_one_work+0x2c8/0x700 [19748.247624] worker_thread+0x296/0x6e0 [19748.249121] ? rescuer_thread+0x570/0x570 [19748.251420] kthread+0x1d1/0x200 [19748.252826] ? set_kthread_struct+0x70/0x70 [19748.254487] ret_from_fork+0x1f/0x30 [19752.057861] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [19752.062189] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [19752.067146] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19752.067408] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19752.067415] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19752.067417] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19752.067431] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19752.067436] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19752.067440] RAX: 0000000000002989 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19752.067443] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19752.067445] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19752.067447] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19752.067449] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19752.067452] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19752.067637] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19752.067639] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19752.067647] Call Trace: [19752.067653] [19752.067659] ? show_regs.cold.9+0x22/0x2f [19752.067812] ? watchdog_timer_fn.cold.6+0x62/0x10a [19752.067819] ? __hrtimer_run_queues+0x1c2/0x4d0 [19752.067824] ? watchdog+0x40/0x40 [19752.067829] ? hrtimer_interrupt+0x128/0x2c0 [19752.067833] ? smp_apic_timer_interrupt+0xa6/0x270 [19752.067844] ? apic_timer_interrupt+0xf/0x20 [19752.067849] [19752.067853] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19752.067860] do_raw_spin_lock+0x177/0x190 [19752.067865] _raw_spin_lock+0x21/0x30 [19752.067871] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19752.068129] ? do_raw_spin_unlock+0x75/0x190 [19752.068135] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19752.069794] ? do_wait_intr+0xf0/0xf0 [19752.069806] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19752.075578] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19752.075662] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19752.075670] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19752.075674] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19752.075688] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19752.075692] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19752.075697] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19752.075700] RDX: 00000000000021ea RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19752.075702] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19752.075704] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19752.075707] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19752.075709] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19752.075712] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19752.075714] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19752.075722] Call Trace: [19752.075727] [19752.075733] ? show_regs.cold.9+0x22/0x2f [19752.075742] ? watchdog_timer_fn.cold.6+0x62/0x10a [19752.075748] ? __hrtimer_run_queues+0x1c2/0x4d0 [19752.075752] ? watchdog+0x40/0x40 [19752.075756] ? hrtimer_interrupt+0x128/0x2c0 [19752.075760] ? smp_apic_timer_interrupt+0xa6/0x270 [19752.075768] ? apic_timer_interrupt+0xf/0x20 [19752.075772] [19752.075775] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19752.075779] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19752.075784] do_raw_spin_lock+0x177/0x190 [19752.075788] _raw_spin_lock+0x21/0x30 [19752.075794] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19752.076224] obd_mod_free+0x46/0x140 [ptlrpc] [19752.076422] mdc_commit_open+0x91/0xb0 [mdc] [19752.076456] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19752.076884] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19752.077098] after_reply+0xe2c/0x1780 [ptlrpc] [19752.077294] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19752.077478] ? __next_timer_interrupt+0x160/0x160 [19752.077503] ptlrpcd+0xa82/0xe40 [ptlrpc] [19752.077708] ? do_wait_intr+0xf0/0xf0 [19752.077717] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19752.077915] kthread+0x1d1/0x200 [19752.077921] ? set_kthread_struct+0x70/0x70 [19752.077924] ret_from_fork+0x1f/0x30 [19752.367421] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19752.370524] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19752.375046] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19752.378062] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19752.380512] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19752.382257] mdc_intent_lock+0x353/0x7e0 [mdc] [19752.384341] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19752.385882] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19752.391194] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19752.395864] lmv_intent_open+0x338/0x1150 [lmv] [19752.398068] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19752.400176] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19752.402733] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19752.404923] ll_intent_lock+0xc5/0xaa0 [lustre] [19752.407034] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19752.409353] ll_lookup_it+0xe0e/0x2270 [lustre] [19752.412041] ? do_raw_spin_unlock+0x75/0x190 [19752.414893] ? _raw_spin_unlock+0x12/0x30 [19752.416505] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19752.419725] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19752.421960] ll_atomic_open+0x36a/0xd70 [lustre] [19752.423999] lookup_open+0x356/0xb40 [19752.425610] do_last+0x313/0xfc0 [19752.427115] ? nd_jump_root+0xe5/0x160 [19752.429844] ? path_init+0x437/0x520 [19752.431447] path_openat+0xf7/0x500 [19752.433027] ? do_raw_spin_unlock+0x75/0x190 [19752.436395] do_filp_open+0x99/0x140 [19752.438203] ? getname_flags+0x6e/0x330 [19752.441476] ? __check_object_size+0xff/0x256 [19752.444367] ? do_raw_spin_unlock+0x75/0x190 [19752.446262] ? _raw_spin_unlock+0x12/0x30 [19752.447496] do_sys_openat2+0x2b4/0x410 [19752.449654] do_sys_open+0x73/0xa0 [19752.451897] __x64_sys_openat+0x24/0x30 [19752.456562] do_syscall_64+0xc1/0x440 [19752.462004] entry_SYSCALL_64_after_hwframe+0x49/0xae [19752.464933] RIP: 0033:0x7f53425a5332 [19752.467016] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19752.480411] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19752.484112] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19752.487201] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19752.490101] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19752.494712] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19752.497905] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19753.440105] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17116s! [19753.443818] Showing busy workqueues and worker pools: [19753.447733] workqueue events: flags=0x0 [19753.449925] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19753.453833] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19753.477048] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19753.487855] pending: cpu_latency_work [ptlrpc] [19753.494834] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19753.497995] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19753.506990] workqueue events_long: flags=0x0 [19753.512215] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19753.519299] pending: lu_objects_flush [obdclass] [19753.526544] workqueue events_unbound: flags=0x2 [19753.533140] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19753.536403] pending: flush_memcg_stats_dwork, idle_cull_fn [19753.546459] workqueue events_power_efficient: flags=0x80 [19753.548493] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19753.559524] pending: check_lifetime, do_cache_clean [19753.565471] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19753.570785] pending: neigh_periodic_work, neigh_periodic_work [19753.574516] workqueue mm_percpu_wq: flags=0x8 [19753.576492] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19753.584289] pending: vmstat_update, lru_add_drain_per_cpu [19753.587246] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19753.591002] pending: vmstat_update, lru_add_drain_per_cpu [19753.594096] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19753.607147] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19753.613191] workqueue xprtiod: flags=0xa [19753.615961] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19753.618270] pending: xprt_autoclose, xs_error_handle [19753.621995] workqueue ipv6_addrconf: flags=0x40008 [19753.626991] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19753.629935] pending: addrconf_verify_work [19753.631872] workqueue knbd0-recv: flags=0x1a [19753.634199] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19753.639354] in-flight: 651:recv_work [19753.641314] workqueue ptlrpc_pinger: flags=0x2 [19753.645263] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19753.648106] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19753.652895] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17116s workers=4 idle: 15979 31824 8 [19753.656481] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19744s workers=2 idle: 654 [19760.058030] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19760.061020] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19760.079720] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19760.085107] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19760.088937] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19760.090872] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19760.098135] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19760.100404] RAX: 0000000000006317 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19760.102786] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19760.105692] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19760.108715] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19760.111578] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19760.113811] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19760.115991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19760.117988] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19760.120861] Call Trace: [19760.121723] [19760.122430] ? show_regs.cold.9+0x22/0x2f [19760.123812] ? watchdog_timer_fn.cold.6+0x62/0x10a [19760.125657] ? __hrtimer_run_queues+0x1c2/0x4d0 [19760.127109] ? watchdog+0x40/0x40 [19760.128048] ? hrtimer_interrupt+0x128/0x2c0 [19760.129173] ? smp_apic_timer_interrupt+0xa6/0x270 [19760.130418] ? apic_timer_interrupt+0xf/0x20 [19760.131660] [19760.132302] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19760.133872] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19760.136138] do_raw_spin_lock+0x177/0x190 [19760.137918] _raw_spin_lock+0x21/0x30 [19760.139428] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19760.142244] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19760.144795] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19760.147505] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19760.150517] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19760.153012] ? woken_wake_function+0x30/0x30 [19760.154400] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19760.156761] kthread+0x1d1/0x200 [19760.158002] ? set_kthread_struct+0x70/0x70 [19760.159700] ret_from_fork+0x1f/0x30 [19780.052019] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [19780.056786] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [19780.062997] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [19780.063011] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19780.063080] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19780.063086] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19780.063089] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19780.063112] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19780.063116] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19780.063120] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19780.063123] RDX: 0000000000001981 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19780.063126] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19780.063127] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19780.063130] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19780.063132] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19780.063135] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19780.063137] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19780.063146] Call Trace: [19780.063152] [19780.063158] ? show_regs.cold.9+0x22/0x2f [19780.063170] ? watchdog_timer_fn.cold.6+0x62/0x10a [19780.063179] ? __hrtimer_run_queues+0x1c2/0x4d0 [19780.063184] ? watchdog+0x40/0x40 [19780.063189] ? hrtimer_interrupt+0x128/0x2c0 [19780.063193] ? smp_apic_timer_interrupt+0xa6/0x270 [19780.063203] ? apic_timer_interrupt+0xf/0x20 [19780.063220] [19780.063224] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19780.063229] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19780.063234] do_raw_spin_lock+0x177/0x190 [19780.063239] _raw_spin_lock+0x21/0x30 [19780.063247] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19780.063511] obd_mod_free+0x46/0x140 [ptlrpc] [19780.063710] mdc_commit_open+0x91/0xb0 [mdc] [19780.063761] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19780.063957] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19780.064173] after_reply+0xe2c/0x1780 [ptlrpc] [19780.064378] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19780.064579] ? __next_timer_interrupt+0x160/0x160 [19780.064589] ptlrpcd+0xa82/0xe40 [ptlrpc] [19780.064793] ? do_wait_intr+0xf0/0xf0 [19780.064802] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19780.065007] kthread+0x1d1/0x200 [19780.065014] ? set_kthread_struct+0x70/0x70 [19780.065018] ret_from_fork+0x1f/0x30 [19780.068142] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) [19780.087531] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19780.087607] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19780.087614] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19780.087617] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19780.087631] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19780.087635] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19780.087640] RAX: 0000000000000085 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19780.087649] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19780.087651] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19780.087653] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19780.087655] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19780.087658] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19780.087661] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19780.087663] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19780.087671] Call Trace: [19780.087680] [19780.087686] ? show_regs.cold.9+0x22/0x2f [19780.087696] ? watchdog_timer_fn.cold.6+0x62/0x10a [19780.087702] ? __hrtimer_run_queues+0x1c2/0x4d0 [19780.087706] ? watchdog+0x40/0x40 [19780.087711] ? hrtimer_interrupt+0x128/0x2c0 [19780.087715] ? smp_apic_timer_interrupt+0xa6/0x270 [19780.087723] ? apic_timer_interrupt+0xf/0x20 [19780.087732] [19780.087735] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19780.087742] do_raw_spin_lock+0x177/0x190 [19780.087747] _raw_spin_lock+0x21/0x30 [19780.087752] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19780.088035] ? do_raw_spin_unlock+0x75/0x190 [19780.088041] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19780.088490] ? do_wait_intr+0xf0/0xf0 [19780.088499] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19780.088794] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19780.089300] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19780.100946] obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19780.100999] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19780.101007] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19780.101014] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19780.102459] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19780.102481] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19780.102486] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19780.102491] RAX: 0000000000004401 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19780.102493] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19780.102495] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19780.102497] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19780.102500] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19780.102503] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19780.102516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19780.102519] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19780.102527] Call Trace: [19780.102533] [19780.102539] ? show_regs.cold.9+0x22/0x2f [19780.102549] ? watchdog_timer_fn.cold.6+0x62/0x10a [19780.102562] ? __hrtimer_run_queues+0x1c2/0x4d0 [19780.102567] ? watchdog+0x40/0x40 [19780.102572] ? hrtimer_interrupt+0x128/0x2c0 [19780.102577] ? smp_apic_timer_interrupt+0xa6/0x270 [19780.102585] ? apic_timer_interrupt+0xf/0x20 [19780.102590] [19780.102593] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19780.102598] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19780.102603] ? newidle_balance.isra.93+0x207/0x580 [19780.102608] do_raw_spin_lock+0x177/0x190 [19780.102612] _raw_spin_lock+0x21/0x30 [19780.102617] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19780.151487] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19780.151615] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19780.151655] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19780.151690] mdc_intent_lock+0x353/0x7e0 [mdc] [19780.151721] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19780.151803] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19780.152074] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19780.152526] lmv_intent_open+0x338/0x1150 [lmv] [19780.152559] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19780.152660] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19780.152743] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19780.152766] ll_intent_lock+0xc5/0xaa0 [lustre] [19780.152851] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19780.152934] ll_lookup_it+0xe0e/0x2270 [lustre] [19780.161404] process_one_work+0x2c8/0x700 [19780.161426] worker_thread+0x296/0x6e0 [19780.161431] ? rescuer_thread+0x570/0x570 [19780.161435] kthread+0x1d1/0x200 [19780.161440] ? set_kthread_struct+0x70/0x70 [19780.161444] ret_from_fork+0x1f/0x30 [19781.093997] ? do_raw_spin_unlock+0x75/0x190 [19781.095857] ? _raw_spin_unlock+0x12/0x30 [19781.100457] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19781.105066] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19781.115182] ll_atomic_open+0x36a/0xd70 [lustre] [19781.121763] lookup_open+0x356/0xb40 [19781.124664] do_last+0x313/0xfc0 [19781.135260] ? nd_jump_root+0xe5/0x160 [19781.141164] ? path_init+0x437/0x520 [19781.142471] path_openat+0xf7/0x500 [19781.159921] ? do_raw_spin_unlock+0x75/0x190 [19781.164710] do_filp_open+0x99/0x140 [19781.169878] ? getname_flags+0x6e/0x330 [19781.183417] ? __check_object_size+0xff/0x256 [19781.188320] ? do_raw_spin_unlock+0x75/0x190 [19781.194469] ? _raw_spin_unlock+0x12/0x30 [19781.200178] do_sys_openat2+0x2b4/0x410 [19781.205664] do_sys_open+0x73/0xa0 [19781.206898] __x64_sys_openat+0x24/0x30 [19781.214049] do_syscall_64+0xc1/0x440 [19781.221896] entry_SYSCALL_64_after_hwframe+0x49/0xae [19781.230597] RIP: 0033:0x7f53425a5332 [19781.238024] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19781.266700] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19781.280784] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19781.290022] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19781.299383] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19781.311711] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19781.319936] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19784.169889] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17147s! [19784.179964] Showing busy workqueues and worker pools: [19784.186934] workqueue events: flags=0x0 [19784.189188] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19784.196998] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19784.226900] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19784.238684] pending: cpu_latency_work [ptlrpc] [19784.242727] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19784.247363] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19784.253287] workqueue events_long: flags=0x0 [19784.264966] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19784.278472] pending: lu_objects_flush [obdclass] [19784.280681] workqueue events_unbound: flags=0x2 [19784.285277] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19784.295404] pending: flush_memcg_stats_dwork, idle_cull_fn [19784.302471] workqueue events_power_efficient: flags=0x80 [19784.323700] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19784.332227] pending: check_lifetime, do_cache_clean [19784.337322] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19784.346502] pending: neigh_periodic_work, neigh_periodic_work [19784.363200] workqueue mm_percpu_wq: flags=0x8 [19784.372159] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19784.377534] pending: vmstat_update, lru_add_drain_per_cpu [19784.386737] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19784.398818] pending: vmstat_update, lru_add_drain_per_cpu [19784.411449] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19784.425544] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19784.439086] workqueue xprtiod: flags=0xa [19784.447530] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19784.451442] pending: xprt_autoclose, xs_error_handle [19784.453982] workqueue ipv6_addrconf: flags=0x40008 [19784.459109] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19784.475632] pending: addrconf_verify_work [19784.490677] workqueue knbd0-recv: flags=0x1a [19784.498686] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19784.506270] in-flight: 651:recv_work [19784.507711] workqueue ptlrpc_pinger: flags=0x2 [19784.511777] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19784.529394] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19784.538109] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17147s workers=4 idle: 15979 31824 8 [19784.540973] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19774s workers=2 idle: 654 [19788.062123] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [19788.066465] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19788.118868] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19788.124763] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19788.132523] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19788.136502] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19788.157020] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19788.163678] RAX: 0000000000006a9c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19788.170305] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19788.175674] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19788.180323] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19788.182696] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19788.197236] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19788.202711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19788.208508] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19788.219295] Call Trace: [19788.222095] [19788.222659] ? show_regs.cold.9+0x22/0x2f [19788.223688] ? watchdog_timer_fn.cold.6+0x62/0x10a [19788.225716] ? __hrtimer_run_queues+0x1c2/0x4d0 [19788.227146] ? watchdog+0x40/0x40 [19788.229052] ? hrtimer_interrupt+0x128/0x2c0 [19788.230525] ? smp_apic_timer_interrupt+0xa6/0x270 [19788.232664] ? apic_timer_interrupt+0xf/0x20 [19788.234554] [19788.236417] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19788.239147] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19788.241146] do_raw_spin_lock+0x177/0x190 [19788.242760] _raw_spin_lock+0x21/0x30 [19788.244682] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19788.250034] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19788.252829] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19788.256515] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19788.261535] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19788.267422] ? woken_wake_function+0x30/0x30 [19788.268921] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19788.272550] kthread+0x1d1/0x200 [19788.273924] ? set_kthread_struct+0x70/0x70 [19788.275918] ret_from_fork+0x1f/0x30 [19794.866594] rcu: INFO: rcu_sched self-detected stall on CPU [19794.877558] rcu: 2-....: (10009304 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [19794.887227] (t=17700440 jiffies g=274185 q=68437) [19794.891134] rcu: rcu_sched kthread starved for 17158806 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [19794.903293] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [19794.911074] rcu: RCU grace-period kthread stack dump: [19794.917765] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [19794.924548] Call Trace: [19794.926019] __schedule+0x351/0xcb0 [19794.928256] schedule+0xc0/0x180 [19794.931359] schedule_timeout+0xb4/0x190 [19794.935733] ? do_raw_spin_unlock+0x75/0x190 [19794.939196] ? __next_timer_interrupt+0x160/0x160 [19794.940807] ? prepare_to_swait_event+0xf3/0x140 [19794.947296] rcu_gp_kthread+0x8dd/0x15b0 [19794.950342] ? force_qs_rnp+0x2c0/0x2c0 [19794.952353] kthread+0x1d1/0x200 [19794.956341] ? set_kthread_struct+0x70/0x70 [19794.959005] ret_from_fork+0x1f/0x30 [19794.961164] rcu: Stack dump where RCU GP kthread last ran: [19794.964672] Sending NMI from CPU 2 to CPUs 0: [19794.968706] NMI backtrace for cpu 0 [19794.968712] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19794.968714] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19794.968716] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19794.968719] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19794.968720] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19794.968723] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 [19794.968724] RAX: 0000000000004b04 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19794.968726] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19794.968727] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19794.968728] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19794.968730] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19794.968731] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19794.968733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19794.968735] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19794.968736] Call Trace: [19794.968737] [19794.968738] ? show_regs.cold.9+0x22/0x2f [19794.968739] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [19794.968746] ? nmi_cpu_backtrace_handler+0x15/0x40 [19794.968747] ? nmi_handle+0xbd/0x210 [19794.968748] ? default_do_nmi+0x4b/0x1b0 [19794.968749] ? do_nmi+0x280/0x380 [19794.968750] ? end_repeat_nmi+0x16/0x1f [19794.968752] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19794.968753] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19794.968754] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19794.968755] [19794.968756] ? newidle_balance.isra.93+0x207/0x580 [19794.968757] do_raw_spin_lock+0x177/0x190 [19794.968758] _raw_spin_lock+0x21/0x30 [19794.968759] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19794.968760] process_one_work+0x2c8/0x700 [19794.968762] worker_thread+0x296/0x6e0 [19794.968763] ? rescuer_thread+0x570/0x570 [19794.968764] kthread+0x1d1/0x200 [19794.968765] ? set_kthread_struct+0x70/0x70 [19794.968767] ret_from_fork+0x1f/0x30 [19794.971095] Sending NMI from CPU 2 to CPUs 1: [19795.050209] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [19795.051166] NMI backtrace for cpu 2 [19795.058232] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19795.062048] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19795.065374] Call Trace: [19795.066101] [19795.066625] ? dump_stack+0xbb/0x10e [19795.067654] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [19795.069381] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [19795.071477] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [19795.073759] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [19795.075802] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [19795.078403] ? rcu_dump_cpu_stacks+0x11b/0x171 [19795.079616] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [19795.081818] ? update_process_times+0x6d/0xc0 [19795.085390] ? tick_sched_handle.isra.18+0x27/0x90 [19795.087632] ? tick_sched_timer+0x53/0xb0 [19795.089315] ? __hrtimer_run_queues+0x1c2/0x4d0 [19795.097077] ? tick_sched_handle.isra.18+0x90/0x90 [19795.098731] ? hrtimer_interrupt+0x128/0x2c0 [19795.100241] ? smp_apic_timer_interrupt+0xa6/0x270 [19795.107349] ? apic_timer_interrupt+0xf/0x20 [19795.109544] [19795.110363] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19795.117441] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19795.121373] ? do_raw_spin_lock+0x177/0x190 [19795.127651] ? _raw_spin_lock+0x21/0x30 [19795.128881] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19795.130746] ? obd_mod_free+0x46/0x140 [ptlrpc] [19795.136792] ? mdc_commit_open+0x91/0xb0 [mdc] [19795.143841] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19795.147682] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19795.154719] ? after_reply+0xe2c/0x1780 [ptlrpc] [19795.157680] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19795.164604] ? __next_timer_interrupt+0x160/0x160 [19795.167414] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [19795.170265] ? do_wait_intr+0xf0/0xf0 [19795.171893] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19795.175072] ? kthread+0x1d1/0x200 [19795.176387] ? set_kthread_struct+0x70/0x70 [19795.178512] ? ret_from_fork+0x1f/0x30 [19814.880353] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17177s! [19814.882931] Showing busy workqueues and worker pools: [19814.884207] workqueue events: flags=0x0 [19814.885257] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19814.887060] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19814.890378] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19814.892333] pending: cpu_latency_work [ptlrpc] [19814.894141] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19814.896044] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19814.899014] workqueue events_long: flags=0x0 [19814.900130] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19814.901948] pending: lu_objects_flush [obdclass] [19814.903316] workqueue events_unbound: flags=0x2 [19814.904321] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19814.907324] pending: flush_memcg_stats_dwork, idle_cull_fn [19814.909792] workqueue events_power_efficient: flags=0x80 [19814.911822] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19814.913859] pending: check_lifetime, do_cache_clean [19814.917030] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19814.921506] pending: neigh_periodic_work, neigh_periodic_work [19814.923827] workqueue mm_percpu_wq: flags=0x8 [19814.928345] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19814.932352] pending: vmstat_update, lru_add_drain_per_cpu [19814.934585] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19814.937990] pending: vmstat_update, lru_add_drain_per_cpu [19814.941679] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19814.946056] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19814.951331] workqueue xprtiod: flags=0xa [19814.957027] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19814.962375] pending: xprt_autoclose, xs_error_handle [19814.964303] workqueue ipv6_addrconf: flags=0x40008 [19814.965426] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19814.971025] pending: addrconf_verify_work [19814.974129] workqueue knbd0-recv: flags=0x1a [19814.976476] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19814.984481] in-flight: 651:recv_work [19814.990892] workqueue ptlrpc_pinger: flags=0x2 [19814.992923] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19814.999483] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19815.003410] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17177s workers=4 idle: 15979 31824 8 [19815.010543] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19805s workers=2 idle: 654 [19816.058073] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [19816.061118] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19816.080722] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19816.086043] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19816.090115] RIP: 0010:native_safe_halt+0x2e/0x50 [19816.092114] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19816.100906] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19816.104276] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19816.107075] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19816.109897] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19816.113855] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19816.117727] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19816.120657] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19816.124434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19816.127253] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19816.131257] Call Trace: [19816.132407] [19816.133448] ? show_regs.cold.9+0x22/0x2f [19816.135567] ? watchdog_timer_fn.cold.6+0x62/0x10a [19816.137767] ? __hrtimer_run_queues+0x1c2/0x4d0 [19816.140104] ? watchdog+0x40/0x40 [19816.141768] ? hrtimer_interrupt+0x128/0x2c0 [19816.143659] ? smp_apic_timer_interrupt+0xa6/0x270 [19816.145698] ? apic_timer_interrupt+0xf/0x20 [19816.147834] [19816.148793] ? native_safe_halt+0x2e/0x50 [19816.150634] kvm_wait+0xcd/0xe0 [19816.151958] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19816.155674] do_raw_spin_lock+0x177/0x190 [19816.157417] _raw_spin_lock+0x21/0x30 [19816.159479] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19816.162929] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19816.165943] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19816.169421] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19816.172521] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19816.175389] ? woken_wake_function+0x30/0x30 [19816.177659] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19816.180151] kthread+0x1d1/0x200 [19816.181859] ? set_kthread_struct+0x70/0x70 [19816.186596] ret_from_fork+0x1f/0x30 [19820.055027] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [19820.057024] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [19820.057993] Modules linked in: [19820.060215] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19820.061933] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19820.061941] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19820.061944] RIP: 0010:native_safe_halt+0x2e/0x50 [19820.061955] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19820.061960] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19820.061964] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19820.061967] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19820.061970] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19820.061972] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19820.061975] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19820.061978] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19820.061981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19820.061983] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19820.061992] Call Trace: [19820.061997] [19820.062002] ? show_regs.cold.9+0x22/0x2f [19820.062012] ? watchdog_timer_fn.cold.6+0x62/0x10a [19820.062025] ? __hrtimer_run_queues+0x1c2/0x4d0 [19820.062030] ? watchdog+0x40/0x40 [19820.062034] ? hrtimer_interrupt+0x128/0x2c0 [19820.062039] ? smp_apic_timer_interrupt+0xa6/0x270 [19820.062045] ? apic_timer_interrupt+0xf/0x20 [19820.062050] [19820.062053] ? native_safe_halt+0x2e/0x50 [19820.062057] kvm_wait+0xcd/0xe0 [19820.062065] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19820.062072] do_raw_spin_lock+0x177/0x190 [19820.062077] _raw_spin_lock+0x21/0x30 [19820.062083] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19820.062366] obd_mod_free+0x46/0x140 [ptlrpc] [19820.063071] mdc_commit_open+0x91/0xb0 [mdc] [19820.063107] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19820.063312] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19820.063515] after_reply+0xe2c/0x1780 [ptlrpc] [19820.065177] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19820.065260] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19820.065267] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19820.065270] RIP: 0010:native_safe_halt+0x2e/0x50 [19820.087769] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19820.091364] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19820.091376] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19820.091383] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19820.091385] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19820.091388] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19820.091390] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19820.091392] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19820.091609] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19820.091613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19820.091616] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19820.091624] Call Trace: [19820.091630] [19820.091636] ? show_regs.cold.9+0x22/0x2f [19820.091648] ? watchdog_timer_fn.cold.6+0x62/0x10a [19820.091655] ? __hrtimer_run_queues+0x1c2/0x4d0 [19820.091659] ? watchdog+0x40/0x40 [19820.091664] ? hrtimer_interrupt+0x128/0x2c0 [19820.091668] ? smp_apic_timer_interrupt+0xa6/0x270 [19820.091676] ? apic_timer_interrupt+0xf/0x20 [19820.091680] [19820.091682] ? native_safe_halt+0x2e/0x50 [19820.091687] kvm_wait+0xcd/0xe0 [19820.091693] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19820.091702] do_raw_spin_lock+0x177/0x190 [19820.091706] _raw_spin_lock+0x21/0x30 [19820.091710] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19820.092087] ? do_raw_spin_unlock+0x75/0x190 [19820.092093] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19820.092288] ? do_wait_intr+0xf0/0xf0 [19820.092296] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19820.092596] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19820.097574] ? __next_timer_interrupt+0x160/0x160 [19820.097597] ptlrpcd+0xa82/0xe40 [ptlrpc] [19820.097921] ? do_wait_intr+0xf0/0xf0 [19820.097930] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19820.100073] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19820.100330] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19820.101466] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19820.101509] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19820.101540] mdc_intent_lock+0x353/0x7e0 [mdc] [19820.101569] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19820.101656] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19820.102743] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19820.102783] lmv_intent_open+0x338/0x1150 [lmv] [19820.102815] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19820.102917] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19820.103015] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19820.103036] ll_intent_lock+0xc5/0xaa0 [lustre] [19820.103123] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19820.103211] ll_lookup_it+0xe0e/0x2270 [lustre] [19820.103296] ? do_raw_spin_unlock+0x75/0x190 [19820.103304] ? _raw_spin_unlock+0x12/0x30 [19820.103310] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19820.110451] kthread+0x1d1/0x200 [19820.110470] ? set_kthread_struct+0x70/0x70 [19820.110474] ret_from_fork+0x1f/0x30 [19820.318775] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19820.320370] ll_atomic_open+0x36a/0xd70 [lustre] [19820.321804] lookup_open+0x356/0xb40 [19820.322943] do_last+0x313/0xfc0 [19820.323816] ? nd_jump_root+0xe5/0x160 [19820.324904] ? path_init+0x437/0x520 [19820.326084] path_openat+0xf7/0x500 [19820.327122] ? do_raw_spin_unlock+0x75/0x190 [19820.328311] do_filp_open+0x99/0x140 [19820.329504] ? getname_flags+0x6e/0x330 [19820.330572] ? __check_object_size+0xff/0x256 [19820.331810] ? do_raw_spin_unlock+0x75/0x190 [19820.333274] ? _raw_spin_unlock+0x12/0x30 [19820.334773] do_sys_openat2+0x2b4/0x410 [19820.336085] do_sys_open+0x73/0xa0 [19820.337026] __x64_sys_openat+0x24/0x30 [19820.338135] do_syscall_64+0xc1/0x440 [19820.339483] entry_SYSCALL_64_after_hwframe+0x49/0xae [19820.340925] RIP: 0033:0x7f53425a5332 [19820.342070] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19820.348970] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19820.351178] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19820.352997] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19820.355050] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19820.356981] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19820.359038] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19840.052582] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [19840.055277] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19840.077191] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19840.089079] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19840.096700] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19840.105606] RIP: 0010:native_safe_halt+0x2e/0x50 [19840.110544] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19840.120735] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19840.123538] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19840.126250] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [19840.129379] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19840.132600] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19840.135566] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19840.138637] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19840.141850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19840.144228] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19840.146960] Call Trace: [19840.147889] [19840.148680] ? show_regs.cold.9+0x22/0x2f [19840.149911] ? watchdog_timer_fn.cold.6+0x62/0x10a [19840.151647] ? __hrtimer_run_queues+0x1c2/0x4d0 [19840.153117] ? watchdog+0x40/0x40 [19840.154373] ? hrtimer_interrupt+0x128/0x2c0 [19840.156407] ? smp_apic_timer_interrupt+0xa6/0x270 [19840.158642] ? apic_timer_interrupt+0xf/0x20 [19840.159703] [19840.160266] ? native_safe_halt+0x2e/0x50 [19840.161447] kvm_wait+0xcd/0xe0 [19840.162487] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19840.164504] ? newidle_balance.isra.93+0x207/0x580 [19840.166500] do_raw_spin_lock+0x177/0x190 [19840.167714] _raw_spin_lock+0x21/0x30 [19840.168926] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19840.172584] process_one_work+0x2c8/0x700 [19840.174014] worker_thread+0x296/0x6e0 [19840.175583] ? rescuer_thread+0x570/0x570 [19840.176835] kthread+0x1d1/0x200 [19840.177804] ? set_kthread_struct+0x70/0x70 [19840.179460] ret_from_fork+0x1f/0x30 [19844.059217] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [19844.061691] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19844.078500] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19844.083493] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19844.086660] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19844.088422] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19844.097708] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19844.100525] RAX: 0000000000006486 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19844.103690] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19844.113498] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19844.115898] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19844.118615] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19844.122132] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19844.124082] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19844.126513] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19844.135060] Call Trace: [19844.140864] [19844.142208] ? show_regs.cold.9+0x22/0x2f [19844.145707] ? watchdog_timer_fn.cold.6+0x62/0x10a [19844.147773] ? __hrtimer_run_queues+0x1c2/0x4d0 [19844.149909] ? watchdog+0x40/0x40 [19844.151663] ? hrtimer_interrupt+0x128/0x2c0 [19844.155478] ? smp_apic_timer_interrupt+0xa6/0x270 [19844.162451] ? apic_timer_interrupt+0xf/0x20 [19844.164336] [19844.164793] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19844.168639] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19844.171967] do_raw_spin_lock+0x177/0x190 [19844.173889] _raw_spin_lock+0x21/0x30 [19844.180346] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19844.183493] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19844.187641] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19844.192056] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19844.196004] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19844.199410] ? woken_wake_function+0x30/0x30 [19844.201752] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19844.205404] kthread+0x1d1/0x200 [19844.207819] ? set_kthread_struct+0x70/0x70 [19844.209891] ret_from_fork+0x1f/0x30 [19845.088459] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17207s! [19845.096629] Showing busy workqueues and worker pools: [19845.107871] workqueue events: flags=0x0 [19845.114207] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19845.123030] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19845.134859] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19845.143105] pending: cpu_latency_work [ptlrpc] [19845.159554] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19845.169784] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19845.182915] workqueue events_long: flags=0x0 [19845.184904] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19845.197882] pending: lu_objects_flush [obdclass] [19845.199773] workqueue events_unbound: flags=0x2 [19845.202089] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19845.204362] pending: flush_memcg_stats_dwork, idle_cull_fn [19845.206619] workqueue events_power_efficient: flags=0x80 [19845.208753] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19845.212235] pending: check_lifetime, do_cache_clean [19845.214457] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19845.216928] pending: neigh_periodic_work, neigh_periodic_work [19845.219268] workqueue mm_percpu_wq: flags=0x8 [19845.221033] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19845.223983] pending: vmstat_update, lru_add_drain_per_cpu [19845.225815] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19845.242613] pending: vmstat_update, lru_add_drain_per_cpu [19845.245868] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19845.249020] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19845.252251] workqueue xprtiod: flags=0xa [19845.254053] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19845.259617] pending: xprt_autoclose, xs_error_handle [19845.263491] workqueue ipv6_addrconf: flags=0x40008 [19845.266595] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19845.271178] pending: addrconf_verify_work [19845.273934] workqueue knbd0-recv: flags=0x1a [19845.275774] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19845.278600] in-flight: 651:recv_work [19845.279698] workqueue ptlrpc_pinger: flags=0x2 [19845.281244] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19845.292208] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19845.295818] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17208s workers=4 idle: 15979 31824 8 [19845.302308] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19835s workers=2 idle: 654 [19848.055143] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [19848.057108] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [19848.058865] Modules linked in: lustre(O) [19848.061496] Modules linked in: [19848.061497] osp(O) ofd(O) lod(O) [19848.064091] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic [19848.065176] mdt(O) [19848.066056] crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19848.066076] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19848.066083] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19848.066086] RIP: 0010:native_safe_halt+0x2e/0x50 [19848.066099] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19848.066103] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19848.066107] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19848.066110] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19848.066112] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19848.066114] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19848.066117] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19848.066121] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19848.086864] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19848.087006] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19848.087014] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19848.087016] RIP: 0010:native_safe_halt+0x2e/0x50 [19848.087029] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19848.087033] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19848.087038] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19848.087040] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19848.087042] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19848.087044] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19848.087047] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19848.087050] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19848.087053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19848.087055] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19848.087065] Call Trace: [19848.087071] [19848.087077] ? show_regs.cold.9+0x22/0x2f [19848.087086] ? watchdog_timer_fn.cold.6+0x62/0x10a [19848.087092] ? __hrtimer_run_queues+0x1c2/0x4d0 [19848.087096] ? watchdog+0x40/0x40 [19848.087100] ? hrtimer_interrupt+0x128/0x2c0 [19848.087103] ? smp_apic_timer_interrupt+0xa6/0x270 [19848.087110] ? apic_timer_interrupt+0xf/0x20 [19848.087114] [19848.087118] ? native_safe_halt+0x2e/0x50 [19848.087121] kvm_wait+0xcd/0xe0 [19848.087129] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19848.087138] do_raw_spin_lock+0x177/0x190 [19848.087143] _raw_spin_lock+0x21/0x30 [19848.087147] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19848.087644] ? do_raw_spin_unlock+0x75/0x190 [19848.087650] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19848.088085] ? do_wait_intr+0xf0/0xf0 [19848.088094] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19848.091491] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19848.103118] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19848.103608] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19848.103837] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19848.104079] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19848.104120] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19848.104149] mdc_intent_lock+0x353/0x7e0 [mdc] [19848.104179] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19848.104509] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19848.104880] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19848.105046] lmv_intent_open+0x338/0x1150 [lmv] [19848.105077] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19848.105192] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19848.120509] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19848.120528] Call Trace: [19848.120536] [19848.120553] ? show_regs.cold.9+0x22/0x2f [19848.120569] ? watchdog_timer_fn.cold.6+0x62/0x10a [19848.120577] ? __hrtimer_run_queues+0x1c2/0x4d0 [19848.120581] ? watchdog+0x40/0x40 [19848.120585] ? hrtimer_interrupt+0x128/0x2c0 [19848.120590] ? smp_apic_timer_interrupt+0xa6/0x270 [19848.120598] ? apic_timer_interrupt+0xf/0x20 [19848.120604] [19848.120607] ? native_safe_halt+0x2e/0x50 [19848.120611] kvm_wait+0xcd/0xe0 [19848.120619] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19848.120627] do_raw_spin_lock+0x177/0x190 [19848.120632] _raw_spin_lock+0x21/0x30 [19848.120638] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19848.121328] obd_mod_free+0x46/0x140 [ptlrpc] [19848.121654] mdc_commit_open+0x91/0xb0 [mdc] [19848.121695] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19848.122795] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19848.123172] after_reply+0xe2c/0x1780 [ptlrpc] [19848.123400] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19848.123861] ? __next_timer_interrupt+0x160/0x160 [19848.123871] ptlrpcd+0xa82/0xe40 [ptlrpc] [19848.124211] ? do_wait_intr+0xf0/0xf0 [19848.124220] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19848.124537] kthread+0x1d1/0x200 [19848.131365] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19848.133853] ? set_kthread_struct+0x70/0x70 [19848.145506] ll_intent_lock+0xc5/0xaa0 [lustre] [19848.152274] ret_from_fork+0x1f/0x30 [19848.387358] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19848.394221] ll_lookup_it+0xe0e/0x2270 [lustre] [19848.397424] ? do_raw_spin_unlock+0x75/0x190 [19848.400868] ? _raw_spin_unlock+0x12/0x30 [19848.405479] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19848.408626] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19848.411451] ll_atomic_open+0x36a/0xd70 [lustre] [19848.413437] lookup_open+0x356/0xb40 [19848.415655] do_last+0x313/0xfc0 [19848.416725] ? nd_jump_root+0xe5/0x160 [19848.418151] ? path_init+0x437/0x520 [19848.419482] path_openat+0xf7/0x500 [19848.420680] ? do_raw_spin_unlock+0x75/0x190 [19848.422256] do_filp_open+0x99/0x140 [19848.423506] ? getname_flags+0x6e/0x330 [19848.425145] ? __check_object_size+0xff/0x256 [19848.426623] ? do_raw_spin_unlock+0x75/0x190 [19848.428012] ? _raw_spin_unlock+0x12/0x30 [19848.431726] do_sys_openat2+0x2b4/0x410 [19848.435242] do_sys_open+0x73/0xa0 [19848.437497] __x64_sys_openat+0x24/0x30 [19848.438847] do_syscall_64+0xc1/0x440 [19848.440323] entry_SYSCALL_64_after_hwframe+0x49/0xae [19848.445812] RIP: 0033:0x7f53425a5332 [19848.446968] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19848.455132] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19848.459412] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19848.462546] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19848.467227] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19848.472584] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19848.476848] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19872.053167] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [19872.056921] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19872.059616] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [19872.089029] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19872.089045] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19872.089053] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19872.089359] RIP: 0010:native_safe_halt+0x2e/0x50 [19872.089369] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19872.089374] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19872.089379] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19872.089381] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [19872.089384] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19872.089386] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19872.089388] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19872.089392] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19872.089395] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19872.089397] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19872.089406] Call Trace: [19872.089413] [19872.089418] ? show_regs.cold.9+0x22/0x2f [19872.089428] ? watchdog_timer_fn.cold.6+0x62/0x10a [19872.089435] ? __hrtimer_run_queues+0x1c2/0x4d0 [19872.089439] ? watchdog+0x40/0x40 [19872.089443] ? hrtimer_interrupt+0x128/0x2c0 [19872.089448] ? smp_apic_timer_interrupt+0xa6/0x270 [19872.089454] ? apic_timer_interrupt+0xf/0x20 [19872.089459] [19872.089462] ? native_safe_halt+0x2e/0x50 [19872.089466] kvm_wait+0xcd/0xe0 [19872.089474] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19872.089481] ? newidle_balance.isra.93+0x207/0x580 [19872.089500] do_raw_spin_lock+0x177/0x190 [19872.089506] _raw_spin_lock+0x21/0x30 [19872.089511] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19872.089715] process_one_work+0x2c8/0x700 [19872.089723] worker_thread+0x296/0x6e0 [19872.089727] ? rescuer_thread+0x570/0x570 [19872.089731] kthread+0x1d1/0x200 [19872.089735] ? set_kthread_struct+0x70/0x70 [19872.089739] ret_from_fork+0x1f/0x30 [19872.250173] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19872.275776] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19872.281604] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19872.284732] RIP: 0010:native_safe_halt+0x2e/0x50 [19872.286420] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19872.293225] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19872.296185] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19872.299572] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19872.301851] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19872.304602] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19872.307141] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19872.310092] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19872.313655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19872.316299] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19872.321726] Call Trace: [19872.323663] [19872.324302] ? show_regs.cold.9+0x22/0x2f [19872.325378] ? watchdog_timer_fn.cold.6+0x62/0x10a [19872.330730] ? __hrtimer_run_queues+0x1c2/0x4d0 [19872.333221] ? watchdog+0x40/0x40 [19872.335379] ? hrtimer_interrupt+0x128/0x2c0 [19872.337752] ? smp_apic_timer_interrupt+0xa6/0x270 [19872.340188] ? apic_timer_interrupt+0xf/0x20 [19872.343629] [19872.344163] ? native_safe_halt+0x2e/0x50 [19872.347268] kvm_wait+0xcd/0xe0 [19872.349938] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19872.353620] do_raw_spin_lock+0x177/0x190 [19872.355756] _raw_spin_lock+0x21/0x30 [19872.357836] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19872.359618] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19872.365247] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19872.369744] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19872.375904] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19872.378685] ? woken_wake_function+0x30/0x30 [19872.380374] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19872.383271] kthread+0x1d1/0x200 [19872.384640] ? set_kthread_struct+0x70/0x70 [19872.386132] ret_from_fork+0x1f/0x30 [19875.808046] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17238s! [19875.811672] Showing busy workqueues and worker pools: [19875.814184] workqueue events: flags=0x0 [19875.816159] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19875.818750] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19875.823914] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19875.826618] pending: cpu_latency_work [ptlrpc] [19875.829020] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19875.831870] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19875.835825] workqueue events_long: flags=0x0 [19875.837977] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19875.840902] pending: lu_objects_flush [obdclass] [19875.843372] workqueue events_unbound: flags=0x2 [19875.845650] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19875.848291] pending: flush_memcg_stats_dwork, idle_cull_fn [19875.850717] workqueue events_power_efficient: flags=0x80 [19875.853133] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19875.855943] pending: check_lifetime, do_cache_clean [19875.857774] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19875.860927] pending: neigh_periodic_work, neigh_periodic_work [19875.864063] workqueue mm_percpu_wq: flags=0x8 [19875.866045] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19875.868630] pending: vmstat_update, lru_add_drain_per_cpu [19875.870867] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19875.873734] pending: vmstat_update, lru_add_drain_per_cpu [19875.876017] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19875.879053] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19875.881282] workqueue xprtiod: flags=0xa [19875.882563] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19875.885059] pending: xprt_autoclose, xs_error_handle [19875.887392] workqueue ipv6_addrconf: flags=0x40008 [19875.889212] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19875.891733] pending: addrconf_verify_work [19875.893713] workqueue knbd0-recv: flags=0x1a [19875.895161] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19875.897788] in-flight: 651:recv_work [19875.899245] workqueue ptlrpc_pinger: flags=0x2 [19875.901325] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19875.903894] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19875.906250] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17238s workers=4 idle: 15979 31824 8 [19875.909046] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19866s workers=2 idle: 654 [19876.055041] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [19876.062357] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19876.064302] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [19876.082851] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19876.082865] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19876.082868] RIP: 0010:native_safe_halt+0x2e/0x50 [19876.082882] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19876.082885] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [19876.082891] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19876.082893] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19876.082895] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19876.082897] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19876.082900] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19876.082903] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19876.082907] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19876.082909] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19876.082918] Call Trace: [19876.082922] [19876.082927] ? show_regs.cold.9+0x22/0x2f [19876.082937] ? watchdog_timer_fn.cold.6+0x62/0x10a [19876.082943] ? __hrtimer_run_queues+0x1c2/0x4d0 [19876.082947] ? watchdog+0x40/0x40 [19876.082951] ? hrtimer_interrupt+0x128/0x2c0 [19876.082954] ? smp_apic_timer_interrupt+0xa6/0x270 [19876.082960] ? apic_timer_interrupt+0xf/0x20 [19876.082964] [19876.082967] ? native_safe_halt+0x2e/0x50 [19876.082970] kvm_wait+0xcd/0xe0 [19876.082977] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19876.082986] do_raw_spin_lock+0x177/0x190 [19876.082991] _raw_spin_lock+0x21/0x30 [19876.082995] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19876.083394] ? do_raw_spin_unlock+0x75/0x190 [19876.083400] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19876.083672] ? do_wait_intr+0xf0/0xf0 [19876.083681] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19876.083881] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19876.084212] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19876.084445] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19876.084562] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19876.084601] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19876.084630] mdc_intent_lock+0x353/0x7e0 [mdc] [19876.084657] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19876.084739] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19876.084965] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19876.084995] lmv_intent_open+0x338/0x1150 [lmv] [19876.085037] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19876.085124] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19876.085209] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19876.085229] ll_intent_lock+0xc5/0xaa0 [lustre] [19876.085319] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19876.085656] ll_lookup_it+0xe0e/0x2270 [lustre] [19876.085751] ? do_raw_spin_unlock+0x75/0x190 [19876.085758] ? _raw_spin_unlock+0x12/0x30 [19876.085765] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19876.085792] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19876.085810] ll_atomic_open+0x36a/0xd70 [lustre] [19876.085889] lookup_open+0x356/0xb40 [19876.085898] do_last+0x313/0xfc0 [19876.085903] ? nd_jump_root+0xe5/0x160 [19876.085908] ? path_init+0x437/0x520 [19876.085913] path_openat+0xf7/0x500 [19876.085918] ? do_raw_spin_unlock+0x75/0x190 [19876.085923] do_filp_open+0x99/0x140 [19876.085930] ? getname_flags+0x6e/0x330 [19876.085935] ? __check_object_size+0xff/0x256 [19876.085942] ? do_raw_spin_unlock+0x75/0x190 [19876.085946] ? _raw_spin_unlock+0x12/0x30 [19876.085951] do_sys_openat2+0x2b4/0x410 [19876.085969] do_sys_open+0x73/0xa0 [19876.085975] __x64_sys_openat+0x24/0x30 [19876.085979] do_syscall_64+0xc1/0x440 [19876.085985] entry_SYSCALL_64_after_hwframe+0x49/0xae [19876.085992] RIP: 0033:0x7f53425a5332 [19876.086339] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19876.086345] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19876.086349] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19876.086352] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19876.086354] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19876.086356] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19876.086358] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19876.272736] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19876.290524] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19876.294500] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19876.301938] RIP: 0010:native_safe_halt+0x2e/0x50 [19876.305517] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19876.313341] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19876.316574] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19876.319708] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19876.322347] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19876.325114] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19876.329211] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19876.333862] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19876.339516] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19876.342762] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19876.349227] Call Trace: [19876.350847] [19876.354283] ? show_regs.cold.9+0x22/0x2f [19876.356572] ? watchdog_timer_fn.cold.6+0x62/0x10a [19876.362334] ? __hrtimer_run_queues+0x1c2/0x4d0 [19876.367623] ? watchdog+0x40/0x40 [19876.368916] ? hrtimer_interrupt+0x128/0x2c0 [19876.370974] ? smp_apic_timer_interrupt+0xa6/0x270 [19876.373590] ? apic_timer_interrupt+0xf/0x20 [19876.375640] [19876.376686] ? native_safe_halt+0x2e/0x50 [19876.379128] kvm_wait+0xcd/0xe0 [19876.380795] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19876.382962] do_raw_spin_lock+0x177/0x190 [19876.384754] _raw_spin_lock+0x21/0x30 [19876.386570] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19876.389575] obd_mod_free+0x46/0x140 [ptlrpc] [19876.392174] mdc_commit_open+0x91/0xb0 [mdc] [19876.393980] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19876.396822] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19876.400165] after_reply+0xe2c/0x1780 [ptlrpc] [19876.402785] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19876.405600] ? __next_timer_interrupt+0x160/0x160 [19876.407774] ptlrpcd+0xa82/0xe40 [ptlrpc] [19876.410188] ? do_wait_intr+0xf0/0xf0 [19876.411793] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19876.414592] kthread+0x1d1/0x200 [19876.416147] ? set_kthread_struct+0x70/0x70 [19876.418237] ret_from_fork+0x1f/0x30 [19900.052027] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [19900.057727] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19900.058030] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19900.093891] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19900.093905] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19900.093912] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19900.094294] RIP: 0010:native_safe_halt+0x2e/0x50 [19900.094302] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19900.094306] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19900.094310] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19900.094312] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [19900.094314] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19900.094316] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19900.094318] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19900.094321] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19900.094323] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19900.094325] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19900.094333] Call Trace: [19900.094338] [19900.094343] ? show_regs.cold.9+0x22/0x2f [19900.100590] Modules linked in: lustre(O) [19900.111626] ? watchdog_timer_fn.cold.6+0x62/0x10a [19900.111644] ? __hrtimer_run_queues+0x1c2/0x4d0 [19900.111649] ? watchdog+0x40/0x40 [19900.111654] ? hrtimer_interrupt+0x128/0x2c0 [19900.111659] ? smp_apic_timer_interrupt+0xa6/0x270 [19900.111667] ? apic_timer_interrupt+0xf/0x20 [19900.111672] [19900.111677] ? native_safe_halt+0x2e/0x50 [19900.111681] kvm_wait+0xcd/0xe0 [19900.111690] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19900.111698] ? newidle_balance.isra.93+0x207/0x580 [19900.111706] do_raw_spin_lock+0x177/0x190 [19900.111711] _raw_spin_lock+0x21/0x30 [19900.111716] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19900.112030] process_one_work+0x2c8/0x700 [19900.112037] worker_thread+0x296/0x6e0 [19900.112041] ? rescuer_thread+0x570/0x570 [19900.112044] kthread+0x1d1/0x200 [19900.112049] ? set_kthread_struct+0x70/0x70 [19900.112052] ret_from_fork+0x1f/0x30 [19900.253693] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19900.268641] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19900.272169] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19900.274848] RIP: 0010:native_safe_halt+0x2e/0x50 [19900.276395] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19900.282517] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19900.284880] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19900.287245] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19900.289635] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19900.291765] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19900.293905] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19900.296052] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19900.298608] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19900.300382] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19900.302292] Call Trace: [19900.302834] [19900.303416] ? show_regs.cold.9+0x22/0x2f [19900.304670] ? watchdog_timer_fn.cold.6+0x62/0x10a [19900.306385] ? __hrtimer_run_queues+0x1c2/0x4d0 [19900.307972] ? watchdog+0x40/0x40 [19900.309404] ? hrtimer_interrupt+0x128/0x2c0 [19900.311344] ? smp_apic_timer_interrupt+0xa6/0x270 [19900.313474] ? apic_timer_interrupt+0xf/0x20 [19900.315717] [19900.319629] ? native_safe_halt+0x2e/0x50 [19900.322489] kvm_wait+0xcd/0xe0 [19900.324169] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19900.326659] do_raw_spin_lock+0x177/0x190 [19900.328333] _raw_spin_lock+0x21/0x30 [19900.329776] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19900.333164] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19900.335467] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19900.338216] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19900.342469] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19900.344407] ? woken_wake_function+0x30/0x30 [19900.346635] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19900.349737] kthread+0x1d1/0x200 [19900.351157] ? set_kthread_struct+0x70/0x70 [19900.353167] ret_from_fork+0x1f/0x30 [19904.055066] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [19904.057027] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [19904.059272] Modules linked in: [19904.062013] Modules linked in: [19904.062016] lustre(O) osp(O) [19904.063365] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19904.063450] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19904.063458] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19904.063461] RIP: 0010:native_safe_halt+0x2e/0x50 [19904.063476] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19904.063481] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19904.063486] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19904.063488] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19904.063491] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19904.063493] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19904.063496] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19904.063499] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19904.063503] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19904.063505] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19904.063514] Call Trace: [19904.063521] [19904.063527] ? show_regs.cold.9+0x22/0x2f [19904.063539] ? watchdog_timer_fn.cold.6+0x62/0x10a [19904.063554] ? __hrtimer_run_queues+0x1c2/0x4d0 [19904.063558] ? watchdog+0x40/0x40 [19904.063563] ? hrtimer_interrupt+0x128/0x2c0 [19904.063567] ? smp_apic_timer_interrupt+0xa6/0x270 [19904.063574] ? apic_timer_interrupt+0xf/0x20 [19904.063584] [19904.063586] ? native_safe_halt+0x2e/0x50 [19904.063590] kvm_wait+0xcd/0xe0 [19904.063599] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19904.063612] do_raw_spin_lock+0x177/0x190 [19904.063617] _raw_spin_lock+0x21/0x30 [19904.063623] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19904.065750] ofd(O) [19904.067076] obd_mod_free+0x46/0x140 [ptlrpc] [19904.067367] mdc_commit_open+0x91/0xb0 [mdc] [19904.067400] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19904.067591] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19904.067873] after_reply+0xe2c/0x1780 [ptlrpc] [19904.068090] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19904.068351] ? __next_timer_interrupt+0x160/0x160 [19904.068361] ptlrpcd+0xa82/0xe40 [ptlrpc] [19904.068635] ? do_wait_intr+0xf0/0xf0 [19904.068644] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19904.068865] kthread+0x1d1/0x200 [19904.068871] ? set_kthread_struct+0x70/0x70 [19904.068876] ret_from_fork+0x1f/0x30 [19904.271855] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19904.323832] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19904.332575] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19904.340080] RIP: 0010:native_safe_halt+0x2e/0x50 [19904.341629] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19904.357847] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19904.361564] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19904.364379] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19904.367026] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19904.377849] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19904.380779] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19904.384176] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19904.387657] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19904.396389] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19904.406097] Call Trace: [19904.410717] [19904.411715] ? show_regs.cold.9+0x22/0x2f [19904.413778] ? watchdog_timer_fn.cold.6+0x62/0x10a [19904.415696] ? __hrtimer_run_queues+0x1c2/0x4d0 [19904.416799] ? watchdog+0x40/0x40 [19904.421439] ? hrtimer_interrupt+0x128/0x2c0 [19904.423087] ? smp_apic_timer_interrupt+0xa6/0x270 [19904.425757] ? apic_timer_interrupt+0xf/0x20 [19904.428754] [19904.430628] ? native_safe_halt+0x2e/0x50 [19904.431779] kvm_wait+0xcd/0xe0 [19904.434319] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19904.436734] do_raw_spin_lock+0x177/0x190 [19904.438352] _raw_spin_lock+0x21/0x30 [19904.440842] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19904.445848] ? do_raw_spin_unlock+0x75/0x190 [19904.447696] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19904.449642] ? do_wait_intr+0xf0/0xf0 [19904.453747] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19904.456073] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19904.458250] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19904.465508] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19904.467645] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19904.470341] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19904.472508] mdc_intent_lock+0x353/0x7e0 [mdc] [19904.474487] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19904.476588] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19904.478219] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19904.480855] lmv_intent_open+0x338/0x1150 [lmv] [19904.482770] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19904.485253] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19904.490637] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19904.492719] ll_intent_lock+0xc5/0xaa0 [lustre] [19904.497967] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19904.499746] ll_lookup_it+0xe0e/0x2270 [lustre] [19904.503216] ? do_raw_spin_unlock+0x75/0x190 [19904.506178] ? _raw_spin_unlock+0x12/0x30 [19904.507983] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19904.510051] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19904.512044] ll_atomic_open+0x36a/0xd70 [lustre] [19904.513297] lookup_open+0x356/0xb40 [19904.514500] do_last+0x313/0xfc0 [19904.515286] ? nd_jump_root+0xe5/0x160 [19904.517311] ? path_init+0x437/0x520 [19904.518421] path_openat+0xf7/0x500 [19904.520602] ? do_raw_spin_unlock+0x75/0x190 [19904.522759] do_filp_open+0x99/0x140 [19904.523986] ? getname_flags+0x6e/0x330 [19904.528468] ? __check_object_size+0xff/0x256 [19904.530090] ? do_raw_spin_unlock+0x75/0x190 [19904.533083] ? _raw_spin_unlock+0x12/0x30 [19904.535081] do_sys_openat2+0x2b4/0x410 [19904.536775] do_sys_open+0x73/0xa0 [19904.538075] __x64_sys_openat+0x24/0x30 [19904.541249] do_syscall_64+0xc1/0x440 [19904.544053] entry_SYSCALL_64_after_hwframe+0x49/0xae [19904.546636] RIP: 0033:0x7f53425a5332 [19904.548763] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19904.555607] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19904.558421] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19904.560815] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19904.564513] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19904.568241] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19904.571808] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19906.017447] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17268s! [19906.031202] Showing busy workqueues and worker pools: [19906.041131] workqueue events: flags=0x0 [19906.042950] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19906.045870] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19906.057607] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19906.063875] pending: cpu_latency_work [ptlrpc] [19906.071828] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19906.076345] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19906.083610] workqueue events_long: flags=0x0 [19906.086466] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19906.090701] pending: lu_objects_flush [obdclass] [19906.092824] workqueue events_unbound: flags=0x2 [19906.094564] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19906.097984] pending: flush_memcg_stats_dwork, idle_cull_fn [19906.100304] workqueue events_power_efficient: flags=0x80 [19906.103478] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19906.107274] pending: check_lifetime, do_cache_clean [19906.109333] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19906.111862] pending: neigh_periodic_work, neigh_periodic_work [19906.115342] workqueue mm_percpu_wq: flags=0x8 [19906.117399] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19906.122030] pending: vmstat_update, lru_add_drain_per_cpu [19906.124192] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19906.128362] pending: vmstat_update, lru_add_drain_per_cpu [19906.131350] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19906.138362] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19906.142571] workqueue xprtiod: flags=0xa [19906.144716] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19906.149362] pending: xprt_autoclose, xs_error_handle [19906.153332] workqueue ipv6_addrconf: flags=0x40008 [19906.155991] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19906.158266] pending: addrconf_verify_work [19906.163151] workqueue knbd0-recv: flags=0x1a [19906.166262] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19906.171134] in-flight: 651:recv_work [19906.172610] workqueue ptlrpc_pinger: flags=0x2 [19906.176344] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19906.180607] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19906.182503] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17269s workers=4 idle: 15979 31824 8 [19906.189856] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19896s workers=2 idle: 654 [19928.060352] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19928.068083] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19928.102375] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19928.112308] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19928.122428] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19928.127044] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19928.142546] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19928.145478] RAX: 0000000000007883 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19928.150125] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19928.152960] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19928.159458] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19928.164837] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19928.172001] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19928.179946] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19928.184350] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19928.190399] Call Trace: [19928.192690] [19928.193914] ? show_regs.cold.9+0x22/0x2f [19928.197550] ? watchdog_timer_fn.cold.6+0x62/0x10a [19928.203695] ? __hrtimer_run_queues+0x1c2/0x4d0 [19928.209155] ? watchdog+0x40/0x40 [19928.210910] ? hrtimer_interrupt+0x128/0x2c0 [19928.216896] ? smp_apic_timer_interrupt+0xa6/0x270 [19928.219485] ? apic_timer_interrupt+0xf/0x20 [19928.224843] [19928.226653] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19928.235834] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19928.240980] do_raw_spin_lock+0x177/0x190 [19928.245146] _raw_spin_lock+0x21/0x30 [19928.250618] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19928.264870] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19928.268892] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19928.279797] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19928.288840] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19928.295941] ? woken_wake_function+0x30/0x30 [19928.300386] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19928.303928] kthread+0x1d1/0x200 [19928.309259] ? set_kthread_struct+0x70/0x70 [19928.311459] ret_from_fork+0x1f/0x30 [19932.052031] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [19932.055038] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [19932.055807] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19932.055860] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19932.057307] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [19932.057317] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19932.057396] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19932.057401] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19932.057404] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19932.057419] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19932.057424] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19932.057429] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19932.057431] RDX: 00000000000059a8 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19932.057433] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19932.059192] Modules linked in: [19932.060406] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19932.060412] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19932.060416] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19932.060419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19932.060422] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19932.060430] Call Trace: [19932.060446] [19932.060453] ? show_regs.cold.9+0x22/0x2f [19932.060471] ? watchdog_timer_fn.cold.6+0x62/0x10a [19932.060478] ? __hrtimer_run_queues+0x1c2/0x4d0 [19932.060483] ? watchdog+0x40/0x40 [19932.060488] ? hrtimer_interrupt+0x128/0x2c0 [19932.060491] ? smp_apic_timer_interrupt+0xa6/0x270 [19932.060500] ? apic_timer_interrupt+0xf/0x20 [19932.060504] [19932.060607] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19932.060620] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19932.060625] do_raw_spin_lock+0x177/0x190 [19932.060631] _raw_spin_lock+0x21/0x30 [19932.060639] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19932.060904] obd_mod_free+0x46/0x140 [ptlrpc] [19932.061935] mdc_commit_open+0x91/0xb0 [mdc] [19932.061981] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19932.062819] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19932.063123] after_reply+0xe2c/0x1780 [ptlrpc] [19932.063532] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19932.063733] ? __next_timer_interrupt+0x160/0x160 [19932.063744] ptlrpcd+0xa82/0xe40 [ptlrpc] [19932.063954] ? do_wait_intr+0xf0/0xf0 [19932.063965] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19932.064192] kthread+0x1d1/0x200 [19932.064200] ? set_kthread_struct+0x70/0x70 [19932.064204] ret_from_fork+0x1f/0x30 [19932.077763] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19932.077779] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19932.082246] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19932.082325] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19932.082332] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19932.082335] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19932.082349] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19932.082353] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19932.082358] RAX: 0000000000006688 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19932.082361] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19932.082363] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19932.082365] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19932.082368] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19932.082371] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19932.082374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19932.082377] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19932.082384] Call Trace: [19932.082389] [19932.082399] ? show_regs.cold.9+0x22/0x2f [19932.082408] ? watchdog_timer_fn.cold.6+0x62/0x10a [19932.082413] ? __hrtimer_run_queues+0x1c2/0x4d0 [19932.082417] ? watchdog+0x40/0x40 [19932.082421] ? hrtimer_interrupt+0x128/0x2c0 [19932.082426] ? smp_apic_timer_interrupt+0xa6/0x270 [19932.082434] ? apic_timer_interrupt+0xf/0x20 [19932.082450] [19932.082453] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19932.082460] do_raw_spin_lock+0x177/0x190 [19932.082464] _raw_spin_lock+0x21/0x30 [19932.082469] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19932.082742] ? do_raw_spin_unlock+0x75/0x190 [19932.082747] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19932.085368] [19932.085375] RIP: 0010:native_safe_halt+0x2e/0x50 [19932.085388] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19932.085392] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19932.085396] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19932.085398] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19932.085401] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19932.085403] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19932.085405] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19932.085408] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19932.085411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19932.085413] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19932.085419] Call Trace: [19932.085423] [19932.085427] ? show_regs.cold.9+0x22/0x2f [19932.104621] ? do_wait_intr+0xf0/0xf0 [19932.104645] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19932.105309] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19932.109866] ? watchdog_timer_fn.cold.6+0x62/0x10a [19932.109888] ? __hrtimer_run_queues+0x1c2/0x4d0 [19932.109893] ? watchdog+0x40/0x40 [19932.109897] ? hrtimer_interrupt+0x128/0x2c0 [19932.109902] ? smp_apic_timer_interrupt+0xa6/0x270 [19932.109910] ? apic_timer_interrupt+0xf/0x20 [19932.109915] [19932.109919] ? native_safe_halt+0x2e/0x50 [19932.109924] kvm_wait+0xcd/0xe0 [19932.109934] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19932.109941] ? newidle_balance.isra.93+0x207/0x580 [19932.109948] do_raw_spin_lock+0x177/0x190 [19932.109952] _raw_spin_lock+0x21/0x30 [19932.109956] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19932.113768] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19932.114035] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19932.114206] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19932.114246] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19932.114277] mdc_intent_lock+0x353/0x7e0 [mdc] [19932.114304] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19932.114389] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19932.114854] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19932.114893] lmv_intent_open+0x338/0x1150 [lmv] [19932.114942] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19932.115200] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19932.115291] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19932.115319] ll_intent_lock+0xc5/0xaa0 [lustre] [19932.115413] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19932.115564] ll_lookup_it+0xe0e/0x2270 [lustre] [19932.115752] ? do_raw_spin_unlock+0x75/0x190 [19932.115761] ? _raw_spin_unlock+0x12/0x30 [19932.115768] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19932.115795] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19932.115813] ll_atomic_open+0x36a/0xd70 [lustre] [19932.115900] lookup_open+0x356/0xb40 [19932.115913] do_last+0x313/0xfc0 [19932.115918] ? nd_jump_root+0xe5/0x160 [19932.115922] ? path_init+0x437/0x520 [19932.115925] path_openat+0xf7/0x500 [19932.115930] ? do_raw_spin_unlock+0x75/0x190 [19932.115934] do_filp_open+0x99/0x140 [19932.115939] ? getname_flags+0x6e/0x330 [19932.115943] ? __check_object_size+0xff/0x256 [19932.115949] ? do_raw_spin_unlock+0x75/0x190 [19932.115952] ? _raw_spin_unlock+0x12/0x30 [19932.115956] do_sys_openat2+0x2b4/0x410 [19932.115960] do_sys_open+0x73/0xa0 [19932.115963] __x64_sys_openat+0x24/0x30 [19932.115966] do_syscall_64+0xc1/0x440 [19932.115972] entry_SYSCALL_64_after_hwframe+0x49/0xae [19932.115977] RIP: 0033:0x7f53425a5332 [19932.115983] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19932.115987] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19932.115991] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19932.115994] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19932.115996] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19932.115998] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19932.116000] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19932.361901] process_one_work+0x2c8/0x700 [19932.363027] worker_thread+0x296/0x6e0 [19932.364017] ? rescuer_thread+0x570/0x570 [19932.365512] kthread+0x1d1/0x200 [19932.366605] ? set_kthread_struct+0x70/0x70 [19932.368331] ret_from_fork+0x1f/0x30 [19936.224050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17299s! [19936.227759] Showing busy workqueues and worker pools: [19936.230736] workqueue events: flags=0x0 [19936.233663] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19936.237875] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19936.243945] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19936.247154] pending: cpu_latency_work [ptlrpc] [19936.250420] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19936.252590] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19936.255262] workqueue events_long: flags=0x0 [19936.256641] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19936.258617] pending: lu_objects_flush [obdclass] [19936.260076] workqueue events_unbound: flags=0x2 [19936.261194] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19936.262788] pending: flush_memcg_stats_dwork, idle_cull_fn [19936.264735] workqueue events_power_efficient: flags=0x80 [19936.266191] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19936.267917] pending: check_lifetime, do_cache_clean [19936.269162] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19936.270910] pending: neigh_periodic_work, neigh_periodic_work [19936.272737] workqueue mm_percpu_wq: flags=0x8 [19936.274115] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19936.277135] pending: vmstat_update, lru_add_drain_per_cpu [19936.279302] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19936.281467] pending: vmstat_update, lru_add_drain_per_cpu [19936.282917] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19936.284844] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19936.286614] workqueue xprtiod: flags=0xa [19936.287591] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19936.289588] pending: xprt_autoclose, xs_error_handle [19936.291648] workqueue ipv6_addrconf: flags=0x40008 [19936.293563] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19936.295265] pending: addrconf_verify_work [19936.297034] workqueue knbd0-recv: flags=0x1a [19936.298935] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19936.301627] in-flight: 651:recv_work [19936.302670] workqueue ptlrpc_pinger: flags=0x2 [19936.303820] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19936.305909] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19936.308063] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17299s workers=4 idle: 15979 31824 8 [19936.310084] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19926s workers=2 idle: 654 [19956.058053] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19956.061524] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19956.076264] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19956.080095] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19956.082948] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19956.085283] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19956.091490] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19956.094371] RAX: 0000000000006712 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19956.097183] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19956.099500] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19956.102172] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19956.104486] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19956.107502] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19956.110091] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19956.112581] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19956.115282] Call Trace: [19956.116494] [19956.117504] ? show_regs.cold.9+0x22/0x2f [19956.118626] ? watchdog_timer_fn.cold.6+0x62/0x10a [19956.120245] ? __hrtimer_run_queues+0x1c2/0x4d0 [19956.121823] ? watchdog+0x40/0x40 [19956.122923] ? hrtimer_interrupt+0x128/0x2c0 [19956.124499] ? smp_apic_timer_interrupt+0xa6/0x270 [19956.125924] ? apic_timer_interrupt+0xf/0x20 [19956.127389] [19956.128184] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19956.129992] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19956.131835] do_raw_spin_lock+0x177/0x190 [19956.133154] _raw_spin_lock+0x21/0x30 [19956.134999] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19956.137347] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19956.139756] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19956.142129] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19956.144384] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19956.146116] ? woken_wake_function+0x30/0x30 [19956.147374] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19956.149387] kthread+0x1d1/0x200 [19956.150406] ? set_kthread_struct+0x70/0x70 [19956.151644] ret_from_fork+0x1f/0x30 [19960.055032] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [19960.057110] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [19960.064169] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19960.064262] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19960.064269] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19960.064272] RIP: 0010:native_safe_halt+0x2e/0x50 [19960.064285] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [19960.064289] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19960.064293] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19960.064296] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19960.064298] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19960.064300] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19960.064302] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19960.064305] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19960.064308] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19960.064310] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19960.064319] Call Trace: [19960.064325] [19960.064329] ? show_regs.cold.9+0x22/0x2f [19960.064339] ? watchdog_timer_fn.cold.6+0x62/0x10a [19960.064345] ? __hrtimer_run_queues+0x1c2/0x4d0 [19960.064349] ? watchdog+0x40/0x40 [19960.064354] ? hrtimer_interrupt+0x128/0x2c0 [19960.064358] ? smp_apic_timer_interrupt+0xa6/0x270 [19960.064364] ? apic_timer_interrupt+0xf/0x20 [19960.064369] [19960.064372] ? native_safe_halt+0x2e/0x50 [19960.064376] kvm_wait+0xcd/0xe0 [19960.064383] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19960.064391] do_raw_spin_lock+0x177/0x190 [19960.064395] _raw_spin_lock+0x21/0x30 [19960.064399] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19960.073053] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19960.073113] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19960.073119] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19960.073120] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [19960.073132] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [19960.073136] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19960.073141] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [19960.073144] RDX: 0000000000003eea RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [19960.073148] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [19960.098445] ? do_raw_spin_unlock+0x75/0x190 [19960.098474] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19960.098804] ? do_wait_intr+0xf0/0xf0 [19960.098818] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19960.099108] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19960.099517] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19960.099769] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19960.099928] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19960.100111] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19960.100143] mdc_intent_lock+0x353/0x7e0 [mdc] [19960.100172] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19960.111323] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [19960.115026] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19960.123851] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [19960.132588] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19960.132642] lmv_intent_open+0x338/0x1150 [lmv] [19960.132674] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19960.132786] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19960.132870] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19960.132892] ll_intent_lock+0xc5/0xaa0 [lustre] [19960.133103] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19960.133200] ll_lookup_it+0xe0e/0x2270 [lustre] [19960.133352] ? do_raw_spin_unlock+0x75/0x190 [19960.133360] ? _raw_spin_unlock+0x12/0x30 [19960.133367] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19960.139837] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [19960.146123] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19960.146176] ll_atomic_open+0x36a/0xd70 [lustre] [19960.151518] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19960.155170] lookup_open+0x356/0xb40 [19960.155189] do_last+0x313/0xfc0 [19960.155195] ? nd_jump_root+0xe5/0x160 [19960.155199] ? path_init+0x437/0x520 [19960.155204] path_openat+0xf7/0x500 [19960.155272] ? do_raw_spin_unlock+0x75/0x190 [19960.155279] do_filp_open+0x99/0x140 [19960.155285] ? getname_flags+0x6e/0x330 [19960.155299] ? __check_object_size+0xff/0x256 [19960.155310] ? do_raw_spin_unlock+0x75/0x190 [19960.155314] ? _raw_spin_unlock+0x12/0x30 [19960.155320] do_sys_openat2+0x2b4/0x410 [19960.155325] do_sys_open+0x73/0xa0 [19960.155329] __x64_sys_openat+0x24/0x30 [19960.155332] do_syscall_64+0xc1/0x440 [19960.155339] entry_SYSCALL_64_after_hwframe+0x49/0xae [19960.155346] RIP: 0033:0x7f53425a5332 [19960.155355] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19960.155358] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19960.155364] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19960.155366] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19960.155368] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19960.155370] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19960.155373] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19960.459885] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [19960.462430] Call Trace: [19960.466203] [19960.466931] ? show_regs.cold.9+0x22/0x2f [19960.468783] ? watchdog_timer_fn.cold.6+0x62/0x10a [19960.470671] ? __hrtimer_run_queues+0x1c2/0x4d0 [19960.472461] ? watchdog+0x40/0x40 [19960.473256] ? hrtimer_interrupt+0x128/0x2c0 [19960.474684] ? smp_apic_timer_interrupt+0xa6/0x270 [19960.475682] ? apic_timer_interrupt+0xf/0x20 [19960.476959] [19960.477899] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19960.480276] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19960.482536] do_raw_spin_lock+0x177/0x190 [19960.484272] _raw_spin_lock+0x21/0x30 [19960.485458] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19960.487285] obd_mod_free+0x46/0x140 [ptlrpc] [19960.489378] mdc_commit_open+0x91/0xb0 [mdc] [19960.491062] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19960.492322] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19960.493867] after_reply+0xe2c/0x1780 [ptlrpc] [19960.495225] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19960.499867] ? __next_timer_interrupt+0x160/0x160 [19960.501704] ptlrpcd+0xa82/0xe40 [ptlrpc] [19960.508280] ? do_wait_intr+0xf0/0xf0 [19960.510785] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19960.515533] kthread+0x1d1/0x200 [19960.516524] ? set_kthread_struct+0x70/0x70 [19960.518448] ret_from_fork+0x1f/0x30 [19964.052072] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [19964.058081] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19964.080020] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19964.085075] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19964.088940] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19964.092015] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19964.094072] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19964.102459] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [19964.106069] RAX: 0000000000004586 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19964.109145] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19964.112105] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19964.114810] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19964.117991] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19964.120531] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19964.123316] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19964.125302] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19964.129192] Call Trace: [19964.129916] [19964.130545] ? show_regs.cold.9+0x22/0x2f [19964.131706] ? watchdog_timer_fn.cold.6+0x62/0x10a [19964.133213] ? __hrtimer_run_queues+0x1c2/0x4d0 [19964.134558] ? watchdog+0x40/0x40 [19964.135650] ? hrtimer_interrupt+0x128/0x2c0 [19964.137093] ? smp_apic_timer_interrupt+0xa6/0x270 [19964.139306] ? apic_timer_interrupt+0xf/0x20 [19964.140769] [19964.141448] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19964.143397] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19964.145654] ? newidle_balance.isra.93+0x207/0x580 [19964.147357] do_raw_spin_lock+0x177/0x190 [19964.148956] _raw_spin_lock+0x21/0x30 [19964.150407] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19964.152710] process_one_work+0x2c8/0x700 [19964.154242] worker_thread+0x296/0x6e0 [19964.155843] ? rescuer_thread+0x570/0x570 [19964.158164] kthread+0x1d1/0x200 [19964.159815] ? set_kthread_struct+0x70/0x70 [19964.161693] ret_from_fork+0x1f/0x30 [19966.431601] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17329s! [19966.435662] Showing busy workqueues and worker pools: [19966.441105] workqueue events: flags=0x0 [19966.442828] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19966.456396] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19966.467819] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19966.473437] pending: cpu_latency_work [ptlrpc] [19966.477016] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19966.480032] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19966.485127] workqueue events_long: flags=0x0 [19966.487218] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19966.490992] pending: lu_objects_flush [obdclass] [19966.495732] workqueue events_unbound: flags=0x2 [19966.498378] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19966.502357] pending: flush_memcg_stats_dwork, idle_cull_fn [19966.505678] workqueue events_power_efficient: flags=0x80 [19966.508764] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19966.511977] pending: check_lifetime, do_cache_clean [19966.516804] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19966.522984] pending: neigh_periodic_work, neigh_periodic_work [19966.527504] workqueue mm_percpu_wq: flags=0x8 [19966.529934] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19966.533358] pending: vmstat_update, lru_add_drain_per_cpu [19966.539014] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19966.547070] pending: vmstat_update, lru_add_drain_per_cpu [19966.550264] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19966.556238] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19966.559968] workqueue xprtiod: flags=0xa [19966.565027] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19966.574303] pending: xprt_autoclose, xs_error_handle [19966.581811] workqueue ipv6_addrconf: flags=0x40008 [19966.587643] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19966.594301] pending: addrconf_verify_work [19966.598207] workqueue knbd0-recv: flags=0x1a [19966.601417] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19966.612713] in-flight: 651:recv_work [19966.617436] workqueue ptlrpc_pinger: flags=0x2 [19966.628478] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19966.637550] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19966.644583] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17329s workers=4 idle: 15979 31824 8 [19966.650532] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19957s workers=2 idle: 654 [19974.871429] rcu: INFO: rcu_sched self-detected stall on CPU [19974.879506] rcu: 2-....: (10144159 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [19974.887427] (t=17880445 jiffies g=274185 q=68437) [19974.891051] rcu: rcu_sched kthread starved for 17338811 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [19974.897088] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [19974.906657] rcu: RCU grace-period kthread stack dump: [19974.909262] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [19974.919335] Call Trace: [19974.922968] __schedule+0x351/0xcb0 [19974.924342] schedule+0xc0/0x180 [19974.926897] schedule_timeout+0xb4/0x190 [19974.928634] ? do_raw_spin_unlock+0x75/0x190 [19974.931900] ? __next_timer_interrupt+0x160/0x160 [19974.933982] ? prepare_to_swait_event+0xf3/0x140 [19974.936131] rcu_gp_kthread+0x8dd/0x15b0 [19974.938048] ? force_qs_rnp+0x2c0/0x2c0 [19974.943072] kthread+0x1d1/0x200 [19974.945403] ? set_kthread_struct+0x70/0x70 [19974.947855] ret_from_fork+0x1f/0x30 [19974.949601] rcu: Stack dump where RCU GP kthread last ran: [19974.954239] NMI backtrace for cpu 2 [19974.956442] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19974.964067] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19974.971827] Call Trace: [19974.974247] [19974.974995] ? dump_stack+0xbb/0x10e [19974.977605] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [19974.981557] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [19974.985938] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [19974.989313] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [19974.995221] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [19974.998445] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [19975.002338] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [19975.006011] ? update_process_times+0x6d/0xc0 [19975.007175] ? tick_sched_handle.isra.18+0x27/0x90 [19975.010984] ? tick_sched_timer+0x53/0xb0 [19975.014636] ? __hrtimer_run_queues+0x1c2/0x4d0 [19975.019545] ? tick_sched_handle.isra.18+0x90/0x90 [19975.021264] ? hrtimer_interrupt+0x128/0x2c0 [19975.026494] ? smp_apic_timer_interrupt+0xa6/0x270 [19975.031349] ? apic_timer_interrupt+0xf/0x20 [19975.033489] [19975.036933] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19975.039248] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19975.045386] ? do_raw_spin_lock+0x177/0x190 [19975.050301] ? _raw_spin_lock+0x21/0x30 [19975.055309] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19975.059765] ? obd_mod_free+0x46/0x140 [ptlrpc] [19975.064655] ? mdc_commit_open+0x91/0xb0 [mdc] [19975.068390] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19975.075147] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19975.078517] ? after_reply+0xe2c/0x1780 [ptlrpc] [19975.083153] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19975.086591] ? __next_timer_interrupt+0x160/0x160 [19975.089360] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [19975.093895] ? do_wait_intr+0xf0/0xf0 [19975.097963] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19975.100636] ? kthread+0x1d1/0x200 [19975.103429] ? set_kthread_struct+0x70/0x70 [19975.105653] ? ret_from_fork+0x1f/0x30 [19975.110165] Sending NMI from CPU 2 to CPUs 1: [19975.122313] NMI backtrace for cpu 1 [19975.122322] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19975.122324] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19975.122326] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19975.122328] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19975.122330] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [19975.122333] RAX: 0000000000003297 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19975.122334] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [19975.122336] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [19975.122337] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [19975.122338] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [19975.122339] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [19975.122341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19975.122342] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [19975.122343] Call Trace: [19975.122344] [19975.122344] ? show_regs.cold.9+0x22/0x2f [19975.122345] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [19975.122346] ? nmi_cpu_backtrace_handler+0x15/0x40 [19975.122347] ? nmi_handle+0xbd/0x210 [19975.122348] ? default_do_nmi+0x4b/0x1b0 [19975.122349] ? do_nmi+0x280/0x380 [19975.122350] ? end_repeat_nmi+0x16/0x1f [19975.122351] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19975.122352] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19975.122353] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19975.122354] [19975.122355] do_raw_spin_lock+0x177/0x190 [19975.122356] _raw_spin_lock+0x21/0x30 [19975.122356] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [19975.122357] ? do_raw_spin_unlock+0x75/0x190 [19975.122358] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [19975.122359] ? do_wait_intr+0xf0/0xf0 [19975.122360] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [19975.122361] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [19975.122362] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19975.122363] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19975.122364] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19975.122365] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [19975.122366] mdc_intent_lock+0x353/0x7e0 [mdc] [19975.122367] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19975.122368] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [19975.122369] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [19975.122370] lmv_intent_open+0x338/0x1150 [lmv] [19975.122371] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19975.122372] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19975.122372] lmv_intent_lock+0x1dd/0x4c0 [lmv] [19975.122373] ll_intent_lock+0xc5/0xaa0 [lustre] [19975.122374] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [19975.122375] ll_lookup_it+0xe0e/0x2270 [lustre] [19975.122376] ? do_raw_spin_unlock+0x75/0x190 [19975.122377] ? _raw_spin_unlock+0x12/0x30 [19975.122378] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [19975.122379] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [19975.122380] ll_atomic_open+0x36a/0xd70 [lustre] [19975.122381] lookup_open+0x356/0xb40 [19975.122382] do_last+0x313/0xfc0 [19975.122383] ? nd_jump_root+0xe5/0x160 [19975.122384] ? path_init+0x437/0x520 [19975.122385] path_openat+0xf7/0x500 [19975.122385] ? do_raw_spin_unlock+0x75/0x190 [19975.122387] do_filp_open+0x99/0x140 [19975.122387] ? getname_flags+0x6e/0x330 [19975.122388] ? __check_object_size+0xff/0x256 [19975.122389] ? do_raw_spin_unlock+0x75/0x190 [19975.122390] ? _raw_spin_unlock+0x12/0x30 [19975.122391] do_sys_openat2+0x2b4/0x410 [19975.122392] do_sys_open+0x73/0xa0 [19975.122393] __x64_sys_openat+0x24/0x30 [19975.122394] do_syscall_64+0xc1/0x440 [19975.122395] entry_SYSCALL_64_after_hwframe+0x49/0xae [19975.122395] RIP: 0033:0x7f53425a5332 [19975.122397] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [19975.122398] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [19975.122400] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [19975.122402] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [19975.122403] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [19975.122404] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [19975.122405] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [19975.123205] NMI backtrace for cpu 2 [19975.344894] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19975.352220] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19975.354991] Call Trace: [19975.355627] [19975.357133] ? dump_stack+0xbb/0x10e [19975.358778] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [19975.360248] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [19975.362282] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [19975.366187] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [19975.370151] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [19975.373311] ? rcu_dump_cpu_stacks+0x11b/0x171 [19975.377665] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [19975.379940] ? update_process_times+0x6d/0xc0 [19975.382088] ? tick_sched_handle.isra.18+0x27/0x90 [19975.383902] ? tick_sched_timer+0x53/0xb0 [19975.385939] ? __hrtimer_run_queues+0x1c2/0x4d0 [19975.388251] ? tick_sched_handle.isra.18+0x90/0x90 [19975.390358] ? hrtimer_interrupt+0x128/0x2c0 [19975.393571] ? smp_apic_timer_interrupt+0xa6/0x270 [19975.396623] ? apic_timer_interrupt+0xf/0x20 [19975.398358] [19975.399181] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [19975.401396] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [19975.403307] ? do_raw_spin_lock+0x177/0x190 [19975.405285] ? _raw_spin_lock+0x21/0x30 [19975.406431] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [19975.408513] ? obd_mod_free+0x46/0x140 [ptlrpc] [19975.410266] ? mdc_commit_open+0x91/0xb0 [mdc] [19975.411595] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [19975.413526] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [19975.415472] ? after_reply+0xe2c/0x1780 [ptlrpc] [19975.417216] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [19975.418991] ? __next_timer_interrupt+0x160/0x160 [19975.420373] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [19975.421941] ? do_wait_intr+0xf0/0xf0 [19975.423140] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [19975.424842] ? kthread+0x1d1/0x200 [19975.425908] ? set_kthread_struct+0x70/0x70 [19975.427035] ? ret_from_fork+0x1f/0x30 [19984.063614] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [19984.066975] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19984.100054] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19984.105696] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19984.110277] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19984.113614] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19984.135483] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [19984.143792] RAX: 0000000000000986 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19984.147236] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [19984.154767] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [19984.159559] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [19984.170824] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [19984.180080] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [19984.185883] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19984.189763] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [19984.192418] Call Trace: [19984.193274] [19984.193962] ? show_regs.cold.9+0x22/0x2f [19984.197425] ? watchdog_timer_fn.cold.6+0x62/0x10a [19984.199598] ? __hrtimer_run_queues+0x1c2/0x4d0 [19984.201621] ? watchdog+0x40/0x40 [19984.203146] ? hrtimer_interrupt+0x128/0x2c0 [19984.205068] ? smp_apic_timer_interrupt+0xa6/0x270 [19984.207687] ? apic_timer_interrupt+0xf/0x20 [19984.209044] [19984.209990] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19984.212118] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19984.214604] do_raw_spin_lock+0x177/0x190 [19984.216205] _raw_spin_lock+0x21/0x30 [19984.217703] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [19984.220796] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [19984.223356] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [19984.226021] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [19984.233405] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [19984.240028] ? woken_wake_function+0x30/0x30 [19984.242416] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [19984.244997] kthread+0x1d1/0x200 [19984.245881] ? set_kthread_struct+0x70/0x70 [19984.248812] ret_from_fork+0x1f/0x30 [19992.052622] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [19992.061510] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [19992.124589] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [19992.136812] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [19992.144232] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [19992.151468] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19992.163332] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [19992.186829] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [19992.191336] RAX: 0000000000007f4f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [19992.194290] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [19992.197933] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [19992.207275] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [19992.213788] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [19992.224480] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [19992.233962] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [19992.243288] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [19992.250889] Call Trace: [19992.252675] [19992.254934] ? show_regs.cold.9+0x22/0x2f [19992.259298] ? watchdog_timer_fn.cold.6+0x62/0x10a [19992.262360] ? __hrtimer_run_queues+0x1c2/0x4d0 [19992.264345] ? watchdog+0x40/0x40 [19992.266808] ? hrtimer_interrupt+0x128/0x2c0 [19992.269446] ? smp_apic_timer_interrupt+0xa6/0x270 [19992.272154] ? apic_timer_interrupt+0xf/0x20 [19992.274349] [19992.275524] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [19992.278442] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [19992.281962] ? newidle_balance.isra.93+0x207/0x580 [19992.296932] do_raw_spin_lock+0x177/0x190 [19992.314625] _raw_spin_lock+0x21/0x30 [19992.318359] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [19992.325897] process_one_work+0x2c8/0x700 [19992.332737] worker_thread+0x296/0x6e0 [19992.338162] ? rescuer_thread+0x570/0x570 [19992.346064] kthread+0x1d1/0x200 [19992.359153] ? set_kthread_struct+0x70/0x70 [19992.366290] ret_from_fork+0x1f/0x30 [19997.152038] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17360s! [19997.155963] Showing busy workqueues and worker pools: [19997.159674] workqueue events: flags=0x0 [19997.162712] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [19997.169710] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [19997.175799] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19997.195904] pending: cpu_latency_work [ptlrpc] [19997.205207] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [19997.218554] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [19997.235221] workqueue events_long: flags=0x0 [19997.240778] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [19997.254479] pending: lu_objects_flush [obdclass] [19997.263310] workqueue events_unbound: flags=0x2 [19997.271959] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [19997.280194] pending: flush_memcg_stats_dwork, idle_cull_fn [19997.285937] workqueue events_power_efficient: flags=0x80 [19997.295770] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19997.302398] pending: check_lifetime, do_cache_clean [19997.305868] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19997.314935] pending: neigh_periodic_work, neigh_periodic_work [19997.323754] workqueue mm_percpu_wq: flags=0x8 [19997.329476] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19997.347215] pending: vmstat_update, lru_add_drain_per_cpu [19997.359261] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [19997.368096] pending: vmstat_update, lru_add_drain_per_cpu [19997.374782] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [19997.382479] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [19997.396565] workqueue xprtiod: flags=0xa [19997.403116] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [19997.410746] pending: xprt_autoclose, xs_error_handle [19997.415732] workqueue ipv6_addrconf: flags=0x40008 [19997.417475] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [19997.425970] pending: addrconf_verify_work [19997.428363] workqueue knbd0-recv: flags=0x1a [19997.430591] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [19997.434180] in-flight: 651:recv_work [19997.436566] workqueue ptlrpc_pinger: flags=0x2 [19997.437897] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [19997.441703] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [19997.452279] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17360s workers=4 idle: 15979 31824 8 [19997.465159] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=19987s workers=2 idle: 654 [20000.055988] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20000.058213] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20000.060055] Modules linked in: [20000.071058] Modules linked in: [20000.071063] lustre(O) [20000.074925] lustre(O) [20000.076512] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20000.076575] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20000.076582] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20000.076585] RIP: 0010:native_safe_halt+0x2e/0x50 [20000.076597] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20000.076601] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20000.076605] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20000.076607] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20000.076609] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20000.076611] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20000.076613] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20000.076616] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20000.076618] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20000.076620] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20000.076628] Call Trace: [20000.076634] [20000.076639] ? show_regs.cold.9+0x22/0x2f [20000.076651] ? watchdog_timer_fn.cold.6+0x62/0x10a [20000.076657] ? __hrtimer_run_queues+0x1c2/0x4d0 [20000.076661] ? watchdog+0x40/0x40 [20000.076666] ? hrtimer_interrupt+0x128/0x2c0 [20000.076670] ? smp_apic_timer_interrupt+0xa6/0x270 [20000.076676] ? apic_timer_interrupt+0xf/0x20 [20000.076681] [20000.076685] ? native_safe_halt+0x2e/0x50 [20000.076689] kvm_wait+0xcd/0xe0 [20000.076696] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20000.076705] do_raw_spin_lock+0x177/0x190 [20000.076709] _raw_spin_lock+0x21/0x30 [20000.076714] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20000.077163] ? do_raw_spin_unlock+0x75/0x190 [20000.077169] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20000.077538] ? do_wait_intr+0xf0/0xf0 [20000.077548] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20000.077745] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20000.078106] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20000.078622] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20000.078740] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20000.078784] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20000.078811] mdc_intent_lock+0x353/0x7e0 [mdc] [20000.078954] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20000.079040] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20000.079386] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20000.079419] lmv_intent_open+0x338/0x1150 [lmv] [20000.079533] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20000.079619] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20000.079703] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20000.079723] ll_intent_lock+0xc5/0xaa0 [lustre] [20000.079805] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20000.080030] ll_lookup_it+0xe0e/0x2270 [lustre] [20000.080120] ? do_raw_spin_unlock+0x75/0x190 [20000.080427] ? _raw_spin_unlock+0x12/0x30 [20000.080433] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20000.085611] osp(O) [20000.086760] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20000.086804] ll_atomic_open+0x36a/0xd70 [lustre] [20000.087050] lookup_open+0x356/0xb40 [20000.087063] do_last+0x313/0xfc0 [20000.087070] ? nd_jump_root+0xe5/0x160 [20000.169636] ofd(O) [20000.174687] ? path_init+0x437/0x520 [20000.174704] path_openat+0xf7/0x500 [20000.174711] ? do_raw_spin_unlock+0x75/0x190 [20000.174717] do_filp_open+0x99/0x140 [20000.174724] ? getname_flags+0x6e/0x330 [20000.174729] ? __check_object_size+0xff/0x256 [20000.174735] ? do_raw_spin_unlock+0x75/0x190 [20000.174739] ? _raw_spin_unlock+0x12/0x30 [20000.174745] do_sys_openat2+0x2b4/0x410 [20000.174750] do_sys_open+0x73/0xa0 [20000.174754] __x64_sys_openat+0x24/0x30 [20000.174758] do_syscall_64+0xc1/0x440 [20000.174769] entry_SYSCALL_64_after_hwframe+0x49/0xae [20000.174776] RIP: 0033:0x7f53425a5332 [20000.174785] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20000.174789] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20000.174795] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20000.174798] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20000.174800] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20000.174802] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20000.174805] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20000.461091] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20000.507573] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20000.515762] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20000.526223] RIP: 0010:native_safe_halt+0x2e/0x50 [20000.538067] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20000.560857] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [20000.572452] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20000.578782] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20000.588212] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20000.602779] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20000.614894] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20000.622940] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20000.636410] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20000.643214] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20000.655551] Call Trace: [20000.658768] [20000.662932] ? show_regs.cold.9+0x22/0x2f [20000.666996] ? watchdog_timer_fn.cold.6+0x62/0x10a [20000.677340] ? __hrtimer_run_queues+0x1c2/0x4d0 [20000.685944] ? watchdog+0x40/0x40 [20000.690677] ? hrtimer_interrupt+0x128/0x2c0 [20000.704192] ? smp_apic_timer_interrupt+0xa6/0x270 [20000.712171] ? apic_timer_interrupt+0xf/0x20 [20000.719783] [20000.723659] ? native_safe_halt+0x2e/0x50 [20000.729311] kvm_wait+0xcd/0xe0 [20000.739530] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20000.748109] do_raw_spin_lock+0x177/0x190 [20000.754423] _raw_spin_lock+0x21/0x30 [20000.758530] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20000.762684] obd_mod_free+0x46/0x140 [ptlrpc] [20000.771543] mdc_commit_open+0x91/0xb0 [mdc] [20000.775552] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20000.779675] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20000.789552] after_reply+0xe2c/0x1780 [ptlrpc] [20000.795117] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20000.800020] ? __next_timer_interrupt+0x160/0x160 [20000.804216] ptlrpcd+0xa82/0xe40 [ptlrpc] [20000.807111] ? do_wait_intr+0xf0/0xf0 [20000.809094] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20000.823646] kthread+0x1d1/0x200 [20000.829889] ? set_kthread_struct+0x70/0x70 [20000.833751] ret_from_fork+0x1f/0x30 [20012.058020] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20012.063864] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20012.098728] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20012.110585] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20012.120753] RIP: 0010:native_safe_halt+0x2e/0x50 [20012.123627] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20012.144000] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20012.153360] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20012.166777] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20012.175312] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20012.186033] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20012.188432] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20012.198825] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20012.208439] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20012.219704] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20012.225976] Call Trace: [20012.227620] [20012.228386] ? show_regs.cold.9+0x22/0x2f [20012.229742] ? watchdog_timer_fn.cold.6+0x62/0x10a [20012.237819] ? __hrtimer_run_queues+0x1c2/0x4d0 [20012.246257] ? watchdog+0x40/0x40 [20012.247425] ? hrtimer_interrupt+0x128/0x2c0 [20012.248775] ? smp_apic_timer_interrupt+0xa6/0x270 [20012.255566] ? apic_timer_interrupt+0xf/0x20 [20012.260164] [20012.261485] ? native_safe_halt+0x2e/0x50 [20012.265773] kvm_wait+0xcd/0xe0 [20012.269377] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20012.275477] do_raw_spin_lock+0x177/0x190 [20012.280816] _raw_spin_lock+0x21/0x30 [20012.285583] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20012.292841] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20012.296420] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20012.300775] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20012.306448] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20012.309381] ? woken_wake_function+0x30/0x30 [20012.311161] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20012.314577] kthread+0x1d1/0x200 [20012.324585] ? set_kthread_struct+0x70/0x70 [20012.327607] ret_from_fork+0x1f/0x30 [20024.058498] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20024.069588] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20024.145687] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20024.161409] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20024.170287] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20024.179458] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20024.188150] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20024.204498] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20024.216921] RAX: 0000000000007d86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20024.230353] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [20024.239631] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20024.243274] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20024.249222] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20024.252009] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20024.257031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20024.260806] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20024.264625] Call Trace: [20024.266955] [20024.267937] ? show_regs.cold.9+0x22/0x2f [20024.269924] ? watchdog_timer_fn.cold.6+0x62/0x10a [20024.273695] ? __hrtimer_run_queues+0x1c2/0x4d0 [20024.276540] ? watchdog+0x40/0x40 [20024.278593] ? hrtimer_interrupt+0x128/0x2c0 [20024.282963] ? smp_apic_timer_interrupt+0xa6/0x270 [20024.290416] ? apic_timer_interrupt+0xf/0x20 [20024.294034] [20024.296579] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20024.299474] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20024.302162] ? newidle_balance.isra.93+0x207/0x580 [20024.304785] do_raw_spin_lock+0x177/0x190 [20024.307511] _raw_spin_lock+0x21/0x30 [20024.309350] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20024.318915] process_one_work+0x2c8/0x700 [20024.320998] worker_thread+0x296/0x6e0 [20024.322766] ? rescuer_thread+0x570/0x570 [20024.324489] kthread+0x1d1/0x200 [20024.325645] ? set_kthread_struct+0x70/0x70 [20024.327315] ret_from_fork+0x1f/0x30 [20027.872119] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17390s! [20027.879112] Showing busy workqueues and worker pools: [20027.880916] workqueue events: flags=0x0 [20027.882644] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20027.885348] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20027.890598] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20027.893623] pending: cpu_latency_work [ptlrpc] [20027.895510] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20027.897510] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20027.902104] workqueue events_long: flags=0x0 [20027.903734] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20027.906997] pending: lu_objects_flush [obdclass] [20027.909804] workqueue events_unbound: flags=0x2 [20027.914595] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20027.919314] pending: flush_memcg_stats_dwork, idle_cull_fn [20027.925242] workqueue events_power_efficient: flags=0x80 [20027.927839] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20027.935302] pending: check_lifetime, do_cache_clean [20027.937225] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20027.939975] pending: neigh_periodic_work, neigh_periodic_work [20027.941630] workqueue mm_percpu_wq: flags=0x8 [20027.943293] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20027.945603] pending: vmstat_update, lru_add_drain_per_cpu [20027.947234] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20027.950476] pending: vmstat_update, lru_add_drain_per_cpu [20027.953117] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20027.956749] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20027.960201] workqueue xprtiod: flags=0xa [20027.961970] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20027.964566] pending: xprt_autoclose, xs_error_handle [20027.966990] workqueue ipv6_addrconf: flags=0x40008 [20027.968695] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20027.971651] pending: addrconf_verify_work [20027.973586] workqueue knbd0-recv: flags=0x1a [20027.980554] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20027.984805] in-flight: 651:recv_work [20027.986222] workqueue ptlrpc_pinger: flags=0x2 [20027.991840] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20027.996562] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20028.001281] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17390s workers=4 idle: 15979 31824 8 [20028.005063] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20018s workers=2 idle: 654 [20028.055021] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20028.056892] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20028.057095] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20028.077657] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20028.077671] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20028.077673] RIP: 0010:native_safe_halt+0x2e/0x50 [20028.077684] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20028.077688] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [20028.077693] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20028.077694] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20028.077696] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20028.077697] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20028.077699] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20028.077702] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20028.077704] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20028.077705] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20028.077712] Call Trace: [20028.077716] [20028.077721] ? show_regs.cold.9+0x22/0x2f [20028.077731] ? watchdog_timer_fn.cold.6+0x62/0x10a [20028.077737] ? __hrtimer_run_queues+0x1c2/0x4d0 [20028.077741] ? watchdog+0x40/0x40 [20028.077746] ? hrtimer_interrupt+0x128/0x2c0 [20028.077750] ? smp_apic_timer_interrupt+0xa6/0x270 [20028.077962] ? apic_timer_interrupt+0xf/0x20 [20028.077966] [20028.077969] ? native_safe_halt+0x2e/0x50 [20028.077972] kvm_wait+0xcd/0xe0 [20028.077979] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20028.077988] do_raw_spin_lock+0x177/0x190 [20028.077992] _raw_spin_lock+0x21/0x30 [20028.077996] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20028.084102] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20028.084180] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20028.084188] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20028.084192] RIP: 0010:native_safe_halt+0x2e/0x50 [20028.084205] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20028.084209] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20028.084213] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20028.084215] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20028.084217] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20028.084219] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20028.084221] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20028.084224] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20028.084226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20028.084229] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20028.084242] Call Trace: [20028.084248] [20028.084254] ? show_regs.cold.9+0x22/0x2f [20028.087446] ? do_raw_spin_unlock+0x75/0x190 [20028.090260] ? watchdog_timer_fn.cold.6+0x62/0x10a [20028.091441] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20028.108579] ? __hrtimer_run_queues+0x1c2/0x4d0 [20028.111277] ? do_wait_intr+0xf0/0xf0 [20028.111295] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20028.119199] ? watchdog+0x40/0x40 [20028.122257] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20028.122504] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20028.122647] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20028.127305] ? hrtimer_interrupt+0x128/0x2c0 [20028.138605] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20028.138650] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20028.138670] mdc_intent_lock+0x353/0x7e0 [mdc] [20028.138689] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20028.138811] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20028.139097] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20028.139124] lmv_intent_open+0x338/0x1150 [lmv] [20028.139149] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20028.139232] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20028.139311] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20028.139333] ll_intent_lock+0xc5/0xaa0 [lustre] [20028.139389] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20028.139443] ll_lookup_it+0xe0e/0x2270 [lustre] [20028.139498] ? do_raw_spin_unlock+0x75/0x190 [20028.139505] ? _raw_spin_unlock+0x12/0x30 [20028.139510] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20028.139531] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20028.139544] ll_atomic_open+0x36a/0xd70 [lustre] [20028.139599] lookup_open+0x356/0xb40 [20028.139606] do_last+0x313/0xfc0 [20028.139609] ? nd_jump_root+0xe5/0x160 [20028.139613] ? path_init+0x437/0x520 [20028.139616] path_openat+0xf7/0x500 [20028.139619] ? do_raw_spin_unlock+0x75/0x190 [20028.139622] do_filp_open+0x99/0x140 [20028.139627] ? getname_flags+0x6e/0x330 [20028.139630] ? __check_object_size+0xff/0x256 [20028.139636] ? do_raw_spin_unlock+0x75/0x190 [20028.139639] ? _raw_spin_unlock+0x12/0x30 [20028.139642] do_sys_openat2+0x2b4/0x410 [20028.139644] do_sys_open+0x73/0xa0 [20028.139647] __x64_sys_openat+0x24/0x30 [20028.139649] do_syscall_64+0xc1/0x440 [20028.139654] entry_SYSCALL_64_after_hwframe+0x49/0xae [20028.139658] RIP: 0033:0x7f53425a5332 [20028.139664] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20028.139667] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20028.139670] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20028.139672] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20028.139673] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20028.139675] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20028.139677] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20028.477920] ? smp_apic_timer_interrupt+0xa6/0x270 [20028.479613] ? apic_timer_interrupt+0xf/0x20 [20028.481299] [20028.482096] ? native_safe_halt+0x2e/0x50 [20028.483433] kvm_wait+0xcd/0xe0 [20028.484600] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20028.486604] do_raw_spin_lock+0x177/0x190 [20028.488249] _raw_spin_lock+0x21/0x30 [20028.490375] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20028.492214] obd_mod_free+0x46/0x140 [ptlrpc] [20028.494054] mdc_commit_open+0x91/0xb0 [mdc] [20028.495567] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20028.497654] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20028.499729] after_reply+0xe2c/0x1780 [ptlrpc] [20028.501642] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20028.503822] ? __next_timer_interrupt+0x160/0x160 [20028.505649] ptlrpcd+0xa82/0xe40 [ptlrpc] [20028.507509] ? do_wait_intr+0xf0/0xf0 [20028.508735] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20028.510866] kthread+0x1d1/0x200 [20028.512012] ? set_kthread_struct+0x70/0x70 [20028.513570] ret_from_fork+0x1f/0x30 [20040.061481] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20040.067842] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20040.112476] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20040.120434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20040.126154] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20040.137120] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20040.160106] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20040.163776] RAX: 0000000000005430 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20040.166772] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20040.169746] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20040.172828] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20040.175859] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20040.179107] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20040.189759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20040.192262] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20040.197186] Call Trace: [20040.199150] [20040.200083] ? show_regs.cold.9+0x22/0x2f [20040.203057] ? watchdog_timer_fn.cold.6+0x62/0x10a [20040.208864] ? __hrtimer_run_queues+0x1c2/0x4d0 [20040.213750] ? watchdog+0x40/0x40 [20040.215324] ? hrtimer_interrupt+0x128/0x2c0 [20040.218177] ? smp_apic_timer_interrupt+0xa6/0x270 [20040.220767] ? apic_timer_interrupt+0xf/0x20 [20040.223934] [20040.225998] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20040.229273] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20040.240233] do_raw_spin_lock+0x177/0x190 [20040.242009] _raw_spin_lock+0x21/0x30 [20040.244909] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20040.248857] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20040.251821] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20040.255992] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20040.259738] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20040.266085] ? woken_wake_function+0x30/0x30 [20040.269854] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20040.272015] kthread+0x1d1/0x200 [20040.274776] ? set_kthread_struct+0x70/0x70 [20040.276391] ret_from_fork+0x1f/0x30 [20052.052137] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20052.055382] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20052.079188] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20052.083927] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20052.090328] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20052.100314] RIP: 0010:native_safe_halt+0x2e/0x50 [20052.101704] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20052.114762] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20052.125935] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20052.129021] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [20052.133858] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20052.137665] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20052.140040] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20052.150887] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20052.156963] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20052.161692] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20052.175918] Call Trace: [20052.176998] [20052.178419] ? show_regs.cold.9+0x22/0x2f [20052.183553] ? watchdog_timer_fn.cold.6+0x62/0x10a [20052.188814] ? __hrtimer_run_queues+0x1c2/0x4d0 [20052.193706] ? watchdog+0x40/0x40 [20052.195443] ? hrtimer_interrupt+0x128/0x2c0 [20052.199715] ? smp_apic_timer_interrupt+0xa6/0x270 [20052.214837] ? apic_timer_interrupt+0xf/0x20 [20052.224608] [20052.225111] ? native_safe_halt+0x2e/0x50 [20052.231375] kvm_wait+0xcd/0xe0 [20052.232287] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20052.233804] ? newidle_balance.isra.93+0x207/0x580 [20052.238385] do_raw_spin_lock+0x177/0x190 [20052.239226] _raw_spin_lock+0x21/0x30 [20052.240352] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20052.245202] process_one_work+0x2c8/0x700 [20052.248726] worker_thread+0x296/0x6e0 [20052.250185] ? rescuer_thread+0x570/0x570 [20052.251821] kthread+0x1d1/0x200 [20052.253023] ? set_kthread_struct+0x70/0x70 [20052.254818] ret_from_fork+0x1f/0x30 [20056.055950] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20056.057541] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20056.060119] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20056.060193] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20056.060199] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20056.060202] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20056.060215] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20056.060218] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20056.060222] RAX: 0000000000003f1f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20056.060225] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20056.060227] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20056.060229] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20056.060231] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20056.060234] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20056.060237] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20056.060239] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20056.060245] Call Trace: [20056.060249] [20056.060255] ? show_regs.cold.9+0x22/0x2f [20056.060263] ? watchdog_timer_fn.cold.6+0x62/0x10a [20056.060282] ? __hrtimer_run_queues+0x1c2/0x4d0 [20056.060287] ? watchdog+0x40/0x40 [20056.060291] ? hrtimer_interrupt+0x128/0x2c0 [20056.060296] ? smp_apic_timer_interrupt+0xa6/0x270 [20056.060304] ? apic_timer_interrupt+0xf/0x20 [20056.060308] [20056.060311] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20056.060316] do_raw_spin_lock+0x177/0x190 [20056.060321] _raw_spin_lock+0x21/0x30 [20056.060325] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20056.068942] Modules linked in: lustre(O) [20056.087643] ? do_raw_spin_unlock+0x75/0x190 [20056.087665] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20056.087922] ? do_wait_intr+0xf0/0xf0 [20056.087930] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20056.088121] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20056.088338] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20056.088711] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20056.088808] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20056.088837] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20056.088856] mdc_intent_lock+0x353/0x7e0 [mdc] [20056.088873] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20056.088929] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20056.089097] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20056.089120] lmv_intent_open+0x338/0x1150 [lmv] [20056.089143] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20056.089200] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20056.089254] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20056.089327] ll_intent_lock+0xc5/0xaa0 [lustre] [20056.089383] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20056.089521] ll_lookup_it+0xe0e/0x2270 [lustre] [20056.089606] ? do_raw_spin_unlock+0x75/0x190 [20056.089613] ? _raw_spin_unlock+0x12/0x30 [20056.089619] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20056.089641] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20056.089660] ll_atomic_open+0x36a/0xd70 [lustre] [20056.089743] lookup_open+0x356/0xb40 [20056.089752] do_last+0x313/0xfc0 [20056.089756] ? nd_jump_root+0xe5/0x160 [20056.089761] ? path_init+0x437/0x520 [20056.089765] path_openat+0xf7/0x500 [20056.089769] ? do_raw_spin_unlock+0x75/0x190 [20056.089774] do_filp_open+0x99/0x140 [20056.089780] ? getname_flags+0x6e/0x330 [20056.089784] ? __check_object_size+0xff/0x256 [20056.089791] ? do_raw_spin_unlock+0x75/0x190 [20056.089795] ? _raw_spin_unlock+0x12/0x30 [20056.089799] do_sys_openat2+0x2b4/0x410 [20056.089803] do_sys_open+0x73/0xa0 [20056.089807] __x64_sys_openat+0x24/0x30 [20056.089810] do_syscall_64+0xc1/0x440 [20056.089815] entry_SYSCALL_64_after_hwframe+0x49/0xae [20056.089819] RIP: 0033:0x7f53425a5332 [20056.089826] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20056.089829] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20056.089833] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20056.089836] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20056.089838] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20056.089840] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20056.089842] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20056.362326] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20056.396253] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20056.400583] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20056.407722] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [20056.413252] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [20056.425898] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20056.428108] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20056.430216] RDX: 00000000000031de RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20056.434220] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20056.436372] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20056.442791] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20056.446793] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20056.453209] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20056.455997] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20056.461794] Call Trace: [20056.463789] [20056.464336] ? show_regs.cold.9+0x22/0x2f [20056.465721] ? watchdog_timer_fn.cold.6+0x62/0x10a [20056.473216] ? __hrtimer_run_queues+0x1c2/0x4d0 [20056.474974] ? watchdog+0x40/0x40 [20056.476066] ? hrtimer_interrupt+0x128/0x2c0 [20056.478979] ? smp_apic_timer_interrupt+0xa6/0x270 [20056.482309] ? apic_timer_interrupt+0xf/0x20 [20056.484646] [20056.485512] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20056.486953] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20056.489150] do_raw_spin_lock+0x177/0x190 [20056.492084] _raw_spin_lock+0x21/0x30 [20056.494748] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20056.499680] obd_mod_free+0x46/0x140 [ptlrpc] [20056.502370] mdc_commit_open+0x91/0xb0 [mdc] [20056.504008] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20056.508768] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20056.512207] after_reply+0xe2c/0x1780 [ptlrpc] [20056.513943] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20056.517058] ? __next_timer_interrupt+0x160/0x160 [20056.521131] ptlrpcd+0xa82/0xe40 [ptlrpc] [20056.524768] ? do_wait_intr+0xf0/0xf0 [20056.527110] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20056.532220] kthread+0x1d1/0x200 [20056.535185] ? set_kthread_struct+0x70/0x70 [20056.537522] ret_from_fork+0x1f/0x30 [20058.085673] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17420s! [20058.095362] Showing busy workqueues and worker pools: [20058.097080] workqueue events: flags=0x0 [20058.098510] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20058.101034] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20058.106162] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20058.113471] pending: cpu_latency_work [ptlrpc] [20058.116191] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20058.120172] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20058.130446] workqueue events_long: flags=0x0 [20058.133845] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20058.138133] pending: lu_objects_flush [obdclass] [20058.139640] workqueue events_unbound: flags=0x2 [20058.141300] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20058.143134] pending: flush_memcg_stats_dwork, idle_cull_fn [20058.147188] workqueue events_power_efficient: flags=0x80 [20058.148952] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20058.152066] pending: check_lifetime, do_cache_clean [20058.154293] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20058.156679] pending: neigh_periodic_work, neigh_periodic_work [20058.159151] workqueue mm_percpu_wq: flags=0x8 [20058.160660] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20058.164191] pending: vmstat_update, lru_add_drain_per_cpu [20058.166755] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20058.170226] pending: vmstat_update, lru_add_drain_per_cpu [20058.172742] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20058.176341] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20058.181818] workqueue xprtiod: flags=0xa [20058.183130] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20058.187105] pending: xprt_autoclose, xs_error_handle [20058.189393] workqueue ipv6_addrconf: flags=0x40008 [20058.191818] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20058.194323] pending: addrconf_verify_work [20058.196581] workqueue knbd0-recv: flags=0x1a [20058.198607] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20058.202745] in-flight: 651:recv_work [20058.204382] workqueue ptlrpc_pinger: flags=0x2 [20058.206672] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20058.209198] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20058.216204] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17421s workers=4 idle: 15979 31824 8 [20058.222856] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20048s workers=2 idle: 654 [20068.059424] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20068.063834] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20068.095962] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20068.104649] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20068.110648] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20068.114842] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20068.129491] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20068.136023] RAX: 0000000000006792 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20068.140878] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20068.144884] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20068.151248] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20068.156845] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20068.162788] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20068.168094] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20068.172112] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20068.176052] Call Trace: [20068.177214] [20068.177932] ? show_regs.cold.9+0x22/0x2f [20068.179080] ? watchdog_timer_fn.cold.6+0x62/0x10a [20068.180593] ? __hrtimer_run_queues+0x1c2/0x4d0 [20068.182577] ? watchdog+0x40/0x40 [20068.184737] ? hrtimer_interrupt+0x128/0x2c0 [20068.188002] ? smp_apic_timer_interrupt+0xa6/0x270 [20068.193529] ? apic_timer_interrupt+0xf/0x20 [20068.197428] [20068.198696] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20068.202775] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20068.206903] do_raw_spin_lock+0x177/0x190 [20068.210846] _raw_spin_lock+0x21/0x30 [20068.214555] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20068.219971] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20068.224728] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20068.230216] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20068.234520] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20068.238420] ? woken_wake_function+0x30/0x30 [20068.242301] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20068.246034] kthread+0x1d1/0x200 [20068.246841] ? set_kthread_struct+0x70/0x70 [20068.249422] ret_from_fork+0x1f/0x30 [20084.053025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20084.055039] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20084.055458] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20084.057111] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20084.057126] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20084.057206] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20084.057212] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20084.057215] RIP: 0010:native_safe_halt+0x2e/0x50 [20084.057229] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20084.057234] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20084.057238] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20084.057241] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20084.057243] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20084.057246] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20084.057248] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20084.057251] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20084.057254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20084.057257] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20084.057274] Call Trace: [20084.057281] [20084.057287] ? show_regs.cold.9+0x22/0x2f [20084.057296] ? watchdog_timer_fn.cold.6+0x62/0x10a [20084.057303] ? __hrtimer_run_queues+0x1c2/0x4d0 [20084.057313] ? watchdog+0x40/0x40 [20084.057318] ? hrtimer_interrupt+0x128/0x2c0 [20084.057322] ? smp_apic_timer_interrupt+0xa6/0x270 [20084.057329] ? apic_timer_interrupt+0xf/0x20 [20084.057335] [20084.057338] ? native_safe_halt+0x2e/0x50 [20084.057342] kvm_wait+0xcd/0xe0 [20084.057350] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20084.057359] do_raw_spin_lock+0x177/0x190 [20084.057364] _raw_spin_lock+0x21/0x30 [20084.057369] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20084.057676] obd_mod_free+0x46/0x140 [ptlrpc] [20084.057913] mdc_commit_open+0x91/0xb0 [mdc] [20084.058067] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20084.058283] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20084.058551] after_reply+0xe2c/0x1780 [ptlrpc] [20084.058869] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20084.059333] ? __next_timer_interrupt+0x160/0x160 [20084.059344] ptlrpcd+0xa82/0xe40 [ptlrpc] [20084.059557] ? do_wait_intr+0xf0/0xf0 [20084.059567] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20084.059896] kthread+0x1d1/0x200 [20084.059903] ? set_kthread_struct+0x70/0x70 [20084.060356] ret_from_fork+0x1f/0x30 [20084.080604] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20084.080619] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20084.080625] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20084.081003] RIP: 0010:native_safe_halt+0x2e/0x50 [20084.081013] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20084.081017] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20084.081022] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20084.081025] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [20084.081027] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20084.081029] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20084.081032] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20084.081035] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20084.081038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20084.081040] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20084.081048] Call Trace: [20084.081053] [20084.081058] ? show_regs.cold.9+0x22/0x2f [20084.081067] ? watchdog_timer_fn.cold.6+0x62/0x10a [20084.081072] ? __hrtimer_run_queues+0x1c2/0x4d0 [20084.081076] ? watchdog+0x40/0x40 [20084.081081] ? hrtimer_interrupt+0x128/0x2c0 [20084.081085] ? smp_apic_timer_interrupt+0xa6/0x270 [20084.081092] ? apic_timer_interrupt+0xf/0x20 [20084.081096] [20084.081099] ? native_safe_halt+0x2e/0x50 [20084.081103] kvm_wait+0xcd/0xe0 [20084.081111] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20084.081117] ? newidle_balance.isra.93+0x207/0x580 [20084.081123] do_raw_spin_lock+0x177/0x190 [20084.081127] _raw_spin_lock+0x21/0x30 [20084.081131] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20084.081377] process_one_work+0x2c8/0x700 [20084.081384] worker_thread+0x296/0x6e0 [20084.081388] ? rescuer_thread+0x570/0x570 [20084.081392] kthread+0x1d1/0x200 [20084.081396] ? set_kthread_struct+0x70/0x70 [20084.081400] ret_from_fork+0x1f/0x30 [20084.291492] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20084.313152] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20084.317940] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20084.322270] RIP: 0010:native_safe_halt+0x2e/0x50 [20084.324243] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20084.333537] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20084.337482] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20084.342468] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20084.350438] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20084.354518] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20084.358946] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20084.363559] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20084.368170] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20084.372065] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20084.376494] Call Trace: [20084.377748] [20084.378556] ? show_regs.cold.9+0x22/0x2f [20084.380227] ? watchdog_timer_fn.cold.6+0x62/0x10a [20084.381966] ? __hrtimer_run_queues+0x1c2/0x4d0 [20084.385651] ? watchdog+0x40/0x40 [20084.387294] ? hrtimer_interrupt+0x128/0x2c0 [20084.389231] ? smp_apic_timer_interrupt+0xa6/0x270 [20084.391240] ? apic_timer_interrupt+0xf/0x20 [20084.392903] [20084.393733] ? native_safe_halt+0x2e/0x50 [20084.395476] kvm_wait+0xcd/0xe0 [20084.396693] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20084.398701] do_raw_spin_lock+0x177/0x190 [20084.400044] _raw_spin_lock+0x21/0x30 [20084.401326] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20084.403314] ? do_raw_spin_unlock+0x75/0x190 [20084.404754] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20084.406686] ? do_wait_intr+0xf0/0xf0 [20084.407860] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20084.409675] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20084.411487] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20084.413675] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20084.415467] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20084.417269] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20084.418875] mdc_intent_lock+0x353/0x7e0 [mdc] [20084.420260] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20084.422503] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20084.424750] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20084.426757] lmv_intent_open+0x338/0x1150 [lmv] [20084.428351] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20084.430932] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20084.433325] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20084.435516] ll_intent_lock+0xc5/0xaa0 [lustre] [20084.437754] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20084.440088] ll_lookup_it+0xe0e/0x2270 [lustre] [20084.442181] ? do_raw_spin_unlock+0x75/0x190 [20084.443987] ? _raw_spin_unlock+0x12/0x30 [20084.445661] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20084.447964] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20084.450157] ll_atomic_open+0x36a/0xd70 [lustre] [20084.452397] lookup_open+0x356/0xb40 [20084.454213] do_last+0x313/0xfc0 [20084.455722] ? nd_jump_root+0xe5/0x160 [20084.457375] ? path_init+0x437/0x520 [20084.459058] path_openat+0xf7/0x500 [20084.460527] ? do_raw_spin_unlock+0x75/0x190 [20084.462543] do_filp_open+0x99/0x140 [20084.464530] ? getname_flags+0x6e/0x330 [20084.466462] ? __check_object_size+0xff/0x256 [20084.468342] ? do_raw_spin_unlock+0x75/0x190 [20084.470391] ? _raw_spin_unlock+0x12/0x30 [20084.472238] do_sys_openat2+0x2b4/0x410 [20084.474092] do_sys_open+0x73/0xa0 [20084.475543] __x64_sys_openat+0x24/0x30 [20084.477146] do_syscall_64+0xc1/0x440 [20084.478747] entry_SYSCALL_64_after_hwframe+0x49/0xae [20084.480605] RIP: 0033:0x7f53425a5332 [20084.482330] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20084.489959] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20084.493143] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20084.495510] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20084.497943] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20084.500986] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20084.503678] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20088.291123] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17451s! [20088.297448] Showing busy workqueues and worker pools: [20088.304468] workqueue events: flags=0x0 [20088.307254] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20088.310781] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20088.316047] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20088.318625] pending: cpu_latency_work [ptlrpc] [20088.320502] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20088.322600] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20088.326088] workqueue events_long: flags=0x0 [20088.327340] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20088.329821] pending: lu_objects_flush [obdclass] [20088.331639] workqueue events_unbound: flags=0x2 [20088.334011] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20088.338095] pending: flush_memcg_stats_dwork, idle_cull_fn [20088.340294] workqueue events_power_efficient: flags=0x80 [20088.345675] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20088.358117] pending: check_lifetime, do_cache_clean [20088.359734] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20088.367528] pending: neigh_periodic_work, neigh_periodic_work [20088.370269] workqueue mm_percpu_wq: flags=0x8 [20088.371603] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20088.380148] pending: vmstat_update, lru_add_drain_per_cpu [20088.382035] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20088.384585] pending: vmstat_update, lru_add_drain_per_cpu [20088.388259] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20088.394248] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20088.398764] workqueue xprtiod: flags=0xa [20088.401300] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20088.404033] pending: xprt_autoclose, xs_error_handle [20088.406155] workqueue ipv6_addrconf: flags=0x40008 [20088.408527] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20088.410435] pending: addrconf_verify_work [20088.412258] workqueue knbd0-recv: flags=0x1a [20088.414470] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20088.417130] in-flight: 651:recv_work [20088.418841] workqueue ptlrpc_pinger: flags=0x2 [20088.420849] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20088.423250] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20088.426158] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17451s workers=4 idle: 15979 31824 8 [20088.429310] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20078s workers=2 idle: 654 [20096.058098] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20096.064939] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20096.102193] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20096.113354] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20096.120128] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20096.122795] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20096.139075] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20096.141907] RAX: 0000000000004e60 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20096.156100] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20096.160362] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20096.162533] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20096.171348] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20096.175944] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20096.182761] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20096.186944] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20096.194260] Call Trace: [20096.196867] [20096.197693] ? show_regs.cold.9+0x22/0x2f [20096.199760] ? watchdog_timer_fn.cold.6+0x62/0x10a [20096.204830] ? __hrtimer_run_queues+0x1c2/0x4d0 [20096.207921] ? watchdog+0x40/0x40 [20096.212324] ? hrtimer_interrupt+0x128/0x2c0 [20096.216739] ? smp_apic_timer_interrupt+0xa6/0x270 [20096.222767] ? apic_timer_interrupt+0xf/0x20 [20096.227436] [20096.228261] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20096.242393] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20096.248828] do_raw_spin_lock+0x177/0x190 [20096.250757] _raw_spin_lock+0x21/0x30 [20096.252348] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20096.255869] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20096.262811] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20096.267530] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20096.275247] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20096.280917] ? woken_wake_function+0x30/0x30 [20096.286822] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20096.294343] kthread+0x1d1/0x200 [20096.296214] ? set_kthread_struct+0x70/0x70 [20096.301843] ret_from_fork+0x1f/0x30 [20112.057179] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20112.060747] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20112.061089] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20112.061262] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20112.061268] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20112.061271] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [20112.061509] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [20112.061515] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20112.061520] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20112.061522] RDX: 0000000000005d08 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20112.061525] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20112.061528] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20112.061530] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20112.061533] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20112.061536] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20112.061539] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20112.061546] Call Trace: [20112.061553] [20112.061559] ? show_regs.cold.9+0x22/0x2f [20112.061568] ? watchdog_timer_fn.cold.6+0x62/0x10a [20112.061575] ? __hrtimer_run_queues+0x1c2/0x4d0 [20112.061579] ? watchdog+0x40/0x40 [20112.061583] ? hrtimer_interrupt+0x128/0x2c0 [20112.061587] ? smp_apic_timer_interrupt+0xa6/0x270 [20112.061596] ? apic_timer_interrupt+0xf/0x20 [20112.061600] [20112.061603] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20112.061608] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20112.061613] do_raw_spin_lock+0x177/0x190 [20112.061618] _raw_spin_lock+0x21/0x30 [20112.061624] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20112.062051] obd_mod_free+0x46/0x140 [ptlrpc] [20112.062264] mdc_commit_open+0x91/0xb0 [mdc] [20112.062387] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20112.062607] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20112.062984] after_reply+0xe2c/0x1780 [ptlrpc] [20112.063186] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20112.075485] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [20112.101109] ? __next_timer_interrupt+0x160/0x160 [20112.101206] ptlrpcd+0xa82/0xe40 [ptlrpc] [20112.101600] ? do_wait_intr+0xf0/0xf0 [20112.101610] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20112.101874] kthread+0x1d1/0x200 [20112.101951] ? set_kthread_struct+0x70/0x70 [20112.101956] ret_from_fork+0x1f/0x30 [20112.281096] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20112.296146] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20112.299999] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20112.303422] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20112.305915] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20112.313146] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20112.315888] RAX: 0000000000004483 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20112.318450] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20112.320948] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20112.323525] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20112.325970] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20112.330003] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20112.333141] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20112.335410] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20112.341205] Call Trace: [20112.342213] [20112.343080] ? show_regs.cold.9+0x22/0x2f [20112.345219] ? watchdog_timer_fn.cold.6+0x62/0x10a [20112.347004] ? __hrtimer_run_queues+0x1c2/0x4d0 [20112.349126] ? watchdog+0x40/0x40 [20112.350236] ? hrtimer_interrupt+0x128/0x2c0 [20112.351889] ? smp_apic_timer_interrupt+0xa6/0x270 [20112.353795] ? apic_timer_interrupt+0xf/0x20 [20112.355507] [20112.356226] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20112.358877] do_raw_spin_lock+0x177/0x190 [20112.360658] _raw_spin_lock+0x21/0x30 [20112.363132] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20112.365456] ? do_raw_spin_unlock+0x75/0x190 [20112.367697] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20112.370883] ? do_wait_intr+0xf0/0xf0 [20112.375811] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20112.380096] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20112.384376] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20112.389142] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20112.392966] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20112.399507] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20112.401721] mdc_intent_lock+0x353/0x7e0 [mdc] [20112.404356] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20112.408102] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20112.410698] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20112.413704] lmv_intent_open+0x338/0x1150 [lmv] [20112.415663] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20112.418094] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20112.420528] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20112.423483] ll_intent_lock+0xc5/0xaa0 [lustre] [20112.430691] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20112.432514] ll_lookup_it+0xe0e/0x2270 [lustre] [20112.437953] ? do_raw_spin_unlock+0x75/0x190 [20112.445309] ? _raw_spin_unlock+0x12/0x30 [20112.446790] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20112.456918] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20112.459017] ll_atomic_open+0x36a/0xd70 [lustre] [20112.464407] lookup_open+0x356/0xb40 [20112.466970] do_last+0x313/0xfc0 [20112.473511] ? nd_jump_root+0xe5/0x160 [20112.476650] ? path_init+0x437/0x520 [20112.479976] path_openat+0xf7/0x500 [20112.483641] ? do_raw_spin_unlock+0x75/0x190 [20112.488735] do_filp_open+0x99/0x140 [20112.493640] ? getname_flags+0x6e/0x330 [20112.496888] ? __check_object_size+0xff/0x256 [20112.505237] ? do_raw_spin_unlock+0x75/0x190 [20112.507725] ? _raw_spin_unlock+0x12/0x30 [20112.512422] do_sys_openat2+0x2b4/0x410 [20112.514275] do_sys_open+0x73/0xa0 [20112.515170] __x64_sys_openat+0x24/0x30 [20112.516457] do_syscall_64+0xc1/0x440 [20112.518684] entry_SYSCALL_64_after_hwframe+0x49/0xae [20112.520669] RIP: 0033:0x7f53425a5332 [20112.522479] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20112.536698] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20112.541143] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20112.545618] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20112.549959] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20112.554890] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20112.556827] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20116.052030] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20116.055744] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20116.089849] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20116.099660] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20116.104981] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20116.115854] RIP: 0010:native_safe_halt+0x2e/0x50 [20116.124196] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20116.154468] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20116.160367] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20116.169171] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [20116.173716] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20116.177760] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20116.186296] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20116.190697] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20116.199660] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20116.203004] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20116.206298] Call Trace: [20116.208395] [20116.209362] ? show_regs.cold.9+0x22/0x2f [20116.215298] ? watchdog_timer_fn.cold.6+0x62/0x10a [20116.218569] ? __hrtimer_run_queues+0x1c2/0x4d0 [20116.221097] ? watchdog+0x40/0x40 [20116.222180] ? hrtimer_interrupt+0x128/0x2c0 [20116.227110] ? smp_apic_timer_interrupt+0xa6/0x270 [20116.231244] ? apic_timer_interrupt+0xf/0x20 [20116.233747] [20116.234651] ? native_safe_halt+0x2e/0x50 [20116.237925] kvm_wait+0xcd/0xe0 [20116.239068] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20116.245564] ? newidle_balance.isra.93+0x207/0x580 [20116.248178] do_raw_spin_lock+0x177/0x190 [20116.251032] _raw_spin_lock+0x21/0x30 [20116.255550] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20116.261809] process_one_work+0x2c8/0x700 [20116.266101] worker_thread+0x296/0x6e0 [20116.268028] ? rescuer_thread+0x570/0x570 [20116.270801] kthread+0x1d1/0x200 [20116.275120] ? set_kthread_struct+0x70/0x70 [20116.277454] ret_from_fork+0x1f/0x30 [20118.496634] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17481s! [20118.503830] Showing busy workqueues and worker pools: [20118.506590] workqueue events: flags=0x0 [20118.507934] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20118.514576] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20118.523319] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20118.527994] pending: cpu_latency_work [ptlrpc] [20118.531469] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20118.543515] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20118.555507] workqueue events_long: flags=0x0 [20118.558926] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20118.568465] pending: lu_objects_flush [obdclass] [20118.574154] workqueue events_unbound: flags=0x2 [20118.575655] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20118.578709] pending: flush_memcg_stats_dwork, idle_cull_fn [20118.582313] workqueue events_power_efficient: flags=0x80 [20118.584722] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20118.587610] pending: check_lifetime, do_cache_clean [20118.590031] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20118.592875] pending: neigh_periodic_work, neigh_periodic_work [20118.599030] workqueue mm_percpu_wq: flags=0x8 [20118.605670] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20118.610120] pending: vmstat_update, lru_add_drain_per_cpu [20118.612791] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20118.623512] pending: vmstat_update, lru_add_drain_per_cpu [20118.625555] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20118.639403] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20118.648105] workqueue xprtiod: flags=0xa [20118.649970] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20118.654818] pending: xprt_autoclose, xs_error_handle [20118.660677] workqueue ipv6_addrconf: flags=0x40008 [20118.666886] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20118.671429] pending: addrconf_verify_work [20118.673476] workqueue knbd0-recv: flags=0x1a [20118.675854] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20118.678832] in-flight: 651:recv_work [20118.681516] workqueue ptlrpc_pinger: flags=0x2 [20118.683532] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20118.687086] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20118.691649] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17481s workers=4 idle: 15979 31824 8 [20118.698415] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20109s workers=2 idle: 654 [20124.060947] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20124.066861] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20124.111959] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20124.121729] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20124.129608] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20124.134915] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20124.152993] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20124.159981] RAX: 0000000000006e86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20124.165561] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20124.168036] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20124.177214] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20124.184826] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20124.191058] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20124.204035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20124.212929] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20124.221907] Call Trace: [20124.225546] [20124.227217] ? show_regs.cold.9+0x22/0x2f [20124.232615] ? watchdog_timer_fn.cold.6+0x62/0x10a [20124.239727] ? __hrtimer_run_queues+0x1c2/0x4d0 [20124.243880] ? watchdog+0x40/0x40 [20124.248601] ? hrtimer_interrupt+0x128/0x2c0 [20124.251998] ? smp_apic_timer_interrupt+0xa6/0x270 [20124.257174] ? apic_timer_interrupt+0xf/0x20 [20124.262232] [20124.263906] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20124.268953] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20124.275907] do_raw_spin_lock+0x177/0x190 [20124.280525] _raw_spin_lock+0x21/0x30 [20124.284030] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20124.286921] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20124.298702] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20124.306896] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20124.317568] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20124.329188] ? woken_wake_function+0x30/0x30 [20124.330356] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20124.341327] kthread+0x1d1/0x200 [20124.342498] ? set_kthread_struct+0x70/0x70 [20124.343646] ret_from_fork+0x1f/0x30 [20140.055973] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20140.058019] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20140.062210] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) [20140.064951] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20140.065025] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20140.065033] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20140.065035] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [20140.065050] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [20140.065054] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20140.065059] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20140.065061] RDX: 0000000000004987 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20140.065063] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20140.065065] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20140.065067] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20140.065070] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20140.065073] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20140.065074] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20140.065083] Call Trace: [20140.065089] [20140.065095] ? show_regs.cold.9+0x22/0x2f [20140.065104] ? watchdog_timer_fn.cold.6+0x62/0x10a [20140.065111] ? __hrtimer_run_queues+0x1c2/0x4d0 [20140.065115] ? watchdog+0x40/0x40 [20140.065119] ? hrtimer_interrupt+0x128/0x2c0 [20140.065122] ? smp_apic_timer_interrupt+0xa6/0x270 [20140.065130] ? apic_timer_interrupt+0xf/0x20 [20140.065135] [20140.065137] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20140.065142] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20140.065148] do_raw_spin_lock+0x177/0x190 [20140.065152] _raw_spin_lock+0x21/0x30 [20140.065158] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20140.075524] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw [20140.103740] obd_mod_free+0x46/0x140 [ptlrpc] [20140.110665] libata [20140.116622] mdc_commit_open+0x91/0xb0 [mdc] [20140.122012] dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20140.122035] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20140.122045] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20140.122052] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20140.122070] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20140.122079] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20140.122083] RAX: 0000000000007e13 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20140.122086] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20140.122088] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20140.122091] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20140.122093] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20140.122100] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20140.122107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20140.122109] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20140.122116] Call Trace: [20140.122121] [20140.122131] ? show_regs.cold.9+0x22/0x2f [20140.122144] ? watchdog_timer_fn.cold.6+0x62/0x10a [20140.122150] ? __hrtimer_run_queues+0x1c2/0x4d0 [20140.122159] ? watchdog+0x40/0x40 [20140.122164] ? hrtimer_interrupt+0x128/0x2c0 [20140.122172] ? smp_apic_timer_interrupt+0xa6/0x270 [20140.122182] ? apic_timer_interrupt+0xf/0x20 [20140.122186] [20140.122193] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20140.122205] do_raw_spin_lock+0x177/0x190 [20140.122211] _raw_spin_lock+0x21/0x30 [20140.122216] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20140.122950] ? do_raw_spin_unlock+0x75/0x190 [20140.122961] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20140.123432] ? do_wait_intr+0xf0/0xf0 [20140.123511] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20140.123906] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20140.142584] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20140.150750] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20140.151204] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20140.151402] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20140.151473] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20140.151515] mdc_intent_lock+0x353/0x7e0 [mdc] [20140.151564] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20140.151681] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20140.152205] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20140.152242] lmv_intent_open+0x338/0x1150 [lmv] [20140.152293] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20140.152411] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20140.152519] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20140.152556] ll_intent_lock+0xc5/0xaa0 [lustre] [20140.152893] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20140.153158] ll_lookup_it+0xe0e/0x2270 [lustre] [20140.153421] ? do_raw_spin_unlock+0x75/0x190 [20140.153436] ? _raw_spin_unlock+0x12/0x30 [20140.153444] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20140.153480] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20140.153510] ll_atomic_open+0x36a/0xd70 [lustre] [20140.153633] lookup_open+0x356/0xb40 [20140.153647] do_last+0x313/0xfc0 [20140.153653] ? nd_jump_root+0xe5/0x160 [20140.153662] ? path_init+0x437/0x520 [20140.153667] path_openat+0xf7/0x500 [20140.153673] ? do_raw_spin_unlock+0x75/0x190 [20140.153678] do_filp_open+0x99/0x140 [20140.153689] ? getname_flags+0x6e/0x330 [20140.153698] ? __check_object_size+0xff/0x256 [20140.153706] ? do_raw_spin_unlock+0x75/0x190 [20140.153710] ? _raw_spin_unlock+0x12/0x30 [20140.153719] do_sys_openat2+0x2b4/0x410 [20140.153723] do_sys_open+0x73/0xa0 [20140.153727] __x64_sys_openat+0x24/0x30 [20140.153731] do_syscall_64+0xc1/0x440 [20140.153737] entry_SYSCALL_64_after_hwframe+0x49/0xae [20140.153747] RIP: 0033:0x7f53425a5332 [20140.153755] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20140.153759] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20140.153765] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20140.153768] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20140.153770] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20140.153773] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20140.153775] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20140.563497] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20140.566482] after_reply+0xe2c/0x1780 [ptlrpc] [20140.577195] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20140.580428] ? __next_timer_interrupt+0x160/0x160 [20140.581998] ptlrpcd+0xa82/0xe40 [ptlrpc] [20140.583569] ? do_wait_intr+0xf0/0xf0 [20140.588155] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20140.590488] kthread+0x1d1/0x200 [20140.604451] ? set_kthread_struct+0x70/0x70 [20140.611273] ret_from_fork+0x1f/0x30 [20144.052028] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20144.057846] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20144.099250] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20144.106136] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20144.116779] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20144.120384] RIP: 0010:native_safe_halt+0x2e/0x50 [20144.125625] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20144.139377] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20144.146167] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20144.154010] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [20144.157704] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20144.161216] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20144.163850] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20144.166976] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20144.172518] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20144.176150] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20144.181490] Call Trace: [20144.186042] [20144.186960] ? show_regs.cold.9+0x22/0x2f [20144.189017] ? watchdog_timer_fn.cold.6+0x62/0x10a [20144.191371] ? __hrtimer_run_queues+0x1c2/0x4d0 [20144.193938] ? watchdog+0x40/0x40 [20144.195283] ? hrtimer_interrupt+0x128/0x2c0 [20144.197202] ? smp_apic_timer_interrupt+0xa6/0x270 [20144.199266] ? apic_timer_interrupt+0xf/0x20 [20144.201171] [20144.202277] ? native_safe_halt+0x2e/0x50 [20144.204106] kvm_wait+0xcd/0xe0 [20144.207284] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20144.212440] ? newidle_balance.isra.93+0x207/0x580 [20144.220278] do_raw_spin_lock+0x177/0x190 [20144.226933] _raw_spin_lock+0x21/0x30 [20144.232339] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20144.238196] process_one_work+0x2c8/0x700 [20144.242553] worker_thread+0x296/0x6e0 [20144.247429] ? rescuer_thread+0x570/0x570 [20144.251802] kthread+0x1d1/0x200 [20144.255136] ? set_kthread_struct+0x70/0x70 [20144.257432] ret_from_fork+0x1f/0x30 [20148.704059] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17511s! [20148.707932] Showing busy workqueues and worker pools: [20148.710697] workqueue events: flags=0x0 [20148.711887] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20148.714814] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20148.719697] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20148.722899] pending: cpu_latency_work [ptlrpc] [20148.725232] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20148.728251] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20148.733248] workqueue events_long: flags=0x0 [20148.735588] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20148.738891] pending: lu_objects_flush [obdclass] [20148.742454] workqueue events_unbound: flags=0x2 [20148.744482] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20148.747898] pending: flush_memcg_stats_dwork, idle_cull_fn [20148.751157] workqueue events_power_efficient: flags=0x80 [20148.753447] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20148.756254] pending: check_lifetime, do_cache_clean [20148.759483] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20148.762499] pending: neigh_periodic_work, neigh_periodic_work [20148.766769] workqueue mm_percpu_wq: flags=0x8 [20148.768439] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20148.771906] pending: vmstat_update, lru_add_drain_per_cpu [20148.774328] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20148.776806] pending: vmstat_update, lru_add_drain_per_cpu [20148.778327] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20148.780250] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20148.782764] workqueue xprtiod: flags=0xa [20148.784335] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20148.786847] pending: xprt_autoclose, xs_error_handle [20148.789236] workqueue ipv6_addrconf: flags=0x40008 [20148.791042] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20148.793137] pending: addrconf_verify_work [20148.794616] workqueue knbd0-recv: flags=0x1a [20148.795960] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20148.797899] in-flight: 651:recv_work [20148.799040] workqueue ptlrpc_pinger: flags=0x2 [20148.800362] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20148.802216] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20148.804127] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17511s workers=4 idle: 15979 31824 8 [20148.806341] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20139s workers=2 idle: 654 [20152.060664] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [20152.067417] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20152.101318] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20152.105952] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20152.109955] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20152.112511] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20152.120686] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20152.124082] RAX: 0000000000007dd2 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20152.126991] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20152.129647] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20152.133642] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20152.136672] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20152.140339] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20152.143801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20152.146371] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20152.149510] Call Trace: [20152.150555] [20152.151359] ? show_regs.cold.9+0x22/0x2f [20152.153288] ? watchdog_timer_fn.cold.6+0x62/0x10a [20152.155915] ? __hrtimer_run_queues+0x1c2/0x4d0 [20152.158167] ? watchdog+0x40/0x40 [20152.159340] ? hrtimer_interrupt+0x128/0x2c0 [20152.161758] ? smp_apic_timer_interrupt+0xa6/0x270 [20152.164035] ? apic_timer_interrupt+0xf/0x20 [20152.165267] [20152.165908] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20152.167743] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20152.169910] do_raw_spin_lock+0x177/0x190 [20152.171666] _raw_spin_lock+0x21/0x30 [20152.173354] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20152.175640] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20152.179379] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20152.182014] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20152.184336] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20152.187506] ? woken_wake_function+0x30/0x30 [20152.189144] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20152.192277] kthread+0x1d1/0x200 [20152.196359] ? set_kthread_struct+0x70/0x70 [20152.202497] ret_from_fork+0x1f/0x30 [20154.875712] rcu: INFO: rcu_sched self-detected stall on CPU [20154.879328] rcu: 2-....: (10260147 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [20154.887187] (t=18060449 jiffies g=274185 q=68500) [20154.889392] rcu: rcu_sched kthread starved for 17518815 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [20154.896137] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [20154.900412] rcu: RCU grace-period kthread stack dump: [20154.902218] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [20154.905444] Call Trace: [20154.906355] __schedule+0x351/0xcb0 [20154.908341] schedule+0xc0/0x180 [20154.911468] schedule_timeout+0xb4/0x190 [20154.913525] ? do_raw_spin_unlock+0x75/0x190 [20154.917987] ? __next_timer_interrupt+0x160/0x160 [20154.920175] ? prepare_to_swait_event+0xf3/0x140 [20154.921809] rcu_gp_kthread+0x8dd/0x15b0 [20154.925517] ? force_qs_rnp+0x2c0/0x2c0 [20154.928456] kthread+0x1d1/0x200 [20154.929995] ? set_kthread_struct+0x70/0x70 [20154.931311] ret_from_fork+0x1f/0x30 [20154.932274] rcu: Stack dump where RCU GP kthread last ran: [20154.933886] Sending NMI from CPU 2 to CPUs 1: [20154.935085] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [20154.936427] Sending NMI from CPU 2 to CPUs 1: [20154.939497] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [20154.940458] NMI backtrace for cpu 2 [20154.942928] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20154.950429] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20154.954433] Call Trace: [20154.955291] [20154.956119] ? dump_stack+0xbb/0x10e [20154.957858] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [20154.959569] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [20154.962382] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [20154.964297] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [20154.966454] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [20154.969267] ? rcu_dump_cpu_stacks+0x11b/0x171 [20154.971808] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [20154.977492] ? update_process_times+0x6d/0xc0 [20154.982783] ? tick_sched_handle.isra.18+0x27/0x90 [20154.986514] ? tick_sched_timer+0x53/0xb0 [20154.990614] ? __hrtimer_run_queues+0x1c2/0x4d0 [20154.995120] ? tick_sched_handle.isra.18+0x90/0x90 [20154.996579] ? hrtimer_interrupt+0x128/0x2c0 [20154.997920] ? smp_apic_timer_interrupt+0xa6/0x270 [20154.999561] ? apic_timer_interrupt+0xf/0x20 [20155.002219] [20155.004208] ? native_safe_halt+0x2e/0x50 [20155.006316] ? kvm_wait+0xcd/0xe0 [20155.007759] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20155.010382] ? do_raw_spin_lock+0x177/0x190 [20155.012521] ? _raw_spin_lock+0x21/0x30 [20155.014658] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20155.021435] ? obd_mod_free+0x46/0x140 [ptlrpc] [20155.024103] ? mdc_commit_open+0x91/0xb0 [mdc] [20155.027439] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20155.031437] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20155.033498] ? after_reply+0xe2c/0x1780 [ptlrpc] [20155.038520] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20155.040471] ? __next_timer_interrupt+0x160/0x160 [20155.042297] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [20155.044179] ? do_wait_intr+0xf0/0xf0 [20155.047210] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20155.050268] ? kthread+0x1d1/0x200 [20155.052281] ? set_kthread_struct+0x70/0x70 [20155.054055] ? ret_from_fork+0x1f/0x30 [20176.052056] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20176.056563] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20176.084140] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20176.089088] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20176.093714] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20176.097275] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20176.100852] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20176.115098] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20176.118148] RAX: 0000000000003584 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20176.121769] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [20176.124040] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20176.126260] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20176.129540] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20176.133318] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20176.136954] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20176.138641] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20176.141256] Call Trace: [20176.142596] [20176.143578] ? show_regs.cold.9+0x22/0x2f [20176.146305] ? watchdog_timer_fn.cold.6+0x62/0x10a [20176.149401] ? __hrtimer_run_queues+0x1c2/0x4d0 [20176.150771] ? watchdog+0x40/0x40 [20176.152161] ? hrtimer_interrupt+0x128/0x2c0 [20176.154331] ? smp_apic_timer_interrupt+0xa6/0x270 [20176.157231] ? apic_timer_interrupt+0xf/0x20 [20176.158662] [20176.159543] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20176.161643] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20176.163860] ? newidle_balance.isra.93+0x207/0x580 [20176.165711] do_raw_spin_lock+0x177/0x190 [20176.167084] _raw_spin_lock+0x21/0x30 [20176.168535] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20176.171256] process_one_work+0x2c8/0x700 [20176.172763] worker_thread+0x296/0x6e0 [20176.174352] ? rescuer_thread+0x570/0x570 [20176.175924] kthread+0x1d1/0x200 [20176.177006] ? set_kthread_struct+0x70/0x70 [20176.178460] ret_from_fork+0x1f/0x30 [20178.912100] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17541s! [20178.916636] Showing busy workqueues and worker pools: [20178.920367] workqueue events: flags=0x0 [20178.922071] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20178.929453] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20178.942303] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20178.948260] pending: cpu_latency_work [ptlrpc] [20178.953834] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20178.958925] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20178.964465] workqueue events_long: flags=0x0 [20178.967670] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20178.972765] pending: lu_objects_flush [obdclass] [20178.975095] workqueue events_unbound: flags=0x2 [20178.978237] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20178.982043] pending: flush_memcg_stats_dwork, idle_cull_fn [20178.985707] workqueue events_power_efficient: flags=0x80 [20178.991645] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20178.999811] pending: check_lifetime, do_cache_clean [20179.002669] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20179.009324] pending: neigh_periodic_work, neigh_periodic_work [20179.012022] workqueue mm_percpu_wq: flags=0x8 [20179.014613] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20179.021483] pending: vmstat_update, lru_add_drain_per_cpu [20179.027247] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20179.031358] pending: vmstat_update, lru_add_drain_per_cpu [20179.035132] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20179.040227] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20179.043888] workqueue xprtiod: flags=0xa [20179.047018] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20179.050330] pending: xprt_autoclose, xs_error_handle [20179.057806] workqueue ipv6_addrconf: flags=0x40008 [20179.062506] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20179.068600] pending: addrconf_verify_work [20179.072036] workqueue knbd0-recv: flags=0x1a [20179.074391] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20179.081056] in-flight: 651:recv_work [20179.083014] workqueue ptlrpc_pinger: flags=0x2 [20179.086987] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20179.090471] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20179.096363] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17542s workers=4 idle: 15979 31824 8 [20179.101956] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20169s workers=2 idle: 654 [20180.056320] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [20180.058344] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [20180.059380] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [20180.059389] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20180.059460] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20180.059466] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20180.059469] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [20180.059483] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [20180.059487] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20180.059492] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20180.059494] RDX: 0000000000000c97 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20180.059497] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20180.059499] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20180.059502] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20180.059504] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20180.059507] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20180.059509] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20180.059625] Call Trace: [20180.059631] [20180.059636] ? show_regs.cold.9+0x22/0x2f [20180.059645] ? watchdog_timer_fn.cold.6+0x62/0x10a [20180.059651] ? __hrtimer_run_queues+0x1c2/0x4d0 [20180.059655] ? watchdog+0x40/0x40 [20180.059659] ? hrtimer_interrupt+0x128/0x2c0 [20180.059662] ? smp_apic_timer_interrupt+0xa6/0x270 [20180.059670] ? apic_timer_interrupt+0xf/0x20 [20180.059675] [20180.059677] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20180.059681] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20180.059685] do_raw_spin_lock+0x177/0x190 [20180.059689] _raw_spin_lock+0x21/0x30 [20180.059695] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20180.060178] obd_mod_free+0x46/0x140 [ptlrpc] [20180.060373] mdc_commit_open+0x91/0xb0 [mdc] [20180.060407] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20180.060669] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20180.061048] after_reply+0xe2c/0x1780 [ptlrpc] [20180.061262] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20180.061494] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20180.061561] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20180.061568] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20180.061571] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20180.061582] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20180.061589] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [20180.061586] ? __next_timer_interrupt+0x160/0x160 [20180.061594] ORIG_RAX: ffffffffffffff13 [20180.061600] RAX: 0000000000002e8b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20180.061603] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20180.061605] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20180.061607] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20180.061610] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20180.061613] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20180.061616] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20180.061619] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20180.061600] ptlrpcd+0xa82/0xe40 [ptlrpc] [20180.061800] ? do_wait_intr+0xf0/0xf0 [20180.061818] Call Trace: [20180.061824] [20180.061828] ? show_regs.cold.9+0x22/0x2f [20180.061963] ? watchdog_timer_fn.cold.6+0x62/0x10a [20180.061969] ? __hrtimer_run_queues+0x1c2/0x4d0 [20180.061973] ? watchdog+0x40/0x40 [20180.061977] ? hrtimer_interrupt+0x128/0x2c0 [20180.061982] ? smp_apic_timer_interrupt+0xa6/0x270 [20180.061989] ? apic_timer_interrupt+0xf/0x20 [20180.061993] [20180.061996] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20180.062003] do_raw_spin_lock+0x177/0x190 [20180.062007] _raw_spin_lock+0x21/0x30 [20180.062012] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20180.062413] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20180.062693] kthread+0x1d1/0x200 [20180.062699] ? set_kthread_struct+0x70/0x70 [20180.062703] ret_from_fork+0x1f/0x30 [20180.067638] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20180.067728] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20180.067736] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20180.067739] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20180.067755] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20180.067759] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20180.067764] RAX: 0000000000007f84 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20180.067766] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20180.067769] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20180.067771] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20180.067774] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20180.067778] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20180.067781] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20180.067784] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20180.067793] Call Trace: [20180.067799] [20180.067805] ? show_regs.cold.9+0x22/0x2f [20180.068661] ? watchdog_timer_fn.cold.6+0x62/0x10a [20180.068669] ? __hrtimer_run_queues+0x1c2/0x4d0 [20180.068674] ? watchdog+0x40/0x40 [20180.068679] ? hrtimer_interrupt+0x128/0x2c0 [20180.068683] ? smp_apic_timer_interrupt+0xa6/0x270 [20180.068692] ? apic_timer_interrupt+0xf/0x20 [20180.068697] [20180.068701] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20180.068706] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20180.068714] do_raw_spin_lock+0x177/0x190 [20180.068720] _raw_spin_lock+0x21/0x30 [20180.068725] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20180.069885] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20180.070081] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20180.076357] ? do_raw_spin_unlock+0x75/0x190 [20180.076378] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20180.077002] ? do_wait_intr+0xf0/0xf0 [20180.077013] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20180.077301] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20180.104313] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20180.108812] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20180.113135] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20180.114926] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20180.124354] ? woken_wake_function+0x30/0x30 [20180.127441] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20180.127497] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20180.127529] mdc_intent_lock+0x353/0x7e0 [mdc] [20180.127560] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20180.127686] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20180.128061] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20180.133742] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20180.136523] lmv_intent_open+0x338/0x1150 [lmv] [20180.136566] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20180.140016] kthread+0x1d1/0x200 [20180.142938] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20180.143073] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20180.143186] ll_intent_lock+0xc5/0xaa0 [lustre] [20180.143273] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20180.146648] ? set_kthread_struct+0x70/0x70 [20180.149923] ll_lookup_it+0xe0e/0x2270 [lustre] [20180.150069] ? do_raw_spin_unlock+0x75/0x190 [20180.150078] ? _raw_spin_unlock+0x12/0x30 [20180.150086] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20180.150190] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20180.152319] ret_from_fork+0x1f/0x30 [20180.529759] ll_atomic_open+0x36a/0xd70 [lustre] [20180.531879] lookup_open+0x356/0xb40 [20180.533531] do_last+0x313/0xfc0 [20180.534952] ? nd_jump_root+0xe5/0x160 [20180.537576] ? path_init+0x437/0x520 [20180.538995] path_openat+0xf7/0x500 [20180.540241] ? do_raw_spin_unlock+0x75/0x190 [20180.542353] do_filp_open+0x99/0x140 [20180.543693] ? getname_flags+0x6e/0x330 [20180.545670] ? __check_object_size+0xff/0x256 [20180.547412] ? do_raw_spin_unlock+0x75/0x190 [20180.549193] ? _raw_spin_unlock+0x12/0x30 [20180.550891] do_sys_openat2+0x2b4/0x410 [20180.552396] do_sys_open+0x73/0xa0 [20180.553710] __x64_sys_openat+0x24/0x30 [20180.558465] do_syscall_64+0xc1/0x440 [20180.560383] entry_SYSCALL_64_after_hwframe+0x49/0xae [20180.562400] RIP: 0033:0x7f53425a5332 [20180.567558] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20180.576741] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20180.579816] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20180.584045] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20180.586420] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20180.590384] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20180.592588] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20204.059132] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20204.073222] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20204.132433] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20204.151332] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20204.160100] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20204.175751] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20204.182267] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20204.215207] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20204.222539] RAX: 0000000000007545 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20204.228825] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [20204.237938] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20204.247292] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20204.256495] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20204.265456] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20204.276148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20204.281643] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20204.290220] Call Trace: [20204.293648] [20204.297577] ? show_regs.cold.9+0x22/0x2f [20204.303553] ? watchdog_timer_fn.cold.6+0x62/0x10a [20204.310508] ? __hrtimer_run_queues+0x1c2/0x4d0 [20204.314157] ? watchdog+0x40/0x40 [20204.318307] ? hrtimer_interrupt+0x128/0x2c0 [20204.324919] ? smp_apic_timer_interrupt+0xa6/0x270 [20204.333795] ? apic_timer_interrupt+0xf/0x20 [20204.338568] [20204.341186] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20204.352601] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20204.358553] ? newidle_balance.isra.93+0x207/0x580 [20204.363405] do_raw_spin_lock+0x177/0x190 [20204.370914] _raw_spin_lock+0x21/0x30 [20204.372287] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20204.377650] process_one_work+0x2c8/0x700 [20204.385111] worker_thread+0x296/0x6e0 [20204.390293] ? rescuer_thread+0x570/0x570 [20204.393555] kthread+0x1d1/0x200 [20204.399525] ? set_kthread_struct+0x70/0x70 [20204.403866] ret_from_fork+0x1f/0x30 [20208.055616] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [20208.057011] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [20208.059018] Modules linked in: [20208.061839] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [20208.059028] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20208.063063] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20208.063071] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20208.063074] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20208.063091] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20208.063095] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20208.063099] RAX: 00000000000020b0 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20208.063101] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20208.063104] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20208.063106] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20208.063109] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20208.063112] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20208.063114] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20208.063116] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20208.063126] Call Trace: [20208.063131] [20208.063138] ? show_regs.cold.9+0x22/0x2f [20208.063148] ? watchdog_timer_fn.cold.6+0x62/0x10a [20208.063155] ? __hrtimer_run_queues+0x1c2/0x4d0 [20208.063160] ? watchdog+0x40/0x40 [20208.063163] ? hrtimer_interrupt+0x128/0x2c0 [20208.063167] ? smp_apic_timer_interrupt+0xa6/0x270 [20208.063176] ? apic_timer_interrupt+0xf/0x20 [20208.063180] [20208.063183] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20208.063190] do_raw_spin_lock+0x177/0x190 [20208.063194] _raw_spin_lock+0x21/0x30 [20208.063199] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20208.063492] ? do_raw_spin_unlock+0x75/0x190 [20208.063498] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20208.068718] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20208.068789] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20208.068796] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20208.068798] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [20208.068812] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [20208.068816] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20208.068820] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20208.068823] RDX: 0000000000001328 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20208.068825] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20208.068827] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20208.068829] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20208.068833] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20208.068836] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20208.068839] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20208.068848] Call Trace: [20208.068854] [20208.068861] ? show_regs.cold.9+0x22/0x2f [20208.068872] ? watchdog_timer_fn.cold.6+0x62/0x10a [20208.068879] ? __hrtimer_run_queues+0x1c2/0x4d0 [20208.068884] ? watchdog+0x40/0x40 [20208.068888] ? hrtimer_interrupt+0x128/0x2c0 [20208.068892] ? smp_apic_timer_interrupt+0xa6/0x270 [20208.068905] ? apic_timer_interrupt+0xf/0x20 [20208.068910] [20208.068913] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20208.068918] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20208.068922] do_raw_spin_lock+0x177/0x190 [20208.068927] _raw_spin_lock+0x21/0x30 [20208.068933] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20208.069192] obd_mod_free+0x46/0x140 [ptlrpc] [20208.069380] mdc_commit_open+0x91/0xb0 [mdc] [20208.069419] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20208.070361] ? do_wait_intr+0xf0/0xf0 [20208.070372] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20208.070586] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20208.070829] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20208.071248] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20208.071364] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20208.071404] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20208.071434] mdc_intent_lock+0x353/0x7e0 [mdc] [20208.071461] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20208.071543] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20208.072218] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20208.072416] after_reply+0xe2c/0x1780 [ptlrpc] [20208.072654] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20208.072858] ? __next_timer_interrupt+0x160/0x160 [20208.072868] ptlrpcd+0xa82/0xe40 [ptlrpc] [20208.073069] ? do_wait_intr+0xf0/0xf0 [20208.073079] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20208.073273] kthread+0x1d1/0x200 [20208.073279] ? set_kthread_struct+0x70/0x70 [20208.073284] ret_from_fork+0x1f/0x30 [20208.073499] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20208.073572] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20208.073578] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20208.073581] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20208.073591] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20208.073596] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20208.073600] RAX: 0000000000001d0c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20208.073602] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20208.073614] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20208.073616] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20208.073618] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20208.073621] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20208.073623] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20208.073625] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20208.073633] Call Trace: [20208.073637] [20208.073640] ? show_regs.cold.9+0x22/0x2f [20208.073650] ? watchdog_timer_fn.cold.6+0x62/0x10a [20208.073655] ? __hrtimer_run_queues+0x1c2/0x4d0 [20208.073659] ? watchdog+0x40/0x40 [20208.073663] ? hrtimer_interrupt+0x128/0x2c0 [20208.073667] ? smp_apic_timer_interrupt+0xa6/0x270 [20208.073673] ? apic_timer_interrupt+0xf/0x20 [20208.073678] [20208.073680] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20208.073685] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20208.073689] do_raw_spin_lock+0x177/0x190 [20208.073694] _raw_spin_lock+0x21/0x30 [20208.073698] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20208.079180] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20208.079243] lmv_intent_open+0x338/0x1150 [lmv] [20208.079285] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20208.079412] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20208.131825] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20208.132576] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20208.132807] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20208.133011] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20208.133381] ? woken_wake_function+0x30/0x30 [20208.133392] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20208.138505] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20208.138553] ll_intent_lock+0xc5/0xaa0 [lustre] [20208.140036] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20208.140126] ll_lookup_it+0xe0e/0x2270 [lustre] [20208.140288] ? do_raw_spin_unlock+0x75/0x190 [20208.140296] ? _raw_spin_unlock+0x12/0x30 [20208.140304] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20208.140332] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20208.140353] ll_atomic_open+0x36a/0xd70 [lustre] [20208.140440] lookup_open+0x356/0xb40 [20208.140450] do_last+0x313/0xfc0 [20208.140456] ? nd_jump_root+0xe5/0x160 [20208.140460] ? path_init+0x437/0x520 [20208.140465] path_openat+0xf7/0x500 [20208.140470] ? do_raw_spin_unlock+0x75/0x190 [20208.140475] do_filp_open+0x99/0x140 [20208.140482] ? getname_flags+0x6e/0x330 [20208.140487] ? __check_object_size+0xff/0x256 [20208.140721] ? do_raw_spin_unlock+0x75/0x190 [20208.140725] ? _raw_spin_unlock+0x12/0x30 [20208.140730] do_sys_openat2+0x2b4/0x410 [20208.140735] do_sys_open+0x73/0xa0 [20208.140739] __x64_sys_openat+0x24/0x30 [20208.140743] do_syscall_64+0xc1/0x440 [20208.140750] entry_SYSCALL_64_after_hwframe+0x49/0xae [20208.140757] RIP: 0033:0x7f53425a5332 [20208.140766] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20208.140770] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20208.140776] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20208.140779] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20208.140781] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20208.140783] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20208.140786] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20209.126322] kthread+0x1d1/0x200 [20209.127641] BUG: workqueue lockup - pool [20209.135850] ? set_kthread_struct+0x70/0x70 [20209.135870] ret_from_fork+0x1f/0x30 [20209.160127] cpus=0-3 flags=0x4 nice=0 stuck for 17572s! [20209.164207] Showing busy workqueues and worker pools: [20209.169025] workqueue events: flags=0x0 [20209.173885] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20209.176270] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20209.192927] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20209.199003] pending: cpu_latency_work [ptlrpc] [20209.201102] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20209.209580] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20209.220341] workqueue events_long: flags=0x0 [20209.222425] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20209.234360] pending: lu_objects_flush [obdclass] [20209.239701] workqueue events_unbound: flags=0x2 [20209.249698] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20209.251968] pending: flush_memcg_stats_dwork, idle_cull_fn [20209.258762] workqueue events_power_efficient: flags=0x80 [20209.267350] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20209.273769] pending: check_lifetime, do_cache_clean [20209.275710] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20209.279857] pending: neigh_periodic_work, neigh_periodic_work [20209.290558] workqueue mm_percpu_wq: flags=0x8 [20209.297420] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20209.305599] pending: vmstat_update, lru_add_drain_per_cpu [20209.311780] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20209.320580] pending: vmstat_update, lru_add_drain_per_cpu [20209.324711] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20209.330787] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20209.341295] workqueue xprtiod: flags=0xa [20209.350208] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20209.355060] pending: xprt_autoclose, xs_error_handle [20209.358206] workqueue ipv6_addrconf: flags=0x40008 [20209.362730] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20209.366387] pending: addrconf_verify_work [20209.368661] workqueue knbd0-recv: flags=0x1a [20209.379205] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20209.387201] in-flight: 651:recv_work [20209.389572] workqueue ptlrpc_pinger: flags=0x2 [20209.400267] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20209.406448] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20209.412508] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17572s workers=4 idle: 15979 31824 8 [20209.421494] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20199s workers=2 idle: 654 [20236.055486] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [20236.057557] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [20236.058636] Modules linked in: [20236.062899] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20236.062978] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20236.062985] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20236.062988] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [20236.063007] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [20236.063012] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20236.063016] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20236.063019] RDX: 000000000000716c RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20236.063022] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20236.063024] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20236.063026] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20236.063029] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20236.063032] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20236.063034] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20236.063042] Call Trace: [20236.063047] [20236.063053] ? show_regs.cold.9+0x22/0x2f [20236.063062] ? watchdog_timer_fn.cold.6+0x62/0x10a [20236.063068] ? __hrtimer_run_queues+0x1c2/0x4d0 [20236.063073] ? watchdog+0x40/0x40 [20236.063077] ? hrtimer_interrupt+0x128/0x2c0 [20236.063080] ? smp_apic_timer_interrupt+0xa6/0x270 [20236.063088] ? apic_timer_interrupt+0xf/0x20 [20236.063092] [20236.063095] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20236.063100] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20236.063105] do_raw_spin_lock+0x177/0x190 [20236.063110] _raw_spin_lock+0x21/0x30 [20236.063127] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20236.063285] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [20236.063405] obd_mod_free+0x46/0x140 [ptlrpc] [20236.063288] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [20236.063636] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) [20236.064157] mdc_commit_open+0x91/0xb0 [mdc] [20236.064261] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20236.064579] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20236.064807] after_reply+0xe2c/0x1780 [ptlrpc] [20236.065250] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20236.065557] ? __next_timer_interrupt+0x160/0x160 [20236.065567] ptlrpcd+0xa82/0xe40 [ptlrpc] [20236.065808] ? do_wait_intr+0xf0/0xf0 [20236.065817] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20236.066095] kthread+0x1d1/0x200 [20236.066101] ? set_kthread_struct+0x70/0x70 [20236.066105] ret_from_fork+0x1f/0x30 [20236.067403] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20236.068219] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20236.068227] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20236.068229] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20236.068244] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20236.068248] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20236.068252] RAX: 00000000000079e5 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20236.068255] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20236.068257] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20236.068259] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20236.068262] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20236.068265] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20236.068268] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20236.068270] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20236.068277] Call Trace: [20236.068281] [20236.068289] ? show_regs.cold.9+0x22/0x2f [20236.068297] ? watchdog_timer_fn.cold.6+0x62/0x10a [20236.068303] ? __hrtimer_run_queues+0x1c2/0x4d0 [20236.068307] ? watchdog+0x40/0x40 [20236.068310] ? hrtimer_interrupt+0x128/0x2c0 [20236.068323] ? smp_apic_timer_interrupt+0xa6/0x270 [20236.068331] ? apic_timer_interrupt+0xf/0x20 [20236.068335] [20236.068338] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20236.068344] do_raw_spin_lock+0x177/0x190 [20236.068348] _raw_spin_lock+0x21/0x30 [20236.068353] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20236.068665] ? do_raw_spin_unlock+0x75/0x190 [20236.068672] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20236.068879] ? do_wait_intr+0xf0/0xf0 [20236.068888] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20236.069805] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20236.070035] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20236.070668] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20236.070943] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20236.070981] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20236.071011] mdc_intent_lock+0x353/0x7e0 [mdc] [20236.071038] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20236.071666] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20236.071867] osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20236.071927] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20236.071933] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20236.071936] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20236.071950] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20236.071954] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20236.071958] RAX: 0000000000006034 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20236.071961] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20236.071964] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20236.071966] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20236.071969] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20236.071972] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20236.071974] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20236.071977] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20236.071990] Call Trace: [20236.071994] [20236.071997] ? show_regs.cold.9+0x22/0x2f [20236.072007] ? watchdog_timer_fn.cold.6+0x62/0x10a [20236.072019] ? __hrtimer_run_queues+0x1c2/0x4d0 [20236.072024] ? watchdog+0x40/0x40 [20236.072029] ? hrtimer_interrupt+0x128/0x2c0 [20236.072034] ? smp_apic_timer_interrupt+0xa6/0x270 [20236.072026] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20236.072043] ? apic_timer_interrupt+0xf/0x20 [20236.072051] [20236.072055] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20236.072060] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20236.072070] do_raw_spin_lock+0x177/0x190 [20236.072077] _raw_spin_lock+0x21/0x30 [20236.072065] lmv_intent_open+0x338/0x1150 [lmv] [20236.072213] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20236.072518] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20236.072603] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20236.072629] ll_intent_lock+0xc5/0xaa0 [lustre] [20236.072728] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20236.072814] ll_lookup_it+0xe0e/0x2270 [lustre] [20236.072899] ? do_raw_spin_unlock+0x75/0x190 [20236.072904] ? _raw_spin_unlock+0x12/0x30 [20236.072909] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20236.072936] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20236.072955] ll_atomic_open+0x36a/0xd70 [lustre] [20236.072082] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20236.073320] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20236.073472] lookup_open+0x356/0xb40 [20236.073481] do_last+0x313/0xfc0 [20236.073655] ? nd_jump_root+0xe5/0x160 [20236.073965] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20236.074206] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20236.074586] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20236.074929] ? woken_wake_function+0x30/0x30 [20236.074939] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20236.075169] kthread+0x1d1/0x200 [20236.075175] ? set_kthread_struct+0x70/0x70 [20236.075179] ret_from_fork+0x1f/0x30 [20236.096068] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20236.096160] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20236.096167] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20236.096174] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20236.099719] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20236.099734] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20236.099738] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20236.099784] RAX: 0000000000007c8f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20236.099787] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [20236.099789] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20236.099792] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20236.099794] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20236.099798] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20236.099801] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20236.099804] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20236.099814] Call Trace: [20236.099821] [20236.099826] ? show_regs.cold.9+0x22/0x2f [20236.099836] ? watchdog_timer_fn.cold.6+0x62/0x10a [20236.099844] ? __hrtimer_run_queues+0x1c2/0x4d0 [20236.099848] ? watchdog+0x40/0x40 [20236.099853] ? hrtimer_interrupt+0x128/0x2c0 [20236.099857] ? smp_apic_timer_interrupt+0xa6/0x270 [20236.099866] ? apic_timer_interrupt+0xf/0x20 [20236.099871] [20236.099873] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20236.099879] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20236.099883] ? newidle_balance.isra.93+0x207/0x580 [20236.099888] do_raw_spin_lock+0x177/0x190 [20236.099893] _raw_spin_lock+0x21/0x30 [20236.099903] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20236.100241] process_one_work+0x2c8/0x700 [20236.100248] worker_thread+0x296/0x6e0 [20236.100253] ? rescuer_thread+0x570/0x570 [20236.100256] kthread+0x1d1/0x200 [20236.100261] ? set_kthread_struct+0x70/0x70 [20236.100265] ret_from_fork+0x1f/0x30 [20236.474272] ? path_init+0x437/0x520 [20236.475487] path_openat+0xf7/0x500 [20236.476617] ? do_raw_spin_unlock+0x75/0x190 [20236.477988] do_filp_open+0x99/0x140 [20236.479223] ? getname_flags+0x6e/0x330 [20236.480367] ? __check_object_size+0xff/0x256 [20236.481620] ? do_raw_spin_unlock+0x75/0x190 [20236.482942] ? _raw_spin_unlock+0x12/0x30 [20236.484241] do_sys_openat2+0x2b4/0x410 [20236.485557] do_sys_open+0x73/0xa0 [20236.486770] __x64_sys_openat+0x24/0x30 [20236.488209] do_syscall_64+0xc1/0x440 [20236.489944] entry_SYSCALL_64_after_hwframe+0x49/0xae [20236.491507] RIP: 0033:0x7f53425a5332 [20236.492721] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20236.498836] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20236.501514] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20236.503944] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20236.507070] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20236.509599] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20236.512310] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20239.841175] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17602s! [20239.847787] Showing busy workqueues and worker pools: [20239.850547] workqueue events: flags=0x0 [20239.853374] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20239.856724] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20239.863410] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20239.867773] pending: cpu_latency_work [ptlrpc] [20239.871209] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20239.874607] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20239.879995] workqueue events_long: flags=0x0 [20239.882614] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20239.887425] pending: lu_objects_flush [obdclass] [20239.891089] workqueue events_unbound: flags=0x2 [20239.895554] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20239.899032] pending: flush_memcg_stats_dwork, idle_cull_fn [20239.903296] workqueue events_power_efficient: flags=0x80 [20239.910182] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20239.920232] pending: check_lifetime, do_cache_clean [20239.922630] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20239.928229] pending: neigh_periodic_work, neigh_periodic_work [20239.931266] workqueue mm_percpu_wq: flags=0x8 [20239.936743] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20239.942849] pending: vmstat_update, lru_add_drain_per_cpu [20239.945546] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20239.952008] pending: vmstat_update, lru_add_drain_per_cpu [20239.959511] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20239.964686] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20239.968987] workqueue xprtiod: flags=0xa [20239.970553] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20239.976246] pending: xprt_autoclose, xs_error_handle [20239.981489] workqueue ipv6_addrconf: flags=0x40008 [20239.983713] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20239.989620] pending: addrconf_verify_work [20239.991217] workqueue knbd0-recv: flags=0x1a [20239.992829] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20239.998928] in-flight: 651:recv_work [20240.001276] workqueue ptlrpc_pinger: flags=0x2 [20240.003364] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20240.005895] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20240.009221] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17602s workers=4 idle: 15979 31824 8 [20240.012979] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20230s workers=2 idle: 654 [20264.052046] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20264.055846] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20264.057019] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20264.057029] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20264.057194] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20264.057199] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20264.057202] RIP: 0010:native_safe_halt+0x2e/0x50 [20264.057347] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20264.057351] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20264.057356] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20264.057359] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20264.057361] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20264.057364] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20264.057367] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20264.057369] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20264.057372] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20264.057374] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20264.057382] Call Trace: [20264.057400] [20264.057406] ? show_regs.cold.9+0x22/0x2f [20264.057416] ? watchdog_timer_fn.cold.6+0x62/0x10a [20264.057421] ? __hrtimer_run_queues+0x1c2/0x4d0 [20264.057425] ? watchdog+0x40/0x40 [20264.057430] ? hrtimer_interrupt+0x128/0x2c0 [20264.057433] ? smp_apic_timer_interrupt+0xa6/0x270 [20264.057440] ? apic_timer_interrupt+0xf/0x20 [20264.057445] [20264.057448] ? native_safe_halt+0x2e/0x50 [20264.057452] kvm_wait+0xcd/0xe0 [20264.057460] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20264.057467] do_raw_spin_lock+0x177/0x190 [20264.057472] _raw_spin_lock+0x21/0x30 [20264.057477] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20264.057842] obd_mod_free+0x46/0x140 [ptlrpc] [20264.057996] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20264.058056] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20264.058061] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20264.058067] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20264.058205] mdc_commit_open+0x91/0xb0 [mdc] [20264.058445] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20264.058872] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20264.059079] after_reply+0xe2c/0x1780 [ptlrpc] [20264.059348] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20264.059545] ? __next_timer_interrupt+0x160/0x160 [20264.059554] ptlrpcd+0xa82/0xe40 [ptlrpc] [20264.060931] ? do_wait_intr+0xf0/0xf0 [20264.060943] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20264.061345] kthread+0x1d1/0x200 [20264.061351] ? set_kthread_struct+0x70/0x70 [20264.061355] ret_from_fork+0x1f/0x30 [20264.061483] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20264.061490] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20264.061563] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20264.061568] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20264.061570] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20264.061578] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20264.061583] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20264.061586] RAX: 000000000000090b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20264.061589] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20264.061590] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20264.061592] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20264.061594] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20264.061597] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20264.061600] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20264.061602] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20264.061610] Call Trace: [20264.061614] [20264.061616] ? show_regs.cold.9+0x22/0x2f [20264.061633] ? watchdog_timer_fn.cold.6+0x62/0x10a [20264.061640] ? __hrtimer_run_queues+0x1c2/0x4d0 [20264.061644] ? watchdog+0x40/0x40 [20264.061648] ? hrtimer_interrupt+0x128/0x2c0 [20264.061651] ? smp_apic_timer_interrupt+0xa6/0x270 [20264.061657] ? apic_timer_interrupt+0xf/0x20 [20264.061660] [20264.061663] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20264.061667] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20264.061671] do_raw_spin_lock+0x177/0x190 [20264.061675] _raw_spin_lock+0x21/0x30 [20264.061679] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20264.062922] [20264.062925] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20264.062935] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20264.062939] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20264.062942] RAX: 000000000000720c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20264.062945] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [20264.062947] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20264.062949] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20264.062952] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20264.062954] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20264.062957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20264.062960] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20264.063604] Call Trace: [20264.063611] [20264.063615] ? show_regs.cold.9+0x22/0x2f [20264.063779] ? watchdog_timer_fn.cold.6+0x62/0x10a [20264.063784] ? __hrtimer_run_queues+0x1c2/0x4d0 [20264.063788] ? watchdog+0x40/0x40 [20264.063792] ? hrtimer_interrupt+0x128/0x2c0 [20264.063796] ? smp_apic_timer_interrupt+0xa6/0x270 [20264.063803] ? apic_timer_interrupt+0xf/0x20 [20264.063808] [20264.063810] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20264.063815] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20264.063819] ? newidle_balance.isra.93+0x207/0x580 [20264.063824] do_raw_spin_lock+0x177/0x190 [20264.063829] _raw_spin_lock+0x21/0x30 [20264.063833] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20264.064069] process_one_work+0x2c8/0x700 [20264.064077] worker_thread+0x296/0x6e0 [20264.064081] ? rescuer_thread+0x570/0x570 [20264.064085] kthread+0x1d1/0x200 [20264.064090] ? set_kthread_struct+0x70/0x70 [20264.064093] ret_from_fork+0x1f/0x30 [20264.066331] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20264.073275] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20264.073553] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20264.074247] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20264.074465] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20264.117529] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20264.117545] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20264.117548] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20264.117562] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20264.117567] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20264.117573] RAX: 0000000000005e0a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20264.117575] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20264.117577] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20264.117580] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20264.117582] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20264.117585] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20264.117587] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20264.117589] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20264.117597] Call Trace: [20264.117602] [20264.117607] ? show_regs.cold.9+0x22/0x2f [20264.117616] ? watchdog_timer_fn.cold.6+0x62/0x10a [20264.123516] ? woken_wake_function+0x30/0x30 [20264.123538] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20264.124053] kthread+0x1d1/0x200 [20264.124060] ? set_kthread_struct+0x70/0x70 [20264.124073] ret_from_fork+0x1f/0x30 [20264.858374] ? __hrtimer_run_queues+0x1c2/0x4d0 [20264.864452] ? watchdog+0x40/0x40 [20264.867346] ? hrtimer_interrupt+0x128/0x2c0 [20264.871427] ? smp_apic_timer_interrupt+0xa6/0x270 [20264.874558] ? apic_timer_interrupt+0xf/0x20 [20264.876454] [20264.880047] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20264.882981] do_raw_spin_lock+0x177/0x190 [20264.888646] _raw_spin_lock+0x21/0x30 [20264.891060] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20264.893497] ? do_raw_spin_unlock+0x75/0x190 [20264.898137] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20264.899802] ? do_wait_intr+0xf0/0xf0 [20264.903128] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20264.906408] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20264.912087] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20264.917376] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20264.921926] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20264.925545] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20264.930307] mdc_intent_lock+0x353/0x7e0 [mdc] [20264.933320] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20264.940684] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20264.954594] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20264.960873] lmv_intent_open+0x338/0x1150 [lmv] [20264.962347] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20264.968009] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20264.972429] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20264.976252] ll_intent_lock+0xc5/0xaa0 [lustre] [20264.978298] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20264.981161] ll_lookup_it+0xe0e/0x2270 [lustre] [20264.984275] ? do_raw_spin_unlock+0x75/0x190 [20264.989566] ? _raw_spin_unlock+0x12/0x30 [20264.994534] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20264.997489] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20265.000359] ll_atomic_open+0x36a/0xd70 [lustre] [20265.002507] lookup_open+0x356/0xb40 [20265.003590] do_last+0x313/0xfc0 [20265.007965] ? nd_jump_root+0xe5/0x160 [20265.010295] ? path_init+0x437/0x520 [20265.012302] path_openat+0xf7/0x500 [20265.013671] ? do_raw_spin_unlock+0x75/0x190 [20265.021489] do_filp_open+0x99/0x140 [20265.022655] ? getname_flags+0x6e/0x330 [20265.023859] ? __check_object_size+0xff/0x256 [20265.027402] ? do_raw_spin_unlock+0x75/0x190 [20265.032876] ? _raw_spin_unlock+0x12/0x30 [20265.036889] do_sys_openat2+0x2b4/0x410 [20265.040367] do_sys_open+0x73/0xa0 [20265.043595] __x64_sys_openat+0x24/0x30 [20265.047480] do_syscall_64+0xc1/0x440 [20265.049526] entry_SYSCALL_64_after_hwframe+0x49/0xae [20265.052520] RIP: 0033:0x7f53425a5332 [20265.054539] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20265.067562] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20265.072360] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20265.077529] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20265.081915] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20265.086485] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20265.093571] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20270.048744] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17632s! [20270.051473] Showing busy workqueues and worker pools: [20270.056635] workqueue events: flags=0x0 [20270.061072] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20270.066930] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20270.088497] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20270.093189] pending: cpu_latency_work [ptlrpc] [20270.097262] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20270.102048] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20270.116110] workqueue events_long: flags=0x0 [20270.118148] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20270.125920] pending: lu_objects_flush [obdclass] [20270.130498] workqueue events_unbound: flags=0x2 [20270.133166] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20270.141481] pending: flush_memcg_stats_dwork, idle_cull_fn [20270.145740] workqueue events_power_efficient: flags=0x80 [20270.153648] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20270.160153] pending: check_lifetime, do_cache_clean [20270.164069] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20270.170731] pending: neigh_periodic_work, neigh_periodic_work [20270.179670] workqueue mm_percpu_wq: flags=0x8 [20270.183025] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20270.187865] pending: vmstat_update, lru_add_drain_per_cpu [20270.191124] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20270.201971] pending: vmstat_update, lru_add_drain_per_cpu [20270.204178] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20270.210832] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20270.214412] workqueue xprtiod: flags=0xa [20270.216242] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20270.222048] pending: xprt_autoclose, xs_error_handle [20270.226216] workqueue ipv6_addrconf: flags=0x40008 [20270.233111] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20270.237293] pending: addrconf_verify_work [20270.239791] workqueue knbd0-recv: flags=0x1a [20270.241309] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20270.246170] in-flight: 651:recv_work [20270.247800] workqueue ptlrpc_pinger: flags=0x2 [20270.249920] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20270.252963] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20270.256229] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17633s workers=4 idle: 15979 31824 8 [20270.260349] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20260s workers=2 idle: 654 [20292.055029] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20292.057029] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20292.057422] Modules linked in: [20292.058024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20292.058030] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20292.058115] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20292.058121] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20292.058124] RIP: 0010:native_safe_halt+0x2e/0x50 [20292.058149] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20292.058154] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20292.058158] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20292.058161] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20292.058163] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20292.058165] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20292.058168] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20292.058171] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20292.058173] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20292.058176] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20292.058184] Call Trace: [20292.058190] [20292.058196] ? show_regs.cold.9+0x22/0x2f [20292.058206] ? watchdog_timer_fn.cold.6+0x62/0x10a [20292.058212] ? __hrtimer_run_queues+0x1c2/0x4d0 [20292.058216] ? watchdog+0x40/0x40 [20292.058220] ? hrtimer_interrupt+0x128/0x2c0 [20292.058224] ? smp_apic_timer_interrupt+0xa6/0x270 [20292.058231] ? apic_timer_interrupt+0xf/0x20 [20292.058235] [20292.058238] ? native_safe_halt+0x2e/0x50 [20292.058242] kvm_wait+0xcd/0xe0 [20292.058249] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20292.058257] do_raw_spin_lock+0x177/0x190 [20292.058262] _raw_spin_lock+0x21/0x30 [20292.058267] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20292.058553] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20292.058772] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20292.058992] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20292.059212] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20292.059425] ? woken_wake_function+0x30/0x30 [20292.059434] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20292.059642] kthread+0x1d1/0x200 [20292.059648] ? set_kthread_struct+0x70/0x70 [20292.059652] ret_from_fork+0x1f/0x30 [20292.060505] Modules linked in: [20292.067951] lustre(O) [20292.070558] lustre(O) [20292.103726] osp(O) [20292.107442] osp(O) [20292.112862] ofd(O) [20292.114793] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20292.114868] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20292.114875] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20292.114878] RIP: 0010:native_safe_halt+0x2e/0x50 [20292.114893] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20292.114897] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20292.114901] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20292.114903] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20292.114907] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20292.114909] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20292.114911] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20292.114915] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20292.114917] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20292.114920] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20292.114928] Call Trace: [20292.114934] [20292.114940] ? show_regs.cold.9+0x22/0x2f [20292.114950] ? watchdog_timer_fn.cold.6+0x62/0x10a [20292.114957] ? __hrtimer_run_queues+0x1c2/0x4d0 [20292.114961] ? watchdog+0x40/0x40 [20292.114966] ? hrtimer_interrupt+0x128/0x2c0 [20292.114970] ? smp_apic_timer_interrupt+0xa6/0x270 [20292.114977] ? apic_timer_interrupt+0xf/0x20 [20292.114982] [20292.114985] ? native_safe_halt+0x2e/0x50 [20292.114989] kvm_wait+0xcd/0xe0 [20292.114997] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20292.115004] do_raw_spin_lock+0x177/0x190 [20292.115008] _raw_spin_lock+0x21/0x30 [20292.115014] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20292.115412] obd_mod_free+0x46/0x140 [ptlrpc] [20292.115612] mdc_commit_open+0x91/0xb0 [mdc] [20292.115651] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20292.115856] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20292.116056] after_reply+0xe2c/0x1780 [ptlrpc] [20292.116413] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20292.116606] ? __next_timer_interrupt+0x160/0x160 [20292.116617] ptlrpcd+0xa82/0xe40 [ptlrpc] [20292.116823] ? do_wait_intr+0xf0/0xf0 [20292.116832] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20292.117036] kthread+0x1d1/0x200 [20292.117042] ? set_kthread_struct+0x70/0x70 [20292.117046] ret_from_fork+0x1f/0x30 [20292.347867] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20292.368840] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20292.373530] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20292.376989] RIP: 0010:native_safe_halt+0x2e/0x50 [20292.378852] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20292.389513] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20292.392623] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20292.397834] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20292.405170] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20292.408563] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20292.412748] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20292.416847] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20292.421237] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20292.424255] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20292.429059] Call Trace: [20292.432908] [20292.434024] ? show_regs.cold.9+0x22/0x2f [20292.439575] ? watchdog_timer_fn.cold.6+0x62/0x10a [20292.443021] ? __hrtimer_run_queues+0x1c2/0x4d0 [20292.446058] ? watchdog+0x40/0x40 [20292.447925] ? hrtimer_interrupt+0x128/0x2c0 [20292.451051] ? smp_apic_timer_interrupt+0xa6/0x270 [20292.453831] ? apic_timer_interrupt+0xf/0x20 [20292.456272] [20292.458302] ? native_safe_halt+0x2e/0x50 [20292.461609] kvm_wait+0xcd/0xe0 [20292.465004] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20292.467708] do_raw_spin_lock+0x177/0x190 [20292.469607] _raw_spin_lock+0x21/0x30 [20292.471437] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20292.474390] ? do_raw_spin_unlock+0x75/0x190 [20292.477430] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20292.480612] ? do_wait_intr+0xf0/0xf0 [20292.486013] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20292.489614] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20292.493860] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20292.496240] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20292.499537] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20292.503068] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20292.506572] mdc_intent_lock+0x353/0x7e0 [mdc] [20292.508585] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20292.512911] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20292.519246] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20292.522568] lmv_intent_open+0x338/0x1150 [lmv] [20292.525863] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20292.529732] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20292.534245] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20292.537119] ll_intent_lock+0xc5/0xaa0 [lustre] [20292.539910] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20292.542980] ll_lookup_it+0xe0e/0x2270 [lustre] [20292.546325] ? do_raw_spin_unlock+0x75/0x190 [20292.553444] ? _raw_spin_unlock+0x12/0x30 [20292.556016] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20292.560248] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20292.564963] ll_atomic_open+0x36a/0xd70 [lustre] [20292.568105] lookup_open+0x356/0xb40 [20292.572527] do_last+0x313/0xfc0 [20292.573645] ? nd_jump_root+0xe5/0x160 [20292.578876] ? path_init+0x437/0x520 [20292.580872] path_openat+0xf7/0x500 [20292.582629] ? do_raw_spin_unlock+0x75/0x190 [20292.584874] do_filp_open+0x99/0x140 [20292.587925] ? getname_flags+0x6e/0x330 [20292.590035] ? __check_object_size+0xff/0x256 [20292.593087] ? do_raw_spin_unlock+0x75/0x190 [20292.597302] ? _raw_spin_unlock+0x12/0x30 [20292.599960] do_sys_openat2+0x2b4/0x410 [20292.602701] do_sys_open+0x73/0xa0 [20292.604577] __x64_sys_openat+0x24/0x30 [20292.606327] do_syscall_64+0xc1/0x440 [20292.608023] entry_SYSCALL_64_after_hwframe+0x49/0xae [20292.611361] RIP: 0033:0x7f53425a5332 [20292.613972] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20292.626174] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20292.629980] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20292.633901] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20292.638332] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20292.642236] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20292.644323] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20296.052473] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [20296.059545] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20296.099516] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20296.109854] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20296.116533] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20296.120452] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20296.127796] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20296.141950] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20296.148458] RAX: 0000000000007e57 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20296.152354] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [20296.156057] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20296.159919] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20296.163763] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20296.169016] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20296.178300] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20296.182426] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20296.185755] Call Trace: [20296.186605] [20296.187472] ? show_regs.cold.9+0x22/0x2f [20296.189328] ? watchdog_timer_fn.cold.6+0x62/0x10a [20296.191304] ? __hrtimer_run_queues+0x1c2/0x4d0 [20296.193037] ? watchdog+0x40/0x40 [20296.194510] ? hrtimer_interrupt+0x128/0x2c0 [20296.196837] ? smp_apic_timer_interrupt+0xa6/0x270 [20296.198894] ? apic_timer_interrupt+0xf/0x20 [20296.201044] [20296.202865] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20296.206786] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20296.209844] ? newidle_balance.isra.93+0x207/0x580 [20296.211297] do_raw_spin_lock+0x177/0x190 [20296.212531] _raw_spin_lock+0x21/0x30 [20296.215285] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20296.217227] process_one_work+0x2c8/0x700 [20296.219813] worker_thread+0x296/0x6e0 [20296.222475] ? rescuer_thread+0x570/0x570 [20296.224241] kthread+0x1d1/0x200 [20296.225275] ? set_kthread_struct+0x70/0x70 [20296.226975] ret_from_fork+0x1f/0x30 [20300.768256] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17663s! [20300.775594] Showing busy workqueues and worker pools: [20300.780670] workqueue events: flags=0x0 [20300.784731] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20300.789355] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20300.795630] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20300.799780] pending: cpu_latency_work [ptlrpc] [20300.801559] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20300.803222] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20300.807418] workqueue events_long: flags=0x0 [20300.809279] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20300.812506] pending: lu_objects_flush [obdclass] [20300.814811] workqueue events_unbound: flags=0x2 [20300.816838] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20300.819236] pending: flush_memcg_stats_dwork, idle_cull_fn [20300.822452] workqueue events_power_efficient: flags=0x80 [20300.824847] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20300.827201] pending: check_lifetime, do_cache_clean [20300.829251] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20300.832118] pending: neigh_periodic_work, neigh_periodic_work [20300.834427] workqueue mm_percpu_wq: flags=0x8 [20300.836185] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20300.838658] pending: vmstat_update, lru_add_drain_per_cpu [20300.840895] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20300.843850] pending: vmstat_update, lru_add_drain_per_cpu [20300.845826] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20300.849277] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20300.851896] workqueue xprtiod: flags=0xa [20300.854204] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20300.857492] pending: xprt_autoclose, xs_error_handle [20300.863121] workqueue ipv6_addrconf: flags=0x40008 [20300.865954] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20300.868884] pending: addrconf_verify_work [20300.872343] workqueue knbd0-recv: flags=0x1a [20300.873908] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20300.885355] in-flight: 651:recv_work [20300.890861] workqueue ptlrpc_pinger: flags=0x2 [20300.895935] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20300.898143] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20300.909631] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17663s workers=4 idle: 15979 31824 8 [20300.916385] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20291s workers=2 idle: 654 [20320.055535] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20320.057027] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20320.058058] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20320.058067] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20320.058146] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20320.058151] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20320.058155] RIP: 0010:native_safe_halt+0x2e/0x50 [20320.058168] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20320.058173] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20320.058178] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20320.058180] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20320.058183] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20320.058185] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20320.058188] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20320.058191] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20320.058195] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20320.058198] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20320.058206] Call Trace: [20320.058213] [20320.058219] ? show_regs.cold.9+0x22/0x2f [20320.058229] ? watchdog_timer_fn.cold.6+0x62/0x10a [20320.058236] ? __hrtimer_run_queues+0x1c2/0x4d0 [20320.058241] ? watchdog+0x40/0x40 [20320.058246] ? hrtimer_interrupt+0x128/0x2c0 [20320.058250] ? smp_apic_timer_interrupt+0xa6/0x270 [20320.058257] ? apic_timer_interrupt+0xf/0x20 [20320.058262] [20320.058266] ? native_safe_halt+0x2e/0x50 [20320.058270] kvm_wait+0xcd/0xe0 [20320.058278] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20320.058287] do_raw_spin_lock+0x177/0x190 [20320.058292] _raw_spin_lock+0x21/0x30 [20320.058298] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20320.058846] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20320.059206] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20320.059449] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20320.061191] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20320.061626] ? woken_wake_function+0x30/0x30 [20320.061637] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20320.062157] kthread+0x1d1/0x200 [20320.062164] ? set_kthread_struct+0x70/0x70 [20320.062169] ret_from_fork+0x1f/0x30 [20320.063615] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) [20320.068014] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20320.068098] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20320.068105] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20320.068109] RIP: 0010:native_safe_halt+0x2e/0x50 [20320.068123] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20320.068126] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20320.068130] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20320.068133] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20320.068135] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20320.068138] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20320.068140] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20320.068143] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20320.068146] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20320.068148] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20320.068156] Call Trace: [20320.068162] [20320.068168] ? show_regs.cold.9+0x22/0x2f [20320.068178] ? watchdog_timer_fn.cold.6+0x62/0x10a [20320.068185] ? __hrtimer_run_queues+0x1c2/0x4d0 [20320.068189] ? watchdog+0x40/0x40 [20320.068193] ? hrtimer_interrupt+0x128/0x2c0 [20320.068197] ? smp_apic_timer_interrupt+0xa6/0x270 [20320.068204] ? apic_timer_interrupt+0xf/0x20 [20320.068209] [20320.068211] ? native_safe_halt+0x2e/0x50 [20320.068215] kvm_wait+0xcd/0xe0 [20320.068222] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20320.068230] do_raw_spin_lock+0x177/0x190 [20320.068235] _raw_spin_lock+0x21/0x30 [20320.068240] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20320.068807] obd_mod_free+0x46/0x140 [ptlrpc] [20320.071803] fld(O) [20320.100888] mdc_commit_open+0x91/0xb0 [mdc] [20320.100938] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20320.101188] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20320.101394] after_reply+0xe2c/0x1780 [ptlrpc] [20320.101608] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20320.102004] ? __next_timer_interrupt+0x160/0x160 [20320.102015] ptlrpcd+0xa82/0xe40 [ptlrpc] [20320.102273] ? do_wait_intr+0xf0/0xf0 [20320.102283] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20320.102561] kthread+0x1d1/0x200 [20320.102568] ? set_kthread_struct+0x70/0x70 [20320.102572] ret_from_fork+0x1f/0x30 [20320.345082] ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20320.355938] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20320.359957] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20320.362815] RIP: 0010:native_safe_halt+0x2e/0x50 [20320.364687] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20320.370309] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20320.373629] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20320.376234] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20320.378032] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20320.379972] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20320.383080] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20320.385088] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20320.387450] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20320.389093] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20320.391085] Call Trace: [20320.391743] [20320.392312] ? show_regs.cold.9+0x22/0x2f [20320.393545] ? watchdog_timer_fn.cold.6+0x62/0x10a [20320.394791] ? __hrtimer_run_queues+0x1c2/0x4d0 [20320.396275] ? watchdog+0x40/0x40 [20320.397635] ? hrtimer_interrupt+0x128/0x2c0 [20320.398794] ? smp_apic_timer_interrupt+0xa6/0x270 [20320.400175] ? apic_timer_interrupt+0xf/0x20 [20320.401297] [20320.401906] ? native_safe_halt+0x2e/0x50 [20320.403147] kvm_wait+0xcd/0xe0 [20320.403976] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20320.405320] do_raw_spin_lock+0x177/0x190 [20320.406501] _raw_spin_lock+0x21/0x30 [20320.407489] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20320.409363] ? do_raw_spin_unlock+0x75/0x190 [20320.410637] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20320.412174] ? do_wait_intr+0xf0/0xf0 [20320.413157] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20320.414735] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20320.416233] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20320.417958] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20320.419576] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20320.421171] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20320.422725] mdc_intent_lock+0x353/0x7e0 [mdc] [20320.423945] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20320.425545] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20320.427618] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20320.429897] lmv_intent_open+0x338/0x1150 [lmv] [20320.431484] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20320.435212] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20320.436623] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20320.438939] ll_intent_lock+0xc5/0xaa0 [lustre] [20320.441162] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20320.443063] ll_lookup_it+0xe0e/0x2270 [lustre] [20320.444551] ? do_raw_spin_unlock+0x75/0x190 [20320.446669] ? _raw_spin_unlock+0x12/0x30 [20320.448521] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20320.451238] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20320.453923] ll_atomic_open+0x36a/0xd70 [lustre] [20320.456076] lookup_open+0x356/0xb40 [20320.457656] do_last+0x313/0xfc0 [20320.460412] ? nd_jump_root+0xe5/0x160 [20320.461598] ? path_init+0x437/0x520 [20320.463287] path_openat+0xf7/0x500 [20320.465124] ? do_raw_spin_unlock+0x75/0x190 [20320.467602] do_filp_open+0x99/0x140 [20320.468788] ? getname_flags+0x6e/0x330 [20320.470264] ? __check_object_size+0xff/0x256 [20320.473307] ? do_raw_spin_unlock+0x75/0x190 [20320.474464] ? _raw_spin_unlock+0x12/0x30 [20320.476483] do_sys_openat2+0x2b4/0x410 [20320.478277] do_sys_open+0x73/0xa0 [20320.480123] __x64_sys_openat+0x24/0x30 [20320.482819] do_syscall_64+0xc1/0x440 [20320.484351] entry_SYSCALL_64_after_hwframe+0x49/0xae [20320.486625] RIP: 0033:0x7f53425a5332 [20320.488078] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20320.498381] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20320.502405] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20320.506391] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20320.509307] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20320.515200] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20320.518590] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20328.054204] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20328.062188] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20328.102155] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20328.113157] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20328.118652] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20328.122236] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20328.128232] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20328.140888] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20328.148025] RAX: 0000000000006c9f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20328.151886] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [20328.157874] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20328.160651] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20328.163613] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20328.166855] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20328.171613] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20328.173908] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20328.176821] Call Trace: [20328.177856] [20328.178843] ? show_regs.cold.9+0x22/0x2f [20328.180341] ? watchdog_timer_fn.cold.6+0x62/0x10a [20328.186283] ? __hrtimer_run_queues+0x1c2/0x4d0 [20328.191536] ? watchdog+0x40/0x40 [20328.193077] ? hrtimer_interrupt+0x128/0x2c0 [20328.195317] ? smp_apic_timer_interrupt+0xa6/0x270 [20328.197094] ? apic_timer_interrupt+0xf/0x20 [20328.201494] [20328.202306] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20328.204166] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20328.209648] ? newidle_balance.isra.93+0x207/0x580 [20328.211251] do_raw_spin_lock+0x177/0x190 [20328.218586] _raw_spin_lock+0x21/0x30 [20328.219890] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20328.226187] process_one_work+0x2c8/0x700 [20328.231219] worker_thread+0x296/0x6e0 [20328.236980] ? rescuer_thread+0x570/0x570 [20328.240136] kthread+0x1d1/0x200 [20328.244970] ? set_kthread_struct+0x70/0x70 [20328.246405] ret_from_fork+0x1f/0x30 [20330.976264] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17693s! [20330.983663] Showing busy workqueues and worker pools: [20330.986681] workqueue events: flags=0x0 [20330.989120] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20330.993185] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20331.009514] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20331.015807] pending: cpu_latency_work [ptlrpc] [20331.023848] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20331.033018] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20331.040775] workqueue events_long: flags=0x0 [20331.043079] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20331.045847] pending: lu_objects_flush [obdclass] [20331.048683] workqueue events_unbound: flags=0x2 [20331.051377] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20331.054821] pending: flush_memcg_stats_dwork, idle_cull_fn [20331.058118] workqueue events_power_efficient: flags=0x80 [20331.061510] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20331.066057] pending: check_lifetime, do_cache_clean [20331.069933] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20331.074107] pending: neigh_periodic_work, neigh_periodic_work [20331.077701] workqueue mm_percpu_wq: flags=0x8 [20331.079188] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20331.087686] pending: vmstat_update, lru_add_drain_per_cpu [20331.092707] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20331.097143] pending: vmstat_update, lru_add_drain_per_cpu [20331.103289] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20331.108253] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20331.113347] workqueue xprtiod: flags=0xa [20331.114686] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20331.120468] pending: xprt_autoclose, xs_error_handle [20331.122352] workqueue ipv6_addrconf: flags=0x40008 [20331.129722] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20331.133049] pending: addrconf_verify_work [20331.137024] workqueue knbd0-recv: flags=0x1a [20331.138891] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20331.142616] in-flight: 651:recv_work [20331.144043] workqueue ptlrpc_pinger: flags=0x2 [20331.146618] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20331.150393] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20331.156714] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17694s workers=4 idle: 15979 31824 8 [20331.159425] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20321s workers=2 idle: 654 [20334.878656] rcu: INFO: rcu_sched self-detected stall on CPU [20334.884056] rcu: 2-....: (10393903 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [20334.889645] (t=18240452 jiffies g=274185 q=68535) [20334.894776] rcu: rcu_sched kthread starved for 17698818 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [20334.905566] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [20334.911133] rcu: RCU grace-period kthread stack dump: [20334.914285] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [20334.920587] Call Trace: [20334.921906] __schedule+0x351/0xcb0 [20334.923200] schedule+0xc0/0x180 [20334.924288] schedule_timeout+0xb4/0x190 [20334.925360] ? do_raw_spin_unlock+0x75/0x190 [20334.926602] ? __next_timer_interrupt+0x160/0x160 [20334.928032] ? prepare_to_swait_event+0xf3/0x140 [20334.929915] rcu_gp_kthread+0x8dd/0x15b0 [20334.931705] ? force_qs_rnp+0x2c0/0x2c0 [20334.935194] kthread+0x1d1/0x200 [20334.936285] ? set_kthread_struct+0x70/0x70 [20334.938615] ret_from_fork+0x1f/0x30 [20334.940246] rcu: Stack dump where RCU GP kthread last ran: [20334.943092] Sending NMI from CPU 2 to CPUs 3: [20334.945866] NMI backtrace for cpu 3 [20334.945873] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20334.945876] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20334.945877] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20334.945879] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20334.945881] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 [20334.945884] RAX: 000000000000711a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20334.945886] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20334.945887] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20334.945889] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20334.945893] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20334.945895] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20334.945896] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20334.945898] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20334.945899] Call Trace: [20334.945900] [20334.945901] ? show_regs.cold.9+0x22/0x2f [20334.945902] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [20334.945904] ? nmi_cpu_backtrace_handler+0x15/0x40 [20334.945905] ? nmi_handle+0xbd/0x210 [20334.945906] ? default_do_nmi+0x4b/0x1b0 [20334.945907] ? do_nmi+0x280/0x380 [20334.945908] ? end_repeat_nmi+0x16/0x1f [20334.945909] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20334.945911] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20334.945912] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20334.945913] [20334.945914] do_raw_spin_lock+0x177/0x190 [20334.945915] _raw_spin_lock+0x21/0x30 [20334.945916] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20334.945918] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20334.945919] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20334.945920] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20334.945921] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20334.945922] ? woken_wake_function+0x30/0x30 [20334.945923] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20334.945924] kthread+0x1d1/0x200 [20334.945925] ? set_kthread_struct+0x70/0x70 [20334.945926] ret_from_fork+0x1f/0x30 [20334.946735] Sending NMI from CPU 2 to CPUs 1: [20335.041869] NMI backtrace for cpu 1 [20335.041876] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20335.041878] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20335.041880] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [20335.041882] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [20335.041884] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [20335.041886] RAX: 00000000000012c1 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20335.041888] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20335.041889] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20335.041890] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20335.041891] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20335.041893] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20335.041894] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20335.041895] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20335.041896] Call Trace: [20335.041897] [20335.041898] ? show_regs.cold.9+0x22/0x2f [20335.041898] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [20335.041899] ? nmi_cpu_backtrace_handler+0x15/0x40 [20335.041900] ? nmi_handle+0xbd/0x210 [20335.041901] ? default_do_nmi+0x4b/0x1b0 [20335.041902] ? do_nmi+0x280/0x380 [20335.041903] ? end_repeat_nmi+0x16/0x1f [20335.041904] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [20335.041905] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [20335.041906] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [20335.041907] [20335.041908] do_raw_spin_lock+0x177/0x190 [20335.041909] _raw_spin_lock+0x21/0x30 [20335.041910] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20335.041910] ? do_raw_spin_unlock+0x75/0x190 [20335.041911] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20335.041912] ? do_wait_intr+0xf0/0xf0 [20335.041913] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20335.041914] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20335.041915] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20335.041916] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20335.041917] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20335.041918] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20335.041919] mdc_intent_lock+0x353/0x7e0 [mdc] [20335.041920] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20335.041921] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20335.041922] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20335.041923] lmv_intent_open+0x338/0x1150 [lmv] [20335.041924] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20335.041925] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20335.041925] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20335.041926] ll_intent_lock+0xc5/0xaa0 [lustre] [20335.041927] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20335.041928] ll_lookup_it+0xe0e/0x2270 [lustre] [20335.041929] ? do_raw_spin_unlock+0x75/0x190 [20335.041930] ? _raw_spin_unlock+0x12/0x30 [20335.041931] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20335.041932] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20335.041932] ll_atomic_open+0x36a/0xd70 [lustre] [20335.041933] lookup_open+0x356/0xb40 [20335.041934] do_last+0x313/0xfc0 [20335.041935] ? nd_jump_root+0xe5/0x160 [20335.041936] ? path_init+0x437/0x520 [20335.041937] path_openat+0xf7/0x500 [20335.041938] ? do_raw_spin_unlock+0x75/0x190 [20335.041938] do_filp_open+0x99/0x140 [20335.041939] ? getname_flags+0x6e/0x330 [20335.041940] ? __check_object_size+0xff/0x256 [20335.041941] ? do_raw_spin_unlock+0x75/0x190 [20335.041942] ? _raw_spin_unlock+0x12/0x30 [20335.041943] do_sys_openat2+0x2b4/0x410 [20335.041944] do_sys_open+0x73/0xa0 [20335.041944] __x64_sys_openat+0x24/0x30 [20335.041945] do_syscall_64+0xc1/0x440 [20335.041946] entry_SYSCALL_64_after_hwframe+0x49/0xae [20335.041947] RIP: 0033:0x7f53425a5332 [20335.041948] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20335.041950] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20335.041952] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20335.041953] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20335.041954] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20335.041955] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20335.041957] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20335.042204] NMI backtrace for cpu 2 [20335.295356] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20335.307961] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20335.311019] Call Trace: [20335.312579] [20335.313255] ? dump_stack+0xbb/0x10e [20335.316594] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [20335.318439] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [20335.320186] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [20335.322621] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [20335.327155] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [20335.333270] ? rcu_dump_cpu_stacks+0x11b/0x171 [20335.336820] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [20335.342039] ? update_process_times+0x6d/0xc0 [20335.345133] ? tick_sched_handle.isra.18+0x27/0x90 [20335.352240] ? tick_sched_timer+0x53/0xb0 [20335.355205] ? __hrtimer_run_queues+0x1c2/0x4d0 [20335.358124] ? tick_sched_handle.isra.18+0x90/0x90 [20335.361673] ? hrtimer_interrupt+0x128/0x2c0 [20335.363177] ? smp_apic_timer_interrupt+0xa6/0x270 [20335.365591] ? apic_timer_interrupt+0xf/0x20 [20335.367350] [20335.368127] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20335.370890] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20335.378197] ? do_raw_spin_lock+0x177/0x190 [20335.380478] ? _raw_spin_lock+0x21/0x30 [20335.383267] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20335.387570] ? obd_mod_free+0x46/0x140 [ptlrpc] [20335.392178] ? mdc_commit_open+0x91/0xb0 [mdc] [20335.397404] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20335.400168] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20335.405252] ? after_reply+0xe2c/0x1780 [ptlrpc] [20335.410842] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20335.415580] ? __next_timer_interrupt+0x160/0x160 [20335.416996] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [20335.418334] ? do_wait_intr+0xf0/0xf0 [20335.430070] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20335.434487] ? kthread+0x1d1/0x200 [20335.438264] ? set_kthread_struct+0x70/0x70 [20335.442865] ? ret_from_fork+0x1f/0x30 [20356.052032] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [20356.054568] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20356.076000] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20356.081153] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20356.085292] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20356.088554] RIP: 0010:native_safe_halt+0x2e/0x50 [20356.091151] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20356.099697] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20356.103323] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20356.105837] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [20356.108350] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20356.111249] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20356.114436] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20356.119070] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20356.127691] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20356.129604] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20356.134414] Call Trace: [20356.136484] [20356.137491] ? show_regs.cold.9+0x22/0x2f [20356.138922] ? watchdog_timer_fn.cold.6+0x62/0x10a [20356.145878] ? __hrtimer_run_queues+0x1c2/0x4d0 [20356.151572] ? watchdog+0x40/0x40 [20356.155542] ? hrtimer_interrupt+0x128/0x2c0 [20356.157161] ? smp_apic_timer_interrupt+0xa6/0x270 [20356.161098] ? apic_timer_interrupt+0xf/0x20 [20356.162306] [20356.163006] ? native_safe_halt+0x2e/0x50 [20356.173487] kvm_wait+0xcd/0xe0 [20356.179463] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20356.187896] ? newidle_balance.isra.93+0x207/0x580 [20356.196202] do_raw_spin_lock+0x177/0x190 [20356.198671] _raw_spin_lock+0x21/0x30 [20356.199952] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20356.211876] process_one_work+0x2c8/0x700 [20356.217280] worker_thread+0x296/0x6e0 [20356.221864] ? rescuer_thread+0x570/0x570 [20356.223911] kthread+0x1d1/0x200 [20356.225435] ? set_kthread_struct+0x70/0x70 [20356.228901] ret_from_fork+0x1f/0x30 [20360.055066] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20360.057019] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20360.058207] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20360.058216] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20360.058610] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20360.058615] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20360.058618] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20360.058633] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20360.058637] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20360.058641] RAX: 0000000000005016 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20360.058644] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20360.058646] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20360.058648] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20360.058650] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20360.058653] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20360.058655] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20360.058657] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20360.058665] Call Trace: [20360.058670] [20360.058675] ? show_regs.cold.9+0x22/0x2f [20360.058684] ? watchdog_timer_fn.cold.6+0x62/0x10a [20360.058690] ? __hrtimer_run_queues+0x1c2/0x4d0 [20360.058694] ? watchdog+0x40/0x40 [20360.058698] ? hrtimer_interrupt+0x128/0x2c0 [20360.058702] ? smp_apic_timer_interrupt+0xa6/0x270 [20360.058710] ? apic_timer_interrupt+0xf/0x20 [20360.058715] [20360.058717] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20360.058721] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20360.058727] do_raw_spin_lock+0x177/0x190 [20360.058732] _raw_spin_lock+0x21/0x30 [20360.058737] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20360.059207] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20360.059650] Modules linked in: [20360.059735] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20360.059972] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20360.060192] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20360.060672] ? woken_wake_function+0x30/0x30 [20360.060680] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20360.061311] kthread+0x1d1/0x200 [20360.061318] ? set_kthread_struct+0x70/0x70 [20360.061435] ret_from_fork+0x1f/0x30 [20360.065874] Modules linked in: [20360.071586] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [20360.095455] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) [20360.102575] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20360.102703] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20360.102711] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20360.102714] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20360.102728] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20360.102732] RSP: 0018:ffffbb5381503500 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [20360.102736] RAX: 0000000000007f1f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20360.102739] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20360.102742] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20360.102744] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20360.102747] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20360.102750] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20360.102753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20360.102756] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20360.102764] Call Trace: [20360.102770] [20360.102776] ? show_regs.cold.9+0x22/0x2f [20360.102784] ? watchdog_timer_fn.cold.6+0x62/0x10a [20360.102790] ? __hrtimer_run_queues+0x1c2/0x4d0 [20360.102801] ? watchdog+0x40/0x40 [20360.102805] ? hrtimer_interrupt+0x128/0x2c0 [20360.102810] ? smp_apic_timer_interrupt+0xa6/0x270 [20360.102818] ? apic_timer_interrupt+0xf/0x20 [20360.102823] [20360.102826] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20360.102833] do_raw_spin_lock+0x177/0x190 [20360.102838] _raw_spin_lock+0x21/0x30 [20360.102843] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20360.103870] ? do_raw_spin_unlock+0x75/0x190 [20360.103879] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20360.105472] ? do_wait_intr+0xf0/0xf0 [20360.105480] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20360.105630] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20360.105774] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20360.107238] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20360.107373] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20360.107428] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20360.107459] mdc_intent_lock+0x353/0x7e0 [mdc] [20360.107487] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20360.107661] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20360.108171] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20360.108202] lmv_intent_open+0x338/0x1150 [lmv] [20360.108232] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20360.108700] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20360.108783] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20360.108807] ll_intent_lock+0xc5/0xaa0 [lustre] [20360.108891] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20360.114059] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20360.114106] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20360.114112] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20360.114116] RIP: 0010:native_safe_halt+0x2e/0x50 [20360.114138] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20360.114143] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [20360.114151] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20360.114153] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20360.114155] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20360.114158] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20360.114160] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20360.114163] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20360.114166] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20360.114168] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20360.114177] Call Trace: [20360.114182] [20360.114188] ? show_regs.cold.9+0x22/0x2f [20360.114198] ? watchdog_timer_fn.cold.6+0x62/0x10a [20360.114205] ? __hrtimer_run_queues+0x1c2/0x4d0 [20360.114209] ? watchdog+0x40/0x40 [20360.114214] ? hrtimer_interrupt+0x128/0x2c0 [20360.114218] ? smp_apic_timer_interrupt+0xa6/0x270 [20360.114225] ? apic_timer_interrupt+0xf/0x20 [20360.114229] [20360.114232] ? native_safe_halt+0x2e/0x50 [20360.114236] kvm_wait+0xcd/0xe0 [20360.114243] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20360.114250] do_raw_spin_lock+0x177/0x190 [20360.114255] _raw_spin_lock+0x21/0x30 [20360.114260] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20360.114709] obd_mod_free+0x46/0x140 [ptlrpc] [20360.115052] mdc_commit_open+0x91/0xb0 [mdc] [20360.115089] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20360.115385] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20360.115915] after_reply+0xe2c/0x1780 [ptlrpc] [20360.116297] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20360.116496] ? __next_timer_interrupt+0x160/0x160 [20360.116506] ptlrpcd+0xa82/0xe40 [ptlrpc] [20360.116835] ? do_wait_intr+0xf0/0xf0 [20360.116844] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20360.117078] kthread+0x1d1/0x200 [20360.117084] ? set_kthread_struct+0x70/0x70 [20360.117088] ret_from_fork+0x1f/0x30 [20360.444994] ll_lookup_it+0xe0e/0x2270 [lustre] [20360.447018] ? do_raw_spin_unlock+0x75/0x190 [20360.449246] ? _raw_spin_unlock+0x12/0x30 [20360.450808] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20360.453192] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20360.455379] ll_atomic_open+0x36a/0xd70 [lustre] [20360.457352] lookup_open+0x356/0xb40 [20360.458837] do_last+0x313/0xfc0 [20360.460005] ? nd_jump_root+0xe5/0x160 [20360.461213] ? path_init+0x437/0x520 [20360.462369] path_openat+0xf7/0x500 [20360.463867] ? do_raw_spin_unlock+0x75/0x190 [20360.465620] do_filp_open+0x99/0x140 [20360.467211] ? getname_flags+0x6e/0x330 [20360.468910] ? __check_object_size+0xff/0x256 [20360.470856] ? do_raw_spin_unlock+0x75/0x190 [20360.472639] ? _raw_spin_unlock+0x12/0x30 [20360.473887] do_sys_openat2+0x2b4/0x410 [20360.475750] do_sys_open+0x73/0xa0 [20360.476870] __x64_sys_openat+0x24/0x30 [20360.478435] do_syscall_64+0xc1/0x440 [20360.479815] entry_SYSCALL_64_after_hwframe+0x49/0xae [20360.481433] RIP: 0033:0x7f53425a5332 [20360.482713] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20360.488726] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20360.491419] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20360.494401] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20360.496760] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20360.499541] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20360.502008] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20361.183074] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17724s! [20361.194555] Showing busy workqueues and worker pools: [20361.197429] workqueue events: flags=0x0 [20361.198712] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20361.202392] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20361.212077] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20361.215211] pending: cpu_latency_work [ptlrpc] [20361.220600] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20361.224003] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20361.228192] workqueue events_long: flags=0x0 [20361.230541] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20361.232976] pending: lu_objects_flush [obdclass] [20361.238558] workqueue events_unbound: flags=0x2 [20361.240145] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20361.248048] pending: flush_memcg_stats_dwork, idle_cull_fn [20361.249696] workqueue events_power_efficient: flags=0x80 [20361.251110] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20361.253236] pending: check_lifetime, do_cache_clean [20361.255318] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20361.258142] pending: neigh_periodic_work, neigh_periodic_work [20361.260529] workqueue mm_percpu_wq: flags=0x8 [20361.262321] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20361.269155] pending: vmstat_update, lru_add_drain_per_cpu [20361.272745] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20361.280076] pending: vmstat_update, lru_add_drain_per_cpu [20361.287351] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20361.292818] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20361.298637] workqueue xprtiod: flags=0xa [20361.300028] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20361.309093] pending: xprt_autoclose, xs_error_handle [20361.312860] workqueue ipv6_addrconf: flags=0x40008 [20361.319367] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20361.324342] pending: addrconf_verify_work [20361.328595] workqueue knbd0-recv: flags=0x1a [20361.332886] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20361.339514] in-flight: 651:recv_work [20361.345635] workqueue ptlrpc_pinger: flags=0x2 [20361.347704] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20361.351715] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20361.357656] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17724s workers=4 idle: 15979 31824 8 [20361.362024] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20351s workers=2 idle: 654 [20388.052024] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20388.054661] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20388.055360] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20388.057023] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20388.057328] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20388.057397] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20388.057404] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20388.057406] RIP: 0010:native_safe_halt+0x2e/0x50 [20388.057420] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20388.057425] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [20388.057429] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20388.057432] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20388.057434] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20388.057436] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20388.057439] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20388.057442] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20388.057445] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20388.057463] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20388.057471] Call Trace: [20388.057477] [20388.057482] ? show_regs.cold.9+0x22/0x2f [20388.057492] ? watchdog_timer_fn.cold.6+0x62/0x10a [20388.057499] ? __hrtimer_run_queues+0x1c2/0x4d0 [20388.057504] ? watchdog+0x40/0x40 [20388.057509] ? hrtimer_interrupt+0x128/0x2c0 [20388.057513] ? smp_apic_timer_interrupt+0xa6/0x270 [20388.057520] ? apic_timer_interrupt+0xf/0x20 [20388.057526] [20388.057529] ? native_safe_halt+0x2e/0x50 [20388.057533] kvm_wait+0xcd/0xe0 [20388.057540] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20388.057548] do_raw_spin_lock+0x177/0x190 [20388.057553] _raw_spin_lock+0x21/0x30 [20388.057559] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20388.057900] obd_mod_free+0x46/0x140 [ptlrpc] [20388.058195] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20388.058201] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20388.058260] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20388.058266] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20388.058268] RIP: 0010:native_safe_halt+0x2e/0x50 [20388.058275] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20388.058279] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20388.058283] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20388.058286] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20388.058288] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20388.058290] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20388.058292] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20388.058295] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20388.058298] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20388.058301] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20388.058308] Call Trace: [20388.058311] [20388.058314] ? show_regs.cold.9+0x22/0x2f [20388.058321] ? watchdog_timer_fn.cold.6+0x62/0x10a [20388.058325] ? __hrtimer_run_queues+0x1c2/0x4d0 [20388.058329] ? watchdog+0x40/0x40 [20388.058333] ? hrtimer_interrupt+0x128/0x2c0 [20388.058337] ? smp_apic_timer_interrupt+0xa6/0x270 [20388.058342] ? apic_timer_interrupt+0xf/0x20 [20388.058347] [20388.058350] ? native_safe_halt+0x2e/0x50 [20388.058354] kvm_wait+0xcd/0xe0 [20388.058360] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20388.058365] do_raw_spin_lock+0x177/0x190 [20388.058369] _raw_spin_lock+0x21/0x30 [20388.058373] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20388.058650] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20388.058855] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20388.059132] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20388.059601] mdc_commit_open+0x91/0xb0 [mdc] [20388.059738] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20388.059946] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20388.060252] after_reply+0xe2c/0x1780 [ptlrpc] [20388.061242] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20388.061509] ? __next_timer_interrupt+0x160/0x160 [20388.061521] ptlrpcd+0xa82/0xe40 [ptlrpc] [20388.061865] ? do_wait_intr+0xf0/0xf0 [20388.061874] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20388.062185] kthread+0x1d1/0x200 [20388.062191] ? set_kthread_struct+0x70/0x70 [20388.062195] ret_from_fork+0x1f/0x30 [20388.062295] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20388.062551] ? woken_wake_function+0x30/0x30 [20388.062560] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20388.062773] kthread+0x1d1/0x200 [20388.062778] ? set_kthread_struct+0x70/0x70 [20388.062782] ret_from_fork+0x1f/0x30 [20388.075092] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20388.075106] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20388.075112] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20388.078159] Modules linked in: lustre(O) osp(O) [20388.081842] [20388.097381] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20388.097521] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20388.097529] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20388.097533] RIP: 0010:native_safe_halt+0x2e/0x50 [20388.097548] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20388.097552] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20388.097556] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20388.097558] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20388.097560] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20388.097562] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20388.097565] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20388.097568] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20388.097570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20388.097573] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20388.097582] Call Trace: [20388.097587] [20388.097592] ? show_regs.cold.9+0x22/0x2f [20388.097603] ? watchdog_timer_fn.cold.6+0x62/0x10a [20388.097610] ? __hrtimer_run_queues+0x1c2/0x4d0 [20388.097615] ? watchdog+0x40/0x40 [20388.097619] ? hrtimer_interrupt+0x128/0x2c0 [20388.097624] ? smp_apic_timer_interrupt+0xa6/0x270 [20388.097631] ? apic_timer_interrupt+0xf/0x20 [20388.097635] [20388.097639] ? native_safe_halt+0x2e/0x50 [20388.097643] kvm_wait+0xcd/0xe0 [20388.097650] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20388.097660] do_raw_spin_lock+0x177/0x190 [20388.097664] _raw_spin_lock+0x21/0x30 [20388.097669] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20388.097939] ? do_raw_spin_unlock+0x75/0x190 [20388.097946] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20388.098155] ? do_wait_intr+0xf0/0xf0 [20388.098165] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20388.098366] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20388.098704] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20388.098980] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20388.099105] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20388.103203] RIP: 0010:native_safe_halt+0x2e/0x50 [20388.103227] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20388.103230] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20388.103236] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20388.103238] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [20388.103240] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20388.103242] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20388.103244] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20388.103247] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20388.103250] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20388.103252] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20388.103261] Call Trace: [20388.103268] [20388.103274] ? show_regs.cold.9+0x22/0x2f [20388.103284] ? watchdog_timer_fn.cold.6+0x62/0x10a [20388.103291] ? __hrtimer_run_queues+0x1c2/0x4d0 [20388.103296] ? watchdog+0x40/0x40 [20388.103300] ? hrtimer_interrupt+0x128/0x2c0 [20388.103304] ? smp_apic_timer_interrupt+0xa6/0x270 [20388.103310] ? apic_timer_interrupt+0xf/0x20 [20388.103314] [20388.103316] ? native_safe_halt+0x2e/0x50 [20388.103320] kvm_wait+0xcd/0xe0 [20388.103327] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20388.103334] ? newidle_balance.isra.93+0x207/0x580 [20388.103342] do_raw_spin_lock+0x177/0x190 [20388.103347] _raw_spin_lock+0x21/0x30 [20388.103351] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20388.103627] process_one_work+0x2c8/0x700 [20388.103635] worker_thread+0x296/0x6e0 [20388.103639] ? rescuer_thread+0x570/0x570 [20388.103642] kthread+0x1d1/0x200 [20388.103646] ? set_kthread_struct+0x70/0x70 [20388.103650] ret_from_fork+0x1f/0x30 [20388.538331] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20388.540397] mdc_intent_lock+0x353/0x7e0 [mdc] [20388.542756] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20388.545217] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20388.547976] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20388.550892] lmv_intent_open+0x338/0x1150 [lmv] [20388.552954] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20388.555519] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20388.557993] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20388.559906] ll_intent_lock+0xc5/0xaa0 [lustre] [20388.562234] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20388.564683] ll_lookup_it+0xe0e/0x2270 [lustre] [20388.567250] ? do_raw_spin_unlock+0x75/0x190 [20388.569162] ? _raw_spin_unlock+0x12/0x30 [20388.571112] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20388.573216] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20388.575336] ll_atomic_open+0x36a/0xd70 [lustre] [20388.577247] lookup_open+0x356/0xb40 [20388.578788] do_last+0x313/0xfc0 [20388.580153] ? nd_jump_root+0xe5/0x160 [20388.581688] ? path_init+0x437/0x520 [20388.583160] path_openat+0xf7/0x500 [20388.584684] ? do_raw_spin_unlock+0x75/0x190 [20388.586557] do_filp_open+0x99/0x140 [20388.588088] ? getname_flags+0x6e/0x330 [20388.589698] ? __check_object_size+0xff/0x256 [20388.591433] ? do_raw_spin_unlock+0x75/0x190 [20388.593248] ? _raw_spin_unlock+0x12/0x30 [20388.594895] do_sys_openat2+0x2b4/0x410 [20388.596968] do_sys_open+0x73/0xa0 [20388.598506] __x64_sys_openat+0x24/0x30 [20388.600541] do_syscall_64+0xc1/0x440 [20388.602107] entry_SYSCALL_64_after_hwframe+0x49/0xae [20388.604195] RIP: 0033:0x7f53425a5332 [20388.605775] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20388.613938] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20388.617112] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20388.619858] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20388.623095] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20388.626289] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20388.629185] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20391.398050] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17754s! [20391.401982] Showing busy workqueues and worker pools: [20391.408183] workqueue events: flags=0x0 [20391.413130] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20391.420865] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20391.441575] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20391.454141] pending: cpu_latency_work [ptlrpc] [20391.461466] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20391.474784] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20391.485913] workqueue events_long: flags=0x0 [20391.497320] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20391.503724] pending: lu_objects_flush [obdclass] [20391.509559] workqueue events_unbound: flags=0x2 [20391.516015] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20391.520686] pending: flush_memcg_stats_dwork, idle_cull_fn [20391.525936] workqueue events_power_efficient: flags=0x80 [20391.528689] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20391.536957] pending: check_lifetime, do_cache_clean [20391.538885] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20391.545840] pending: neigh_periodic_work, neigh_periodic_work [20391.560505] workqueue mm_percpu_wq: flags=0x8 [20391.561760] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20391.565189] pending: vmstat_update, lru_add_drain_per_cpu [20391.566896] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20391.575372] pending: vmstat_update, lru_add_drain_per_cpu [20391.582683] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20391.589329] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20391.593763] workqueue xprtiod: flags=0xa [20391.599279] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20391.605242] pending: xprt_autoclose, xs_error_handle [20391.610319] workqueue ipv6_addrconf: flags=0x40008 [20391.613673] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20391.617915] pending: addrconf_verify_work [20391.626553] workqueue knbd0-recv: flags=0x1a [20391.629905] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20391.635574] in-flight: 651:recv_work [20391.639489] workqueue ptlrpc_pinger: flags=0x2 [20391.642072] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20391.646219] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20391.648796] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17754s workers=4 idle: 15979 31824 8 [20391.651924] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20382s workers=2 idle: 654 [20416.052320] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20416.056645] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [20416.058524] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20416.058890] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20416.059560] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20416.059567] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) [20416.059602] rpcsec_gss_krb5 [20416.059606] ksocklnd(O) [20416.059609] auth_rpcgss [20416.059611] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss [20416.059645] nfsv4 [20416.059648] nfsv4 [20416.059652] dns_resolver [20416.059654] dns_resolver [20416.059658] intel_rapl_msr [20416.059659] intel_rapl_msr [20416.059673] intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20416.059738] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20416.059745] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20416.059935] intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20416.059963] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20416.059975] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20416.059978] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [20416.060006] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [20416.060010] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20416.060014] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20416.060017] RDX: 000000000000709d RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20416.060019] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20416.060021] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20416.060024] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20416.060027] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20416.060031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20416.060033] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20416.060041] Call Trace: [20416.060054] [20416.060059] ? show_regs.cold.9+0x22/0x2f [20416.060067] ? watchdog_timer_fn.cold.6+0x62/0x10a [20416.060072] ? __hrtimer_run_queues+0x1c2/0x4d0 [20416.060075] ? watchdog+0x40/0x40 [20416.060078] ? hrtimer_interrupt+0x128/0x2c0 [20416.060080] ? smp_apic_timer_interrupt+0xa6/0x270 [20416.060088] ? apic_timer_interrupt+0xf/0x20 [20416.060092] [20416.060094] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20416.060097] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20416.060100] do_raw_spin_lock+0x177/0x190 [20416.060103] _raw_spin_lock+0x21/0x30 [20416.060107] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20416.060354] obd_mod_free+0x46/0x140 [ptlrpc] [20416.060558] mdc_commit_open+0x91/0xb0 [mdc] [20416.060601] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20416.060804] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20416.061014] after_reply+0xe2c/0x1780 [ptlrpc] [20416.061216] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20416.061474] ? __next_timer_interrupt+0x160/0x160 [20416.061484] ptlrpcd+0xa82/0xe40 [ptlrpc] [20416.061688] ? do_wait_intr+0xf0/0xf0 [20416.061698] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20416.061897] kthread+0x1d1/0x200 [20416.061904] ? set_kthread_struct+0x70/0x70 [20416.061907] ret_from_fork+0x1f/0x30 [20416.059751] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20416.064152] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20416.064168] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20416.064172] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20416.064176] RAX: 0000000000007716 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20416.064179] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [20416.064181] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20416.064182] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20416.064184] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20416.064188] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20416.064191] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20416.064193] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20416.064203] Call Trace: [20416.064208] [20416.064213] ? show_regs.cold.9+0x22/0x2f [20416.064221] ? watchdog_timer_fn.cold.6+0x62/0x10a [20416.064227] ? __hrtimer_run_queues+0x1c2/0x4d0 [20416.064232] ? watchdog+0x40/0x40 [20416.064236] ? hrtimer_interrupt+0x128/0x2c0 [20416.064240] ? smp_apic_timer_interrupt+0xa6/0x270 [20416.064248] ? apic_timer_interrupt+0xf/0x20 [20416.064251] [20416.064255] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20416.064260] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20416.064264] ? newidle_balance.isra.93+0x207/0x580 [20416.064270] do_raw_spin_lock+0x177/0x190 [20416.064276] _raw_spin_lock+0x21/0x30 [20416.064281] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20416.066156] process_one_work+0x2c8/0x700 [20416.066164] worker_thread+0x296/0x6e0 [20416.066169] ? rescuer_thread+0x570/0x570 [20416.066173] kthread+0x1d1/0x200 [20416.066177] ? set_kthread_struct+0x70/0x70 [20416.066181] ret_from_fork+0x1f/0x30 [20416.079625] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20416.079708] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20416.079715] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20416.079719] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20416.079732] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20416.079736] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20416.079740] RAX: 0000000000007e90 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20416.079743] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20416.079745] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20416.079748] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20416.079750] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20416.079754] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20416.079757] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20416.079759] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20416.079768] Call Trace: [20416.079774] [20416.079780] ? show_regs.cold.9+0x22/0x2f [20416.079789] ? watchdog_timer_fn.cold.6+0x62/0x10a [20416.079795] ? __hrtimer_run_queues+0x1c2/0x4d0 [20416.079800] ? watchdog+0x40/0x40 [20416.079805] ? hrtimer_interrupt+0x128/0x2c0 [20416.079809] ? smp_apic_timer_interrupt+0xa6/0x270 [20416.079818] ? apic_timer_interrupt+0xf/0x20 [20416.079823] [20416.079826] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20416.079833] do_raw_spin_lock+0x177/0x190 [20416.079839] _raw_spin_lock+0x21/0x30 [20416.079844] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20416.080332] ? do_raw_spin_unlock+0x75/0x190 [20416.080338] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20416.086351] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20416.086427] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20416.086435] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20416.086438] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20416.086451] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20416.086456] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20416.086460] RAX: 0000000000006597 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20416.086462] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20416.086464] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20416.086466] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20416.086468] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20416.086471] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20416.086474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20416.086476] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20416.086487] Call Trace: [20416.086492] [20416.086498] ? show_regs.cold.9+0x22/0x2f [20416.086508] ? watchdog_timer_fn.cold.6+0x62/0x10a [20416.086515] ? __hrtimer_run_queues+0x1c2/0x4d0 [20416.086520] ? watchdog+0x40/0x40 [20416.086524] ? hrtimer_interrupt+0x128/0x2c0 [20416.086529] ? smp_apic_timer_interrupt+0xa6/0x270 [20416.086537] ? apic_timer_interrupt+0xf/0x20 [20416.086541] [20416.086544] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20416.086548] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20416.086554] do_raw_spin_lock+0x177/0x190 [20416.086559] _raw_spin_lock+0x21/0x30 [20416.086564] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20416.090494] ? do_wait_intr+0xf0/0xf0 [20416.090521] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20416.090837] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20416.091061] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20416.091269] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20416.091390] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20416.091433] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20416.091460] mdc_intent_lock+0x353/0x7e0 [mdc] [20416.091487] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20416.091571] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20416.091779] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20416.091810] lmv_intent_open+0x338/0x1150 [lmv] [20416.091841] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20416.091926] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20416.100796] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20416.114278] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20416.114328] ll_intent_lock+0xc5/0xaa0 [lustre] [20416.114446] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20416.117405] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20416.119538] ll_lookup_it+0xe0e/0x2270 [lustre] [20416.119661] ? do_raw_spin_unlock+0x75/0x190 [20416.119670] ? _raw_spin_unlock+0x12/0x30 [20416.119678] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20416.119703] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20416.119722] ll_atomic_open+0x36a/0xd70 [lustre] [20416.121660] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20416.127050] lookup_open+0x356/0xb40 [20416.127077] do_last+0x313/0xfc0 [20416.127083] ? nd_jump_root+0xe5/0x160 [20416.127088] ? path_init+0x437/0x520 [20416.127093] path_openat+0xf7/0x500 [20416.127098] ? do_raw_spin_unlock+0x75/0x190 [20416.127106] do_filp_open+0x99/0x140 [20416.127113] ? getname_flags+0x6e/0x330 [20416.127118] ? __check_object_size+0xff/0x256 [20416.127126] ? do_raw_spin_unlock+0x75/0x190 [20416.127129] ? _raw_spin_unlock+0x12/0x30 [20416.127136] do_sys_openat2+0x2b4/0x410 [20416.127243] do_sys_open+0x73/0xa0 [20416.127255] __x64_sys_openat+0x24/0x30 [20416.128233] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20416.128535] ? woken_wake_function+0x30/0x30 [20416.128543] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20416.128761] kthread+0x1d1/0x200 [20416.128949] ? set_kthread_struct+0x70/0x70 [20416.128953] ret_from_fork+0x1f/0x30 [20416.983510] do_syscall_64+0xc1/0x440 [20416.984856] entry_SYSCALL_64_after_hwframe+0x49/0xae [20416.986614] RIP: 0033:0x7f53425a5332 [20416.987732] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20416.999233] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20417.004103] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20417.007297] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20417.011510] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20417.015117] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20417.018785] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20422.111179] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17785s! [20422.115847] Showing busy workqueues and worker pools: [20422.123402] workqueue events: flags=0x0 [20422.126188] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20422.135403] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20422.147813] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20422.152959] pending: cpu_latency_work [ptlrpc] [20422.160968] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20422.168919] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20422.179592] workqueue events_long: flags=0x0 [20422.183936] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20422.192201] pending: lu_objects_flush [obdclass] [20422.193724] workqueue events_unbound: flags=0x2 [20422.201787] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20422.203777] pending: flush_memcg_stats_dwork, idle_cull_fn [20422.213520] workqueue events_power_efficient: flags=0x80 [20422.219490] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20422.225952] pending: check_lifetime, do_cache_clean [20422.230743] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20422.239369] pending: neigh_periodic_work, neigh_periodic_work [20422.247099] workqueue mm_percpu_wq: flags=0x8 [20422.250472] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20422.255837] pending: vmstat_update, lru_add_drain_per_cpu [20422.263973] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20422.271225] pending: vmstat_update, lru_add_drain_per_cpu [20422.276427] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20422.282771] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20422.292018] workqueue xprtiod: flags=0xa [20422.294497] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20422.302767] pending: xprt_autoclose, xs_error_handle [20422.309454] workqueue ipv6_addrconf: flags=0x40008 [20422.316793] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20422.322348] pending: addrconf_verify_work [20422.329671] workqueue knbd0-recv: flags=0x1a [20422.334762] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20422.344433] in-flight: 651:recv_work [20422.345764] workqueue ptlrpc_pinger: flags=0x2 [20422.352179] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20422.359012] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20422.368064] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17785s workers=4 idle: 15979 31824 8 [20422.374736] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20412s workers=2 idle: 654 [20444.059540] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [20444.060076] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [20444.060817] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [20444.060824] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20444.060897] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20444.060903] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20444.060906] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20444.060920] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20444.060924] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20444.060929] RAX: 0000000000007137 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20444.060932] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20444.060934] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20444.060936] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20444.060939] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20444.060942] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20444.060945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20444.060947] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20444.060956] Call Trace: [20444.060961] [20444.060966] ? show_regs.cold.9+0x22/0x2f [20444.060975] ? watchdog_timer_fn.cold.6+0x62/0x10a [20444.060982] ? __hrtimer_run_queues+0x1c2/0x4d0 [20444.060986] ? watchdog+0x40/0x40 [20444.060990] ? hrtimer_interrupt+0x128/0x2c0 [20444.060994] ? smp_apic_timer_interrupt+0xa6/0x270 [20444.061002] ? apic_timer_interrupt+0xf/0x20 [20444.061008] [20444.061010] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20444.061017] do_raw_spin_lock+0x177/0x190 [20444.061022] _raw_spin_lock+0x21/0x30 [20444.061028] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20444.061287] ? do_raw_spin_unlock+0x75/0x190 [20444.061293] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20444.061517] ? do_wait_intr+0xf0/0xf0 [20444.061526] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20444.061718] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20444.061930] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20444.062144] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20444.062258] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20444.062294] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20444.062324] mdc_intent_lock+0x353/0x7e0 [mdc] [20444.062354] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20444.062438] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20444.062657] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20444.062687] lmv_intent_open+0x338/0x1150 [lmv] [20444.062717] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20444.062809] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20444.062899] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20444.062923] ll_intent_lock+0xc5/0xaa0 [lustre] [20444.063015] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20444.063105] ll_lookup_it+0xe0e/0x2270 [lustre] [20444.063197] ? do_raw_spin_unlock+0x75/0x190 [20444.063203] ? _raw_spin_unlock+0x12/0x30 [20444.063209] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20444.063235] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20444.063256] ll_atomic_open+0x36a/0xd70 [lustre] [20444.063346] lookup_open+0x356/0xb40 [20444.063355] do_last+0x313/0xfc0 [20444.063361] ? nd_jump_root+0xe5/0x160 [20444.063366] ? path_init+0x437/0x520 [20444.063370] path_openat+0xf7/0x500 [20444.063376] ? do_raw_spin_unlock+0x75/0x190 [20444.063380] do_filp_open+0x99/0x140 [20444.063386] ? getname_flags+0x6e/0x330 [20444.063391] ? __check_object_size+0xff/0x256 [20444.063399] ? do_raw_spin_unlock+0x75/0x190 [20444.063403] ? _raw_spin_unlock+0x12/0x30 [20444.063408] do_sys_openat2+0x2b4/0x410 [20444.063411] do_sys_open+0x73/0xa0 [20444.063415] __x64_sys_openat+0x24/0x30 [20444.063419] do_syscall_64+0xc1/0x440 [20444.063427] entry_SYSCALL_64_after_hwframe+0x49/0xae [20444.063433] RIP: 0033:0x7f53425a5332 [20444.063440] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20444.063444] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20444.063449] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20444.063451] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20444.063453] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20444.063455] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20444.063457] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20444.063971] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20444.064038] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20444.064044] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20444.064046] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20444.064055] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20444.067927] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20444.068010] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20444.068018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20444.068021] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [20444.068035] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [20444.068039] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20444.068044] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20444.068047] RDX: 00000000000033a9 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20444.068049] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20444.068052] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20444.068054] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20444.068057] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20444.068060] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20444.068063] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20444.068072] Call Trace: [20444.068077] [20444.068082] ? show_regs.cold.9+0x22/0x2f [20444.068092] ? watchdog_timer_fn.cold.6+0x62/0x10a [20444.068098] ? __hrtimer_run_queues+0x1c2/0x4d0 [20444.068102] ? watchdog+0x40/0x40 [20444.068106] ? hrtimer_interrupt+0x128/0x2c0 [20444.068111] ? smp_apic_timer_interrupt+0xa6/0x270 [20444.068118] ? apic_timer_interrupt+0xf/0x20 [20444.068123] [20444.068126] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20444.068131] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20444.068136] do_raw_spin_lock+0x177/0x190 [20444.068140] _raw_spin_lock+0x21/0x30 [20444.068147] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20444.076851] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20444.076866] RAX: 0000000000007f7f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20444.076869] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20444.076871] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20444.076873] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20444.076876] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20444.076880] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20444.076883] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20444.076886] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20444.076895] Call Trace: [20444.076902] [20444.076909] ? show_regs.cold.9+0x22/0x2f [20444.099708] obd_mod_free+0x46/0x140 [ptlrpc] [20444.099983] mdc_commit_open+0x91/0xb0 [mdc] [20444.100019] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20444.117607] ? watchdog_timer_fn.cold.6+0x62/0x10a [20444.123316] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20444.123596] after_reply+0xe2c/0x1780 [ptlrpc] [20444.123868] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20444.128094] ? __hrtimer_run_queues+0x1c2/0x4d0 [20444.128113] ? watchdog+0x40/0x40 [20444.128119] ? hrtimer_interrupt+0x128/0x2c0 [20444.128124] ? smp_apic_timer_interrupt+0xa6/0x270 [20444.128132] ? apic_timer_interrupt+0xf/0x20 [20444.128136] [20444.128141] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20444.128149] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20444.128157] do_raw_spin_lock+0x177/0x190 [20444.128164] _raw_spin_lock+0x21/0x30 [20444.128169] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20444.141496] ? __next_timer_interrupt+0x160/0x160 [20444.141518] ptlrpcd+0xa82/0xe40 [ptlrpc] [20444.141922] ? do_wait_intr+0xf0/0xf0 [20444.141932] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20444.142606] kthread+0x1d1/0x200 [20444.142613] ? set_kthread_struct+0x70/0x70 [20444.142618] ret_from_fork+0x1f/0x30 [20444.651357] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20444.653005] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20444.654818] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20444.656249] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20444.657795] ? woken_wake_function+0x30/0x30 [20444.658910] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20444.660436] kthread+0x1d1/0x200 [20444.661272] ? set_kthread_struct+0x70/0x70 [20444.662390] ret_from_fork+0x1f/0x30 [20448.052023] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20448.053859] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20448.067534] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20448.070320] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20448.072869] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20448.074937] RIP: 0010:native_safe_halt+0x2e/0x50 [20448.076208] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20448.081245] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20448.083562] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20448.085579] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [20448.087651] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20448.090223] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20448.092524] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20448.094784] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20448.097000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20448.098798] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20448.100603] Call Trace: [20448.101500] [20448.102042] ? show_regs.cold.9+0x22/0x2f [20448.103381] ? watchdog_timer_fn.cold.6+0x62/0x10a [20448.105318] ? __hrtimer_run_queues+0x1c2/0x4d0 [20448.106824] ? watchdog+0x40/0x40 [20448.107913] ? hrtimer_interrupt+0x128/0x2c0 [20448.109165] ? smp_apic_timer_interrupt+0xa6/0x270 [20448.110970] ? apic_timer_interrupt+0xf/0x20 [20448.112226] [20448.113232] ? native_safe_halt+0x2e/0x50 [20448.114474] kvm_wait+0xcd/0xe0 [20448.115370] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20448.117057] ? newidle_balance.isra.93+0x207/0x580 [20448.118779] do_raw_spin_lock+0x177/0x190 [20448.120107] _raw_spin_lock+0x21/0x30 [20448.120980] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20448.122936] process_one_work+0x2c8/0x700 [20448.124067] worker_thread+0x296/0x6e0 [20448.125164] ? rescuer_thread+0x570/0x570 [20448.126163] kthread+0x1d1/0x200 [20448.127190] ? set_kthread_struct+0x70/0x70 [20448.128552] ret_from_fork+0x1f/0x30 [20452.833036] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17815s! [20452.839942] Showing busy workqueues and worker pools: [20452.842390] workqueue events: flags=0x0 [20452.843753] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20452.845862] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20452.850778] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20452.854027] pending: cpu_latency_work [ptlrpc] [20452.856115] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20452.858138] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20452.862275] workqueue events_long: flags=0x0 [20452.863552] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20452.866090] pending: lu_objects_flush [obdclass] [20452.868742] workqueue events_unbound: flags=0x2 [20452.870224] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20452.871980] pending: flush_memcg_stats_dwork, idle_cull_fn [20452.875511] workqueue events_power_efficient: flags=0x80 [20452.878291] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20452.883386] pending: check_lifetime, do_cache_clean [20452.887611] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20452.892793] pending: neigh_periodic_work, neigh_periodic_work [20452.905875] workqueue mm_percpu_wq: flags=0x8 [20452.909739] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20452.913750] pending: vmstat_update, lru_add_drain_per_cpu [20452.918556] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20452.922631] pending: vmstat_update, lru_add_drain_per_cpu [20452.926183] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20452.930381] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20452.933926] workqueue xprtiod: flags=0xa [20452.935901] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20452.939530] pending: xprt_autoclose, xs_error_handle [20452.945890] workqueue ipv6_addrconf: flags=0x40008 [20452.949273] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20452.953951] pending: addrconf_verify_work [20452.956656] workqueue knbd0-recv: flags=0x1a [20452.958078] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20452.960310] in-flight: 651:recv_work [20452.964150] workqueue ptlrpc_pinger: flags=0x2 [20452.966728] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20452.971854] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20452.975594] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17815s workers=4 idle: 15979 31824 8 [20452.980300] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20443s workers=2 idle: 654 [20472.056306] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [20472.058997] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [20472.060188] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20472.060261] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20472.060267] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20472.060270] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20472.060284] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20472.060289] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20472.060293] RAX: 0000000000007d29 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20472.060296] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20472.060298] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20472.060301] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20472.060303] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20472.060307] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20472.060310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20472.060312] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20472.060321] Call Trace: [20472.060327] [20472.060334] ? show_regs.cold.9+0x22/0x2f [20472.060343] ? watchdog_timer_fn.cold.6+0x62/0x10a [20472.060350] ? __hrtimer_run_queues+0x1c2/0x4d0 [20472.060355] ? watchdog+0x40/0x40 [20472.060359] ? hrtimer_interrupt+0x128/0x2c0 [20472.060365] ? smp_apic_timer_interrupt+0xa6/0x270 [20472.060373] ? apic_timer_interrupt+0xf/0x20 [20472.060378] [20472.060381] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20472.060389] do_raw_spin_lock+0x177/0x190 [20472.060394] _raw_spin_lock+0x21/0x30 [20472.060399] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20472.061127] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [20472.061134] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20472.061202] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20472.061207] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20472.061209] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [20472.061218] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [20472.061222] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20472.061227] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20472.061229] RDX: 0000000000000f6b RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20472.061231] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20472.061233] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20472.061236] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20472.061238] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20472.061241] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20472.061244] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20472.061251] Call Trace: [20472.061255] [20472.061258] ? show_regs.cold.9+0x22/0x2f [20472.061266] ? watchdog_timer_fn.cold.6+0x62/0x10a [20472.061272] ? __hrtimer_run_queues+0x1c2/0x4d0 [20472.061276] ? watchdog+0x40/0x40 [20472.061281] ? hrtimer_interrupt+0x128/0x2c0 [20472.061285] ? smp_apic_timer_interrupt+0xa6/0x270 [20472.061292] ? apic_timer_interrupt+0xf/0x20 [20472.061297] [20472.061299] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20472.061304] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20472.061308] do_raw_spin_lock+0x177/0x190 [20472.061312] _raw_spin_lock+0x21/0x30 [20472.061318] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20472.061546] obd_mod_free+0x46/0x140 [ptlrpc] [20472.061752] mdc_commit_open+0x91/0xb0 [mdc] [20472.061788] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20472.062010] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20472.062208] after_reply+0xe2c/0x1780 [ptlrpc] [20472.062402] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20472.062602] ? __next_timer_interrupt+0x160/0x160 [20472.062611] ptlrpcd+0xa82/0xe40 [ptlrpc] [20472.062819] ? do_wait_intr+0xf0/0xf0 [20472.062829] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20472.063052] kthread+0x1d1/0x200 [20472.063059] ? set_kthread_struct+0x70/0x70 [20472.063062] ret_from_fork+0x1f/0x30 [20472.068249] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [20472.091619] ? do_raw_spin_unlock+0x75/0x190 [20472.100935] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20472.101021] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20472.101030] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20472.101033] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20472.101049] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20472.101054] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20472.101061] RAX: 0000000000004993 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20472.101064] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20472.101066] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20472.101068] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20472.101071] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20472.101074] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20472.101077] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20472.101079] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20472.101128] Call Trace: [20472.101135] [20472.101140] ? show_regs.cold.9+0x22/0x2f [20472.101149] ? watchdog_timer_fn.cold.6+0x62/0x10a [20472.101156] ? __hrtimer_run_queues+0x1c2/0x4d0 [20472.101161] ? watchdog+0x40/0x40 [20472.101166] ? hrtimer_interrupt+0x128/0x2c0 [20472.101170] ? smp_apic_timer_interrupt+0xa6/0x270 [20472.101178] ? apic_timer_interrupt+0xf/0x20 [20472.101183] [20472.101186] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20472.101190] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20472.101197] do_raw_spin_lock+0x177/0x190 [20472.101203] _raw_spin_lock+0x21/0x30 [20472.101208] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20472.105815] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20472.110814] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20472.119329] ? do_wait_intr+0xf0/0xf0 [20472.119354] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20472.119637] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20472.119994] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20472.120215] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20472.120407] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20472.120458] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20472.120494] mdc_intent_lock+0x353/0x7e0 [mdc] [20472.120532] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20472.120640] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20472.120895] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20472.120923] lmv_intent_open+0x338/0x1150 [lmv] [20472.120957] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20472.124765] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20472.127915] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20472.128105] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20472.128138] ll_intent_lock+0xc5/0xaa0 [lustre] [20472.128227] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20472.137116] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20472.140711] ll_lookup_it+0xe0e/0x2270 [lustre] [20472.144120] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20472.144422] ? woken_wake_function+0x30/0x30 [20472.144433] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20472.144638] kthread+0x1d1/0x200 [20472.144644] ? set_kthread_struct+0x70/0x70 [20472.144648] ret_from_fork+0x1f/0x30 [20472.425199] ? do_raw_spin_unlock+0x75/0x190 [20472.429504] ? _raw_spin_unlock+0x12/0x30 [20472.434670] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20472.436231] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20472.440193] ll_atomic_open+0x36a/0xd70 [lustre] [20472.441616] lookup_open+0x356/0xb40 [20472.445616] do_last+0x313/0xfc0 [20472.447344] ? nd_jump_root+0xe5/0x160 [20472.448760] ? path_init+0x437/0x520 [20472.450213] path_openat+0xf7/0x500 [20472.451424] ? do_raw_spin_unlock+0x75/0x190 [20472.452863] do_filp_open+0x99/0x140 [20472.454292] ? getname_flags+0x6e/0x330 [20472.455805] ? __check_object_size+0xff/0x256 [20472.457451] ? do_raw_spin_unlock+0x75/0x190 [20472.458819] ? _raw_spin_unlock+0x12/0x30 [20472.460447] do_sys_openat2+0x2b4/0x410 [20472.461720] do_sys_open+0x73/0xa0 [20472.463113] __x64_sys_openat+0x24/0x30 [20472.464696] do_syscall_64+0xc1/0x440 [20472.466090] entry_SYSCALL_64_after_hwframe+0x49/0xae [20472.468344] RIP: 0033:0x7f53425a5332 [20472.469664] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20472.477144] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20472.479446] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20472.483511] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20472.487421] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20472.491124] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20472.494425] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20480.052024] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20480.054143] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20480.074619] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20480.079055] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20480.082769] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20480.085527] RIP: 0010:native_safe_halt+0x2e/0x50 [20480.087364] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20480.096108] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20480.098721] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20480.101577] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [20480.104624] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20480.107197] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20480.109835] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20480.113531] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20480.116732] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20480.119323] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20480.122669] Call Trace: [20480.124023] [20480.124708] ? show_regs.cold.9+0x22/0x2f [20480.126035] ? watchdog_timer_fn.cold.6+0x62/0x10a [20480.127587] ? __hrtimer_run_queues+0x1c2/0x4d0 [20480.129266] ? watchdog+0x40/0x40 [20480.130342] ? hrtimer_interrupt+0x128/0x2c0 [20480.131794] ? smp_apic_timer_interrupt+0xa6/0x270 [20480.133446] ? apic_timer_interrupt+0xf/0x20 [20480.134977] [20480.135766] ? native_safe_halt+0x2e/0x50 [20480.137419] kvm_wait+0xcd/0xe0 [20480.138676] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20480.140528] ? newidle_balance.isra.93+0x207/0x580 [20480.142484] do_raw_spin_lock+0x177/0x190 [20480.144271] _raw_spin_lock+0x21/0x30 [20480.145853] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20480.148922] process_one_work+0x2c8/0x700 [20480.150838] worker_thread+0x296/0x6e0 [20480.152383] ? rescuer_thread+0x570/0x570 [20480.154084] kthread+0x1d1/0x200 [20480.156055] ? set_kthread_struct+0x70/0x70 [20480.157460] ret_from_fork+0x1f/0x30 [20483.039778] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17845s! [20483.045278] Showing busy workqueues and worker pools: [20483.048712] workqueue events: flags=0x0 [20483.051287] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20483.055217] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20483.062787] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20483.067033] pending: cpu_latency_work [ptlrpc] [20483.069928] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20483.073471] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20483.078321] workqueue events_long: flags=0x0 [20483.080750] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20483.084161] pending: lu_objects_flush [obdclass] [20483.087048] workqueue events_unbound: flags=0x2 [20483.090219] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20483.092740] pending: flush_memcg_stats_dwork, idle_cull_fn [20483.096182] workqueue events_power_efficient: flags=0x80 [20483.097727] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20483.100402] pending: check_lifetime, do_cache_clean [20483.104827] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20483.111240] pending: neigh_periodic_work, neigh_periodic_work [20483.114954] workqueue mm_percpu_wq: flags=0x8 [20483.116254] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20483.122137] pending: vmstat_update, lru_add_drain_per_cpu [20483.127612] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20483.132346] pending: vmstat_update, lru_add_drain_per_cpu [20483.137220] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20483.142594] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20483.146211] workqueue xprtiod: flags=0xa [20483.148645] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20483.152691] pending: xprt_autoclose, xs_error_handle [20483.156121] workqueue ipv6_addrconf: flags=0x40008 [20483.160275] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20483.165710] pending: addrconf_verify_work [20483.168925] workqueue knbd0-recv: flags=0x1a [20483.172075] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20483.176804] in-flight: 651:recv_work [20483.180102] workqueue ptlrpc_pinger: flags=0x2 [20483.183989] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20483.194346] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20483.201106] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17846s workers=4 idle: 15979 31824 8 [20483.204296] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20473s workers=2 idle: 654 [20500.055037] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [20500.058030] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [20500.058163] Modules linked in: [20500.058654] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [20500.058661] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20500.058741] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20500.058747] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20500.058750] RIP: 0010:native_safe_halt+0x2e/0x50 [20500.058763] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20500.058767] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20500.058771] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20500.058773] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20500.058775] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20500.058777] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20500.058779] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20500.058782] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20500.058785] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20500.058787] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20500.058796] Call Trace: [20500.058803] [20500.058809] ? show_regs.cold.9+0x22/0x2f [20500.058820] ? watchdog_timer_fn.cold.6+0x62/0x10a [20500.058827] ? __hrtimer_run_queues+0x1c2/0x4d0 [20500.058838] ? watchdog+0x40/0x40 [20500.058847] ? hrtimer_interrupt+0x128/0x2c0 [20500.058855] ? smp_apic_timer_interrupt+0xa6/0x270 [20500.058868] ? apic_timer_interrupt+0xf/0x20 [20500.058878] [20500.058881] ? native_safe_halt+0x2e/0x50 [20500.058892] kvm_wait+0xcd/0xe0 [20500.058912] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20500.058924] do_raw_spin_lock+0x177/0x190 [20500.058932] _raw_spin_lock+0x21/0x30 [20500.058941] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20500.059566] obd_mod_free+0x46/0x140 [ptlrpc] [20500.059782] mdc_commit_open+0x91/0xb0 [mdc] [20500.059819] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20500.060143] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20500.060358] after_reply+0xe2c/0x1780 [ptlrpc] [20500.060558] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20500.060759] ? __next_timer_interrupt+0x160/0x160 [20500.060768] ptlrpcd+0xa82/0xe40 [ptlrpc] [20500.060995] ? do_wait_intr+0xf0/0xf0 [20500.061004] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20500.061224] kthread+0x1d1/0x200 [20500.061229] ? set_kthread_struct+0x70/0x70 [20500.061233] ret_from_fork+0x1f/0x30 [20500.061698] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [20500.062809] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20500.062893] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20500.062899] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20500.062903] RIP: 0010:native_safe_halt+0x2e/0x50 [20500.062914] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20500.062918] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20500.062922] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20500.062925] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20500.062927] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20500.062933] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20500.062935] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20500.062938] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20500.062942] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20500.062944] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20500.062952] Call Trace: [20500.062956] [20500.062960] ? show_regs.cold.9+0x22/0x2f [20500.062969] ? watchdog_timer_fn.cold.6+0x62/0x10a [20500.062974] ? __hrtimer_run_queues+0x1c2/0x4d0 [20500.062983] ? watchdog+0x40/0x40 [20500.062987] ? hrtimer_interrupt+0x128/0x2c0 [20500.062991] ? smp_apic_timer_interrupt+0xa6/0x270 [20500.062998] ? apic_timer_interrupt+0xf/0x20 [20500.063002] [20500.063005] ? native_safe_halt+0x2e/0x50 [20500.063009] kvm_wait+0xcd/0xe0 [20500.063016] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20500.063024] do_raw_spin_lock+0x177/0x190 [20500.063028] _raw_spin_lock+0x21/0x30 [20500.063032] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20500.063544] ? do_raw_spin_unlock+0x75/0x190 [20500.063550] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20500.063761] ? do_wait_intr+0xf0/0xf0 [20500.063770] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20500.064043] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20500.064554] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20500.064925] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20500.065088] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20500.065127] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20500.065163] mdc_intent_lock+0x353/0x7e0 [mdc] [20500.065197] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20500.065492] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20500.065903] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20500.065940] lmv_intent_open+0x338/0x1150 [lmv] [20500.065986] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20500.066095] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20500.066183] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20500.066205] ll_intent_lock+0xc5/0xaa0 [lustre] [20500.072194] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20500.102554] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20500.109583] [20500.109595] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20500.109606] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20500.109610] RIP: 0010:native_safe_halt+0x2e/0x50 [20500.109625] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20500.109629] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20500.109635] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20500.109637] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20500.109639] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20500.109642] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20500.109645] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20500.109648] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20500.109651] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20500.109653] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20500.109662] Call Trace: [20500.109753] [20500.109760] ? show_regs.cold.9+0x22/0x2f [20500.109769] ? watchdog_timer_fn.cold.6+0x62/0x10a [20500.109775] ? __hrtimer_run_queues+0x1c2/0x4d0 [20500.109780] ? watchdog+0x40/0x40 [20500.109784] ? hrtimer_interrupt+0x128/0x2c0 [20500.109854] ? smp_apic_timer_interrupt+0xa6/0x270 [20500.109862] ? apic_timer_interrupt+0xf/0x20 [20500.109867] [20500.109870] ? native_safe_halt+0x2e/0x50 [20500.109875] kvm_wait+0xcd/0xe0 [20500.109883] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20500.109893] do_raw_spin_lock+0x177/0x190 [20500.109898] _raw_spin_lock+0x21/0x30 [20500.109903] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20500.110751] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20500.111025] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20500.111563] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20500.116956] ll_lookup_it+0xe0e/0x2270 [lustre] [20500.119418] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20500.119921] ? woken_wake_function+0x30/0x30 [20500.119931] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20500.120217] kthread+0x1d1/0x200 [20500.120223] ? set_kthread_struct+0x70/0x70 [20500.120227] ret_from_fork+0x1f/0x30 [20500.583679] ? do_raw_spin_unlock+0x75/0x190 [20500.585176] ? _raw_spin_unlock+0x12/0x30 [20500.586910] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20500.588405] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20500.592744] ll_atomic_open+0x36a/0xd70 [lustre] [20500.594991] lookup_open+0x356/0xb40 [20500.596183] do_last+0x313/0xfc0 [20500.599740] ? nd_jump_root+0xe5/0x160 [20500.600873] ? path_init+0x437/0x520 [20500.603311] path_openat+0xf7/0x500 [20500.606930] ? do_raw_spin_unlock+0x75/0x190 [20500.609995] do_filp_open+0x99/0x140 [20500.611880] ? getname_flags+0x6e/0x330 [20500.613529] ? __check_object_size+0xff/0x256 [20500.616584] ? do_raw_spin_unlock+0x75/0x190 [20500.620891] ? _raw_spin_unlock+0x12/0x30 [20500.622572] do_sys_openat2+0x2b4/0x410 [20500.625462] do_sys_open+0x73/0xa0 [20500.627093] __x64_sys_openat+0x24/0x30 [20500.628811] do_syscall_64+0xc1/0x440 [20500.631303] entry_SYSCALL_64_after_hwframe+0x49/0xae [20500.633985] RIP: 0033:0x7f53425a5332 [20500.635458] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20500.643305] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20500.648976] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20500.652168] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20500.655318] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20500.658543] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20500.662208] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20508.052474] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20508.057852] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20508.099690] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20508.103255] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20508.106002] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20508.108461] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20508.110106] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20508.115690] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20508.119105] RAX: 0000000000006844 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20508.121335] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [20508.123718] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20508.127009] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20508.129516] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20508.132080] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20508.134636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20508.136706] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20508.139568] Call Trace: [20508.140421] [20508.141111] ? show_regs.cold.9+0x22/0x2f [20508.142604] ? watchdog_timer_fn.cold.6+0x62/0x10a [20508.144037] ? __hrtimer_run_queues+0x1c2/0x4d0 [20508.145326] ? watchdog+0x40/0x40 [20508.146586] ? hrtimer_interrupt+0x128/0x2c0 [20508.148180] ? smp_apic_timer_interrupt+0xa6/0x270 [20508.150049] ? apic_timer_interrupt+0xf/0x20 [20508.151452] [20508.152211] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20508.153952] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20508.155915] ? newidle_balance.isra.93+0x207/0x580 [20508.157301] do_raw_spin_lock+0x177/0x190 [20508.158548] _raw_spin_lock+0x21/0x30 [20508.159817] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20508.162049] process_one_work+0x2c8/0x700 [20508.163552] worker_thread+0x296/0x6e0 [20508.164772] ? rescuer_thread+0x570/0x570 [20508.165931] kthread+0x1d1/0x200 [20508.167871] ? set_kthread_struct+0x70/0x70 [20508.174127] ret_from_fork+0x1f/0x30 [20513.248070] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17876s! [20513.250917] Showing busy workqueues and worker pools: [20513.255590] workqueue events: flags=0x0 [20513.259597] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20513.262903] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20513.271279] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20513.277461] pending: cpu_latency_work [ptlrpc] [20513.278897] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20513.293920] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20513.300650] workqueue events_long: flags=0x0 [20513.302842] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20513.306148] pending: lu_objects_flush [obdclass] [20513.310787] workqueue events_unbound: flags=0x2 [20513.312557] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20513.318112] pending: flush_memcg_stats_dwork, idle_cull_fn [20513.321465] workqueue events_power_efficient: flags=0x80 [20513.324247] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20513.327747] pending: check_lifetime, do_cache_clean [20513.336664] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20513.342622] pending: neigh_periodic_work, neigh_periodic_work [20513.350197] workqueue mm_percpu_wq: flags=0x8 [20513.352120] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20513.360148] pending: vmstat_update, lru_add_drain_per_cpu [20513.363192] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20513.369657] pending: vmstat_update, lru_add_drain_per_cpu [20513.375899] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20513.384543] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20513.389506] workqueue xprtiod: flags=0xa [20513.393214] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20513.401434] pending: xprt_autoclose, xs_error_handle [20513.403157] workqueue ipv6_addrconf: flags=0x40008 [20513.410231] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20513.413763] pending: addrconf_verify_work [20513.415889] workqueue knbd0-recv: flags=0x1a [20513.417092] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20513.421371] in-flight: 651:recv_work [20513.426602] workqueue ptlrpc_pinger: flags=0x2 [20513.428224] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20513.433573] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20513.440095] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17876s workers=4 idle: 15979 31824 8 [20513.447661] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20503s workers=2 idle: 654 [20514.881059] rcu: INFO: rcu_sched self-detected stall on CPU [20514.883602] rcu: 2-....: (10529417 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [20514.887811] (t=18420455 jiffies g=274185 q=68552) [20514.889832] rcu: rcu_sched kthread starved for 17878821 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [20514.897925] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [20514.905834] rcu: RCU grace-period kthread stack dump: [20514.911556] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [20514.918074] Call Trace: [20514.920999] __schedule+0x351/0xcb0 [20514.924522] schedule+0xc0/0x180 [20514.925417] schedule_timeout+0xb4/0x190 [20514.927942] ? do_raw_spin_unlock+0x75/0x190 [20514.931557] ? __next_timer_interrupt+0x160/0x160 [20514.934200] ? prepare_to_swait_event+0xf3/0x140 [20514.938876] rcu_gp_kthread+0x8dd/0x15b0 [20514.941561] ? force_qs_rnp+0x2c0/0x2c0 [20514.946897] kthread+0x1d1/0x200 [20514.950024] ? set_kthread_struct+0x70/0x70 [20514.953818] ret_from_fork+0x1f/0x30 [20514.954984] rcu: Stack dump where RCU GP kthread last ran: [20514.959797] Sending NMI from CPU 2 to CPUs 3: [20514.963909] NMI backtrace for cpu 3 [20514.963917] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20514.963919] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20514.963921] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20514.963922] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20514.963925] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 [20514.963927] RAX: 0000000000007f1a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20514.963929] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20514.963930] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20514.963932] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20514.963933] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20514.963935] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20514.963937] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20514.963938] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20514.963939] Call Trace: [20514.963940] [20514.963941] ? show_regs.cold.9+0x22/0x2f [20514.963943] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [20514.963944] ? nmi_cpu_backtrace_handler+0x15/0x40 [20514.963945] ? nmi_handle+0xbd/0x210 [20514.963947] ? default_do_nmi+0x4b/0x1b0 [20514.963948] ? do_nmi+0x280/0x380 [20514.963949] ? end_repeat_nmi+0x16/0x1f [20514.963950] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20514.963952] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20514.963953] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20514.963954] [20514.963955] do_raw_spin_lock+0x177/0x190 [20514.963957] _raw_spin_lock+0x21/0x30 [20514.963958] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20514.963959] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20514.963960] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20514.963962] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20514.963963] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20514.963964] ? woken_wake_function+0x30/0x30 [20514.963966] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20514.963967] kthread+0x1d1/0x200 [20514.963968] ? set_kthread_struct+0x70/0x70 [20514.963970] ret_from_fork+0x1f/0x30 [20514.969546] Sending NMI from CPU 2 to CPUs 1: [20515.177413] NMI backtrace for cpu 1 [20515.177421] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20515.177424] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20515.177426] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20515.177428] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20515.177431] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [20515.177434] RAX: 0000000000004a9c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20515.177435] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20515.177437] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20515.177439] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20515.177440] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20515.177442] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20515.177443] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20515.177445] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20515.177446] Call Trace: [20515.177447] [20515.177448] ? show_regs.cold.9+0x22/0x2f [20515.177450] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [20515.177451] ? nmi_cpu_backtrace_handler+0x15/0x40 [20515.177452] ? nmi_handle+0xbd/0x210 [20515.177453] ? default_do_nmi+0x4b/0x1b0 [20515.177454] ? do_nmi+0x280/0x380 [20515.177455] ? end_repeat_nmi+0x16/0x1f [20515.177456] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20515.177458] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20515.177459] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20515.177460] [20515.177461] do_raw_spin_lock+0x177/0x190 [20515.177463] _raw_spin_lock+0x21/0x30 [20515.177464] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20515.177465] ? do_raw_spin_unlock+0x75/0x190 [20515.177466] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20515.177468] ? do_wait_intr+0xf0/0xf0 [20515.177469] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20515.177470] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20515.177471] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20515.177473] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20515.177475] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20515.177476] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20515.177477] mdc_intent_lock+0x353/0x7e0 [mdc] [20515.177478] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20515.177480] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20515.177481] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20515.177483] lmv_intent_open+0x338/0x1150 [lmv] [20515.177484] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20515.177485] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20515.177486] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20515.177487] ll_intent_lock+0xc5/0xaa0 [lustre] [20515.177489] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20515.177490] ll_lookup_it+0xe0e/0x2270 [lustre] [20515.177491] ? do_raw_spin_unlock+0x75/0x190 [20515.177492] ? _raw_spin_unlock+0x12/0x30 [20515.177493] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20515.177494] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20515.177495] ll_atomic_open+0x36a/0xd70 [lustre] [20515.177496] lookup_open+0x356/0xb40 [20515.177497] do_last+0x313/0xfc0 [20515.177498] ? nd_jump_root+0xe5/0x160 [20515.177499] ? path_init+0x437/0x520 [20515.177500] path_openat+0xf7/0x500 [20515.177501] ? do_raw_spin_unlock+0x75/0x190 [20515.177502] do_filp_open+0x99/0x140 [20515.177503] ? getname_flags+0x6e/0x330 [20515.177503] ? __check_object_size+0xff/0x256 [20515.177504] ? do_raw_spin_unlock+0x75/0x190 [20515.177505] ? _raw_spin_unlock+0x12/0x30 [20515.177506] do_sys_openat2+0x2b4/0x410 [20515.177507] do_sys_open+0x73/0xa0 [20515.177508] __x64_sys_openat+0x24/0x30 [20515.177509] do_syscall_64+0xc1/0x440 [20515.177510] entry_SYSCALL_64_after_hwframe+0x49/0xae [20515.177511] RIP: 0033:0x7f53425a5332 [20515.177513] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20515.177515] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20515.177518] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20515.177520] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20515.177522] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20515.177523] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20515.177525] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20515.179656] NMI backtrace for cpu 2 [20515.471372] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20515.478106] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20515.485530] Call Trace: [20515.488395] [20515.490577] ? dump_stack+0xbb/0x10e [20515.492289] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [20515.493752] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [20515.499138] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [20515.502123] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [20515.510063] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [20515.513389] ? rcu_dump_cpu_stacks+0x11b/0x171 [20515.516595] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [20515.519416] ? update_process_times+0x6d/0xc0 [20515.522748] ? tick_sched_handle.isra.18+0x27/0x90 [20515.525185] ? tick_sched_timer+0x53/0xb0 [20515.526211] ? __hrtimer_run_queues+0x1c2/0x4d0 [20515.527854] ? tick_sched_handle.isra.18+0x90/0x90 [20515.529547] ? hrtimer_interrupt+0x128/0x2c0 [20515.530811] ? smp_apic_timer_interrupt+0xa6/0x270 [20515.532962] ? apic_timer_interrupt+0xf/0x20 [20515.534442] [20515.535196] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20515.537169] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20515.538867] ? do_raw_spin_lock+0x177/0x190 [20515.540174] ? _raw_spin_lock+0x21/0x30 [20515.541365] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20515.543034] ? obd_mod_free+0x46/0x140 [ptlrpc] [20515.545928] ? mdc_commit_open+0x91/0xb0 [mdc] [20515.547805] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20515.550287] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20515.552534] ? after_reply+0xe2c/0x1780 [ptlrpc] [20515.555262] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20515.558838] ? __next_timer_interrupt+0x160/0x160 [20515.564995] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [20515.567148] ? do_wait_intr+0xf0/0xf0 [20515.570498] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20515.574824] ? kthread+0x1d1/0x200 [20515.576570] ? set_kthread_struct+0x70/0x70 [20515.578142] ? ret_from_fork+0x1f/0x30 [20540.052026] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20540.055009] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20540.055247] Modules linked in: [20540.057017] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20540.057027] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20540.057104] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20540.057110] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20540.057113] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [20540.057127] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [20540.057131] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20540.057136] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20540.057139] RDX: 0000000000004e27 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20540.057141] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20540.057143] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20540.057145] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20540.057149] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20540.057152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20540.057154] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20540.057162] Call Trace: [20540.057168] [20540.057174] ? show_regs.cold.9+0x22/0x2f [20540.057183] ? watchdog_timer_fn.cold.6+0x62/0x10a [20540.057189] ? __hrtimer_run_queues+0x1c2/0x4d0 [20540.057193] ? watchdog+0x40/0x40 [20540.057198] ? hrtimer_interrupt+0x128/0x2c0 [20540.057202] ? smp_apic_timer_interrupt+0xa6/0x270 [20540.057213] ? apic_timer_interrupt+0xf/0x20 [20540.057217] [20540.057220] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20540.057224] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20540.057228] do_raw_spin_lock+0x177/0x190 [20540.057233] _raw_spin_lock+0x21/0x30 [20540.057239] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20540.057632] obd_mod_free+0x46/0x140 [ptlrpc] [20540.057907] mdc_commit_open+0x91/0xb0 [mdc] [20540.058012] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20540.058209] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20540.058333] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20540.058418] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20540.058423] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20540.058425] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20540.058434] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20540.058437] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20540.058441] RAX: 0000000000004657 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20540.058443] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20540.058445] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20540.058447] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20540.058501] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20540.058504] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20540.058507] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20540.058510] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20540.058517] Call Trace: [20540.058520] [20540.058523] ? show_regs.cold.9+0x22/0x2f [20540.058531] ? watchdog_timer_fn.cold.6+0x62/0x10a [20540.058536] ? __hrtimer_run_queues+0x1c2/0x4d0 [20540.058541] ? watchdog+0x40/0x40 [20540.058545] ? hrtimer_interrupt+0x128/0x2c0 [20540.058405] after_reply+0xe2c/0x1780 [ptlrpc] [20540.058548] ? smp_apic_timer_interrupt+0xa6/0x270 [20540.058557] ? apic_timer_interrupt+0xf/0x20 [20540.058562] [20540.058565] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20540.058579] do_raw_spin_lock+0x177/0x190 [20540.058588] _raw_spin_lock+0x21/0x30 [20540.058593] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20540.059034] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20540.059234] ? __next_timer_interrupt+0x160/0x160 [20540.059244] ptlrpcd+0xa82/0xe40 [ptlrpc] [20540.059576] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20540.059583] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [20540.059591] ? do_wait_intr+0xf0/0xf0 [20540.059597] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20540.059658] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20540.059663] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20540.059666] RIP: 0010:native_safe_halt+0x2e/0x50 [20540.059674] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20540.059679] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20540.059684] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20540.059686] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20540.059689] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20540.059691] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20540.059694] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20540.059696] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20540.059699] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20540.059702] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20540.059605] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20540.059711] Call Trace: [20540.059715] [20540.059718] ? show_regs.cold.9+0x22/0x2f [20540.059725] ? watchdog_timer_fn.cold.6+0x62/0x10a [20540.059731] ? __hrtimer_run_queues+0x1c2/0x4d0 [20540.059735] ? watchdog+0x40/0x40 [20540.059739] ? hrtimer_interrupt+0x128/0x2c0 [20540.059743] ? smp_apic_timer_interrupt+0xa6/0x270 [20540.059749] ? apic_timer_interrupt+0xf/0x20 [20540.059754] [20540.059757] ? native_safe_halt+0x2e/0x50 [20540.059761] kvm_wait+0xcd/0xe0 [20540.059768] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20540.059775] do_raw_spin_lock+0x177/0x190 [20540.059781] _raw_spin_lock+0x21/0x30 [20540.060026] kthread+0x1d1/0x200 [20540.060032] ? set_kthread_struct+0x70/0x70 [20540.060036] ret_from_fork+0x1f/0x30 [20540.059784] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20540.061482] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20540.061553] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20540.061560] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20540.061567] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20540.062090] RIP: 0010:native_safe_halt+0x2e/0x50 [20540.062099] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20540.062104] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20540.062108] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20540.062110] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [20540.062113] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20540.062115] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20540.062117] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20540.062121] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20540.062124] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20540.062126] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20540.062135] Call Trace: [20540.062140] [20540.062144] ? show_regs.cold.9+0x22/0x2f [20540.062155] ? watchdog_timer_fn.cold.6+0x62/0x10a [20540.062161] ? __hrtimer_run_queues+0x1c2/0x4d0 [20540.062166] ? watchdog+0x40/0x40 [20540.062171] ? hrtimer_interrupt+0x128/0x2c0 [20540.062175] ? smp_apic_timer_interrupt+0xa6/0x270 [20540.062181] ? apic_timer_interrupt+0xf/0x20 [20540.062186] [20540.062189] ? native_safe_halt+0x2e/0x50 [20540.062193] kvm_wait+0xcd/0xe0 [20540.062201] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20540.062207] ? newidle_balance.isra.93+0x207/0x580 [20540.062213] do_raw_spin_lock+0x177/0x190 [20540.062217] _raw_spin_lock+0x21/0x30 [20540.062221] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20540.062439] process_one_work+0x2c8/0x700 [20540.062446] worker_thread+0x296/0x6e0 [20540.062451] ? rescuer_thread+0x570/0x570 [20540.062455] kthread+0x1d1/0x200 [20540.062459] ? set_kthread_struct+0x70/0x70 [20540.062463] ret_from_fork+0x1f/0x30 [20540.066359] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20540.066965] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20540.067217] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20540.067383] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20540.067552] ? woken_wake_function+0x30/0x30 [20540.067562] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20540.067740] kthread+0x1d1/0x200 [20540.067745] ? set_kthread_struct+0x70/0x70 [20540.067748] ret_from_fork+0x1f/0x30 [20540.604628] ? do_raw_spin_unlock+0x75/0x190 [20540.605704] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20540.614577] ? do_wait_intr+0xf0/0xf0 [20540.615617] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20540.617313] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20540.620493] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20540.627197] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20540.630927] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20540.633909] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20540.635480] mdc_intent_lock+0x353/0x7e0 [mdc] [20540.637753] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20540.642277] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20540.647430] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20540.652175] lmv_intent_open+0x338/0x1150 [lmv] [20540.654186] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20540.658758] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20540.661621] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20540.663566] ll_intent_lock+0xc5/0xaa0 [lustre] [20540.667147] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20540.670677] ll_lookup_it+0xe0e/0x2270 [lustre] [20540.673222] ? do_raw_spin_unlock+0x75/0x190 [20540.677527] ? _raw_spin_unlock+0x12/0x30 [20540.679361] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20540.682824] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20540.686362] ll_atomic_open+0x36a/0xd70 [lustre] [20540.688392] lookup_open+0x356/0xb40 [20540.689920] do_last+0x313/0xfc0 [20540.691269] ? nd_jump_root+0xe5/0x160 [20540.692753] ? path_init+0x437/0x520 [20540.694023] path_openat+0xf7/0x500 [20540.694998] ? do_raw_spin_unlock+0x75/0x190 [20540.696355] do_filp_open+0x99/0x140 [20540.697462] ? getname_flags+0x6e/0x330 [20540.698518] ? __check_object_size+0xff/0x256 [20540.700027] ? do_raw_spin_unlock+0x75/0x190 [20540.701212] ? _raw_spin_unlock+0x12/0x30 [20540.702505] do_sys_openat2+0x2b4/0x410 [20540.703560] do_sys_open+0x73/0xa0 [20540.704603] __x64_sys_openat+0x24/0x30 [20540.706254] do_syscall_64+0xc1/0x440 [20540.707262] entry_SYSCALL_64_after_hwframe+0x49/0xae [20540.708650] RIP: 0033:0x7f53425a5332 [20540.710275] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20540.720649] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20540.726527] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20540.732656] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20540.738585] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20540.743429] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20540.747662] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20543.462890] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17906s! [20543.467133] Showing busy workqueues and worker pools: [20543.468983] workqueue events: flags=0x0 [20543.470498] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20543.473445] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20543.480091] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20543.484095] pending: cpu_latency_work [ptlrpc] [20543.487131] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20543.490032] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20543.494132] workqueue events_long: flags=0x0 [20543.495904] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20543.498478] pending: lu_objects_flush [obdclass] [20543.500086] workqueue events_unbound: flags=0x2 [20543.501426] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20543.503652] pending: flush_memcg_stats_dwork, idle_cull_fn [20543.505730] workqueue events_power_efficient: flags=0x80 [20543.508377] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20543.510725] pending: check_lifetime, do_cache_clean [20543.512683] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20543.515100] pending: neigh_periodic_work, neigh_periodic_work [20543.517200] workqueue mm_percpu_wq: flags=0x8 [20543.518725] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20543.521249] pending: vmstat_update, lru_add_drain_per_cpu [20543.523184] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20543.525290] pending: vmstat_update, lru_add_drain_per_cpu [20543.527164] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20543.529451] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20543.532116] workqueue xprtiod: flags=0xa [20543.533548] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20543.535447] pending: xprt_autoclose, xs_error_handle [20543.537102] workqueue ipv6_addrconf: flags=0x40008 [20543.538943] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20543.541619] pending: addrconf_verify_work [20543.543514] workqueue knbd0-recv: flags=0x1a [20543.545658] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20543.548293] in-flight: 651:recv_work [20543.549961] workqueue ptlrpc_pinger: flags=0x2 [20543.551697] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20543.554042] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20543.556781] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17906s workers=4 idle: 15979 31824 8 [20543.560029] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20534s workers=2 idle: 654 [20568.052703] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20568.055119] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20568.057588] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20568.057604] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20568.057793] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20568.057799] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20568.057802] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [20568.057827] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [20568.057831] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20568.057835] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20568.057838] RDX: 0000000000005670 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20568.057839] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20568.057841] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20568.057843] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20568.057846] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20568.057849] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20568.057851] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20568.058456] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20568.058463] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [20568.058557] Call Trace: [20568.058578] [20568.058584] ? show_regs.cold.9+0x22/0x2f [20568.058595] ? watchdog_timer_fn.cold.6+0x62/0x10a [20568.058602] ? __hrtimer_run_queues+0x1c2/0x4d0 [20568.058606] ? watchdog+0x40/0x40 [20568.058610] ? hrtimer_interrupt+0x128/0x2c0 [20568.058613] ? smp_apic_timer_interrupt+0xa6/0x270 [20568.058622] ? apic_timer_interrupt+0xf/0x20 [20568.058627] [20568.058630] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20568.058636] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20568.058686] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20568.058691] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20568.058694] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20568.058702] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20568.058706] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20568.058711] RAX: 0000000000004520 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20568.058713] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20568.058716] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20568.058718] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20568.059103] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20568.059106] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20568.059109] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20568.059112] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20568.059118] Call Trace: [20568.059121] [20568.059124] ? show_regs.cold.9+0x22/0x2f [20568.059131] ? watchdog_timer_fn.cold.6+0x62/0x10a [20568.059136] ? __hrtimer_run_queues+0x1c2/0x4d0 [20568.059139] ? watchdog+0x40/0x40 [20568.059144] ? hrtimer_interrupt+0x128/0x2c0 [20568.059148] ? smp_apic_timer_interrupt+0xa6/0x270 [20568.059154] ? apic_timer_interrupt+0xf/0x20 [20568.059159] [20568.059161] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20568.059166] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20568.059172] do_raw_spin_lock+0x177/0x190 [20568.059177] _raw_spin_lock+0x21/0x30 [20568.059183] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20568.059387] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20568.059393] do_raw_spin_lock+0x177/0x190 [20568.059397] _raw_spin_lock+0x21/0x30 [20568.059681] Modules linked in: lustre(O) [20568.059402] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20568.059891] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20568.060340] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20568.060623] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20568.060854] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20568.061194] ? woken_wake_function+0x30/0x30 [20568.061205] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20568.061418] kthread+0x1d1/0x200 [20568.061424] ? set_kthread_struct+0x70/0x70 [20568.061428] ret_from_fork+0x1f/0x30 [20568.059848] obd_mod_free+0x46/0x140 [ptlrpc] [20568.061758] mdc_commit_open+0x91/0xb0 [mdc] [20568.061799] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20568.062023] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20568.062317] after_reply+0xe2c/0x1780 [ptlrpc] [20568.062530] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20568.062745] ? __next_timer_interrupt+0x160/0x160 [20568.062754] ptlrpcd+0xa82/0xe40 [ptlrpc] [20568.063316] ? do_wait_intr+0xf0/0xf0 [20568.063326] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20568.063535] kthread+0x1d1/0x200 [20568.063541] ? set_kthread_struct+0x70/0x70 [20568.063545] ret_from_fork+0x1f/0x30 [20568.064665] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20568.064863] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20568.064870] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20568.064873] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20568.064885] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20568.064889] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20568.064894] RAX: 0000000000007a63 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20568.065011] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20568.065013] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20568.065016] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20568.065018] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20568.065022] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20568.065027] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20568.065030] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20568.065038] Call Trace: [20568.065043] [20568.065047] ? show_regs.cold.9+0x22/0x2f [20568.065056] ? watchdog_timer_fn.cold.6+0x62/0x10a [20568.065062] ? __hrtimer_run_queues+0x1c2/0x4d0 [20568.065066] ? watchdog+0x40/0x40 [20568.065076] ? hrtimer_interrupt+0x128/0x2c0 [20568.065081] ? smp_apic_timer_interrupt+0xa6/0x270 [20568.065088] ? apic_timer_interrupt+0xf/0x20 [20568.065092] [20568.065096] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20568.065106] do_raw_spin_lock+0x177/0x190 [20568.065111] _raw_spin_lock+0x21/0x30 [20568.065116] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20568.066252] ? do_raw_spin_unlock+0x75/0x190 [20568.066259] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20568.066609] ? do_wait_intr+0xf0/0xf0 [20568.066637] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20568.067200] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20568.067611] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20568.067885] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20568.068077] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20568.068165] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20568.068197] mdc_intent_lock+0x353/0x7e0 [mdc] [20568.068227] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20568.068462] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20568.084568] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20568.084653] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20568.084661] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20568.084667] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20568.108555] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20568.108722] lmv_intent_open+0x338/0x1150 [lmv] [20568.108766] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20568.108916] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20568.108999] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20568.109021] ll_intent_lock+0xc5/0xaa0 [lustre] [20568.109126] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20568.109214] ll_lookup_it+0xe0e/0x2270 [lustre] [20568.109297] ? do_raw_spin_unlock+0x75/0x190 [20568.109430] ? _raw_spin_unlock+0x12/0x30 [20568.109438] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20568.109466] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20568.109499] ll_atomic_open+0x36a/0xd70 [lustre] [20568.109611] lookup_open+0x356/0xb40 [20568.109621] do_last+0x313/0xfc0 [20568.109631] ? nd_jump_root+0xe5/0x160 [20568.109636] ? path_init+0x437/0x520 [20568.109641] path_openat+0xf7/0x500 [20568.109650] ? do_raw_spin_unlock+0x75/0x190 [20568.109655] do_filp_open+0x99/0x140 [20568.109665] ? getname_flags+0x6e/0x330 [20568.109676] ? __check_object_size+0xff/0x256 [20568.109687] ? do_raw_spin_unlock+0x75/0x190 [20568.109691] ? _raw_spin_unlock+0x12/0x30 [20568.109696] do_sys_openat2+0x2b4/0x410 [20568.109700] do_sys_open+0x73/0xa0 [20568.109709] __x64_sys_openat+0x24/0x30 [20568.109713] do_syscall_64+0xc1/0x440 [20568.109719] entry_SYSCALL_64_after_hwframe+0x49/0xae [20568.109729] RIP: 0033:0x7f53425a5332 [20568.109756] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20568.109760] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20568.109769] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20568.109772] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20568.109775] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20568.109777] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20568.109780] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20568.554087] [20568.554637] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20568.556674] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20568.562017] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20568.564225] RAX: 0000000000003a86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20568.566729] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [20568.568997] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20568.571171] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20568.573717] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20568.575903] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20568.578848] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20568.580708] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20568.582938] Call Trace: [20568.583674] [20568.584556] ? show_regs.cold.9+0x22/0x2f [20568.585806] ? watchdog_timer_fn.cold.6+0x62/0x10a [20568.587182] ? __hrtimer_run_queues+0x1c2/0x4d0 [20568.588736] ? watchdog+0x40/0x40 [20568.589782] ? hrtimer_interrupt+0x128/0x2c0 [20568.591035] ? smp_apic_timer_interrupt+0xa6/0x270 [20568.592624] ? apic_timer_interrupt+0xf/0x20 [20568.593851] [20568.594444] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20568.596116] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20568.598118] ? newidle_balance.isra.93+0x207/0x580 [20568.599587] do_raw_spin_lock+0x177/0x190 [20568.600988] _raw_spin_lock+0x21/0x30 [20568.602260] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20568.604979] process_one_work+0x2c8/0x700 [20568.606721] worker_thread+0x296/0x6e0 [20568.608024] ? rescuer_thread+0x570/0x570 [20568.609537] kthread+0x1d1/0x200 [20568.610708] ? set_kthread_struct+0x70/0x70 [20568.612173] ret_from_fork+0x1f/0x30 [20573.663079] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17936s! [20573.667026] Showing busy workqueues and worker pools: [20573.669950] workqueue events: flags=0x0 [20573.673143] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20573.677102] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20573.682995] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20573.686712] pending: cpu_latency_work [ptlrpc] [20573.689168] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20573.692852] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20573.697353] workqueue events_long: flags=0x0 [20573.699535] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20573.702351] pending: lu_objects_flush [obdclass] [20573.704862] workqueue events_unbound: flags=0x2 [20573.706846] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20573.709943] pending: flush_memcg_stats_dwork, idle_cull_fn [20573.711966] workqueue events_power_efficient: flags=0x80 [20573.714433] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20573.717170] pending: check_lifetime, do_cache_clean [20573.719613] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20573.722374] pending: neigh_periodic_work, neigh_periodic_work [20573.725352] workqueue mm_percpu_wq: flags=0x8 [20573.728052] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20573.731230] pending: vmstat_update, lru_add_drain_per_cpu [20573.734758] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20573.739690] pending: vmstat_update, lru_add_drain_per_cpu [20573.743066] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20573.747302] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20573.750480] workqueue xprtiod: flags=0xa [20573.752027] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20573.755244] pending: xprt_autoclose, xs_error_handle [20573.758198] workqueue ipv6_addrconf: flags=0x40008 [20573.760926] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20573.764018] pending: addrconf_verify_work [20573.766098] workqueue knbd0-recv: flags=0x1a [20573.768132] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20573.770927] in-flight: 651:recv_work [20573.772869] workqueue ptlrpc_pinger: flags=0x2 [20573.774845] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20573.777847] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20573.780549] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17936s workers=4 idle: 15979 31824 8 [20573.784120] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20564s workers=2 idle: 654 [20596.055114] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20596.057683] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20596.060285] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20596.060298] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20596.060374] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20596.060379] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20596.060381] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20596.060395] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20596.060399] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20596.060403] RAX: 000000000000140c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20596.060405] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20596.060408] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20596.060409] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20596.060412] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20596.060414] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20596.060417] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20596.060419] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20596.060428] Call Trace: [20596.060433] [20596.060439] ? show_regs.cold.9+0x22/0x2f [20596.060448] ? watchdog_timer_fn.cold.6+0x62/0x10a [20596.060454] ? __hrtimer_run_queues+0x1c2/0x4d0 [20596.060458] ? watchdog+0x40/0x40 [20596.060462] ? hrtimer_interrupt+0x128/0x2c0 [20596.060466] ? smp_apic_timer_interrupt+0xa6/0x270 [20596.060474] ? apic_timer_interrupt+0xf/0x20 [20596.060479] [20596.060482] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20596.060499] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20596.060505] do_raw_spin_lock+0x177/0x190 [20596.060510] _raw_spin_lock+0x21/0x30 [20596.060514] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20596.060983] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20596.061183] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20596.062145] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20596.062362] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20596.064039] ? woken_wake_function+0x30/0x30 [20596.064050] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20596.064307] kthread+0x1d1/0x200 [20596.064313] ? set_kthread_struct+0x70/0x70 [20596.064318] ret_from_fork+0x1f/0x30 [20596.066232] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20596.066314] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20596.066321] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20596.066323] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20596.066337] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20596.066341] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20596.066346] RAX: 000000000000313f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20596.066349] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20596.066351] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20596.066353] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20596.066356] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20596.066359] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20596.066362] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20596.066365] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20596.066380] Call Trace: [20596.066385] [20596.066390] ? show_regs.cold.9+0x22/0x2f [20596.066398] ? watchdog_timer_fn.cold.6+0x62/0x10a [20596.066403] ? __hrtimer_run_queues+0x1c2/0x4d0 [20596.066408] ? watchdog+0x40/0x40 [20596.066412] ? hrtimer_interrupt+0x128/0x2c0 [20596.066416] ? smp_apic_timer_interrupt+0xa6/0x270 [20596.066423] ? apic_timer_interrupt+0xf/0x20 [20596.066428] [20596.066431] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20596.066437] do_raw_spin_lock+0x177/0x190 [20596.066441] _raw_spin_lock+0x21/0x30 [20596.066446] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20596.066731] ? do_raw_spin_unlock+0x75/0x190 [20596.066737] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20596.069235] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20596.069305] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20596.069312] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20596.069315] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [20596.069327] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [20596.069331] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20596.069335] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20596.069337] RDX: 0000000000006edf RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20596.069339] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20596.069341] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20596.069343] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20596.069346] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20596.069348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20596.069350] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20596.069357] Call Trace: [20596.069361] [20596.069365] ? show_regs.cold.9+0x22/0x2f [20596.072920] ? do_wait_intr+0xf0/0xf0 [20596.072939] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20596.073320] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20596.073565] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20596.073779] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20596.074002] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20596.074040] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20596.074069] mdc_intent_lock+0x353/0x7e0 [mdc] [20596.074096] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20596.074200] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20596.074493] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20596.074523] lmv_intent_open+0x338/0x1150 [lmv] [20596.074552] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20596.074653] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20596.074741] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20596.074763] ll_intent_lock+0xc5/0xaa0 [lustre] [20596.074937] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20596.075015] ll_lookup_it+0xe0e/0x2270 [lustre] [20596.075095] ? do_raw_spin_unlock+0x75/0x190 [20596.075102] ? _raw_spin_unlock+0x12/0x30 [20596.075109] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20596.075178] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20596.092928] ? watchdog_timer_fn.cold.6+0x62/0x10a [20596.092992] ? __hrtimer_run_queues+0x1c2/0x4d0 [20596.092997] ? watchdog+0x40/0x40 [20596.093002] ? hrtimer_interrupt+0x128/0x2c0 [20596.093006] ? smp_apic_timer_interrupt+0xa6/0x270 [20596.093015] ? apic_timer_interrupt+0xf/0x20 [20596.093020] [20596.093025] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20596.093031] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20596.093036] do_raw_spin_lock+0x177/0x190 [20596.093043] _raw_spin_lock+0x21/0x30 [20596.093050] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20596.098496] ll_atomic_open+0x36a/0xd70 [lustre] [20596.098629] lookup_open+0x356/0xb40 [20596.098648] do_last+0x313/0xfc0 [20596.098654] ? nd_jump_root+0xe5/0x160 [20596.098660] ? path_init+0x437/0x520 [20596.098665] path_openat+0xf7/0x500 [20596.098671] ? do_raw_spin_unlock+0x75/0x190 [20596.098677] do_filp_open+0x99/0x140 [20596.098685] ? getname_flags+0x6e/0x330 [20596.098690] ? __check_object_size+0xff/0x256 [20596.098699] ? do_raw_spin_unlock+0x75/0x190 [20596.098704] ? _raw_spin_unlock+0x12/0x30 [20596.098711] do_sys_openat2+0x2b4/0x410 [20596.098715] do_sys_open+0x73/0xa0 [20596.098720] __x64_sys_openat+0x24/0x30 [20596.098724] do_syscall_64+0xc1/0x440 [20596.098731] entry_SYSCALL_64_after_hwframe+0x49/0xae [20596.098738] RIP: 0033:0x7f53425a5332 [20596.098745] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20596.098749] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20596.098755] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20596.098757] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20596.098760] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20596.098762] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20596.098764] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20596.580675] obd_mod_free+0x46/0x140 [ptlrpc] [20596.582603] mdc_commit_open+0x91/0xb0 [mdc] [20596.584616] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20596.586911] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20596.590106] after_reply+0xe2c/0x1780 [ptlrpc] [20596.593335] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20596.595584] ? __next_timer_interrupt+0x160/0x160 [20596.597154] ptlrpcd+0xa82/0xe40 [ptlrpc] [20596.598766] ? do_wait_intr+0xf0/0xf0 [20596.601911] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20596.606989] kthread+0x1d1/0x200 [20596.607918] ? set_kthread_struct+0x70/0x70 [20596.609495] ret_from_fork+0x1f/0x30 [20600.052031] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [20600.054939] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20600.082021] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20600.090655] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20600.097962] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20600.100965] RIP: 0010:native_safe_halt+0x2e/0x50 [20600.106204] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20600.117237] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20600.122996] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20600.125485] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [20600.127928] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20600.131484] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20600.134191] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20600.136870] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20600.140252] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20600.142677] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20600.147133] Call Trace: [20600.148221] [20600.149191] ? show_regs.cold.9+0x22/0x2f [20600.151049] ? watchdog_timer_fn.cold.6+0x62/0x10a [20600.154028] ? __hrtimer_run_queues+0x1c2/0x4d0 [20600.156221] ? watchdog+0x40/0x40 [20600.158873] ? hrtimer_interrupt+0x128/0x2c0 [20600.161591] ? smp_apic_timer_interrupt+0xa6/0x270 [20600.167677] ? apic_timer_interrupt+0xf/0x20 [20600.171397] [20600.173596] ? native_safe_halt+0x2e/0x50 [20600.178256] kvm_wait+0xcd/0xe0 [20600.181234] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20600.186661] ? newidle_balance.isra.93+0x207/0x580 [20600.191387] do_raw_spin_lock+0x177/0x190 [20600.194905] _raw_spin_lock+0x21/0x30 [20600.198281] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20600.203435] process_one_work+0x2c8/0x700 [20600.205636] worker_thread+0x296/0x6e0 [20600.207539] ? rescuer_thread+0x570/0x570 [20600.209484] kthread+0x1d1/0x200 [20600.211042] ? set_kthread_struct+0x70/0x70 [20600.213418] ret_from_fork+0x1f/0x30 [20603.871899] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17966s! [20603.875245] Showing busy workqueues and worker pools: [20603.884195] workqueue events: flags=0x0 [20603.885267] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20603.887368] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20603.899855] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20603.905177] pending: cpu_latency_work [ptlrpc] [20603.909711] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20603.914056] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20603.919774] workqueue events_long: flags=0x0 [20603.921136] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20603.923169] pending: lu_objects_flush [obdclass] [20603.927129] workqueue events_unbound: flags=0x2 [20603.929268] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20603.933658] pending: flush_memcg_stats_dwork, idle_cull_fn [20603.937810] workqueue events_power_efficient: flags=0x80 [20603.939932] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20603.944353] pending: check_lifetime, do_cache_clean [20603.946221] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20603.948451] pending: neigh_periodic_work, neigh_periodic_work [20603.950577] workqueue mm_percpu_wq: flags=0x8 [20603.951978] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20603.954223] pending: vmstat_update, lru_add_drain_per_cpu [20603.957077] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20603.960579] pending: vmstat_update, lru_add_drain_per_cpu [20603.964102] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20603.965965] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20603.968816] workqueue xprtiod: flags=0xa [20603.969824] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20603.972423] pending: xprt_autoclose, xs_error_handle [20603.975635] workqueue ipv6_addrconf: flags=0x40008 [20603.977584] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20603.979911] pending: addrconf_verify_work [20603.982474] workqueue knbd0-recv: flags=0x1a [20603.983567] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20603.985222] in-flight: 651:recv_work [20603.986609] workqueue ptlrpc_pinger: flags=0x2 [20603.988552] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20603.991836] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20603.996312] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17966s workers=4 idle: 15979 31824 8 [20604.001880] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20594s workers=2 idle: 654 [20624.055029] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20624.057811] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20624.058122] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20624.058129] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20624.058209] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20624.058214] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20624.058216] RIP: 0010:native_safe_halt+0x2e/0x50 [20624.058229] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20624.058233] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20624.058238] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20624.058240] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20624.058242] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20624.058245] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20624.058247] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20624.058250] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20624.058253] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20624.058255] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20624.058264] Call Trace: [20624.058269] [20624.058275] ? show_regs.cold.9+0x22/0x2f [20624.058285] ? watchdog_timer_fn.cold.6+0x62/0x10a [20624.058293] ? __hrtimer_run_queues+0x1c2/0x4d0 [20624.058297] ? watchdog+0x40/0x40 [20624.058302] ? hrtimer_interrupt+0x128/0x2c0 [20624.058306] ? smp_apic_timer_interrupt+0xa6/0x270 [20624.058313] ? apic_timer_interrupt+0xf/0x20 [20624.058338] [20624.058341] ? native_safe_halt+0x2e/0x50 [20624.058346] kvm_wait+0xcd/0xe0 [20624.058353] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20624.058362] do_raw_spin_lock+0x177/0x190 [20624.058366] _raw_spin_lock+0x21/0x30 [20624.058372] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20624.058834] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20624.059306] Modules linked in: lustre(O) [20624.059189] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20624.059313] osp(O) ofd(O) [20624.059433] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20624.059658] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20624.059942] ? woken_wake_function+0x30/0x30 [20624.059951] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20624.059317] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20624.061349] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20624.061356] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20624.061359] RIP: 0010:native_safe_halt+0x2e/0x50 [20624.061369] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20624.061373] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20624.061378] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20624.061381] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20624.061383] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20624.061385] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20624.061387] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20624.061390] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20624.061392] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20624.061395] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20624.061405] Call Trace: [20624.061409] [20624.061412] ? show_regs.cold.9+0x22/0x2f [20624.061423] ? watchdog_timer_fn.cold.6+0x62/0x10a [20624.061429] ? __hrtimer_run_queues+0x1c2/0x4d0 [20624.061433] ? watchdog+0x40/0x40 [20624.061437] ? hrtimer_interrupt+0x128/0x2c0 [20624.061441] ? smp_apic_timer_interrupt+0xa6/0x270 [20624.061448] ? apic_timer_interrupt+0xf/0x20 [20624.061453] [20624.061455] ? native_safe_halt+0x2e/0x50 [20624.061459] kvm_wait+0xcd/0xe0 [20624.061467] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20624.061474] do_raw_spin_lock+0x177/0x190 [20624.061479] _raw_spin_lock+0x21/0x30 [20624.061483] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20624.062635] ? do_raw_spin_unlock+0x75/0x190 [20624.062658] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20624.066548] ? do_wait_intr+0xf0/0xf0 [20624.066560] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20624.066866] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20624.067099] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20624.067334] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20624.067448] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20624.067723] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20624.067790] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20624.067797] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20624.067800] RIP: 0010:native_safe_halt+0x2e/0x50 [20624.067812] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20624.067816] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [20624.067821] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20624.067823] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20624.067826] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20624.067828] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20624.067831] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20624.067834] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20624.067837] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20624.067840] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20624.067849] Call Trace: [20624.067853] [20624.067857] ? show_regs.cold.9+0x22/0x2f [20624.067866] ? watchdog_timer_fn.cold.6+0x62/0x10a [20624.067872] ? __hrtimer_run_queues+0x1c2/0x4d0 [20624.067877] ? watchdog+0x40/0x40 [20624.067881] ? hrtimer_interrupt+0x128/0x2c0 [20624.067885] ? smp_apic_timer_interrupt+0xa6/0x270 [20624.067891] ? apic_timer_interrupt+0xf/0x20 [20624.067896] [20624.067899] ? native_safe_halt+0x2e/0x50 [20624.067904] kvm_wait+0xcd/0xe0 [20624.067911] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20624.067917] do_raw_spin_lock+0x177/0x190 [20624.067922] _raw_spin_lock+0x21/0x30 [20624.067927] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20624.068135] obd_mod_free+0x46/0x140 [ptlrpc] [20624.072998] kthread+0x1d1/0x200 [20624.073025] ? set_kthread_struct+0x70/0x70 [20624.073010] mdc_commit_open+0x91/0xb0 [mdc] [20624.073030] ret_from_fork+0x1f/0x30 [20624.073047] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20624.073374] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20624.073641] after_reply+0xe2c/0x1780 [ptlrpc] [20624.073919] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20624.074131] ? __next_timer_interrupt+0x160/0x160 [20624.074140] ptlrpcd+0xa82/0xe40 [ptlrpc] [20624.074888] ? do_wait_intr+0xf0/0xf0 [20624.074898] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20624.075819] kthread+0x1d1/0x200 [20624.075826] ? set_kthread_struct+0x70/0x70 [20624.075831] ret_from_fork+0x1f/0x30 [20624.620709] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20624.622767] mdc_intent_lock+0x353/0x7e0 [mdc] [20624.624613] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20624.627117] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20624.630046] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20624.633011] lmv_intent_open+0x338/0x1150 [lmv] [20624.635179] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20624.638801] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20624.641288] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20624.643439] ll_intent_lock+0xc5/0xaa0 [lustre] [20624.645481] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20624.648144] ll_lookup_it+0xe0e/0x2270 [lustre] [20624.650602] ? do_raw_spin_unlock+0x75/0x190 [20624.652831] ? _raw_spin_unlock+0x12/0x30 [20624.654543] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20624.658219] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20624.662156] ll_atomic_open+0x36a/0xd70 [lustre] [20624.664571] lookup_open+0x356/0xb40 [20624.670144] do_last+0x313/0xfc0 [20624.671337] ? nd_jump_root+0xe5/0x160 [20624.674467] ? path_init+0x437/0x520 [20624.676438] path_openat+0xf7/0x500 [20624.679150] ? do_raw_spin_unlock+0x75/0x190 [20624.682853] do_filp_open+0x99/0x140 [20624.685825] ? getname_flags+0x6e/0x330 [20624.687913] ? __check_object_size+0xff/0x256 [20624.692815] ? do_raw_spin_unlock+0x75/0x190 [20624.695303] ? _raw_spin_unlock+0x12/0x30 [20624.697545] do_sys_openat2+0x2b4/0x410 [20624.699513] do_sys_open+0x73/0xa0 [20624.702922] __x64_sys_openat+0x24/0x30 [20624.704907] do_syscall_64+0xc1/0x440 [20624.706175] entry_SYSCALL_64_after_hwframe+0x49/0xae [20624.710262] RIP: 0033:0x7f53425a5332 [20624.714637] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20624.726215] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20624.733435] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20624.738276] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20624.742228] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20624.748061] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20624.751446] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20628.052896] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20628.066191] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20628.141792] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20628.155488] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20628.161218] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20628.169376] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20628.178444] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20628.197486] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20628.211441] RAX: 0000000000007e11 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20628.217546] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [20628.222060] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20628.231422] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20628.244606] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20628.251525] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20628.265223] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20628.269585] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20628.277126] Call Trace: [20628.278433] [20628.279532] ? show_regs.cold.9+0x22/0x2f [20628.282936] ? watchdog_timer_fn.cold.6+0x62/0x10a [20628.288187] ? __hrtimer_run_queues+0x1c2/0x4d0 [20628.294552] ? watchdog+0x40/0x40 [20628.297861] ? hrtimer_interrupt+0x128/0x2c0 [20628.302231] ? smp_apic_timer_interrupt+0xa6/0x270 [20628.305118] ? apic_timer_interrupt+0xf/0x20 [20628.311836] [20628.313449] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20628.323224] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20628.326498] ? newidle_balance.isra.93+0x207/0x580 [20628.328950] do_raw_spin_lock+0x177/0x190 [20628.330552] _raw_spin_lock+0x21/0x30 [20628.332628] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20628.334893] process_one_work+0x2c8/0x700 [20628.336750] worker_thread+0x296/0x6e0 [20628.338106] ? rescuer_thread+0x570/0x570 [20628.339056] kthread+0x1d1/0x200 [20628.340053] ? set_kthread_struct+0x70/0x70 [20628.341078] ret_from_fork+0x1f/0x30 [20634.080029] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 17996s! [20634.082224] Showing busy workqueues and worker pools: [20634.085011] workqueue events: flags=0x0 [20634.086334] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20634.088859] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20634.097017] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20634.100007] pending: cpu_latency_work [ptlrpc] [20634.102546] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20634.105287] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20634.109877] workqueue events_long: flags=0x0 [20634.111751] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20634.116215] pending: lu_objects_flush [obdclass] [20634.120092] workqueue events_unbound: flags=0x2 [20634.123901] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20634.127820] pending: flush_memcg_stats_dwork, idle_cull_fn [20634.129576] workqueue events_power_efficient: flags=0x80 [20634.133891] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20634.138238] pending: check_lifetime, do_cache_clean [20634.139883] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20634.143662] pending: neigh_periodic_work, neigh_periodic_work [20634.148226] workqueue mm_percpu_wq: flags=0x8 [20634.149864] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20634.152840] pending: vmstat_update, lru_add_drain_per_cpu [20634.155274] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20634.157997] pending: vmstat_update, lru_add_drain_per_cpu [20634.160788] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20634.163584] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20634.166970] workqueue xprtiod: flags=0xa [20634.168316] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20634.171311] pending: xprt_autoclose, xs_error_handle [20634.173865] workqueue ipv6_addrconf: flags=0x40008 [20634.176045] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20634.179127] pending: addrconf_verify_work [20634.180756] workqueue knbd0-recv: flags=0x1a [20634.182885] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20634.185051] in-flight: 651:recv_work [20634.187652] workqueue ptlrpc_pinger: flags=0x2 [20634.190183] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20634.194020] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20634.195804] pool 8: cpus=0-3 flags=0x4 nice=0 hung=17997s workers=4 idle: 15979 31824 8 [20634.198867] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20624s workers=2 idle: 654 [20652.055089] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20652.057195] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20652.058283] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20652.058290] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20652.058631] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20652.058635] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20652.058637] RIP: 0010:native_safe_halt+0x2e/0x50 [20652.058654] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20652.058657] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20652.058660] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20652.058661] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20652.058663] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20652.058664] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20652.058666] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20652.058669] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20652.058671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20652.058672] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20652.058678] Call Trace: [20652.058688] [20652.058693] ? show_regs.cold.9+0x22/0x2f [20652.058701] ? watchdog_timer_fn.cold.6+0x62/0x10a [20652.058705] ? __hrtimer_run_queues+0x1c2/0x4d0 [20652.058708] ? watchdog+0x40/0x40 [20652.058711] ? hrtimer_interrupt+0x128/0x2c0 [20652.058714] ? smp_apic_timer_interrupt+0xa6/0x270 [20652.058718] ? apic_timer_interrupt+0xf/0x20 [20652.058722] [20652.058724] ? native_safe_halt+0x2e/0x50 [20652.058726] kvm_wait+0xcd/0xe0 [20652.058732] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20652.058739] do_raw_spin_lock+0x177/0x190 [20652.058742] _raw_spin_lock+0x21/0x30 [20652.058745] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20652.059047] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20652.059584] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20652.059790] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20652.059904] Modules linked in: lustre(O) [20652.060011] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20652.060298] ? woken_wake_function+0x30/0x30 [20652.060308] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20652.060493] kthread+0x1d1/0x200 [20652.060499] ? set_kthread_struct+0x70/0x70 [20652.060503] ret_from_fork+0x1f/0x30 [20652.064012] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20652.064113] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20652.064121] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20652.064124] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [20652.068191] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20652.068270] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20652.068278] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20652.068281] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20652.068295] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20652.068299] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20652.068304] RAX: 0000000000003723 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20652.068307] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20652.068309] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20652.068312] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20652.068315] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20652.068318] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20652.068322] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20652.068324] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20652.068334] Call Trace: [20652.068340] [20652.068346] ? show_regs.cold.9+0x22/0x2f [20652.068355] ? watchdog_timer_fn.cold.6+0x62/0x10a [20652.068362] ? __hrtimer_run_queues+0x1c2/0x4d0 [20652.068366] ? watchdog+0x40/0x40 [20652.068371] ? hrtimer_interrupt+0x128/0x2c0 [20652.068375] ? smp_apic_timer_interrupt+0xa6/0x270 [20652.068384] ? apic_timer_interrupt+0xf/0x20 [20652.068389] [20652.068392] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20652.068399] do_raw_spin_lock+0x177/0x190 [20652.068404] _raw_spin_lock+0x21/0x30 [20652.096106] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [20652.096128] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20652.096135] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20652.096137] RDX: 000000000000409a RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20652.096140] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20652.096142] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20652.096145] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20652.096148] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20652.096152] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20652.096154] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20652.096163] Call Trace: [20652.096173] [20652.096179] ? show_regs.cold.9+0x22/0x2f [20652.096192] ? watchdog_timer_fn.cold.6+0x62/0x10a [20652.096199] ? __hrtimer_run_queues+0x1c2/0x4d0 [20652.096202] ? watchdog+0x40/0x40 [20652.096284] ? hrtimer_interrupt+0x128/0x2c0 [20652.096289] ? smp_apic_timer_interrupt+0xa6/0x270 [20652.096297] ? apic_timer_interrupt+0xf/0x20 [20652.096301] [20652.096310] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20652.096317] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20652.096322] do_raw_spin_lock+0x177/0x190 [20652.096326] _raw_spin_lock+0x21/0x30 [20652.096333] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20652.096942] obd_mod_free+0x46/0x140 [ptlrpc] [20652.105088] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20652.110281] mdc_commit_open+0x91/0xb0 [mdc] [20652.110325] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20652.110732] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20652.110979] after_reply+0xe2c/0x1780 [ptlrpc] [20652.111222] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20652.111352] ? __next_timer_interrupt+0x160/0x160 [20652.111360] ptlrpcd+0xa82/0xe40 [ptlrpc] [20652.111603] ? do_wait_intr+0xf0/0xf0 [20652.111611] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20652.113483] ? do_raw_spin_unlock+0x75/0x190 [20652.113498] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20652.113760] ? do_wait_intr+0xf0/0xf0 [20652.113768] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20652.120244] kthread+0x1d1/0x200 [20652.120260] ? set_kthread_struct+0x70/0x70 [20652.120265] ret_from_fork+0x1f/0x30 [20652.353669] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20652.355657] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20652.357824] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20652.359706] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20652.362008] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20652.363562] mdc_intent_lock+0x353/0x7e0 [mdc] [20652.365198] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20652.367295] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20652.370552] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20652.373836] lmv_intent_open+0x338/0x1150 [lmv] [20652.377314] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20652.380278] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20652.382363] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20652.383825] ll_intent_lock+0xc5/0xaa0 [lustre] [20652.385656] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20652.387355] ll_lookup_it+0xe0e/0x2270 [lustre] [20652.389199] ? do_raw_spin_unlock+0x75/0x190 [20652.390633] ? _raw_spin_unlock+0x12/0x30 [20652.392007] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20652.393680] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20652.395358] ll_atomic_open+0x36a/0xd70 [lustre] [20652.396799] lookup_open+0x356/0xb40 [20652.398021] do_last+0x313/0xfc0 [20652.399263] ? nd_jump_root+0xe5/0x160 [20652.400663] ? path_init+0x437/0x520 [20652.401963] path_openat+0xf7/0x500 [20652.403080] ? do_raw_spin_unlock+0x75/0x190 [20652.404620] do_filp_open+0x99/0x140 [20652.405858] ? getname_flags+0x6e/0x330 [20652.407617] ? __check_object_size+0xff/0x256 [20652.409217] ? do_raw_spin_unlock+0x75/0x190 [20652.411681] ? _raw_spin_unlock+0x12/0x30 [20652.415248] do_sys_openat2+0x2b4/0x410 [20652.417840] do_sys_open+0x73/0xa0 [20652.420275] __x64_sys_openat+0x24/0x30 [20652.423234] do_syscall_64+0xc1/0x440 [20652.424861] entry_SYSCALL_64_after_hwframe+0x49/0xae [20652.426591] RIP: 0033:0x7f53425a5332 [20652.427937] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20652.433992] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20652.436365] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20652.438598] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20652.440973] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20652.444351] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20652.448190] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20660.052296] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [20660.060849] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20660.100297] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20660.109046] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20660.124915] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20660.135381] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20660.138537] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20660.161421] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20660.175558] RAX: 0000000000006f0f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20660.185864] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [20660.197382] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20660.205153] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20660.212529] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20660.227542] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20660.244288] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20660.253875] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20660.261870] Call Trace: [20660.263301] [20660.273219] ? show_regs.cold.9+0x22/0x2f [20660.282920] ? watchdog_timer_fn.cold.6+0x62/0x10a [20660.290564] ? __hrtimer_run_queues+0x1c2/0x4d0 [20660.293665] ? watchdog+0x40/0x40 [20660.295612] ? hrtimer_interrupt+0x128/0x2c0 [20660.300931] ? smp_apic_timer_interrupt+0xa6/0x270 [20660.304145] ? apic_timer_interrupt+0xf/0x20 [20660.320549] [20660.322535] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20660.327952] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20660.330678] ? newidle_balance.isra.93+0x207/0x580 [20660.336961] do_raw_spin_lock+0x177/0x190 [20660.338882] _raw_spin_lock+0x21/0x30 [20660.340542] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20660.346935] process_one_work+0x2c8/0x700 [20660.350924] worker_thread+0x296/0x6e0 [20660.354460] ? rescuer_thread+0x570/0x570 [20660.357061] kthread+0x1d1/0x200 [20660.362106] ? set_kthread_struct+0x70/0x70 [20660.365847] ret_from_fork+0x1f/0x30 [20664.288086] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18027s! [20664.290598] Showing busy workqueues and worker pools: [20664.292886] workqueue events: flags=0x0 [20664.296746] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20664.302379] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20664.306277] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20664.308233] pending: cpu_latency_work [ptlrpc] [20664.310191] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20664.312303] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20664.315519] workqueue events_long: flags=0x0 [20664.317206] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20664.319024] pending: lu_objects_flush [obdclass] [20664.320801] workqueue events_unbound: flags=0x2 [20664.322099] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20664.324195] pending: flush_memcg_stats_dwork, idle_cull_fn [20664.326382] workqueue events_power_efficient: flags=0x80 [20664.328521] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20664.330414] pending: check_lifetime, do_cache_clean [20664.331922] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20664.334002] pending: neigh_periodic_work, neigh_periodic_work [20664.336089] workqueue mm_percpu_wq: flags=0x8 [20664.337516] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20664.339379] pending: vmstat_update, lru_add_drain_per_cpu [20664.341595] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20664.343660] pending: vmstat_update, lru_add_drain_per_cpu [20664.345338] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20664.347610] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20664.349611] workqueue xprtiod: flags=0xa [20664.350756] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20664.352916] pending: xprt_autoclose, xs_error_handle [20664.354648] workqueue ipv6_addrconf: flags=0x40008 [20664.356244] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20664.358197] pending: addrconf_verify_work [20664.359762] workqueue knbd0-recv: flags=0x1a [20664.360996] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20664.363079] in-flight: 651:recv_work [20664.364533] workqueue ptlrpc_pinger: flags=0x2 [20664.365985] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20664.368352] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20664.370259] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18027s workers=4 idle: 15979 31824 8 [20664.372707] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20654s workers=2 idle: 654 [20680.057304] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20680.058376] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20680.060715] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20680.060792] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20680.060801] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20680.060805] RIP: 0010:native_safe_halt+0x2e/0x50 [20680.060819] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20680.060823] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20680.060828] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20680.060830] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20680.060833] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20680.060835] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20680.060837] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20680.060840] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20680.060843] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20680.060846] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20680.062236] Call Trace: [20680.064731] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20680.064743] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20680.064813] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20680.064818] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20680.064820] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20680.064833] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20680.064837] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20680.064841] RAX: 000000000000690e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20680.064843] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20680.064846] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20680.064848] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20680.064850] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20680.064853] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20680.064856] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20680.064858] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20680.064866] Call Trace: [20680.064872] [20680.064876] ? show_regs.cold.9+0x22/0x2f [20680.064885] ? watchdog_timer_fn.cold.6+0x62/0x10a [20680.064890] ? __hrtimer_run_queues+0x1c2/0x4d0 [20680.064894] ? watchdog+0x40/0x40 [20680.064898] ? hrtimer_interrupt+0x128/0x2c0 [20680.064901] ? smp_apic_timer_interrupt+0xa6/0x270 [20680.064909] ? apic_timer_interrupt+0xf/0x20 [20680.064914] [20680.064916] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20680.064924] do_raw_spin_lock+0x177/0x190 [20680.064928] _raw_spin_lock+0x21/0x30 [20680.064932] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20680.066506] ? do_raw_spin_unlock+0x75/0x190 [20680.066514] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20680.066723] ? do_wait_intr+0xf0/0xf0 [20680.066733] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20680.066986] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20680.068670] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20680.068890] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20680.069119] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20680.069733] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20680.069769] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20680.069799] mdc_intent_lock+0x353/0x7e0 [mdc] [20680.069826] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20680.069925] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20680.070171] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20680.070677] lmv_intent_open+0x338/0x1150 [lmv] [20680.070705] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20680.070802] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20680.070901] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20680.070925] ll_intent_lock+0xc5/0xaa0 [lustre] [20680.071009] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20680.071110] ll_lookup_it+0xe0e/0x2270 [lustre] [20680.071990] ? do_raw_spin_unlock+0x75/0x190 [20680.072006] ? _raw_spin_unlock+0x12/0x30 [20680.072013] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20680.072038] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20680.072147] ll_atomic_open+0x36a/0xd70 [lustre] [20680.072509] lookup_open+0x356/0xb40 [20680.072517] do_last+0x313/0xfc0 [20680.072522] ? nd_jump_root+0xe5/0x160 [20680.072597] ? path_init+0x437/0x520 [20680.072601] path_openat+0xf7/0x500 [20680.072606] ? do_raw_spin_unlock+0x75/0x190 [20680.072611] do_filp_open+0x99/0x140 [20680.072616] ? getname_flags+0x6e/0x330 [20680.072620] ? __check_object_size+0xff/0x256 [20680.072627] ? do_raw_spin_unlock+0x75/0x190 [20680.072631] ? _raw_spin_unlock+0x12/0x30 [20680.072635] do_sys_openat2+0x2b4/0x410 [20680.072638] do_sys_open+0x73/0xa0 [20680.072643] __x64_sys_openat+0x24/0x30 [20680.072646] do_syscall_64+0xc1/0x440 [20680.072652] entry_SYSCALL_64_after_hwframe+0x49/0xae [20680.072657] RIP: 0033:0x7f53425a5332 [20680.072666] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20680.072670] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20680.072676] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20680.072678] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20680.072680] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20680.072683] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20680.072685] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20680.108671] [20680.130519] [20680.136260] ? show_regs.cold.9+0x22/0x2f [20680.136279] ? watchdog_timer_fn.cold.6+0x62/0x10a [20680.136286] ? __hrtimer_run_queues+0x1c2/0x4d0 [20680.136291] ? watchdog+0x40/0x40 [20680.136296] ? hrtimer_interrupt+0x128/0x2c0 [20680.136300] ? smp_apic_timer_interrupt+0xa6/0x270 [20680.136308] ? apic_timer_interrupt+0xf/0x20 [20680.136313] [20680.136486] ? native_safe_halt+0x2e/0x50 [20680.136492] kvm_wait+0xcd/0xe0 [20680.136501] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20680.136508] do_raw_spin_lock+0x177/0x190 [20680.136513] _raw_spin_lock+0x21/0x30 [20680.136635] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20680.136938] obd_mod_free+0x46/0x140 [ptlrpc] [20680.137455] mdc_commit_open+0x91/0xb0 [mdc] [20680.137492] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20680.141200] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20680.154138] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20680.154586] after_reply+0xe2c/0x1780 [ptlrpc] [20680.155312] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20680.155563] ? __next_timer_interrupt+0x160/0x160 [20680.155574] ptlrpcd+0xa82/0xe40 [ptlrpc] [20680.156148] ? do_wait_intr+0xf0/0xf0 [20680.156160] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20680.161710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20680.161721] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20680.161735] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20680.161740] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20680.161745] RAX: 0000000000006721 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20680.161748] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20680.161750] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20680.161752] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20680.161754] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20680.161758] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20680.161760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20680.161763] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20680.161771] Call Trace: [20680.161776] [20680.161781] ? show_regs.cold.9+0x22/0x2f [20680.161790] ? watchdog_timer_fn.cold.6+0x62/0x10a [20680.161797] ? __hrtimer_run_queues+0x1c2/0x4d0 [20680.161802] ? watchdog+0x40/0x40 [20680.161807] ? hrtimer_interrupt+0x128/0x2c0 [20680.161810] ? smp_apic_timer_interrupt+0xa6/0x270 [20680.161818] ? apic_timer_interrupt+0xf/0x20 [20680.161823] [20680.161825] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20680.161830] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20680.161837] do_raw_spin_lock+0x177/0x190 [20680.161842] _raw_spin_lock+0x21/0x30 [20680.161847] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20680.162119] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20680.166289] kthread+0x1d1/0x200 [20680.166462] ? set_kthread_struct+0x70/0x70 [20680.166469] ret_from_fork+0x1f/0x30 [20680.798601] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20680.801652] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20680.804114] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20680.806829] ? woken_wake_function+0x30/0x30 [20680.808881] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20680.811609] kthread+0x1d1/0x200 [20680.812988] ? set_kthread_struct+0x70/0x70 [20680.815531] ret_from_fork+0x1f/0x30 [20692.058147] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20692.066171] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20692.126387] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20692.135059] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20692.139761] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20692.143248] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20692.146401] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20692.157711] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20692.161429] RAX: 0000000000006bfc RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20692.164584] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [20692.167981] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20692.177314] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20692.189658] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20692.203162] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20692.214671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20692.219726] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20692.227007] Call Trace: [20692.231486] [20692.232460] ? show_regs.cold.9+0x22/0x2f [20692.242683] ? watchdog_timer_fn.cold.6+0x62/0x10a [20692.245406] ? __hrtimer_run_queues+0x1c2/0x4d0 [20692.251109] ? watchdog+0x40/0x40 [20692.257523] ? hrtimer_interrupt+0x128/0x2c0 [20692.260084] ? smp_apic_timer_interrupt+0xa6/0x270 [20692.265752] ? apic_timer_interrupt+0xf/0x20 [20692.267921] [20692.269187] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20692.276846] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20692.284756] ? newidle_balance.isra.93+0x207/0x580 [20692.290600] do_raw_spin_lock+0x177/0x190 [20692.293370] _raw_spin_lock+0x21/0x30 [20692.298008] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20692.312164] process_one_work+0x2c8/0x700 [20692.321478] worker_thread+0x296/0x6e0 [20692.323636] ? rescuer_thread+0x570/0x570 [20692.332081] kthread+0x1d1/0x200 [20692.336983] ? set_kthread_struct+0x70/0x70 [20692.338995] ret_from_fork+0x1f/0x30 [20694.495053] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18057s! [20694.498387] Showing busy workqueues and worker pools: [20694.500956] workqueue events: flags=0x0 [20694.502747] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20694.506694] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20694.512459] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20694.514998] pending: cpu_latency_work [ptlrpc] [20694.517416] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20694.520625] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20694.524315] workqueue events_long: flags=0x0 [20694.525958] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20694.528720] pending: lu_objects_flush [obdclass] [20694.530878] workqueue events_unbound: flags=0x2 [20694.533342] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20694.536230] pending: flush_memcg_stats_dwork, idle_cull_fn [20694.538621] workqueue events_power_efficient: flags=0x80 [20694.540911] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20694.543685] pending: check_lifetime, do_cache_clean [20694.546293] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20694.548988] pending: neigh_periodic_work, neigh_periodic_work [20694.551522] workqueue mm_percpu_wq: flags=0x8 [20694.553429] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20694.556474] pending: vmstat_update, lru_add_drain_per_cpu [20694.559313] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20694.562294] pending: vmstat_update, lru_add_drain_per_cpu [20694.564207] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20694.568027] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20694.571221] workqueue xprtiod: flags=0xa [20694.574537] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20694.577401] pending: xprt_autoclose, xs_error_handle [20694.579592] workqueue ipv6_addrconf: flags=0x40008 [20694.582989] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20694.586607] pending: addrconf_verify_work [20694.588662] workqueue knbd0-recv: flags=0x1a [20694.590682] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20694.594410] in-flight: 651:recv_work [20694.596373] workqueue ptlrpc_pinger: flags=0x2 [20694.598614] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20694.601276] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20694.604154] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18057s workers=4 idle: 15979 31824 8 [20694.610659] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20685s workers=2 idle: 654 [20694.886590] rcu: INFO: rcu_sched self-detected stall on CPU [20694.892858] rcu: 2-....: (10657260 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [20694.900242] (t=18600460 jiffies g=274185 q=68622) [20694.905750] rcu: rcu_sched kthread starved for 18058826 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [20694.916665] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [20694.921537] rcu: RCU grace-period kthread stack dump: [20694.924831] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [20694.928204] Call Trace: [20694.928935] __schedule+0x351/0xcb0 [20694.929818] schedule+0xc0/0x180 [20694.930815] schedule_timeout+0xb4/0x190 [20694.931793] ? do_raw_spin_unlock+0x75/0x190 [20694.933445] ? __next_timer_interrupt+0x160/0x160 [20694.938403] ? prepare_to_swait_event+0xf3/0x140 [20694.941680] rcu_gp_kthread+0x8dd/0x15b0 [20694.943006] ? force_qs_rnp+0x2c0/0x2c0 [20694.944293] kthread+0x1d1/0x200 [20694.945851] ? set_kthread_struct+0x70/0x70 [20694.947330] ret_from_fork+0x1f/0x30 [20694.948459] rcu: Stack dump where RCU GP kthread last ran: [20694.950416] Sending NMI from CPU 2 to CPUs 0: [20694.952567] NMI backtrace for cpu 0 [20694.952573] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20694.952576] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20694.952577] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20694.952773] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20694.952777] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20694.952779] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 [20694.952782] RAX: 0000000000007072 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20694.952783] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [20694.952785] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20694.952786] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20694.952788] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20694.952789] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20694.952790] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20694.952791] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20694.952793] Call Trace: [20694.952793] [20694.952795] ? show_regs.cold.9+0x22/0x2f [20694.952796] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [20694.952797] ? nmi_cpu_backtrace_handler+0x15/0x40 [20694.952799] ? nmi_handle+0xbd/0x210 [20694.952800] ? default_do_nmi+0x4b/0x1b0 [20694.952801] ? do_nmi+0x280/0x380 [20694.952802] ? end_repeat_nmi+0x16/0x1f [20694.952803] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20694.952805] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20694.952806] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20694.952807] [20694.952808] ? newidle_balance.isra.93+0x207/0x580 [20694.952809] do_raw_spin_lock+0x177/0x190 [20694.952810] _raw_spin_lock+0x21/0x30 [20694.952811] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20694.952812] process_one_work+0x2c8/0x700 [20694.952814] worker_thread+0x296/0x6e0 [20694.952815] ? rescuer_thread+0x570/0x570 [20694.952816] kthread+0x1d1/0x200 [20694.952817] ? set_kthread_struct+0x70/0x70 [20694.952818] ret_from_fork+0x1f/0x30 [20694.952888] Sending NMI from CPU 2 to CPUs 1: [20695.025247] NMI backtrace for cpu 1 [20695.025255] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20695.025257] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20695.025258] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20695.025260] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20695.025262] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [20695.025264] RAX: 0000000000000688 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20695.025266] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20695.025267] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20695.025269] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20695.025270] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20695.025272] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20695.025273] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20695.025275] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20695.025276] Call Trace: [20695.025277] [20695.025278] ? show_regs.cold.9+0x22/0x2f [20695.025279] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [20695.025280] ? nmi_cpu_backtrace_handler+0x15/0x40 [20695.025282] ? nmi_handle+0xbd/0x210 [20695.025283] ? default_do_nmi+0x4b/0x1b0 [20695.025284] ? do_nmi+0x280/0x380 [20695.025285] ? end_repeat_nmi+0x16/0x1f [20695.025286] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20695.025287] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20695.025288] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20695.025290] [20695.025291] do_raw_spin_lock+0x177/0x190 [20695.025292] _raw_spin_lock+0x21/0x30 [20695.025293] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20695.025295] ? do_raw_spin_unlock+0x75/0x190 [20695.025296] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20695.025297] ? do_wait_intr+0xf0/0xf0 [20695.025298] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20695.025299] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20695.025300] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20695.025301] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20695.025302] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20695.025304] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20695.025305] mdc_intent_lock+0x353/0x7e0 [mdc] [20695.025306] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20695.025307] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20695.025308] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20695.025309] lmv_intent_open+0x338/0x1150 [lmv] [20695.025310] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20695.025311] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20695.025312] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20695.025313] ll_intent_lock+0xc5/0xaa0 [lustre] [20695.025314] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20695.025315] ll_lookup_it+0xe0e/0x2270 [lustre] [20695.025316] ? do_raw_spin_unlock+0x75/0x190 [20695.025317] ? _raw_spin_unlock+0x12/0x30 [20695.025318] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20695.025319] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20695.025320] ll_atomic_open+0x36a/0xd70 [lustre] [20695.025321] lookup_open+0x356/0xb40 [20695.025322] do_last+0x313/0xfc0 [20695.025323] ? nd_jump_root+0xe5/0x160 [20695.025324] ? path_init+0x437/0x520 [20695.025325] path_openat+0xf7/0x500 [20695.025326] ? do_raw_spin_unlock+0x75/0x190 [20695.025327] do_filp_open+0x99/0x140 [20695.025328] ? getname_flags+0x6e/0x330 [20695.025329] ? __check_object_size+0xff/0x256 [20695.025330] ? do_raw_spin_unlock+0x75/0x190 [20695.025331] ? _raw_spin_unlock+0x12/0x30 [20695.025332] do_sys_openat2+0x2b4/0x410 [20695.025333] do_sys_open+0x73/0xa0 [20695.025334] __x64_sys_openat+0x24/0x30 [20695.025335] do_syscall_64+0xc1/0x440 [20695.025336] entry_SYSCALL_64_after_hwframe+0x49/0xae [20695.025336] RIP: 0033:0x7f53425a5332 [20695.025338] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20695.025340] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20695.025343] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20695.025345] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20695.025346] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20695.025348] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20695.025349] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20695.026012] NMI backtrace for cpu 2 [20695.192446] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20695.196606] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20695.200526] Call Trace: [20695.201369] [20695.201952] ? dump_stack+0xbb/0x10e [20695.203015] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [20695.204836] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [20695.206112] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [20695.207284] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [20695.208968] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [20695.211160] ? rcu_dump_cpu_stacks+0x11b/0x171 [20695.212588] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [20695.214239] ? update_process_times+0x6d/0xc0 [20695.215866] ? tick_sched_handle.isra.18+0x27/0x90 [20695.217926] ? tick_sched_timer+0x53/0xb0 [20695.219702] ? __hrtimer_run_queues+0x1c2/0x4d0 [20695.224504] ? tick_sched_handle.isra.18+0x90/0x90 [20695.226475] ? hrtimer_interrupt+0x128/0x2c0 [20695.228247] ? smp_apic_timer_interrupt+0xa6/0x270 [20695.230403] ? apic_timer_interrupt+0xf/0x20 [20695.232012] [20695.232750] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20695.234619] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20695.236407] ? do_raw_spin_lock+0x177/0x190 [20695.238323] ? _raw_spin_lock+0x21/0x30 [20695.239394] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20695.241135] ? obd_mod_free+0x46/0x140 [ptlrpc] [20695.243195] ? mdc_commit_open+0x91/0xb0 [mdc] [20695.244805] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20695.246523] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20695.248347] ? after_reply+0xe2c/0x1780 [ptlrpc] [20695.252178] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20695.254938] ? __next_timer_interrupt+0x160/0x160 [20695.256703] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [20695.262156] ? do_wait_intr+0xf0/0xf0 [20695.263015] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20695.268761] ? kthread+0x1d1/0x200 [20695.271433] ? set_kthread_struct+0x70/0x70 [20695.273974] ? ret_from_fork+0x1f/0x30 [20708.058005] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20708.068637] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20708.083488] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20708.086294] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20708.088723] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20708.090248] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20708.095128] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20708.096960] RAX: 00000000000060de RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20708.098550] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20708.100485] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20708.102235] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20708.104046] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20708.105752] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20708.107949] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20708.109494] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20708.111444] Call Trace: [20708.112167] [20708.112635] ? show_regs.cold.9+0x22/0x2f [20708.113696] ? watchdog_timer_fn.cold.6+0x62/0x10a [20708.115238] ? __hrtimer_run_queues+0x1c2/0x4d0 [20708.116371] ? watchdog+0x40/0x40 [20708.117398] ? hrtimer_interrupt+0x128/0x2c0 [20708.118594] ? smp_apic_timer_interrupt+0xa6/0x270 [20708.120141] ? apic_timer_interrupt+0xf/0x20 [20708.121782] [20708.122456] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20708.124344] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20708.126108] do_raw_spin_lock+0x177/0x190 [20708.127531] _raw_spin_lock+0x21/0x30 [20708.128781] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20708.131199] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20708.133286] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20708.135425] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20708.137695] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20708.139764] ? woken_wake_function+0x30/0x30 [20708.141014] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20708.142844] kthread+0x1d1/0x200 [20708.143761] ? set_kthread_struct+0x70/0x70 [20708.145251] ret_from_fork+0x1f/0x30 [20720.053061] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [20720.056774] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [20720.058087] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20720.058170] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20720.058177] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20720.058183] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20720.058522] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20720.058536] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20720.058540] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20720.058544] RAX: 0000000000001a94 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20720.058549] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [20720.058551] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20720.058553] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20720.058556] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20720.058558] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20720.058562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20720.058564] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20720.058572] Call Trace: [20720.058576] [20720.058581] ? show_regs.cold.9+0x22/0x2f [20720.058590] ? watchdog_timer_fn.cold.6+0x62/0x10a [20720.058595] ? __hrtimer_run_queues+0x1c2/0x4d0 [20720.058602] ? watchdog+0x40/0x40 [20720.058607] ? hrtimer_interrupt+0x128/0x2c0 [20720.058610] ? smp_apic_timer_interrupt+0xa6/0x270 [20720.058617] ? apic_timer_interrupt+0xf/0x20 [20720.058622] [20720.058625] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20720.058629] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20720.058635] ? newidle_balance.isra.93+0x207/0x580 [20720.058653] do_raw_spin_lock+0x177/0x190 [20720.058659] _raw_spin_lock+0x21/0x30 [20720.058663] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20720.059144] process_one_work+0x2c8/0x700 [20720.059151] worker_thread+0x296/0x6e0 [20720.059155] ? rescuer_thread+0x570/0x570 [20720.059159] kthread+0x1d1/0x200 [20720.059163] ? set_kthread_struct+0x70/0x70 [20720.059169] ret_from_fork+0x1f/0x30 [20720.059405] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [20720.059411] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20720.059478] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20720.059482] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20720.059484] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [20720.059492] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [20720.059496] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20720.059499] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20720.059502] RDX: 0000000000005f70 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20720.059504] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20720.059507] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20720.059509] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20720.059511] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20720.059514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20720.059516] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20720.059522] Call Trace: [20720.059525] [20720.059527] ? show_regs.cold.9+0x22/0x2f [20720.059533] ? watchdog_timer_fn.cold.6+0x62/0x10a [20720.059537] ? __hrtimer_run_queues+0x1c2/0x4d0 [20720.059540] ? watchdog+0x40/0x40 [20720.059544] ? hrtimer_interrupt+0x128/0x2c0 [20720.059548] ? smp_apic_timer_interrupt+0xa6/0x270 [20720.059552] ? apic_timer_interrupt+0xf/0x20 [20720.059557] [20720.059559] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20720.059563] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20720.059567] do_raw_spin_lock+0x177/0x190 [20720.059571] _raw_spin_lock+0x21/0x30 [20720.059574] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20720.060137] obd_mod_free+0x46/0x140 [ptlrpc] [20720.060373] mdc_commit_open+0x91/0xb0 [mdc] [20720.060413] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20720.060618] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20720.066963] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20720.067025] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20720.067031] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20720.067033] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20720.084595] after_reply+0xe2c/0x1780 [ptlrpc] [20720.088433] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20720.088445] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20720.088452] RAX: 0000000000007487 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20720.088455] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20720.088457] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20720.091715] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20720.093741] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20720.093747] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20720.093750] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20720.093753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20720.093755] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20720.093763] Call Trace: [20720.093769] [20720.093774] ? show_regs.cold.9+0x22/0x2f [20720.093785] ? watchdog_timer_fn.cold.6+0x62/0x10a [20720.093791] ? __hrtimer_run_queues+0x1c2/0x4d0 [20720.093795] ? watchdog+0x40/0x40 [20720.093799] ? hrtimer_interrupt+0x128/0x2c0 [20720.093803] ? smp_apic_timer_interrupt+0xa6/0x270 [20720.093810] ? apic_timer_interrupt+0xf/0x20 [20720.093815] [20720.093817] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20720.093824] do_raw_spin_lock+0x177/0x190 [20720.093829] _raw_spin_lock+0x21/0x30 [20720.093833] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20720.100014] ? __next_timer_interrupt+0x160/0x160 [20720.110863] ? do_raw_spin_unlock+0x75/0x190 [20720.114586] ptlrpcd+0xa82/0xe40 [ptlrpc] [20720.115274] ? do_wait_intr+0xf0/0xf0 [20720.115288] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20720.115556] kthread+0x1d1/0x200 [20720.115561] ? set_kthread_struct+0x70/0x70 [20720.115565] ret_from_fork+0x1f/0x30 [20720.419027] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20720.425093] ? do_wait_intr+0xf0/0xf0 [20720.426378] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20720.433299] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20720.440872] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20720.446168] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20720.450373] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20720.452878] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20720.455270] mdc_intent_lock+0x353/0x7e0 [mdc] [20720.459416] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20720.462275] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20720.470357] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20720.472323] lmv_intent_open+0x338/0x1150 [lmv] [20720.479249] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20720.488925] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20720.490539] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20720.497131] ll_intent_lock+0xc5/0xaa0 [lustre] [20720.503012] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20720.510124] ll_lookup_it+0xe0e/0x2270 [lustre] [20720.517249] ? do_raw_spin_unlock+0x75/0x190 [20720.524003] ? _raw_spin_unlock+0x12/0x30 [20720.527925] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20720.533003] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20720.539214] ll_atomic_open+0x36a/0xd70 [lustre] [20720.544704] lookup_open+0x356/0xb40 [20720.546852] do_last+0x313/0xfc0 [20720.548880] ? nd_jump_root+0xe5/0x160 [20720.551670] ? path_init+0x437/0x520 [20720.553939] path_openat+0xf7/0x500 [20720.554983] ? do_raw_spin_unlock+0x75/0x190 [20720.564822] do_filp_open+0x99/0x140 [20720.567105] ? getname_flags+0x6e/0x330 [20720.569615] ? __check_object_size+0xff/0x256 [20720.573425] ? do_raw_spin_unlock+0x75/0x190 [20720.576917] ? _raw_spin_unlock+0x12/0x30 [20720.579950] do_sys_openat2+0x2b4/0x410 [20720.584419] do_sys_open+0x73/0xa0 [20720.585671] __x64_sys_openat+0x24/0x30 [20720.587037] do_syscall_64+0xc1/0x440 [20720.589032] entry_SYSCALL_64_after_hwframe+0x49/0xae [20720.591049] RIP: 0033:0x7f53425a5332 [20720.593997] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20720.612108] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20720.617016] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20720.623738] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20720.631115] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20720.636729] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20720.639364] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20724.704039] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18087s! [20724.708535] Showing busy workqueues and worker pools: [20724.711393] workqueue events: flags=0x0 [20724.714724] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20724.718485] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20724.725546] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20724.733281] pending: cpu_latency_work [ptlrpc] [20724.736765] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20724.739786] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20724.743216] workqueue events_long: flags=0x0 [20724.744471] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20724.746610] pending: lu_objects_flush [obdclass] [20724.748619] workqueue events_unbound: flags=0x2 [20724.749935] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20724.752635] pending: flush_memcg_stats_dwork, idle_cull_fn [20724.756071] workqueue events_power_efficient: flags=0x80 [20724.759171] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20724.762053] pending: check_lifetime, do_cache_clean [20724.764031] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20724.766290] pending: neigh_periodic_work, neigh_periodic_work [20724.769807] workqueue mm_percpu_wq: flags=0x8 [20724.771330] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20724.778683] pending: vmstat_update, lru_add_drain_per_cpu [20724.780787] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20724.784169] pending: vmstat_update, lru_add_drain_per_cpu [20724.786271] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20724.789264] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20724.792024] workqueue xprtiod: flags=0xa [20724.795618] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20724.798649] pending: xprt_autoclose, xs_error_handle [20724.802896] workqueue ipv6_addrconf: flags=0x40008 [20724.806612] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20724.812375] pending: addrconf_verify_work [20724.814577] workqueue knbd0-recv: flags=0x1a [20724.815952] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20724.818610] in-flight: 651:recv_work [20724.821698] workqueue ptlrpc_pinger: flags=0x2 [20724.823262] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20724.828236] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20724.835440] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18087s workers=4 idle: 15979 31824 8 [20724.840131] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20715s workers=2 idle: 654 [20736.058642] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20736.063063] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20736.100182] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20736.106885] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20736.112063] RIP: 0010:native_safe_halt+0x2e/0x50 [20736.116292] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20736.131348] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20736.140332] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20736.149728] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20736.151535] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20736.158642] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20736.164748] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20736.169776] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20736.174711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20736.177831] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20736.181814] Call Trace: [20736.182896] [20736.184237] ? show_regs.cold.9+0x22/0x2f [20736.186425] ? watchdog_timer_fn.cold.6+0x62/0x10a [20736.190340] ? __hrtimer_run_queues+0x1c2/0x4d0 [20736.193270] ? watchdog+0x40/0x40 [20736.194712] ? hrtimer_interrupt+0x128/0x2c0 [20736.199787] ? smp_apic_timer_interrupt+0xa6/0x270 [20736.203161] ? apic_timer_interrupt+0xf/0x20 [20736.206166] [20736.207228] ? native_safe_halt+0x2e/0x50 [20736.209646] kvm_wait+0xcd/0xe0 [20736.211720] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20736.215121] do_raw_spin_lock+0x177/0x190 [20736.218757] _raw_spin_lock+0x21/0x30 [20736.228462] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20736.235723] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20736.239111] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20736.245165] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20736.248638] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20736.254037] ? woken_wake_function+0x30/0x30 [20736.255598] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20736.261259] kthread+0x1d1/0x200 [20736.265626] ? set_kthread_struct+0x70/0x70 [20736.274079] ret_from_fork+0x1f/0x30 [20748.056167] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [20748.060601] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [20748.060678] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20748.060755] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20748.060761] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20748.060764] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20748.060778] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20748.060783] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20748.060944] RAX: 000000000000428b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20748.060949] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20748.060952] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20748.060954] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20748.060956] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20748.060960] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20748.060963] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20748.060966] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20748.060975] Call Trace: [20748.060981] [20748.060987] ? show_regs.cold.9+0x22/0x2f [20748.060998] ? watchdog_timer_fn.cold.6+0x62/0x10a [20748.061016] ? __hrtimer_run_queues+0x1c2/0x4d0 [20748.061021] ? watchdog+0x40/0x40 [20748.061071] ? hrtimer_interrupt+0x128/0x2c0 [20748.061077] ? smp_apic_timer_interrupt+0xa6/0x270 [20748.061086] ? apic_timer_interrupt+0xf/0x20 [20748.061091] [20748.061095] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20748.061103] do_raw_spin_lock+0x177/0x190 [20748.061108] _raw_spin_lock+0x21/0x30 [20748.061114] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20748.069763] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) [20748.120771] ? do_raw_spin_unlock+0x75/0x190 [20748.120792] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20748.121262] ? do_wait_intr+0xf0/0xf0 [20748.121272] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20748.122141] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20748.128676] mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20748.128735] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20748.128742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20748.128745] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [20748.128759] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [20748.128764] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20748.128773] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20748.128776] RDX: 0000000000006ff4 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20748.128778] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20748.128780] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20748.128782] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20748.128785] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20748.128789] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20748.128791] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20748.128799] Call Trace: [20748.128804] [20748.128809] ? show_regs.cold.9+0x22/0x2f [20748.128818] ? watchdog_timer_fn.cold.6+0x62/0x10a [20748.128824] ? __hrtimer_run_queues+0x1c2/0x4d0 [20748.128828] ? watchdog+0x40/0x40 [20748.128833] ? hrtimer_interrupt+0x128/0x2c0 [20748.128836] ? smp_apic_timer_interrupt+0xa6/0x270 [20748.128843] ? apic_timer_interrupt+0xf/0x20 [20748.128848] [20748.128850] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20748.128855] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20748.128859] do_raw_spin_lock+0x177/0x190 [20748.128864] _raw_spin_lock+0x21/0x30 [20748.128869] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20748.129461] obd_mod_free+0x46/0x140 [ptlrpc] [20748.135590] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20748.135905] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20748.136022] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20748.136077] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20748.136110] mdc_intent_lock+0x353/0x7e0 [mdc] [20748.136138] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20748.136504] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20748.136913] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20748.137003] lmv_intent_open+0x338/0x1150 [lmv] [20748.137049] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20748.137142] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20748.137500] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20748.137537] ll_intent_lock+0xc5/0xaa0 [lustre] [20748.137689] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20748.137778] ll_lookup_it+0xe0e/0x2270 [lustre] [20748.137933] ? do_raw_spin_unlock+0x75/0x190 [20748.137946] ? _raw_spin_unlock+0x12/0x30 [20748.137953] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20748.137980] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20748.138001] ll_atomic_open+0x36a/0xd70 [lustre] [20748.138147] lookup_open+0x356/0xb40 [20748.138264] do_last+0x313/0xfc0 [20748.138273] ? nd_jump_root+0xe5/0x160 [20748.138277] ? path_init+0x437/0x520 [20748.138282] path_openat+0xf7/0x500 [20748.138287] ? do_raw_spin_unlock+0x75/0x190 [20748.138292] do_filp_open+0x99/0x140 [20748.138299] ? getname_flags+0x6e/0x330 [20748.138304] ? __check_object_size+0xff/0x256 [20748.138312] ? do_raw_spin_unlock+0x75/0x190 [20748.138428] ? _raw_spin_unlock+0x12/0x30 [20748.138433] do_sys_openat2+0x2b4/0x410 [20748.138438] do_sys_open+0x73/0xa0 [20748.138443] __x64_sys_openat+0x24/0x30 [20748.138447] do_syscall_64+0xc1/0x440 [20748.138453] entry_SYSCALL_64_after_hwframe+0x49/0xae [20748.138459] RIP: 0033:0x7f53425a5332 [20748.138467] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20748.138470] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20748.138475] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20748.138477] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20748.138479] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20748.138482] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20748.138484] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20748.665971] mdc_commit_open+0x91/0xb0 [mdc] [20748.667480] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20748.669720] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20748.671935] after_reply+0xe2c/0x1780 [ptlrpc] [20748.674279] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20748.676626] ? __next_timer_interrupt+0x160/0x160 [20748.679235] ptlrpcd+0xa82/0xe40 [ptlrpc] [20748.687873] ? do_wait_intr+0xf0/0xf0 [20748.690297] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20748.694085] kthread+0x1d1/0x200 [20748.695384] ? set_kthread_struct+0x70/0x70 [20748.700572] ret_from_fork+0x1f/0x30 [20752.052027] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20752.054531] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20752.110318] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20752.113556] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20752.116391] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20752.122024] RIP: 0010:native_safe_halt+0x2e/0x50 [20752.124195] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20752.133958] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20752.136811] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20752.139713] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [20752.142421] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20752.146071] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20752.149053] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20752.156057] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20752.157931] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20752.175664] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20752.184061] Call Trace: [20752.185682] [20752.189296] ? show_regs.cold.9+0x22/0x2f [20752.190745] ? watchdog_timer_fn.cold.6+0x62/0x10a [20752.191915] ? __hrtimer_run_queues+0x1c2/0x4d0 [20752.193284] ? watchdog+0x40/0x40 [20752.194437] ? hrtimer_interrupt+0x128/0x2c0 [20752.195643] ? smp_apic_timer_interrupt+0xa6/0x270 [20752.197105] ? apic_timer_interrupt+0xf/0x20 [20752.198211] [20752.198861] ? native_safe_halt+0x2e/0x50 [20752.200000] kvm_wait+0xcd/0xe0 [20752.201198] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20752.203283] ? newidle_balance.isra.93+0x207/0x580 [20752.205846] do_raw_spin_lock+0x177/0x190 [20752.207454] _raw_spin_lock+0x21/0x30 [20752.209056] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20752.211978] process_one_work+0x2c8/0x700 [20752.214244] worker_thread+0x296/0x6e0 [20752.216279] ? rescuer_thread+0x570/0x570 [20752.217663] kthread+0x1d1/0x200 [20752.218731] ? set_kthread_struct+0x70/0x70 [20752.220000] ret_from_fork+0x1f/0x30 [20754.921293] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18117s! [20754.935063] Showing busy workqueues and worker pools: [20754.936528] workqueue events: flags=0x0 [20754.937652] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20754.947365] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20754.966725] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20754.976266] pending: cpu_latency_work [ptlrpc] [20754.988698] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20754.991181] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20755.001733] workqueue events_long: flags=0x0 [20755.005165] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20755.014004] pending: lu_objects_flush [obdclass] [20755.021854] workqueue events_unbound: flags=0x2 [20755.025279] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20755.029814] pending: flush_memcg_stats_dwork, idle_cull_fn [20755.036320] workqueue events_power_efficient: flags=0x80 [20755.040921] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20755.048592] pending: check_lifetime, do_cache_clean [20755.055927] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20755.063202] pending: neigh_periodic_work, neigh_periodic_work [20755.067020] workqueue mm_percpu_wq: flags=0x8 [20755.069524] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20755.072732] pending: vmstat_update, lru_add_drain_per_cpu [20755.075703] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20755.078664] pending: vmstat_update, lru_add_drain_per_cpu [20755.081241] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20755.093268] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20755.103161] workqueue xprtiod: flags=0xa [20755.109704] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20755.115279] pending: xprt_autoclose, xs_error_handle [20755.118690] workqueue ipv6_addrconf: flags=0x40008 [20755.120422] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20755.129841] pending: addrconf_verify_work [20755.132637] workqueue knbd0-recv: flags=0x1a [20755.136919] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20755.144370] in-flight: 651:recv_work [20755.146332] workqueue ptlrpc_pinger: flags=0x2 [20755.148754] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20755.150940] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20755.154021] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18118s workers=4 idle: 15979 31824 8 [20755.157493] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20745s workers=2 idle: 654 [20764.058117] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [20764.061759] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20764.106190] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20764.116593] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20764.120890] RIP: 0010:native_safe_halt+0x2e/0x50 [20764.131123] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20764.148515] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20764.159720] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20764.163559] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20764.167468] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20764.171187] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20764.174818] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20764.178688] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20764.182533] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20764.185002] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20764.188286] Call Trace: [20764.189323] [20764.190230] ? show_regs.cold.9+0x22/0x2f [20764.192154] ? watchdog_timer_fn.cold.6+0x62/0x10a [20764.194459] ? __hrtimer_run_queues+0x1c2/0x4d0 [20764.199305] ? watchdog+0x40/0x40 [20764.203731] ? hrtimer_interrupt+0x128/0x2c0 [20764.206110] ? smp_apic_timer_interrupt+0xa6/0x270 [20764.210302] ? apic_timer_interrupt+0xf/0x20 [20764.212300] [20764.215257] ? native_safe_halt+0x2e/0x50 [20764.216753] kvm_wait+0xcd/0xe0 [20764.217762] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20764.220834] do_raw_spin_lock+0x177/0x190 [20764.223216] _raw_spin_lock+0x21/0x30 [20764.225177] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20764.230712] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20764.237766] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20764.245070] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20764.252792] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20764.261693] ? woken_wake_function+0x30/0x30 [20764.267343] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20764.271230] kthread+0x1d1/0x200 [20764.272781] ? set_kthread_struct+0x70/0x70 [20764.275360] ret_from_fork+0x1f/0x30 [20776.055013] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [20776.059464] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) [20776.059511] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [20776.059521] Modules linked in: [20776.085298] rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20776.085346] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20776.085354] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20776.085357] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20776.085373] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20776.092700] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20776.092772] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20776.092779] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20776.092782] RIP: 0010:native_safe_halt+0x2e/0x50 [20776.092795] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20776.092799] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20776.092803] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20776.092806] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20776.092808] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20776.092811] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20776.092813] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20776.092816] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20776.092819] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20776.092822] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20776.092830] Call Trace: [20776.092836] [20776.092977] ? show_regs.cold.9+0x22/0x2f [20776.092989] ? watchdog_timer_fn.cold.6+0x62/0x10a [20776.092995] ? __hrtimer_run_queues+0x1c2/0x4d0 [20776.092999] ? watchdog+0x40/0x40 [20776.093004] ? hrtimer_interrupt+0x128/0x2c0 [20776.093008] ? smp_apic_timer_interrupt+0xa6/0x270 [20776.093015] ? apic_timer_interrupt+0xf/0x20 [20776.093019] [20776.093022] ? native_safe_halt+0x2e/0x50 [20776.093027] kvm_wait+0xcd/0xe0 [20776.093034] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20776.093041] do_raw_spin_lock+0x177/0x190 [20776.093046] _raw_spin_lock+0x21/0x30 [20776.093052] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20776.094280] obd_mod_free+0x46/0x140 [ptlrpc] [20776.094496] mdc_commit_open+0x91/0xb0 [mdc] [20776.094533] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20776.094744] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20776.094958] after_reply+0xe2c/0x1780 [ptlrpc] [20776.095153] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20776.095352] ? __next_timer_interrupt+0x160/0x160 [20776.095364] ptlrpcd+0xa82/0xe40 [ptlrpc] [20776.095570] ? do_wait_intr+0xf0/0xf0 [20776.095579] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20776.097640] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20776.097653] RAX: 00000000000036d8 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20776.097656] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20776.097717] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20776.097720] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20776.097722] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20776.097725] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20776.097729] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20776.097731] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20776.097740] Call Trace: [20776.097748] [20776.097753] ? show_regs.cold.9+0x22/0x2f [20776.097766] ? watchdog_timer_fn.cold.6+0x62/0x10a [20776.097773] ? __hrtimer_run_queues+0x1c2/0x4d0 [20776.097776] ? watchdog+0x40/0x40 [20776.097780] ? hrtimer_interrupt+0x128/0x2c0 [20776.097784] ? smp_apic_timer_interrupt+0xa6/0x270 [20776.097792] ? apic_timer_interrupt+0xf/0x20 [20776.097796] [20776.097799] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20776.097807] do_raw_spin_lock+0x177/0x190 [20776.097811] _raw_spin_lock+0x21/0x30 [20776.097816] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20776.098070] ? do_raw_spin_unlock+0x75/0x190 [20776.098076] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20776.120876] kthread+0x1d1/0x200 [20776.120894] ? set_kthread_struct+0x70/0x70 [20776.120899] ret_from_fork+0x1f/0x30 [20776.439616] ? do_wait_intr+0xf0/0xf0 [20776.445078] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20776.447567] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20776.451765] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20776.454506] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20776.457431] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20776.459718] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20776.462183] mdc_intent_lock+0x353/0x7e0 [mdc] [20776.464987] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20776.467327] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20776.469961] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20776.472249] lmv_intent_open+0x338/0x1150 [lmv] [20776.474077] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20776.479577] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20776.481361] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20776.483361] ll_intent_lock+0xc5/0xaa0 [lustre] [20776.485290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20776.487178] ll_lookup_it+0xe0e/0x2270 [lustre] [20776.489013] ? do_raw_spin_unlock+0x75/0x190 [20776.490746] ? _raw_spin_unlock+0x12/0x30 [20776.492147] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20776.494246] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20776.496227] ll_atomic_open+0x36a/0xd70 [lustre] [20776.499913] lookup_open+0x356/0xb40 [20776.504814] do_last+0x313/0xfc0 [20776.506209] ? nd_jump_root+0xe5/0x160 [20776.507575] ? path_init+0x437/0x520 [20776.512332] path_openat+0xf7/0x500 [20776.513887] ? do_raw_spin_unlock+0x75/0x190 [20776.515624] do_filp_open+0x99/0x140 [20776.517282] ? getname_flags+0x6e/0x330 [20776.518564] ? __check_object_size+0xff/0x256 [20776.520954] ? do_raw_spin_unlock+0x75/0x190 [20776.522836] ? _raw_spin_unlock+0x12/0x30 [20776.524767] do_sys_openat2+0x2b4/0x410 [20776.526439] do_sys_open+0x73/0xa0 [20776.528035] __x64_sys_openat+0x24/0x30 [20776.529903] do_syscall_64+0xc1/0x440 [20776.531568] entry_SYSCALL_64_after_hwframe+0x49/0xae [20776.533714] RIP: 0033:0x7f53425a5332 [20776.535000] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20776.544438] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20776.552010] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20776.554606] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20776.557396] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20776.559584] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20776.562489] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20780.053010] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20780.065606] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20780.110634] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20780.116561] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20780.120872] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20780.124067] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20780.127048] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20780.135791] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20780.139327] RAX: 0000000000007b0b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20780.142540] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [20780.147128] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20780.154474] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20780.158719] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20780.162911] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20780.165727] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20780.167962] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20780.175153] Call Trace: [20780.176756] [20780.177351] ? show_regs.cold.9+0x22/0x2f [20780.179531] ? watchdog_timer_fn.cold.6+0x62/0x10a [20780.183095] ? __hrtimer_run_queues+0x1c2/0x4d0 [20780.185176] ? watchdog+0x40/0x40 [20780.187560] ? hrtimer_interrupt+0x128/0x2c0 [20780.192467] ? smp_apic_timer_interrupt+0xa6/0x270 [20780.195033] ? apic_timer_interrupt+0xf/0x20 [20780.197098] [20780.198166] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20780.202797] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20780.205576] ? newidle_balance.isra.93+0x207/0x580 [20780.207873] do_raw_spin_lock+0x177/0x190 [20780.214252] _raw_spin_lock+0x21/0x30 [20780.218122] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20780.221184] process_one_work+0x2c8/0x700 [20780.222824] worker_thread+0x296/0x6e0 [20780.226088] ? rescuer_thread+0x570/0x570 [20780.229501] kthread+0x1d1/0x200 [20780.231427] ? set_kthread_struct+0x70/0x70 [20780.233148] ret_from_fork+0x1f/0x30 [20785.640777] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18148s! [20785.645365] Showing busy workqueues and worker pools: [20785.646716] workqueue events: flags=0x0 [20785.648079] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20785.650887] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20785.656725] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20785.661627] pending: cpu_latency_work [ptlrpc] [20785.664301] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20785.670444] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20785.677126] workqueue events_long: flags=0x0 [20785.682373] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20785.697182] pending: lu_objects_flush [obdclass] [20785.698706] workqueue events_unbound: flags=0x2 [20785.699973] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20785.702960] pending: flush_memcg_stats_dwork, idle_cull_fn [20785.709061] workqueue events_power_efficient: flags=0x80 [20785.712303] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20785.715623] pending: check_lifetime, do_cache_clean [20785.718443] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20785.721345] pending: neigh_periodic_work, neigh_periodic_work [20785.724251] workqueue mm_percpu_wq: flags=0x8 [20785.726825] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20785.730031] pending: vmstat_update, lru_add_drain_per_cpu [20785.732885] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20785.735966] pending: vmstat_update, lru_add_drain_per_cpu [20785.738067] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20785.740418] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20785.743854] workqueue xprtiod: flags=0xa [20785.745962] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20785.749357] pending: xprt_autoclose, xs_error_handle [20785.752287] workqueue ipv6_addrconf: flags=0x40008 [20785.758154] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20785.763935] pending: addrconf_verify_work [20785.765156] workqueue knbd0-recv: flags=0x1a [20785.767204] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20785.770636] in-flight: 651:recv_work [20785.772330] workqueue ptlrpc_pinger: flags=0x2 [20785.774313] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20785.777727] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20785.781888] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18148s workers=4 idle: 15979 31824 8 [20785.786990] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20776s workers=2 idle: 654 [20792.058066] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [20792.061804] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20792.082570] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20792.086354] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20792.093790] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20792.096275] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20792.103132] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20792.105649] RAX: 0000000000001b8c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20792.108254] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20792.110687] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20792.112640] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20792.114703] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20792.116470] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20792.118338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20792.119942] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20792.122185] Call Trace: [20792.122832] [20792.123233] ? show_regs.cold.9+0x22/0x2f [20792.124137] ? watchdog_timer_fn.cold.6+0x62/0x10a [20792.125340] ? __hrtimer_run_queues+0x1c2/0x4d0 [20792.126727] ? watchdog+0x40/0x40 [20792.127909] ? hrtimer_interrupt+0x128/0x2c0 [20792.129466] ? smp_apic_timer_interrupt+0xa6/0x270 [20792.131112] ? apic_timer_interrupt+0xf/0x20 [20792.132640] [20792.133396] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20792.135284] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20792.137227] do_raw_spin_lock+0x177/0x190 [20792.138650] _raw_spin_lock+0x21/0x30 [20792.140032] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20792.142436] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20792.144521] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20792.146651] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20792.148728] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20792.150884] ? woken_wake_function+0x30/0x30 [20792.152235] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20792.154283] kthread+0x1d1/0x200 [20792.155543] ? set_kthread_struct+0x70/0x70 [20792.157132] ret_from_fork+0x1f/0x30 [20804.055021] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [20804.059123] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20804.059752] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [20804.112551] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20804.112564] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20804.112567] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [20804.112626] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [20804.112630] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20804.112636] RAX: 00000000000049c6 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20804.112638] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20804.112640] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20804.112642] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20804.112644] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20804.112648] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20804.112650] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20804.112653] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20804.112661] Call Trace: [20804.112680] [20804.112685] ? show_regs.cold.9+0x22/0x2f [20804.112693] ? watchdog_timer_fn.cold.6+0x62/0x10a [20804.112697] ? __hrtimer_run_queues+0x1c2/0x4d0 [20804.112700] ? watchdog+0x40/0x40 [20804.112703] ? hrtimer_interrupt+0x128/0x2c0 [20804.112706] ? smp_apic_timer_interrupt+0xa6/0x270 [20804.112712] ? apic_timer_interrupt+0xf/0x20 [20804.112715] [20804.112717] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [20804.112724] do_raw_spin_lock+0x177/0x190 [20804.112727] _raw_spin_lock+0x21/0x30 [20804.112730] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20804.112953] ? do_raw_spin_unlock+0x75/0x190 [20804.112958] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20804.113129] ? do_wait_intr+0xf0/0xf0 [20804.113136] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20804.115189] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20804.115409] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20804.123349] Modules linked in: [20804.129697] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20804.129982] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20804.130029] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20804.130060] mdc_intent_lock+0x353/0x7e0 [mdc] [20804.130087] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20804.130307] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20804.130662] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20804.130693] lmv_intent_open+0x338/0x1150 [lmv] [20804.130719] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20804.130965] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20804.131052] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20804.131075] ll_intent_lock+0xc5/0xaa0 [lustre] [20804.131328] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20804.131429] ll_lookup_it+0xe0e/0x2270 [lustre] [20804.131515] ? do_raw_spin_unlock+0x75/0x190 [20804.131523] ? _raw_spin_unlock+0x12/0x30 [20804.131529] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20804.131558] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20804.131815] ll_atomic_open+0x36a/0xd70 [lustre] [20804.131912] lookup_open+0x356/0xb40 [20804.131921] do_last+0x313/0xfc0 [20804.131926] ? nd_jump_root+0xe5/0x160 [20804.131930] ? path_init+0x437/0x520 [20804.131934] path_openat+0xf7/0x500 [20804.131939] ? do_raw_spin_unlock+0x75/0x190 [20804.131944] do_filp_open+0x99/0x140 [20804.131950] ? getname_flags+0x6e/0x330 [20804.131956] ? __check_object_size+0xff/0x256 [20804.131962] ? do_raw_spin_unlock+0x75/0x190 [20804.131966] ? _raw_spin_unlock+0x12/0x30 [20804.131971] do_sys_openat2+0x2b4/0x410 [20804.131975] do_sys_open+0x73/0xa0 [20804.131979] __x64_sys_openat+0x24/0x30 [20804.131982] do_syscall_64+0xc1/0x440 [20804.131989] entry_SYSCALL_64_after_hwframe+0x49/0xae [20804.131995] RIP: 0033:0x7f53425a5332 [20804.132047] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20804.132052] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20804.132057] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20804.132060] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20804.132062] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20804.132064] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20804.132066] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20804.442917] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20804.463875] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20804.469810] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20804.473975] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [20804.476748] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [20804.487281] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20804.491277] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20804.493969] RDX: 0000000000007e3a RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20804.499848] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20804.502985] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20804.505906] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20804.510259] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20804.513304] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20804.517923] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20804.521106] Call Trace: [20804.522134] [20804.523027] ? show_regs.cold.9+0x22/0x2f [20804.525014] ? watchdog_timer_fn.cold.6+0x62/0x10a [20804.527694] ? __hrtimer_run_queues+0x1c2/0x4d0 [20804.529525] ? watchdog+0x40/0x40 [20804.531079] ? hrtimer_interrupt+0x128/0x2c0 [20804.536040] ? smp_apic_timer_interrupt+0xa6/0x270 [20804.538683] ? apic_timer_interrupt+0xf/0x20 [20804.540651] [20804.541827] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20804.544326] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20804.547994] do_raw_spin_lock+0x177/0x190 [20804.549638] _raw_spin_lock+0x21/0x30 [20804.552223] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20804.554938] obd_mod_free+0x46/0x140 [ptlrpc] [20804.558036] mdc_commit_open+0x91/0xb0 [mdc] [20804.559874] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20804.561821] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20804.569913] after_reply+0xe2c/0x1780 [ptlrpc] [20804.572995] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20804.578124] ? __next_timer_interrupt+0x160/0x160 [20804.581338] ptlrpcd+0xa82/0xe40 [ptlrpc] [20804.584000] ? do_wait_intr+0xf0/0xf0 [20804.587295] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20804.592386] kthread+0x1d1/0x200 [20804.594064] ? set_kthread_struct+0x70/0x70 [20804.596076] ret_from_fork+0x1f/0x30 [20812.053754] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20812.058733] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20812.071811] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20812.075811] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20812.077919] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20812.079554] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20812.080925] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20812.086106] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20812.088729] RAX: 0000000000007085 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20812.091514] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [20812.093207] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20812.094823] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20812.096796] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20812.098599] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20812.101634] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20812.103693] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20812.105627] Call Trace: [20812.107169] [20812.107863] ? show_regs.cold.9+0x22/0x2f [20812.109141] ? watchdog_timer_fn.cold.6+0x62/0x10a [20812.110473] ? __hrtimer_run_queues+0x1c2/0x4d0 [20812.111895] ? watchdog+0x40/0x40 [20812.112754] ? hrtimer_interrupt+0x128/0x2c0 [20812.114217] ? smp_apic_timer_interrupt+0xa6/0x270 [20812.115606] ? apic_timer_interrupt+0xf/0x20 [20812.117199] [20812.117911] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20812.119634] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20812.121313] ? newidle_balance.isra.93+0x207/0x580 [20812.122784] do_raw_spin_lock+0x177/0x190 [20812.124131] _raw_spin_lock+0x21/0x30 [20812.127247] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20812.131230] process_one_work+0x2c8/0x700 [20812.132414] worker_thread+0x296/0x6e0 [20812.136554] ? rescuer_thread+0x570/0x570 [20812.137769] kthread+0x1d1/0x200 [20812.138858] ? set_kthread_struct+0x70/0x70 [20812.142381] ret_from_fork+0x1f/0x30 [20815.841394] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18178s! [20815.845171] Showing busy workqueues and worker pools: [20815.848390] workqueue events: flags=0x0 [20815.850507] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20815.855456] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20815.866187] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20815.872819] pending: cpu_latency_work [ptlrpc] [20815.876280] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20815.879875] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20815.887034] workqueue events_long: flags=0x0 [20815.890334] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20815.894763] pending: lu_objects_flush [obdclass] [20815.897408] workqueue events_unbound: flags=0x2 [20815.899397] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20815.901642] pending: flush_memcg_stats_dwork, idle_cull_fn [20815.903499] workqueue events_power_efficient: flags=0x80 [20815.907269] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20815.913781] pending: check_lifetime, do_cache_clean [20815.921366] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20815.928888] pending: neigh_periodic_work, neigh_periodic_work [20815.932478] workqueue mm_percpu_wq: flags=0x8 [20815.935416] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20815.943307] pending: vmstat_update, lru_add_drain_per_cpu [20815.945137] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20815.960393] pending: vmstat_update, lru_add_drain_per_cpu [20815.969607] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20815.981660] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20815.993758] workqueue xprtiod: flags=0xa [20815.995652] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20815.998457] pending: xprt_autoclose, xs_error_handle [20816.000723] workqueue ipv6_addrconf: flags=0x40008 [20816.005865] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20816.008398] pending: addrconf_verify_work [20816.010669] workqueue knbd0-recv: flags=0x1a [20816.012418] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20816.015196] in-flight: 651:recv_work [20816.017115] workqueue ptlrpc_pinger: flags=0x2 [20816.019109] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20816.021736] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20816.024332] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18178s workers=4 idle: 15979 31824 8 [20816.029318] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20806s workers=2 idle: 654 [20820.060121] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [20820.064872] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20820.102282] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20820.105879] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20820.108873] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20820.111277] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20820.117749] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20820.120358] RAX: 0000000000007c86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20820.123004] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20820.125532] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20820.127843] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20820.130566] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20820.133035] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20820.135825] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20820.137953] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20820.140364] Call Trace: [20820.141291] [20820.141843] ? show_regs.cold.9+0x22/0x2f [20820.143007] ? watchdog_timer_fn.cold.6+0x62/0x10a [20820.144515] ? __hrtimer_run_queues+0x1c2/0x4d0 [20820.146071] ? watchdog+0x40/0x40 [20820.147218] ? hrtimer_interrupt+0x128/0x2c0 [20820.148729] ? smp_apic_timer_interrupt+0xa6/0x270 [20820.150428] ? apic_timer_interrupt+0xf/0x20 [20820.151754] [20820.152543] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20820.154351] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20820.156289] do_raw_spin_lock+0x177/0x190 [20820.157752] _raw_spin_lock+0x21/0x30 [20820.159628] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20820.162606] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20820.166139] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20820.168436] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20820.172547] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20820.174854] ? woken_wake_function+0x30/0x30 [20820.176484] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20820.179454] kthread+0x1d1/0x200 [20820.180654] ? set_kthread_struct+0x70/0x70 [20820.182256] ret_from_fork+0x1f/0x30 [20832.060168] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20832.061859] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20832.065865] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20832.065921] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20832.065927] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20832.065929] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20832.065940] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20832.065943] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20832.065946] RAX: 0000000000005805 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20832.065948] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20832.065949] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20832.065951] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20832.065953] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20832.065955] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20832.065957] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20832.065959] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20832.065965] Call Trace: [20832.065969] [20832.065975] ? show_regs.cold.9+0x22/0x2f [20832.065982] ? watchdog_timer_fn.cold.6+0x62/0x10a [20832.065986] ? __hrtimer_run_queues+0x1c2/0x4d0 [20832.065989] ? watchdog+0x40/0x40 [20832.065992] ? hrtimer_interrupt+0x128/0x2c0 [20832.065995] ? smp_apic_timer_interrupt+0xa6/0x270 [20832.066001] ? apic_timer_interrupt+0xf/0x20 [20832.066004] [20832.066007] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20832.066014] do_raw_spin_lock+0x177/0x190 [20832.066019] _raw_spin_lock+0x21/0x30 [20832.066023] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20832.066457] ? do_raw_spin_unlock+0x75/0x190 [20832.066463] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20832.066748] ? do_wait_intr+0xf0/0xf0 [20832.066755] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20832.066978] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20832.067436] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20832.067646] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20832.067771] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20832.067811] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20832.067838] mdc_intent_lock+0x353/0x7e0 [mdc] [20832.067922] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20832.067997] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20832.068254] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20832.075297] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash [20832.089399] lmv_intent_open+0x338/0x1150 [lmv] [20832.095042] dm_log dm_mod sha512_ssse3 sha512_generic [20832.095061] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20832.098780] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20832.098999] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20832.099082] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20832.099121] ll_intent_lock+0xc5/0xaa0 [lustre] [20832.099283] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20832.099436] ll_lookup_it+0xe0e/0x2270 [lustre] [20832.099522] ? do_raw_spin_unlock+0x75/0x190 [20832.099536] ? _raw_spin_unlock+0x12/0x30 [20832.099543] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20832.099572] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20832.099592] ll_atomic_open+0x36a/0xd70 [lustre] [20832.099699] lookup_open+0x356/0xb40 [20832.099709] do_last+0x313/0xfc0 [20832.099714] ? nd_jump_root+0xe5/0x160 [20832.099719] ? path_init+0x437/0x520 [20832.099724] path_openat+0xf7/0x500 [20832.102607] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20832.110283] ? do_raw_spin_unlock+0x75/0x190 [20832.110362] do_filp_open+0x99/0x140 [20832.110370] ? getname_flags+0x6e/0x330 [20832.110375] ? __check_object_size+0xff/0x256 [20832.110383] ? do_raw_spin_unlock+0x75/0x190 [20832.110386] ? _raw_spin_unlock+0x12/0x30 [20832.110393] do_sys_openat2+0x2b4/0x410 [20832.110397] do_sys_open+0x73/0xa0 [20832.110401] __x64_sys_openat+0x24/0x30 [20832.110405] do_syscall_64+0xc1/0x440 [20832.110410] entry_SYSCALL_64_after_hwframe+0x49/0xae [20832.110420] RIP: 0033:0x7f53425a5332 [20832.110428] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20832.110431] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20832.110437] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20832.110439] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20832.110441] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20832.110443] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20832.110446] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20832.382660] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [20832.385007] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [20832.394094] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20832.399194] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20832.402789] RDX: 00000000000011b0 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20832.407800] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20832.411030] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20832.413630] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20832.415992] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20832.418560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20832.420249] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20832.422594] Call Trace: [20832.423250] [20832.424345] ? show_regs.cold.9+0x22/0x2f [20832.426180] ? watchdog_timer_fn.cold.6+0x62/0x10a [20832.428085] ? __hrtimer_run_queues+0x1c2/0x4d0 [20832.430165] ? watchdog+0x40/0x40 [20832.432247] ? hrtimer_interrupt+0x128/0x2c0 [20832.434443] ? smp_apic_timer_interrupt+0xa6/0x270 [20832.436808] ? apic_timer_interrupt+0xf/0x20 [20832.439261] [20832.440775] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20832.443391] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20832.451084] do_raw_spin_lock+0x177/0x190 [20832.456665] _raw_spin_lock+0x21/0x30 [20832.461847] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20832.465778] obd_mod_free+0x46/0x140 [ptlrpc] [20832.469839] mdc_commit_open+0x91/0xb0 [mdc] [20832.472445] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20832.475803] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20832.478601] after_reply+0xe2c/0x1780 [ptlrpc] [20832.481085] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20832.484379] ? __next_timer_interrupt+0x160/0x160 [20832.487496] ptlrpcd+0xa82/0xe40 [ptlrpc] [20832.490066] ? do_wait_intr+0xf0/0xf0 [20832.491947] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20832.494807] kthread+0x1d1/0x200 [20832.496578] ? set_kthread_struct+0x70/0x70 [20832.498494] ret_from_fork+0x1f/0x30 [20840.052107] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20840.055138] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20840.074856] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20840.080026] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20840.084098] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20840.087098] RIP: 0010:native_safe_halt+0x2e/0x50 [20840.089251] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20840.098040] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20840.101103] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20840.104189] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [20840.107282] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20840.110632] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20840.113459] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20840.117213] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20840.121424] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20840.124801] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20840.128338] Call Trace: [20840.129751] [20840.130536] ? show_regs.cold.9+0x22/0x2f [20840.132720] ? watchdog_timer_fn.cold.6+0x62/0x10a [20840.135000] ? __hrtimer_run_queues+0x1c2/0x4d0 [20840.137220] ? watchdog+0x40/0x40 [20840.138659] ? hrtimer_interrupt+0x128/0x2c0 [20840.141019] ? smp_apic_timer_interrupt+0xa6/0x270 [20840.143236] ? apic_timer_interrupt+0xf/0x20 [20840.145296] [20840.146278] ? native_safe_halt+0x2e/0x50 [20840.148611] kvm_wait+0xcd/0xe0 [20840.150246] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20840.152792] ? newidle_balance.isra.93+0x207/0x580 [20840.155265] do_raw_spin_lock+0x177/0x190 [20840.157280] _raw_spin_lock+0x21/0x30 [20840.159106] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20840.162220] process_one_work+0x2c8/0x700 [20840.164146] worker_thread+0x296/0x6e0 [20840.166070] ? rescuer_thread+0x570/0x570 [20840.168149] kthread+0x1d1/0x200 [20840.169581] ? set_kthread_struct+0x70/0x70 [20840.171958] ret_from_fork+0x1f/0x30 [20846.049081] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18208s! [20846.052031] Showing busy workqueues and worker pools: [20846.054248] workqueue events: flags=0x0 [20846.055976] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20846.058995] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20846.072390] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20846.078709] pending: cpu_latency_work [ptlrpc] [20846.082521] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20846.087816] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20846.096217] workqueue events_long: flags=0x0 [20846.100923] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20846.103135] pending: lu_objects_flush [obdclass] [20846.106923] workqueue events_unbound: flags=0x2 [20846.108673] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20846.111922] pending: flush_memcg_stats_dwork, idle_cull_fn [20846.114956] workqueue events_power_efficient: flags=0x80 [20846.116285] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20846.118416] pending: check_lifetime, do_cache_clean [20846.120565] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20846.123281] pending: neigh_periodic_work, neigh_periodic_work [20846.125708] workqueue mm_percpu_wq: flags=0x8 [20846.127607] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20846.130246] pending: vmstat_update, lru_add_drain_per_cpu [20846.132537] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20846.135337] pending: vmstat_update, lru_add_drain_per_cpu [20846.137625] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20846.140238] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20846.142974] workqueue xprtiod: flags=0xa [20846.144610] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20846.147004] pending: xprt_autoclose, xs_error_handle [20846.149249] workqueue ipv6_addrconf: flags=0x40008 [20846.151184] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20846.153413] pending: addrconf_verify_work [20846.155296] workqueue knbd0-recv: flags=0x1a [20846.157176] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20846.159901] in-flight: 651:recv_work [20846.161687] workqueue ptlrpc_pinger: flags=0x2 [20846.163780] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20846.166388] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20846.169637] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18209s workers=4 idle: 15979 31824 8 [20846.175818] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20836s workers=2 idle: 654 [20848.058692] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [20848.062770] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20848.087267] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20848.095334] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20848.101662] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20848.105853] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20848.114909] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20848.118881] RAX: 0000000000003f8a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20848.121504] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20848.124954] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20848.128245] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20848.131509] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20848.135081] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20848.138596] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20848.141245] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20848.145125] Call Trace: [20848.145939] [20848.146757] ? show_regs.cold.9+0x22/0x2f [20848.148696] ? watchdog_timer_fn.cold.6+0x62/0x10a [20848.151365] ? __hrtimer_run_queues+0x1c2/0x4d0 [20848.155016] ? watchdog+0x40/0x40 [20848.156209] ? hrtimer_interrupt+0x128/0x2c0 [20848.158692] ? smp_apic_timer_interrupt+0xa6/0x270 [20848.161060] ? apic_timer_interrupt+0xf/0x20 [20848.163530] [20848.164289] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20848.166316] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20848.168754] do_raw_spin_lock+0x177/0x190 [20848.170251] _raw_spin_lock+0x21/0x30 [20848.172077] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20848.174883] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20848.177075] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20848.179983] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20848.182378] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20848.184851] ? woken_wake_function+0x30/0x30 [20848.186342] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20848.189621] kthread+0x1d1/0x200 [20848.190938] ? set_kthread_struct+0x70/0x70 [20848.193039] ret_from_fork+0x1f/0x30 [20860.055200] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20860.058535] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20860.061795] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20860.062003] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20860.062011] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20860.062014] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20860.062029] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20860.062034] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20860.062040] RAX: 0000000000003d89 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20860.062043] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20860.062046] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20860.062048] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20860.062051] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20860.062055] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20860.062058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20860.062060] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20860.062068] Call Trace: [20860.062074] [20860.062080] ? show_regs.cold.9+0x22/0x2f [20860.062090] ? watchdog_timer_fn.cold.6+0x62/0x10a [20860.062097] ? __hrtimer_run_queues+0x1c2/0x4d0 [20860.062101] ? watchdog+0x40/0x40 [20860.062106] ? hrtimer_interrupt+0x128/0x2c0 [20860.062110] ? smp_apic_timer_interrupt+0xa6/0x270 [20860.062119] ? apic_timer_interrupt+0xf/0x20 [20860.062124] [20860.062128] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20860.062136] do_raw_spin_lock+0x177/0x190 [20860.062141] _raw_spin_lock+0x21/0x30 [20860.062146] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20860.062439] ? do_raw_spin_unlock+0x75/0x190 [20860.062445] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20860.062662] ? do_wait_intr+0xf0/0xf0 [20860.062671] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20860.062893] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20860.064228] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20860.064463] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20860.064586] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20860.064625] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20860.064657] mdc_intent_lock+0x353/0x7e0 [mdc] [20860.064686] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20860.064778] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20860.071277] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20860.071367] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20860.071375] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20860.071378] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [20860.071395] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [20860.071399] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20860.071404] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20860.071407] RDX: 00000000000000ac RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20860.071410] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20860.071412] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20860.071415] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20860.071418] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20860.071421] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20860.071424] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20860.071433] Call Trace: [20860.071439] [20860.071446] ? show_regs.cold.9+0x22/0x2f [20860.071457] ? watchdog_timer_fn.cold.6+0x62/0x10a [20860.071464] ? __hrtimer_run_queues+0x1c2/0x4d0 [20860.071469] ? watchdog+0x40/0x40 [20860.071474] ? hrtimer_interrupt+0x128/0x2c0 [20860.071478] ? smp_apic_timer_interrupt+0xa6/0x270 [20860.071493] ? apic_timer_interrupt+0xf/0x20 [20860.071499] [20860.071502] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20860.071507] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20860.071513] do_raw_spin_lock+0x177/0x190 [20860.071517] _raw_spin_lock+0x21/0x30 [20860.071523] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20860.071827] obd_mod_free+0x46/0x140 [ptlrpc] [20860.072166] mdc_commit_open+0x91/0xb0 [mdc] [20860.072266] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20860.072455] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20860.072695] after_reply+0xe2c/0x1780 [ptlrpc] [20860.115753] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20860.115808] lmv_intent_open+0x338/0x1150 [lmv] [20860.116278] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20860.121544] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20860.121832] ? __next_timer_interrupt+0x160/0x160 [20860.121844] ptlrpcd+0xa82/0xe40 [ptlrpc] [20860.122140] ? do_wait_intr+0xf0/0xf0 [20860.122150] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20860.123105] kthread+0x1d1/0x200 [20860.123111] ? set_kthread_struct+0x70/0x70 [20860.123117] ret_from_fork+0x1f/0x30 [20860.340101] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20860.343056] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20860.344965] ll_intent_lock+0xc5/0xaa0 [lustre] [20860.347065] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20860.349267] ll_lookup_it+0xe0e/0x2270 [lustre] [20860.351398] ? do_raw_spin_unlock+0x75/0x190 [20860.353254] ? _raw_spin_unlock+0x12/0x30 [20860.354871] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20860.357042] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20860.359252] ll_atomic_open+0x36a/0xd70 [lustre] [20860.361359] lookup_open+0x356/0xb40 [20860.362973] do_last+0x313/0xfc0 [20860.364345] ? nd_jump_root+0xe5/0x160 [20860.365897] ? path_init+0x437/0x520 [20860.367539] path_openat+0xf7/0x500 [20860.369001] ? do_raw_spin_unlock+0x75/0x190 [20860.370788] do_filp_open+0x99/0x140 [20860.372430] ? getname_flags+0x6e/0x330 [20860.374033] ? __check_object_size+0xff/0x256 [20860.376003] ? do_raw_spin_unlock+0x75/0x190 [20860.377660] ? _raw_spin_unlock+0x12/0x30 [20860.379425] do_sys_openat2+0x2b4/0x410 [20860.380954] do_sys_open+0x73/0xa0 [20860.382611] __x64_sys_openat+0x24/0x30 [20860.384307] do_syscall_64+0xc1/0x440 [20860.385784] entry_SYSCALL_64_after_hwframe+0x49/0xae [20860.387644] RIP: 0033:0x7f53425a5332 [20860.389752] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20860.398038] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20860.400768] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20860.403834] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20860.407027] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20860.409744] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20860.412897] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20872.062690] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20872.077785] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20872.119725] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20872.128313] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20872.135960] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20872.141310] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20872.144335] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20872.161885] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20872.168072] RAX: 0000000000006fb5 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20872.172672] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [20872.178679] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20872.185733] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20872.189754] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20872.198934] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20872.204838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20872.209902] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20872.216652] Call Trace: [20872.222560] [20872.223187] ? show_regs.cold.9+0x22/0x2f [20872.225650] ? watchdog_timer_fn.cold.6+0x62/0x10a [20872.228864] ? __hrtimer_run_queues+0x1c2/0x4d0 [20872.235763] ? watchdog+0x40/0x40 [20872.238850] ? hrtimer_interrupt+0x128/0x2c0 [20872.242241] ? smp_apic_timer_interrupt+0xa6/0x270 [20872.252463] ? apic_timer_interrupt+0xf/0x20 [20872.254813] [20872.256913] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20872.260653] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20872.263773] ? newidle_balance.isra.93+0x207/0x580 [20872.265452] do_raw_spin_lock+0x177/0x190 [20872.272582] _raw_spin_lock+0x21/0x30 [20872.273892] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20872.297913] process_one_work+0x2c8/0x700 [20872.300895] worker_thread+0x296/0x6e0 [20872.304802] ? rescuer_thread+0x570/0x570 [20872.311788] kthread+0x1d1/0x200 [20872.319618] ? set_kthread_struct+0x70/0x70 [20872.321017] ret_from_fork+0x1f/0x30 [20874.889094] rcu: INFO: rcu_sched self-detected stall on CPU [20874.891334] rcu: 2-....: (10787131 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [20874.894986] (t=18780463 jiffies g=274185 q=68642) [20874.896554] rcu: rcu_sched kthread starved for 18238829 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [20874.899638] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [20874.902506] rcu: RCU grace-period kthread stack dump: [20874.904362] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [20874.908094] Call Trace: [20874.909195] __schedule+0x351/0xcb0 [20874.910301] schedule+0xc0/0x180 [20874.911328] schedule_timeout+0xb4/0x190 [20874.912484] ? do_raw_spin_unlock+0x75/0x190 [20874.914471] ? __next_timer_interrupt+0x160/0x160 [20874.916575] ? prepare_to_swait_event+0xf3/0x140 [20874.918746] rcu_gp_kthread+0x8dd/0x15b0 [20874.920495] ? force_qs_rnp+0x2c0/0x2c0 [20874.922544] kthread+0x1d1/0x200 [20874.924250] ? set_kthread_struct+0x70/0x70 [20874.926528] ret_from_fork+0x1f/0x30 [20874.928365] rcu: Stack dump where RCU GP kthread last ran: [20874.930542] Sending NMI from CPU 2 to CPUs 0: [20874.932850] NMI backtrace for cpu 0 skipped: idling at native_safe_halt+0x2e/0x50 [20874.935423] Sending NMI from CPU 2 to CPUs 1: [20874.937532] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [20874.940287] NMI backtrace for cpu 2 [20874.941595] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20874.945593] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20874.949525] Call Trace: [20874.950447] [20874.950995] ? dump_stack+0xbb/0x10e [20874.951845] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [20874.954605] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [20874.957147] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [20874.959154] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [20874.961382] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [20874.963472] ? rcu_dump_cpu_stacks+0x11b/0x171 [20874.965535] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [20874.967539] ? update_process_times+0x6d/0xc0 [20874.969634] ? tick_sched_handle.isra.18+0x27/0x90 [20874.970909] ? tick_sched_timer+0x53/0xb0 [20874.972841] ? __hrtimer_run_queues+0x1c2/0x4d0 [20874.974342] ? tick_sched_handle.isra.18+0x90/0x90 [20874.976345] ? hrtimer_interrupt+0x128/0x2c0 [20874.978183] ? smp_apic_timer_interrupt+0xa6/0x270 [20874.979877] ? apic_timer_interrupt+0xf/0x20 [20874.981402] [20874.982198] ? native_safe_halt+0x2e/0x50 [20874.983422] ? kvm_wait+0xcd/0xe0 [20874.984668] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20874.987212] ? do_raw_spin_lock+0x177/0x190 [20874.988678] ? _raw_spin_lock+0x21/0x30 [20874.990142] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20874.992432] ? obd_mod_free+0x46/0x140 [ptlrpc] [20874.994551] ? mdc_commit_open+0x91/0xb0 [mdc] [20874.996502] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20874.998865] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20875.001361] ? after_reply+0xe2c/0x1780 [ptlrpc] [20875.003298] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20875.005457] ? __next_timer_interrupt+0x160/0x160 [20875.007370] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [20875.009333] ? do_wait_intr+0xf0/0xf0 [20875.010544] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20875.012541] ? kthread+0x1d1/0x200 [20875.013703] ? set_kthread_struct+0x70/0x70 [20875.014723] ? ret_from_fork+0x1f/0x30 [20876.063805] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [20876.071074] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20876.111398] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20876.119022] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20876.127815] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20876.138038] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20876.161433] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20876.165331] RAX: 0000000000007c1c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20876.176692] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20876.179189] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20876.182060] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20876.184782] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20876.188208] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20876.191233] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20876.201981] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20876.205051] Call Trace: [20876.206447] [20876.207921] ? show_regs.cold.9+0x22/0x2f [20876.210740] ? watchdog_timer_fn.cold.6+0x62/0x10a [20876.219226] ? __hrtimer_run_queues+0x1c2/0x4d0 [20876.221339] ? watchdog+0x40/0x40 [20876.224870] ? hrtimer_interrupt+0x128/0x2c0 [20876.227309] ? smp_apic_timer_interrupt+0xa6/0x270 [20876.230491] ? apic_timer_interrupt+0xf/0x20 [20876.236681] [20876.244010] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20876.245504] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20876.250197] do_raw_spin_lock+0x177/0x190 [20876.251591] _raw_spin_lock+0x21/0x30 [20876.256628] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18239s! [20876.258542] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20876.263126] Showing busy workqueues and worker pools: [20876.263135] workqueue events: flags=0x0 [20876.263141] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20876.263155] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20876.263696] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20876.263705] pending: cpu_latency_work [ptlrpc] [20876.263988] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20876.263997] pending: vmstat_shepherd, osc_grant_work_handler [osc] [20876.266848] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20876.267652] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20876.267863] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20876.271246] , sec_gc_main [ptlrpc] [20876.271668] workqueue events_long: flags=0x0 [20876.271674] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20876.271686] pending: lu_objects_flush [obdclass] [20876.272635] workqueue events_unbound: flags=0x2 [20876.272639] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20876.272648] pending: flush_memcg_stats_dwork, idle_cull_fn [20876.272663] workqueue events_power_efficient: flags=0x80 [20876.272666] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20876.272675] pending: check_lifetime, do_cache_clean [20876.272687] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20876.272694] pending: neigh_periodic_work, neigh_periodic_work [20876.272706] workqueue mm_percpu_wq: flags=0x8 [20876.272709] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20876.272715] pending: vmstat_update, lru_add_drain_per_cpu [20876.272727] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20876.272734] pending: vmstat_update, lru_add_drain_per_cpu [20876.272742] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20876.272748] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20876.273127] workqueue xprtiod: flags=0xa [20876.275221] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20876.278484] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20876.278503] pending: xprt_autoclose, xs_error_handle [20876.278524] workqueue ipv6_addrconf: flags=0x40008 [20876.278526] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20876.278534] pending: addrconf_verify_work [20876.278548] workqueue knbd0-recv: flags=0x1a [20876.278550] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20876.278557] in-flight: 651:recv_work [20876.278566] workqueue ptlrpc_pinger: flags=0x2 [20876.278568] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20876.278573] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20876.279172] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18239s workers=4 idle: 15979 31824 8 [20876.279187] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20866s workers=2 idle: 654 [20876.405996] ? woken_wake_function+0x30/0x30 [20876.408833] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20876.413205] kthread+0x1d1/0x200 [20876.414310] ? set_kthread_struct+0x70/0x70 [20876.416808] ret_from_fork+0x1f/0x30 [20900.057979] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20900.064765] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20900.072115] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20900.072760] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20900.072768] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20900.072771] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20900.072785] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20900.072790] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20900.072794] RAX: 0000000000006c88 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20900.072797] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20900.072799] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20900.072802] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20900.072804] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20900.072808] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20900.072811] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20900.072814] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20900.072823] Call Trace: [20900.072829] [20900.072835] ? show_regs.cold.9+0x22/0x2f [20900.072846] ? watchdog_timer_fn.cold.6+0x62/0x10a [20900.072852] ? __hrtimer_run_queues+0x1c2/0x4d0 [20900.072856] ? watchdog+0x40/0x40 [20900.072861] ? hrtimer_interrupt+0x128/0x2c0 [20900.072865] ? smp_apic_timer_interrupt+0xa6/0x270 [20900.072875] ? apic_timer_interrupt+0xf/0x20 [20900.072880] [20900.072883] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20900.073216] do_raw_spin_lock+0x177/0x190 [20900.073221] _raw_spin_lock+0x21/0x30 [20900.073227] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20900.073847] ? do_raw_spin_unlock+0x75/0x190 [20900.083502] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20900.083580] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20900.083588] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20900.083591] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [20900.083605] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [20900.083610] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20900.083614] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20900.083616] RDX: 0000000000007bca RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20900.083643] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20900.083645] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20900.083648] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20900.083651] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20900.083654] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20900.083656] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20900.083665] Call Trace: [20900.083681] [20900.083688] ? show_regs.cold.9+0x22/0x2f [20900.083699] ? watchdog_timer_fn.cold.6+0x62/0x10a [20900.083705] ? __hrtimer_run_queues+0x1c2/0x4d0 [20900.083709] ? watchdog+0x40/0x40 [20900.083713] ? hrtimer_interrupt+0x128/0x2c0 [20900.083717] ? smp_apic_timer_interrupt+0xa6/0x270 [20900.083725] ? apic_timer_interrupt+0xf/0x20 [20900.083730] [20900.083732] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20900.083737] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20900.083742] do_raw_spin_lock+0x177/0x190 [20900.083746] _raw_spin_lock+0x21/0x30 [20900.083758] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20900.084159] obd_mod_free+0x46/0x140 [ptlrpc] [20900.084352] mdc_commit_open+0x91/0xb0 [mdc] [20900.084484] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20900.084859] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20900.085174] after_reply+0xe2c/0x1780 [ptlrpc] [20900.085363] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20900.085566] ? __next_timer_interrupt+0x160/0x160 [20900.140673] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20900.140959] ? do_wait_intr+0xf0/0xf0 [20900.140970] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20900.141165] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20900.141377] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20900.142726] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20900.142851] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20900.153413] ptlrpcd+0xa82/0xe40 [ptlrpc] [20900.162668] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20900.162730] mdc_intent_lock+0x353/0x7e0 [mdc] [20900.162758] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20900.162956] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20900.166577] ? do_wait_intr+0xf0/0xf0 [20900.166600] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20900.166953] kthread+0x1d1/0x200 [20900.166960] ? set_kthread_struct+0x70/0x70 [20900.166964] ret_from_fork+0x1f/0x30 [20900.569858] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20900.577371] lmv_intent_open+0x338/0x1150 [lmv] [20900.580268] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20900.583745] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20900.592360] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20900.595774] ll_intent_lock+0xc5/0xaa0 [lustre] [20900.600219] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20900.604093] ll_lookup_it+0xe0e/0x2270 [lustre] [20900.607036] ? do_raw_spin_unlock+0x75/0x190 [20900.609532] ? _raw_spin_unlock+0x12/0x30 [20900.614465] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20900.618773] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20900.622404] ll_atomic_open+0x36a/0xd70 [lustre] [20900.626684] lookup_open+0x356/0xb40 [20900.632335] do_last+0x313/0xfc0 [20900.633678] ? nd_jump_root+0xe5/0x160 [20900.637834] ? path_init+0x437/0x520 [20900.643212] path_openat+0xf7/0x500 [20900.646025] ? do_raw_spin_unlock+0x75/0x190 [20900.650535] do_filp_open+0x99/0x140 [20900.651896] ? getname_flags+0x6e/0x330 [20900.654708] ? __check_object_size+0xff/0x256 [20900.659015] ? do_raw_spin_unlock+0x75/0x190 [20900.660860] ? _raw_spin_unlock+0x12/0x30 [20900.662142] do_sys_openat2+0x2b4/0x410 [20900.669940] do_sys_open+0x73/0xa0 [20900.673214] __x64_sys_openat+0x24/0x30 [20900.679061] do_syscall_64+0xc1/0x440 [20900.685086] entry_SYSCALL_64_after_hwframe+0x49/0xae [20900.689278] RIP: 0033:0x7f53425a5332 [20900.690558] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20900.707754] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20900.713239] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20900.717269] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20900.721602] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20900.725763] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20900.729303] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20904.052204] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20904.058037] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20904.058283] Modules linked in: [20904.062264] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20904.062499] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20904.062506] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20904.062509] RIP: 0010:native_safe_halt+0x2e/0x50 [20904.062554] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20904.062560] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20904.062565] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20904.062567] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20904.062570] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20904.062572] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20904.062575] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20904.062578] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20904.062581] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20904.062584] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20904.062592] Call Trace: [20904.062609] [20904.062616] ? show_regs.cold.9+0x22/0x2f [20904.062628] ? watchdog_timer_fn.cold.6+0x62/0x10a [20904.062636] ? __hrtimer_run_queues+0x1c2/0x4d0 [20904.062640] ? watchdog+0x40/0x40 [20904.062645] ? hrtimer_interrupt+0x128/0x2c0 [20904.062649] ? smp_apic_timer_interrupt+0xa6/0x270 [20904.062655] ? apic_timer_interrupt+0xf/0x20 [20904.062660] [20904.062662] ? native_safe_halt+0x2e/0x50 [20904.062810] kvm_wait+0xcd/0xe0 [20904.062820] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20904.062829] do_raw_spin_lock+0x177/0x190 [20904.062834] _raw_spin_lock+0x21/0x30 [20904.062839] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20904.063296] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20904.063582] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20904.064048] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20904.064537] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20904.067841] lustre(O) [20904.104726] ? woken_wake_function+0x30/0x30 [20904.104750] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20904.105593] kthread+0x1d1/0x200 [20904.105600] ? set_kthread_struct+0x70/0x70 [20904.105604] ret_from_fork+0x1f/0x30 [20904.263563] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20904.309253] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20904.316096] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20904.324863] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20904.328037] RIP: 0010:native_safe_halt+0x2e/0x50 [20904.330145] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20904.341118] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20904.346209] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20904.350096] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [20904.356478] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20904.360728] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20904.367045] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20904.369922] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20904.373374] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20904.375924] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20904.378846] Call Trace: [20904.379888] [20904.380604] ? show_regs.cold.9+0x22/0x2f [20904.383171] ? watchdog_timer_fn.cold.6+0x62/0x10a [20904.385228] ? __hrtimer_run_queues+0x1c2/0x4d0 [20904.387702] ? watchdog+0x40/0x40 [20904.390773] ? hrtimer_interrupt+0x128/0x2c0 [20904.392801] ? smp_apic_timer_interrupt+0xa6/0x270 [20904.396855] ? apic_timer_interrupt+0xf/0x20 [20904.400022] [20904.400958] ? native_safe_halt+0x2e/0x50 [20904.402694] kvm_wait+0xcd/0xe0 [20904.404134] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20904.407163] ? newidle_balance.isra.93+0x207/0x580 [20904.410580] do_raw_spin_lock+0x177/0x190 [20904.412863] _raw_spin_lock+0x21/0x30 [20904.414370] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20904.417751] process_one_work+0x2c8/0x700 [20904.420251] worker_thread+0x296/0x6e0 [20904.422828] ? rescuer_thread+0x570/0x570 [20904.424727] kthread+0x1d1/0x200 [20904.425946] ? set_kthread_struct+0x70/0x70 [20904.430012] ret_from_fork+0x1f/0x30 [20906.464239] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18269s! [20906.469893] Showing busy workqueues and worker pools: [20906.473847] workqueue events: flags=0x0 [20906.478047] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20906.483086] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20906.488185] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20906.490165] pending: cpu_latency_work [ptlrpc] [20906.492856] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20906.496257] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20906.500064] workqueue events_long: flags=0x0 [20906.502837] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20906.509871] pending: lu_objects_flush [obdclass] [20906.516923] workqueue events_unbound: flags=0x2 [20906.519675] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20906.523132] pending: flush_memcg_stats_dwork, idle_cull_fn [20906.525166] workqueue events_power_efficient: flags=0x80 [20906.530797] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20906.535341] pending: check_lifetime, do_cache_clean [20906.536992] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20906.544013] pending: neigh_periodic_work, neigh_periodic_work [20906.549188] workqueue mm_percpu_wq: flags=0x8 [20906.552247] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20906.559545] pending: vmstat_update, lru_add_drain_per_cpu [20906.562324] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20906.564362] pending: vmstat_update, lru_add_drain_per_cpu [20906.565928] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20906.568234] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20906.570706] workqueue xprtiod: flags=0xa [20906.573585] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20906.577973] pending: xprt_autoclose, xs_error_handle [20906.581400] workqueue ipv6_addrconf: flags=0x40008 [20906.585937] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20906.588999] pending: addrconf_verify_work [20906.590692] workqueue knbd0-recv: flags=0x1a [20906.592259] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20906.596180] in-flight: 651:recv_work [20906.597177] workqueue ptlrpc_pinger: flags=0x2 [20906.598209] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20906.602441] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20906.608084] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18269s workers=4 idle: 15979 31824 8 [20906.616395] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20897s workers=2 idle: 654 [20928.057031] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20928.057142] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20928.061062] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20928.061197] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20928.061207] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20928.061210] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [20928.061226] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [20928.061230] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20928.061235] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20928.061238] RDX: 0000000000001283 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20928.061240] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20928.061243] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20928.061246] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20928.061249] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20928.061252] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20928.061255] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20928.061263] Call Trace: [20928.061269] [20928.061276] ? show_regs.cold.9+0x22/0x2f [20928.061286] ? watchdog_timer_fn.cold.6+0x62/0x10a [20928.061293] ? __hrtimer_run_queues+0x1c2/0x4d0 [20928.061297] ? watchdog+0x40/0x40 [20928.061302] ? hrtimer_interrupt+0x128/0x2c0 [20928.061307] ? smp_apic_timer_interrupt+0xa6/0x270 [20928.061315] ? apic_timer_interrupt+0xf/0x20 [20928.061320] [20928.061322] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [20928.061326] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20928.061331] do_raw_spin_lock+0x177/0x190 [20928.061336] _raw_spin_lock+0x21/0x30 [20928.061342] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20928.061815] obd_mod_free+0x46/0x140 [ptlrpc] [20928.062295] mdc_commit_open+0x91/0xb0 [mdc] [20928.062336] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20928.062548] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20928.063247] after_reply+0xe2c/0x1780 [ptlrpc] [20928.063456] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20928.063954] ? __next_timer_interrupt+0x160/0x160 [20928.063966] ptlrpcd+0xa82/0xe40 [ptlrpc] [20928.064239] ? do_wait_intr+0xf0/0xf0 [20928.064249] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20928.064454] kthread+0x1d1/0x200 [20928.064461] ? set_kthread_struct+0x70/0x70 [20928.064466] ret_from_fork+0x1f/0x30 [20928.392726] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20928.434677] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20928.447987] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20928.460600] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20928.464018] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20928.485979] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20928.491518] RAX: 0000000000007c85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20928.502640] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20928.511598] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20928.515832] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20928.524779] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20928.530828] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20928.540828] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20928.550550] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20928.558199] Call Trace: [20928.560887] [20928.562442] ? show_regs.cold.9+0x22/0x2f [20928.566281] ? watchdog_timer_fn.cold.6+0x62/0x10a [20928.570424] ? __hrtimer_run_queues+0x1c2/0x4d0 [20928.573469] ? watchdog+0x40/0x40 [20928.577522] ? hrtimer_interrupt+0x128/0x2c0 [20928.579977] ? smp_apic_timer_interrupt+0xa6/0x270 [20928.581968] ? apic_timer_interrupt+0xf/0x20 [20928.583836] [20928.584773] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20928.587175] do_raw_spin_lock+0x177/0x190 [20928.588835] _raw_spin_lock+0x21/0x30 [20928.590255] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20928.594427] ? do_raw_spin_unlock+0x75/0x190 [20928.596268] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20928.598607] ? do_wait_intr+0xf0/0xf0 [20928.600407] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20928.602764] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20928.604935] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20928.610390] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20928.612479] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20928.615464] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20928.617586] mdc_intent_lock+0x353/0x7e0 [mdc] [20928.619326] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20928.620798] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20928.628643] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20928.630714] lmv_intent_open+0x338/0x1150 [lmv] [20928.633760] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20928.638236] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20928.646676] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20928.651920] ll_intent_lock+0xc5/0xaa0 [lustre] [20928.656429] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20928.658922] ll_lookup_it+0xe0e/0x2270 [lustre] [20928.665040] ? do_raw_spin_unlock+0x75/0x190 [20928.666880] ? _raw_spin_unlock+0x12/0x30 [20928.673841] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20928.684096] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20928.688830] ll_atomic_open+0x36a/0xd70 [lustre] [20928.692953] lookup_open+0x356/0xb40 [20928.696739] do_last+0x313/0xfc0 [20928.699658] ? nd_jump_root+0xe5/0x160 [20928.703011] ? path_init+0x437/0x520 [20928.704328] path_openat+0xf7/0x500 [20928.705291] ? do_raw_spin_unlock+0x75/0x190 [20928.708232] do_filp_open+0x99/0x140 [20928.713821] ? getname_flags+0x6e/0x330 [20928.715762] ? __check_object_size+0xff/0x256 [20928.719750] ? do_raw_spin_unlock+0x75/0x190 [20928.732402] ? _raw_spin_unlock+0x12/0x30 [20928.734900] do_sys_openat2+0x2b4/0x410 [20928.737619] do_sys_open+0x73/0xa0 [20928.743823] __x64_sys_openat+0x24/0x30 [20928.747852] do_syscall_64+0xc1/0x440 [20928.751901] entry_SYSCALL_64_after_hwframe+0x49/0xae [20928.758222] RIP: 0033:0x7f53425a5332 [20928.762699] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20928.777114] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20928.781842] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20928.787861] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20928.794710] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20928.800842] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20928.805724] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20932.052104] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20932.058025] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20932.059732] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20932.059807] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20932.059815] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20932.059822] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20932.060593] RIP: 0010:native_safe_halt+0x2e/0x50 [20932.060605] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20932.060608] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20932.060618] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20932.060621] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [20932.060623] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20932.060626] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20932.060628] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20932.060632] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20932.060635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20932.060637] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20932.060644] Call Trace: [20932.060650] [20932.060655] ? show_regs.cold.9+0x22/0x2f [20932.060664] ? watchdog_timer_fn.cold.6+0x62/0x10a [20932.060670] ? __hrtimer_run_queues+0x1c2/0x4d0 [20932.060674] ? watchdog+0x40/0x40 [20932.060678] ? hrtimer_interrupt+0x128/0x2c0 [20932.060681] ? smp_apic_timer_interrupt+0xa6/0x270 [20932.060688] ? apic_timer_interrupt+0xf/0x20 [20932.060692] [20932.060694] ? native_safe_halt+0x2e/0x50 [20932.060698] kvm_wait+0xcd/0xe0 [20932.060703] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20932.060709] ? newidle_balance.isra.93+0x207/0x580 [20932.060714] do_raw_spin_lock+0x177/0x190 [20932.060717] _raw_spin_lock+0x21/0x30 [20932.060721] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20932.060969] process_one_work+0x2c8/0x700 [20932.060976] worker_thread+0x296/0x6e0 [20932.060980] ? rescuer_thread+0x570/0x570 [20932.060990] kthread+0x1d1/0x200 [20932.061001] ? set_kthread_struct+0x70/0x70 [20932.061004] ret_from_fork+0x1f/0x30 [20932.213766] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20932.236101] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20932.241395] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20932.245618] RIP: 0010:native_safe_halt+0x2e/0x50 [20932.247772] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20932.256134] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20932.259889] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20932.263843] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20932.266761] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20932.270829] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20932.274641] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20932.279398] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20932.283626] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20932.286389] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20932.290143] Call Trace: [20932.291720] [20932.292631] ? show_regs.cold.9+0x22/0x2f [20932.294387] ? watchdog_timer_fn.cold.6+0x62/0x10a [20932.296588] ? __hrtimer_run_queues+0x1c2/0x4d0 [20932.298831] ? watchdog+0x40/0x40 [20932.300136] ? hrtimer_interrupt+0x128/0x2c0 [20932.301815] ? smp_apic_timer_interrupt+0xa6/0x270 [20932.303892] ? apic_timer_interrupt+0xf/0x20 [20932.306658] [20932.307849] ? native_safe_halt+0x2e/0x50 [20932.310145] kvm_wait+0xcd/0xe0 [20932.311865] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20932.314768] do_raw_spin_lock+0x177/0x190 [20932.318264] _raw_spin_lock+0x21/0x30 [20932.319685] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20932.324184] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20932.327556] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20932.330885] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20932.336465] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20932.344133] ? woken_wake_function+0x30/0x30 [20932.346335] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20932.354736] kthread+0x1d1/0x200 [20932.355888] ? set_kthread_struct+0x70/0x70 [20932.361460] ret_from_fork+0x1f/0x30 [20936.677912] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18299s! [20936.683334] Showing busy workqueues and worker pools: [20936.685127] workqueue events: flags=0x0 [20936.687030] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20936.694516] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20936.701356] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20936.709299] pending: cpu_latency_work [ptlrpc] [20936.714017] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20936.718813] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20936.726761] workqueue events_long: flags=0x0 [20936.731724] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20936.734912] pending: lu_objects_flush [obdclass] [20936.737919] workqueue events_unbound: flags=0x2 [20936.739840] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20936.745800] pending: flush_memcg_stats_dwork, idle_cull_fn [20936.748585] workqueue events_power_efficient: flags=0x80 [20936.752040] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20936.758228] pending: check_lifetime, do_cache_clean [20936.760089] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20936.774677] pending: neigh_periodic_work, neigh_periodic_work [20936.785460] workqueue mm_percpu_wq: flags=0x8 [20936.786874] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20936.792726] pending: vmstat_update, lru_add_drain_per_cpu [20936.801796] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20936.808308] pending: vmstat_update, lru_add_drain_per_cpu [20936.812753] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20936.817099] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20936.822702] workqueue xprtiod: flags=0xa [20936.823740] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20936.830051] pending: xprt_autoclose, xs_error_handle [20936.839475] workqueue ipv6_addrconf: flags=0x40008 [20936.848748] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20936.854984] pending: addrconf_verify_work [20936.856538] workqueue knbd0-recv: flags=0x1a [20936.858446] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20936.861276] in-flight: 651:recv_work [20936.862989] workqueue ptlrpc_pinger: flags=0x2 [20936.864596] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20936.867026] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20936.869407] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18299s workers=4 idle: 15979 31824 8 [20936.872616] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20927s workers=2 idle: 654 [20956.055524] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20956.057019] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20956.061979] Modules linked in: [20956.064423] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20956.064503] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20956.064511] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20956.064532] RIP: 0010:native_safe_halt+0x2e/0x50 [20956.064545] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20956.064550] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [20956.064555] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20956.064557] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20956.064559] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20956.064562] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20956.064565] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20956.064568] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20956.064571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20956.064573] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20956.064581] Call Trace: [20956.064587] [20956.064593] ? show_regs.cold.9+0x22/0x2f [20956.064604] ? watchdog_timer_fn.cold.6+0x62/0x10a [20956.064611] ? __hrtimer_run_queues+0x1c2/0x4d0 [20956.069368] lustre(O) [20956.084948] ? watchdog+0x40/0x40 [20956.108583] osp(O) [20956.111799] ? hrtimer_interrupt+0x128/0x2c0 [20956.111817] ? smp_apic_timer_interrupt+0xa6/0x270 [20956.111826] ? apic_timer_interrupt+0xf/0x20 [20956.111831] [20956.111837] ? native_safe_halt+0x2e/0x50 [20956.118914] ofd(O) [20956.126721] kvm_wait+0xcd/0xe0 [20956.126744] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20956.126754] do_raw_spin_lock+0x177/0x190 [20956.126759] _raw_spin_lock+0x21/0x30 [20956.126768] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20956.136692] lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20956.136762] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20956.136769] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20956.136772] RIP: 0010:native_safe_halt+0x2e/0x50 [20956.136786] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20956.136789] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20956.136794] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20956.136797] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20956.136799] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20956.136801] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20956.136803] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20956.136806] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20956.136809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20956.136811] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20956.136819] Call Trace: [20956.136826] [20956.136832] ? show_regs.cold.9+0x22/0x2f [20956.136842] ? watchdog_timer_fn.cold.6+0x62/0x10a [20956.136848] ? __hrtimer_run_queues+0x1c2/0x4d0 [20956.136852] ? watchdog+0x40/0x40 [20956.136856] ? hrtimer_interrupt+0x128/0x2c0 [20956.136860] ? smp_apic_timer_interrupt+0xa6/0x270 [20956.136867] ? apic_timer_interrupt+0xf/0x20 [20956.136871] [20956.136874] ? native_safe_halt+0x2e/0x50 [20956.136878] kvm_wait+0xcd/0xe0 [20956.136886] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20956.136895] do_raw_spin_lock+0x177/0x190 [20956.136900] _raw_spin_lock+0x21/0x30 [20956.136904] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20956.141117] obd_mod_free+0x46/0x140 [ptlrpc] [20956.164604] ? do_raw_spin_unlock+0x75/0x190 [20956.167046] mdc_commit_open+0x91/0xb0 [mdc] [20956.167093] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20956.190241] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20956.194902] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20956.195167] after_reply+0xe2c/0x1780 [ptlrpc] [20956.195408] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20956.195629] ? __next_timer_interrupt+0x160/0x160 [20956.195640] ptlrpcd+0xa82/0xe40 [ptlrpc] [20956.208239] ? do_wait_intr+0xf0/0xf0 [20956.210685] ? do_wait_intr+0xf0/0xf0 [20956.210703] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20956.210905] kthread+0x1d1/0x200 [20956.210909] ? set_kthread_struct+0x70/0x70 [20956.210912] ret_from_fork+0x1f/0x30 [20956.716896] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20956.721420] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20956.727965] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20956.735349] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20956.739108] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20956.745244] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20956.750955] mdc_intent_lock+0x353/0x7e0 [mdc] [20956.753816] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20956.762982] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20956.769673] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20956.778242] lmv_intent_open+0x338/0x1150 [lmv] [20956.785027] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20956.789285] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20956.796864] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20956.799953] ll_intent_lock+0xc5/0xaa0 [lustre] [20956.804112] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20956.808003] ll_lookup_it+0xe0e/0x2270 [lustre] [20956.812177] ? do_raw_spin_unlock+0x75/0x190 [20956.815995] ? _raw_spin_unlock+0x12/0x30 [20956.818828] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20956.822113] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20956.826928] ll_atomic_open+0x36a/0xd70 [lustre] [20956.829005] lookup_open+0x356/0xb40 [20956.831868] do_last+0x313/0xfc0 [20956.833789] ? nd_jump_root+0xe5/0x160 [20956.837060] ? path_init+0x437/0x520 [20956.838861] path_openat+0xf7/0x500 [20956.840895] ? do_raw_spin_unlock+0x75/0x190 [20956.845002] do_filp_open+0x99/0x140 [20956.848060] ? getname_flags+0x6e/0x330 [20956.853658] ? __check_object_size+0xff/0x256 [20956.857912] ? do_raw_spin_unlock+0x75/0x190 [20956.861957] ? _raw_spin_unlock+0x12/0x30 [20956.866023] do_sys_openat2+0x2b4/0x410 [20956.868897] do_sys_open+0x73/0xa0 [20956.875884] __x64_sys_openat+0x24/0x30 [20956.879181] do_syscall_64+0xc1/0x440 [20956.880492] entry_SYSCALL_64_after_hwframe+0x49/0xae [20956.889555] RIP: 0033:0x7f53425a5332 [20956.891412] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20956.904900] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20956.910679] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20956.917793] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20956.922843] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20956.934643] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20956.943007] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20960.059876] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20960.069323] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20960.106987] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20960.113570] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20960.118764] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20960.128082] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20960.145975] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20960.151071] RAX: 0000000000007606 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20960.157077] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20960.163375] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20960.166837] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20960.171106] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20960.179839] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20960.191991] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20960.201067] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20960.203899] Call Trace: [20960.206025] [20960.207412] ? show_regs.cold.9+0x22/0x2f [20960.214757] ? watchdog_timer_fn.cold.6+0x62/0x10a [20960.217779] ? __hrtimer_run_queues+0x1c2/0x4d0 [20960.225109] ? watchdog+0x40/0x40 [20960.227744] ? hrtimer_interrupt+0x128/0x2c0 [20960.230100] ? smp_apic_timer_interrupt+0xa6/0x270 [20960.235238] ? apic_timer_interrupt+0xf/0x20 [20960.238107] [20960.238988] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20960.242365] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20960.244207] do_raw_spin_lock+0x177/0x190 [20960.250029] _raw_spin_lock+0x21/0x30 [20960.255041] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20960.260937] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20960.265073] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20960.272694] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20960.278433] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20960.282889] ? woken_wake_function+0x30/0x30 [20960.297749] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20960.303660] kthread+0x1d1/0x200 [20960.306533] ? set_kthread_struct+0x70/0x70 [20960.311409] ret_from_fork+0x1f/0x30 [20964.052099] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [20964.055301] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20964.089130] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20964.097299] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20964.106900] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20964.112315] RIP: 0010:native_safe_halt+0x2e/0x50 [20964.119996] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20964.137010] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20964.145301] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20964.154614] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [20964.163059] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20964.167992] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20964.171955] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20964.181339] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20964.190275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20964.195351] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20964.202113] Call Trace: [20964.203250] [20964.207627] ? show_regs.cold.9+0x22/0x2f [20964.211528] ? watchdog_timer_fn.cold.6+0x62/0x10a [20964.214173] ? __hrtimer_run_queues+0x1c2/0x4d0 [20964.216671] ? watchdog+0x40/0x40 [20964.219127] ? hrtimer_interrupt+0x128/0x2c0 [20964.223436] ? smp_apic_timer_interrupt+0xa6/0x270 [20964.224852] ? apic_timer_interrupt+0xf/0x20 [20964.227320] [20964.227919] ? native_safe_halt+0x2e/0x50 [20964.231467] kvm_wait+0xcd/0xe0 [20964.232911] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20964.235027] ? newidle_balance.isra.93+0x207/0x580 [20964.240266] do_raw_spin_lock+0x177/0x190 [20964.243582] _raw_spin_lock+0x21/0x30 [20964.250870] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20964.252992] process_one_work+0x2c8/0x700 [20964.257969] worker_thread+0x296/0x6e0 [20964.260051] ? rescuer_thread+0x570/0x570 [20964.262660] kthread+0x1d1/0x200 [20964.266623] ? set_kthread_struct+0x70/0x70 [20964.270451] ret_from_fork+0x1f/0x30 [20966.879322] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18329s! [20966.886608] Showing busy workqueues and worker pools: [20966.889177] workqueue events: flags=0x0 [20966.892725] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20966.895888] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20966.903701] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20966.907223] pending: cpu_latency_work [ptlrpc] [20966.910350] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20966.914006] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20966.920830] workqueue events_long: flags=0x0 [20966.922543] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20966.927989] pending: lu_objects_flush [obdclass] [20966.931176] workqueue events_unbound: flags=0x2 [20966.933134] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20966.935837] pending: flush_memcg_stats_dwork, idle_cull_fn [20966.942542] workqueue events_power_efficient: flags=0x80 [20966.945653] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20966.950183] pending: check_lifetime, do_cache_clean [20966.952532] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20966.956225] pending: neigh_periodic_work, neigh_periodic_work [20966.959010] workqueue mm_percpu_wq: flags=0x8 [20966.961224] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20966.964750] pending: vmstat_update, lru_add_drain_per_cpu [20966.967602] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20966.970822] pending: vmstat_update, lru_add_drain_per_cpu [20966.973619] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20966.976628] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20966.979637] workqueue xprtiod: flags=0xa [20966.981430] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20966.984740] pending: xprt_autoclose, xs_error_handle [20966.987503] workqueue ipv6_addrconf: flags=0x40008 [20966.989649] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20966.992400] pending: addrconf_verify_work [20966.994195] workqueue knbd0-recv: flags=0x1a [20966.995915] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20966.998587] in-flight: 651:recv_work [20967.000772] workqueue ptlrpc_pinger: flags=0x2 [20967.003241] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20967.006219] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20967.009423] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18329s workers=4 idle: 15979 31824 8 [20967.013259] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20957s workers=2 idle: 654 [20984.055093] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [20984.057036] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [20984.066629] Modules linked in: [20984.069760] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20984.069844] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20984.069851] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20984.069854] RIP: 0010:native_safe_halt+0x2e/0x50 [20984.069869] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20984.069873] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20984.069878] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [20984.069880] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [20984.069882] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [20984.069884] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [20984.069887] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [20984.069890] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [20984.069892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20984.069894] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [20984.069903] Call Trace: [20984.069908] [20984.069913] ? show_regs.cold.9+0x22/0x2f [20984.069923] ? watchdog_timer_fn.cold.6+0x62/0x10a [20984.069929] ? __hrtimer_run_queues+0x1c2/0x4d0 [20984.069933] ? watchdog+0x40/0x40 [20984.069937] ? hrtimer_interrupt+0x128/0x2c0 [20984.069941] ? smp_apic_timer_interrupt+0xa6/0x270 [20984.069949] ? apic_timer_interrupt+0xf/0x20 [20984.069954] [20984.069957] ? native_safe_halt+0x2e/0x50 [20984.069960] kvm_wait+0xcd/0xe0 [20984.069968] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [20984.069974] do_raw_spin_lock+0x177/0x190 [20984.069978] _raw_spin_lock+0x21/0x30 [20984.069983] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [20984.070262] obd_mod_free+0x46/0x140 [ptlrpc] [20984.070454] mdc_commit_open+0x91/0xb0 [mdc] [20984.070492] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [20984.070689] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [20984.076455] lustre(O) [20984.112798] after_reply+0xe2c/0x1780 [ptlrpc] [20984.117151] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [20984.117880] ? __next_timer_interrupt+0x160/0x160 [20984.117891] ptlrpcd+0xa82/0xe40 [ptlrpc] [20984.118117] ? do_wait_intr+0xf0/0xf0 [20984.118127] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [20984.137839] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20984.137912] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20984.137917] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20984.137920] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20984.137935] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20984.137939] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20984.137944] RAX: 0000000000004df5 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20984.137947] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [20984.137949] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [20984.137950] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [20984.137953] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [20984.137956] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [20984.137958] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20984.137961] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [20984.137970] Call Trace: [20984.137976] [20984.137981] ? show_regs.cold.9+0x22/0x2f [20984.137991] ? watchdog_timer_fn.cold.6+0x62/0x10a [20984.137997] ? __hrtimer_run_queues+0x1c2/0x4d0 [20984.138002] ? watchdog+0x40/0x40 [20984.138006] ? hrtimer_interrupt+0x128/0x2c0 [20984.138010] ? smp_apic_timer_interrupt+0xa6/0x270 [20984.142139] kthread+0x1d1/0x200 [20984.142158] ? set_kthread_struct+0x70/0x70 [20984.142162] ret_from_fork+0x1f/0x30 [20984.513477] ? apic_timer_interrupt+0xf/0x20 [20984.518089] [20984.519932] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20984.525470] do_raw_spin_lock+0x177/0x190 [20984.532670] _raw_spin_lock+0x21/0x30 [20984.537831] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [20984.547781] ? do_raw_spin_unlock+0x75/0x190 [20984.552902] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [20984.559489] ? do_wait_intr+0xf0/0xf0 [20984.563881] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [20984.571257] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [20984.572786] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20984.587290] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20984.596813] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20984.610433] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [20984.618273] mdc_intent_lock+0x353/0x7e0 [mdc] [20984.623533] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20984.626795] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [20984.633854] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [20984.638139] lmv_intent_open+0x338/0x1150 [lmv] [20984.641445] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20984.650299] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20984.653316] lmv_intent_lock+0x1dd/0x4c0 [lmv] [20984.656747] ll_intent_lock+0xc5/0xaa0 [lustre] [20984.664505] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [20984.676910] ll_lookup_it+0xe0e/0x2270 [lustre] [20984.683464] ? do_raw_spin_unlock+0x75/0x190 [20984.696458] ? _raw_spin_unlock+0x12/0x30 [20984.706156] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [20984.711530] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [20984.713765] ll_atomic_open+0x36a/0xd70 [lustre] [20984.724900] lookup_open+0x356/0xb40 [20984.727776] do_last+0x313/0xfc0 [20984.729602] ? nd_jump_root+0xe5/0x160 [20984.731370] ? path_init+0x437/0x520 [20984.735995] path_openat+0xf7/0x500 [20984.741481] ? do_raw_spin_unlock+0x75/0x190 [20984.743791] do_filp_open+0x99/0x140 [20984.748818] ? getname_flags+0x6e/0x330 [20984.752517] ? __check_object_size+0xff/0x256 [20984.755008] ? do_raw_spin_unlock+0x75/0x190 [20984.758645] ? _raw_spin_unlock+0x12/0x30 [20984.760779] do_sys_openat2+0x2b4/0x410 [20984.765720] do_sys_open+0x73/0xa0 [20984.767852] __x64_sys_openat+0x24/0x30 [20984.769984] do_syscall_64+0xc1/0x440 [20984.772107] entry_SYSCALL_64_after_hwframe+0x49/0xae [20984.780960] RIP: 0033:0x7f53425a5332 [20984.784650] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [20984.802456] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [20984.812570] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [20984.821171] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [20984.834143] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [20984.854461] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [20984.864127] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [20988.058013] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [20988.061820] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20988.083916] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20988.088052] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20988.094609] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20988.101770] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [20988.117420] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [20988.124913] RAX: 0000000000002e86 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20988.129568] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [20988.136273] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [20988.139607] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [20988.144514] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [20988.149232] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [20988.156105] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20988.160141] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [20988.165933] Call Trace: [20988.167923] [20988.170100] ? show_regs.cold.9+0x22/0x2f [20988.172061] ? watchdog_timer_fn.cold.6+0x62/0x10a [20988.177721] ? __hrtimer_run_queues+0x1c2/0x4d0 [20988.179255] ? watchdog+0x40/0x40 [20988.181519] ? hrtimer_interrupt+0x128/0x2c0 [20988.185445] ? smp_apic_timer_interrupt+0xa6/0x270 [20988.188590] ? apic_timer_interrupt+0xf/0x20 [20988.192469] [20988.193687] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [20988.197601] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20988.201273] do_raw_spin_lock+0x177/0x190 [20988.202629] _raw_spin_lock+0x21/0x30 [20988.204787] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [20988.208058] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [20988.210523] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [20988.214015] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [20988.216862] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [20988.218876] ? woken_wake_function+0x30/0x30 [20988.220481] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [20988.222685] kthread+0x1d1/0x200 [20988.225869] ? set_kthread_struct+0x70/0x70 [20988.227574] ret_from_fork+0x1f/0x30 [20992.052080] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [20992.056682] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [20992.085158] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [20992.092839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [20992.097011] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [20992.102832] RIP: 0010:native_safe_halt+0x2e/0x50 [20992.109198] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [20992.120218] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [20992.126125] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [20992.130843] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [20992.136804] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [20992.142692] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [20992.146126] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [20992.150107] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [20992.155131] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [20992.158226] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [20992.162455] Call Trace: [20992.164139] [20992.165396] ? show_regs.cold.9+0x22/0x2f [20992.168325] ? watchdog_timer_fn.cold.6+0x62/0x10a [20992.171200] ? __hrtimer_run_queues+0x1c2/0x4d0 [20992.173658] ? watchdog+0x40/0x40 [20992.175174] ? hrtimer_interrupt+0x128/0x2c0 [20992.177192] ? smp_apic_timer_interrupt+0xa6/0x270 [20992.179141] ? apic_timer_interrupt+0xf/0x20 [20992.180716] [20992.181805] ? native_safe_halt+0x2e/0x50 [20992.184171] kvm_wait+0xcd/0xe0 [20992.185827] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [20992.188998] ? newidle_balance.isra.93+0x207/0x580 [20992.191813] do_raw_spin_lock+0x177/0x190 [20992.194245] _raw_spin_lock+0x21/0x30 [20992.196259] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [20992.200079] process_one_work+0x2c8/0x700 [20992.202138] worker_thread+0x296/0x6e0 [20992.205197] ? rescuer_thread+0x570/0x570 [20992.207330] kthread+0x1d1/0x200 [20992.209462] ? set_kthread_struct+0x70/0x70 [20992.212056] ret_from_fork+0x1f/0x30 [20997.087706] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18359s! [20997.098925] Showing busy workqueues and worker pools: [20997.101677] workqueue events: flags=0x0 [20997.105735] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [20997.113534] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [20997.128946] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20997.137296] pending: cpu_latency_work [ptlrpc] [20997.147187] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [20997.156455] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [20997.163270] workqueue events_long: flags=0x0 [20997.166840] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [20997.177695] pending: lu_objects_flush [obdclass] [20997.179759] workqueue events_unbound: flags=0x2 [20997.181658] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [20997.186221] pending: flush_memcg_stats_dwork, idle_cull_fn [20997.192727] workqueue events_power_efficient: flags=0x80 [20997.195392] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20997.198764] pending: check_lifetime, do_cache_clean [20997.201553] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20997.208630] pending: neigh_periodic_work, neigh_periodic_work [20997.218049] workqueue mm_percpu_wq: flags=0x8 [20997.224686] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20997.228188] pending: vmstat_update, lru_add_drain_per_cpu [20997.231313] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [20997.237358] pending: vmstat_update, lru_add_drain_per_cpu [20997.240167] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [20997.243358] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [20997.246446] workqueue xprtiod: flags=0xa [20997.248151] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [20997.250615] pending: xprt_autoclose, xs_error_handle [20997.253090] workqueue ipv6_addrconf: flags=0x40008 [20997.258987] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [20997.261159] pending: addrconf_verify_work [20997.270271] workqueue knbd0-recv: flags=0x1a [20997.273295] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [20997.282892] in-flight: 651:recv_work [20997.285362] workqueue ptlrpc_pinger: flags=0x2 [20997.288172] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [20997.291454] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [20997.299755] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18360s workers=4 idle: 15979 31824 8 [20997.308237] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=20987s workers=2 idle: 654 [21012.055195] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [21012.059121] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [21012.059189] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21012.059269] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21012.059274] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21012.059276] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21012.059288] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21012.059291] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21012.059295] RAX: 0000000000005c1b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21012.059296] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21012.059298] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21012.059299] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21012.059301] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21012.059304] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21012.059306] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21012.059307] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21012.059313] Call Trace: [21012.059317] [21012.059322] ? show_regs.cold.9+0x22/0x2f [21012.059328] ? watchdog_timer_fn.cold.6+0x62/0x10a [21012.059332] ? __hrtimer_run_queues+0x1c2/0x4d0 [21012.059335] ? watchdog+0x40/0x40 [21012.059338] ? hrtimer_interrupt+0x128/0x2c0 [21012.059340] ? smp_apic_timer_interrupt+0xa6/0x270 [21012.059346] ? apic_timer_interrupt+0xf/0x20 [21012.059350] [21012.059351] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21012.059356] do_raw_spin_lock+0x177/0x190 [21012.059360] _raw_spin_lock+0x21/0x30 [21012.059363] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21012.059756] ? do_raw_spin_unlock+0x75/0x190 [21012.059762] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21012.059968] ? do_wait_intr+0xf0/0xf0 [21012.059977] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21012.060210] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21012.060504] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21012.060825] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21012.060934] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21012.060974] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21012.061005] mdc_intent_lock+0x353/0x7e0 [mdc] [21012.061034] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21012.061120] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21012.061413] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21012.061443] lmv_intent_open+0x338/0x1150 [lmv] [21012.061476] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21012.061658] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21012.061764] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21012.061793] ll_intent_lock+0xc5/0xaa0 [lustre] [21012.061881] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21012.061965] ll_lookup_it+0xe0e/0x2270 [lustre] [21012.062053] ? do_raw_spin_unlock+0x75/0x190 [21012.062059] ? _raw_spin_unlock+0x12/0x30 [21012.062066] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21012.062096] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21012.062118] ll_atomic_open+0x36a/0xd70 [lustre] [21012.062207] lookup_open+0x356/0xb40 [21012.062217] do_last+0x313/0xfc0 [21012.062223] ? nd_jump_root+0xe5/0x160 [21012.062227] ? path_init+0x437/0x520 [21012.062232] path_openat+0xf7/0x500 [21012.062237] ? do_raw_spin_unlock+0x75/0x190 [21012.062242] do_filp_open+0x99/0x140 [21012.062248] ? getname_flags+0x6e/0x330 [21012.062253] ? __check_object_size+0xff/0x256 [21012.065504] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21012.065587] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21012.065594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21012.065597] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21012.065611] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21012.065621] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21012.065625] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21012.065628] RDX: 0000000000007f8a RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21012.107985] ? do_raw_spin_unlock+0x75/0x190 [21012.121357] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21012.121369] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21012.121372] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21012.121375] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21012.121379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21012.121382] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21012.121391] Call Trace: [21012.121398] [21012.121404] ? show_regs.cold.9+0x22/0x2f [21012.121417] ? watchdog_timer_fn.cold.6+0x62/0x10a [21012.121424] ? __hrtimer_run_queues+0x1c2/0x4d0 [21012.121429] ? watchdog+0x40/0x40 [21012.121434] ? hrtimer_interrupt+0x128/0x2c0 [21012.121438] ? smp_apic_timer_interrupt+0xa6/0x270 [21012.121446] ? apic_timer_interrupt+0xf/0x20 [21012.121451] [21012.121454] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21012.121461] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21012.121466] do_raw_spin_lock+0x177/0x190 [21012.121471] _raw_spin_lock+0x21/0x30 [21012.121478] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21012.122041] obd_mod_free+0x46/0x140 [ptlrpc] [21012.122386] mdc_commit_open+0x91/0xb0 [mdc] [21012.122424] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21012.122713] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21012.123141] after_reply+0xe2c/0x1780 [ptlrpc] [21012.123430] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21012.131922] ? _raw_spin_unlock+0x12/0x30 [21012.135202] ? __next_timer_interrupt+0x160/0x160 [21012.135291] ptlrpcd+0xa82/0xe40 [ptlrpc] [21012.153780] do_sys_openat2+0x2b4/0x410 [21012.158999] ? do_wait_intr+0xf0/0xf0 [21012.159024] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21012.159471] kthread+0x1d1/0x200 [21012.159479] ? set_kthread_struct+0x70/0x70 [21012.159483] ret_from_fork+0x1f/0x30 [21012.645636] do_sys_open+0x73/0xa0 [21012.647889] __x64_sys_openat+0x24/0x30 [21012.649741] do_syscall_64+0xc1/0x440 [21012.651584] entry_SYSCALL_64_after_hwframe+0x49/0xae [21012.654437] RIP: 0033:0x7f53425a5332 [21012.656717] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21012.667884] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21012.671880] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21012.678412] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21012.686385] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21012.692858] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21012.697192] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21016.058011] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [21016.061796] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21016.093766] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21016.098919] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21016.107104] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21016.115698] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21016.128441] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [21016.131081] RAX: 0000000000000a0f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21016.134136] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21016.140761] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21016.142571] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21016.152808] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21016.155330] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21016.159132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21016.161147] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21016.165799] Call Trace: [21016.166555] [21016.167046] ? show_regs.cold.9+0x22/0x2f [21016.171333] ? watchdog_timer_fn.cold.6+0x62/0x10a [21016.176742] ? __hrtimer_run_queues+0x1c2/0x4d0 [21016.178774] ? watchdog+0x40/0x40 [21016.180425] ? hrtimer_interrupt+0x128/0x2c0 [21016.184099] ? smp_apic_timer_interrupt+0xa6/0x270 [21016.186435] ? apic_timer_interrupt+0xf/0x20 [21016.188366] [21016.189332] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21016.191704] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21016.193990] do_raw_spin_lock+0x177/0x190 [21016.195794] _raw_spin_lock+0x21/0x30 [21016.197407] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21016.201653] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21016.204699] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21016.207605] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21016.211009] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21016.215708] ? woken_wake_function+0x30/0x30 [21016.216975] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21016.222504] kthread+0x1d1/0x200 [21016.223656] ? set_kthread_struct+0x70/0x70 [21016.226132] ret_from_fork+0x1f/0x30 [21024.052162] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [21024.055994] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21024.083048] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21024.088492] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21024.092974] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21024.096734] RIP: 0010:native_safe_halt+0x2e/0x50 [21024.099446] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21024.109309] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21024.113176] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21024.117459] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [21024.121270] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21024.125203] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21024.128713] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21024.132772] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21024.136963] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21024.139849] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21024.143349] Call Trace: [21024.144275] [21024.144975] ? show_regs.cold.9+0x22/0x2f [21024.146884] ? watchdog_timer_fn.cold.6+0x62/0x10a [21024.149107] ? __hrtimer_run_queues+0x1c2/0x4d0 [21024.151417] ? watchdog+0x40/0x40 [21024.153084] ? hrtimer_interrupt+0x128/0x2c0 [21024.154942] ? smp_apic_timer_interrupt+0xa6/0x270 [21024.157090] ? apic_timer_interrupt+0xf/0x20 [21024.159452] [21024.160692] ? native_safe_halt+0x2e/0x50 [21024.162909] kvm_wait+0xcd/0xe0 [21024.164804] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21024.167487] ? newidle_balance.isra.93+0x207/0x580 [21024.169954] do_raw_spin_lock+0x177/0x190 [21024.172028] _raw_spin_lock+0x21/0x30 [21024.174026] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21024.177220] process_one_work+0x2c8/0x700 [21024.179298] worker_thread+0x296/0x6e0 [21024.181200] ? rescuer_thread+0x570/0x570 [21024.183407] kthread+0x1d1/0x200 [21024.185029] ? set_kthread_struct+0x70/0x70 [21024.187024] ret_from_fork+0x1f/0x30 [21027.811049] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18390s! [21027.814015] Showing busy workqueues and worker pools: [21027.830543] workqueue events: flags=0x0 [21027.834058] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21027.841100] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21027.856775] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21027.860901] pending: cpu_latency_work [ptlrpc] [21027.864585] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21027.868018] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21027.873517] workqueue events_long: flags=0x0 [21027.875843] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21027.879240] pending: lu_objects_flush [obdclass] [21027.881924] workqueue events_unbound: flags=0x2 [21027.888737] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21027.892929] pending: flush_memcg_stats_dwork, idle_cull_fn [21027.896009] workqueue events_power_efficient: flags=0x80 [21027.902615] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21027.910391] pending: check_lifetime, do_cache_clean [21027.912140] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21027.920552] pending: neigh_periodic_work, neigh_periodic_work [21027.939777] workqueue mm_percpu_wq: flags=0x8 [21027.942771] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21027.946206] pending: vmstat_update, lru_add_drain_per_cpu [21027.950012] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21027.953385] pending: vmstat_update, lru_add_drain_per_cpu [21027.959833] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21027.966994] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21027.974170] workqueue xprtiod: flags=0xa [21027.976606] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21027.982540] pending: xprt_autoclose, xs_error_handle [21027.986375] workqueue ipv6_addrconf: flags=0x40008 [21027.991616] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21027.996107] pending: addrconf_verify_work [21027.998858] workqueue knbd0-recv: flags=0x1a [21028.008694] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21028.015565] in-flight: 651:recv_work [21028.018799] workqueue ptlrpc_pinger: flags=0x2 [21028.021824] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21028.025717] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21028.033682] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18390s workers=4 idle: 15979 31824 8 [21028.043221] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21018s workers=2 idle: 654 [21040.057298] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [21040.060280] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [21040.060472] Modules linked in: [21040.064500] Modules linked in: [21040.064504] lustre(O) [21040.065663] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21040.065734] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21040.065742] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21040.065745] RIP: 0010:native_safe_halt+0x2e/0x50 [21040.065759] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21040.065764] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21040.065768] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21040.065770] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21040.065773] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21040.065775] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21040.065777] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21040.065780] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21040.065783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21040.065785] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21040.065795] Call Trace: [21040.065800] [21040.065806] ? show_regs.cold.9+0x22/0x2f [21040.065815] ? watchdog_timer_fn.cold.6+0x62/0x10a [21040.065821] ? __hrtimer_run_queues+0x1c2/0x4d0 [21040.065825] ? watchdog+0x40/0x40 [21040.065829] ? hrtimer_interrupt+0x128/0x2c0 [21040.065833] ? smp_apic_timer_interrupt+0xa6/0x270 [21040.065840] ? apic_timer_interrupt+0xf/0x20 [21040.065845] [21040.065848] ? native_safe_halt+0x2e/0x50 [21040.065853] kvm_wait+0xcd/0xe0 [21040.065860] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21040.065869] do_raw_spin_lock+0x177/0x190 [21040.065874] _raw_spin_lock+0x21/0x30 [21040.065879] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21040.066164] ? do_raw_spin_unlock+0x75/0x190 [21040.066170] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21040.066381] ? do_wait_intr+0xf0/0xf0 [21040.066390] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21040.069534] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21040.069614] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21040.069620] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21040.069623] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21040.069635] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21040.069638] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21040.069643] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21040.069646] RDX: 00000000000073d0 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21040.069648] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21040.069656] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21040.069658] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21040.069661] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21040.069664] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21040.069667] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21040.069675] Call Trace: [21040.069680] [21040.069685] ? show_regs.cold.9+0x22/0x2f [21040.069700] ? watchdog_timer_fn.cold.6+0x62/0x10a [21040.069706] ? __hrtimer_run_queues+0x1c2/0x4d0 [21040.069710] ? watchdog+0x40/0x40 [21040.069715] ? hrtimer_interrupt+0x128/0x2c0 [21040.069719] ? smp_apic_timer_interrupt+0xa6/0x270 [21040.069728] ? apic_timer_interrupt+0xf/0x20 [21040.069732] [21040.069735] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21040.069739] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21040.069744] do_raw_spin_lock+0x177/0x190 [21040.069749] _raw_spin_lock+0x21/0x30 [21040.069754] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21040.070503] obd_mod_free+0x46/0x140 [ptlrpc] [21040.070718] mdc_commit_open+0x91/0xb0 [mdc] [21040.070759] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21040.071221] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21040.071587] after_reply+0xe2c/0x1780 [ptlrpc] [21040.071785] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21040.073534] ? __next_timer_interrupt+0x160/0x160 [21040.073544] ptlrpcd+0xa82/0xe40 [ptlrpc] [21040.073743] ? do_wait_intr+0xf0/0xf0 [21040.073754] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21040.108225] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21040.122489] kthread+0x1d1/0x200 [21040.126409] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21040.128453] ? set_kthread_struct+0x70/0x70 [21040.138317] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21040.138441] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21040.138484] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21040.138517] mdc_intent_lock+0x353/0x7e0 [mdc] [21040.138548] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21040.138635] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21040.140004] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21040.146107] ret_from_fork+0x1f/0x30 [21040.522476] lmv_intent_open+0x338/0x1150 [lmv] [21040.524525] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21040.525786] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21040.529570] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21040.530840] ll_intent_lock+0xc5/0xaa0 [lustre] [21040.534249] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21040.535767] ll_lookup_it+0xe0e/0x2270 [lustre] [21040.537510] ? do_raw_spin_unlock+0x75/0x190 [21040.539176] ? _raw_spin_unlock+0x12/0x30 [21040.541125] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21040.543265] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21040.545468] ll_atomic_open+0x36a/0xd70 [lustre] [21040.550803] lookup_open+0x356/0xb40 [21040.552657] do_last+0x313/0xfc0 [21040.554130] ? nd_jump_root+0xe5/0x160 [21040.555496] ? path_init+0x437/0x520 [21040.556827] path_openat+0xf7/0x500 [21040.558738] ? do_raw_spin_unlock+0x75/0x190 [21040.561952] do_filp_open+0x99/0x140 [21040.563273] ? getname_flags+0x6e/0x330 [21040.566570] ? __check_object_size+0xff/0x256 [21040.569690] ? do_raw_spin_unlock+0x75/0x190 [21040.572001] ? _raw_spin_unlock+0x12/0x30 [21040.574069] do_sys_openat2+0x2b4/0x410 [21040.576052] do_sys_open+0x73/0xa0 [21040.577349] __x64_sys_openat+0x24/0x30 [21040.579152] do_syscall_64+0xc1/0x440 [21040.580994] entry_SYSCALL_64_after_hwframe+0x49/0xae [21040.582814] RIP: 0033:0x7f53425a5332 [21040.584764] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21040.595195] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21040.598905] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21040.601657] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21040.603542] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21040.605424] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21040.607414] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21044.058016] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [21044.062797] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21044.086093] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21044.091754] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21044.097008] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21044.100468] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21044.109278] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21044.112921] RAX: 0000000000001fd4 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21044.116402] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21044.119698] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21044.123327] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21044.127879] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21044.130822] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21044.134910] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21044.137543] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21044.140815] Call Trace: [21044.141925] [21044.142947] ? show_regs.cold.9+0x22/0x2f [21044.145070] ? watchdog_timer_fn.cold.6+0x62/0x10a [21044.148675] ? __hrtimer_run_queues+0x1c2/0x4d0 [21044.150790] ? watchdog+0x40/0x40 [21044.152936] ? hrtimer_interrupt+0x128/0x2c0 [21044.154757] ? smp_apic_timer_interrupt+0xa6/0x270 [21044.156830] ? apic_timer_interrupt+0xf/0x20 [21044.159763] [21044.161448] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21044.164558] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21044.167462] do_raw_spin_lock+0x177/0x190 [21044.169425] _raw_spin_lock+0x21/0x30 [21044.171346] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21044.174331] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21044.178381] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21044.181410] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21044.184301] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21044.187334] ? woken_wake_function+0x30/0x30 [21044.189414] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21044.192544] kthread+0x1d1/0x200 [21044.194286] ? set_kthread_struct+0x70/0x70 [21044.196457] ret_from_fork+0x1f/0x30 [21054.896707] rcu: INFO: rcu_sched self-detected stall on CPU [21054.905637] rcu: 2-....: (10906387 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [21054.916021] (t=18960470 jiffies g=274185 q=68709) [21054.921152] rcu: rcu_sched kthread starved for 18418836 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [21054.933963] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [21054.938148] rcu: RCU grace-period kthread stack dump: [21054.940594] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [21054.944714] Call Trace: [21054.945925] __schedule+0x351/0xcb0 [21054.947580] schedule+0xc0/0x180 [21054.948895] schedule_timeout+0xb4/0x190 [21054.950661] ? do_raw_spin_unlock+0x75/0x190 [21054.952608] ? __next_timer_interrupt+0x160/0x160 [21054.956884] ? prepare_to_swait_event+0xf3/0x140 [21054.959037] rcu_gp_kthread+0x8dd/0x15b0 [21054.962552] ? force_qs_rnp+0x2c0/0x2c0 [21054.964634] kthread+0x1d1/0x200 [21054.966383] ? set_kthread_struct+0x70/0x70 [21054.968507] ret_from_fork+0x1f/0x30 [21054.972430] rcu: Stack dump where RCU GP kthread last ran: [21054.976771] Sending NMI from CPU 2 to CPUs 1: [21054.992594] NMI backtrace for cpu 1 [21054.992602] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21054.992604] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21054.992606] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21054.992608] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21054.992611] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [21054.992613] RAX: 0000000000006608 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21054.992615] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21054.992617] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21054.992618] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21054.992620] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21054.992622] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21054.992624] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21054.992625] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21054.992627] Call Trace: [21054.992628] [21054.992629] ? show_regs.cold.9+0x22/0x2f [21054.992630] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [21054.992631] ? nmi_cpu_backtrace_handler+0x15/0x40 [21054.992633] ? nmi_handle+0xbd/0x210 [21054.992634] ? default_do_nmi+0x4b/0x1b0 [21054.992635] ? do_nmi+0x280/0x380 [21054.992637] ? end_repeat_nmi+0x16/0x1f [21054.992638] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21054.992640] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21054.992641] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21054.992643] [21054.992644] do_raw_spin_lock+0x177/0x190 [21054.992646] _raw_spin_lock+0x21/0x30 [21054.992647] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21054.992648] ? do_raw_spin_unlock+0x75/0x190 [21054.992650] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21054.992651] ? do_wait_intr+0xf0/0xf0 [21054.992653] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21054.992654] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21054.992655] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21054.992657] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21054.992658] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21054.992660] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21054.992792] mdc_intent_lock+0x353/0x7e0 [mdc] [21054.992793] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21054.992794] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21054.992796] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21054.992797] lmv_intent_open+0x338/0x1150 [lmv] [21054.992798] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21054.992799] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21054.992801] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21054.992802] ll_intent_lock+0xc5/0xaa0 [lustre] [21054.992803] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21054.992805] ll_lookup_it+0xe0e/0x2270 [lustre] [21054.992806] ? do_raw_spin_unlock+0x75/0x190 [21054.992807] ? _raw_spin_unlock+0x12/0x30 [21054.992808] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21054.992809] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21054.992810] ll_atomic_open+0x36a/0xd70 [lustre] [21054.992811] lookup_open+0x356/0xb40 [21054.992813] do_last+0x313/0xfc0 [21054.992814] ? nd_jump_root+0xe5/0x160 [21054.992815] ? path_init+0x437/0x520 [21054.992816] path_openat+0xf7/0x500 [21054.992817] ? do_raw_spin_unlock+0x75/0x190 [21054.992818] do_filp_open+0x99/0x140 [21054.992819] ? getname_flags+0x6e/0x330 [21054.992820] ? __check_object_size+0xff/0x256 [21054.992822] ? do_raw_spin_unlock+0x75/0x190 [21054.992823] ? _raw_spin_unlock+0x12/0x30 [21054.992824] do_sys_openat2+0x2b4/0x410 [21054.992825] do_sys_open+0x73/0xa0 [21054.992826] __x64_sys_openat+0x24/0x30 [21054.992827] do_syscall_64+0xc1/0x440 [21054.992828] entry_SYSCALL_64_after_hwframe+0x49/0xae [21054.992829] RIP: 0033:0x7f53425a5332 [21054.992831] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21054.992833] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21054.992836] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21054.992837] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21054.992838] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21054.992840] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21054.992841] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21054.993736] Sending NMI from CPU 2 to CPUs 1: [21055.381629] NMI backtrace for cpu 1 [21055.381638] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21055.381640] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21055.381642] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21055.381644] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21055.381646] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 [21055.381648] RAX: 0000000000007115 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21055.381649] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21055.381651] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21055.381652] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21055.381665] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21055.381667] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21055.381668] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21055.381669] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21055.381671] Call Trace: [21055.381671] [21055.381672] ? show_regs.cold.9+0x22/0x2f [21055.381673] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [21055.381674] ? nmi_cpu_backtrace_handler+0x15/0x40 [21055.381676] ? nmi_handle+0xbd/0x210 [21055.381676] ? default_do_nmi+0x4b/0x1b0 [21055.381677] ? do_nmi+0x280/0x380 [21055.381894] ? end_repeat_nmi+0x16/0x1f [21055.381896] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21055.381897] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21055.381898] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21055.381899] [21055.381900] do_raw_spin_lock+0x177/0x190 [21055.381901] _raw_spin_lock+0x21/0x30 [21055.381903] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21055.381904] ? do_raw_spin_unlock+0x75/0x190 [21055.381905] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21055.381906] ? do_wait_intr+0xf0/0xf0 [21055.381907] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21055.381908] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21055.381909] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21055.381911] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21055.381912] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21055.381913] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21055.381914] mdc_intent_lock+0x353/0x7e0 [mdc] [21055.381916] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21055.381917] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21055.381918] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21055.381919] lmv_intent_open+0x338/0x1150 [lmv] [21055.381921] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21055.381922] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21055.381923] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21055.381924] ll_intent_lock+0xc5/0xaa0 [lustre] [21055.381926] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21055.381927] ll_lookup_it+0xe0e/0x2270 [lustre] [21055.381928] ? do_raw_spin_unlock+0x75/0x190 [21055.381929] ? _raw_spin_unlock+0x12/0x30 [21055.381930] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21055.381931] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21055.381933] ll_atomic_open+0x36a/0xd70 [lustre] [21055.381934] lookup_open+0x356/0xb40 [21055.382113] do_last+0x313/0xfc0 [21055.382114] ? nd_jump_root+0xe5/0x160 [21055.382115] ? path_init+0x437/0x520 [21055.382116] path_openat+0xf7/0x500 [21055.382117] ? do_raw_spin_unlock+0x75/0x190 [21055.382118] do_filp_open+0x99/0x140 [21055.382119] ? getname_flags+0x6e/0x330 [21055.382120] ? __check_object_size+0xff/0x256 [21055.382121] ? do_raw_spin_unlock+0x75/0x190 [21055.382122] ? _raw_spin_unlock+0x12/0x30 [21055.382123] do_sys_openat2+0x2b4/0x410 [21055.382124] do_sys_open+0x73/0xa0 [21055.382125] __x64_sys_openat+0x24/0x30 [21055.382126] do_syscall_64+0xc1/0x440 [21055.382128] entry_SYSCALL_64_after_hwframe+0x49/0xae [21055.382129] RIP: 0033:0x7f53425a5332 [21055.382131] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21055.382133] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21055.382136] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21055.382138] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21055.382140] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21055.382141] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21055.382143] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21055.382150] NMI backtrace for cpu 2 [21055.674846] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21055.682065] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21055.690928] Call Trace: [21055.694001] [21055.694845] ? dump_stack+0xbb/0x10e [21055.696074] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [21055.704049] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [21055.709665] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [21055.717095] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [21055.720732] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [21055.734782] ? rcu_dump_cpu_stacks+0x11b/0x171 [21055.736549] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [21055.739382] ? update_process_times+0x6d/0xc0 [21055.741189] ? tick_sched_handle.isra.18+0x27/0x90 [21055.743246] ? tick_sched_timer+0x53/0xb0 [21055.745025] ? __hrtimer_run_queues+0x1c2/0x4d0 [21055.748149] ? tick_sched_handle.isra.18+0x90/0x90 [21055.751252] ? hrtimer_interrupt+0x128/0x2c0 [21055.753577] ? smp_apic_timer_interrupt+0xa6/0x270 [21055.756592] ? apic_timer_interrupt+0xf/0x20 [21055.758867] [21055.759691] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21055.762280] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21055.765183] ? do_raw_spin_lock+0x177/0x190 [21055.767303] ? _raw_spin_lock+0x21/0x30 [21055.769150] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21055.772222] ? obd_mod_free+0x46/0x140 [ptlrpc] [21055.780989] ? mdc_commit_open+0x91/0xb0 [mdc] [21055.785738] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21055.792495] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21055.800870] ? after_reply+0xe2c/0x1780 [ptlrpc] [21055.807437] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21055.814516] ? __next_timer_interrupt+0x160/0x160 [21055.819872] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [21055.828153] ? do_wait_intr+0xf0/0xf0 [21055.833986] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21055.846342] ? kthread+0x1d1/0x200 [21055.851105] ? set_kthread_struct+0x70/0x70 [21055.856710] ? ret_from_fork+0x1f/0x30 [21056.052095] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [21056.057624] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21056.086039] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21056.091306] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21056.093994] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21056.106710] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21056.124683] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21056.147540] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21056.153616] RAX: 0000000000007109 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21056.169277] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [21056.184387] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21056.208886] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21056.215897] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21056.226687] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21056.239854] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21056.241786] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21056.254372] Call Trace: [21056.260568] [21056.261302] ? show_regs.cold.9+0x22/0x2f [21056.269929] ? watchdog_timer_fn.cold.6+0x62/0x10a [21056.276029] ? __hrtimer_run_queues+0x1c2/0x4d0 [21056.279574] ? watchdog+0x40/0x40 [21056.283109] ? hrtimer_interrupt+0x128/0x2c0 [21056.287383] ? smp_apic_timer_interrupt+0xa6/0x270 [21056.290610] ? apic_timer_interrupt+0xf/0x20 [21056.299945] [21056.301267] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21056.309185] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21056.313242] ? newidle_balance.isra.93+0x207/0x580 [21056.314722] do_raw_spin_lock+0x177/0x190 [21056.327540] _raw_spin_lock+0x21/0x30 [21056.335737] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21056.337779] process_one_work+0x2c8/0x700 [21056.342215] worker_thread+0x296/0x6e0 [21056.348497] ? rescuer_thread+0x570/0x570 [21056.350497] kthread+0x1d1/0x200 [21056.351614] ? set_kthread_struct+0x70/0x70 [21056.353144] ret_from_fork+0x1f/0x30 [21058.528362] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18421s! [21058.531139] Showing busy workqueues and worker pools: [21058.541219] workqueue events: flags=0x0 [21058.543249] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21058.553276] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21058.575590] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21058.584795] pending: cpu_latency_work [ptlrpc] [21058.589680] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21058.596389] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21058.601573] workqueue events_long: flags=0x0 [21058.608903] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21058.618609] pending: lu_objects_flush [obdclass] [21058.620575] workqueue events_unbound: flags=0x2 [21058.625122] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21058.631270] pending: flush_memcg_stats_dwork, idle_cull_fn [21058.634250] workqueue events_power_efficient: flags=0x80 [21058.638074] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21058.643014] pending: check_lifetime, do_cache_clean [21058.646001] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21058.652802] pending: neigh_periodic_work, neigh_periodic_work [21058.656113] workqueue mm_percpu_wq: flags=0x8 [21058.665665] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21058.673455] pending: vmstat_update, lru_add_drain_per_cpu [21058.679369] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21058.683137] pending: vmstat_update, lru_add_drain_per_cpu [21058.685213] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21058.688823] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21058.692514] workqueue xprtiod: flags=0xa [21058.693865] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21058.701552] pending: xprt_autoclose, xs_error_handle [21058.705557] workqueue ipv6_addrconf: flags=0x40008 [21058.710441] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21058.714196] pending: addrconf_verify_work [21058.717552] workqueue knbd0-recv: flags=0x1a [21058.727704] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21058.735431] in-flight: 651:recv_work [21058.737700] workqueue ptlrpc_pinger: flags=0x2 [21058.741122] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21058.744171] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21058.753135] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18421s workers=4 idle: 15979 31824 8 [21058.760625] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21049s workers=2 idle: 654 [21072.059608] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [21072.074823] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21072.130963] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21072.145978] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21072.159100] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21072.166307] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21072.185069] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21072.199052] RAX: 0000000000007b08 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21072.202450] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21072.212739] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21072.224795] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21072.231325] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21072.241242] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21072.253348] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21072.263843] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21072.272234] Call Trace: [21072.276631] [21072.278619] ? show_regs.cold.9+0x22/0x2f [21072.282449] ? watchdog_timer_fn.cold.6+0x62/0x10a [21072.287415] ? __hrtimer_run_queues+0x1c2/0x4d0 [21072.292396] ? watchdog+0x40/0x40 [21072.296021] ? hrtimer_interrupt+0x128/0x2c0 [21072.301113] ? smp_apic_timer_interrupt+0xa6/0x270 [21072.302787] ? apic_timer_interrupt+0xf/0x20 [21072.308472] [21072.311634] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21072.320176] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21072.328733] do_raw_spin_lock+0x177/0x190 [21072.334700] _raw_spin_lock+0x21/0x30 [21072.339030] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21072.345646] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21072.348610] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21072.357156] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21072.364756] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21072.370337] ? woken_wake_function+0x30/0x30 [21072.374512] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21072.381120] kthread+0x1d1/0x200 [21072.383993] ? set_kthread_struct+0x70/0x70 [21072.388821] ret_from_fork+0x1f/0x30 [21080.057410] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [21080.059587] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [21080.061717] Modules linked in: lustre(O) [21080.065762] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log [21080.067807] osp(O) ofd(O) lod(O) mdt(O) [21080.095705] dm_mod sha512_ssse3 sha512_generic [21080.097900] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) [21080.099990] [21080.099998] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21080.103645] lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel [21080.104816] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21080.104824] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21080.104838] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21080.104842] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21080.104848] RAX: 0000000000007614 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21080.104850] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21080.104852] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21080.104854] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21080.104857] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21080.104860] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21080.104862] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21080.104864] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21080.104873] Call Trace: [21080.104878] [21080.104884] ? show_regs.cold.9+0x22/0x2f [21080.104894] ? watchdog_timer_fn.cold.6+0x62/0x10a [21080.104900] ? __hrtimer_run_queues+0x1c2/0x4d0 [21080.104905] ? watchdog+0x40/0x40 [21080.104909] ? hrtimer_interrupt+0x128/0x2c0 [21080.104913] ? smp_apic_timer_interrupt+0xa6/0x270 [21080.104920] ? apic_timer_interrupt+0xf/0x20 [21080.104926] [21080.104928] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21080.104935] do_raw_spin_lock+0x177/0x190 [21080.104940] _raw_spin_lock+0x21/0x30 [21080.104945] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21080.105371] ? do_raw_spin_unlock+0x75/0x190 [21080.105378] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21080.105770] ? do_wait_intr+0xf0/0xf0 [21080.105780] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21080.106047] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21080.106409] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21080.106725] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21080.106845] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21080.106885] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21080.106918] mdc_intent_lock+0x353/0x7e0 [mdc] [21080.106948] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21080.107124] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21080.107428] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21080.107457] lmv_intent_open+0x338/0x1150 [lmv] [21080.107488] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21080.107752] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21080.107839] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21080.107864] ll_intent_lock+0xc5/0xaa0 [lustre] [21080.108025] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21080.108111] ll_lookup_it+0xe0e/0x2270 [lustre] [21080.108264] ? do_raw_spin_unlock+0x75/0x190 [21080.108272] ? _raw_spin_unlock+0x12/0x30 [21080.108279] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21080.108307] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21080.108328] ll_atomic_open+0x36a/0xd70 [lustre] [21080.108475] lookup_open+0x356/0xb40 [21080.108484] do_last+0x313/0xfc0 [21080.108489] ? nd_jump_root+0xe5/0x160 [21080.108493] ? path_init+0x437/0x520 [21080.108498] path_openat+0xf7/0x500 [21080.108502] ? do_raw_spin_unlock+0x75/0x190 [21080.108507] do_filp_open+0x99/0x140 [21080.108513] ? getname_flags+0x6e/0x330 [21080.108517] ? __check_object_size+0xff/0x256 [21080.108535] ? do_raw_spin_unlock+0x75/0x190 [21080.108539] ? _raw_spin_unlock+0x12/0x30 [21080.108544] do_sys_openat2+0x2b4/0x410 [21080.108548] do_sys_open+0x73/0xa0 [21080.108552] __x64_sys_openat+0x24/0x30 [21080.108556] do_syscall_64+0xc1/0x440 [21080.108562] entry_SYSCALL_64_after_hwframe+0x49/0xae [21080.108568] RIP: 0033:0x7f53425a5332 [21080.108616] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21080.120891] ata_piix [21080.136521] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21080.136544] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21080.136547] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21080.136549] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21080.136551] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21080.136554] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21080.422667] serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21080.426629] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21080.445954] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21080.456823] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21080.459450] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21080.476797] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21080.492670] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21080.498404] RDX: 0000000000006618 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21080.502915] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21080.513534] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21080.517936] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21080.522733] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21080.526519] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21080.528440] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21080.536473] Call Trace: [21080.538781] [21080.540170] ? show_regs.cold.9+0x22/0x2f [21080.542605] ? watchdog_timer_fn.cold.6+0x62/0x10a [21080.547412] ? __hrtimer_run_queues+0x1c2/0x4d0 [21080.548813] ? watchdog+0x40/0x40 [21080.553168] ? hrtimer_interrupt+0x128/0x2c0 [21080.558317] ? smp_apic_timer_interrupt+0xa6/0x270 [21080.560824] ? apic_timer_interrupt+0xf/0x20 [21080.564916] [21080.567819] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21080.571490] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21080.576067] do_raw_spin_lock+0x177/0x190 [21080.579282] _raw_spin_lock+0x21/0x30 [21080.582394] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21080.587503] obd_mod_free+0x46/0x140 [ptlrpc] [21080.591710] mdc_commit_open+0x91/0xb0 [mdc] [21080.598917] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21080.606149] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21080.617292] after_reply+0xe2c/0x1780 [ptlrpc] [21080.621022] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21080.623006] ? __next_timer_interrupt+0x160/0x160 [21080.631852] ptlrpcd+0xa82/0xe40 [ptlrpc] [21080.634849] ? do_wait_intr+0xf0/0xf0 [21080.638964] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21080.645158] kthread+0x1d1/0x200 [21080.648328] ? set_kthread_struct+0x70/0x70 [21080.654600] ret_from_fork+0x1f/0x30 [21084.052038] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [21084.056337] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21084.106079] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21084.114784] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21084.120738] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21084.126867] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21084.134705] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21084.145156] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21084.149171] RAX: 0000000000003b79 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21084.153436] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [21084.159859] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21084.165807] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21084.169500] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21084.177566] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21084.182638] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21084.185573] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21084.191001] Call Trace: [21084.192249] [21084.195392] ? show_regs.cold.9+0x22/0x2f [21084.197816] ? watchdog_timer_fn.cold.6+0x62/0x10a [21084.202893] ? __hrtimer_run_queues+0x1c2/0x4d0 [21084.205020] ? watchdog+0x40/0x40 [21084.206529] ? hrtimer_interrupt+0x128/0x2c0 [21084.208894] ? smp_apic_timer_interrupt+0xa6/0x270 [21084.210214] ? apic_timer_interrupt+0xf/0x20 [21084.213144] [21084.214142] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21084.218797] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21084.226964] ? newidle_balance.isra.93+0x207/0x580 [21084.228558] do_raw_spin_lock+0x177/0x190 [21084.230013] _raw_spin_lock+0x21/0x30 [21084.231192] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21084.234243] process_one_work+0x2c8/0x700 [21084.236526] worker_thread+0x296/0x6e0 [21084.237998] ? rescuer_thread+0x570/0x570 [21084.239576] kthread+0x1d1/0x200 [21084.240940] ? set_kthread_struct+0x70/0x70 [21084.242689] ret_from_fork+0x1f/0x30 [21089.255140] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18452s! [21089.259605] Showing busy workqueues and worker pools: [21089.264108] workqueue events: flags=0x0 [21089.275721] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21089.280210] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21089.287510] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21089.297415] pending: cpu_latency_work [ptlrpc] [21089.299467] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21089.310037] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21089.316251] workqueue events_long: flags=0x0 [21089.317989] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21089.324196] pending: lu_objects_flush [obdclass] [21089.331796] workqueue events_unbound: flags=0x2 [21089.334483] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21089.340526] pending: flush_memcg_stats_dwork, idle_cull_fn [21089.345774] workqueue events_power_efficient: flags=0x80 [21089.348685] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21089.359624] pending: check_lifetime, do_cache_clean [21089.362728] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21089.373051] pending: neigh_periodic_work, neigh_periodic_work [21089.384350] workqueue mm_percpu_wq: flags=0x8 [21089.385988] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21089.395339] pending: vmstat_update, lru_add_drain_per_cpu [21089.398471] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21089.404526] pending: vmstat_update, lru_add_drain_per_cpu [21089.408236] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21089.414083] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21089.419467] workqueue xprtiod: flags=0xa [21089.424415] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21089.428414] pending: xprt_autoclose, xs_error_handle [21089.433719] workqueue ipv6_addrconf: flags=0x40008 [21089.435679] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21089.438963] pending: addrconf_verify_work [21089.440854] workqueue knbd0-recv: flags=0x1a [21089.442832] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21089.448526] in-flight: 651:recv_work [21089.454932] workqueue ptlrpc_pinger: flags=0x2 [21089.458646] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21089.462088] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21089.471137] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18452s workers=4 idle: 15979 31824 8 [21089.475013] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21079s workers=2 idle: 654 [21100.058028] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [21100.064635] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21100.122075] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21100.140806] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21100.153445] RIP: 0010:native_safe_halt+0x2e/0x50 [21100.157353] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21100.186324] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21100.196799] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21100.205276] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21100.211516] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21100.220324] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21100.232975] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21100.240416] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21100.246063] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21100.249679] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21100.264967] Call Trace: [21100.269739] [21100.274561] ? show_regs.cold.9+0x22/0x2f [21100.277908] ? watchdog_timer_fn.cold.6+0x62/0x10a [21100.280283] ? __hrtimer_run_queues+0x1c2/0x4d0 [21100.282502] ? watchdog+0x40/0x40 [21100.283828] ? hrtimer_interrupt+0x128/0x2c0 [21100.286338] ? smp_apic_timer_interrupt+0xa6/0x270 [21100.288412] ? apic_timer_interrupt+0xf/0x20 [21100.291564] [21100.294791] ? native_safe_halt+0x2e/0x50 [21100.299394] kvm_wait+0xcd/0xe0 [21100.300399] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21100.302838] do_raw_spin_lock+0x177/0x190 [21100.304585] _raw_spin_lock+0x21/0x30 [21100.306735] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21100.311710] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21100.317838] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21100.324837] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21100.331734] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21100.338723] ? woken_wake_function+0x30/0x30 [21100.342724] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21100.347628] kthread+0x1d1/0x200 [21100.349695] ? set_kthread_struct+0x70/0x70 [21100.351742] ret_from_fork+0x1f/0x30 [21108.055011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [21108.060525] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [21108.069685] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21108.070038] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21108.070045] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21108.070048] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [21108.070061] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [21108.070065] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21108.070070] RAX: 0000000000005d17 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21108.070072] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21108.070074] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21108.070077] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21108.070079] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21108.070082] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21108.070084] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21108.070087] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21108.070096] Call Trace: [21108.070101] [21108.070106] ? show_regs.cold.9+0x22/0x2f [21108.070115] ? watchdog_timer_fn.cold.6+0x62/0x10a [21108.070122] ? __hrtimer_run_queues+0x1c2/0x4d0 [21108.070126] ? watchdog+0x40/0x40 [21108.070131] ? hrtimer_interrupt+0x128/0x2c0 [21108.070135] ? smp_apic_timer_interrupt+0xa6/0x270 [21108.070143] ? apic_timer_interrupt+0xf/0x20 [21108.070148] [21108.070151] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [21108.070158] do_raw_spin_lock+0x177/0x190 [21108.070163] _raw_spin_lock+0x21/0x30 [21108.070168] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21108.070439] ? do_raw_spin_unlock+0x75/0x190 [21108.070446] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21108.081151] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21108.081229] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21108.081237] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21108.081240] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21108.081254] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21108.081258] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21108.081263] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21108.081266] RDX: 00000000000062d8 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21108.081268] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21108.081270] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21108.081272] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21108.081275] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21108.081278] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21108.081280] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21108.081289] Call Trace: [21108.081294] [21108.081301] ? show_regs.cold.9+0x22/0x2f [21108.081310] ? watchdog_timer_fn.cold.6+0x62/0x10a [21108.081316] ? __hrtimer_run_queues+0x1c2/0x4d0 [21108.081320] ? watchdog+0x40/0x40 [21108.081325] ? hrtimer_interrupt+0x128/0x2c0 [21108.081328] ? smp_apic_timer_interrupt+0xa6/0x270 [21108.081336] ? apic_timer_interrupt+0xf/0x20 [21108.081340] [21108.081343] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21108.081348] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21108.081352] do_raw_spin_lock+0x177/0x190 [21108.081357] _raw_spin_lock+0x21/0x30 [21108.081363] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21108.105825] ? do_wait_intr+0xf0/0xf0 [21108.105909] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21108.106317] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21108.106785] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21108.107138] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21108.107254] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21108.107297] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21108.107328] mdc_intent_lock+0x353/0x7e0 [mdc] [21108.107358] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21108.107443] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21108.107660] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21108.107689] lmv_intent_open+0x338/0x1150 [lmv] [21108.107721] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21108.107808] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21108.113100] obd_mod_free+0x46/0x140 [ptlrpc] [21108.113661] mdc_commit_open+0x91/0xb0 [mdc] [21108.113701] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21108.118000] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21108.118117] ll_intent_lock+0xc5/0xaa0 [lustre] [21108.118246] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21108.118332] ll_lookup_it+0xe0e/0x2270 [lustre] [21108.118418] ? do_raw_spin_unlock+0x75/0x190 [21108.118426] ? _raw_spin_unlock+0x12/0x30 [21108.118434] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21108.118463] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21108.118484] ll_atomic_open+0x36a/0xd70 [lustre] [21108.118568] lookup_open+0x356/0xb40 [21108.118577] do_last+0x313/0xfc0 [21108.118582] ? nd_jump_root+0xe5/0x160 [21108.118587] ? path_init+0x437/0x520 [21108.118591] path_openat+0xf7/0x500 [21108.118596] ? do_raw_spin_unlock+0x75/0x190 [21108.118601] do_filp_open+0x99/0x140 [21108.118607] ? getname_flags+0x6e/0x330 [21108.118612] ? __check_object_size+0xff/0x256 [21108.118619] ? do_raw_spin_unlock+0x75/0x190 [21108.118623] ? _raw_spin_unlock+0x12/0x30 [21108.118627] do_sys_openat2+0x2b4/0x410 [21108.118631] do_sys_open+0x73/0xa0 [21108.118636] __x64_sys_openat+0x24/0x30 [21108.118640] do_syscall_64+0xc1/0x440 [21108.118647] entry_SYSCALL_64_after_hwframe+0x49/0xae [21108.118653] RIP: 0033:0x7f53425a5332 [21108.118700] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21108.118705] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21108.118710] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21108.118713] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21108.118716] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21108.118718] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21108.118720] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21108.704183] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21108.708695] after_reply+0xe2c/0x1780 [ptlrpc] [21108.715644] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21108.718536] ? __next_timer_interrupt+0x160/0x160 [21108.720880] ptlrpcd+0xa82/0xe40 [ptlrpc] [21108.722660] ? do_wait_intr+0xf0/0xf0 [21108.726769] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21108.730516] kthread+0x1d1/0x200 [21108.732200] ? set_kthread_struct+0x70/0x70 [21108.734051] ret_from_fork+0x1f/0x30 [21116.053075] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [21116.061365] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21116.076136] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21116.079856] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21116.082933] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21116.085883] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21116.087960] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21116.094177] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21116.096751] RAX: 0000000000001185 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21116.098891] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [21116.101003] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21116.103268] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21116.105573] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21116.108164] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21116.111139] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21116.113309] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21116.116682] Call Trace: [21116.117823] [21116.118879] ? show_regs.cold.9+0x22/0x2f [21116.120673] ? watchdog_timer_fn.cold.6+0x62/0x10a [21116.122704] ? __hrtimer_run_queues+0x1c2/0x4d0 [21116.124585] ? watchdog+0x40/0x40 [21116.125694] ? hrtimer_interrupt+0x128/0x2c0 [21116.127192] ? smp_apic_timer_interrupt+0xa6/0x270 [21116.128656] ? apic_timer_interrupt+0xf/0x20 [21116.130174] [21116.130938] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21116.132859] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21116.134568] ? newidle_balance.isra.93+0x207/0x580 [21116.136142] do_raw_spin_lock+0x177/0x190 [21116.137527] _raw_spin_lock+0x21/0x30 [21116.138677] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21116.140650] process_one_work+0x2c8/0x700 [21116.142062] worker_thread+0x296/0x6e0 [21116.143328] ? rescuer_thread+0x570/0x570 [21116.144704] kthread+0x1d1/0x200 [21116.145871] ? set_kthread_struct+0x70/0x70 [21116.147378] ret_from_fork+0x1f/0x30 [21119.969219] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18482s! [21119.988565] Showing busy workqueues and worker pools: [21119.998431] workqueue events: flags=0x0 [21119.999788] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21120.010840] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21120.023291] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21120.030398] pending: cpu_latency_work [ptlrpc] [21120.034339] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21120.039583] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21120.050290] workqueue events_long: flags=0x0 [21120.055948] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21120.060933] pending: lu_objects_flush [obdclass] [21120.065218] workqueue events_unbound: flags=0x2 [21120.072286] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21120.075371] pending: flush_memcg_stats_dwork, idle_cull_fn [21120.078825] workqueue events_power_efficient: flags=0x80 [21120.085217] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21120.091303] pending: check_lifetime, do_cache_clean [21120.095524] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21120.100193] pending: neigh_periodic_work, neigh_periodic_work [21120.110964] workqueue mm_percpu_wq: flags=0x8 [21120.123808] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21120.135358] pending: vmstat_update, lru_add_drain_per_cpu [21120.146466] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21120.160621] pending: vmstat_update, lru_add_drain_per_cpu [21120.162534] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21120.168491] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21120.173441] workqueue xprtiod: flags=0xa [21120.177061] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21120.179317] pending: xprt_autoclose, xs_error_handle [21120.182288] workqueue ipv6_addrconf: flags=0x40008 [21120.184625] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21120.191841] pending: addrconf_verify_work [21120.195731] workqueue knbd0-recv: flags=0x1a [21120.198555] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21120.203567] in-flight: 651:recv_work [21120.207387] workqueue ptlrpc_pinger: flags=0x2 [21120.209598] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21120.213123] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21120.220372] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18483s workers=4 idle: 15979 31824 8 [21120.225011] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21110s workers=2 idle: 654 [21128.058010] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [21128.064334] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21128.095070] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21128.098936] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21128.103484] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21128.105902] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21128.112760] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21128.115471] RAX: 0000000000001e6d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21128.117791] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21128.120318] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21128.122771] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21128.127464] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21128.130275] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21128.142422] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21128.147345] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21128.150947] Call Trace: [21128.152117] [21128.152992] ? show_regs.cold.9+0x22/0x2f [21128.154794] ? watchdog_timer_fn.cold.6+0x62/0x10a [21128.157032] ? __hrtimer_run_queues+0x1c2/0x4d0 [21128.159053] ? watchdog+0x40/0x40 [21128.166073] ? hrtimer_interrupt+0x128/0x2c0 [21128.169421] ? smp_apic_timer_interrupt+0xa6/0x270 [21128.173027] ? apic_timer_interrupt+0xf/0x20 [21128.174440] [21128.184756] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21128.187344] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21128.190272] do_raw_spin_lock+0x177/0x190 [21128.197708] _raw_spin_lock+0x21/0x30 [21128.198971] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21128.206032] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21128.210132] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21128.213298] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21128.223326] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21128.225351] ? woken_wake_function+0x30/0x30 [21128.231960] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21128.234423] kthread+0x1d1/0x200 [21128.236063] ? set_kthread_struct+0x70/0x70 [21128.237459] ret_from_fork+0x1f/0x30 [21136.055014] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [21136.057116] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [21136.061506] Modules linked in: [21136.068961] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21136.069054] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21136.069066] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21136.069069] RIP: 0010:native_safe_halt+0x2e/0x50 [21136.069126] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21136.069131] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21136.069136] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21136.069139] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21136.069141] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21136.069143] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21136.069146] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21136.069149] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21136.069151] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21136.069154] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21136.069161] Call Trace: [21136.069172] [21136.069181] ? show_regs.cold.9+0x22/0x2f [21136.069191] ? watchdog_timer_fn.cold.6+0x62/0x10a [21136.069197] ? __hrtimer_run_queues+0x1c2/0x4d0 [21136.069201] ? watchdog+0x40/0x40 [21136.069206] ? hrtimer_interrupt+0x128/0x2c0 [21136.069214] ? smp_apic_timer_interrupt+0xa6/0x270 [21136.069221] ? apic_timer_interrupt+0xf/0x20 [21136.069225] [21136.069228] ? native_safe_halt+0x2e/0x50 [21136.069232] kvm_wait+0xcd/0xe0 [21136.069239] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21136.069249] do_raw_spin_lock+0x177/0x190 [21136.069275] _raw_spin_lock+0x21/0x30 [21136.069281] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21136.070311] obd_mod_free+0x46/0x140 [ptlrpc] [21136.073024] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21136.073166] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21136.073173] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21136.073176] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21136.073190] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21136.073194] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21136.073270] RAX: 0000000000005a11 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21136.073274] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21136.073276] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21136.073279] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21136.073281] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21136.073285] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21136.073288] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21136.073291] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21136.073299] Call Trace: [21136.073304] [21136.073309] ? show_regs.cold.9+0x22/0x2f [21136.073318] ? watchdog_timer_fn.cold.6+0x62/0x10a [21136.073324] ? __hrtimer_run_queues+0x1c2/0x4d0 [21136.073329] ? watchdog+0x40/0x40 [21136.073334] ? hrtimer_interrupt+0x128/0x2c0 [21136.073338] ? smp_apic_timer_interrupt+0xa6/0x270 [21136.073346] ? apic_timer_interrupt+0xf/0x20 [21136.073351] [21136.073354] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21136.073430] do_raw_spin_lock+0x177/0x190 [21136.073438] _raw_spin_lock+0x21/0x30 [21136.073443] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21136.074041] ? do_raw_spin_unlock+0x75/0x190 [21136.074048] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21136.074341] ? do_wait_intr+0xf0/0xf0 [21136.074352] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21136.074888] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21136.106675] mdc_commit_open+0x91/0xb0 [mdc] [21136.106738] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21136.112909] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21136.113283] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21136.113638] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21136.113810] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21136.120203] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21136.122250] mdc_intent_lock+0x353/0x7e0 [mdc] [21136.122300] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21136.122443] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21136.122886] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21136.122918] lmv_intent_open+0x338/0x1150 [lmv] [21136.137507] after_reply+0xe2c/0x1780 [ptlrpc] [21136.145718] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21136.149344] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21136.149944] ? __next_timer_interrupt+0x160/0x160 [21136.149960] ptlrpcd+0xa82/0xe40 [ptlrpc] [21136.154835] ? do_wait_intr+0xf0/0xf0 [21136.154851] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21136.159923] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21136.160070] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21136.160115] ll_intent_lock+0xc5/0xaa0 [lustre] [21136.160209] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21136.160525] ll_lookup_it+0xe0e/0x2270 [lustre] [21136.160610] ? do_raw_spin_unlock+0x75/0x190 [21136.160618] ? _raw_spin_unlock+0x12/0x30 [21136.160625] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21136.160650] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21136.160669] ll_atomic_open+0x36a/0xd70 [lustre] [21136.160768] lookup_open+0x356/0xb40 [21136.160778] do_last+0x313/0xfc0 [21136.160783] ? nd_jump_root+0xe5/0x160 [21136.160788] ? path_init+0x437/0x520 [21136.160793] path_openat+0xf7/0x500 [21136.160804] ? do_raw_spin_unlock+0x75/0x190 [21136.160809] do_filp_open+0x99/0x140 [21136.160816] ? getname_flags+0x6e/0x330 [21136.160821] ? __check_object_size+0xff/0x256 [21136.160832] ? do_raw_spin_unlock+0x75/0x190 [21136.160836] ? _raw_spin_unlock+0x12/0x30 [21136.160840] do_sys_openat2+0x2b4/0x410 [21136.160845] do_sys_open+0x73/0xa0 [21136.160853] __x64_sys_openat+0x24/0x30 [21136.160857] do_syscall_64+0xc1/0x440 [21136.160864] entry_SYSCALL_64_after_hwframe+0x49/0xae [21136.160870] RIP: 0033:0x7f53425a5332 [21136.160878] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21136.160886] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21136.160892] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21136.160894] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21136.160896] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21136.160899] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21136.160900] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21136.525136] kthread+0x1d1/0x200 [21136.526739] ? set_kthread_struct+0x70/0x70 [21136.528591] ret_from_fork+0x1f/0x30 [21148.052377] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [21148.058386] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21148.099700] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21148.107629] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21148.112792] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21148.118335] RIP: 0010:native_safe_halt+0x2e/0x50 [21148.127557] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21148.148434] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21148.158290] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21148.162574] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [21148.172538] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21148.176398] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21148.179921] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21148.183259] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21148.188999] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21148.197416] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21148.204949] Call Trace: [21148.205850] [21148.206558] ? show_regs.cold.9+0x22/0x2f [21148.207932] ? watchdog_timer_fn.cold.6+0x62/0x10a [21148.209631] ? __hrtimer_run_queues+0x1c2/0x4d0 [21148.211228] ? watchdog+0x40/0x40 [21148.212359] ? hrtimer_interrupt+0x128/0x2c0 [21148.213746] ? smp_apic_timer_interrupt+0xa6/0x270 [21148.215443] ? apic_timer_interrupt+0xf/0x20 [21148.216902] [21148.217660] ? native_safe_halt+0x2e/0x50 [21148.218998] kvm_wait+0xcd/0xe0 [21148.220366] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21148.222025] ? newidle_balance.isra.93+0x207/0x580 [21148.223642] do_raw_spin_lock+0x177/0x190 [21148.224956] _raw_spin_lock+0x21/0x30 [21148.226291] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21148.236492] process_one_work+0x2c8/0x700 [21148.242669] worker_thread+0x296/0x6e0 [21148.245885] ? rescuer_thread+0x570/0x570 [21148.253486] kthread+0x1d1/0x200 [21148.262451] ? set_kthread_struct+0x70/0x70 [21148.263811] ret_from_fork+0x1f/0x30 [21150.692073] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18513s! [21150.694685] Showing busy workqueues and worker pools: [21150.696418] workqueue events: flags=0x0 [21150.697684] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21150.700272] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21150.706028] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21150.709518] pending: cpu_latency_work [ptlrpc] [21150.712072] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21150.714865] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21150.719493] workqueue events_long: flags=0x0 [21150.721765] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21150.725227] pending: lu_objects_flush [obdclass] [21150.727510] workqueue events_unbound: flags=0x2 [21150.729867] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21150.732057] pending: flush_memcg_stats_dwork, idle_cull_fn [21150.734384] workqueue events_power_efficient: flags=0x80 [21150.736247] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21150.738686] pending: check_lifetime, do_cache_clean [21150.740616] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21150.751858] pending: neigh_periodic_work, neigh_periodic_work [21150.754176] workqueue mm_percpu_wq: flags=0x8 [21150.755860] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21150.758207] pending: vmstat_update, lru_add_drain_per_cpu [21150.759979] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21150.762607] pending: vmstat_update, lru_add_drain_per_cpu [21150.764706] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21150.766985] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21150.771903] workqueue xprtiod: flags=0xa [21150.773388] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21150.775829] pending: xprt_autoclose, xs_error_handle [21150.777708] workqueue ipv6_addrconf: flags=0x40008 [21150.779530] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21150.781861] pending: addrconf_verify_work [21150.783267] workqueue knbd0-recv: flags=0x1a [21150.784634] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21150.786710] in-flight: 651:recv_work [21150.788019] workqueue ptlrpc_pinger: flags=0x2 [21150.789540] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21150.791513] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21150.793711] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18513s workers=4 idle: 15979 31824 8 [21150.796589] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21141s workers=2 idle: 654 [21156.060134] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [21156.063888] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21156.103295] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21156.121414] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21156.126719] RIP: 0010:native_safe_halt+0x2e/0x50 [21156.130361] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21156.148366] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21156.156007] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21156.166363] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21156.169930] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21156.173851] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21156.177715] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21156.185568] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21156.188448] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21156.190612] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21156.195861] Call Trace: [21156.201354] [21156.201976] ? show_regs.cold.9+0x22/0x2f [21156.203413] ? watchdog_timer_fn.cold.6+0x62/0x10a [21156.208348] ? __hrtimer_run_queues+0x1c2/0x4d0 [21156.215498] ? watchdog+0x40/0x40 [21156.223888] ? hrtimer_interrupt+0x128/0x2c0 [21156.227365] ? smp_apic_timer_interrupt+0xa6/0x270 [21156.231255] ? apic_timer_interrupt+0xf/0x20 [21156.232736] [21156.234205] ? native_safe_halt+0x2e/0x50 [21156.239832] kvm_wait+0xcd/0xe0 [21156.245534] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21156.249699] do_raw_spin_lock+0x177/0x190 [21156.254637] _raw_spin_lock+0x21/0x30 [21156.257332] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21156.262750] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21156.268217] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21156.272516] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21156.276309] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21156.279956] ? woken_wake_function+0x30/0x30 [21156.283600] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21156.291783] kthread+0x1d1/0x200 [21156.293006] ? set_kthread_struct+0x70/0x70 [21156.298552] ret_from_fork+0x1f/0x30 [21164.057361] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [21164.058570] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [21164.062598] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21164.062680] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21164.062689] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21164.062692] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21164.062714] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21164.062720] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21164.062726] RAX: 0000000000007f81 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21164.062729] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21164.062731] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21164.062734] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21164.062736] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21164.062740] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21164.062743] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21164.062745] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21164.062755] Call Trace: [21164.062760] [21164.062766] ? show_regs.cold.9+0x22/0x2f [21164.062775] ? watchdog_timer_fn.cold.6+0x62/0x10a [21164.062780] ? __hrtimer_run_queues+0x1c2/0x4d0 [21164.062785] ? watchdog+0x40/0x40 [21164.062789] ? hrtimer_interrupt+0x128/0x2c0 [21164.062793] ? smp_apic_timer_interrupt+0xa6/0x270 [21164.062802] ? apic_timer_interrupt+0xf/0x20 [21164.062807] [21164.062810] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21164.062821] do_raw_spin_lock+0x177/0x190 [21164.062830] _raw_spin_lock+0x21/0x30 [21164.062841] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21164.063321] ? do_raw_spin_unlock+0x75/0x190 [21164.063328] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21164.063552] ? do_wait_intr+0xf0/0xf0 [21164.063559] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21164.063752] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21164.069991] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21164.070074] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21164.070081] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21164.070084] RIP: 0010:native_safe_halt+0x2e/0x50 [21164.070215] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21164.070221] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21164.070227] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21164.070229] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21164.070231] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21164.070233] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21164.070236] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21164.070239] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21164.070242] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21164.070245] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21164.070253] Call Trace: [21164.070259] [21164.070265] ? show_regs.cold.9+0x22/0x2f [21164.070275] ? watchdog_timer_fn.cold.6+0x62/0x10a [21164.070282] ? __hrtimer_run_queues+0x1c2/0x4d0 [21164.070286] ? watchdog+0x40/0x40 [21164.070291] ? hrtimer_interrupt+0x128/0x2c0 [21164.070371] ? smp_apic_timer_interrupt+0xa6/0x270 [21164.070379] ? apic_timer_interrupt+0xf/0x20 [21164.070384] [21164.070387] ? native_safe_halt+0x2e/0x50 [21164.070391] kvm_wait+0xcd/0xe0 [21164.070399] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21164.070406] do_raw_spin_lock+0x177/0x190 [21164.070411] _raw_spin_lock+0x21/0x30 [21164.070417] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21164.071554] obd_mod_free+0x46/0x140 [ptlrpc] [21164.071847] mdc_commit_open+0x91/0xb0 [mdc] [21164.071898] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21164.098345] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21164.098653] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21164.098778] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21164.098819] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21164.098850] mdc_intent_lock+0x353/0x7e0 [mdc] [21164.098889] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21164.098979] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21164.099205] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21164.099235] lmv_intent_open+0x338/0x1150 [lmv] [21164.099264] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21164.099356] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21164.099443] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21164.099465] ll_intent_lock+0xc5/0xaa0 [lustre] [21164.099549] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21164.108701] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21164.113035] ll_lookup_it+0xe0e/0x2270 [lustre] [21164.113193] ? do_raw_spin_unlock+0x75/0x190 [21164.113202] ? _raw_spin_unlock+0x12/0x30 [21164.113209] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21164.113245] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21164.113272] ll_atomic_open+0x36a/0xd70 [lustre] [21164.113359] lookup_open+0x356/0xb40 [21164.113369] do_last+0x313/0xfc0 [21164.113374] ? nd_jump_root+0xe5/0x160 [21164.113379] ? path_init+0x437/0x520 [21164.113384] path_openat+0xf7/0x500 [21164.113389] ? do_raw_spin_unlock+0x75/0x190 [21164.113394] do_filp_open+0x99/0x140 [21164.113401] ? getname_flags+0x6e/0x330 [21164.113406] ? __check_object_size+0xff/0x256 [21164.113413] ? do_raw_spin_unlock+0x75/0x190 [21164.113417] ? _raw_spin_unlock+0x12/0x30 [21164.113423] do_sys_openat2+0x2b4/0x410 [21164.113507] do_sys_open+0x73/0xa0 [21164.113515] __x64_sys_openat+0x24/0x30 [21164.113519] do_syscall_64+0xc1/0x440 [21164.113531] entry_SYSCALL_64_after_hwframe+0x49/0xae [21164.113538] RIP: 0033:0x7f53425a5332 [21164.113596] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21164.113602] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21164.113608] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21164.113611] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21164.113613] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21164.113615] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21164.113618] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21164.624975] after_reply+0xe2c/0x1780 [ptlrpc] [21164.626825] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21164.634427] ? __next_timer_interrupt+0x160/0x160 [21164.639076] ptlrpcd+0xa82/0xe40 [ptlrpc] [21164.640714] ? do_wait_intr+0xf0/0xf0 [21164.643008] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21164.645461] kthread+0x1d1/0x200 [21164.646785] ? set_kthread_struct+0x70/0x70 [21164.648637] ret_from_fork+0x1f/0x30 [21176.052021] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [21176.056054] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21176.082610] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21176.088261] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21176.093268] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21176.096851] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21176.099861] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21176.108804] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21176.112087] RAX: 000000000000490b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21176.115174] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [21176.118143] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21176.121613] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21176.124902] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21176.128010] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21176.131766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21176.134144] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21176.137600] Call Trace: [21176.138775] [21176.139642] ? show_regs.cold.9+0x22/0x2f [21176.141601] ? watchdog_timer_fn.cold.6+0x62/0x10a [21176.144454] ? __hrtimer_run_queues+0x1c2/0x4d0 [21176.146954] ? watchdog+0x40/0x40 [21176.148876] ? hrtimer_interrupt+0x128/0x2c0 [21176.150867] ? smp_apic_timer_interrupt+0xa6/0x270 [21176.152296] ? apic_timer_interrupt+0xf/0x20 [21176.154020] [21176.155116] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21176.158136] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21176.161267] ? newidle_balance.isra.93+0x207/0x580 [21176.166015] do_raw_spin_lock+0x177/0x190 [21176.172043] _raw_spin_lock+0x21/0x30 [21176.173474] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21176.178030] process_one_work+0x2c8/0x700 [21176.180147] worker_thread+0x296/0x6e0 [21176.181518] ? rescuer_thread+0x570/0x570 [21176.183478] kthread+0x1d1/0x200 [21176.184979] ? set_kthread_struct+0x70/0x70 [21176.187074] ret_from_fork+0x1f/0x30 [21180.900460] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18543s! [21180.913459] Showing busy workqueues and worker pools: [21180.917110] workqueue events: flags=0x0 [21180.920942] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21180.925573] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21180.941686] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21180.948105] pending: cpu_latency_work [ptlrpc] [21180.954345] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21180.961782] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21180.966664] workqueue events_long: flags=0x0 [21180.970692] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21180.974492] pending: lu_objects_flush [obdclass] [21180.977075] workqueue events_unbound: flags=0x2 [21180.979084] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21180.981726] pending: flush_memcg_stats_dwork, idle_cull_fn [21180.987979] workqueue events_power_efficient: flags=0x80 [21180.989907] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21180.992096] pending: check_lifetime, do_cache_clean [21180.998180] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21181.001874] pending: neigh_periodic_work, neigh_periodic_work [21181.004798] workqueue mm_percpu_wq: flags=0x8 [21181.007450] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21181.013499] pending: vmstat_update, lru_add_drain_per_cpu [21181.018019] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21181.020958] pending: vmstat_update, lru_add_drain_per_cpu [21181.024890] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21181.030390] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21181.035546] workqueue xprtiod: flags=0xa [21181.036539] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21181.040111] pending: xprt_autoclose, xs_error_handle [21181.042147] workqueue ipv6_addrconf: flags=0x40008 [21181.044881] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21181.048990] pending: addrconf_verify_work [21181.056266] workqueue knbd0-recv: flags=0x1a [21181.058061] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21181.062949] in-flight: 651:recv_work [21181.065373] workqueue ptlrpc_pinger: flags=0x2 [21181.066946] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21181.070074] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21181.073506] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18543s workers=4 idle: 15979 31824 8 [21181.075980] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21171s workers=2 idle: 654 [21184.058263] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [21184.062506] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21184.089255] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21184.096576] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21184.101718] RIP: 0010:native_safe_halt+0x2e/0x50 [21184.105368] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21184.117576] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21184.122084] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21184.127595] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21184.131423] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21184.135014] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21184.139182] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21184.143983] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21184.149406] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21184.152673] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21184.158045] Call Trace: [21184.159743] [21184.161224] ? show_regs.cold.9+0x22/0x2f [21184.163480] ? watchdog_timer_fn.cold.6+0x62/0x10a [21184.166810] ? __hrtimer_run_queues+0x1c2/0x4d0 [21184.169309] ? watchdog+0x40/0x40 [21184.170419] ? hrtimer_interrupt+0x128/0x2c0 [21184.172675] ? smp_apic_timer_interrupt+0xa6/0x270 [21184.175346] ? apic_timer_interrupt+0xf/0x20 [21184.177660] [21184.178738] ? native_safe_halt+0x2e/0x50 [21184.181438] kvm_wait+0xcd/0xe0 [21184.183081] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21184.185843] do_raw_spin_lock+0x177/0x190 [21184.188481] _raw_spin_lock+0x21/0x30 [21184.190488] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21184.194414] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21184.198398] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21184.201755] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21184.211946] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21184.217405] ? woken_wake_function+0x30/0x30 [21184.221212] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21184.225225] kthread+0x1d1/0x200 [21184.227484] ? set_kthread_struct+0x70/0x70 [21184.230486] ret_from_fork+0x1f/0x30 [21192.055050] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [21192.057185] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [21192.063898] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21192.063984] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21192.063992] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21192.063996] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21192.064054] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21192.064061] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21192.064066] RAX: 000000000000391c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21192.064068] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21192.064072] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21192.064073] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21192.064075] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21192.064079] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21192.064082] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21192.064085] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21192.064093] Call Trace: [21192.064098] [21192.064105] ? show_regs.cold.9+0x22/0x2f [21192.064115] ? watchdog_timer_fn.cold.6+0x62/0x10a [21192.064122] ? __hrtimer_run_queues+0x1c2/0x4d0 [21192.064126] ? watchdog+0x40/0x40 [21192.073173] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21192.073263] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21192.073272] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21192.073275] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21192.110749] ? hrtimer_interrupt+0x128/0x2c0 [21192.110775] ? smp_apic_timer_interrupt+0xa6/0x270 [21192.110784] ? apic_timer_interrupt+0xf/0x20 [21192.110790] [21192.110795] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21192.110807] do_raw_spin_lock+0x177/0x190 [21192.110814] _raw_spin_lock+0x21/0x30 [21192.110820] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21192.111377] ? do_raw_spin_unlock+0x75/0x190 [21192.111499] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21192.120080] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21192.120100] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21192.120107] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21192.120113] RDX: 000000000000779b RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21192.120116] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21192.120119] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21192.120121] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21192.120124] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21192.120127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21192.120130] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21192.120138] Call Trace: [21192.120147] [21192.120152] ? show_regs.cold.9+0x22/0x2f [21192.120165] ? watchdog_timer_fn.cold.6+0x62/0x10a [21192.120171] ? __hrtimer_run_queues+0x1c2/0x4d0 [21192.120175] ? watchdog+0x40/0x40 [21192.120179] ? hrtimer_interrupt+0x128/0x2c0 [21192.120183] ? smp_apic_timer_interrupt+0xa6/0x270 [21192.120191] ? apic_timer_interrupt+0xf/0x20 [21192.120196] [21192.120199] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21192.120205] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21192.120209] do_raw_spin_lock+0x177/0x190 [21192.120214] _raw_spin_lock+0x21/0x30 [21192.120222] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21192.120863] obd_mod_free+0x46/0x140 [ptlrpc] [21192.121106] mdc_commit_open+0x91/0xb0 [mdc] [21192.121153] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21192.121598] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21192.121930] after_reply+0xe2c/0x1780 [ptlrpc] [21192.122139] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21192.128243] ? do_wait_intr+0xf0/0xf0 [21192.128269] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21192.128804] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21192.129030] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21192.129880] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21192.130077] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21192.130116] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21192.130141] mdc_intent_lock+0x353/0x7e0 [mdc] [21192.130165] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21192.130251] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21192.130642] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21192.130674] lmv_intent_open+0x338/0x1150 [lmv] [21192.130774] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21192.130864] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21192.130940] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21192.130960] ll_intent_lock+0xc5/0xaa0 [lustre] [21192.131038] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21192.131121] ll_lookup_it+0xe0e/0x2270 [lustre] [21192.131198] ? do_raw_spin_unlock+0x75/0x190 [21192.131205] ? _raw_spin_unlock+0x12/0x30 [21192.131212] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21192.131239] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21192.131256] ll_atomic_open+0x36a/0xd70 [lustre] [21192.131342] lookup_open+0x356/0xb40 [21192.131352] do_last+0x313/0xfc0 [21192.131357] ? nd_jump_root+0xe5/0x160 [21192.131361] ? path_init+0x437/0x520 [21192.131366] path_openat+0xf7/0x500 [21192.131371] ? do_raw_spin_unlock+0x75/0x190 [21192.131376] do_filp_open+0x99/0x140 [21192.131608] ? getname_flags+0x6e/0x330 [21192.131612] ? __check_object_size+0xff/0x256 [21192.131619] ? do_raw_spin_unlock+0x75/0x190 [21192.131622] ? _raw_spin_unlock+0x12/0x30 [21192.131626] do_sys_openat2+0x2b4/0x410 [21192.131630] do_sys_open+0x73/0xa0 [21192.131633] __x64_sys_openat+0x24/0x30 [21192.131636] do_syscall_64+0xc1/0x440 [21192.131642] entry_SYSCALL_64_after_hwframe+0x49/0xae [21192.131647] RIP: 0033:0x7f53425a5332 [21192.131654] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21192.131657] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21192.131662] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21192.131664] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21192.131666] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21192.131668] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21192.131670] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21192.425913] ? __next_timer_interrupt+0x160/0x160 [21192.430131] ptlrpcd+0xa82/0xe40 [ptlrpc] [21192.433567] ? do_wait_intr+0xf0/0xf0 [21192.435115] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21192.437905] kthread+0x1d1/0x200 [21192.439002] ? set_kthread_struct+0x70/0x70 [21192.441097] ret_from_fork+0x1f/0x30 [21208.052610] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [21208.058606] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21208.101469] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21208.114333] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21208.120592] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21208.130359] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21208.137374] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21208.157455] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21208.167464] RAX: 0000000000005615 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21208.174321] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [21208.189702] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21208.199258] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21208.203733] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21208.207950] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21208.212604] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21208.215018] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21208.218763] Call Trace: [21208.219584] [21208.222399] ? show_regs.cold.9+0x22/0x2f [21208.223797] ? watchdog_timer_fn.cold.6+0x62/0x10a [21208.231347] ? __hrtimer_run_queues+0x1c2/0x4d0 [21208.239623] ? watchdog+0x40/0x40 [21208.240827] ? hrtimer_interrupt+0x128/0x2c0 [21208.247934] ? smp_apic_timer_interrupt+0xa6/0x270 [21208.251430] ? apic_timer_interrupt+0xf/0x20 [21208.252781] [21208.254350] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21208.257188] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21208.261795] ? newidle_balance.isra.93+0x207/0x580 [21208.266837] do_raw_spin_lock+0x177/0x190 [21208.269977] _raw_spin_lock+0x21/0x30 [21208.277841] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21208.286548] process_one_work+0x2c8/0x700 [21208.288283] worker_thread+0x296/0x6e0 [21208.289594] ? rescuer_thread+0x570/0x570 [21208.292569] kthread+0x1d1/0x200 [21208.294468] ? set_kthread_struct+0x70/0x70 [21208.302998] ret_from_fork+0x1f/0x30 [21211.107374] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18574s! [21211.112440] Showing busy workqueues and worker pools: [21211.114744] workqueue events: flags=0x0 [21211.116427] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21211.119528] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21211.124734] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21211.127851] pending: cpu_latency_work [ptlrpc] [21211.130328] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21211.132467] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21211.136010] workqueue events_long: flags=0x0 [21211.137906] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21211.140487] pending: lu_objects_flush [obdclass] [21211.149671] workqueue events_unbound: flags=0x2 [21211.151169] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21211.160953] pending: flush_memcg_stats_dwork, idle_cull_fn [21211.171008] workqueue events_power_efficient: flags=0x80 [21211.179838] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21211.190482] pending: check_lifetime, do_cache_clean [21211.199363] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21211.203566] pending: neigh_periodic_work, neigh_periodic_work [21211.210477] workqueue mm_percpu_wq: flags=0x8 [21211.212582] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21211.217005] pending: vmstat_update, lru_add_drain_per_cpu [21211.220470] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21211.225506] pending: vmstat_update, lru_add_drain_per_cpu [21211.230340] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21211.234564] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21211.242351] workqueue xprtiod: flags=0xa [21211.243629] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21211.251469] pending: xprt_autoclose, xs_error_handle [21211.256866] workqueue ipv6_addrconf: flags=0x40008 [21211.259540] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21211.264874] pending: addrconf_verify_work [21211.266409] workqueue knbd0-recv: flags=0x1a [21211.269638] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21211.272769] in-flight: 651:recv_work [21211.275215] workqueue ptlrpc_pinger: flags=0x2 [21211.279422] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21211.282928] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21211.288879] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18574s workers=4 idle: 15979 31824 8 [21211.291740] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21201s workers=2 idle: 654 [21212.060445] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [21212.064445] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21212.089671] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21212.094287] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21212.097981] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21212.100665] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21212.108165] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [21212.111670] RAX: 0000000000007f92 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21212.114875] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21212.117557] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21212.124077] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21212.128514] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21212.131003] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21212.138917] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21212.148679] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21212.160501] Call Trace: [21212.163736] [21212.167511] ? show_regs.cold.9+0x22/0x2f [21212.172720] ? watchdog_timer_fn.cold.6+0x62/0x10a [21212.177475] ? __hrtimer_run_queues+0x1c2/0x4d0 [21212.186192] ? watchdog+0x40/0x40 [21212.188905] ? hrtimer_interrupt+0x128/0x2c0 [21212.191175] ? smp_apic_timer_interrupt+0xa6/0x270 [21212.192933] ? apic_timer_interrupt+0xf/0x20 [21212.194365] [21212.195304] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21212.198854] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21212.201902] do_raw_spin_lock+0x177/0x190 [21212.204283] _raw_spin_lock+0x21/0x30 [21212.206899] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21212.215568] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21212.217476] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21212.221740] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21212.227823] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21212.231430] ? woken_wake_function+0x30/0x30 [21212.234500] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21212.242929] kthread+0x1d1/0x200 [21212.245629] ? set_kthread_struct+0x70/0x70 [21212.247727] ret_from_fork+0x1f/0x30 [21220.055217] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [21220.057575] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [21220.058841] Modules linked in: [21220.062458] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21220.062611] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21220.062619] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21220.062622] RIP: 0010:native_safe_halt+0x2e/0x50 [21220.062635] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21220.062639] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21220.062644] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21220.062647] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21220.062649] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21220.062651] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21220.062654] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21220.062657] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21220.062660] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21220.062663] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21220.062671] Call Trace: [21220.062677] [21220.062683] ? show_regs.cold.9+0x22/0x2f [21220.062693] ? watchdog_timer_fn.cold.6+0x62/0x10a [21220.062699] ? __hrtimer_run_queues+0x1c2/0x4d0 [21220.062705] ? watchdog+0x40/0x40 [21220.062709] ? hrtimer_interrupt+0x128/0x2c0 [21220.062713] ? smp_apic_timer_interrupt+0xa6/0x270 [21220.062720] ? apic_timer_interrupt+0xf/0x20 [21220.062724] [21220.062727] ? native_safe_halt+0x2e/0x50 [21220.062730] kvm_wait+0xcd/0xe0 [21220.062739] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21220.062747] do_raw_spin_lock+0x177/0x190 [21220.062751] _raw_spin_lock+0x21/0x30 [21220.062757] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21220.063149] obd_mod_free+0x46/0x140 [ptlrpc] [21220.063353] mdc_commit_open+0x91/0xb0 [mdc] [21220.063455] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21220.065647] lustre(O) [21220.081361] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21220.087475] osp(O) [21220.090521] after_reply+0xe2c/0x1780 [ptlrpc] [21220.092336] ofd(O) [21220.099270] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21220.103545] lod(O) [21220.105989] ? __next_timer_interrupt+0x160/0x160 [21220.110702] mdt(O) [21220.113427] ptlrpcd+0xa82/0xe40 [ptlrpc] [21220.118697] mdd(O) [21220.121283] ? do_wait_intr+0xf0/0xf0 [21220.125000] mgs(O) [21220.127115] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21220.130556] lquota(O) [21220.131491] kthread+0x1d1/0x200 [21220.132577] lfsck(O) [21220.134271] ? set_kthread_struct+0x70/0x70 [21220.134285] ret_from_fork+0x1f/0x30 [21220.222573] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21220.239257] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21220.244386] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21220.248291] RIP: 0010:native_safe_halt+0x2e/0x50 [21220.250210] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21220.259940] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21220.263372] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21220.266127] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21220.269199] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21220.271982] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21220.273929] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21220.276550] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21220.279674] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21220.282091] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21220.284974] Call Trace: [21220.286158] [21220.286886] ? show_regs.cold.9+0x22/0x2f [21220.288667] ? watchdog_timer_fn.cold.6+0x62/0x10a [21220.290552] ? __hrtimer_run_queues+0x1c2/0x4d0 [21220.292326] ? watchdog+0x40/0x40 [21220.293666] ? hrtimer_interrupt+0x128/0x2c0 [21220.295535] ? smp_apic_timer_interrupt+0xa6/0x270 [21220.297621] ? apic_timer_interrupt+0xf/0x20 [21220.299008] [21220.299510] ? native_safe_halt+0x2e/0x50 [21220.300944] kvm_wait+0xcd/0xe0 [21220.302301] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21220.304669] do_raw_spin_lock+0x177/0x190 [21220.306169] _raw_spin_lock+0x21/0x30 [21220.307965] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21220.310366] ? do_raw_spin_unlock+0x75/0x190 [21220.312010] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21220.314435] ? do_wait_intr+0xf0/0xf0 [21220.315696] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21220.317819] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21220.320258] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21220.323600] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21220.326049] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21220.328347] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21220.330316] mdc_intent_lock+0x353/0x7e0 [mdc] [21220.331975] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21220.334253] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21220.337220] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21220.340112] lmv_intent_open+0x338/0x1150 [lmv] [21220.345437] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21220.357278] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21220.360460] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21220.362302] ll_intent_lock+0xc5/0xaa0 [lustre] [21220.365454] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21220.369955] ll_lookup_it+0xe0e/0x2270 [lustre] [21220.371876] ? do_raw_spin_unlock+0x75/0x190 [21220.374528] ? _raw_spin_unlock+0x12/0x30 [21220.376349] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21220.378560] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21220.380856] ll_atomic_open+0x36a/0xd70 [lustre] [21220.383230] lookup_open+0x356/0xb40 [21220.384675] do_last+0x313/0xfc0 [21220.386167] ? nd_jump_root+0xe5/0x160 [21220.387788] ? path_init+0x437/0x520 [21220.389379] path_openat+0xf7/0x500 [21220.391144] ? do_raw_spin_unlock+0x75/0x190 [21220.393072] do_filp_open+0x99/0x140 [21220.394516] ? getname_flags+0x6e/0x330 [21220.397367] ? __check_object_size+0xff/0x256 [21220.399541] ? do_raw_spin_unlock+0x75/0x190 [21220.401373] ? _raw_spin_unlock+0x12/0x30 [21220.403195] do_sys_openat2+0x2b4/0x410 [21220.405007] do_sys_open+0x73/0xa0 [21220.406950] __x64_sys_openat+0x24/0x30 [21220.408674] do_syscall_64+0xc1/0x440 [21220.410284] entry_SYSCALL_64_after_hwframe+0x49/0xae [21220.412548] RIP: 0033:0x7f53425a5332 [21220.414105] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21220.422374] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21220.425368] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21220.428536] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21220.431724] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21220.435154] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21220.438305] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21234.900610] rcu: INFO: rcu_sched self-detected stall on CPU [21234.906567] rcu: 2-....: (11018329 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [21234.912024] (t=19140474 jiffies g=274185 q=68745) [21234.917300] rcu: rcu_sched kthread starved for 18598840 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [21234.925488] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [21234.937179] rcu: RCU grace-period kthread stack dump: [21234.941841] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [21234.952406] Call Trace: [21234.953546] __schedule+0x351/0xcb0 [21234.959313] schedule+0xc0/0x180 [21234.961758] schedule_timeout+0xb4/0x190 [21234.963481] ? do_raw_spin_unlock+0x75/0x190 [21234.969169] ? __next_timer_interrupt+0x160/0x160 [21234.971543] ? prepare_to_swait_event+0xf3/0x140 [21234.977155] rcu_gp_kthread+0x8dd/0x15b0 [21234.979469] ? force_qs_rnp+0x2c0/0x2c0 [21234.985319] kthread+0x1d1/0x200 [21234.991330] ? set_kthread_struct+0x70/0x70 [21234.995819] ret_from_fork+0x1f/0x30 [21235.000104] rcu: Stack dump where RCU GP kthread last ran: [21235.008700] NMI backtrace for cpu 2 [21235.013072] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21235.025576] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21235.036226] Call Trace: [21235.039069] [21235.039743] ? dump_stack+0xbb/0x10e [21235.046538] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [21235.055277] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [21235.063758] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [21235.069220] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [21235.072918] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [21235.079211] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [21235.086127] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [21235.088251] ? update_process_times+0x6d/0xc0 [21235.092303] ? tick_sched_handle.isra.18+0x27/0x90 [21235.095160] ? tick_sched_timer+0x53/0xb0 [21235.097766] ? __hrtimer_run_queues+0x1c2/0x4d0 [21235.100717] ? tick_sched_handle.isra.18+0x90/0x90 [21235.102909] ? hrtimer_interrupt+0x128/0x2c0 [21235.105285] ? smp_apic_timer_interrupt+0xa6/0x270 [21235.110975] ? apic_timer_interrupt+0xf/0x20 [21235.114651] [21235.115272] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21235.117009] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21235.121560] ? do_raw_spin_lock+0x177/0x190 [21235.132578] ? _raw_spin_lock+0x21/0x30 [21235.139487] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21235.143750] ? obd_mod_free+0x46/0x140 [ptlrpc] [21235.154207] ? mdc_commit_open+0x91/0xb0 [mdc] [21235.159118] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21235.164466] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21235.170713] ? after_reply+0xe2c/0x1780 [ptlrpc] [21235.176834] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21235.180215] ? __next_timer_interrupt+0x160/0x160 [21235.184642] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [21235.187927] ? do_wait_intr+0xf0/0xf0 [21235.190905] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21235.194428] ? kthread+0x1d1/0x200 [21235.196096] ? set_kthread_struct+0x70/0x70 [21235.198147] ? ret_from_fork+0x1f/0x30 [21235.200481] Sending NMI from CPU 2 to CPUs 1: [21235.204045] NMI backtrace for cpu 1 [21235.204051] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21235.204053] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21235.204054] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21235.204056] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21235.204058] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [21235.204061] RAX: 0000000000003fb8 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21235.204062] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21235.204064] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21235.204065] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21235.204067] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21235.204068] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21235.204070] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21235.204071] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21235.204072] Call Trace: [21235.204073] [21235.204074] ? show_regs.cold.9+0x22/0x2f [21235.204075] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [21235.204076] ? nmi_cpu_backtrace_handler+0x15/0x40 [21235.204077] ? nmi_handle+0xbd/0x210 [21235.204078] ? default_do_nmi+0x4b/0x1b0 [21235.204079] ? do_nmi+0x280/0x380 [21235.204080] ? end_repeat_nmi+0x16/0x1f [21235.204082] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21235.204083] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21235.204084] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21235.204086] [21235.204087] do_raw_spin_lock+0x177/0x190 [21235.204088] _raw_spin_lock+0x21/0x30 [21235.204089] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21235.204090] ? do_raw_spin_unlock+0x75/0x190 [21235.204091] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21235.204092] ? do_wait_intr+0xf0/0xf0 [21235.204093] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21235.204095] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21235.204096] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21235.204097] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21235.204098] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21235.204100] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21235.204100] mdc_intent_lock+0x353/0x7e0 [mdc] [21235.204102] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21235.204103] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21235.204105] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21235.204106] lmv_intent_open+0x338/0x1150 [lmv] [21235.204107] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21235.204109] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21235.204110] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21235.204111] ll_intent_lock+0xc5/0xaa0 [lustre] [21235.204112] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21235.204113] ll_lookup_it+0xe0e/0x2270 [lustre] [21235.204114] ? do_raw_spin_unlock+0x75/0x190 [21235.204115] ? _raw_spin_unlock+0x12/0x30 [21235.204117] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21235.204118] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21235.204119] ll_atomic_open+0x36a/0xd70 [lustre] [21235.204120] lookup_open+0x356/0xb40 [21235.204121] do_last+0x313/0xfc0 [21235.204122] ? nd_jump_root+0xe5/0x160 [21235.204123] ? path_init+0x437/0x520 [21235.204124] path_openat+0xf7/0x500 [21235.204125] ? do_raw_spin_unlock+0x75/0x190 [21235.204127] do_filp_open+0x99/0x140 [21235.204128] ? getname_flags+0x6e/0x330 [21235.204129] ? __check_object_size+0xff/0x256 [21235.204130] ? do_raw_spin_unlock+0x75/0x190 [21235.204131] ? _raw_spin_unlock+0x12/0x30 [21235.204133] do_sys_openat2+0x2b4/0x410 [21235.204134] do_sys_open+0x73/0xa0 [21235.204135] __x64_sys_openat+0x24/0x30 [21235.204136] do_syscall_64+0xc1/0x440 [21235.204138] entry_SYSCALL_64_after_hwframe+0x49/0xae [21235.204139] RIP: 0033:0x7f53425a5332 [21235.204140] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21235.204142] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21235.204145] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21235.204147] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21235.204148] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21235.204150] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21235.204151] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21235.206054] NMI backtrace for cpu 2 [21235.578868] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21235.582882] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21235.585994] Call Trace: [21235.586949] [21235.587682] ? dump_stack+0xbb/0x10e [21235.589097] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [21235.591040] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [21235.593366] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [21235.596781] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [21235.599894] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [21235.604010] ? rcu_dump_cpu_stacks+0x11b/0x171 [21235.606608] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [21235.611213] ? update_process_times+0x6d/0xc0 [21235.615033] ? tick_sched_handle.isra.18+0x27/0x90 [21235.618003] ? tick_sched_timer+0x53/0xb0 [21235.619599] ? __hrtimer_run_queues+0x1c2/0x4d0 [21235.621137] ? tick_sched_handle.isra.18+0x90/0x90 [21235.624590] ? hrtimer_interrupt+0x128/0x2c0 [21235.628315] ? smp_apic_timer_interrupt+0xa6/0x270 [21235.630105] ? apic_timer_interrupt+0xf/0x20 [21235.631795] [21235.632428] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21235.634383] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21235.636418] ? do_raw_spin_lock+0x177/0x190 [21235.637544] ? _raw_spin_lock+0x21/0x30 [21235.638609] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21235.640394] ? obd_mod_free+0x46/0x140 [ptlrpc] [21235.641871] ? mdc_commit_open+0x91/0xb0 [mdc] [21235.643112] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21235.644812] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21235.646328] ? after_reply+0xe2c/0x1780 [ptlrpc] [21235.648058] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21235.649928] ? __next_timer_interrupt+0x160/0x160 [21235.651209] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [21235.652794] ? do_wait_intr+0xf0/0xf0 [21235.653575] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21235.656249] ? kthread+0x1d1/0x200 [21235.657127] ? set_kthread_struct+0x70/0x70 [21235.658266] ? ret_from_fork+0x1f/0x30 [21236.052275] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [21236.055786] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21236.085203] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21236.097141] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21236.102866] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21236.114038] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21236.116174] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21236.131287] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21236.135315] RAX: 0000000000005b89 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21236.141398] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [21236.146929] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21236.150255] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21236.154092] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21236.157301] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21236.160510] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21236.166259] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21236.174023] Call Trace: [21236.176194] [21236.179355] ? show_regs.cold.9+0x22/0x2f [21236.183528] ? watchdog_timer_fn.cold.6+0x62/0x10a [21236.186803] ? __hrtimer_run_queues+0x1c2/0x4d0 [21236.189287] ? watchdog+0x40/0x40 [21236.192480] ? hrtimer_interrupt+0x128/0x2c0 [21236.195377] ? smp_apic_timer_interrupt+0xa6/0x270 [21236.201245] ? apic_timer_interrupt+0xf/0x20 [21236.206144] [21236.207237] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21236.209221] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21236.213723] ? newidle_balance.isra.93+0x207/0x580 [21236.215400] do_raw_spin_lock+0x177/0x190 [21236.219136] _raw_spin_lock+0x21/0x30 [21236.220442] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21236.222811] process_one_work+0x2c8/0x700 [21236.225598] worker_thread+0x296/0x6e0 [21236.232858] ? rescuer_thread+0x570/0x570 [21236.236239] kthread+0x1d1/0x200 [21236.242942] ? set_kthread_struct+0x70/0x70 [21236.247085] ret_from_fork+0x1f/0x30 [21240.058022] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [21240.062291] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21240.123470] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21240.141529] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21240.166819] RIP: 0010:native_safe_halt+0x2e/0x50 [21240.177238] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21240.214946] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21240.224088] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21240.237353] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21240.245296] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21240.261395] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21240.269458] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21240.281729] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21240.293667] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21240.301328] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21240.318167] Call Trace: [21240.321155] [21240.323516] ? show_regs.cold.9+0x22/0x2f [21240.330080] ? watchdog_timer_fn.cold.6+0x62/0x10a [21240.336477] ? __hrtimer_run_queues+0x1c2/0x4d0 [21240.342323] ? watchdog+0x40/0x40 [21240.347256] ? hrtimer_interrupt+0x128/0x2c0 [21240.352164] ? smp_apic_timer_interrupt+0xa6/0x270 [21240.361491] ? apic_timer_interrupt+0xf/0x20 [21240.367584] [21240.371811] ? native_safe_halt+0x2e/0x50 [21240.377400] kvm_wait+0xcd/0xe0 [21240.385651] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21240.395783] do_raw_spin_lock+0x177/0x190 [21240.402805] _raw_spin_lock+0x21/0x30 [21240.405264] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21240.409685] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21240.420921] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21240.431525] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21240.433709] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21240.447979] ? woken_wake_function+0x30/0x30 [21240.451103] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21240.461120] kthread+0x1d1/0x200 [21240.468491] ? set_kthread_struct+0x70/0x70 [21240.478207] ret_from_fork+0x1f/0x30 [21241.311503] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18604s! [21241.320569] Showing busy workqueues and worker pools: [21241.323559] workqueue events: flags=0x0 [21241.325976] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21241.333211] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21241.341244] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21241.344682] pending: cpu_latency_work [ptlrpc] [21241.346828] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21241.349734] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21241.354261] workqueue events_long: flags=0x0 [21241.356117] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21241.359213] pending: lu_objects_flush [obdclass] [21241.361777] workqueue events_unbound: flags=0x2 [21241.363668] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21241.365942] pending: flush_memcg_stats_dwork, idle_cull_fn [21241.368062] workqueue events_power_efficient: flags=0x80 [21241.369645] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21241.372412] pending: check_lifetime, do_cache_clean [21241.374726] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21241.377516] pending: neigh_periodic_work, neigh_periodic_work [21241.380044] workqueue mm_percpu_wq: flags=0x8 [21241.381441] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21241.384228] pending: vmstat_update, lru_add_drain_per_cpu [21241.386630] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21241.388750] pending: vmstat_update, lru_add_drain_per_cpu [21241.391238] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21241.393960] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21241.398900] workqueue xprtiod: flags=0xa [21241.404287] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21241.408058] pending: xprt_autoclose, xs_error_handle [21241.416182] workqueue ipv6_addrconf: flags=0x40008 [21241.422127] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21241.428545] pending: addrconf_verify_work [21241.430553] workqueue knbd0-recv: flags=0x1a [21241.435748] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21241.440633] in-flight: 651:recv_work [21241.442597] workqueue ptlrpc_pinger: flags=0x2 [21241.444875] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21241.448855] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21241.452241] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18604s workers=4 idle: 15979 31824 8 [21241.456120] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21231s workers=2 idle: 654 [21260.055279] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [21260.059048] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [21260.060650] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21260.060743] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21260.060750] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21260.060753] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21260.060768] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21260.060773] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21260.060778] RAX: 0000000000001b7e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21260.060781] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21260.060783] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21260.060785] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21260.060788] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21260.060792] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21260.060795] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21260.060797] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21260.060806] Call Trace: [21260.060812] [21260.060819] ? show_regs.cold.9+0x22/0x2f [21260.060828] ? watchdog_timer_fn.cold.6+0x62/0x10a [21260.060835] ? __hrtimer_run_queues+0x1c2/0x4d0 [21260.060840] ? watchdog+0x40/0x40 [21260.060844] ? hrtimer_interrupt+0x128/0x2c0 [21260.060849] ? smp_apic_timer_interrupt+0xa6/0x270 [21260.060857] ? apic_timer_interrupt+0xf/0x20 [21260.060862] [21260.060865] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21260.060873] do_raw_spin_lock+0x177/0x190 [21260.060878] _raw_spin_lock+0x21/0x30 [21260.060883] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21260.061804] ? do_raw_spin_unlock+0x75/0x190 [21260.061811] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21260.062038] ? do_wait_intr+0xf0/0xf0 [21260.062048] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21260.062514] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21260.062980] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21260.063287] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21260.063684] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21260.063794] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21260.063825] mdc_intent_lock+0x353/0x7e0 [mdc] [21260.063852] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21260.064147] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21260.064671] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21260.064709] lmv_intent_open+0x338/0x1150 [lmv] [21260.069590] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata [21260.096990] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21260.097273] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21260.097364] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21260.097395] ll_intent_lock+0xc5/0xaa0 [lustre] [21260.097697] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21260.097783] ll_lookup_it+0xe0e/0x2270 [lustre] [21260.097870] ? do_raw_spin_unlock+0x75/0x190 [21260.097878] ? _raw_spin_unlock+0x12/0x30 [21260.097886] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21260.097973] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21260.097994] ll_atomic_open+0x36a/0xd70 [lustre] [21260.098164] lookup_open+0x356/0xb40 [21260.098178] do_last+0x313/0xfc0 [21260.098184] ? nd_jump_root+0xe5/0x160 [21260.098332] ? path_init+0x437/0x520 [21260.098338] path_openat+0xf7/0x500 [21260.098345] ? do_raw_spin_unlock+0x75/0x190 [21260.098350] do_filp_open+0x99/0x140 [21260.098356] ? getname_flags+0x6e/0x330 [21260.098470] ? __check_object_size+0xff/0x256 [21260.104567] dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21260.104589] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21260.104596] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21260.104601] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21260.110177] ? do_raw_spin_unlock+0x75/0x190 [21260.112946] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21260.124023] ? _raw_spin_unlock+0x12/0x30 [21260.124042] do_sys_openat2+0x2b4/0x410 [21260.124050] do_sys_open+0x73/0xa0 [21260.124055] __x64_sys_openat+0x24/0x30 [21260.124059] do_syscall_64+0xc1/0x440 [21260.124066] entry_SYSCALL_64_after_hwframe+0x49/0xae [21260.124137] RIP: 0033:0x7f53425a5332 [21260.124145] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21260.124149] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21260.124155] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21260.124158] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21260.124160] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21260.124163] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21260.124165] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21260.483930] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21260.489029] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21260.491773] RDX: 000000000000084b RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21260.495014] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21260.506124] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21260.514132] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21260.517161] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21260.523637] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21260.526021] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21260.537070] Call Trace: [21260.538496] [21260.539097] ? show_regs.cold.9+0x22/0x2f [21260.540528] ? watchdog_timer_fn.cold.6+0x62/0x10a [21260.546204] ? __hrtimer_run_queues+0x1c2/0x4d0 [21260.551071] ? watchdog+0x40/0x40 [21260.552558] ? hrtimer_interrupt+0x128/0x2c0 [21260.555196] ? smp_apic_timer_interrupt+0xa6/0x270 [21260.561074] ? apic_timer_interrupt+0xf/0x20 [21260.564399] [21260.565571] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21260.567809] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21260.579070] do_raw_spin_lock+0x177/0x190 [21260.583612] _raw_spin_lock+0x21/0x30 [21260.585091] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21260.588533] obd_mod_free+0x46/0x140 [ptlrpc] [21260.591380] mdc_commit_open+0x91/0xb0 [mdc] [21260.596478] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21260.601134] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21260.604749] after_reply+0xe2c/0x1780 [ptlrpc] [21260.609962] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21260.613708] ? __next_timer_interrupt+0x160/0x160 [21260.615992] ptlrpcd+0xa82/0xe40 [ptlrpc] [21260.618673] ? do_wait_intr+0xf0/0xf0 [21260.620518] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21260.622934] kthread+0x1d1/0x200 [21260.626075] ? set_kthread_struct+0x70/0x70 [21260.627939] ret_from_fork+0x1f/0x30 [21268.052026] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [21268.058037] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [21268.058567] Modules linked in: lustre(O) [21268.063820] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21268.063905] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21268.063912] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21268.063915] RIP: 0010:native_safe_halt+0x2e/0x50 [21268.063928] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21268.063936] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21268.063940] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21268.063942] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21268.063945] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21268.063947] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21268.063950] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21268.063953] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21268.063956] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21268.063963] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21268.063970] Call Trace: [21268.063976] [21268.063982] ? show_regs.cold.9+0x22/0x2f [21268.063997] ? watchdog_timer_fn.cold.6+0x62/0x10a [21268.064004] ? __hrtimer_run_queues+0x1c2/0x4d0 [21268.064010] ? watchdog+0x40/0x40 [21268.064014] ? hrtimer_interrupt+0x128/0x2c0 [21268.064019] ? smp_apic_timer_interrupt+0xa6/0x270 [21268.064026] ? apic_timer_interrupt+0xf/0x20 [21268.064031] [21268.064034] ? native_safe_halt+0x2e/0x50 [21268.064039] kvm_wait+0xcd/0xe0 [21268.064047] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21268.064067] do_raw_spin_lock+0x177/0x190 [21268.064072] _raw_spin_lock+0x21/0x30 [21268.064076] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21268.064371] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21268.064561] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21268.064769] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21268.064979] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21268.065520] ? woken_wake_function+0x30/0x30 [21268.065529] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21268.065854] kthread+0x1d1/0x200 [21268.065861] ? set_kthread_struct+0x70/0x70 [21268.065864] ret_from_fork+0x1f/0x30 [21268.203809] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21268.222916] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21268.229304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21268.232803] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21268.239228] RIP: 0010:native_safe_halt+0x2e/0x50 [21268.240856] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21268.254272] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [21268.257802] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21268.262522] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [21268.266321] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21268.270835] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21268.273470] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21268.279457] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21268.282873] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21268.285348] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21268.288343] Call Trace: [21268.289427] [21268.290308] ? show_regs.cold.9+0x22/0x2f [21268.291843] ? watchdog_timer_fn.cold.6+0x62/0x10a [21268.301883] ? __hrtimer_run_queues+0x1c2/0x4d0 [21268.303562] ? watchdog+0x40/0x40 [21268.308115] ? hrtimer_interrupt+0x128/0x2c0 [21268.311329] ? smp_apic_timer_interrupt+0xa6/0x270 [21268.313403] ? apic_timer_interrupt+0xf/0x20 [21268.315327] [21268.316267] ? native_safe_halt+0x2e/0x50 [21268.317797] kvm_wait+0xcd/0xe0 [21268.322479] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21268.324709] ? newidle_balance.isra.93+0x207/0x580 [21268.328315] do_raw_spin_lock+0x177/0x190 [21268.330762] _raw_spin_lock+0x21/0x30 [21268.333503] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21268.336332] process_one_work+0x2c8/0x700 [21268.338158] worker_thread+0x296/0x6e0 [21268.340192] ? rescuer_thread+0x570/0x570 [21268.342249] kthread+0x1d1/0x200 [21268.343990] ? set_kthread_struct+0x70/0x70 [21268.346255] ret_from_fork+0x1f/0x30 [21271.520108] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18634s! [21271.524017] Showing busy workqueues and worker pools: [21271.529207] workqueue events: flags=0x0 [21271.536704] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21271.545532] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21271.555464] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21271.563532] pending: cpu_latency_work [ptlrpc] [21271.569955] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21271.572184] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21271.591529] workqueue events_long: flags=0x0 [21271.594630] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21271.599475] pending: lu_objects_flush [obdclass] [21271.602527] workqueue events_unbound: flags=0x2 [21271.606335] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21271.609564] pending: flush_memcg_stats_dwork, idle_cull_fn [21271.621072] workqueue events_power_efficient: flags=0x80 [21271.623327] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21271.628371] pending: check_lifetime, do_cache_clean [21271.630173] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21271.645140] pending: neigh_periodic_work, neigh_periodic_work [21271.649562] workqueue mm_percpu_wq: flags=0x8 [21271.651693] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21271.655206] pending: vmstat_update, lru_add_drain_per_cpu [21271.658218] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21271.660508] pending: vmstat_update, lru_add_drain_per_cpu [21271.664226] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21271.666937] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21271.673341] workqueue xprtiod: flags=0xa [21271.689970] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21271.692187] pending: xprt_autoclose, xs_error_handle [21271.704160] workqueue ipv6_addrconf: flags=0x40008 [21271.709463] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21271.712167] pending: addrconf_verify_work [21271.713811] workqueue knbd0-recv: flags=0x1a [21271.715597] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21271.717976] in-flight: 651:recv_work [21271.719370] workqueue ptlrpc_pinger: flags=0x2 [21271.724924] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21271.727337] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21271.729394] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18634s workers=4 idle: 15979 31824 8 [21271.731661] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21262s workers=2 idle: 654 [21288.055157] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [21288.058805] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [21288.065997] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21288.066075] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21288.066082] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21288.066085] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [21288.066101] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [21288.066105] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21288.066109] RAX: 0000000000004e5f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21288.066111] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21288.066114] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21288.066116] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21288.066118] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21288.066122] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21288.066125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21288.066128] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21288.066136] Call Trace: [21288.066141] [21288.066147] ? show_regs.cold.9+0x22/0x2f [21288.066158] ? watchdog_timer_fn.cold.6+0x62/0x10a [21288.066164] ? __hrtimer_run_queues+0x1c2/0x4d0 [21288.066168] ? watchdog+0x40/0x40 [21288.066172] ? hrtimer_interrupt+0x128/0x2c0 [21288.066176] ? smp_apic_timer_interrupt+0xa6/0x270 [21288.066184] ? apic_timer_interrupt+0xf/0x20 [21288.066190] [21288.066192] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [21288.066199] do_raw_spin_lock+0x177/0x190 [21288.066205] _raw_spin_lock+0x21/0x30 [21288.066210] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21288.066479] ? do_raw_spin_unlock+0x75/0x190 [21288.066485] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21288.067005] ? do_wait_intr+0xf0/0xf0 [21288.067016] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21288.067211] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21288.067424] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21288.067633] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21288.071168] Modules linked in: [21288.111296] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21288.121152] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21288.121882] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21288.121890] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21288.121893] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21288.121907] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21288.121911] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21288.121915] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21288.121918] RDX: 00000000000014c9 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21288.121920] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21288.121923] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21288.121925] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21288.121929] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21288.121931] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21288.121934] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21288.121943] Call Trace: [21288.121948] [21288.121954] ? show_regs.cold.9+0x22/0x2f [21288.121963] ? watchdog_timer_fn.cold.6+0x62/0x10a [21288.121969] ? __hrtimer_run_queues+0x1c2/0x4d0 [21288.121973] ? watchdog+0x40/0x40 [21288.121978] ? hrtimer_interrupt+0x128/0x2c0 [21288.121982] ? smp_apic_timer_interrupt+0xa6/0x270 [21288.121990] ? apic_timer_interrupt+0xf/0x20 [21288.121994] [21288.121998] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21288.122003] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21288.122008] do_raw_spin_lock+0x177/0x190 [21288.122014] _raw_spin_lock+0x21/0x30 [21288.122019] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21288.122603] obd_mod_free+0x46/0x140 [ptlrpc] [21288.130036] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21288.132388] mdc_commit_open+0x91/0xb0 [mdc] [21288.150683] mdc_intent_lock+0x353/0x7e0 [mdc] [21288.152759] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21288.152990] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21288.153151] after_reply+0xe2c/0x1780 [ptlrpc] [21288.153295] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21288.153470] ? __next_timer_interrupt+0x160/0x160 [21288.153478] ptlrpcd+0xa82/0xe40 [ptlrpc] [21288.153618] ? do_wait_intr+0xf0/0xf0 [21288.153627] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21288.153840] kthread+0x1d1/0x200 [21288.153847] ? set_kthread_struct+0x70/0x70 [21288.153850] ret_from_fork+0x1f/0x30 [21288.320329] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21288.322308] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21288.326230] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21288.329468] lmv_intent_open+0x338/0x1150 [lmv] [21288.334274] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21288.338488] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21288.344204] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21288.346896] ll_intent_lock+0xc5/0xaa0 [lustre] [21288.350283] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21288.354644] ll_lookup_it+0xe0e/0x2270 [lustre] [21288.357265] ? do_raw_spin_unlock+0x75/0x190 [21288.358846] ? _raw_spin_unlock+0x12/0x30 [21288.361164] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21288.363893] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21288.367616] ll_atomic_open+0x36a/0xd70 [lustre] [21288.370265] lookup_open+0x356/0xb40 [21288.372214] do_last+0x313/0xfc0 [21288.373650] ? nd_jump_root+0xe5/0x160 [21288.375638] ? path_init+0x437/0x520 [21288.378553] path_openat+0xf7/0x500 [21288.380208] ? do_raw_spin_unlock+0x75/0x190 [21288.381590] do_filp_open+0x99/0x140 [21288.383199] ? getname_flags+0x6e/0x330 [21288.385683] ? __check_object_size+0xff/0x256 [21288.388556] ? do_raw_spin_unlock+0x75/0x190 [21288.392285] ? _raw_spin_unlock+0x12/0x30 [21288.394690] do_sys_openat2+0x2b4/0x410 [21288.398799] do_sys_open+0x73/0xa0 [21288.400677] __x64_sys_openat+0x24/0x30 [21288.402114] do_syscall_64+0xc1/0x440 [21288.403339] entry_SYSCALL_64_after_hwframe+0x49/0xae [21288.408561] RIP: 0033:0x7f53425a5332 [21288.410375] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21288.425636] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21288.429654] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21288.434597] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21288.440350] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21288.444198] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21288.452545] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21296.052056] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [21296.055748] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21296.058024] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [21296.099378] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21296.099392] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21296.099399] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21296.105700] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21296.106054] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21296.106062] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21296.106065] RIP: 0010:native_safe_halt+0x2e/0x50 [21296.106079] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21296.106083] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21296.106087] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21296.106089] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21296.106091] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21296.106093] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21296.106096] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21296.106099] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21296.106102] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21296.106105] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21296.106114] Call Trace: [21296.106120] [21296.106126] ? show_regs.cold.9+0x22/0x2f [21296.106135] ? watchdog_timer_fn.cold.6+0x62/0x10a [21296.106141] ? __hrtimer_run_queues+0x1c2/0x4d0 [21296.106146] ? watchdog+0x40/0x40 [21296.106149] ? hrtimer_interrupt+0x128/0x2c0 [21296.106153] ? smp_apic_timer_interrupt+0xa6/0x270 [21296.106159] ? apic_timer_interrupt+0xf/0x20 [21296.106164] [21296.106167] ? native_safe_halt+0x2e/0x50 [21296.106170] kvm_wait+0xcd/0xe0 [21296.106178] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21296.106187] do_raw_spin_lock+0x177/0x190 [21296.106193] _raw_spin_lock+0x21/0x30 [21296.106197] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21296.108770] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21296.110139] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21296.110370] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21296.112278] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21296.119892] RIP: 0010:native_safe_halt+0x2e/0x50 [21296.119915] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21296.119920] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21296.119925] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21296.119928] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [21296.119930] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21296.119932] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21296.119934] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21296.119937] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21296.119940] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21296.119943] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21296.119962] Call Trace: [21296.119969] [21296.119974] ? show_regs.cold.9+0x22/0x2f [21296.119985] ? watchdog_timer_fn.cold.6+0x62/0x10a [21296.119991] ? __hrtimer_run_queues+0x1c2/0x4d0 [21296.119994] ? watchdog+0x40/0x40 [21296.119999] ? hrtimer_interrupt+0x128/0x2c0 [21296.120003] ? smp_apic_timer_interrupt+0xa6/0x270 [21296.120010] ? apic_timer_interrupt+0xf/0x20 [21296.120015] [21296.120017] ? native_safe_halt+0x2e/0x50 [21296.120021] kvm_wait+0xcd/0xe0 [21296.120028] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21296.120033] ? newidle_balance.isra.93+0x207/0x580 [21296.120040] do_raw_spin_lock+0x177/0x190 [21296.120044] _raw_spin_lock+0x21/0x30 [21296.120048] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21296.120310] process_one_work+0x2c8/0x700 [21296.120317] worker_thread+0x296/0x6e0 [21296.120321] ? rescuer_thread+0x570/0x570 [21296.120325] kthread+0x1d1/0x200 [21296.120400] ? set_kthread_struct+0x70/0x70 [21296.120407] ret_from_fork+0x1f/0x30 [21296.514922] ? woken_wake_function+0x30/0x30 [21296.521707] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21296.540870] kthread+0x1d1/0x200 [21296.544343] ? set_kthread_struct+0x70/0x70 [21296.548990] ret_from_fork+0x1f/0x30 [21302.239094] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18665s! [21302.242126] Showing busy workqueues and worker pools: [21302.244531] workqueue events: flags=0x0 [21302.246171] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21302.249157] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21302.254148] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21302.257238] pending: cpu_latency_work [ptlrpc] [21302.259223] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21302.262197] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21302.265451] workqueue events_long: flags=0x0 [21302.267312] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21302.270032] pending: lu_objects_flush [obdclass] [21302.272162] workqueue events_unbound: flags=0x2 [21302.274115] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21302.276825] pending: flush_memcg_stats_dwork, idle_cull_fn [21302.279121] workqueue events_power_efficient: flags=0x80 [21302.281912] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21302.284760] pending: check_lifetime, do_cache_clean [21302.287008] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21302.289790] pending: neigh_periodic_work, neigh_periodic_work [21302.292123] workqueue mm_percpu_wq: flags=0x8 [21302.293817] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21302.296387] pending: vmstat_update, lru_add_drain_per_cpu [21302.298864] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21302.301785] pending: vmstat_update, lru_add_drain_per_cpu [21302.304082] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21302.306993] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21302.309535] workqueue xprtiod: flags=0xa [21302.311050] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21302.313631] pending: xprt_autoclose, xs_error_handle [21302.315606] workqueue ipv6_addrconf: flags=0x40008 [21302.317633] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21302.320325] pending: addrconf_verify_work [21302.321754] workqueue knbd0-recv: flags=0x1a [21302.323989] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21302.326642] in-flight: 651:recv_work [21302.328338] workqueue ptlrpc_pinger: flags=0x2 [21302.330828] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21302.333021] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21302.335825] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18665s workers=4 idle: 15979 31824 8 [21302.339108] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21292s workers=2 idle: 654 [21316.055890] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [21316.057061] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [21316.062001] Modules linked in: [21316.064933] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21316.065021] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21316.065031] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21316.065034] RIP: 0010:native_safe_halt+0x2e/0x50 [21316.065048] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21316.065053] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21316.065058] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21316.065061] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21316.065063] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21316.065066] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21316.065069] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21316.065073] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21316.065076] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21316.065078] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21316.065087] Call Trace: [21316.065093] [21316.065100] ? show_regs.cold.9+0x22/0x2f [21316.065110] ? watchdog_timer_fn.cold.6+0x62/0x10a [21316.065117] ? __hrtimer_run_queues+0x1c2/0x4d0 [21316.065121] ? watchdog+0x40/0x40 [21316.065127] ? hrtimer_interrupt+0x128/0x2c0 [21316.065131] ? smp_apic_timer_interrupt+0xa6/0x270 [21316.065138] ? apic_timer_interrupt+0xf/0x20 [21316.065144] [21316.065147] ? native_safe_halt+0x2e/0x50 [21316.065152] kvm_wait+0xcd/0xe0 [21316.065159] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21316.065166] do_raw_spin_lock+0x177/0x190 [21316.065170] _raw_spin_lock+0x21/0x30 [21316.065176] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21316.065489] obd_mod_free+0x46/0x140 [ptlrpc] [21316.065695] mdc_commit_open+0x91/0xb0 [mdc] [21316.065733] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21316.065933] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21316.066130] after_reply+0xe2c/0x1780 [ptlrpc] [21316.066346] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21316.066538] ? __next_timer_interrupt+0x160/0x160 [21316.066547] ptlrpcd+0xa82/0xe40 [ptlrpc] [21316.066745] ? do_wait_intr+0xf0/0xf0 [21316.066753] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21316.067035] kthread+0x1d1/0x200 [21316.067042] ? set_kthread_struct+0x70/0x70 [21316.067045] ret_from_fork+0x1f/0x30 [21316.348052] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21316.379579] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21316.389505] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21316.398967] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21316.405990] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21316.417499] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21316.426293] RAX: 0000000000005c49 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21316.430809] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21316.434349] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21316.438824] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21316.445461] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21316.447075] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21316.458944] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21316.464331] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21316.470471] Call Trace: [21316.472830] [21316.474594] ? show_regs.cold.9+0x22/0x2f [21316.476010] ? watchdog_timer_fn.cold.6+0x62/0x10a [21316.481970] ? __hrtimer_run_queues+0x1c2/0x4d0 [21316.484980] ? watchdog+0x40/0x40 [21316.489106] ? hrtimer_interrupt+0x128/0x2c0 [21316.493499] ? smp_apic_timer_interrupt+0xa6/0x270 [21316.495002] ? apic_timer_interrupt+0xf/0x20 [21316.505879] [21316.512750] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21316.517035] do_raw_spin_lock+0x177/0x190 [21316.522546] _raw_spin_lock+0x21/0x30 [21316.524570] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21316.526925] ? do_raw_spin_unlock+0x75/0x190 [21316.528756] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21316.531967] ? do_wait_intr+0xf0/0xf0 [21316.533517] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21316.536295] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21316.538932] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21316.541587] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21316.543913] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21316.546537] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21316.548067] mdc_intent_lock+0x353/0x7e0 [mdc] [21316.549448] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21316.551096] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21316.552890] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21316.554941] lmv_intent_open+0x338/0x1150 [lmv] [21316.556783] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21316.559799] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21316.564593] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21316.566016] ll_intent_lock+0xc5/0xaa0 [lustre] [21316.567649] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21316.572945] ll_lookup_it+0xe0e/0x2270 [lustre] [21316.574733] ? do_raw_spin_unlock+0x75/0x190 [21316.576519] ? _raw_spin_unlock+0x12/0x30 [21316.578956] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21316.581046] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21316.583448] ll_atomic_open+0x36a/0xd70 [lustre] [21316.585526] lookup_open+0x356/0xb40 [21316.586872] do_last+0x313/0xfc0 [21316.588300] ? nd_jump_root+0xe5/0x160 [21316.589905] ? path_init+0x437/0x520 [21316.591188] path_openat+0xf7/0x500 [21316.592471] ? do_raw_spin_unlock+0x75/0x190 [21316.594007] do_filp_open+0x99/0x140 [21316.595313] ? getname_flags+0x6e/0x330 [21316.596834] ? __check_object_size+0xff/0x256 [21316.598677] ? do_raw_spin_unlock+0x75/0x190 [21316.600304] ? _raw_spin_unlock+0x12/0x30 [21316.601731] do_sys_openat2+0x2b4/0x410 [21316.603120] do_sys_open+0x73/0xa0 [21316.604438] __x64_sys_openat+0x24/0x30 [21316.605735] do_syscall_64+0xc1/0x440 [21316.606978] entry_SYSCALL_64_after_hwframe+0x49/0xae [21316.608935] RIP: 0033:0x7f53425a5332 [21316.610343] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21316.617020] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21316.619588] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21316.626517] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21316.629307] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21316.632027] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21316.634606] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21324.059585] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [21324.069876] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21324.130291] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21324.143995] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21324.157091] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21324.161575] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21324.215169] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21324.227568] RAX: 0000000000007b18 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21324.235781] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21324.251189] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21324.255690] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21324.266730] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21324.278694] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21324.287847] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21324.297787] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21324.303254] Call Trace: [21324.304613] [21324.310820] ? show_regs.cold.9+0x22/0x2f [21324.314778] ? watchdog_timer_fn.cold.6+0x62/0x10a [21324.318513] ? __hrtimer_run_queues+0x1c2/0x4d0 [21324.320347] ? watchdog+0x40/0x40 [21324.321485] ? hrtimer_interrupt+0x128/0x2c0 [21324.333260] ? smp_apic_timer_interrupt+0xa6/0x270 [21324.336304] ? apic_timer_interrupt+0xf/0x20 [21324.345376] [21324.347509] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21324.352658] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21324.359945] do_raw_spin_lock+0x177/0x190 [21324.361222] _raw_spin_lock+0x21/0x30 [21324.367793] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21324.369962] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21324.382415] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21324.388535] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21324.394346] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21324.395962] ? woken_wake_function+0x30/0x30 [21324.397057] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21324.417991] kthread+0x1d1/0x200 [21324.425043] ? set_kthread_struct+0x70/0x70 [21324.427045] ret_from_fork+0x1f/0x30 [21328.052480] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [21328.061167] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21328.110950] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21328.125180] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21328.136720] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21328.144792] RIP: 0010:native_safe_halt+0x2e/0x50 [21328.150462] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21328.167420] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21328.172336] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21328.176236] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [21328.183582] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21328.188536] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21328.195534] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21328.199285] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21328.207384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21328.209535] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21328.211915] Call Trace: [21328.212966] [21328.213605] ? show_regs.cold.9+0x22/0x2f [21328.215426] ? watchdog_timer_fn.cold.6+0x62/0x10a [21328.217166] ? __hrtimer_run_queues+0x1c2/0x4d0 [21328.218425] ? watchdog+0x40/0x40 [21328.219455] ? hrtimer_interrupt+0x128/0x2c0 [21328.220564] ? smp_apic_timer_interrupt+0xa6/0x270 [21328.221964] ? apic_timer_interrupt+0xf/0x20 [21328.223126] [21328.223540] ? native_safe_halt+0x2e/0x50 [21328.224218] kvm_wait+0xcd/0xe0 [21328.224771] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21328.225712] ? newidle_balance.isra.93+0x207/0x580 [21328.227092] do_raw_spin_lock+0x177/0x190 [21328.228591] _raw_spin_lock+0x21/0x30 [21328.229886] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21328.232117] process_one_work+0x2c8/0x700 [21328.233178] worker_thread+0x296/0x6e0 [21328.234110] ? rescuer_thread+0x570/0x570 [21328.235567] kthread+0x1d1/0x200 [21328.236744] ? set_kthread_struct+0x70/0x70 [21328.237961] ret_from_fork+0x1f/0x30 [21332.448076] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18695s! [21332.451921] Showing busy workqueues and worker pools: [21332.454260] workqueue events: flags=0x0 [21332.455774] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21332.459719] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21332.466643] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21332.472280] pending: cpu_latency_work [ptlrpc] [21332.475236] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21332.478922] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21332.483637] workqueue events_long: flags=0x0 [21332.486127] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21332.489248] pending: lu_objects_flush [obdclass] [21332.491698] workqueue events_unbound: flags=0x2 [21332.493647] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21332.496639] pending: flush_memcg_stats_dwork, idle_cull_fn [21332.499504] workqueue events_power_efficient: flags=0x80 [21332.502305] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21332.505844] pending: check_lifetime, do_cache_clean [21332.508328] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21332.511210] pending: neigh_periodic_work, neigh_periodic_work [21332.514375] workqueue mm_percpu_wq: flags=0x8 [21332.516449] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21332.519679] pending: vmstat_update, lru_add_drain_per_cpu [21332.522244] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21332.525288] pending: vmstat_update, lru_add_drain_per_cpu [21332.528065] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21332.531078] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21332.533307] workqueue xprtiod: flags=0xa [21332.534950] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21332.538217] pending: xprt_autoclose, xs_error_handle [21332.540402] workqueue ipv6_addrconf: flags=0x40008 [21332.542681] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21332.545546] pending: addrconf_verify_work [21332.547328] workqueue knbd0-recv: flags=0x1a [21332.549033] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21332.552589] in-flight: 651:recv_work [21332.554160] workqueue ptlrpc_pinger: flags=0x2 [21332.556120] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21332.559533] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21332.563087] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18695s workers=4 idle: 15979 31824 8 [21332.567014] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21323s workers=2 idle: 654 [21344.055035] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [21344.057027] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [21344.059336] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21344.059414] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21344.059421] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21344.059425] RIP: 0010:native_safe_halt+0x2e/0x50 [21344.059440] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21344.059444] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21344.059449] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21344.059452] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21344.059454] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21344.059456] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21344.059459] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21344.059462] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21344.059465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21344.059468] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21344.059475] Call Trace: [21344.059481] [21344.059487] ? show_regs.cold.9+0x22/0x2f [21344.059497] ? watchdog_timer_fn.cold.6+0x62/0x10a [21344.059504] ? __hrtimer_run_queues+0x1c2/0x4d0 [21344.059509] ? watchdog+0x40/0x40 [21344.059514] ? hrtimer_interrupt+0x128/0x2c0 [21344.059517] ? smp_apic_timer_interrupt+0xa6/0x270 [21344.059539] ? apic_timer_interrupt+0xf/0x20 [21344.059544] [21344.059547] ? native_safe_halt+0x2e/0x50 [21344.059552] kvm_wait+0xcd/0xe0 [21344.059560] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21344.059569] do_raw_spin_lock+0x177/0x190 [21344.059573] _raw_spin_lock+0x21/0x30 [21344.059578] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21344.059898] ? do_raw_spin_unlock+0x75/0x190 [21344.059916] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21344.060214] ? do_wait_intr+0xf0/0xf0 [21344.060228] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21344.060431] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21344.060748] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21344.060974] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21344.067337] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21344.067445] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21344.067452] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21344.067455] RIP: 0010:native_safe_halt+0x2e/0x50 [21344.067469] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21344.067474] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21344.067478] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21344.067481] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21344.067483] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21344.067485] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21344.067487] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21344.067490] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21344.067494] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21344.067496] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21344.067506] Call Trace: [21344.067510] [21344.067516] ? show_regs.cold.9+0x22/0x2f [21344.067524] ? watchdog_timer_fn.cold.6+0x62/0x10a [21344.067530] ? __hrtimer_run_queues+0x1c2/0x4d0 [21344.067532] ? watchdog+0x40/0x40 [21344.067535] ? hrtimer_interrupt+0x128/0x2c0 [21344.067538] ? smp_apic_timer_interrupt+0xa6/0x270 [21344.067543] ? apic_timer_interrupt+0xf/0x20 [21344.067546] [21344.067548] ? native_safe_halt+0x2e/0x50 [21344.067550] kvm_wait+0xcd/0xe0 [21344.067557] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21344.067562] do_raw_spin_lock+0x177/0x190 [21344.067564] _raw_spin_lock+0x21/0x30 [21344.067569] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21344.103070] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21344.108367] obd_mod_free+0x46/0x140 [ptlrpc] [21344.113674] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21344.115212] mdc_commit_open+0x91/0xb0 [mdc] [21344.124276] mdc_intent_lock+0x353/0x7e0 [mdc] [21344.126887] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21344.129990] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21344.132417] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21344.136078] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21344.138588] after_reply+0xe2c/0x1780 [ptlrpc] [21344.138884] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21344.139078] ? __next_timer_interrupt+0x160/0x160 [21344.139090] ptlrpcd+0xa82/0xe40 [ptlrpc] [21344.142592] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21344.145467] ? do_wait_intr+0xf0/0xf0 [21344.145490] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21344.145819] kthread+0x1d1/0x200 [21344.145825] ? set_kthread_struct+0x70/0x70 [21344.145829] ret_from_fork+0x1f/0x30 [21344.472770] lmv_intent_open+0x338/0x1150 [lmv] [21344.474538] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21344.490283] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21344.499341] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21344.502283] ll_intent_lock+0xc5/0xaa0 [lustre] [21344.505157] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21344.510974] ll_lookup_it+0xe0e/0x2270 [lustre] [21344.515616] ? do_raw_spin_unlock+0x75/0x190 [21344.521795] ? _raw_spin_unlock+0x12/0x30 [21344.525291] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21344.529238] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21344.531611] ll_atomic_open+0x36a/0xd70 [lustre] [21344.538854] lookup_open+0x356/0xb40 [21344.542100] do_last+0x313/0xfc0 [21344.543240] ? nd_jump_root+0xe5/0x160 [21344.546049] ? path_init+0x437/0x520 [21344.547522] path_openat+0xf7/0x500 [21344.553873] ? do_raw_spin_unlock+0x75/0x190 [21344.555815] do_filp_open+0x99/0x140 [21344.558295] ? getname_flags+0x6e/0x330 [21344.563264] ? __check_object_size+0xff/0x256 [21344.569841] ? do_raw_spin_unlock+0x75/0x190 [21344.571397] ? _raw_spin_unlock+0x12/0x30 [21344.573474] do_sys_openat2+0x2b4/0x410 [21344.575080] do_sys_open+0x73/0xa0 [21344.576439] __x64_sys_openat+0x24/0x30 [21344.578338] do_syscall_64+0xc1/0x440 [21344.581137] entry_SYSCALL_64_after_hwframe+0x49/0xae [21344.586060] RIP: 0033:0x7f53425a5332 [21344.589449] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21344.598009] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21344.600970] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21344.603547] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21344.606695] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21344.609513] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21344.611984] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21352.058185] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [21352.061192] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21352.083720] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21352.088285] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21352.092273] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21352.095122] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21352.103339] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21352.106483] RAX: 000000000000651f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21352.109803] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21352.112880] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21352.116242] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21352.119281] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21352.122324] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21352.126163] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21352.128651] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21352.131515] Call Trace: [21352.133099] [21352.134351] ? show_regs.cold.9+0x22/0x2f [21352.136199] ? watchdog_timer_fn.cold.6+0x62/0x10a [21352.138457] ? __hrtimer_run_queues+0x1c2/0x4d0 [21352.140477] ? watchdog+0x40/0x40 [21352.141965] ? hrtimer_interrupt+0x128/0x2c0 [21352.144314] ? smp_apic_timer_interrupt+0xa6/0x270 [21352.161313] ? apic_timer_interrupt+0xf/0x20 [21352.163025] [21352.165756] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21352.168945] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21352.175510] do_raw_spin_lock+0x177/0x190 [21352.181238] _raw_spin_lock+0x21/0x30 [21352.183932] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21352.187621] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21352.190520] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21352.193233] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21352.196517] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21352.199293] ? woken_wake_function+0x30/0x30 [21352.201417] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21352.204553] kthread+0x1d1/0x200 [21352.206000] ? set_kthread_struct+0x70/0x70 [21352.208315] ret_from_fork+0x1f/0x30 [21360.058848] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [21360.065078] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21360.106502] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21360.111930] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21360.115510] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21360.118466] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21360.126559] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21360.139468] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21360.142118] RAX: 000000000000378d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21360.148943] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [21360.159394] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21360.168848] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21360.180663] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21360.189463] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21360.192292] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21360.203169] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21360.208599] Call Trace: [21360.209775] [21360.210741] ? show_regs.cold.9+0x22/0x2f [21360.212812] ? watchdog_timer_fn.cold.6+0x62/0x10a [21360.215523] ? __hrtimer_run_queues+0x1c2/0x4d0 [21360.217753] ? watchdog+0x40/0x40 [21360.220504] ? hrtimer_interrupt+0x128/0x2c0 [21360.223595] ? smp_apic_timer_interrupt+0xa6/0x270 [21360.228275] ? apic_timer_interrupt+0xf/0x20 [21360.229919] [21360.235365] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21360.238430] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21360.241536] ? newidle_balance.isra.93+0x207/0x580 [21360.246268] do_raw_spin_lock+0x177/0x190 [21360.250947] _raw_spin_lock+0x21/0x30 [21360.253369] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21360.257397] process_one_work+0x2c8/0x700 [21360.262660] worker_thread+0x296/0x6e0 [21360.269948] ? rescuer_thread+0x570/0x570 [21360.275479] kthread+0x1d1/0x200 [21360.280610] ? set_kthread_struct+0x70/0x70 [21360.287282] ret_from_fork+0x1f/0x30 [21362.658717] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18725s! [21362.664147] Showing busy workqueues and worker pools: [21362.668130] workqueue events: flags=0x0 [21362.672890] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21362.675856] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21362.689500] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21362.692941] pending: cpu_latency_work [ptlrpc] [21362.696313] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21362.699923] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21362.707824] workqueue events_long: flags=0x0 [21362.709731] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21362.714644] pending: lu_objects_flush [obdclass] [21362.717500] workqueue events_unbound: flags=0x2 [21362.720209] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21362.724976] pending: flush_memcg_stats_dwork, idle_cull_fn [21362.734563] workqueue events_power_efficient: flags=0x80 [21362.739044] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21362.745455] pending: check_lifetime, do_cache_clean [21362.748920] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21362.754860] pending: neigh_periodic_work, neigh_periodic_work [21362.759278] workqueue mm_percpu_wq: flags=0x8 [21362.762095] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21362.767601] pending: vmstat_update, lru_add_drain_per_cpu [21362.772113] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21362.777960] pending: vmstat_update, lru_add_drain_per_cpu [21362.786650] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21362.791372] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21362.803895] workqueue xprtiod: flags=0xa [21362.808662] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21362.812699] pending: xprt_autoclose, xs_error_handle [21362.820448] workqueue ipv6_addrconf: flags=0x40008 [21362.822604] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21362.828446] pending: addrconf_verify_work [21362.834759] workqueue knbd0-recv: flags=0x1a [21362.836990] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21362.843070] in-flight: 651:recv_work [21362.845379] workqueue ptlrpc_pinger: flags=0x2 [21362.850279] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21362.855972] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21362.866706] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18725s workers=4 idle: 15979 31824 8 [21362.872475] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21353s workers=2 idle: 654 [21372.055100] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [21372.057011] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [21372.058440] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21372.058611] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21372.058620] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21372.058623] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21372.058636] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21372.058641] RSP: 0018:ffffbb5381503500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [21372.058645] RAX: 0000000000001186 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21372.058648] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21372.058650] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21372.058653] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21372.058655] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21372.058658] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21372.058662] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21372.058664] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21372.058672] Call Trace: [21372.058676] [21372.058686] ? show_regs.cold.9+0x22/0x2f [21372.058695] ? watchdog_timer_fn.cold.6+0x62/0x10a [21372.058701] ? __hrtimer_run_queues+0x1c2/0x4d0 [21372.058705] ? watchdog+0x40/0x40 [21372.058710] ? hrtimer_interrupt+0x128/0x2c0 [21372.058714] ? smp_apic_timer_interrupt+0xa6/0x270 [21372.058725] ? apic_timer_interrupt+0xf/0x20 [21372.058730] [21372.058733] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21372.058739] do_raw_spin_lock+0x177/0x190 [21372.058743] _raw_spin_lock+0x21/0x30 [21372.058748] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21372.064651] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21372.064729] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21372.064737] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21372.064741] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21372.064755] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21372.064759] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21372.064764] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21372.064767] RDX: 000000000000220a RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21372.064769] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21372.064772] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21372.064774] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21372.064777] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21372.064780] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21372.064783] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21372.064792] Call Trace: [21372.064798] [21372.064804] ? show_regs.cold.9+0x22/0x2f [21372.064828] ? watchdog_timer_fn.cold.6+0x62/0x10a [21372.064835] ? __hrtimer_run_queues+0x1c2/0x4d0 [21372.064839] ? watchdog+0x40/0x40 [21372.064843] ? hrtimer_interrupt+0x128/0x2c0 [21372.064847] ? smp_apic_timer_interrupt+0xa6/0x270 [21372.064855] ? apic_timer_interrupt+0xf/0x20 [21372.064860] [21372.064863] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21372.064868] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21372.064872] do_raw_spin_lock+0x177/0x190 [21372.064877] _raw_spin_lock+0x21/0x30 [21372.064884] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21372.065148] obd_mod_free+0x46/0x140 [ptlrpc] [21372.065415] mdc_commit_open+0x91/0xb0 [mdc] [21372.065454] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21372.096639] ? do_raw_spin_unlock+0x75/0x190 [21372.096660] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21372.101439] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21372.106218] ? do_wait_intr+0xf0/0xf0 [21372.109905] after_reply+0xe2c/0x1780 [ptlrpc] [21372.120455] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21372.123615] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21372.126875] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21372.129009] ? __next_timer_interrupt+0x160/0x160 [21372.129026] ptlrpcd+0xa82/0xe40 [ptlrpc] [21372.131182] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21372.133644] ? do_wait_intr+0xf0/0xf0 [21372.133664] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21372.133913] kthread+0x1d1/0x200 [21372.133919] ? set_kthread_struct+0x70/0x70 [21372.133923] ret_from_fork+0x1f/0x30 [21372.302752] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21372.304532] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21372.313590] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21372.319360] mdc_intent_lock+0x353/0x7e0 [mdc] [21372.321904] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21372.324605] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21372.327596] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21372.331501] lmv_intent_open+0x338/0x1150 [lmv] [21372.333527] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21372.336542] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21372.339115] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21372.340578] ll_intent_lock+0xc5/0xaa0 [lustre] [21372.342600] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21372.345408] ll_lookup_it+0xe0e/0x2270 [lustre] [21372.347609] ? do_raw_spin_unlock+0x75/0x190 [21372.349910] ? _raw_spin_unlock+0x12/0x30 [21372.351771] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21372.354446] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21372.357615] ll_atomic_open+0x36a/0xd70 [lustre] [21372.359896] lookup_open+0x356/0xb40 [21372.361677] do_last+0x313/0xfc0 [21372.368981] ? nd_jump_root+0xe5/0x160 [21372.371597] ? path_init+0x437/0x520 [21372.372610] path_openat+0xf7/0x500 [21372.373581] ? do_raw_spin_unlock+0x75/0x190 [21372.390306] do_filp_open+0x99/0x140 [21372.391381] ? getname_flags+0x6e/0x330 [21372.392500] ? __check_object_size+0xff/0x256 [21372.400280] ? do_raw_spin_unlock+0x75/0x190 [21372.407204] ? _raw_spin_unlock+0x12/0x30 [21372.412580] do_sys_openat2+0x2b4/0x410 [21372.415560] do_sys_open+0x73/0xa0 [21372.421416] __x64_sys_openat+0x24/0x30 [21372.422615] do_syscall_64+0xc1/0x440 [21372.432156] entry_SYSCALL_64_after_hwframe+0x49/0xae [21372.438795] RIP: 0033:0x7f53425a5332 [21372.450867] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21372.467506] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21372.473829] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21372.478123] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21372.485758] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21372.495724] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21372.504814] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21380.058023] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [21380.060974] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21380.080939] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21380.084805] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21380.088285] RIP: 0010:native_safe_halt+0x2e/0x50 [21380.089799] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21380.094995] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [21380.097045] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21380.099004] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21380.100870] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21380.102766] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21380.104595] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21380.106524] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21380.108603] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21380.110179] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21380.111911] Call Trace: [21380.112639] [21380.113207] ? show_regs.cold.9+0x22/0x2f [21380.114293] ? watchdog_timer_fn.cold.6+0x62/0x10a [21380.115653] ? __hrtimer_run_queues+0x1c2/0x4d0 [21380.116784] ? watchdog+0x40/0x40 [21380.117775] ? hrtimer_interrupt+0x128/0x2c0 [21380.118928] ? smp_apic_timer_interrupt+0xa6/0x270 [21380.120268] ? apic_timer_interrupt+0xf/0x20 [21380.121436] [21380.121980] ? native_safe_halt+0x2e/0x50 [21380.123046] kvm_wait+0xcd/0xe0 [21380.123903] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21380.125275] do_raw_spin_lock+0x177/0x190 [21380.126325] _raw_spin_lock+0x21/0x30 [21380.127321] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21380.129081] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21380.131230] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21380.133045] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21380.134838] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21380.136619] ? woken_wake_function+0x30/0x30 [21380.137811] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21380.139840] kthread+0x1d1/0x200 [21380.140803] ? set_kthread_struct+0x70/0x70 [21380.141943] ret_from_fork+0x1f/0x30 [21388.052635] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [21388.055576] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21388.073518] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21388.078129] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21388.081378] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21388.084711] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21388.087226] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21388.095541] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21388.098593] RAX: 00000000000042a1 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21388.101940] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [21388.104544] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21388.107297] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21388.110180] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21388.113345] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21388.116863] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21388.119414] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21388.123276] Call Trace: [21388.124477] [21388.125421] ? show_regs.cold.9+0x22/0x2f [21388.127502] ? watchdog_timer_fn.cold.6+0x62/0x10a [21388.130056] ? __hrtimer_run_queues+0x1c2/0x4d0 [21388.131791] ? watchdog+0x40/0x40 [21388.133389] ? hrtimer_interrupt+0x128/0x2c0 [21388.135105] ? smp_apic_timer_interrupt+0xa6/0x270 [21388.137047] ? apic_timer_interrupt+0xf/0x20 [21388.138590] [21388.139396] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21388.141749] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21388.144255] ? newidle_balance.isra.93+0x207/0x580 [21388.146342] do_raw_spin_lock+0x177/0x190 [21388.147603] _raw_spin_lock+0x21/0x30 [21388.148984] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21388.151922] process_one_work+0x2c8/0x700 [21388.153757] worker_thread+0x296/0x6e0 [21388.155135] ? rescuer_thread+0x570/0x570 [21388.156609] kthread+0x1d1/0x200 [21388.158276] ? set_kthread_struct+0x70/0x70 [21388.159978] ret_from_fork+0x1f/0x30 [21393.377235] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18756s! [21393.380486] Showing busy workqueues and worker pools: [21393.383783] workqueue events: flags=0x0 [21393.387891] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21393.391543] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21393.402623] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21393.409595] pending: cpu_latency_work [ptlrpc] [21393.414133] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21393.417511] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21393.424827] workqueue events_long: flags=0x0 [21393.426818] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21393.432737] pending: lu_objects_flush [obdclass] [21393.435906] workqueue events_unbound: flags=0x2 [21393.438322] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21393.442569] pending: flush_memcg_stats_dwork, idle_cull_fn [21393.453147] workqueue events_power_efficient: flags=0x80 [21393.459641] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21393.463683] pending: check_lifetime, do_cache_clean [21393.466337] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21393.469794] pending: neigh_periodic_work, neigh_periodic_work [21393.474619] workqueue mm_percpu_wq: flags=0x8 [21393.476727] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21393.480790] pending: vmstat_update, lru_add_drain_per_cpu [21393.486746] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21393.499431] pending: vmstat_update, lru_add_drain_per_cpu [21393.510139] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21393.518316] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21393.520821] workqueue xprtiod: flags=0xa [21393.525396] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21393.533835] pending: xprt_autoclose, xs_error_handle [21393.539207] workqueue ipv6_addrconf: flags=0x40008 [21393.546302] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21393.548633] pending: addrconf_verify_work [21393.553571] workqueue knbd0-recv: flags=0x1a [21393.555275] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21393.558512] in-flight: 651:recv_work [21393.560285] workqueue ptlrpc_pinger: flags=0x2 [21393.561847] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21393.570043] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21393.572151] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18756s workers=4 idle: 15979 31824 8 [21393.579343] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21384s workers=2 idle: 654 [21400.055322] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [21400.057866] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [21400.060398] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21400.060475] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21400.060482] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21400.060485] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21400.060500] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21400.060504] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21400.060509] RAX: 000000000000599a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21400.060512] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21400.060514] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21400.060516] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21400.060519] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21400.060522] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21400.060525] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21400.060527] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21400.060536] Call Trace: [21400.060542] [21400.060548] ? show_regs.cold.9+0x22/0x2f [21400.060557] ? watchdog_timer_fn.cold.6+0x62/0x10a [21400.060563] ? __hrtimer_run_queues+0x1c2/0x4d0 [21400.060566] ? watchdog+0x40/0x40 [21400.060571] ? hrtimer_interrupt+0x128/0x2c0 [21400.060575] ? smp_apic_timer_interrupt+0xa6/0x270 [21400.060582] ? apic_timer_interrupt+0xf/0x20 [21400.060586] [21400.060589] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21400.060595] do_raw_spin_lock+0x177/0x190 [21400.060599] _raw_spin_lock+0x21/0x30 [21400.060603] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21400.064714] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21400.064799] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21400.064806] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21400.064809] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21400.064824] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21400.064829] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21400.064834] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21400.064837] RDX: 0000000000001652 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21400.064839] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21400.064841] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21400.064844] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21400.064847] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21400.064850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21400.064853] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21400.064861] Call Trace: [21400.064867] [21400.064873] ? show_regs.cold.9+0x22/0x2f [21400.064883] ? watchdog_timer_fn.cold.6+0x62/0x10a [21400.064890] ? __hrtimer_run_queues+0x1c2/0x4d0 [21400.064894] ? watchdog+0x40/0x40 [21400.064898] ? hrtimer_interrupt+0x128/0x2c0 [21400.064903] ? smp_apic_timer_interrupt+0xa6/0x270 [21400.064911] ? apic_timer_interrupt+0xf/0x20 [21400.064926] [21400.064929] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21400.064935] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21400.064940] do_raw_spin_lock+0x177/0x190 [21400.064945] _raw_spin_lock+0x21/0x30 [21400.064950] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21400.065245] obd_mod_free+0x46/0x140 [ptlrpc] [21400.065682] mdc_commit_open+0x91/0xb0 [mdc] [21400.065717] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21400.065939] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21400.066156] after_reply+0xe2c/0x1780 [ptlrpc] [21400.066530] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21400.066733] ? __next_timer_interrupt+0x160/0x160 [21400.066755] ptlrpcd+0xa82/0xe40 [ptlrpc] [21400.066972] ? do_wait_intr+0xf0/0xf0 [21400.066982] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21400.067306] kthread+0x1d1/0x200 [21400.067312] ? set_kthread_struct+0x70/0x70 [21400.067316] ret_from_fork+0x1f/0x30 [21400.579637] ? do_raw_spin_unlock+0x75/0x190 [21400.594024] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21400.596230] ? do_wait_intr+0xf0/0xf0 [21400.605297] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21400.611959] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21400.625514] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21400.632811] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21400.634232] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21400.637548] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21400.645513] mdc_intent_lock+0x353/0x7e0 [mdc] [21400.648388] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21400.654852] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21400.663424] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21400.669331] lmv_intent_open+0x338/0x1150 [lmv] [21400.678683] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21400.688775] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21400.690641] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21400.697993] ll_intent_lock+0xc5/0xaa0 [lustre] [21400.702477] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21400.706700] ll_lookup_it+0xe0e/0x2270 [lustre] [21400.709232] ? do_raw_spin_unlock+0x75/0x190 [21400.712097] ? _raw_spin_unlock+0x12/0x30 [21400.715278] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21400.724166] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21400.734140] ll_atomic_open+0x36a/0xd70 [lustre] [21400.742300] lookup_open+0x356/0xb40 [21400.743593] do_last+0x313/0xfc0 [21400.748326] ? nd_jump_root+0xe5/0x160 [21400.751238] ? path_init+0x437/0x520 [21400.753171] path_openat+0xf7/0x500 [21400.758159] ? do_raw_spin_unlock+0x75/0x190 [21400.763145] do_filp_open+0x99/0x140 [21400.767167] ? getname_flags+0x6e/0x330 [21400.772238] ? __check_object_size+0xff/0x256 [21400.774349] ? do_raw_spin_unlock+0x75/0x190 [21400.778680] ? _raw_spin_unlock+0x12/0x30 [21400.780208] do_sys_openat2+0x2b4/0x410 [21400.782061] do_sys_open+0x73/0xa0 [21400.783293] __x64_sys_openat+0x24/0x30 [21400.785332] do_syscall_64+0xc1/0x440 [21400.786652] entry_SYSCALL_64_after_hwframe+0x49/0xae [21400.791355] RIP: 0033:0x7f53425a5332 [21400.800444] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21400.814560] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21400.823976] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21400.832614] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21400.839972] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21400.842189] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21400.849338] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21408.058099] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [21408.061266] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21408.082392] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21408.087497] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21408.091322] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21408.093849] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21408.101730] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21408.104705] RAX: 0000000000006591 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21408.107496] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21408.111426] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21408.116291] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21408.120416] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21408.124728] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21408.128418] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21408.131070] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21408.134112] Call Trace: [21408.135543] [21408.136440] ? show_regs.cold.9+0x22/0x2f [21408.138027] ? watchdog_timer_fn.cold.6+0x62/0x10a [21408.140450] ? __hrtimer_run_queues+0x1c2/0x4d0 [21408.142436] ? watchdog+0x40/0x40 [21408.143879] ? hrtimer_interrupt+0x128/0x2c0 [21408.145527] ? smp_apic_timer_interrupt+0xa6/0x270 [21408.147462] ? apic_timer_interrupt+0xf/0x20 [21408.150004] [21408.151268] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21408.153409] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21408.155530] do_raw_spin_lock+0x177/0x190 [21408.158229] _raw_spin_lock+0x21/0x30 [21408.159916] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21408.163250] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21408.166217] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21408.168658] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21408.171306] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21408.173492] ? woken_wake_function+0x30/0x30 [21408.175846] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21408.178471] kthread+0x1d1/0x200 [21408.179994] ? set_kthread_struct+0x70/0x70 [21408.181660] ret_from_fork+0x1f/0x30 [21414.903125] rcu: INFO: rcu_sched self-detected stall on CPU [21414.905352] rcu: 2-....: (11131823 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [21414.910285] (t=19320477 jiffies g=274185 q=68799) [21414.912342] rcu: rcu_sched kthread starved for 18778843 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [21414.916980] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [21414.920129] rcu: RCU grace-period kthread stack dump: [21414.922275] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [21414.926242] Call Trace: [21414.929189] __schedule+0x351/0xcb0 [21414.931365] schedule+0xc0/0x180 [21414.932924] schedule_timeout+0xb4/0x190 [21414.936037] ? do_raw_spin_unlock+0x75/0x190 [21414.939876] ? __next_timer_interrupt+0x160/0x160 [21414.942043] ? prepare_to_swait_event+0xf3/0x140 [21414.944186] rcu_gp_kthread+0x8dd/0x15b0 [21414.945823] ? force_qs_rnp+0x2c0/0x2c0 [21414.947396] kthread+0x1d1/0x200 [21414.949015] ? set_kthread_struct+0x70/0x70 [21414.950868] ret_from_fork+0x1f/0x30 [21414.952831] rcu: Stack dump where RCU GP kthread last ran: [21414.955235] Sending NMI from CPU 2 to CPUs 0: [21414.958238] NMI backtrace for cpu 0 [21414.958244] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21414.958246] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21414.958248] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21414.958251] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21414.958252] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21414.958255] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 [21414.958257] RAX: 0000000000003b83 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21414.958258] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [21414.958260] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21414.958262] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21414.958263] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21414.958265] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21414.958266] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21414.958268] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21414.958269] Call Trace: [21414.958270] [21414.958271] ? show_regs.cold.9+0x22/0x2f [21414.958272] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [21414.958273] ? nmi_cpu_backtrace_handler+0x15/0x40 [21414.958274] ? nmi_handle+0xbd/0x210 [21414.958275] ? default_do_nmi+0x4b/0x1b0 [21414.958276] ? do_nmi+0x280/0x380 [21414.958277] ? end_repeat_nmi+0x16/0x1f [21414.958278] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21414.958279] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21414.958281] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21414.958282] [21414.958284] ? newidle_balance.isra.93+0x207/0x580 [21414.958285] do_raw_spin_lock+0x177/0x190 [21414.958287] _raw_spin_lock+0x21/0x30 [21414.958288] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21414.958289] process_one_work+0x2c8/0x700 [21414.958290] worker_thread+0x296/0x6e0 [21414.958291] ? rescuer_thread+0x570/0x570 [21414.958292] kthread+0x1d1/0x200 [21414.958294] ? set_kthread_struct+0x70/0x70 [21414.958295] ret_from_fork+0x1f/0x30 [21414.960193] Sending NMI from CPU 2 to CPUs 1: [21415.073296] NMI backtrace for cpu 1 [21415.073302] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21415.073304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21415.073306] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21415.073308] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21415.073310] RSP: 0018:ffffbb5381503500 EFLAGS: 00000212 [21415.073313] RAX: 0000000000006446 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21415.073314] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21415.073315] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21415.073317] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21415.073319] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21415.075563] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21415.075566] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21415.075567] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21415.075568] Call Trace: [21415.075569] [21415.075570] ? show_regs.cold.9+0x22/0x2f [21415.075571] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [21415.075572] ? nmi_cpu_backtrace_handler+0x15/0x40 [21415.075574] ? nmi_handle+0xbd/0x210 [21415.075575] ? default_do_nmi+0x4b/0x1b0 [21415.075576] ? do_nmi+0x280/0x380 [21415.075577] ? end_repeat_nmi+0x16/0x1f [21415.075578] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21415.075579] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21415.075580] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21415.075581] [21415.075581] do_raw_spin_lock+0x177/0x190 [21415.075582] _raw_spin_lock+0x21/0x30 [21415.075583] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21415.075584] ? do_raw_spin_unlock+0x75/0x190 [21415.075585] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21415.075586] ? do_wait_intr+0xf0/0xf0 [21415.075587] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21415.075588] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21415.075589] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21415.075590] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21415.075592] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21415.075593] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21415.075593] mdc_intent_lock+0x353/0x7e0 [mdc] [21415.075594] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21415.075595] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21415.075596] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21415.075597] lmv_intent_open+0x338/0x1150 [lmv] [21415.075599] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21415.075600] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21415.075601] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21415.075602] ll_intent_lock+0xc5/0xaa0 [lustre] [21415.075604] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21415.075605] ll_lookup_it+0xe0e/0x2270 [lustre] [21415.075606] ? do_raw_spin_unlock+0x75/0x190 [21415.075607] ? _raw_spin_unlock+0x12/0x30 [21415.075608] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21415.075609] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21415.075610] ll_atomic_open+0x36a/0xd70 [lustre] [21415.075611] lookup_open+0x356/0xb40 [21415.075612] do_last+0x313/0xfc0 [21415.075613] ? nd_jump_root+0xe5/0x160 [21415.075614] ? path_init+0x437/0x520 [21415.075615] path_openat+0xf7/0x500 [21415.075616] ? do_raw_spin_unlock+0x75/0x190 [21415.075617] do_filp_open+0x99/0x140 [21415.075618] ? getname_flags+0x6e/0x330 [21415.075619] ? __check_object_size+0xff/0x256 [21415.075620] ? do_raw_spin_unlock+0x75/0x190 [21415.075621] ? _raw_spin_unlock+0x12/0x30 [21415.075622] do_sys_openat2+0x2b4/0x410 [21415.075623] do_sys_open+0x73/0xa0 [21415.075624] __x64_sys_openat+0x24/0x30 [21415.075625] do_syscall_64+0xc1/0x440 [21415.075626] entry_SYSCALL_64_after_hwframe+0x49/0xae [21415.075627] RIP: 0033:0x7f53425a5332 [21415.075628] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21415.075630] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21415.075633] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21415.075635] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21415.075636] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21415.075637] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21415.075638] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21415.075645] NMI backtrace for cpu 2 [21415.414832] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21415.429823] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21415.439768] Call Trace: [21415.441352] [21415.442369] ? dump_stack+0xbb/0x10e [21415.448966] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [21415.457898] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [21415.465539] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [21415.477040] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [21415.478821] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [21415.487968] ? rcu_dump_cpu_stacks+0x11b/0x171 [21415.500638] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [21415.503148] ? update_process_times+0x6d/0xc0 [21415.506278] ? tick_sched_handle.isra.18+0x27/0x90 [21415.508195] ? tick_sched_timer+0x53/0xb0 [21415.514492] ? __hrtimer_run_queues+0x1c2/0x4d0 [21415.528136] ? tick_sched_handle.isra.18+0x90/0x90 [21415.530199] ? hrtimer_interrupt+0x128/0x2c0 [21415.533943] ? smp_apic_timer_interrupt+0xa6/0x270 [21415.535573] ? apic_timer_interrupt+0xf/0x20 [21415.536984] [21415.546415] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21415.559106] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21415.567847] ? do_raw_spin_lock+0x177/0x190 [21415.570870] ? _raw_spin_lock+0x21/0x30 [21415.576305] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21415.584367] ? obd_mod_free+0x46/0x140 [ptlrpc] [21415.592925] ? mdc_commit_open+0x91/0xb0 [mdc] [21415.601313] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21415.607809] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21415.612649] ? after_reply+0xe2c/0x1780 [ptlrpc] [21415.619992] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21415.623137] ? __next_timer_interrupt+0x160/0x160 [21415.627861] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [21415.636677] ? do_wait_intr+0xf0/0xf0 [21415.639020] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21415.646403] ? kthread+0x1d1/0x200 [21415.648966] ? set_kthread_struct+0x70/0x70 [21415.659699] ? ret_from_fork+0x1f/0x30 [21423.583303] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18786s! [21423.603772] Showing busy workqueues and worker pools: [21423.606389] workqueue events: flags=0x0 [21423.612112] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21423.618799] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21423.630930] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21423.637574] pending: cpu_latency_work [ptlrpc] [21423.646915] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21423.649227] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21423.660657] workqueue events_long: flags=0x0 [21423.662188] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21423.677656] pending: lu_objects_flush [obdclass] [21423.683497] workqueue events_unbound: flags=0x2 [21423.686864] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21423.696824] pending: flush_memcg_stats_dwork, idle_cull_fn [21423.706019] workqueue events_power_efficient: flags=0x80 [21423.712534] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21423.719959] pending: check_lifetime, do_cache_clean [21423.725939] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21423.727934] pending: neigh_periodic_work, neigh_periodic_work [21423.740217] workqueue mm_percpu_wq: flags=0x8 [21423.745312] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21423.751220] pending: vmstat_update, lru_add_drain_per_cpu [21423.753400] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21423.755919] pending: vmstat_update, lru_add_drain_per_cpu [21423.760697] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21423.763631] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21423.766926] workqueue xprtiod: flags=0xa [21423.771822] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21423.776552] pending: xprt_autoclose, xs_error_handle [21423.781157] workqueue ipv6_addrconf: flags=0x40008 [21423.783590] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21423.787873] pending: addrconf_verify_work [21423.791112] workqueue knbd0-recv: flags=0x1a [21423.793169] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21423.799586] in-flight: 651:recv_work [21423.800923] workqueue ptlrpc_pinger: flags=0x2 [21423.803594] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21423.808291] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21423.813914] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18786s workers=4 idle: 15979 31824 8 [21423.822075] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21414s workers=2 idle: 654 [21436.058009] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [21436.061553] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21436.083025] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21436.089756] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21436.094400] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21436.096640] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21436.102249] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21436.104598] RAX: 00000000000055d7 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21436.106486] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21436.108701] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21436.111075] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21436.115000] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21436.117955] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21436.124877] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21436.130267] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21436.133414] Call Trace: [21436.138669] [21436.139433] ? show_regs.cold.9+0x22/0x2f [21436.141644] ? watchdog_timer_fn.cold.6+0x62/0x10a [21436.143024] ? __hrtimer_run_queues+0x1c2/0x4d0 [21436.147873] ? watchdog+0x40/0x40 [21436.148928] ? hrtimer_interrupt+0x128/0x2c0 [21436.151015] ? smp_apic_timer_interrupt+0xa6/0x270 [21436.154215] ? apic_timer_interrupt+0xf/0x20 [21436.158205] [21436.158885] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21436.163611] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21436.166381] do_raw_spin_lock+0x177/0x190 [21436.168083] _raw_spin_lock+0x21/0x30 [21436.170929] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21436.173749] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21436.175645] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21436.177702] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21436.179646] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21436.181575] ? woken_wake_function+0x30/0x30 [21436.182946] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21436.185289] kthread+0x1d1/0x200 [21436.186336] ? set_kthread_struct+0x70/0x70 [21436.187718] ret_from_fork+0x1f/0x30 [21440.055037] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [21440.057935] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21440.060511] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [21440.083578] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21440.083594] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21440.083597] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21440.083610] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21440.083615] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21440.083620] RAX: 0000000000006c5b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21440.083623] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21440.083625] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21440.083627] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21440.083630] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21440.083633] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21440.083636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21440.083638] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21440.083648] Call Trace: [21440.083655] [21440.083660] ? show_regs.cold.9+0x22/0x2f [21440.083670] ? watchdog_timer_fn.cold.6+0x62/0x10a [21440.083677] ? __hrtimer_run_queues+0x1c2/0x4d0 [21440.083681] ? watchdog+0x40/0x40 [21440.083687] ? hrtimer_interrupt+0x128/0x2c0 [21440.083691] ? smp_apic_timer_interrupt+0xa6/0x270 [21440.083699] ? apic_timer_interrupt+0xf/0x20 [21440.083704] [21440.083708] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21440.083716] do_raw_spin_lock+0x177/0x190 [21440.083720] _raw_spin_lock+0x21/0x30 [21440.083726] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21440.083991] ? do_raw_spin_unlock+0x75/0x190 [21440.083997] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21440.084275] ? do_wait_intr+0xf0/0xf0 [21440.084283] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21440.084493] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21440.084717] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21440.084929] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21440.085037] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21440.085205] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21440.085231] mdc_intent_lock+0x353/0x7e0 [mdc] [21440.085256] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21440.085341] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21440.085556] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21440.085583] lmv_intent_open+0x338/0x1150 [lmv] [21440.085614] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21440.085702] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21440.085782] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21440.085806] ll_intent_lock+0xc5/0xaa0 [lustre] [21440.085899] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21440.085988] ll_lookup_it+0xe0e/0x2270 [lustre] [21440.086154] ? do_raw_spin_unlock+0x75/0x190 [21440.086161] ? _raw_spin_unlock+0x12/0x30 [21440.086167] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21440.086192] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21440.086214] ll_atomic_open+0x36a/0xd70 [lustre] [21440.086308] lookup_open+0x356/0xb40 [21440.086318] do_last+0x313/0xfc0 [21440.086323] ? nd_jump_root+0xe5/0x160 [21440.086328] ? path_init+0x437/0x520 [21440.086333] path_openat+0xf7/0x500 [21440.086339] ? do_raw_spin_unlock+0x75/0x190 [21440.086344] do_filp_open+0x99/0x140 [21440.086351] ? getname_flags+0x6e/0x330 [21440.086356] ? __check_object_size+0xff/0x256 [21440.086363] ? do_raw_spin_unlock+0x75/0x190 [21440.086368] ? _raw_spin_unlock+0x12/0x30 [21440.086373] do_sys_openat2+0x2b4/0x410 [21440.086378] do_sys_open+0x73/0xa0 [21440.086383] __x64_sys_openat+0x24/0x30 [21440.086387] do_syscall_64+0xc1/0x440 [21440.086394] entry_SYSCALL_64_after_hwframe+0x49/0xae [21440.086400] RIP: 0033:0x7f53425a5332 [21440.086407] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21440.086411] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21440.086416] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21440.086419] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21440.086422] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21440.086425] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21440.086427] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21440.518311] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21440.555853] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21440.561910] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21440.571984] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21440.575379] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21440.590275] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21440.597338] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21440.604832] RDX: 0000000000001e12 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21440.610406] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21440.613520] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21440.618481] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21440.623494] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21440.628787] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21440.634564] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21440.641279] Call Trace: [21440.642720] [21440.644005] ? show_regs.cold.9+0x22/0x2f [21440.646232] ? watchdog_timer_fn.cold.6+0x62/0x10a [21440.648360] ? __hrtimer_run_queues+0x1c2/0x4d0 [21440.651298] ? watchdog+0x40/0x40 [21440.653255] ? hrtimer_interrupt+0x128/0x2c0 [21440.655543] ? smp_apic_timer_interrupt+0xa6/0x270 [21440.657709] ? apic_timer_interrupt+0xf/0x20 [21440.664022] [21440.666264] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21440.671264] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21440.674759] do_raw_spin_lock+0x177/0x190 [21440.676983] _raw_spin_lock+0x21/0x30 [21440.678581] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21440.682045] obd_mod_free+0x46/0x140 [ptlrpc] [21440.691314] mdc_commit_open+0x91/0xb0 [mdc] [21440.695627] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21440.699711] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21440.702837] after_reply+0xe2c/0x1780 [ptlrpc] [21440.707429] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21440.712756] ? __next_timer_interrupt+0x160/0x160 [21440.717856] ptlrpcd+0xa82/0xe40 [ptlrpc] [21440.723640] ? do_wait_intr+0xf0/0xf0 [21440.724980] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21440.729997] kthread+0x1d1/0x200 [21440.733658] ? set_kthread_struct+0x70/0x70 [21440.737537] ret_from_fork+0x1f/0x30 [21448.052832] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [21448.059103] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21448.101625] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21448.111679] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21448.119501] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21448.126956] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21448.137001] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21448.156949] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21448.162649] RAX: 0000000000007d12 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21448.168650] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [21448.181464] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21448.188841] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21448.198525] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21448.202816] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21448.211101] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21448.215188] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21448.219282] Call Trace: [21448.221192] [21448.222965] ? show_regs.cold.9+0x22/0x2f [21448.224968] ? watchdog_timer_fn.cold.6+0x62/0x10a [21448.227301] ? __hrtimer_run_queues+0x1c2/0x4d0 [21448.230144] ? watchdog+0x40/0x40 [21448.231908] ? hrtimer_interrupt+0x128/0x2c0 [21448.236066] ? smp_apic_timer_interrupt+0xa6/0x270 [21448.240713] ? apic_timer_interrupt+0xf/0x20 [21448.243539] [21448.248210] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21448.252681] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21448.257111] ? newidle_balance.isra.93+0x207/0x580 [21448.268232] do_raw_spin_lock+0x177/0x190 [21448.274833] _raw_spin_lock+0x21/0x30 [21448.278824] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21448.281571] process_one_work+0x2c8/0x700 [21448.289033] worker_thread+0x296/0x6e0 [21448.290711] ? rescuer_thread+0x570/0x570 [21448.293459] kthread+0x1d1/0x200 [21448.295043] ? set_kthread_struct+0x70/0x70 [21448.297482] ret_from_fork+0x1f/0x30 [21454.306022] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18817s! [21454.312640] Showing busy workqueues and worker pools: [21454.316243] workqueue events: flags=0x0 [21454.319340] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21454.324100] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21454.335859] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21454.339182] pending: cpu_latency_work [ptlrpc] [21454.343982] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21454.350690] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21454.354415] workqueue events_long: flags=0x0 [21454.360395] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21454.365721] pending: lu_objects_flush [obdclass] [21454.368280] workqueue events_unbound: flags=0x2 [21454.372429] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21454.376005] pending: flush_memcg_stats_dwork, idle_cull_fn [21454.381620] workqueue events_power_efficient: flags=0x80 [21454.385226] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21454.387442] pending: check_lifetime, do_cache_clean [21454.391306] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21454.395681] pending: neigh_periodic_work, neigh_periodic_work [21454.397642] workqueue mm_percpu_wq: flags=0x8 [21454.401232] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21454.406050] pending: vmstat_update, lru_add_drain_per_cpu [21454.409659] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21454.412730] pending: vmstat_update, lru_add_drain_per_cpu [21454.416754] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21454.423261] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21454.428146] workqueue xprtiod: flags=0xa [21454.430014] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21454.434062] pending: xprt_autoclose, xs_error_handle [21454.436720] workqueue ipv6_addrconf: flags=0x40008 [21454.444463] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21454.449004] pending: addrconf_verify_work [21454.452636] workqueue knbd0-recv: flags=0x1a [21454.456075] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21454.459294] in-flight: 651:recv_work [21454.463010] workqueue ptlrpc_pinger: flags=0x2 [21454.465326] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21454.470420] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21454.474618] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18817s workers=4 idle: 15979 31824 8 [21454.478072] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21444s workers=2 idle: 654 [21464.058036] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [21464.061915] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21464.101082] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21464.120194] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21464.127614] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21464.133973] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21464.145292] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21464.149938] RAX: 0000000000005a8e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21464.154192] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21464.157532] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21464.161856] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21464.166699] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21464.170722] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21464.175456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21464.178834] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21464.183491] Call Trace: [21464.185561] [21464.186942] ? show_regs.cold.9+0x22/0x2f [21464.189371] ? watchdog_timer_fn.cold.6+0x62/0x10a [21464.191607] ? __hrtimer_run_queues+0x1c2/0x4d0 [21464.193983] ? watchdog+0x40/0x40 [21464.196732] ? hrtimer_interrupt+0x128/0x2c0 [21464.198974] ? smp_apic_timer_interrupt+0xa6/0x270 [21464.204915] ? apic_timer_interrupt+0xf/0x20 [21464.206949] [21464.209664] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21464.215901] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21464.223819] do_raw_spin_lock+0x177/0x190 [21464.228914] _raw_spin_lock+0x21/0x30 [21464.233587] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21464.238899] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21464.242404] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21464.250606] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21464.257799] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21464.271535] ? woken_wake_function+0x30/0x30 [21464.286594] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21464.294467] kthread+0x1d1/0x200 [21464.299991] ? set_kthread_struct+0x70/0x70 [21464.304904] ret_from_fork+0x1f/0x30 [21468.058098] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [21468.058865] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [21468.070846] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21468.070926] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21468.070934] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21468.070936] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21468.070951] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21468.070955] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21468.070959] RAX: 0000000000003885 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21468.070961] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21468.070964] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21468.070966] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21468.070968] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21468.070971] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21468.070975] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21468.070977] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21468.070987] Call Trace: [21468.070993] [21468.070999] ? show_regs.cold.9+0x22/0x2f [21468.071026] ? watchdog_timer_fn.cold.6+0x62/0x10a [21468.071033] ? __hrtimer_run_queues+0x1c2/0x4d0 [21468.071037] ? watchdog+0x40/0x40 [21468.071041] ? hrtimer_interrupt+0x128/0x2c0 [21468.071046] ? smp_apic_timer_interrupt+0xa6/0x270 [21468.071054] ? apic_timer_interrupt+0xf/0x20 [21468.071059] [21468.071062] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21468.071069] do_raw_spin_lock+0x177/0x190 [21468.071075] _raw_spin_lock+0x21/0x30 [21468.071080] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21468.078356] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) [21468.118611] ? do_raw_spin_unlock+0x75/0x190 [21468.118850] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21468.119105] ? do_wait_intr+0xf0/0xf0 [21468.119234] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21468.119435] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21468.119918] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21468.120372] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21468.120484] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21468.120523] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21468.120780] mdc_intent_lock+0x353/0x7e0 [mdc] [21468.120810] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21468.120895] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21468.121396] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21468.121425] lmv_intent_open+0x338/0x1150 [lmv] [21468.121456] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21468.121546] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21468.121656] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21468.121681] ll_intent_lock+0xc5/0xaa0 [lustre] [21468.121765] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21468.121850] ll_lookup_it+0xe0e/0x2270 [lustre] [21468.121937] ? do_raw_spin_unlock+0x75/0x190 [21468.121946] ? _raw_spin_unlock+0x12/0x30 [21468.121952] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21468.122663] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21468.122684] ll_atomic_open+0x36a/0xd70 [lustre] [21468.122774] lookup_open+0x356/0xb40 [21468.122783] do_last+0x313/0xfc0 [21468.122788] ? nd_jump_root+0xe5/0x160 [21468.122792] ? path_init+0x437/0x520 [21468.122797] path_openat+0xf7/0x500 [21468.122802] ? do_raw_spin_unlock+0x75/0x190 [21468.122808] do_filp_open+0x99/0x140 [21468.122814] ? getname_flags+0x6e/0x330 [21468.122818] ? __check_object_size+0xff/0x256 [21468.122825] ? do_raw_spin_unlock+0x75/0x190 [21468.122828] ? _raw_spin_unlock+0x12/0x30 [21468.122834] do_sys_openat2+0x2b4/0x410 [21468.122839] do_sys_open+0x73/0xa0 [21468.122843] __x64_sys_openat+0x24/0x30 [21468.122847] do_syscall_64+0xc1/0x440 [21468.122854] entry_SYSCALL_64_after_hwframe+0x49/0xae [21468.122860] RIP: 0033:0x7f53425a5332 [21468.122867] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21468.122871] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21468.122877] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21468.122879] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21468.122881] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21468.122883] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21468.122886] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21468.515828] obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21468.527190] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21468.531753] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21468.535366] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21468.537918] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21468.545411] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21468.548501] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21468.551475] RDX: 000000000000028f RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21468.554501] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21468.557292] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21468.559772] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21468.562520] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21468.566012] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21468.568203] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21468.571099] Call Trace: [21468.572136] [21468.572973] ? show_regs.cold.9+0x22/0x2f [21468.574470] ? watchdog_timer_fn.cold.6+0x62/0x10a [21468.576475] ? __hrtimer_run_queues+0x1c2/0x4d0 [21468.578450] ? watchdog+0x40/0x40 [21468.579909] ? hrtimer_interrupt+0x128/0x2c0 [21468.581714] ? smp_apic_timer_interrupt+0xa6/0x270 [21468.583788] ? apic_timer_interrupt+0xf/0x20 [21468.585476] [21468.586054] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21468.587593] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21468.589478] do_raw_spin_lock+0x177/0x190 [21468.591205] _raw_spin_lock+0x21/0x30 [21468.592820] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21468.594995] obd_mod_free+0x46/0x140 [ptlrpc] [21468.597025] mdc_commit_open+0x91/0xb0 [mdc] [21468.598819] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21468.600794] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21468.603289] after_reply+0xe2c/0x1780 [ptlrpc] [21468.605417] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21468.608187] ? __next_timer_interrupt+0x160/0x160 [21468.610819] ptlrpcd+0xa82/0xe40 [ptlrpc] [21468.612441] ? do_wait_intr+0xf0/0xf0 [21468.614520] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21468.617514] kthread+0x1d1/0x200 [21468.620528] ? set_kthread_struct+0x70/0x70 [21468.623782] ret_from_fork+0x1f/0x30 [21480.052050] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [21480.069291] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21480.106039] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21480.119562] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21480.130910] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21480.134750] RIP: 0010:native_safe_halt+0x2e/0x50 [21480.137098] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21480.155856] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21480.166382] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21480.174493] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [21480.179751] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21480.186147] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21480.196818] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21480.206735] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21480.215421] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21480.220441] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21480.225266] Call Trace: [21480.226271] [21480.227151] ? show_regs.cold.9+0x22/0x2f [21480.228595] ? watchdog_timer_fn.cold.6+0x62/0x10a [21480.230899] ? __hrtimer_run_queues+0x1c2/0x4d0 [21480.232604] ? watchdog+0x40/0x40 [21480.236172] ? hrtimer_interrupt+0x128/0x2c0 [21480.240242] ? smp_apic_timer_interrupt+0xa6/0x270 [21480.244207] ? apic_timer_interrupt+0xf/0x20 [21480.247840] [21480.248621] ? native_safe_halt+0x2e/0x50 [21480.250180] kvm_wait+0xcd/0xe0 [21480.253560] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21480.257097] ? newidle_balance.isra.93+0x207/0x580 [21480.262514] do_raw_spin_lock+0x177/0x190 [21480.264404] _raw_spin_lock+0x21/0x30 [21480.265764] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21480.268133] process_one_work+0x2c8/0x700 [21480.269327] worker_thread+0x296/0x6e0 [21480.271818] ? rescuer_thread+0x570/0x570 [21480.275831] kthread+0x1d1/0x200 [21480.279019] ? set_kthread_struct+0x70/0x70 [21480.281686] ret_from_fork+0x1f/0x30 [21484.518497] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18847s! [21484.521985] Showing busy workqueues and worker pools: [21484.524195] workqueue events: flags=0x0 [21484.525936] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21484.528191] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21484.533659] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21484.536542] pending: cpu_latency_work [ptlrpc] [21484.538497] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21484.541128] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21484.548065] workqueue events_long: flags=0x0 [21484.550208] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21484.553615] pending: lu_objects_flush [obdclass] [21484.555957] workqueue events_unbound: flags=0x2 [21484.557967] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21484.560650] pending: flush_memcg_stats_dwork, idle_cull_fn [21484.563441] workqueue events_power_efficient: flags=0x80 [21484.565700] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21484.568754] pending: check_lifetime, do_cache_clean [21484.571485] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21484.574163] pending: neigh_periodic_work, neigh_periodic_work [21484.576196] workqueue mm_percpu_wq: flags=0x8 [21484.578068] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21484.580469] pending: vmstat_update, lru_add_drain_per_cpu [21484.582633] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21484.589822] pending: vmstat_update, lru_add_drain_per_cpu [21484.592519] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21484.595613] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21484.598224] workqueue xprtiod: flags=0xa [21484.599882] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21484.601671] pending: xprt_autoclose, xs_error_handle [21484.603193] workqueue ipv6_addrconf: flags=0x40008 [21484.604660] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21484.606656] pending: addrconf_verify_work [21484.608015] workqueue knbd0-recv: flags=0x1a [21484.609702] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21484.612073] in-flight: 651:recv_work [21484.613659] workqueue ptlrpc_pinger: flags=0x2 [21484.619209] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21484.624647] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21484.627917] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18847s workers=4 idle: 15979 31824 8 [21484.631737] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21475s workers=2 idle: 654 [21492.060639] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [21492.068310] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21492.114496] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21492.124098] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21492.132103] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21492.136710] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21492.163152] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21492.173315] RAX: 0000000000007e09 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21492.181715] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21492.195190] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21492.207187] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21492.220640] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21492.230339] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21492.242029] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21492.253399] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21492.259820] Call Trace: [21492.260863] [21492.262716] ? show_regs.cold.9+0x22/0x2f [21492.267787] ? watchdog_timer_fn.cold.6+0x62/0x10a [21492.273756] ? __hrtimer_run_queues+0x1c2/0x4d0 [21492.279682] ? watchdog+0x40/0x40 [21492.281088] ? hrtimer_interrupt+0x128/0x2c0 [21492.282997] ? smp_apic_timer_interrupt+0xa6/0x270 [21492.285315] ? apic_timer_interrupt+0xf/0x20 [21492.287078] [21492.289531] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21492.292827] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21492.295763] do_raw_spin_lock+0x177/0x190 [21492.298120] _raw_spin_lock+0x21/0x30 [21492.300321] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21492.303609] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21492.306527] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21492.308558] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21492.310754] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21492.313460] ? woken_wake_function+0x30/0x30 [21492.315392] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21492.318042] kthread+0x1d1/0x200 [21492.319391] ? set_kthread_struct+0x70/0x70 [21492.321085] ret_from_fork+0x1f/0x30 [21496.056154] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [21496.057026] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [21496.058971] Modules linked in: [21496.068828] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21496.068911] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21496.068918] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21496.068920] RIP: 0010:native_safe_halt+0x2e/0x50 [21496.068934] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21496.068938] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21496.068943] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21496.068945] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21496.068948] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21496.068950] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21496.068953] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21496.068956] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21496.068960] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21496.068962] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21496.068971] Call Trace: [21496.068977] [21496.068982] ? show_regs.cold.9+0x22/0x2f [21496.068992] ? watchdog_timer_fn.cold.6+0x62/0x10a [21496.068998] ? __hrtimer_run_queues+0x1c2/0x4d0 [21496.069003] ? watchdog+0x40/0x40 [21496.069008] ? hrtimer_interrupt+0x128/0x2c0 [21496.069012] ? smp_apic_timer_interrupt+0xa6/0x270 [21496.069019] ? apic_timer_interrupt+0xf/0x20 [21496.069024] [21496.069028] ? native_safe_halt+0x2e/0x50 [21496.069033] kvm_wait+0xcd/0xe0 [21496.069041] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21496.069048] do_raw_spin_lock+0x177/0x190 [21496.069053] _raw_spin_lock+0x21/0x30 [21496.069058] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21496.072466] obd_mod_free+0x46/0x140 [ptlrpc] [21496.072679] mdc_commit_open+0x91/0xb0 [mdc] [21496.072715] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21496.072913] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21496.073422] after_reply+0xe2c/0x1780 [ptlrpc] [21496.075567] lustre(O) [21496.135353] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21496.135637] ? __next_timer_interrupt+0x160/0x160 [21496.135649] ptlrpcd+0xa82/0xe40 [ptlrpc] [21496.135871] ? do_wait_intr+0xf0/0xf0 [21496.135882] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21496.136115] kthread+0x1d1/0x200 [21496.136121] ? set_kthread_struct+0x70/0x70 [21496.136126] ret_from_fork+0x1f/0x30 [21496.472419] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21496.524008] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21496.548331] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21496.555353] RIP: 0010:native_safe_halt+0x2e/0x50 [21496.564863] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21496.581269] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff13 [21496.587270] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21496.592883] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21496.610083] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21496.622214] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21496.627915] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21496.631757] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21496.636116] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21496.639037] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21496.642703] Call Trace: [21496.645861] [21496.648708] ? show_regs.cold.9+0x22/0x2f [21496.652942] ? watchdog_timer_fn.cold.6+0x62/0x10a [21496.664868] ? __hrtimer_run_queues+0x1c2/0x4d0 [21496.673215] ? watchdog+0x40/0x40 [21496.681383] ? hrtimer_interrupt+0x128/0x2c0 [21496.682835] ? smp_apic_timer_interrupt+0xa6/0x270 [21496.691210] ? apic_timer_interrupt+0xf/0x20 [21496.697346] [21496.701982] ? native_safe_halt+0x2e/0x50 [21496.705132] kvm_wait+0xcd/0xe0 [21496.712412] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21496.721129] do_raw_spin_lock+0x177/0x190 [21496.730114] _raw_spin_lock+0x21/0x30 [21496.735372] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21496.747958] ? do_raw_spin_unlock+0x75/0x190 [21496.752650] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21496.760601] ? do_wait_intr+0xf0/0xf0 [21496.766714] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21496.774695] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21496.782272] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21496.791646] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21496.798276] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21496.800976] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21496.802636] mdc_intent_lock+0x353/0x7e0 [mdc] [21496.804107] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21496.810035] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21496.812258] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21496.823219] lmv_intent_open+0x338/0x1150 [lmv] [21496.825563] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21496.828701] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21496.832540] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21496.839556] ll_intent_lock+0xc5/0xaa0 [lustre] [21496.842595] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21496.854205] ll_lookup_it+0xe0e/0x2270 [lustre] [21496.855784] ? do_raw_spin_unlock+0x75/0x190 [21496.869490] ? _raw_spin_unlock+0x12/0x30 [21496.870955] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21496.888749] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21496.898291] ll_atomic_open+0x36a/0xd70 [lustre] [21496.908886] lookup_open+0x356/0xb40 [21496.913472] do_last+0x313/0xfc0 [21496.916347] ? nd_jump_root+0xe5/0x160 [21496.926662] ? path_init+0x437/0x520 [21496.929686] path_openat+0xf7/0x500 [21496.938221] ? do_raw_spin_unlock+0x75/0x190 [21496.941915] do_filp_open+0x99/0x140 [21496.944606] ? getname_flags+0x6e/0x330 [21496.949253] ? __check_object_size+0xff/0x256 [21496.952403] ? do_raw_spin_unlock+0x75/0x190 [21496.955364] ? _raw_spin_unlock+0x12/0x30 [21496.957614] do_sys_openat2+0x2b4/0x410 [21496.959647] do_sys_open+0x73/0xa0 [21496.963720] __x64_sys_openat+0x24/0x30 [21496.966642] do_syscall_64+0xc1/0x440 [21496.976213] entry_SYSCALL_64_after_hwframe+0x49/0xae [21496.982959] RIP: 0033:0x7f53425a5332 [21496.984933] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21496.999020] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21497.003105] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21497.006735] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21497.012138] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21497.019144] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21497.032521] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21512.053579] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [21512.063173] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21512.092953] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21512.099123] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21512.104599] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21512.108005] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21512.111281] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21512.121954] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21512.126109] RAX: 0000000000005896 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21512.130758] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [21512.134157] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21512.137811] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21512.141448] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21512.148620] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21512.154341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21512.161720] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21512.170088] Call Trace: [21512.171150] [21512.171940] ? show_regs.cold.9+0x22/0x2f [21512.173703] ? watchdog_timer_fn.cold.6+0x62/0x10a [21512.180130] ? __hrtimer_run_queues+0x1c2/0x4d0 [21512.184594] ? watchdog+0x40/0x40 [21512.186516] ? hrtimer_interrupt+0x128/0x2c0 [21512.191530] ? smp_apic_timer_interrupt+0xa6/0x270 [21512.195694] ? apic_timer_interrupt+0xf/0x20 [21512.202274] [21512.203285] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21512.206522] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21512.213110] ? newidle_balance.isra.93+0x207/0x580 [21512.221073] do_raw_spin_lock+0x177/0x190 [21512.225447] _raw_spin_lock+0x21/0x30 [21512.231168] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21512.237405] process_one_work+0x2c8/0x700 [21512.239445] worker_thread+0x296/0x6e0 [21512.244140] ? rescuer_thread+0x570/0x570 [21512.246599] kthread+0x1d1/0x200 [21512.248634] ? set_kthread_struct+0x70/0x70 [21512.251305] ret_from_fork+0x1f/0x30 [21514.727651] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18877s! [21514.733993] Showing busy workqueues and worker pools: [21514.745062] workqueue events: flags=0x0 [21514.750875] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21514.758612] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21514.778459] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21514.788593] pending: cpu_latency_work [ptlrpc] [21514.795394] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21514.799352] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21514.810140] workqueue events_long: flags=0x0 [21514.814377] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21514.822955] pending: lu_objects_flush [obdclass] [21514.832923] workqueue events_unbound: flags=0x2 [21514.839412] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21514.846633] pending: flush_memcg_stats_dwork, idle_cull_fn [21514.851232] workqueue events_power_efficient: flags=0x80 [21514.859069] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21514.865910] pending: check_lifetime, do_cache_clean [21514.867848] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21514.878635] pending: neigh_periodic_work, neigh_periodic_work [21514.885177] workqueue mm_percpu_wq: flags=0x8 [21514.889527] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21514.893064] pending: vmstat_update, lru_add_drain_per_cpu [21514.896066] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21514.899640] pending: vmstat_update, lru_add_drain_per_cpu [21514.902572] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21514.906049] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21514.915802] workqueue xprtiod: flags=0xa [21514.920727] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21514.927016] pending: xprt_autoclose, xs_error_handle [21514.929598] workqueue ipv6_addrconf: flags=0x40008 [21514.931210] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21514.940129] pending: addrconf_verify_work [21514.946025] workqueue knbd0-recv: flags=0x1a [21514.952963] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21514.964324] in-flight: 651:recv_work [21514.966886] workqueue ptlrpc_pinger: flags=0x2 [21514.969305] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21514.978650] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21514.991243] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18877s workers=4 idle: 15979 31824 8 [21514.996299] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21505s workers=2 idle: 654 [21520.063126] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [21520.066799] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21520.095289] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21520.098781] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21520.101599] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21520.104343] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21520.124196] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21520.127317] RAX: 0000000000007783 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21520.131228] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21520.134855] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21520.138084] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21520.141393] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21520.144863] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21520.148313] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21520.150982] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21520.153926] Call Trace: [21520.155047] [21520.155928] ? show_regs.cold.9+0x22/0x2f [21520.157525] ? watchdog_timer_fn.cold.6+0x62/0x10a [21520.159567] ? __hrtimer_run_queues+0x1c2/0x4d0 [21520.161173] ? watchdog+0x40/0x40 [21520.162249] ? hrtimer_interrupt+0x128/0x2c0 [21520.163941] ? smp_apic_timer_interrupt+0xa6/0x270 [21520.165559] ? apic_timer_interrupt+0xf/0x20 [21520.167167] [21520.168025] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21520.169802] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21520.172525] do_raw_spin_lock+0x177/0x190 [21520.175322] _raw_spin_lock+0x21/0x30 [21520.177236] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21520.180834] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21520.183750] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21520.186021] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21520.188618] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21520.192632] ? woken_wake_function+0x30/0x30 [21520.194935] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21520.198096] kthread+0x1d1/0x200 [21520.199499] ? set_kthread_struct+0x70/0x70 [21520.202427] ret_from_fork+0x1f/0x30 [21524.055230] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [21524.063797] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21524.066235] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [21524.123142] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21524.123155] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21524.123158] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21524.123171] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21524.123175] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21524.123181] RAX: 000000000000540e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21524.123183] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21524.123186] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21524.123188] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21524.123190] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21524.123193] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21524.123196] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21524.123198] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21524.123206] Call Trace: [21524.123212] [21524.123218] ? show_regs.cold.9+0x22/0x2f [21524.123228] ? watchdog_timer_fn.cold.6+0x62/0x10a [21524.123235] ? __hrtimer_run_queues+0x1c2/0x4d0 [21524.123240] ? watchdog+0x40/0x40 [21524.123244] ? hrtimer_interrupt+0x128/0x2c0 [21524.123247] ? smp_apic_timer_interrupt+0xa6/0x270 [21524.123253] ? apic_timer_interrupt+0xf/0x20 [21524.123258] [21524.123261] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21524.123269] do_raw_spin_lock+0x177/0x190 [21524.123274] _raw_spin_lock+0x21/0x30 [21524.123280] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21524.124189] ? do_raw_spin_unlock+0x75/0x190 [21524.124200] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21524.125169] ? do_wait_intr+0xf0/0xf0 [21524.125180] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21524.125395] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21524.126764] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21524.127268] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21524.127378] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21524.128861] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21524.128893] mdc_intent_lock+0x353/0x7e0 [mdc] [21524.143772] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21524.144120] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21524.144127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21524.144130] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21524.144143] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21524.144147] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21524.144152] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21524.144155] RDX: 00000000000059b2 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21524.144157] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21524.144159] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21524.144162] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21524.144165] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21524.144167] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21524.144169] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21524.144177] Call Trace: [21524.144182] [21524.144187] ? show_regs.cold.9+0x22/0x2f [21524.144195] ? watchdog_timer_fn.cold.6+0x62/0x10a [21524.144201] ? __hrtimer_run_queues+0x1c2/0x4d0 [21524.144205] ? watchdog+0x40/0x40 [21524.144210] ? hrtimer_interrupt+0x128/0x2c0 [21524.144214] ? smp_apic_timer_interrupt+0xa6/0x270 [21524.144222] ? apic_timer_interrupt+0xf/0x20 [21524.144226] [21524.144230] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21524.144234] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21524.144239] do_raw_spin_lock+0x177/0x190 [21524.144243] _raw_spin_lock+0x21/0x30 [21524.144249] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21524.145271] obd_mod_free+0x46/0x140 [ptlrpc] [21524.145675] mdc_commit_open+0x91/0xb0 [mdc] [21524.145715] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21524.158740] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21524.158860] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21524.170337] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21524.170886] after_reply+0xe2c/0x1780 [ptlrpc] [21524.171227] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21524.186381] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21524.186742] lmv_intent_open+0x338/0x1150 [lmv] [21524.186776] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21524.187357] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21524.187643] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21524.187735] ll_intent_lock+0xc5/0xaa0 [lustre] [21524.188031] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21524.188121] ll_lookup_it+0xe0e/0x2270 [lustre] [21524.189350] ? do_raw_spin_unlock+0x75/0x190 [21524.189360] ? _raw_spin_unlock+0x12/0x30 [21524.189367] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21524.189394] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21524.190157] ll_atomic_open+0x36a/0xd70 [lustre] [21524.190776] lookup_open+0x356/0xb40 [21524.190795] do_last+0x313/0xfc0 [21524.190801] ? nd_jump_root+0xe5/0x160 [21524.190810] ? path_init+0x437/0x520 [21524.190815] path_openat+0xf7/0x500 [21524.190820] ? do_raw_spin_unlock+0x75/0x190 [21524.190827] do_filp_open+0x99/0x140 [21524.190833] ? getname_flags+0x6e/0x330 [21524.190838] ? __check_object_size+0xff/0x256 [21524.190846] ? do_raw_spin_unlock+0x75/0x190 [21524.190850] ? _raw_spin_unlock+0x12/0x30 [21524.190855] do_sys_openat2+0x2b4/0x410 [21524.190860] do_sys_open+0x73/0xa0 [21524.190869] __x64_sys_openat+0x24/0x30 [21524.190873] do_syscall_64+0xc1/0x440 [21524.190880] entry_SYSCALL_64_after_hwframe+0x49/0xae [21524.190887] RIP: 0033:0x7f53425a5332 [21524.190896] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21524.190900] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21524.190916] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21524.191222] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21524.191226] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21524.191228] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21524.191231] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21524.992894] ? __next_timer_interrupt+0x160/0x160 [21524.994640] ptlrpcd+0xa82/0xe40 [ptlrpc] [21524.996541] ? do_wait_intr+0xf0/0xf0 [21524.999007] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21525.001888] kthread+0x1d1/0x200 [21525.003989] ? set_kthread_struct+0x70/0x70 [21525.010424] ret_from_fork+0x1f/0x30 [21540.058997] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [21540.065980] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21540.110495] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21540.127069] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21540.146108] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21540.158596] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21540.167477] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21540.184728] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21540.193474] RAX: 0000000000001b1e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21540.199871] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [21540.202485] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21540.206082] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21540.211141] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21540.214077] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21540.218610] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21540.221431] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21540.224570] Call Trace: [21540.225577] [21540.226572] ? show_regs.cold.9+0x22/0x2f [21540.228496] ? watchdog_timer_fn.cold.6+0x62/0x10a [21540.236482] ? __hrtimer_run_queues+0x1c2/0x4d0 [21540.248655] ? watchdog+0x40/0x40 [21540.251714] ? hrtimer_interrupt+0x128/0x2c0 [21540.253178] ? smp_apic_timer_interrupt+0xa6/0x270 [21540.254731] ? apic_timer_interrupt+0xf/0x20 [21540.271636] [21540.272581] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21540.288035] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21540.289768] ? newidle_balance.isra.93+0x207/0x580 [21540.299275] do_raw_spin_lock+0x177/0x190 [21540.301912] _raw_spin_lock+0x21/0x30 [21540.305647] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21540.312201] process_one_work+0x2c8/0x700 [21540.315178] worker_thread+0x296/0x6e0 [21540.317644] ? rescuer_thread+0x570/0x570 [21540.319058] kthread+0x1d1/0x200 [21540.328825] ? set_kthread_struct+0x70/0x70 [21540.330081] ret_from_fork+0x1f/0x30 [21545.441026] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18908s! [21545.443470] Showing busy workqueues and worker pools: [21545.445534] workqueue events: flags=0x0 [21545.446835] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21545.448967] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21545.452572] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21545.454345] pending: cpu_latency_work [ptlrpc] [21545.456133] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21545.458369] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21545.461193] workqueue events_long: flags=0x0 [21545.462994] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21545.465135] pending: lu_objects_flush [obdclass] [21545.467104] workqueue events_unbound: flags=0x2 [21545.468662] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21545.470696] pending: flush_memcg_stats_dwork, idle_cull_fn [21545.472880] workqueue events_power_efficient: flags=0x80 [21545.474757] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21545.477506] pending: check_lifetime, do_cache_clean [21545.479223] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21545.481636] pending: neigh_periodic_work, neigh_periodic_work [21545.483709] workqueue mm_percpu_wq: flags=0x8 [21545.485406] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21545.487813] pending: vmstat_update, lru_add_drain_per_cpu [21545.489938] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21545.492178] pending: vmstat_update, lru_add_drain_per_cpu [21545.494062] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21545.496387] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21545.498811] workqueue xprtiod: flags=0xa [21545.500106] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21545.502165] pending: xprt_autoclose, xs_error_handle [21545.504236] workqueue ipv6_addrconf: flags=0x40008 [21545.505989] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21545.508552] pending: addrconf_verify_work [21545.510103] workqueue knbd0-recv: flags=0x1a [21545.511714] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21545.514505] in-flight: 651:recv_work [21545.516041] workqueue ptlrpc_pinger: flags=0x2 [21545.517797] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21545.519997] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21545.522519] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18908s workers=4 idle: 15979 31824 8 [21545.525231] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21535s workers=2 idle: 654 [21548.058693] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [21548.070478] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21548.115572] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21548.123515] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21548.134245] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21548.136481] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21548.159982] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21548.163798] RAX: 000000000000385e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21548.166835] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21548.169904] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21548.173910] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21548.177538] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21548.181909] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21548.186403] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21548.188615] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21548.191546] Call Trace: [21548.192772] [21548.193480] ? show_regs.cold.9+0x22/0x2f [21548.195045] ? watchdog_timer_fn.cold.6+0x62/0x10a [21548.198106] ? __hrtimer_run_queues+0x1c2/0x4d0 [21548.200546] ? watchdog+0x40/0x40 [21548.202332] ? hrtimer_interrupt+0x128/0x2c0 [21548.204659] ? smp_apic_timer_interrupt+0xa6/0x270 [21548.206767] ? apic_timer_interrupt+0xf/0x20 [21548.208855] [21548.209939] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21548.212543] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21548.214709] do_raw_spin_lock+0x177/0x190 [21548.216693] _raw_spin_lock+0x21/0x30 [21548.218595] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21548.221585] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21548.224961] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21548.228051] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21548.231530] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21548.234851] ? woken_wake_function+0x30/0x30 [21548.237316] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21548.241134] kthread+0x1d1/0x200 [21548.243045] ? set_kthread_struct+0x70/0x70 [21548.245475] ret_from_fork+0x1f/0x30 [21552.057064] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [21552.063269] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [21552.063380] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21552.063460] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21552.063466] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21552.063469] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21552.063483] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21552.063487] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21552.063492] RAX: 0000000000004190 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21552.063494] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21552.063496] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21552.063499] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21552.063501] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21552.063504] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21552.063507] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21552.063509] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21552.063518] Call Trace: [21552.063524] [21552.063530] ? show_regs.cold.9+0x22/0x2f [21552.063539] ? watchdog_timer_fn.cold.6+0x62/0x10a [21552.063548] ? __hrtimer_run_queues+0x1c2/0x4d0 [21552.063552] ? watchdog+0x40/0x40 [21552.063558] ? hrtimer_interrupt+0x128/0x2c0 [21552.063562] ? smp_apic_timer_interrupt+0xa6/0x270 [21552.063570] ? apic_timer_interrupt+0xf/0x20 [21552.063574] [21552.063578] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21552.063586] do_raw_spin_lock+0x177/0x190 [21552.063591] _raw_spin_lock+0x21/0x30 [21552.063596] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21552.064597] ? do_raw_spin_unlock+0x75/0x190 [21552.064604] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21552.065423] ? do_wait_intr+0xf0/0xf0 [21552.065434] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21552.068902] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21552.070335] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21552.070557] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21552.075605] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21552.075678] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21552.075685] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21552.075688] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21552.075701] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21552.075704] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21552.075708] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21552.075711] RDX: 0000000000005224 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21552.075713] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21552.075715] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21552.075717] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21552.075720] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21552.075722] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21552.075724] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21552.075731] Call Trace: [21552.075736] [21552.075741] ? show_regs.cold.9+0x22/0x2f [21552.076030] ? watchdog_timer_fn.cold.6+0x62/0x10a [21552.076037] ? __hrtimer_run_queues+0x1c2/0x4d0 [21552.076041] ? watchdog+0x40/0x40 [21552.076045] ? hrtimer_interrupt+0x128/0x2c0 [21552.076049] ? smp_apic_timer_interrupt+0xa6/0x270 [21552.076057] ? apic_timer_interrupt+0xf/0x20 [21552.076062] [21552.076065] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21552.076069] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21552.076074] do_raw_spin_lock+0x177/0x190 [21552.076142] _raw_spin_lock+0x21/0x30 [21552.076151] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21552.079637] obd_mod_free+0x46/0x140 [ptlrpc] [21552.132358] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21552.132486] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21552.132518] mdc_intent_lock+0x353/0x7e0 [mdc] [21552.132550] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21552.133215] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21552.145482] mdc_commit_open+0x91/0xb0 [mdc] [21552.148874] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21552.153065] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21552.153387] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21552.153600] after_reply+0xe2c/0x1780 [ptlrpc] [21552.154110] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21552.154313] ? __next_timer_interrupt+0x160/0x160 [21552.154335] ptlrpcd+0xa82/0xe40 [ptlrpc] [21552.154537] ? do_wait_intr+0xf0/0xf0 [21552.154547] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21552.154751] kthread+0x1d1/0x200 [21552.154756] ? set_kthread_struct+0x70/0x70 [21552.154760] ret_from_fork+0x1f/0x30 [21552.581933] lmv_intent_open+0x338/0x1150 [lmv] [21552.584703] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21552.588454] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21552.590533] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21552.592216] ll_intent_lock+0xc5/0xaa0 [lustre] [21552.594459] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21552.596476] ll_lookup_it+0xe0e/0x2270 [lustre] [21552.598284] ? do_raw_spin_unlock+0x75/0x190 [21552.599863] ? _raw_spin_unlock+0x12/0x30 [21552.601430] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21552.603394] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21552.605203] ll_atomic_open+0x36a/0xd70 [lustre] [21552.607568] lookup_open+0x356/0xb40 [21552.608870] do_last+0x313/0xfc0 [21552.610047] ? nd_jump_root+0xe5/0x160 [21552.611497] ? path_init+0x437/0x520 [21552.612909] path_openat+0xf7/0x500 [21552.614224] ? do_raw_spin_unlock+0x75/0x190 [21552.615722] do_filp_open+0x99/0x140 [21552.617640] ? getname_flags+0x6e/0x330 [21552.618856] ? __check_object_size+0xff/0x256 [21552.623590] ? do_raw_spin_unlock+0x75/0x190 [21552.625052] ? _raw_spin_unlock+0x12/0x30 [21552.626853] do_sys_openat2+0x2b4/0x410 [21552.629392] do_sys_open+0x73/0xa0 [21552.630648] __x64_sys_openat+0x24/0x30 [21552.632218] do_syscall_64+0xc1/0x440 [21552.633616] entry_SYSCALL_64_after_hwframe+0x49/0xae [21552.635522] RIP: 0033:0x7f53425a5332 [21552.636821] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21552.644296] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21552.646924] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21552.653012] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21552.655833] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21552.658420] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21552.660193] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21572.052765] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [21572.054793] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21572.116615] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21572.125031] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21572.135704] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21572.140082] RIP: 0010:native_safe_halt+0x2e/0x50 [21572.142208] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21572.166994] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21572.176973] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21572.185977] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [21572.194727] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21572.204358] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21572.212225] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21572.222565] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21572.231580] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21572.239586] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21572.248083] Call Trace: [21572.250359] [21572.251441] ? show_regs.cold.9+0x22/0x2f [21572.256694] ? watchdog_timer_fn.cold.6+0x62/0x10a [21572.264089] ? __hrtimer_run_queues+0x1c2/0x4d0 [21572.270575] ? watchdog+0x40/0x40 [21572.272775] ? hrtimer_interrupt+0x128/0x2c0 [21572.276663] ? smp_apic_timer_interrupt+0xa6/0x270 [21572.284249] ? apic_timer_interrupt+0xf/0x20 [21572.292798] [21572.296168] ? native_safe_halt+0x2e/0x50 [21572.300211] kvm_wait+0xcd/0xe0 [21572.302268] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21572.312966] ? newidle_balance.isra.93+0x207/0x580 [21572.316299] do_raw_spin_lock+0x177/0x190 [21572.322441] _raw_spin_lock+0x21/0x30 [21572.327197] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21572.347129] process_one_work+0x2c8/0x700 [21572.354718] worker_thread+0x296/0x6e0 [21572.362699] ? rescuer_thread+0x570/0x570 [21572.371335] kthread+0x1d1/0x200 [21572.377420] ? set_kthread_struct+0x70/0x70 [21572.383926] ret_from_fork+0x1f/0x30 [21575.651132] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18938s! [21575.654634] Showing busy workqueues and worker pools: [21575.656716] workqueue events: flags=0x0 [21575.658430] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21575.661161] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21575.666836] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21575.670000] pending: cpu_latency_work [ptlrpc] [21575.672540] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21575.676509] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21575.680663] workqueue events_long: flags=0x0 [21575.682567] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21575.685544] pending: lu_objects_flush [obdclass] [21575.688318] workqueue events_unbound: flags=0x2 [21575.690394] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21575.693790] pending: flush_memcg_stats_dwork, idle_cull_fn [21575.696463] workqueue events_power_efficient: flags=0x80 [21575.698884] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21575.701738] pending: check_lifetime, do_cache_clean [21575.704065] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21575.707253] pending: neigh_periodic_work, neigh_periodic_work [21575.710025] workqueue mm_percpu_wq: flags=0x8 [21575.711826] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21575.714582] pending: vmstat_update, lru_add_drain_per_cpu [21575.716860] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21575.719328] pending: vmstat_update, lru_add_drain_per_cpu [21575.721919] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21575.724814] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21575.727437] workqueue xprtiod: flags=0xa [21575.729331] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21575.731636] pending: xprt_autoclose, xs_error_handle [21575.733970] workqueue ipv6_addrconf: flags=0x40008 [21575.736088] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21575.738523] pending: addrconf_verify_work [21575.740545] workqueue knbd0-recv: flags=0x1a [21575.742289] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21575.745290] in-flight: 651:recv_work [21575.747155] workqueue ptlrpc_pinger: flags=0x2 [21575.749152] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21575.751844] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21575.754580] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18938s workers=4 idle: 15979 31824 8 [21575.758158] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21566s workers=2 idle: 654 [21576.058111] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [21576.061794] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21576.081394] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21576.086934] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21576.091070] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21576.093454] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21576.102357] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21576.105844] RAX: 0000000000003299 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21576.109047] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21576.112347] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21576.115633] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21576.119431] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21576.122644] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21576.126531] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21576.128819] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21576.132110] Call Trace: [21576.132943] [21576.133645] ? show_regs.cold.9+0x22/0x2f [21576.134899] ? watchdog_timer_fn.cold.6+0x62/0x10a [21576.136690] ? __hrtimer_run_queues+0x1c2/0x4d0 [21576.138605] ? watchdog+0x40/0x40 [21576.140403] ? hrtimer_interrupt+0x128/0x2c0 [21576.142364] ? smp_apic_timer_interrupt+0xa6/0x270 [21576.144346] ? apic_timer_interrupt+0xf/0x20 [21576.145984] [21576.147027] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21576.149571] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21576.152524] do_raw_spin_lock+0x177/0x190 [21576.154783] _raw_spin_lock+0x21/0x30 [21576.156235] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21576.159480] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21576.162200] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21576.164848] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21576.167435] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21576.170003] ? woken_wake_function+0x30/0x30 [21576.172287] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21576.175489] kthread+0x1d1/0x200 [21576.176873] ? set_kthread_struct+0x70/0x70 [21576.179156] ret_from_fork+0x1f/0x30 [21580.056373] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [21580.057676] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [21580.060209] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21580.060289] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21580.060295] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21580.060298] RIP: 0010:native_safe_halt+0x2e/0x50 [21580.060315] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21580.060320] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21580.060325] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21580.060327] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21580.060330] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21580.060332] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21580.060334] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21580.060337] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21580.060340] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21580.060342] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21580.060350] Call Trace: [21580.060355] [21580.060361] ? show_regs.cold.9+0x22/0x2f [21580.060372] ? watchdog_timer_fn.cold.6+0x62/0x10a [21580.060387] ? __hrtimer_run_queues+0x1c2/0x4d0 [21580.060392] ? watchdog+0x40/0x40 [21580.060396] ? hrtimer_interrupt+0x128/0x2c0 [21580.060400] ? smp_apic_timer_interrupt+0xa6/0x270 [21580.060407] ? apic_timer_interrupt+0xf/0x20 [21580.060416] [21580.060419] ? native_safe_halt+0x2e/0x50 [21580.060424] kvm_wait+0xcd/0xe0 [21580.060431] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21580.060439] do_raw_spin_lock+0x177/0x190 [21580.060444] _raw_spin_lock+0x21/0x30 [21580.060449] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21580.060879] ? do_raw_spin_unlock+0x75/0x190 [21580.060886] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21580.064919] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21580.065006] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21580.065013] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21580.065016] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21580.065193] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21580.065197] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21580.065202] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21580.065204] RDX: 000000000000132e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21580.065206] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21580.065208] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21580.065210] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21580.065213] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21580.065216] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21580.065218] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21580.065225] Call Trace: [21580.065230] [21580.065234] ? show_regs.cold.9+0x22/0x2f [21580.065243] ? watchdog_timer_fn.cold.6+0x62/0x10a [21580.065249] ? __hrtimer_run_queues+0x1c2/0x4d0 [21580.065253] ? watchdog+0x40/0x40 [21580.065256] ? hrtimer_interrupt+0x128/0x2c0 [21580.065260] ? smp_apic_timer_interrupt+0xa6/0x270 [21580.065268] ? apic_timer_interrupt+0xf/0x20 [21580.065272] [21580.065275] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21580.065279] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21580.065284] do_raw_spin_lock+0x177/0x190 [21580.065288] _raw_spin_lock+0x21/0x30 [21580.065294] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21580.065752] obd_mod_free+0x46/0x140 [ptlrpc] [21580.100245] ? do_wait_intr+0xf0/0xf0 [21580.100265] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21580.100522] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21580.100749] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21580.100967] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21580.101079] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21580.101118] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21580.101147] mdc_intent_lock+0x353/0x7e0 [mdc] [21580.101175] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21580.101264] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21580.101481] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21580.101511] lmv_intent_open+0x338/0x1150 [lmv] [21580.101542] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21580.101884] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21580.102061] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21580.102086] ll_intent_lock+0xc5/0xaa0 [lustre] [21580.102164] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21580.102248] ll_lookup_it+0xe0e/0x2270 [lustre] [21580.102337] ? do_raw_spin_unlock+0x75/0x190 [21580.102345] ? _raw_spin_unlock+0x12/0x30 [21580.102352] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21580.102378] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21580.102397] ll_atomic_open+0x36a/0xd70 [lustre] [21580.102479] lookup_open+0x356/0xb40 [21580.102489] do_last+0x313/0xfc0 [21580.102495] ? nd_jump_root+0xe5/0x160 [21580.102499] ? path_init+0x437/0x520 [21580.102504] path_openat+0xf7/0x500 [21580.102509] ? do_raw_spin_unlock+0x75/0x190 [21580.102515] do_filp_open+0x99/0x140 [21580.102522] ? getname_flags+0x6e/0x330 [21580.102526] ? __check_object_size+0xff/0x256 [21580.102533] ? do_raw_spin_unlock+0x75/0x190 [21580.102538] ? _raw_spin_unlock+0x12/0x30 [21580.102542] do_sys_openat2+0x2b4/0x410 [21580.102547] do_sys_open+0x73/0xa0 [21580.102552] __x64_sys_openat+0x24/0x30 [21580.102555] do_syscall_64+0xc1/0x440 [21580.102562] entry_SYSCALL_64_after_hwframe+0x49/0xae [21580.102569] RIP: 0033:0x7f53425a5332 [21580.103112] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21580.112517] mdc_commit_open+0x91/0xb0 [mdc] [21580.112563] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21580.119502] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21580.119516] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21580.119518] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21580.119521] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21580.119523] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21580.119526] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21580.483697] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21580.486048] after_reply+0xe2c/0x1780 [ptlrpc] [21580.488201] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21580.490548] ? __next_timer_interrupt+0x160/0x160 [21580.493221] ptlrpcd+0xa82/0xe40 [ptlrpc] [21580.495416] ? do_wait_intr+0xf0/0xf0 [21580.497062] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21580.499659] kthread+0x1d1/0x200 [21580.502781] ? set_kthread_struct+0x70/0x70 [21580.504447] ret_from_fork+0x1f/0x30 [21594.909979] rcu: INFO: rcu_sched self-detected stall on CPU [21594.918307] rcu: 2-....: (11229800 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [21594.924755] (t=19500483 jiffies g=274185 q=68913) [21594.936569] rcu: rcu_sched kthread starved for 18958849 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=3 [21594.944019] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [21594.952083] rcu: RCU grace-period kthread stack dump: [21594.955951] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [21594.960235] Call Trace: [21594.964327] __schedule+0x351/0xcb0 [21594.967096] schedule+0xc0/0x180 [21594.977897] schedule_timeout+0xb4/0x190 [21594.979676] ? do_raw_spin_unlock+0x75/0x190 [21594.983274] ? __next_timer_interrupt+0x160/0x160 [21594.988006] ? prepare_to_swait_event+0xf3/0x140 [21594.999614] rcu_gp_kthread+0x8dd/0x15b0 [21595.001581] ? force_qs_rnp+0x2c0/0x2c0 [21595.006430] kthread+0x1d1/0x200 [21595.011778] ? set_kthread_struct+0x70/0x70 [21595.017882] ret_from_fork+0x1f/0x30 [21595.021090] rcu: Stack dump where RCU GP kthread last ran: [21595.034497] Sending NMI from CPU 2 to CPUs 3: [21595.035662] NMI backtrace for cpu 3 skipped: idling at native_safe_halt+0x2e/0x50 [21595.037112] Sending NMI from CPU 2 to CPUs 1: [21595.049598] NMI backtrace for cpu 1 [21595.049606] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21595.049608] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21595.049609] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21595.049611] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21595.049613] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [21595.049615] RAX: 0000000000002587 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21595.049617] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21595.049618] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21595.049620] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21595.049621] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21595.049622] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21595.049624] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21595.049625] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21595.049626] Call Trace: [21595.049627] [21595.049628] ? show_regs.cold.9+0x22/0x2f [21595.049629] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [21595.049630] ? nmi_cpu_backtrace_handler+0x15/0x40 [21595.049631] ? nmi_handle+0xbd/0x210 [21595.049632] ? default_do_nmi+0x4b/0x1b0 [21595.049632] ? do_nmi+0x280/0x380 [21595.049633] ? end_repeat_nmi+0x16/0x1f [21595.049828] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21595.049829] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21595.049831] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21595.049832] [21595.049833] do_raw_spin_lock+0x177/0x190 [21595.049834] _raw_spin_lock+0x21/0x30 [21595.049835] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21595.049836] ? do_raw_spin_unlock+0x75/0x190 [21595.049837] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21595.049838] ? do_wait_intr+0xf0/0xf0 [21595.049839] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21595.049840] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21595.049841] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21595.049842] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21595.049844] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21595.049845] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21595.049846] mdc_intent_lock+0x353/0x7e0 [mdc] [21595.049847] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21595.049848] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21595.049849] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21595.049850] lmv_intent_open+0x338/0x1150 [lmv] [21595.049851] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21595.049853] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21595.049854] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21595.049855] ll_intent_lock+0xc5/0xaa0 [lustre] [21595.049856] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21595.049857] ll_lookup_it+0xe0e/0x2270 [lustre] [21595.049859] ? do_raw_spin_unlock+0x75/0x190 [21595.049860] ? _raw_spin_unlock+0x12/0x30 [21595.049861] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21595.049862] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21595.049864] ll_atomic_open+0x36a/0xd70 [lustre] [21595.049864] lookup_open+0x356/0xb40 [21595.049866] do_last+0x313/0xfc0 [21595.049867] ? nd_jump_root+0xe5/0x160 [21595.049868] ? path_init+0x437/0x520 [21595.049869] path_openat+0xf7/0x500 [21595.049870] ? do_raw_spin_unlock+0x75/0x190 [21595.049871] do_filp_open+0x99/0x140 [21595.049872] ? getname_flags+0x6e/0x330 [21595.049873] ? __check_object_size+0xff/0x256 [21595.049874] ? do_raw_spin_unlock+0x75/0x190 [21595.049875] ? _raw_spin_unlock+0x12/0x30 [21595.049876] do_sys_openat2+0x2b4/0x410 [21595.049877] do_sys_open+0x73/0xa0 [21595.049879] __x64_sys_openat+0x24/0x30 [21595.049880] do_syscall_64+0xc1/0x440 [21595.050086] entry_SYSCALL_64_after_hwframe+0x49/0xae [21595.050088] RIP: 0033:0x7f53425a5332 [21595.050090] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21595.050092] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21595.050094] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21595.050096] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21595.050097] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21595.050098] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21595.050100] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21595.054387] NMI backtrace for cpu 2 [21595.289533] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21595.292526] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21595.294890] Call Trace: [21595.295523] [21595.295968] ? dump_stack+0xbb/0x10e [21595.297025] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [21595.298264] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [21595.299436] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [21595.300762] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [21595.302042] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [21595.303480] ? rcu_dump_cpu_stacks+0x11b/0x171 [21595.304852] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [21595.306013] ? update_process_times+0x6d/0xc0 [21595.307075] ? tick_sched_handle.isra.18+0x27/0x90 [21595.308438] ? tick_sched_timer+0x53/0xb0 [21595.309451] ? __hrtimer_run_queues+0x1c2/0x4d0 [21595.310596] ? tick_sched_handle.isra.18+0x90/0x90 [21595.311783] ? hrtimer_interrupt+0x128/0x2c0 [21595.312936] ? smp_apic_timer_interrupt+0xa6/0x270 [21595.314035] ? apic_timer_interrupt+0xf/0x20 [21595.315057] [21595.315539] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21595.316894] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21595.318127] ? do_raw_spin_lock+0x177/0x190 [21595.319147] ? _raw_spin_lock+0x21/0x30 [21595.320046] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21595.321412] ? obd_mod_free+0x46/0x140 [ptlrpc] [21595.322571] ? mdc_commit_open+0x91/0xb0 [mdc] [21595.323558] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21595.325056] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21595.327184] ? after_reply+0xe2c/0x1780 [ptlrpc] [21595.328835] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21595.330162] ? __next_timer_interrupt+0x160/0x160 [21595.331489] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [21595.332813] ? do_wait_intr+0xf0/0xf0 [21595.333773] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21595.335294] ? kthread+0x1d1/0x200 [21595.336282] ? set_kthread_struct+0x70/0x70 [21595.337272] ? ret_from_fork+0x1f/0x30 [21600.052234] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [21600.056008] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21600.094913] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21600.109905] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21600.119111] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21600.124144] RIP: 0010:native_safe_halt+0x2e/0x50 [21600.127059] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21600.146101] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21600.149813] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21600.153204] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [21600.160143] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21600.168052] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21600.177904] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21600.185079] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21600.192219] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21600.197855] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21600.204529] Call Trace: [21600.208887] [21600.211860] ? show_regs.cold.9+0x22/0x2f [21600.214786] ? watchdog_timer_fn.cold.6+0x62/0x10a [21600.219104] ? __hrtimer_run_queues+0x1c2/0x4d0 [21600.221927] ? watchdog+0x40/0x40 [21600.223546] ? hrtimer_interrupt+0x128/0x2c0 [21600.225536] ? smp_apic_timer_interrupt+0xa6/0x270 [21600.229804] ? apic_timer_interrupt+0xf/0x20 [21600.232979] [21600.233736] ? native_safe_halt+0x2e/0x50 [21600.235104] kvm_wait+0xcd/0xe0 [21600.237123] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21600.242141] ? newidle_balance.isra.93+0x207/0x580 [21600.244268] do_raw_spin_lock+0x177/0x190 [21600.246274] _raw_spin_lock+0x21/0x30 [21600.247987] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21600.256264] process_one_work+0x2c8/0x700 [21600.257405] worker_thread+0x296/0x6e0 [21600.261572] ? rescuer_thread+0x570/0x570 [21600.263001] kthread+0x1d1/0x200 [21600.264385] ? set_kthread_struct+0x70/0x70 [21600.266789] ret_from_fork+0x1f/0x30 [21605.858214] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18968s! [21605.862960] Showing busy workqueues and worker pools: [21605.867492] workqueue events: flags=0x0 [21605.868729] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21605.874552] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21605.881836] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21605.883726] pending: cpu_latency_work [ptlrpc] [21605.888072] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21605.898104] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21605.905718] workqueue events_long: flags=0x0 [21605.908595] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21605.914384] pending: lu_objects_flush [obdclass] [21605.918277] workqueue events_unbound: flags=0x2 [21605.921530] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21605.925409] pending: flush_memcg_stats_dwork, idle_cull_fn [21605.928658] workqueue events_power_efficient: flags=0x80 [21605.934336] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21605.939951] pending: check_lifetime, do_cache_clean [21605.943582] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21605.949841] pending: neigh_periodic_work, neigh_periodic_work [21605.954457] workqueue mm_percpu_wq: flags=0x8 [21605.957590] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21605.964598] pending: vmstat_update, lru_add_drain_per_cpu [21605.970032] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21605.976051] pending: vmstat_update, lru_add_drain_per_cpu [21605.978373] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21605.984338] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21605.989987] workqueue xprtiod: flags=0xa [21605.991847] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21605.995867] pending: xprt_autoclose, xs_error_handle [21606.001532] workqueue ipv6_addrconf: flags=0x40008 [21606.003534] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21606.009215] pending: addrconf_verify_work [21606.011020] workqueue knbd0-recv: flags=0x1a [21606.016026] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21606.018958] in-flight: 651:recv_work [21606.022048] workqueue ptlrpc_pinger: flags=0x2 [21606.025582] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21606.028993] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21606.032901] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18968s workers=4 idle: 15979 31824 8 [21606.036619] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21596s workers=2 idle: 654 [21620.055042] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [21620.057268] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [21620.058501] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) [21620.058692] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [21620.058698] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21620.058758] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21620.058764] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21620.058767] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21620.058782] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21620.058786] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21620.058790] RAX: 0000000000007f1f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21620.058793] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21620.058795] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21620.058797] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21620.058799] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21620.058802] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21620.058805] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21620.058807] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21620.059081] Call Trace: [21620.059088] [21620.059093] ? show_regs.cold.9+0x22/0x2f [21620.059104] ? watchdog_timer_fn.cold.6+0x62/0x10a [21620.059111] ? __hrtimer_run_queues+0x1c2/0x4d0 [21620.059116] ? watchdog+0x40/0x40 [21620.059121] ? hrtimer_interrupt+0x128/0x2c0 [21620.059124] ? smp_apic_timer_interrupt+0xa6/0x270 [21620.059132] ? apic_timer_interrupt+0xf/0x20 [21620.059136] [21620.059139] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21620.059144] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21620.059151] do_raw_spin_lock+0x177/0x190 [21620.059155] _raw_spin_lock+0x21/0x30 [21620.059160] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21620.059428] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21620.059621] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21620.062011] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 [21620.061751] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21620.062711] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21620.062943] ? woken_wake_function+0x30/0x30 [21620.062954] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21620.063166] kthread+0x1d1/0x200 [21620.063172] ? set_kthread_struct+0x70/0x70 [21620.063177] ret_from_fork+0x1f/0x30 [21620.070819] obdclass(O) ksocklnd(O) lnet(O) [21620.074035] auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr [21620.118538] libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21620.118582] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21620.118591] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21620.118594] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21620.118606] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21620.118610] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21620.118615] RAX: 0000000000002912 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21620.118617] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21620.118619] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21620.118621] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21620.118623] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21620.118626] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21620.118629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21620.118631] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21620.118639] Call Trace: [21620.118645] [21620.118650] ? show_regs.cold.9+0x22/0x2f [21620.118659] ? watchdog_timer_fn.cold.6+0x62/0x10a [21620.118665] ? __hrtimer_run_queues+0x1c2/0x4d0 [21620.118669] ? watchdog+0x40/0x40 [21620.118673] ? hrtimer_interrupt+0x128/0x2c0 [21620.118677] ? smp_apic_timer_interrupt+0xa6/0x270 [21620.118685] ? apic_timer_interrupt+0xf/0x20 [21620.118689] [21620.118692] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21620.118699] do_raw_spin_lock+0x177/0x190 [21620.118703] _raw_spin_lock+0x21/0x30 [21620.118708] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21620.128057] squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21620.128094] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21620.128101] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21620.128104] RIP: 0010:native_safe_halt+0x2e/0x50 [21620.128124] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21620.128128] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21620.128132] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21620.128135] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21620.128138] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21620.128140] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21620.128143] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21620.128146] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21620.128149] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21620.128151] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21620.128159] Call Trace: [21620.128165] [21620.128171] ? show_regs.cold.9+0x22/0x2f [21620.128180] ? watchdog_timer_fn.cold.6+0x62/0x10a [21620.128186] ? __hrtimer_run_queues+0x1c2/0x4d0 [21620.128191] ? watchdog+0x40/0x40 [21620.128196] ? hrtimer_interrupt+0x128/0x2c0 [21620.128200] ? smp_apic_timer_interrupt+0xa6/0x270 [21620.128207] ? apic_timer_interrupt+0xf/0x20 [21620.128212] [21620.128215] ? native_safe_halt+0x2e/0x50 [21620.128219] kvm_wait+0xcd/0xe0 [21620.128226] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21620.128232] do_raw_spin_lock+0x177/0x190 [21620.128237] _raw_spin_lock+0x21/0x30 [21620.128243] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21620.128745] obd_mod_free+0x46/0x140 [ptlrpc] [21620.129043] mdc_commit_open+0x91/0xb0 [mdc] [21620.129079] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21620.129415] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21620.129617] after_reply+0xe2c/0x1780 [ptlrpc] [21620.129884] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21620.136480] ? do_raw_spin_unlock+0x75/0x190 [21620.136501] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21620.136753] ? do_wait_intr+0xf0/0xf0 [21620.136836] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21620.137628] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21620.140703] ? __next_timer_interrupt+0x160/0x160 [21620.140723] ptlrpcd+0xa82/0xe40 [ptlrpc] [21620.142046] ? do_wait_intr+0xf0/0xf0 [21620.142058] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21620.160613] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21620.161644] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21620.162506] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21620.162549] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21620.162582] mdc_intent_lock+0x353/0x7e0 [mdc] [21620.162621] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21620.162714] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21620.163751] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21620.163862] lmv_intent_open+0x338/0x1150 [lmv] [21620.164001] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21620.164101] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21620.165054] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21620.165081] ll_intent_lock+0xc5/0xaa0 [lustre] [21620.165656] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21620.165739] ll_lookup_it+0xe0e/0x2270 [lustre] [21620.166592] ? do_raw_spin_unlock+0x75/0x190 [21620.166838] ? _raw_spin_unlock+0x12/0x30 [21620.166846] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21620.166871] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21620.167091] ll_atomic_open+0x36a/0xd70 [lustre] [21620.167476] lookup_open+0x356/0xb40 [21620.167486] do_last+0x313/0xfc0 [21620.167491] ? nd_jump_root+0xe5/0x160 [21620.167496] ? path_init+0x437/0x520 [21620.167500] path_openat+0xf7/0x500 [21620.167505] ? do_raw_spin_unlock+0x75/0x190 [21620.167509] do_filp_open+0x99/0x140 [21620.167516] ? getname_flags+0x6e/0x330 [21620.167520] ? __check_object_size+0xff/0x256 [21620.167528] ? do_raw_spin_unlock+0x75/0x190 [21620.167532] ? _raw_spin_unlock+0x12/0x30 [21620.167537] do_sys_openat2+0x2b4/0x410 [21620.167543] do_sys_open+0x73/0xa0 [21620.167547] __x64_sys_openat+0x24/0x30 [21620.168480] do_syscall_64+0xc1/0x440 [21620.168487] entry_SYSCALL_64_after_hwframe+0x49/0xae [21620.168494] RIP: 0033:0x7f53425a5332 [21620.168502] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21620.168506] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21620.168512] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21620.168515] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21620.168517] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21620.168519] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21620.168522] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21620.912934] kthread+0x1d1/0x200 [21620.915621] ? set_kthread_struct+0x70/0x70 [21620.918401] ret_from_fork+0x1f/0x30 [21632.052996] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [21632.060923] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21632.107292] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21632.121930] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21632.128484] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21632.139350] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21632.149521] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21632.176232] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21632.185700] RAX: 0000000000002b1b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21632.197197] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [21632.203329] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21632.209249] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21632.216118] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21632.222001] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21632.228215] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21632.235118] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21632.240453] Call Trace: [21632.241667] [21632.243000] ? show_regs.cold.9+0x22/0x2f [21632.244804] ? watchdog_timer_fn.cold.6+0x62/0x10a [21632.247134] ? __hrtimer_run_queues+0x1c2/0x4d0 [21632.249195] ? watchdog+0x40/0x40 [21632.250841] ? hrtimer_interrupt+0x128/0x2c0 [21632.252912] ? smp_apic_timer_interrupt+0xa6/0x270 [21632.255421] ? apic_timer_interrupt+0xf/0x20 [21632.258148] [21632.259644] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21632.262638] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21632.266436] ? newidle_balance.isra.93+0x207/0x580 [21632.270400] do_raw_spin_lock+0x177/0x190 [21632.272928] _raw_spin_lock+0x21/0x30 [21632.275584] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21632.279195] process_one_work+0x2c8/0x700 [21632.282039] worker_thread+0x296/0x6e0 [21632.284479] ? rescuer_thread+0x570/0x570 [21632.286967] kthread+0x1d1/0x200 [21632.289445] ? set_kthread_struct+0x70/0x70 [21632.292486] ret_from_fork+0x1f/0x30 [21636.064119] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 18998s! [21636.066884] Showing busy workqueues and worker pools: [21636.069024] workqueue events: flags=0x0 [21636.070542] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21636.073460] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21636.078357] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21636.081986] pending: cpu_latency_work [ptlrpc] [21636.084530] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21636.088472] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21636.103922] workqueue events_long: flags=0x0 [21636.106279] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21636.113138] pending: lu_objects_flush [obdclass] [21636.118207] workqueue events_unbound: flags=0x2 [21636.121059] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21636.124800] pending: flush_memcg_stats_dwork, idle_cull_fn [21636.128346] workqueue events_power_efficient: flags=0x80 [21636.131113] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21636.135773] pending: check_lifetime, do_cache_clean [21636.138795] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21636.142090] pending: neigh_periodic_work, neigh_periodic_work [21636.145366] workqueue mm_percpu_wq: flags=0x8 [21636.148527] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21636.156669] pending: vmstat_update, lru_add_drain_per_cpu [21636.160099] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21636.166537] pending: vmstat_update, lru_add_drain_per_cpu [21636.173583] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21636.183762] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21636.189320] workqueue xprtiod: flags=0xa [21636.194227] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21636.204853] pending: xprt_autoclose, xs_error_handle [21636.208158] workqueue ipv6_addrconf: flags=0x40008 [21636.212076] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21636.218341] pending: addrconf_verify_work [21636.224803] workqueue knbd0-recv: flags=0x1a [21636.229347] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21636.238785] in-flight: 651:recv_work [21636.242911] workqueue ptlrpc_pinger: flags=0x2 [21636.248384] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21636.250578] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21636.261317] pool 8: cpus=0-3 flags=0x4 nice=0 hung=18999s workers=4 idle: 15979 31824 8 [21636.270842] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21626s workers=2 idle: 654 [21648.056749] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [21648.057031] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [21648.062502] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [21648.062515] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21648.062599] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21648.062605] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21648.062608] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21648.062622] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21648.062639] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21648.062644] RAX: 000000000000620c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21648.062646] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21648.062649] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21648.062651] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21648.062653] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21648.062656] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21648.062659] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21648.062661] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21648.062669] Call Trace: [21648.062674] [21648.062679] ? show_regs.cold.9+0x22/0x2f [21648.062688] ? watchdog_timer_fn.cold.6+0x62/0x10a [21648.062694] ? __hrtimer_run_queues+0x1c2/0x4d0 [21648.062698] ? watchdog+0x40/0x40 [21648.062703] ? hrtimer_interrupt+0x128/0x2c0 [21648.062707] ? smp_apic_timer_interrupt+0xa6/0x270 [21648.062715] ? apic_timer_interrupt+0xf/0x20 [21648.062721] [21648.062724] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21648.062728] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21648.062735] do_raw_spin_lock+0x177/0x190 [21648.062739] _raw_spin_lock+0x21/0x30 [21648.062744] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21648.063349] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21648.063781] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21648.064531] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21648.065459] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21648.065705] ? woken_wake_function+0x30/0x30 [21648.065715] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21648.066038] kthread+0x1d1/0x200 [21648.066045] ? set_kthread_struct+0x70/0x70 [21648.066049] ret_from_fork+0x1f/0x30 [21648.066294] Modules linked in: [21648.070176] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21648.070252] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21648.070259] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21648.070262] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21648.070277] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21648.070281] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21648.070285] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21648.070288] RDX: 00000000000014a9 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21648.070290] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21648.070292] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21648.070294] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21648.070297] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21648.070300] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21648.070303] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21648.070312] Call Trace: [21648.070458] [21648.070467] ? show_regs.cold.9+0x22/0x2f [21648.070476] ? watchdog_timer_fn.cold.6+0x62/0x10a [21648.070483] ? __hrtimer_run_queues+0x1c2/0x4d0 [21648.070487] ? watchdog+0x40/0x40 [21648.070491] ? hrtimer_interrupt+0x128/0x2c0 [21648.070495] ? smp_apic_timer_interrupt+0xa6/0x270 [21648.070503] ? apic_timer_interrupt+0xf/0x20 [21648.070509] [21648.070511] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21648.070515] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21648.070520] do_raw_spin_lock+0x177/0x190 [21648.070525] _raw_spin_lock+0x21/0x30 [21648.070533] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21648.070862] obd_mod_free+0x46/0x140 [ptlrpc] [21648.071053] mdc_commit_open+0x91/0xb0 [mdc] [21648.071089] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21648.071272] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21648.071571] after_reply+0xe2c/0x1780 [ptlrpc] [21648.071900] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21648.072089] ? __next_timer_interrupt+0x160/0x160 [21648.072099] ptlrpcd+0xa82/0xe40 [ptlrpc] [21648.072510] ? do_wait_intr+0xf0/0xf0 [21648.072521] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21648.085938] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21648.086013] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21648.086021] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21648.086024] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21648.086488] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21648.086493] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21648.086498] RAX: 0000000000005b14 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21648.086500] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21648.086503] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21648.086505] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21648.086508] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21648.086511] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21648.086514] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21648.086516] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21648.086527] Call Trace: [21648.086532] [21648.086537] ? show_regs.cold.9+0x22/0x2f [21648.086546] ? watchdog_timer_fn.cold.6+0x62/0x10a [21648.086553] ? __hrtimer_run_queues+0x1c2/0x4d0 [21648.086557] ? watchdog+0x40/0x40 [21648.086561] ? hrtimer_interrupt+0x128/0x2c0 [21648.086565] ? smp_apic_timer_interrupt+0xa6/0x270 [21648.086574] ? apic_timer_interrupt+0xf/0x20 [21648.086579] [21648.086582] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21648.086590] do_raw_spin_lock+0x177/0x190 [21648.086594] _raw_spin_lock+0x21/0x30 [21648.086599] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21648.124981] kthread+0x1d1/0x200 [21648.125001] ? set_kthread_struct+0x70/0x70 [21648.125005] ret_from_fork+0x1f/0x30 [21648.863657] ? do_raw_spin_unlock+0x75/0x190 [21648.865473] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21648.867613] ? do_wait_intr+0xf0/0xf0 [21648.868672] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21648.870812] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21648.873036] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21648.875353] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21648.877571] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21648.879799] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21648.882292] mdc_intent_lock+0x353/0x7e0 [mdc] [21648.884586] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21648.887253] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21648.890465] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21648.892535] lmv_intent_open+0x338/0x1150 [lmv] [21648.896568] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21648.898886] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21648.904473] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21648.906667] ll_intent_lock+0xc5/0xaa0 [lustre] [21648.908955] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21648.911601] ll_lookup_it+0xe0e/0x2270 [lustre] [21648.913714] ? do_raw_spin_unlock+0x75/0x190 [21648.917851] ? _raw_spin_unlock+0x12/0x30 [21648.920560] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21648.922907] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21648.925382] ll_atomic_open+0x36a/0xd70 [lustre] [21648.927594] lookup_open+0x356/0xb40 [21648.929520] do_last+0x313/0xfc0 [21648.930870] ? nd_jump_root+0xe5/0x160 [21648.932750] ? path_init+0x437/0x520 [21648.933883] path_openat+0xf7/0x500 [21648.934927] ? do_raw_spin_unlock+0x75/0x190 [21648.936795] do_filp_open+0x99/0x140 [21648.938692] ? getname_flags+0x6e/0x330 [21648.940919] ? __check_object_size+0xff/0x256 [21648.942632] ? do_raw_spin_unlock+0x75/0x190 [21648.945785] ? _raw_spin_unlock+0x12/0x30 [21648.947786] do_sys_openat2+0x2b4/0x410 [21648.950111] do_sys_open+0x73/0xa0 [21648.951845] __x64_sys_openat+0x24/0x30 [21648.953730] do_syscall_64+0xc1/0x440 [21648.955965] entry_SYSCALL_64_after_hwframe+0x49/0xae [21648.958247] RIP: 0033:0x7f53425a5332 [21648.959192] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21648.967169] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21648.970676] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21648.976853] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21648.979210] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21648.982818] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21648.990367] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21664.052029] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [21664.056422] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21664.074411] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21664.078887] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21664.082750] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21664.085558] RIP: 0010:native_safe_halt+0x2e/0x50 [21664.087399] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21664.094983] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21664.098192] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21664.100106] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [21664.106297] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21664.110017] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21664.113406] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21664.117403] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21664.120476] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21664.123346] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21664.126570] Call Trace: [21664.127611] [21664.129331] ? show_regs.cold.9+0x22/0x2f [21664.132073] ? watchdog_timer_fn.cold.6+0x62/0x10a [21664.134315] ? __hrtimer_run_queues+0x1c2/0x4d0 [21664.136396] ? watchdog+0x40/0x40 [21664.137799] ? hrtimer_interrupt+0x128/0x2c0 [21664.139351] ? smp_apic_timer_interrupt+0xa6/0x270 [21664.140834] ? apic_timer_interrupt+0xf/0x20 [21664.142115] [21664.143423] ? native_safe_halt+0x2e/0x50 [21664.144829] kvm_wait+0xcd/0xe0 [21664.146218] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21664.150880] ? newidle_balance.isra.93+0x207/0x580 [21664.152867] do_raw_spin_lock+0x177/0x190 [21664.153826] _raw_spin_lock+0x21/0x30 [21664.154695] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21664.158631] process_one_work+0x2c8/0x700 [21664.159512] worker_thread+0x296/0x6e0 [21664.160382] ? rescuer_thread+0x570/0x570 [21664.166300] kthread+0x1d1/0x200 [21664.167387] ? set_kthread_struct+0x70/0x70 [21664.168875] ret_from_fork+0x1f/0x30 [21666.786143] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 19029s! [21666.798044] Showing busy workqueues and worker pools: [21666.800614] workqueue events: flags=0x0 [21666.804239] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21666.817950] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21666.823362] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21666.827608] pending: cpu_latency_work [ptlrpc] [21666.833282] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21666.839033] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21666.848669] workqueue events_long: flags=0x0 [21666.850755] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21666.857159] pending: lu_objects_flush [obdclass] [21666.859823] workqueue events_unbound: flags=0x2 [21666.863954] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21666.867353] pending: flush_memcg_stats_dwork, idle_cull_fn [21666.870286] workqueue events_power_efficient: flags=0x80 [21666.875358] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21666.880539] pending: check_lifetime, do_cache_clean [21666.887726] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21666.891015] pending: neigh_periodic_work, neigh_periodic_work [21666.895779] workqueue mm_percpu_wq: flags=0x8 [21666.897448] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21666.904815] pending: vmstat_update, lru_add_drain_per_cpu [21666.908323] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21666.910925] pending: vmstat_update, lru_add_drain_per_cpu [21666.913753] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21666.919790] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21666.922941] workqueue xprtiod: flags=0xa [21666.928023] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21666.931219] pending: xprt_autoclose, xs_error_handle [21666.934018] workqueue ipv6_addrconf: flags=0x40008 [21666.939938] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21666.942887] pending: addrconf_verify_work [21666.945159] workqueue knbd0-recv: flags=0x1a [21666.951769] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21666.956740] in-flight: 651:recv_work [21666.961624] workqueue ptlrpc_pinger: flags=0x2 [21666.963314] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21666.965953] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21666.968651] pool 8: cpus=0-3 flags=0x4 nice=0 hung=19029s workers=4 idle: 15979 31824 8 [21666.980904] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21657s workers=2 idle: 654 [21676.055011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [21676.057006] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [21676.058015] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [21676.058022] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21676.058157] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21676.058172] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21676.058175] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21676.058189] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21676.058194] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21676.058198] RAX: 0000000000005aaa RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21676.058200] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21676.058203] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21676.058204] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21676.058207] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21676.058210] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21676.058212] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21676.058214] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21676.058223] Call Trace: [21676.058228] [21676.058234] ? show_regs.cold.9+0x22/0x2f [21676.058244] ? watchdog_timer_fn.cold.6+0x62/0x10a [21676.058252] ? __hrtimer_run_queues+0x1c2/0x4d0 [21676.058255] ? watchdog+0x40/0x40 [21676.058260] ? hrtimer_interrupt+0x128/0x2c0 [21676.058264] ? smp_apic_timer_interrupt+0xa6/0x270 [21676.058272] ? apic_timer_interrupt+0xf/0x20 [21676.058277] [21676.058280] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21676.058285] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21676.058291] do_raw_spin_lock+0x177/0x190 [21676.058449] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21676.058510] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21676.058515] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21676.058517] RIP: 0010:__pv_queued_spin_lock_slowpath+0x320/0x6d0 [21676.058525] Code: 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 <48> 83 05 18 d4 dc 02 01 eb e0 48 83 05 06 d4 dc 02 01 48 83 05 be [21676.058529] RSP: 0018:ffffbb5381503500 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13 [21676.058533] RAX: 000000000000568c RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21676.058536] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21676.058538] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21676.058540] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21676.058543] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21676.058545] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21676.058554] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21676.058556] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21676.058561] Call Trace: [21676.058564] [21676.058567] ? show_regs.cold.9+0x22/0x2f [21676.058573] ? watchdog_timer_fn.cold.6+0x62/0x10a [21676.058582] ? __hrtimer_run_queues+0x1c2/0x4d0 [21676.058585] ? watchdog+0x40/0x40 [21676.058589] ? hrtimer_interrupt+0x128/0x2c0 [21676.058593] ? smp_apic_timer_interrupt+0xa6/0x270 [21676.058597] ? apic_timer_interrupt+0xf/0x20 [21676.058602] [21676.058604] ? __pv_queued_spin_lock_slowpath+0x320/0x6d0 [21676.058610] do_raw_spin_lock+0x177/0x190 [21676.058615] _raw_spin_lock+0x21/0x30 [21676.058619] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21676.058998] _raw_spin_lock+0x21/0x30 [21676.059009] ? do_raw_spin_unlock+0x75/0x190 [21676.059019] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21676.059418] ? do_wait_intr+0xf0/0xf0 [21676.059426] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21676.059761] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21676.059009] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21676.060132] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21676.060508] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21676.060688] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21676.060727] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21676.060757] mdc_intent_lock+0x353/0x7e0 [mdc] [21676.060793] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21676.060884] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21676.061336] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21676.061367] lmv_intent_open+0x338/0x1150 [lmv] [21676.061399] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21676.061489] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21676.061573] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21676.061595] ll_intent_lock+0xc5/0xaa0 [lustre] [21676.061810] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21676.061890] ll_lookup_it+0xe0e/0x2270 [lustre] [21676.062006] ? do_raw_spin_unlock+0x75/0x190 [21676.062012] ? _raw_spin_unlock+0x12/0x30 [21676.062019] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21676.062047] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21676.062141] ll_atomic_open+0x36a/0xd70 [lustre] [21676.062246] lookup_open+0x356/0xb40 [21676.062257] do_last+0x313/0xfc0 [21676.062262] ? nd_jump_root+0xe5/0x160 [21676.062267] ? path_init+0x437/0x520 [21676.062272] path_openat+0xf7/0x500 [21676.062277] ? do_raw_spin_unlock+0x75/0x190 [21676.062281] do_filp_open+0x99/0x140 [21676.062288] ? getname_flags+0x6e/0x330 [21676.062293] ? __check_object_size+0xff/0x256 [21676.062300] ? do_raw_spin_unlock+0x75/0x190 [21676.062304] ? _raw_spin_unlock+0x12/0x30 [21676.062309] do_sys_openat2+0x2b4/0x410 [21676.062314] do_sys_open+0x73/0xa0 [21676.062318] __x64_sys_openat+0x24/0x30 [21676.062321] do_syscall_64+0xc1/0x440 [21676.062329] entry_SYSCALL_64_after_hwframe+0x49/0xae [21676.062335] RIP: 0033:0x7f53425a5332 [21676.062418] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21676.062423] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21676.062428] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21676.062430] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21676.062432] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21676.062434] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21676.062437] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21676.065035] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21676.065122] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21676.065129] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21676.065132] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21676.065143] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21676.072622] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21676.073015] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21676.104041] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21676.104056] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21676.104059] RDX: 0000000000000ce6 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21676.104061] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21676.104064] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21676.104066] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21676.104070] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21676.104073] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21676.104076] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21676.104084] Call Trace: [21676.104095] [21676.104102] ? show_regs.cold.9+0x22/0x2f [21676.104114] ? watchdog_timer_fn.cold.6+0x62/0x10a [21676.104121] ? __hrtimer_run_queues+0x1c2/0x4d0 [21676.104126] ? watchdog+0x40/0x40 [21676.104130] ? hrtimer_interrupt+0x128/0x2c0 [21676.104134] ? smp_apic_timer_interrupt+0xa6/0x270 [21676.104142] ? apic_timer_interrupt+0xf/0x20 [21676.104147] [21676.104150] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21676.104157] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21676.104161] do_raw_spin_lock+0x177/0x190 [21676.104166] _raw_spin_lock+0x21/0x30 [21676.104173] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21676.104814] obd_mod_free+0x46/0x140 [ptlrpc] [21676.120107] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21676.120499] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21676.120864] ? woken_wake_function+0x30/0x30 [21676.120874] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21676.121114] kthread+0x1d1/0x200 [21676.121125] ? set_kthread_struct+0x70/0x70 [21676.121129] ret_from_fork+0x1f/0x30 [21676.563120] mdc_commit_open+0x91/0xb0 [mdc] [21676.565411] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21676.567953] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21676.570079] after_reply+0xe2c/0x1780 [ptlrpc] [21676.572036] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21676.574258] ? __next_timer_interrupt+0x160/0x160 [21676.575722] ptlrpcd+0xa82/0xe40 [ptlrpc] [21676.577381] ? do_wait_intr+0xf0/0xf0 [21676.578661] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21676.580561] kthread+0x1d1/0x200 [21676.582010] ? set_kthread_struct+0x70/0x70 [21676.583443] ret_from_fork+0x1f/0x30 [21692.052023] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [21692.055981] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21692.071939] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21692.075520] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21692.078911] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21692.081440] RIP: 0010:native_safe_halt+0x2e/0x50 [21692.083385] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21692.091210] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21692.094293] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21692.097101] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [21692.100162] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21692.102909] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21692.105873] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21692.108933] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21692.112267] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21692.114462] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21692.117446] Call Trace: [21692.118191] [21692.118852] ? show_regs.cold.9+0x22/0x2f [21692.120339] ? watchdog_timer_fn.cold.6+0x62/0x10a [21692.122296] ? __hrtimer_run_queues+0x1c2/0x4d0 [21692.124218] ? watchdog+0x40/0x40 [21692.125782] ? hrtimer_interrupt+0x128/0x2c0 [21692.127482] ? smp_apic_timer_interrupt+0xa6/0x270 [21692.129811] ? apic_timer_interrupt+0xf/0x20 [21692.131532] [21692.132478] ? native_safe_halt+0x2e/0x50 [21692.134077] kvm_wait+0xcd/0xe0 [21692.135360] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21692.137736] ? newidle_balance.isra.93+0x207/0x580 [21692.139437] do_raw_spin_lock+0x177/0x190 [21692.141197] _raw_spin_lock+0x21/0x30 [21692.142818] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21692.145252] process_one_work+0x2c8/0x700 [21692.146973] worker_thread+0x296/0x6e0 [21692.148427] ? rescuer_thread+0x570/0x570 [21692.150205] kthread+0x1d1/0x200 [21692.151512] ? set_kthread_struct+0x70/0x70 [21692.153210] ret_from_fork+0x1f/0x30 [21696.993580] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 19059s! [21697.000318] Showing busy workqueues and worker pools: [21697.004450] workqueue events: flags=0x0 [21697.010520] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21697.018317] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21697.032494] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21697.040596] pending: cpu_latency_work [ptlrpc] [21697.045205] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21697.048881] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21697.056462] workqueue events_long: flags=0x0 [21697.058973] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21697.066741] pending: lu_objects_flush [obdclass] [21697.072154] workqueue events_unbound: flags=0x2 [21697.075753] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21697.079477] pending: flush_memcg_stats_dwork, idle_cull_fn [21697.085317] workqueue events_power_efficient: flags=0x80 [21697.090825] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21697.097458] pending: check_lifetime, do_cache_clean [21697.101202] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21697.106950] pending: neigh_periodic_work, neigh_periodic_work [21697.116475] workqueue mm_percpu_wq: flags=0x8 [21697.120626] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21697.129385] pending: vmstat_update, lru_add_drain_per_cpu [21697.133142] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21697.137604] pending: vmstat_update, lru_add_drain_per_cpu [21697.144450] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21697.147378] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21697.153800] workqueue xprtiod: flags=0xa [21697.155073] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21697.158556] pending: xprt_autoclose, xs_error_handle [21697.161023] workqueue ipv6_addrconf: flags=0x40008 [21697.163073] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21697.167468] pending: addrconf_verify_work [21697.169961] workqueue knbd0-recv: flags=0x1a [21697.171234] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21697.177351] in-flight: 651:recv_work [21697.179819] workqueue ptlrpc_pinger: flags=0x2 [21697.181286] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21697.184438] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21697.187148] pool 8: cpus=0-3 flags=0x4 nice=0 hung=19060s workers=4 idle: 15979 31824 8 [21697.190323] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21687s workers=2 idle: 654 [21704.055037] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [21704.057051] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [21704.058013] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [21704.058019] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21704.058088] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21704.058093] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21704.058096] RIP: 0010:native_safe_halt+0x2e/0x50 [21704.058110] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21704.058113] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21704.058117] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21704.058120] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21704.058122] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21704.058124] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21704.058126] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21704.058129] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21704.058132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21704.058134] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21704.058140] Call Trace: [21704.058146] [21704.058150] ? show_regs.cold.9+0x22/0x2f [21704.058158] ? watchdog_timer_fn.cold.6+0x62/0x10a [21704.058164] ? __hrtimer_run_queues+0x1c2/0x4d0 [21704.058167] ? watchdog+0x40/0x40 [21704.058171] ? hrtimer_interrupt+0x128/0x2c0 [21704.058175] ? smp_apic_timer_interrupt+0xa6/0x270 [21704.058181] ? apic_timer_interrupt+0xf/0x20 [21704.058185] [21704.058187] ? native_safe_halt+0x2e/0x50 [21704.058191] kvm_wait+0xcd/0xe0 [21704.058198] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21704.058206] do_raw_spin_lock+0x177/0x190 [21704.058210] _raw_spin_lock+0x21/0x30 [21704.058213] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21704.058507] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21704.058984] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21704.059199] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21704.059431] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21704.059655] ? woken_wake_function+0x30/0x30 [21704.059663] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21704.059876] kthread+0x1d1/0x200 [21704.059881] ? set_kthread_struct+0x70/0x70 [21704.059884] ret_from_fork+0x1f/0x30 [21704.062165] Modules linked in: lustre(O) [21704.065256] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [21704.068103] osp(O) [21704.082369] mdt(O) [21704.088105] ofd(O) [21704.090789] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common [21704.092822] lod(O) [21704.098749] sb_edac [21704.102346] mdt(O) [21704.104630] rapl [21704.107063] mdd(O) [21704.109156] i2c_piix4 [21704.113202] mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl [21704.115492] pcspkr [21704.119685] i2c_piix4 [21704.121755] squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21704.121783] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21704.121789] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21704.121791] RIP: 0010:native_safe_halt+0x2e/0x50 [21704.121804] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21704.121808] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21704.121813] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21704.121815] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21704.121817] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21704.121819] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21704.126526] pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21704.126557] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21704.127351] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21704.128272] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21704.129646] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21704.129654] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21704.129657] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21704.129665] Call Trace: [21704.129671] [21704.129676] ? show_regs.cold.9+0x22/0x2f [21704.129690] ? watchdog_timer_fn.cold.6+0x62/0x10a [21704.129697] ? __hrtimer_run_queues+0x1c2/0x4d0 [21704.132062] RIP: 0010:native_safe_halt+0x2e/0x50 [21704.133313] ? watchdog+0x40/0x40 [21704.134269] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21704.136238] ? hrtimer_interrupt+0x128/0x2c0 [21704.136251] ? smp_apic_timer_interrupt+0xa6/0x270 [21704.136259] ? apic_timer_interrupt+0xf/0x20 [21704.136263] [21704.136267] ? native_safe_halt+0x2e/0x50 [21704.136270] kvm_wait+0xcd/0xe0 [21704.136279] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21704.136285] do_raw_spin_lock+0x177/0x190 [21704.136289] _raw_spin_lock+0x21/0x30 [21704.136295] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21704.138336] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000202 [21704.139810] obd_mod_free+0x46/0x140 [ptlrpc] [21704.146838] ORIG_RAX: ffffffffffffff13 [21704.146849] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21704.146852] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21704.146854] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21704.146857] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21704.146859] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21704.146863] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21704.146866] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21704.146869] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21704.146876] Call Trace: [21704.146883] [21704.146889] ? show_regs.cold.9+0x22/0x2f [21704.146903] ? watchdog_timer_fn.cold.6+0x62/0x10a [21704.146910] ? __hrtimer_run_queues+0x1c2/0x4d0 [21704.146914] ? watchdog+0x40/0x40 [21704.146919] ? hrtimer_interrupt+0x128/0x2c0 [21704.146923] ? smp_apic_timer_interrupt+0xa6/0x270 [21704.146930] ? apic_timer_interrupt+0xf/0x20 [21704.146934] [21704.146936] ? native_safe_halt+0x2e/0x50 [21704.146940] kvm_wait+0xcd/0xe0 [21704.146947] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21704.146956] do_raw_spin_lock+0x177/0x190 [21704.146959] _raw_spin_lock+0x21/0x30 [21704.146963] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21704.147249] ? do_raw_spin_unlock+0x75/0x190 [21704.147255] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21704.147507] ? do_wait_intr+0xf0/0xf0 [21704.147516] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21704.147704] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21704.147915] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21704.148131] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21704.148246] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21704.148285] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21704.148316] mdc_intent_lock+0x353/0x7e0 [mdc] [21704.148343] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21704.148557] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21704.148777] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21704.148809] lmv_intent_open+0x338/0x1150 [lmv] [21704.148844] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21704.148934] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21704.149019] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21704.149042] ll_intent_lock+0xc5/0xaa0 [lustre] [21704.149128] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21704.149213] ll_lookup_it+0xe0e/0x2270 [lustre] [21704.149299] ? do_raw_spin_unlock+0x75/0x190 [21704.149304] ? _raw_spin_unlock+0x12/0x30 [21704.149311] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21704.149336] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21704.149356] ll_atomic_open+0x36a/0xd70 [lustre] [21704.151872] mdc_commit_open+0x91/0xb0 [mdc] [21704.151920] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21704.154079] lookup_open+0x356/0xb40 [21704.154093] do_last+0x313/0xfc0 [21704.154098] ? nd_jump_root+0xe5/0x160 [21704.154103] ? path_init+0x437/0x520 [21704.154109] path_openat+0xf7/0x500 [21704.154114] ? do_raw_spin_unlock+0x75/0x190 [21704.154120] do_filp_open+0x99/0x140 [21704.154127] ? getname_flags+0x6e/0x330 [21704.154132] ? __check_object_size+0xff/0x256 [21704.154138] ? do_raw_spin_unlock+0x75/0x190 [21704.154142] ? _raw_spin_unlock+0x12/0x30 [21704.154148] do_sys_openat2+0x2b4/0x410 [21704.154152] do_sys_open+0x73/0xa0 [21704.154155] __x64_sys_openat+0x24/0x30 [21704.154158] do_syscall_64+0xc1/0x440 [21704.154165] entry_SYSCALL_64_after_hwframe+0x49/0xae [21704.154171] RIP: 0033:0x7f53425a5332 [21704.154180] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21704.154183] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21704.154189] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21704.154191] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21704.154193] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21704.154195] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21704.154197] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21704.394366] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21704.396637] after_reply+0xe2c/0x1780 [ptlrpc] [21704.398246] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21704.399978] ? __next_timer_interrupt+0x160/0x160 [21704.401588] ptlrpcd+0xa82/0xe40 [ptlrpc] [21704.402944] ? do_wait_intr+0xf0/0xf0 [21704.404063] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21704.406383] kthread+0x1d1/0x200 [21704.407467] ? set_kthread_struct+0x70/0x70 [21704.408781] ret_from_fork+0x1f/0x30 [21724.052021] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [21724.056236] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21724.080982] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21724.087010] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21724.092226] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21724.095735] RIP: 0010:native_safe_halt+0x2e/0x50 [21724.098127] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21724.108056] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21724.111257] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21724.115044] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [21724.118556] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21724.122248] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21724.125762] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21724.129252] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21724.133482] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21724.136630] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21724.140244] Call Trace: [21724.141978] [21724.143134] ? show_regs.cold.9+0x22/0x2f [21724.145414] ? watchdog_timer_fn.cold.6+0x62/0x10a [21724.148080] ? __hrtimer_run_queues+0x1c2/0x4d0 [21724.150720] ? watchdog+0x40/0x40 [21724.152532] ? hrtimer_interrupt+0x128/0x2c0 [21724.154536] ? smp_apic_timer_interrupt+0xa6/0x270 [21724.157093] ? apic_timer_interrupt+0xf/0x20 [21724.159418] [21724.160375] ? native_safe_halt+0x2e/0x50 [21724.162393] kvm_wait+0xcd/0xe0 [21724.164040] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21724.167460] ? newidle_balance.isra.93+0x207/0x580 [21724.169976] do_raw_spin_lock+0x177/0x190 [21724.171976] _raw_spin_lock+0x21/0x30 [21724.173982] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21724.177586] process_one_work+0x2c8/0x700 [21724.179626] worker_thread+0x296/0x6e0 [21724.181626] ? rescuer_thread+0x570/0x570 [21724.183865] kthread+0x1d1/0x200 [21724.185476] ? set_kthread_struct+0x70/0x70 [21724.187590] ret_from_fork+0x1f/0x30 [21727.204220] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 19090s! [21727.210799] Showing busy workqueues and worker pools: [21727.215245] workqueue events: flags=0x0 [21727.218640] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21727.222813] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21727.234566] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21727.244074] pending: cpu_latency_work [ptlrpc] [21727.246959] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21727.254371] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21727.258208] workqueue events_long: flags=0x0 [21727.264046] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21727.267566] pending: lu_objects_flush [obdclass] [21727.270075] workqueue events_unbound: flags=0x2 [21727.274241] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21727.279945] pending: flush_memcg_stats_dwork, idle_cull_fn [21727.284492] workqueue events_power_efficient: flags=0x80 [21727.289864] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21727.293929] pending: check_lifetime, do_cache_clean [21727.299770] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21727.303596] pending: neigh_periodic_work, neigh_periodic_work [21727.308993] workqueue mm_percpu_wq: flags=0x8 [21727.313477] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21727.320729] pending: vmstat_update, lru_add_drain_per_cpu [21727.326934] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21727.332385] pending: vmstat_update, lru_add_drain_per_cpu [21727.342419] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21727.346914] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21727.355378] workqueue xprtiod: flags=0xa [21727.358551] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21727.368054] pending: xprt_autoclose, xs_error_handle [21727.370918] workqueue ipv6_addrconf: flags=0x40008 [21727.378378] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21727.381294] pending: addrconf_verify_work [21727.384645] workqueue knbd0-recv: flags=0x1a [21727.389778] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21727.396586] in-flight: 651:recv_work [21727.402584] workqueue ptlrpc_pinger: flags=0x2 [21727.409956] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21727.412238] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21727.418489] pool 8: cpus=0-3 flags=0x4 nice=0 hung=19090s workers=4 idle: 15979 31824 8 [21727.424186] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21717s workers=2 idle: 654 [21732.055019] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [21732.057010] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [21732.058027] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [21732.058035] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21732.058122] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21732.058127] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21732.058131] RIP: 0010:native_safe_halt+0x2e/0x50 [21732.058158] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21732.058163] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21732.058170] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21732.058173] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21732.058176] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21732.058178] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21732.058181] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21732.058185] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21732.058187] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21732.058190] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21732.058198] Call Trace: [21732.058204] [21732.058209] ? show_regs.cold.9+0x22/0x2f [21732.058221] ? watchdog_timer_fn.cold.6+0x62/0x10a [21732.058227] ? __hrtimer_run_queues+0x1c2/0x4d0 [21732.058232] ? watchdog+0x40/0x40 [21732.058236] ? hrtimer_interrupt+0x128/0x2c0 [21732.058240] ? smp_apic_timer_interrupt+0xa6/0x270 [21732.058246] ? apic_timer_interrupt+0xf/0x20 [21732.058251] [21732.058254] ? native_safe_halt+0x2e/0x50 [21732.058259] kvm_wait+0xcd/0xe0 [21732.058267] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21732.058276] do_raw_spin_lock+0x177/0x190 [21732.058281] _raw_spin_lock+0x21/0x30 [21732.058286] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21732.058528] Modules linked in: lustre(O) [21732.058561] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21732.058775] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21732.059015] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21732.059220] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21732.059429] ? woken_wake_function+0x30/0x30 [21732.059439] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21732.059648] kthread+0x1d1/0x200 [21732.059654] ? set_kthread_struct+0x70/0x70 [21732.059658] ret_from_fork+0x1f/0x30 [21732.062649] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) [21732.068250] osp(O) [21732.083195] mgs(O) [21732.088524] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21732.088704] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21732.088712] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21732.088714] RIP: 0010:native_safe_halt+0x2e/0x50 [21732.088726] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21732.088731] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21732.088736] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21732.088738] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21732.088741] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21732.088743] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21732.088746] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21732.088749] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21732.088753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21732.088755] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21732.088764] Call Trace: [21732.088769] [21732.088774] ? show_regs.cold.9+0x22/0x2f [21732.088784] ? watchdog_timer_fn.cold.6+0x62/0x10a [21732.088790] ? __hrtimer_run_queues+0x1c2/0x4d0 [21732.088800] ? watchdog+0x40/0x40 [21732.088805] ? hrtimer_interrupt+0x128/0x2c0 [21732.088809] ? smp_apic_timer_interrupt+0xa6/0x270 [21732.088816] ? apic_timer_interrupt+0xf/0x20 [21732.088820] [21732.088824] ? native_safe_halt+0x2e/0x50 [21732.088828] kvm_wait+0xcd/0xe0 [21732.088836] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21732.088845] do_raw_spin_lock+0x177/0x190 [21732.088850] _raw_spin_lock+0x21/0x30 [21732.088854] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21732.097424] lquota(O) [21732.099742] ? do_raw_spin_unlock+0x75/0x190 [21732.099761] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21732.109971] lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21732.110045] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21732.110053] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21732.110056] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21732.110072] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21732.110076] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21732.110081] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21732.110083] RDX: 000000000000106a RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21732.110086] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21732.110088] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21732.110091] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21732.110094] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21732.110097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21732.110099] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21732.110110] Call Trace: [21732.110117] [21732.110122] ? show_regs.cold.9+0x22/0x2f [21732.110133] ? watchdog_timer_fn.cold.6+0x62/0x10a [21732.110140] ? __hrtimer_run_queues+0x1c2/0x4d0 [21732.110145] ? watchdog+0x40/0x40 [21732.110149] ? hrtimer_interrupt+0x128/0x2c0 [21732.110153] ? smp_apic_timer_interrupt+0xa6/0x270 [21732.110162] ? apic_timer_interrupt+0xf/0x20 [21732.110167] [21732.110170] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21732.110175] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21732.110179] do_raw_spin_lock+0x177/0x190 [21732.110184] _raw_spin_lock+0x21/0x30 [21732.110189] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21732.110614] obd_mod_free+0x46/0x140 [ptlrpc] [21732.110824] mdc_commit_open+0x91/0xb0 [mdc] [21732.110864] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21732.111182] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21732.111481] after_reply+0xe2c/0x1780 [ptlrpc] [21732.111750] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21732.112130] ? __next_timer_interrupt+0x160/0x160 [21732.112140] ptlrpcd+0xa82/0xe40 [ptlrpc] [21732.112341] ? do_wait_intr+0xf0/0xf0 [21732.112423] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21732.116626] ? do_wait_intr+0xf0/0xf0 [21732.116644] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21732.117181] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21732.117411] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21732.117731] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21732.117871] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21732.117917] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21732.118065] mdc_intent_lock+0x353/0x7e0 [mdc] [21732.118092] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21732.118182] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21732.124115] kthread+0x1d1/0x200 [21732.124135] ? set_kthread_struct+0x70/0x70 [21732.124140] ret_from_fork+0x1f/0x30 [21732.665548] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21732.671513] lmv_intent_open+0x338/0x1150 [lmv] [21732.672748] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21732.680446] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21732.684457] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21732.692532] ll_intent_lock+0xc5/0xaa0 [lustre] [21732.698510] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21732.703061] ll_lookup_it+0xe0e/0x2270 [lustre] [21732.709694] ? do_raw_spin_unlock+0x75/0x190 [21732.712796] ? _raw_spin_unlock+0x12/0x30 [21732.717971] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21732.723606] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21732.727295] ll_atomic_open+0x36a/0xd70 [lustre] [21732.731592] lookup_open+0x356/0xb40 [21732.732767] do_last+0x313/0xfc0 [21732.734873] ? nd_jump_root+0xe5/0x160 [21732.737324] ? path_init+0x437/0x520 [21732.741684] path_openat+0xf7/0x500 [21732.744298] ? do_raw_spin_unlock+0x75/0x190 [21732.749628] do_filp_open+0x99/0x140 [21732.754740] ? getname_flags+0x6e/0x330 [21732.756061] ? __check_object_size+0xff/0x256 [21732.763310] ? do_raw_spin_unlock+0x75/0x190 [21732.769624] ? _raw_spin_unlock+0x12/0x30 [21732.779312] do_sys_openat2+0x2b4/0x410 [21732.780627] do_sys_open+0x73/0xa0 [21732.789807] __x64_sys_openat+0x24/0x30 [21732.791157] do_syscall_64+0xc1/0x440 [21732.799183] entry_SYSCALL_64_after_hwframe+0x49/0xae [21732.807400] RIP: 0033:0x7f53425a5332 [21732.811328] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21732.831259] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21732.841874] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21732.845568] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21732.852435] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21732.863700] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21732.867576] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21752.052646] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [21752.057963] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21752.085792] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21752.090087] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21752.093326] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21752.095721] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21752.097726] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21752.103841] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21752.106510] RAX: 0000000000004c96 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21752.108979] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [21752.111381] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21752.120496] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21752.126795] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21752.130513] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21752.141606] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21752.144333] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21752.151978] Call Trace: [21752.154172] [21752.155247] ? show_regs.cold.9+0x22/0x2f [21752.158367] ? watchdog_timer_fn.cold.6+0x62/0x10a [21752.160676] ? __hrtimer_run_queues+0x1c2/0x4d0 [21752.162806] ? watchdog+0x40/0x40 [21752.164811] ? hrtimer_interrupt+0x128/0x2c0 [21752.167492] ? smp_apic_timer_interrupt+0xa6/0x270 [21752.170299] ? apic_timer_interrupt+0xf/0x20 [21752.172210] [21752.173441] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21752.176476] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21752.179532] ? newidle_balance.isra.93+0x207/0x580 [21752.181637] do_raw_spin_lock+0x177/0x190 [21752.183364] _raw_spin_lock+0x21/0x30 [21752.184974] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21752.187903] process_one_work+0x2c8/0x700 [21752.189846] worker_thread+0x296/0x6e0 [21752.191830] ? rescuer_thread+0x570/0x570 [21752.193782] kthread+0x1d1/0x200 [21752.195696] ? set_kthread_struct+0x70/0x70 [21752.197964] ret_from_fork+0x1f/0x30 [21757.922041] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 19120s! [21757.925744] Showing busy workqueues and worker pools: [21757.927295] workqueue events: flags=0x0 [21757.928997] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21757.932946] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21757.942726] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21757.946864] pending: cpu_latency_work [ptlrpc] [21757.955364] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21757.958906] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21757.963809] workqueue events_long: flags=0x0 [21757.965529] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21757.968462] pending: lu_objects_flush [obdclass] [21757.971738] workqueue events_unbound: flags=0x2 [21757.975123] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21757.979866] pending: flush_memcg_stats_dwork, idle_cull_fn [21757.983304] workqueue events_power_efficient: flags=0x80 [21757.985969] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21757.991886] pending: check_lifetime, do_cache_clean [21757.999199] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21758.007127] pending: neigh_periodic_work, neigh_periodic_work [21758.010232] workqueue mm_percpu_wq: flags=0x8 [21758.011850] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21758.016895] pending: vmstat_update, lru_add_drain_per_cpu [21758.020414] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21758.024162] pending: vmstat_update, lru_add_drain_per_cpu [21758.027379] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21758.032883] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21758.041905] workqueue xprtiod: flags=0xa [21758.046633] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21758.048640] pending: xprt_autoclose, xs_error_handle [21758.053088] workqueue ipv6_addrconf: flags=0x40008 [21758.060940] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21758.066333] pending: addrconf_verify_work [21758.070387] workqueue knbd0-recv: flags=0x1a [21758.072592] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21758.078174] in-flight: 651:recv_work [21758.082254] workqueue ptlrpc_pinger: flags=0x2 [21758.087065] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21758.094112] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21758.098096] pool 8: cpus=0-3 flags=0x4 nice=0 hung=19121s workers=4 idle: 15979 31824 8 [21758.111409] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21748s workers=2 idle: 654 [21760.057870] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [21760.058008] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [21760.059521] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [21760.059533] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21760.059611] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21760.059617] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21760.059619] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21760.059635] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21760.059639] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21760.059644] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21760.059646] RDX: 000000000000682f RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21760.059648] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21760.059650] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21760.059653] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21760.059655] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21760.059659] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21760.059661] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21760.059670] Call Trace: [21760.059676] [21760.059682] ? show_regs.cold.9+0x22/0x2f [21760.059765] ? watchdog_timer_fn.cold.6+0x62/0x10a [21760.059771] ? __hrtimer_run_queues+0x1c2/0x4d0 [21760.059775] ? watchdog+0x40/0x40 [21760.059779] ? hrtimer_interrupt+0x128/0x2c0 [21760.059782] ? smp_apic_timer_interrupt+0xa6/0x270 [21760.059790] ? apic_timer_interrupt+0xf/0x20 [21760.059794] [21760.059796] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21760.059804] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21760.059808] do_raw_spin_lock+0x177/0x190 [21760.059812] _raw_spin_lock+0x21/0x30 [21760.059818] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21760.060080] obd_mod_free+0x46/0x140 [ptlrpc] [21760.061110] mdc_commit_open+0x91/0xb0 [mdc] [21760.061172] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21760.061425] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21760.061647] after_reply+0xe2c/0x1780 [ptlrpc] [21760.062494] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21760.062825] ? __next_timer_interrupt+0x160/0x160 [21760.062836] ptlrpcd+0xa82/0xe40 [ptlrpc] [21760.063041] ? do_wait_intr+0xf0/0xf0 [21760.063051] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21760.063612] kthread+0x1d1/0x200 [21760.063619] ? set_kthread_struct+0x70/0x70 [21760.063623] ret_from_fork+0x1f/0x30 [21760.064426] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) [21760.071726] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21760.071803] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21760.071810] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21760.071814] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21760.071966] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21760.071976] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21760.071981] RAX: 0000000000006f92 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21760.071984] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21760.071989] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21760.071993] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21760.071996] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21760.072000] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21760.072003] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21760.072005] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21760.072027] Call Trace: [21760.072034] [21760.072040] ? show_regs.cold.9+0x22/0x2f [21760.072051] ? watchdog_timer_fn.cold.6+0x62/0x10a [21760.072058] ? __hrtimer_run_queues+0x1c2/0x4d0 [21760.072062] ? watchdog+0x40/0x40 [21760.072067] ? hrtimer_interrupt+0x128/0x2c0 [21760.072070] ? smp_apic_timer_interrupt+0xa6/0x270 [21760.072078] ? apic_timer_interrupt+0xf/0x20 [21760.072088] [21760.072092] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21760.072100] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21760.072107] do_raw_spin_lock+0x177/0x190 [21760.072115] _raw_spin_lock+0x21/0x30 [21760.072120] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21760.072966] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21760.073611] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21760.073980] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21760.074969] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21760.075609] ? woken_wake_function+0x30/0x30 [21760.075619] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21760.082622] mdt(O) [21760.114415] kthread+0x1d1/0x200 [21760.114435] ? set_kthread_struct+0x70/0x70 [21760.114619] ret_from_fork+0x1f/0x30 [21760.359608] mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21760.376550] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21760.381064] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21760.384776] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21760.387275] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21760.399972] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21760.403890] RAX: 0000000000005218 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21760.407194] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21760.410130] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21760.413661] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21760.416671] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21760.420383] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21760.424562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21760.427611] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21760.430839] Call Trace: [21760.431998] [21760.435996] ? show_regs.cold.9+0x22/0x2f [21760.439832] ? watchdog_timer_fn.cold.6+0x62/0x10a [21760.442999] ? __hrtimer_run_queues+0x1c2/0x4d0 [21760.446307] ? watchdog+0x40/0x40 [21760.447565] ? hrtimer_interrupt+0x128/0x2c0 [21760.449173] ? smp_apic_timer_interrupt+0xa6/0x270 [21760.450945] ? apic_timer_interrupt+0xf/0x20 [21760.452616] [21760.453415] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21760.455308] do_raw_spin_lock+0x177/0x190 [21760.456833] _raw_spin_lock+0x21/0x30 [21760.458378] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21760.460669] ? do_raw_spin_unlock+0x75/0x190 [21760.462265] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21760.464149] ? do_wait_intr+0xf0/0xf0 [21760.465329] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21760.467148] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21760.469132] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21760.471325] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21760.473123] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21760.474848] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21760.476222] mdc_intent_lock+0x353/0x7e0 [mdc] [21760.477535] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21760.479506] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21760.481826] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21760.483407] lmv_intent_open+0x338/0x1150 [lmv] [21760.485401] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21760.488109] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21760.490182] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21760.491662] ll_intent_lock+0xc5/0xaa0 [lustre] [21760.495356] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21760.502836] ll_lookup_it+0xe0e/0x2270 [lustre] [21760.504562] ? do_raw_spin_unlock+0x75/0x190 [21760.505809] ? _raw_spin_unlock+0x12/0x30 [21760.506891] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21760.508287] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21760.509857] ll_atomic_open+0x36a/0xd70 [lustre] [21760.511201] lookup_open+0x356/0xb40 [21760.512341] do_last+0x313/0xfc0 [21760.513805] ? nd_jump_root+0xe5/0x160 [21760.515108] ? path_init+0x437/0x520 [21760.516195] path_openat+0xf7/0x500 [21760.517314] ? do_raw_spin_unlock+0x75/0x190 [21760.518985] do_filp_open+0x99/0x140 [21760.520209] ? getname_flags+0x6e/0x330 [21760.521568] ? __check_object_size+0xff/0x256 [21760.526139] ? do_raw_spin_unlock+0x75/0x190 [21760.527782] ? _raw_spin_unlock+0x12/0x30 [21760.530028] do_sys_openat2+0x2b4/0x410 [21760.532253] do_sys_open+0x73/0xa0 [21760.535958] __x64_sys_openat+0x24/0x30 [21760.538307] do_syscall_64+0xc1/0x440 [21760.541777] entry_SYSCALL_64_after_hwframe+0x49/0xae [21760.545170] RIP: 0033:0x7f53425a5332 [21760.547009] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21760.554814] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21760.561250] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21760.570599] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21760.575773] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21760.579374] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21760.583358] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21774.912461] rcu: INFO: rcu_sched self-detected stall on CPU [21774.915353] rcu: 2-....: (11353644 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [21774.920615] (t=19680486 jiffies g=274185 q=68956) [21774.923420] rcu: rcu_sched kthread starved for 19138852 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=1 [21774.927540] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [21774.930074] rcu: RCU grace-period kthread stack dump: [21774.931710] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [21774.935146] Call Trace: [21774.938987] __schedule+0x351/0xcb0 [21774.941493] schedule+0xc0/0x180 [21774.943775] schedule_timeout+0xb4/0x190 [21774.945891] ? do_raw_spin_unlock+0x75/0x190 [21774.948649] ? __next_timer_interrupt+0x160/0x160 [21774.950776] ? prepare_to_swait_event+0xf3/0x140 [21774.957993] rcu_gp_kthread+0x8dd/0x15b0 [21774.959636] ? force_qs_rnp+0x2c0/0x2c0 [21774.962946] kthread+0x1d1/0x200 [21774.966420] ? set_kthread_struct+0x70/0x70 [21774.970596] ret_from_fork+0x1f/0x30 [21774.973845] rcu: Stack dump where RCU GP kthread last ran: [21774.980475] Sending NMI from CPU 2 to CPUs 1: [21774.991780] NMI backtrace for cpu 1 [21774.991788] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21774.991790] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21774.991791] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21774.991793] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21774.991795] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [21774.991798] RAX: 0000000000007887 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21774.991800] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21774.991801] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21774.991802] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21774.991804] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21774.991805] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21774.991806] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21774.991808] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21774.991809] Call Trace: [21774.991810] [21774.991811] ? show_regs.cold.9+0x22/0x2f [21774.991812] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [21774.991813] ? nmi_cpu_backtrace_handler+0x15/0x40 [21774.991814] ? nmi_handle+0xbd/0x210 [21774.991815] ? default_do_nmi+0x4b/0x1b0 [21774.991816] ? do_nmi+0x280/0x380 [21774.991817] ? end_repeat_nmi+0x16/0x1f [21774.991818] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21774.991819] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21774.991821] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21774.991822] [21774.991823] do_raw_spin_lock+0x177/0x190 [21774.991824] _raw_spin_lock+0x21/0x30 [21774.991825] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21774.991826] ? do_raw_spin_unlock+0x75/0x190 [21774.991827] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21774.991828] ? do_wait_intr+0xf0/0xf0 [21774.991829] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21774.991830] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21774.991831] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21774.991833] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21774.991834] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21774.991835] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21774.991836] mdc_intent_lock+0x353/0x7e0 [mdc] [21774.991837] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21774.991838] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21774.991839] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21774.991841] lmv_intent_open+0x338/0x1150 [lmv] [21774.991842] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21774.991843] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21774.991844] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21774.991845] ll_intent_lock+0xc5/0xaa0 [lustre] [21774.991846] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21774.991847] ll_lookup_it+0xe0e/0x2270 [lustre] [21774.991848] ? do_raw_spin_unlock+0x75/0x190 [21774.991849] ? _raw_spin_unlock+0x12/0x30 [21774.991850] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21774.991852] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21774.991853] ll_atomic_open+0x36a/0xd70 [lustre] [21774.991854] lookup_open+0x356/0xb40 [21774.991855] do_last+0x313/0xfc0 [21774.991856] ? nd_jump_root+0xe5/0x160 [21774.991857] ? path_init+0x437/0x520 [21774.991859] path_openat+0xf7/0x500 [21774.991860] ? do_raw_spin_unlock+0x75/0x190 [21774.991861] do_filp_open+0x99/0x140 [21774.991862] ? getname_flags+0x6e/0x330 [21774.991863] ? __check_object_size+0xff/0x256 [21774.991865] ? do_raw_spin_unlock+0x75/0x190 [21774.991866] ? _raw_spin_unlock+0x12/0x30 [21774.991867] do_sys_openat2+0x2b4/0x410 [21774.991868] do_sys_open+0x73/0xa0 [21774.991869] __x64_sys_openat+0x24/0x30 [21774.991870] do_syscall_64+0xc1/0x440 [21774.991871] entry_SYSCALL_64_after_hwframe+0x49/0xae [21774.991872] RIP: 0033:0x7f53425a5332 [21774.991874] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21774.991876] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21774.991879] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21774.991880] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21774.991881] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21774.991882] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21774.991884] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21774.999139] Sending NMI from CPU 2 to CPUs 1: [21775.363032] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [21775.367046] NMI backtrace for cpu 2 [21775.368560] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21775.374857] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21775.381909] Call Trace: [21775.383003] [21775.385579] ? dump_stack+0xbb/0x10e [21775.388699] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [21775.395783] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [21775.400526] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [21775.403649] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [21775.414483] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [21775.416255] ? rcu_dump_cpu_stacks+0x11b/0x171 [21775.420605] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [21775.423823] ? update_process_times+0x6d/0xc0 [21775.430116] ? tick_sched_handle.isra.18+0x27/0x90 [21775.435850] ? tick_sched_timer+0x53/0xb0 [21775.438592] ? __hrtimer_run_queues+0x1c2/0x4d0 [21775.441937] ? tick_sched_handle.isra.18+0x90/0x90 [21775.448067] ? hrtimer_interrupt+0x128/0x2c0 [21775.457741] ? smp_apic_timer_interrupt+0xa6/0x270 [21775.459450] ? apic_timer_interrupt+0xf/0x20 [21775.465501] [21775.466095] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21775.468782] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21775.476953] ? do_raw_spin_lock+0x177/0x190 [21775.479738] ? _raw_spin_lock+0x21/0x30 [21775.482100] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21775.489073] ? obd_mod_free+0x46/0x140 [ptlrpc] [21775.493478] ? mdc_commit_open+0x91/0xb0 [mdc] [21775.494863] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21775.497521] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21775.499382] ? after_reply+0xe2c/0x1780 [ptlrpc] [21775.500728] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21775.502237] ? __next_timer_interrupt+0x160/0x160 [21775.503864] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [21775.505748] ? do_wait_intr+0xf0/0xf0 [21775.507374] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21775.509707] ? kthread+0x1d1/0x200 [21775.511350] ? set_kthread_struct+0x70/0x70 [21775.512988] ? ret_from_fork+0x1f/0x30 [21784.055613] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [21784.064594] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21784.098305] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21784.104254] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21784.112598] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21784.118704] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21784.121925] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21784.145400] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21784.153753] RAX: 0000000000007cd1 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21784.156186] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [21784.158732] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21784.161421] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21784.165399] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21784.178125] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21784.181980] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21784.185523] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21784.188721] Call Trace: [21784.189596] [21784.191942] ? show_regs.cold.9+0x22/0x2f [21784.196433] ? watchdog_timer_fn.cold.6+0x62/0x10a [21784.199997] ? __hrtimer_run_queues+0x1c2/0x4d0 [21784.203329] ? watchdog+0x40/0x40 [21784.204621] ? hrtimer_interrupt+0x128/0x2c0 [21784.206472] ? smp_apic_timer_interrupt+0xa6/0x270 [21784.210541] ? apic_timer_interrupt+0xf/0x20 [21784.212581] [21784.213722] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21784.218586] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21784.221709] ? newidle_balance.isra.93+0x207/0x580 [21784.223667] do_raw_spin_lock+0x177/0x190 [21784.226526] _raw_spin_lock+0x21/0x30 [21784.229802] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21784.233309] process_one_work+0x2c8/0x700 [21784.235384] worker_thread+0x296/0x6e0 [21784.236854] ? rescuer_thread+0x570/0x570 [21784.238424] kthread+0x1d1/0x200 [21784.239864] ? set_kthread_struct+0x70/0x70 [21784.241796] ret_from_fork+0x1f/0x30 [21788.058031] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [21788.060751] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21788.093257] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21788.109484] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21788.117190] RIP: 0010:native_safe_halt+0x2e/0x50 [21788.120609] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21788.131215] BUG: workqueue lockup - pool [21788.136274] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21788.136296] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21788.136299] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21788.136301] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21788.136303] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21788.136306] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21788.136309] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21788.136312] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21788.136315] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21788.136323] Call Trace: [21788.136330] [21788.136336] ? show_regs.cold.9+0x22/0x2f [21788.136349] ? watchdog_timer_fn.cold.6+0x62/0x10a [21788.136356] ? __hrtimer_run_queues+0x1c2/0x4d0 [21788.136360] ? watchdog+0x40/0x40 [21788.136364] ? hrtimer_interrupt+0x128/0x2c0 [21788.136368] ? smp_apic_timer_interrupt+0xa6/0x270 [21788.136376] ? apic_timer_interrupt+0xf/0x20 [21788.136381] [21788.136385] ? native_safe_halt+0x2e/0x50 [21788.136389] kvm_wait+0xcd/0xe0 [21788.136396] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21788.136406] do_raw_spin_lock+0x177/0x190 [21788.136410] _raw_spin_lock+0x21/0x30 [21788.136415] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21788.142142] cpus=0-3 flags=0x4 nice=0 stuck for 19151s! [21788.142165] Showing busy workqueues and worker pools: [21788.142167] workqueue events: flags=0x0 [21788.142170] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21788.144550] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc] [21788.151354] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21788.158197] , nfs4_renew_state [nfsv4], push_to_pool [21788.158427] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21788.158441] pending: cpu_latency_work [ptlrpc] [21788.159302] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21788.159313] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21788.159600] workqueue events_long: flags=0x0 [21788.159603] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21788.159609] pending: lu_objects_flush [obdclass] [21788.159857] workqueue events_unbound: flags=0x2 [21788.159859] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21788.159866] pending: flush_memcg_stats_dwork, idle_cull_fn [21788.159877] workqueue events_power_efficient: flags=0x80 [21788.159880] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21788.159886] pending: check_lifetime, do_cache_clean [21788.159898] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21788.159905] pending: neigh_periodic_work, neigh_periodic_work [21788.159930] workqueue mm_percpu_wq: flags=0x8 [21788.159932] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21788.159937] pending: vmstat_update, lru_add_drain_per_cpu [21788.160172] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21788.160181] pending: vmstat_update, lru_add_drain_per_cpu [21788.160191] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21788.160200] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21788.160225] workqueue xprtiod: flags=0xa [21788.160227] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21788.160233] pending: xprt_autoclose, xs_error_handle [21788.160250] workqueue ipv6_addrconf: flags=0x40008 [21788.160252] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21788.160260] pending: addrconf_verify_work [21788.160332] workqueue knbd0-recv: flags=0x1a [21788.160334] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21788.160341] in-flight: 651:recv_work [21788.160351] workqueue ptlrpc_pinger: flags=0x2 [21788.160353] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21788.160359] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21788.160688] pool 8: cpus=0-3 flags=0x4 nice=0 hung=19151s workers=4 idle: 15979 31824 8 [21788.160701] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21778s workers=2 idle: 654 [21788.446154] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21788.452102] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21788.458451] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21788.462504] ? woken_wake_function+0x30/0x30 [21788.466292] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21788.470739] kthread+0x1d1/0x200 [21788.472641] ? set_kthread_struct+0x70/0x70 [21788.477411] ret_from_fork+0x1f/0x30 [21800.055222] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [21800.057036] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [21800.065489] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21800.065562] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21800.068496] Modules linked in: lustre(O) osp(O) [21800.137414] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21800.142230] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21800.142305] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21800.142312] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21800.142315] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21800.142328] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21800.142332] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21800.142336] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21800.142339] RDX: 0000000000003d98 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21800.142342] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21800.142344] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21800.142347] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21800.142350] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21800.142353] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21800.142356] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21800.142363] Call Trace: [21800.142369] [21800.142374] ? show_regs.cold.9+0x22/0x2f [21800.142382] ? watchdog_timer_fn.cold.6+0x62/0x10a [21800.142388] ? __hrtimer_run_queues+0x1c2/0x4d0 [21800.142392] ? watchdog+0x40/0x40 [21800.142396] ? hrtimer_interrupt+0x128/0x2c0 [21800.142400] ? smp_apic_timer_interrupt+0xa6/0x270 [21800.142408] ? apic_timer_interrupt+0xf/0x20 [21800.142412] [21800.142415] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21800.142420] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21800.142424] do_raw_spin_lock+0x177/0x190 [21800.142429] _raw_spin_lock+0x21/0x30 [21800.142434] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21800.142721] obd_mod_free+0x46/0x140 [ptlrpc] [21800.143049] mdc_commit_open+0x91/0xb0 [mdc] [21800.143091] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21800.143353] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21800.143566] after_reply+0xe2c/0x1780 [ptlrpc] [21800.148187] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21800.148213] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21800.148218] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21800.148224] RAX: 0000000000000e85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21800.148226] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21800.148229] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21800.148232] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21800.151700] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21800.152022] ? __next_timer_interrupt+0x160/0x160 [21800.152032] ptlrpcd+0xa82/0xe40 [ptlrpc] [21800.152248] ? do_wait_intr+0xf0/0xf0 [21800.152258] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21800.152472] kthread+0x1d1/0x200 [21800.152478] ? set_kthread_struct+0x70/0x70 [21800.152482] ret_from_fork+0x1f/0x30 [21800.548546] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21800.551403] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21800.553975] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21800.555389] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21800.558451] Call Trace: [21800.559877] [21800.560954] ? show_regs.cold.9+0x22/0x2f [21800.562256] ? watchdog_timer_fn.cold.6+0x62/0x10a [21800.564298] ? __hrtimer_run_queues+0x1c2/0x4d0 [21800.566167] ? watchdog+0x40/0x40 [21800.567299] ? hrtimer_interrupt+0x128/0x2c0 [21800.569144] ? smp_apic_timer_interrupt+0xa6/0x270 [21800.571191] ? apic_timer_interrupt+0xf/0x20 [21800.572700] [21800.573292] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21800.574887] do_raw_spin_lock+0x177/0x190 [21800.576238] _raw_spin_lock+0x21/0x30 [21800.580148] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21800.583341] ? do_raw_spin_unlock+0x75/0x190 [21800.587110] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21800.590411] ? do_wait_intr+0xf0/0xf0 [21800.591570] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21800.593498] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21800.599255] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21800.603341] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21800.613430] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21800.617512] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21800.619421] mdc_intent_lock+0x353/0x7e0 [mdc] [21800.624323] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21800.629620] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21800.631715] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21800.635659] lmv_intent_open+0x338/0x1150 [lmv] [21800.637195] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21800.639311] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21800.641280] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21800.645702] ll_intent_lock+0xc5/0xaa0 [lustre] [21800.647258] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21800.656089] ll_lookup_it+0xe0e/0x2270 [lustre] [21800.660298] ? do_raw_spin_unlock+0x75/0x190 [21800.663098] ? _raw_spin_unlock+0x12/0x30 [21800.665124] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21800.667441] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21800.670079] ll_atomic_open+0x36a/0xd70 [lustre] [21800.671613] lookup_open+0x356/0xb40 [21800.674684] do_last+0x313/0xfc0 [21800.676371] ? nd_jump_root+0xe5/0x160 [21800.677876] ? path_init+0x437/0x520 [21800.679686] path_openat+0xf7/0x500 [21800.681209] ? do_raw_spin_unlock+0x75/0x190 [21800.683209] do_filp_open+0x99/0x140 [21800.685231] ? getname_flags+0x6e/0x330 [21800.687274] ? __check_object_size+0xff/0x256 [21800.689268] ? do_raw_spin_unlock+0x75/0x190 [21800.693223] ? _raw_spin_unlock+0x12/0x30 [21800.695901] do_sys_openat2+0x2b4/0x410 [21800.697259] do_sys_open+0x73/0xa0 [21800.698931] __x64_sys_openat+0x24/0x30 [21800.700362] do_syscall_64+0xc1/0x440 [21800.702077] entry_SYSCALL_64_after_hwframe+0x49/0xae [21800.704804] RIP: 0033:0x7f53425a5332 [21800.709060] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21800.718067] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21800.721135] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21800.725017] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21800.729192] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21800.732851] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21800.735471] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21816.052025] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [21816.058380] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [21816.059578] Modules linked in: [21816.061951] Modules linked in: [21816.061956] lustre(O) osp(O) [21816.062929] lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21816.063002] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21816.063008] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21816.063011] RIP: 0010:native_safe_halt+0x2e/0x50 [21816.063024] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21816.063028] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21816.063032] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21816.063035] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21816.063037] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21816.063040] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21816.063042] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21816.063046] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21816.063049] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21816.063051] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21816.063059] Call Trace: [21816.063064] [21816.063069] ? show_regs.cold.9+0x22/0x2f [21816.063079] ? watchdog_timer_fn.cold.6+0x62/0x10a [21816.063084] ? __hrtimer_run_queues+0x1c2/0x4d0 [21816.063089] ? watchdog+0x40/0x40 [21816.063093] ? hrtimer_interrupt+0x128/0x2c0 [21816.063098] ? smp_apic_timer_interrupt+0xa6/0x270 [21816.063170] ? apic_timer_interrupt+0xf/0x20 [21816.063175] [21816.063178] ? native_safe_halt+0x2e/0x50 [21816.063181] kvm_wait+0xcd/0xe0 [21816.063189] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21816.063197] do_raw_spin_lock+0x177/0x190 [21816.063201] _raw_spin_lock+0x21/0x30 [21816.063206] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21816.064934] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) [21816.067769] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21816.094151] ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac [21816.099846] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21816.100401] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21816.100738] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21816.100967] ? woken_wake_function+0x30/0x30 [21816.100978] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21816.101332] kthread+0x1d1/0x200 [21816.101338] ? set_kthread_struct+0x70/0x70 [21816.101343] ret_from_fork+0x1f/0x30 [21816.242498] rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21816.252989] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21816.261714] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21816.266908] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21816.274637] RIP: 0010:native_safe_halt+0x2e/0x50 [21816.276110] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21816.288791] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21816.292253] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21816.296646] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [21816.303691] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21816.308750] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21816.311978] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21816.316890] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21816.322811] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21816.329051] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21816.332923] Call Trace: [21816.335992] [21816.337399] ? show_regs.cold.9+0x22/0x2f [21816.340304] ? watchdog_timer_fn.cold.6+0x62/0x10a [21816.343595] ? __hrtimer_run_queues+0x1c2/0x4d0 [21816.347608] ? watchdog+0x40/0x40 [21816.348595] ? hrtimer_interrupt+0x128/0x2c0 [21816.350296] ? smp_apic_timer_interrupt+0xa6/0x270 [21816.353048] ? apic_timer_interrupt+0xf/0x20 [21816.354666] [21816.356200] ? native_safe_halt+0x2e/0x50 [21816.358403] kvm_wait+0xcd/0xe0 [21816.361945] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21816.364767] ? newidle_balance.isra.93+0x207/0x580 [21816.373862] do_raw_spin_lock+0x177/0x190 [21816.376100] _raw_spin_lock+0x21/0x30 [21816.379848] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21816.381888] process_one_work+0x2c8/0x700 [21816.385109] worker_thread+0x296/0x6e0 [21816.388602] ? rescuer_thread+0x570/0x570 [21816.391115] kthread+0x1d1/0x200 [21816.394265] ? set_kthread_struct+0x70/0x70 [21816.398267] ret_from_fork+0x1f/0x30 [21818.336540] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 19181s! [21818.344228] Showing busy workqueues and worker pools: [21818.347966] workqueue events: flags=0x0 [21818.349982] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21818.354108] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21818.362579] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21818.367027] pending: cpu_latency_work [ptlrpc] [21818.368860] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21818.375393] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21818.381410] workqueue events_long: flags=0x0 [21818.383842] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21818.388065] pending: lu_objects_flush [obdclass] [21818.390959] workqueue events_unbound: flags=0x2 [21818.393288] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21818.396586] pending: flush_memcg_stats_dwork, idle_cull_fn [21818.399599] workqueue events_power_efficient: flags=0x80 [21818.407079] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21818.409073] pending: check_lifetime, do_cache_clean [21818.416311] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21818.420274] pending: neigh_periodic_work, neigh_periodic_work [21818.425208] workqueue mm_percpu_wq: flags=0x8 [21818.433787] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21818.439342] pending: vmstat_update, lru_add_drain_per_cpu [21818.442740] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21818.446395] pending: vmstat_update, lru_add_drain_per_cpu [21818.451763] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21818.456742] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21818.469618] workqueue xprtiod: flags=0xa [21818.471168] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21818.478045] pending: xprt_autoclose, xs_error_handle [21818.480711] workqueue ipv6_addrconf: flags=0x40008 [21818.485405] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21818.494404] pending: addrconf_verify_work [21818.498039] workqueue knbd0-recv: flags=0x1a [21818.503634] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21818.511647] in-flight: 651:recv_work [21818.515850] workqueue ptlrpc_pinger: flags=0x2 [21818.520124] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21818.527690] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21818.539633] pool 8: cpus=0-3 flags=0x4 nice=0 hung=19181s workers=4 idle: 15979 31824 8 [21818.548628] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21809s workers=2 idle: 654 [21828.055622] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [21828.057833] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [21828.061083] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21828.061163] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21828.061170] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21828.061173] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21828.061186] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21828.061191] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21828.061195] RAX: 0000000000006658 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21828.061197] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21828.061200] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21828.061202] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21828.061205] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21828.061208] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21828.061211] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21828.061213] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21828.061223] Call Trace: [21828.061228] [21828.061234] ? show_regs.cold.9+0x22/0x2f [21828.061245] ? watchdog_timer_fn.cold.6+0x62/0x10a [21828.061511] ? __hrtimer_run_queues+0x1c2/0x4d0 [21828.061516] ? watchdog+0x40/0x40 [21828.061520] ? hrtimer_interrupt+0x128/0x2c0 [21828.061524] ? smp_apic_timer_interrupt+0xa6/0x270 [21828.061533] ? apic_timer_interrupt+0xf/0x20 [21828.061537] [21828.061540] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21828.061547] do_raw_spin_lock+0x177/0x190 [21828.061553] _raw_spin_lock+0x21/0x30 [21828.061558] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21828.065615] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) [21828.087957] ? do_raw_spin_unlock+0x75/0x190 [21828.087979] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21828.092359] obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21828.092426] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21828.092433] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21828.092435] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21828.092449] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21828.092454] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21828.092459] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21828.092461] RDX: 00000000000024cf RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21828.092463] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21828.092466] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21828.092468] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21828.092471] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21828.092474] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21828.092476] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21828.092486] Call Trace: [21828.092493] [21828.092499] ? show_regs.cold.9+0x22/0x2f [21828.092509] ? watchdog_timer_fn.cold.6+0x62/0x10a [21828.092516] ? __hrtimer_run_queues+0x1c2/0x4d0 [21828.092520] ? watchdog+0x40/0x40 [21828.092524] ? hrtimer_interrupt+0x128/0x2c0 [21828.092529] ? smp_apic_timer_interrupt+0xa6/0x270 [21828.092539] ? apic_timer_interrupt+0xf/0x20 [21828.092544] [21828.092547] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21828.092552] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21828.092557] do_raw_spin_lock+0x177/0x190 [21828.092562] _raw_spin_lock+0x21/0x30 [21828.092569] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21828.092844] obd_mod_free+0x46/0x140 [ptlrpc] [21828.093060] mdc_commit_open+0x91/0xb0 [mdc] [21828.093097] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21828.093420] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21828.093616] after_reply+0xe2c/0x1780 [ptlrpc] [21828.093808] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21828.094001] ? __next_timer_interrupt+0x160/0x160 [21828.094009] ptlrpcd+0xa82/0xe40 [ptlrpc] [21828.094408] ? do_wait_intr+0xf0/0xf0 [21828.094419] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21828.094624] kthread+0x1d1/0x200 [21828.094630] ? set_kthread_struct+0x70/0x70 [21828.094634] ret_from_fork+0x1f/0x30 [21828.317446] ? do_wait_intr+0xf0/0xf0 [21828.318727] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21828.322400] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21828.325164] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21828.327682] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21828.330208] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21828.333506] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21828.336426] mdc_intent_lock+0x353/0x7e0 [mdc] [21828.339363] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21828.342615] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21828.346725] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21828.349332] lmv_intent_open+0x338/0x1150 [lmv] [21828.351844] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21828.354256] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21828.356643] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21828.358521] ll_intent_lock+0xc5/0xaa0 [lustre] [21828.360168] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21828.362046] ll_lookup_it+0xe0e/0x2270 [lustre] [21828.364023] ? do_raw_spin_unlock+0x75/0x190 [21828.366441] ? _raw_spin_unlock+0x12/0x30 [21828.368341] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21828.370296] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21828.372822] ll_atomic_open+0x36a/0xd70 [lustre] [21828.374799] lookup_open+0x356/0xb40 [21828.376420] do_last+0x313/0xfc0 [21828.377595] ? nd_jump_root+0xe5/0x160 [21828.379241] ? path_init+0x437/0x520 [21828.380590] path_openat+0xf7/0x500 [21828.382443] ? do_raw_spin_unlock+0x75/0x190 [21828.384081] do_filp_open+0x99/0x140 [21828.385984] ? getname_flags+0x6e/0x330 [21828.387507] ? __check_object_size+0xff/0x256 [21828.389535] ? do_raw_spin_unlock+0x75/0x190 [21828.391106] ? _raw_spin_unlock+0x12/0x30 [21828.392751] do_sys_openat2+0x2b4/0x410 [21828.394594] do_sys_open+0x73/0xa0 [21828.395783] __x64_sys_openat+0x24/0x30 [21828.397471] do_syscall_64+0xc1/0x440 [21828.398617] entry_SYSCALL_64_after_hwframe+0x49/0xae [21828.400642] RIP: 0033:0x7f53425a5332 [21828.402033] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21828.409649] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21828.412898] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21828.415467] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21828.418122] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21828.420777] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21828.423566] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21844.052028] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [21844.054950] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21844.058035] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [21844.097233] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21844.097249] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21844.097257] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21844.100375] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21844.100462] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21844.100475] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21844.100478] RIP: 0010:native_safe_halt+0x2e/0x50 [21844.100491] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21844.100496] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21844.100501] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21844.100503] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21844.100506] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21844.100509] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21844.100512] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21844.100515] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21844.100518] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21844.100521] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21844.100529] Call Trace: [21844.100535] [21844.100541] ? show_regs.cold.9+0x22/0x2f [21844.100556] ? watchdog_timer_fn.cold.6+0x62/0x10a [21844.100563] ? __hrtimer_run_queues+0x1c2/0x4d0 [21844.100567] ? watchdog+0x40/0x40 [21844.100572] ? hrtimer_interrupt+0x128/0x2c0 [21844.100576] ? smp_apic_timer_interrupt+0xa6/0x270 [21844.100583] ? apic_timer_interrupt+0xf/0x20 [21844.100654] [21844.100657] ? native_safe_halt+0x2e/0x50 [21844.100661] kvm_wait+0xcd/0xe0 [21844.100669] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21844.100677] do_raw_spin_lock+0x177/0x190 [21844.100681] _raw_spin_lock+0x21/0x30 [21844.100686] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21844.101139] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21844.101473] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21844.102328] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21844.102885] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21844.103690] ? woken_wake_function+0x30/0x30 [21844.118212] [21844.118225] RIP: 0010:native_safe_halt+0x2e/0x50 [21844.118245] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21844.118249] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21844.118254] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21844.118256] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [21844.118263] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21844.118265] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21844.118267] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21844.118269] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21844.118273] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21844.118276] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21844.118286] Call Trace: [21844.118292] [21844.118297] ? show_regs.cold.9+0x22/0x2f [21844.118307] ? watchdog_timer_fn.cold.6+0x62/0x10a [21844.118314] ? __hrtimer_run_queues+0x1c2/0x4d0 [21844.118319] ? watchdog+0x40/0x40 [21844.118323] ? hrtimer_interrupt+0x128/0x2c0 [21844.118327] ? smp_apic_timer_interrupt+0xa6/0x270 [21844.118333] ? apic_timer_interrupt+0xf/0x20 [21844.118338] [21844.118341] ? native_safe_halt+0x2e/0x50 [21844.118345] kvm_wait+0xcd/0xe0 [21844.118354] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21844.118367] ? newidle_balance.isra.93+0x207/0x580 [21844.118375] do_raw_spin_lock+0x177/0x190 [21844.118384] _raw_spin_lock+0x21/0x30 [21844.118389] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21844.121784] process_one_work+0x2c8/0x700 [21844.121794] worker_thread+0x296/0x6e0 [21844.121799] ? rescuer_thread+0x570/0x570 [21844.121803] kthread+0x1d1/0x200 [21844.121808] ? set_kthread_struct+0x70/0x70 [21844.121812] ret_from_fork+0x1f/0x30 [21844.477316] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21844.479515] kthread+0x1d1/0x200 [21844.480817] ? set_kthread_struct+0x70/0x70 [21844.482636] ret_from_fork+0x1f/0x30 [21849.067259] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 19211s! [21849.073332] Showing busy workqueues and worker pools: [21849.079410] workqueue events: flags=0x0 [21849.085832] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21849.092140] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21849.102739] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21849.108109] pending: cpu_latency_work [ptlrpc] [21849.112411] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21849.118125] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21849.125591] workqueue events_long: flags=0x0 [21849.130047] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21849.135099] pending: lu_objects_flush [obdclass] [21849.140930] workqueue events_unbound: flags=0x2 [21849.142145] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21849.147872] pending: flush_memcg_stats_dwork, idle_cull_fn [21849.151936] workqueue events_power_efficient: flags=0x80 [21849.154520] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21849.159329] pending: check_lifetime, do_cache_clean [21849.166232] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21849.172965] pending: neigh_periodic_work, neigh_periodic_work [21849.184418] workqueue mm_percpu_wq: flags=0x8 [21849.190405] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21849.192607] pending: vmstat_update, lru_add_drain_per_cpu [21849.198209] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21849.202063] pending: vmstat_update, lru_add_drain_per_cpu [21849.205208] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21849.209200] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21849.211741] workqueue xprtiod: flags=0xa [21849.213173] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21849.216418] pending: xprt_autoclose, xs_error_handle [21849.220314] workqueue ipv6_addrconf: flags=0x40008 [21849.222324] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21849.224789] pending: addrconf_verify_work [21849.226729] workqueue knbd0-recv: flags=0x1a [21849.228590] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21849.233857] in-flight: 651:recv_work [21849.237874] workqueue ptlrpc_pinger: flags=0x2 [21849.242300] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21849.247347] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21849.256229] pool 8: cpus=0-3 flags=0x4 nice=0 hung=19212s workers=4 idle: 15979 31824 8 [21849.259272] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21839s workers=2 idle: 654 [21856.055016] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [21856.058197] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21856.058526] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [21856.087652] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21856.087667] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21856.087671] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21856.087685] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21856.087690] RSP: 0018:ffffbb5381503500 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [21856.087695] RAX: 0000000000002a07 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21856.087697] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21856.087700] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21856.087702] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21856.087704] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21856.087707] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21856.087710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21856.087712] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21856.087720] Call Trace: [21856.087726] [21856.087731] ? show_regs.cold.9+0x22/0x2f [21856.087740] ? watchdog_timer_fn.cold.6+0x62/0x10a [21856.087746] ? __hrtimer_run_queues+0x1c2/0x4d0 [21856.087751] ? watchdog+0x40/0x40 [21856.087755] ? hrtimer_interrupt+0x128/0x2c0 [21856.087861] ? smp_apic_timer_interrupt+0xa6/0x270 [21856.087870] ? apic_timer_interrupt+0xf/0x20 [21856.087875] [21856.087878] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21856.087887] do_raw_spin_lock+0x177/0x190 [21856.087893] _raw_spin_lock+0x21/0x30 [21856.087898] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21856.088180] ? do_raw_spin_unlock+0x75/0x190 [21856.088186] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21856.088380] ? do_wait_intr+0xf0/0xf0 [21856.088389] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21856.088582] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21856.088928] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21856.094553] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21856.094627] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21856.094634] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21856.094637] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21856.094649] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21856.094653] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21856.094657] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21856.094659] RDX: 000000000000133b RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21856.094661] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21856.094663] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21856.094665] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21856.094668] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21856.094671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21856.094674] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21856.094682] Call Trace: [21856.094689] [21856.094696] ? show_regs.cold.9+0x22/0x2f [21856.094705] ? watchdog_timer_fn.cold.6+0x62/0x10a [21856.094712] ? __hrtimer_run_queues+0x1c2/0x4d0 [21856.094715] ? watchdog+0x40/0x40 [21856.094720] ? hrtimer_interrupt+0x128/0x2c0 [21856.094725] ? smp_apic_timer_interrupt+0xa6/0x270 [21856.094734] ? apic_timer_interrupt+0xf/0x20 [21856.094738] [21856.094741] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21856.094746] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21856.094751] do_raw_spin_lock+0x177/0x190 [21856.094756] _raw_spin_lock+0x21/0x30 [21856.094775] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21856.095054] obd_mod_free+0x46/0x140 [ptlrpc] [21856.109200] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21856.109336] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21856.109376] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21856.109405] mdc_intent_lock+0x353/0x7e0 [mdc] [21856.109434] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21856.109520] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21856.111264] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21856.111308] lmv_intent_open+0x338/0x1150 [lmv] [21856.111343] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21856.115602] mdc_commit_open+0x91/0xb0 [mdc] [21856.115649] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21856.116214] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21856.116409] after_reply+0xe2c/0x1780 [ptlrpc] [21856.116608] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21856.116832] ? __next_timer_interrupt+0x160/0x160 [21856.116973] ptlrpcd+0xa82/0xe40 [ptlrpc] [21856.117200] ? do_wait_intr+0xf0/0xf0 [21856.117209] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21856.117410] kthread+0x1d1/0x200 [21856.117416] ? set_kthread_struct+0x70/0x70 [21856.117420] ret_from_fork+0x1f/0x30 [21856.334632] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21856.338547] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21856.341733] ll_intent_lock+0xc5/0xaa0 [lustre] [21856.343136] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21856.344880] ll_lookup_it+0xe0e/0x2270 [lustre] [21856.347002] ? do_raw_spin_unlock+0x75/0x190 [21856.348616] ? _raw_spin_unlock+0x12/0x30 [21856.350153] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21856.352586] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21856.354526] ll_atomic_open+0x36a/0xd70 [lustre] [21856.356540] lookup_open+0x356/0xb40 [21856.358309] do_last+0x313/0xfc0 [21856.360137] ? nd_jump_root+0xe5/0x160 [21856.361663] ? path_init+0x437/0x520 [21856.363198] path_openat+0xf7/0x500 [21856.364459] ? do_raw_spin_unlock+0x75/0x190 [21856.365958] do_filp_open+0x99/0x140 [21856.367466] ? getname_flags+0x6e/0x330 [21856.369302] ? __check_object_size+0xff/0x256 [21856.371742] ? do_raw_spin_unlock+0x75/0x190 [21856.373992] ? _raw_spin_unlock+0x12/0x30 [21856.375956] do_sys_openat2+0x2b4/0x410 [21856.377689] do_sys_open+0x73/0xa0 [21856.379624] __x64_sys_openat+0x24/0x30 [21856.381550] do_syscall_64+0xc1/0x440 [21856.382882] entry_SYSCALL_64_after_hwframe+0x49/0xae [21856.385594] RIP: 0033:0x7f53425a5332 [21856.386879] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21856.393952] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21856.396638] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21856.399097] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21856.401634] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21856.404320] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21856.409507] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21872.061979] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [21872.064874] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21872.127740] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21872.139146] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21872.148470] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21872.150913] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21872.166607] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21872.170893] RAX: 0000000000006e8a RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21872.179429] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21872.188829] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21872.197431] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21872.205759] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21872.215924] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21872.219825] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21872.222502] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21872.228315] Call Trace: [21872.232401] [21872.233924] ? show_regs.cold.9+0x22/0x2f [21872.236806] ? watchdog_timer_fn.cold.6+0x62/0x10a [21872.239876] ? __hrtimer_run_queues+0x1c2/0x4d0 [21872.242966] ? watchdog+0x40/0x40 [21872.244813] ? hrtimer_interrupt+0x128/0x2c0 [21872.247735] ? smp_apic_timer_interrupt+0xa6/0x270 [21872.251055] ? apic_timer_interrupt+0xf/0x20 [21872.252601] [21872.253368] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21872.263888] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21872.272588] do_raw_spin_lock+0x177/0x190 [21872.274020] _raw_spin_lock+0x21/0x30 [21872.280416] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21872.291814] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21872.295637] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21872.301764] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21872.305855] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21872.309240] ? woken_wake_function+0x30/0x30 [21872.311935] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21872.315763] kthread+0x1d1/0x200 [21872.317852] ? set_kthread_struct+0x70/0x70 [21872.320428] ret_from_fork+0x1f/0x30 [21876.053902] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [21876.061680] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21876.107872] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21876.123336] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21876.143403] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21876.150712] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21876.158293] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21876.182558] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21876.186495] RAX: 0000000000007cc1 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21876.195734] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [21876.210177] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21876.221676] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21876.227718] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21876.237852] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21876.249033] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21876.256550] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21876.274513] Call Trace: [21876.279313] [21876.281150] ? show_regs.cold.9+0x22/0x2f [21876.289638] ? watchdog_timer_fn.cold.6+0x62/0x10a [21876.291263] ? __hrtimer_run_queues+0x1c2/0x4d0 [21876.305222] ? watchdog+0x40/0x40 [21876.313816] ? hrtimer_interrupt+0x128/0x2c0 [21876.320859] ? smp_apic_timer_interrupt+0xa6/0x270 [21876.330767] ? apic_timer_interrupt+0xf/0x20 [21876.334790] [21876.336644] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21876.339683] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21876.343804] ? newidle_balance.isra.93+0x207/0x580 [21876.345810] do_raw_spin_lock+0x177/0x190 [21876.347272] _raw_spin_lock+0x21/0x30 [21876.348608] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21876.350679] process_one_work+0x2c8/0x700 [21876.352117] worker_thread+0x296/0x6e0 [21876.353576] ? rescuer_thread+0x570/0x570 [21876.354977] kthread+0x1d1/0x200 [21876.356119] ? set_kthread_struct+0x70/0x70 [21876.357772] ret_from_fork+0x1f/0x30 [21879.785440] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 19242s! [21879.791532] Showing busy workqueues and worker pools: [21879.794163] workqueue events: flags=0x0 [21879.797482] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21879.811577] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21879.827514] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21879.831968] pending: cpu_latency_work [ptlrpc] [21879.833607] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21879.839138] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21879.844587] workqueue events_long: flags=0x0 [21879.847904] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21879.850288] pending: lu_objects_flush [obdclass] [21879.854546] workqueue events_unbound: flags=0x2 [21879.857876] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21879.861035] pending: flush_memcg_stats_dwork, idle_cull_fn [21879.863167] workqueue events_power_efficient: flags=0x80 [21879.865571] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21879.868857] pending: check_lifetime, do_cache_clean [21879.873371] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21879.875831] pending: neigh_periodic_work, neigh_periodic_work [21879.877562] workqueue mm_percpu_wq: flags=0x8 [21879.882575] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21879.886320] pending: vmstat_update, lru_add_drain_per_cpu [21879.889522] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21879.895405] pending: vmstat_update, lru_add_drain_per_cpu [21879.896933] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21879.898331] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21879.904097] workqueue xprtiod: flags=0xa [21879.908245] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21879.913585] pending: xprt_autoclose, xs_error_handle [21879.918497] workqueue ipv6_addrconf: flags=0x40008 [21879.921554] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21879.926469] pending: addrconf_verify_work [21879.928846] workqueue knbd0-recv: flags=0x1a [21879.931807] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21879.934242] in-flight: 651:recv_work [21879.935410] workqueue ptlrpc_pinger: flags=0x2 [21879.939052] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21879.943626] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21879.945832] pool 8: cpus=0-3 flags=0x4 nice=0 hung=19242s workers=4 idle: 15979 31824 8 [21879.950275] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21870s workers=2 idle: 654 [21884.055075] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [21884.057020] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [21884.058329] Modules linked in: [21884.060996] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21884.061071] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21884.061078] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21884.061080] RIP: 0010:native_safe_halt+0x2e/0x50 [21884.061092] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21884.061096] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21884.061101] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21884.061103] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21884.061105] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21884.061108] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21884.061110] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21884.061113] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21884.061117] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21884.061119] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21884.061128] Call Trace: [21884.061134] [21884.062447] lustre(O) [21884.077599] ? show_regs.cold.9+0x22/0x2f [21884.082383] osp(O) [21884.085591] ? watchdog_timer_fn.cold.6+0x62/0x10a [21884.087264] ofd(O) [21884.093667] ? __hrtimer_run_queues+0x1c2/0x4d0 [21884.098657] lod(O) [21884.101529] ? watchdog+0x40/0x40 [21884.105622] mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21884.105691] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21884.105698] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21884.105701] RIP: 0010:native_safe_halt+0x2e/0x50 [21884.105715] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21884.105720] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21884.105724] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21884.105726] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21884.105729] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21884.105731] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21884.105733] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21884.105737] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21884.105739] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21884.105742] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21884.105752] Call Trace: [21884.105757] [21884.105763] ? show_regs.cold.9+0x22/0x2f [21884.105773] ? watchdog_timer_fn.cold.6+0x62/0x10a [21884.105779] ? __hrtimer_run_queues+0x1c2/0x4d0 [21884.105782] ? watchdog+0x40/0x40 [21884.105787] ? hrtimer_interrupt+0x128/0x2c0 [21884.105790] ? smp_apic_timer_interrupt+0xa6/0x270 [21884.105796] ? apic_timer_interrupt+0xf/0x20 [21884.105801] [21884.105804] ? native_safe_halt+0x2e/0x50 [21884.105807] kvm_wait+0xcd/0xe0 [21884.105815] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21884.105823] do_raw_spin_lock+0x177/0x190 [21884.105827] _raw_spin_lock+0x21/0x30 [21884.105832] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21884.111135] ? hrtimer_interrupt+0x128/0x2c0 [21884.111154] ? smp_apic_timer_interrupt+0xa6/0x270 [21884.111163] ? apic_timer_interrupt+0xf/0x20 [21884.111168] [21884.111173] ? native_safe_halt+0x2e/0x50 [21884.111178] kvm_wait+0xcd/0xe0 [21884.111188] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21884.111195] do_raw_spin_lock+0x177/0x190 [21884.113825] ? do_raw_spin_unlock+0x75/0x190 [21884.113923] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21884.114550] ? do_wait_intr+0xf0/0xf0 [21884.114561] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21884.114769] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21884.115407] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21884.119074] _raw_spin_lock+0x21/0x30 [21884.119095] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21884.122853] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21884.122970] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21884.123030] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21884.123061] mdc_intent_lock+0x353/0x7e0 [mdc] [21884.126548] obd_mod_free+0x46/0x140 [ptlrpc] [21884.130356] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21884.130946] mdc_commit_open+0x91/0xb0 [mdc] [21884.132788] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21884.137717] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21884.137986] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21884.138191] after_reply+0xe2c/0x1780 [ptlrpc] [21884.138939] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21884.139136] ? __next_timer_interrupt+0x160/0x160 [21884.139147] ptlrpcd+0xa82/0xe40 [ptlrpc] [21884.139648] ? do_wait_intr+0xf0/0xf0 [21884.139658] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21884.140005] kthread+0x1d1/0x200 [21884.140012] ? set_kthread_struct+0x70/0x70 [21884.140016] ret_from_fork+0x1f/0x30 [21884.568141] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21884.569997] lmv_intent_open+0x338/0x1150 [lmv] [21884.571620] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21884.573616] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21884.575667] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21884.577079] ll_intent_lock+0xc5/0xaa0 [lustre] [21884.578768] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21884.583505] ll_lookup_it+0xe0e/0x2270 [lustre] [21884.585833] ? do_raw_spin_unlock+0x75/0x190 [21884.587459] ? _raw_spin_unlock+0x12/0x30 [21884.588435] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21884.589690] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21884.591223] ll_atomic_open+0x36a/0xd70 [lustre] [21884.593025] lookup_open+0x356/0xb40 [21884.594400] do_last+0x313/0xfc0 [21884.595671] ? nd_jump_root+0xe5/0x160 [21884.597075] ? path_init+0x437/0x520 [21884.598648] path_openat+0xf7/0x500 [21884.599967] ? do_raw_spin_unlock+0x75/0x190 [21884.601816] do_filp_open+0x99/0x140 [21884.603159] ? getname_flags+0x6e/0x330 [21884.604822] ? __check_object_size+0xff/0x256 [21884.606496] ? do_raw_spin_unlock+0x75/0x190 [21884.608450] ? _raw_spin_unlock+0x12/0x30 [21884.610205] do_sys_openat2+0x2b4/0x410 [21884.612185] do_sys_open+0x73/0xa0 [21884.613805] __x64_sys_openat+0x24/0x30 [21884.615937] do_syscall_64+0xc1/0x440 [21884.617247] entry_SYSCALL_64_after_hwframe+0x49/0xae [21884.621204] RIP: 0033:0x7f53425a5332 [21884.623847] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21884.634367] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21884.637049] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21884.639903] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21884.642603] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21884.645133] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21884.647183] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21900.058053] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [21900.063150] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21900.125552] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21900.142305] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21900.144845] RIP: 0010:native_safe_halt+0x2e/0x50 [21900.151920] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21900.176053] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21900.178734] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21900.184907] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21900.187673] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21900.189422] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21900.201545] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21900.211182] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21900.215217] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21900.218139] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21900.222689] Call Trace: [21900.224574] [21900.225368] ? show_regs.cold.9+0x22/0x2f [21900.227114] ? watchdog_timer_fn.cold.6+0x62/0x10a [21900.229157] ? __hrtimer_run_queues+0x1c2/0x4d0 [21900.231497] ? watchdog+0x40/0x40 [21900.232838] ? hrtimer_interrupt+0x128/0x2c0 [21900.234897] ? smp_apic_timer_interrupt+0xa6/0x270 [21900.237582] ? apic_timer_interrupt+0xf/0x20 [21900.239405] [21900.240129] ? native_safe_halt+0x2e/0x50 [21900.241428] kvm_wait+0xcd/0xe0 [21900.242585] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21900.244762] do_raw_spin_lock+0x177/0x190 [21900.246560] _raw_spin_lock+0x21/0x30 [21900.248362] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21900.252236] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21900.254653] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21900.257540] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21900.259905] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21900.262233] ? woken_wake_function+0x30/0x30 [21900.264941] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21900.268024] kthread+0x1d1/0x200 [21900.270104] ? set_kthread_struct+0x70/0x70 [21900.271292] ret_from_fork+0x1f/0x30 [21904.052113] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [21904.056477] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21904.078760] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21904.087066] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21904.089828] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21904.092302] RIP: 0010:native_safe_halt+0x2e/0x50 [21904.093472] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21904.099695] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21904.102658] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21904.105262] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [21904.107810] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21904.110338] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21904.112412] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21904.114897] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21904.117408] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21904.119635] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21904.121867] Call Trace: [21904.122694] [21904.123416] ? show_regs.cold.9+0x22/0x2f [21904.124653] ? watchdog_timer_fn.cold.6+0x62/0x10a [21904.127088] ? __hrtimer_run_queues+0x1c2/0x4d0 [21904.128377] ? watchdog+0x40/0x40 [21904.129836] ? hrtimer_interrupt+0x128/0x2c0 [21904.131348] ? smp_apic_timer_interrupt+0xa6/0x270 [21904.133351] ? apic_timer_interrupt+0xf/0x20 [21904.134821] [21904.135618] ? native_safe_halt+0x2e/0x50 [21904.136771] kvm_wait+0xcd/0xe0 [21904.137788] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21904.140047] ? newidle_balance.isra.93+0x207/0x580 [21904.141552] do_raw_spin_lock+0x177/0x190 [21904.143610] _raw_spin_lock+0x21/0x30 [21904.146784] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21904.149411] process_one_work+0x2c8/0x700 [21904.151321] worker_thread+0x296/0x6e0 [21904.153354] ? rescuer_thread+0x570/0x570 [21904.155307] kthread+0x1d1/0x200 [21904.156962] ? set_kthread_struct+0x70/0x70 [21904.159055] ret_from_fork+0x1f/0x30 [21909.984321] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 19272s! [21909.989519] Showing busy workqueues and worker pools: [21909.992564] workqueue events: flags=0x0 [21909.995794] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21909.998985] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21910.009181] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21910.016064] pending: cpu_latency_work [ptlrpc] [21910.023366] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21910.028131] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21910.034839] workqueue events_long: flags=0x0 [21910.038559] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21910.046519] pending: lu_objects_flush [obdclass] [21910.050256] workqueue events_unbound: flags=0x2 [21910.052482] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21910.056341] pending: flush_memcg_stats_dwork, idle_cull_fn [21910.059770] workqueue events_power_efficient: flags=0x80 [21910.063058] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21910.066619] pending: check_lifetime, do_cache_clean [21910.069105] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21910.072018] pending: neigh_periodic_work, neigh_periodic_work [21910.074727] workqueue mm_percpu_wq: flags=0x8 [21910.076937] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21910.079725] pending: vmstat_update, lru_add_drain_per_cpu [21910.082297] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21910.087789] pending: vmstat_update, lru_add_drain_per_cpu [21910.096331] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21910.103065] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21910.107778] workqueue xprtiod: flags=0xa [21910.109761] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21910.113743] pending: xprt_autoclose, xs_error_handle [21910.121287] workqueue ipv6_addrconf: flags=0x40008 [21910.130068] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21910.135395] pending: addrconf_verify_work [21910.137566] workqueue knbd0-recv: flags=0x1a [21910.140701] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21910.144338] in-flight: 651:recv_work [21910.151452] workqueue ptlrpc_pinger: flags=0x2 [21910.154782] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21910.157545] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21910.160286] pool 8: cpus=0-3 flags=0x4 nice=0 hung=19273s workers=4 idle: 15979 31824 8 [21910.164255] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21900s workers=2 idle: 654 [21912.055754] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [21912.057976] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21912.059194] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [21912.072547] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21912.072561] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21912.072564] RIP: 0010:native_safe_halt+0x2e/0x50 [21912.072575] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21912.072580] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [21912.072585] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21912.072588] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21912.072590] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21912.072591] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21912.072594] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21912.072597] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21912.072599] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21912.072602] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21912.072611] Call Trace: [21912.072617] [21912.072623] ? show_regs.cold.9+0x22/0x2f [21912.072633] ? watchdog_timer_fn.cold.6+0x62/0x10a [21912.072640] ? __hrtimer_run_queues+0x1c2/0x4d0 [21912.072644] ? watchdog+0x40/0x40 [21912.072648] ? hrtimer_interrupt+0x128/0x2c0 [21912.072653] ? smp_apic_timer_interrupt+0xa6/0x270 [21912.072659] ? apic_timer_interrupt+0xf/0x20 [21912.072663] [21912.072666] ? native_safe_halt+0x2e/0x50 [21912.072670] kvm_wait+0xcd/0xe0 [21912.072676] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21912.072686] do_raw_spin_lock+0x177/0x190 [21912.072690] _raw_spin_lock+0x21/0x30 [21912.072694] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21912.072941] ? do_raw_spin_unlock+0x75/0x190 [21912.072945] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21912.073233] ? do_wait_intr+0xf0/0xf0 [21912.073242] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21912.073511] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21912.073720] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21912.073941] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21912.074180] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21912.074217] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21912.074244] mdc_intent_lock+0x353/0x7e0 [mdc] [21912.074269] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21912.074410] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21912.074621] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21912.074648] lmv_intent_open+0x338/0x1150 [lmv] [21912.074677] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21912.074761] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21912.088333] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21912.088411] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21912.088418] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21912.088421] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21912.088436] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21912.088440] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21912.088445] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21912.088448] RDX: 0000000000000c62 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21912.088450] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21912.088452] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21912.088455] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21912.088458] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21912.088461] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21912.088464] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21912.088473] Call Trace: [21912.088480] [21912.088486] ? show_regs.cold.9+0x22/0x2f [21912.088495] ? watchdog_timer_fn.cold.6+0x62/0x10a [21912.088502] ? __hrtimer_run_queues+0x1c2/0x4d0 [21912.088506] ? watchdog+0x40/0x40 [21912.088510] ? hrtimer_interrupt+0x128/0x2c0 [21912.088515] ? smp_apic_timer_interrupt+0xa6/0x270 [21912.088523] ? apic_timer_interrupt+0xf/0x20 [21912.088527] [21912.088530] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21912.088535] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21912.088540] do_raw_spin_lock+0x177/0x190 [21912.088544] _raw_spin_lock+0x21/0x30 [21912.088551] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21912.088865] obd_mod_free+0x46/0x140 [ptlrpc] [21912.094984] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21912.098062] mdc_commit_open+0x91/0xb0 [mdc] [21912.098112] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21912.098378] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21912.108547] ll_intent_lock+0xc5/0xaa0 [lustre] [21912.110556] after_reply+0xe2c/0x1780 [ptlrpc] [21912.116771] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21912.119693] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21912.120278] ? __next_timer_interrupt+0x160/0x160 [21912.120290] ptlrpcd+0xa82/0xe40 [ptlrpc] [21912.124026] ll_lookup_it+0xe0e/0x2270 [lustre] [21912.127409] ? do_wait_intr+0xf0/0xf0 [21912.132973] ? do_raw_spin_unlock+0x75/0x190 [21912.135573] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21912.135957] kthread+0x1d1/0x200 [21912.135966] ? set_kthread_struct+0x70/0x70 [21912.135971] ret_from_fork+0x1f/0x30 [21912.498726] ? _raw_spin_unlock+0x12/0x30 [21912.501834] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21912.505224] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21912.508348] ll_atomic_open+0x36a/0xd70 [lustre] [21912.513641] lookup_open+0x356/0xb40 [21912.516056] do_last+0x313/0xfc0 [21912.518196] ? nd_jump_root+0xe5/0x160 [21912.520719] ? path_init+0x437/0x520 [21912.522841] path_openat+0xf7/0x500 [21912.524877] ? do_raw_spin_unlock+0x75/0x190 [21912.526622] do_filp_open+0x99/0x140 [21912.528228] ? getname_flags+0x6e/0x330 [21912.530407] ? __check_object_size+0xff/0x256 [21912.532766] ? do_raw_spin_unlock+0x75/0x190 [21912.535832] ? _raw_spin_unlock+0x12/0x30 [21912.539634] do_sys_openat2+0x2b4/0x410 [21912.541763] do_sys_open+0x73/0xa0 [21912.543061] __x64_sys_openat+0x24/0x30 [21912.545394] do_syscall_64+0xc1/0x440 [21912.548647] entry_SYSCALL_64_after_hwframe+0x49/0xae [21912.552200] RIP: 0033:0x7f53425a5332 [21912.553924] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21912.564411] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21912.566875] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21912.569130] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21912.572012] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21912.574392] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21912.577144] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21928.058024] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [21928.062675] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21928.098932] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21928.115062] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21928.119643] RIP: 0010:native_safe_halt+0x2e/0x50 [21928.122345] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21928.138799] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21928.144686] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21928.150153] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21928.154751] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21928.161281] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21928.166417] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21928.172332] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21928.176206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21928.183646] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21928.187357] Call Trace: [21928.188945] [21928.189999] ? show_regs.cold.9+0x22/0x2f [21928.192273] ? watchdog_timer_fn.cold.6+0x62/0x10a [21928.194455] ? __hrtimer_run_queues+0x1c2/0x4d0 [21928.199982] ? watchdog+0x40/0x40 [21928.201804] ? hrtimer_interrupt+0x128/0x2c0 [21928.204542] ? smp_apic_timer_interrupt+0xa6/0x270 [21928.207059] ? apic_timer_interrupt+0xf/0x20 [21928.210092] [21928.210883] ? native_safe_halt+0x2e/0x50 [21928.213477] kvm_wait+0xcd/0xe0 [21928.215684] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21928.222270] do_raw_spin_lock+0x177/0x190 [21928.227130] _raw_spin_lock+0x21/0x30 [21928.229533] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21928.233967] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21928.239552] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21928.243584] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21928.246988] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21928.253983] ? woken_wake_function+0x30/0x30 [21928.260724] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21928.268056] kthread+0x1d1/0x200 [21928.270095] ? set_kthread_struct+0x70/0x70 [21928.273727] ret_from_fork+0x1f/0x30 [21936.052046] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [21936.055330] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21936.076600] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21936.079814] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21936.088599] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21936.091912] RIP: 0010:native_safe_halt+0x2e/0x50 [21936.093594] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [21936.100663] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21936.104119] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21936.111240] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [21936.121831] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21936.131696] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21936.133500] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21936.145502] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21936.150684] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21936.154613] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21936.159617] Call Trace: [21936.160271] [21936.160786] ? show_regs.cold.9+0x22/0x2f [21936.161727] ? watchdog_timer_fn.cold.6+0x62/0x10a [21936.164339] ? __hrtimer_run_queues+0x1c2/0x4d0 [21936.165805] ? watchdog+0x40/0x40 [21936.166879] ? hrtimer_interrupt+0x128/0x2c0 [21936.172677] ? smp_apic_timer_interrupt+0xa6/0x270 [21936.174168] ? apic_timer_interrupt+0xf/0x20 [21936.175887] [21936.176538] ? native_safe_halt+0x2e/0x50 [21936.177718] kvm_wait+0xcd/0xe0 [21936.178868] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21936.181816] ? newidle_balance.isra.93+0x207/0x580 [21936.183568] do_raw_spin_lock+0x177/0x190 [21936.184885] _raw_spin_lock+0x21/0x30 [21936.187822] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21936.189799] process_one_work+0x2c8/0x700 [21936.191996] worker_thread+0x296/0x6e0 [21936.193320] ? rescuer_thread+0x570/0x570 [21936.194673] kthread+0x1d1/0x200 [21936.195747] ? set_kthread_struct+0x70/0x70 [21936.201909] ret_from_fork+0x1f/0x30 [21940.055025] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [21940.057378] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [21940.058727] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul [21940.061856] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21940.061954] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21940.061963] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21940.061965] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21940.061979] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21940.061984] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21940.061989] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21940.061992] RDX: 0000000000004807 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21940.061994] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21940.061997] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21940.061999] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21940.062064] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21940.062067] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21940.062070] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21940.062078] Call Trace: [21940.062084] [21940.062090] ? show_regs.cold.9+0x22/0x2f [21940.062104] ? watchdog_timer_fn.cold.6+0x62/0x10a [21940.062115] ? __hrtimer_run_queues+0x1c2/0x4d0 [21940.062120] ? watchdog+0x40/0x40 [21940.062125] ? hrtimer_interrupt+0x128/0x2c0 [21940.062133] ? smp_apic_timer_interrupt+0xa6/0x270 [21940.062143] ? apic_timer_interrupt+0xf/0x20 [21940.062148] [21940.062151] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21940.062159] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21940.062165] do_raw_spin_lock+0x177/0x190 [21940.062172] _raw_spin_lock+0x21/0x30 [21940.062178] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21940.062533] obd_mod_free+0x46/0x140 [ptlrpc] [21940.062767] mdc_commit_open+0x91/0xb0 [mdc] [21940.062811] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21940.063113] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21940.063383] after_reply+0xe2c/0x1780 [ptlrpc] [21940.063648] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21940.064172] ? __next_timer_interrupt+0x160/0x160 [21940.064186] ptlrpcd+0xa82/0xe40 [ptlrpc] [21940.064446] ? do_wait_intr+0xf0/0xf0 [21940.064455] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21940.067997] kthread+0x1d1/0x200 [21940.068020] ? set_kthread_struct+0x70/0x70 [21940.068025] ret_from_fork+0x1f/0x30 [21940.200743] BUG: workqueue lockup - pool [21940.205922] crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21940.205950] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21940.205957] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21940.205960] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21940.205974] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21940.205978] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21940.205983] RAX: 0000000000000025 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21940.205985] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21940.205987] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21940.205989] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21940.205991] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21940.205994] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21940.205996] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21940.205999] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21940.206008] Call Trace: [21940.206013] [21940.206018] ? show_regs.cold.9+0x22/0x2f [21940.206027] ? watchdog_timer_fn.cold.6+0x62/0x10a [21940.206033] ? __hrtimer_run_queues+0x1c2/0x4d0 [21940.206037] ? watchdog+0x40/0x40 [21940.206041] ? hrtimer_interrupt+0x128/0x2c0 [21940.206044] ? smp_apic_timer_interrupt+0xa6/0x270 [21940.206052] ? apic_timer_interrupt+0xf/0x20 [21940.206056] [21940.206059] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21940.206066] do_raw_spin_lock+0x177/0x190 [21940.206070] _raw_spin_lock+0x21/0x30 [21940.206075] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21940.206366] ? do_raw_spin_unlock+0x75/0x190 [21940.206372] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21940.206809] ? do_wait_intr+0xf0/0xf0 [21940.206818] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21940.207020] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21940.207233] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21940.215411] cpus=0-3 flags=0x4 nice=0 stuck for 19303s! [21940.215430] Showing busy workqueues and worker pools: [21940.215432] workqueue events: flags=0x0 [21940.215436] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21940.215443] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21940.215866] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21940.215875] pending: cpu_latency_work [ptlrpc] [21940.216100] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21940.216110] pending: vmstat_shepherd, osc_grant_work_handler [osc] [21940.220238] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21940.220683] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21940.220723] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21940.220757] mdc_intent_lock+0x353/0x7e0 [mdc] [21940.220785] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21940.220871] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21940.224062] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21940.224122] lmv_intent_open+0x338/0x1150 [lmv] [21940.224151] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21940.224257] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21940.224454] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21940.224485] ll_intent_lock+0xc5/0xaa0 [lustre] [21940.224887] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21940.224970] ll_lookup_it+0xe0e/0x2270 [lustre] [21940.225065] ? do_raw_spin_unlock+0x75/0x190 [21940.225073] ? _raw_spin_unlock+0x12/0x30 [21940.225079] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21940.225119] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21940.225137] ll_atomic_open+0x36a/0xd70 [lustre] [21940.225218] lookup_open+0x356/0xb40 [21940.225225] do_last+0x313/0xfc0 [21940.225230] ? nd_jump_root+0xe5/0x160 [21940.225235] ? path_init+0x437/0x520 [21940.225239] path_openat+0xf7/0x500 [21940.225243] ? do_raw_spin_unlock+0x75/0x190 [21940.225248] do_filp_open+0x99/0x140 [21940.225254] ? getname_flags+0x6e/0x330 [21940.225257] ? __check_object_size+0xff/0x256 [21940.225264] ? do_raw_spin_unlock+0x75/0x190 [21940.225267] ? _raw_spin_unlock+0x12/0x30 [21940.225272] do_sys_openat2+0x2b4/0x410 [21940.225276] do_sys_open+0x73/0xa0 [21940.225280] __x64_sys_openat+0x24/0x30 [21940.225470] do_syscall_64+0xc1/0x440 [21940.225482] entry_SYSCALL_64_after_hwframe+0x49/0xae [21940.225488] RIP: 0033:0x7f53425a5332 [21940.225496] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21940.225500] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21940.225505] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21940.225507] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21940.225510] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21940.225512] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21940.225514] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21940.641904] , sec_gc_main [ptlrpc] [21940.643557] workqueue events_long: flags=0x0 [21940.644824] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21940.647576] pending: lu_objects_flush [obdclass] [21940.653473] workqueue events_unbound: flags=0x2 [21940.655009] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21940.667897] pending: flush_memcg_stats_dwork, idle_cull_fn [21940.674947] workqueue events_power_efficient: flags=0x80 [21940.683017] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21940.686464] pending: check_lifetime, do_cache_clean [21940.689271] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21940.696023] pending: neigh_periodic_work, neigh_periodic_work [21940.698935] workqueue mm_percpu_wq: flags=0x8 [21940.701724] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21940.705506] pending: vmstat_update, lru_add_drain_per_cpu [21940.711343] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21940.717902] pending: vmstat_update, lru_add_drain_per_cpu [21940.723923] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21940.728787] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21940.742421] workqueue xprtiod: flags=0xa [21940.746523] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21940.754922] pending: xprt_autoclose, xs_error_handle [21940.761085] workqueue ipv6_addrconf: flags=0x40008 [21940.766362] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21940.773345] pending: addrconf_verify_work [21940.774874] workqueue knbd0-recv: flags=0x1a [21940.781453] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21940.794880] in-flight: 651:recv_work [21940.796150] workqueue ptlrpc_pinger: flags=0x2 [21940.798126] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21940.806238] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21940.811077] pool 8: cpus=0-3 flags=0x4 nice=0 hung=19303s workers=4 idle: 15979 31824 8 [21940.818238] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21931s workers=2 idle: 654 [21954.915027] rcu: INFO: rcu_sched self-detected stall on CPU [21954.917642] rcu: 2-....: (11472040 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [21954.923318] (t=19860489 jiffies g=274185 q=69008) [21954.926797] rcu: rcu_sched kthread starved for 19318855 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2 [21954.933400] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [21954.938387] rcu: RCU grace-period kthread stack dump: [21954.941161] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [21954.953763] Call Trace: [21954.959397] __schedule+0x351/0xcb0 [21954.960489] schedule+0xc0/0x180 [21954.965551] schedule_timeout+0xb4/0x190 [21954.966956] ? do_raw_spin_unlock+0x75/0x190 [21954.968438] ? __next_timer_interrupt+0x160/0x160 [21954.971173] ? prepare_to_swait_event+0xf3/0x140 [21954.972742] rcu_gp_kthread+0x8dd/0x15b0 [21954.974120] ? force_qs_rnp+0x2c0/0x2c0 [21954.975472] kthread+0x1d1/0x200 [21954.976564] ? set_kthread_struct+0x70/0x70 [21954.979430] ret_from_fork+0x1f/0x30 [21954.985124] rcu: Stack dump where RCU GP kthread last ran: [21954.987651] NMI backtrace for cpu 2 [21954.994113] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21955.008346] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21955.018611] Call Trace: [21955.021593] [21955.023340] ? dump_stack+0xbb/0x10e [21955.025292] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [21955.028895] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [21955.030553] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [21955.034724] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [21955.040310] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [21955.046249] ? rcu_check_gp_kthread_starvation+0x16f/0x188 [21955.049877] ? rcu_sched_clock_irq.cold.51+0xce/0x307 [21955.055966] ? update_process_times+0x6d/0xc0 [21955.063732] ? tick_sched_handle.isra.18+0x27/0x90 [21955.068146] ? tick_sched_timer+0x53/0xb0 [21955.070910] ? __hrtimer_run_queues+0x1c2/0x4d0 [21955.075613] ? tick_sched_handle.isra.18+0x90/0x90 [21955.079989] ? hrtimer_interrupt+0x128/0x2c0 [21955.081874] ? smp_apic_timer_interrupt+0xa6/0x270 [21955.085497] ? apic_timer_interrupt+0xf/0x20 [21955.087472] [21955.095741] ? native_safe_halt+0x2e/0x50 [21955.098919] ? kvm_wait+0xcd/0xe0 [21955.103486] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21955.108157] ? do_raw_spin_lock+0x177/0x190 [21955.113271] ? _raw_spin_lock+0x21/0x30 [21955.116787] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21955.124629] ? obd_mod_free+0x46/0x140 [ptlrpc] [21955.129979] ? mdc_commit_open+0x91/0xb0 [mdc] [21955.133637] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21955.136701] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21955.139816] ? after_reply+0xe2c/0x1780 [ptlrpc] [21955.142635] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21955.146992] ? __next_timer_interrupt+0x160/0x160 [21955.150715] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [21955.161087] ? do_wait_intr+0xf0/0xf0 [21955.162425] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21955.167876] ? kthread+0x1d1/0x200 [21955.170372] ? set_kthread_struct+0x70/0x70 [21955.175217] ? ret_from_fork+0x1f/0x30 [21955.180124] Sending NMI from CPU 2 to CPUs 1: [21955.184292] NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0x2e/0x50 [21955.185242] NMI backtrace for cpu 2 [21955.194674] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21955.209208] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21955.215298] Call Trace: [21955.216688] [21955.217854] ? dump_stack+0xbb/0x10e [21955.224164] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [21955.232702] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [21955.234657] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [21955.237004] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [21955.239880] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [21955.242144] ? rcu_dump_cpu_stacks+0x11b/0x171 [21955.243582] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [21955.252591] ? update_process_times+0x6d/0xc0 [21955.258907] ? tick_sched_handle.isra.18+0x27/0x90 [21955.260552] ? tick_sched_timer+0x53/0xb0 [21955.261740] ? __hrtimer_run_queues+0x1c2/0x4d0 [21955.263705] ? tick_sched_handle.isra.18+0x90/0x90 [21955.265201] ? hrtimer_interrupt+0x128/0x2c0 [21955.266615] ? smp_apic_timer_interrupt+0xa6/0x270 [21955.270142] ? apic_timer_interrupt+0xf/0x20 [21955.272546] [21955.273360] ? native_safe_halt+0x2e/0x50 [21955.274862] ? kvm_wait+0xcd/0xe0 [21955.276210] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21955.278316] ? do_raw_spin_lock+0x177/0x190 [21955.282423] ? _raw_spin_lock+0x21/0x30 [21955.283902] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21955.288610] ? obd_mod_free+0x46/0x140 [ptlrpc] [21955.291665] ? mdc_commit_open+0x91/0xb0 [mdc] [21955.293700] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21955.297010] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21955.299568] ? after_reply+0xe2c/0x1780 [ptlrpc] [21955.301653] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21955.304302] ? __next_timer_interrupt+0x160/0x160 [21955.306851] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [21955.309007] ? do_wait_intr+0xf0/0xf0 [21955.312500] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21955.314534] ? kthread+0x1d1/0x200 [21955.319929] ? set_kthread_struct+0x70/0x70 [21955.321966] ? ret_from_fork+0x1f/0x30 [21956.059273] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [21956.068811] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21956.107545] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21956.113452] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21956.118404] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21956.123479] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21956.148834] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21956.152313] RAX: 0000000000006b14 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21956.156790] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21956.163133] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21956.167415] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21956.170758] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21956.178666] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21956.182172] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21956.185786] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21956.193193] Call Trace: [21956.195496] [21956.196525] ? show_regs.cold.9+0x22/0x2f [21956.198451] ? watchdog_timer_fn.cold.6+0x62/0x10a [21956.201837] ? __hrtimer_run_queues+0x1c2/0x4d0 [21956.204289] ? watchdog+0x40/0x40 [21956.206500] ? hrtimer_interrupt+0x128/0x2c0 [21956.209023] ? smp_apic_timer_interrupt+0xa6/0x270 [21956.211949] ? apic_timer_interrupt+0xf/0x20 [21956.213935] [21956.215182] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21956.218216] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21956.220529] do_raw_spin_lock+0x177/0x190 [21956.222401] _raw_spin_lock+0x21/0x30 [21956.223904] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21956.226686] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21956.229367] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21956.232763] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21956.235314] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21956.238227] ? woken_wake_function+0x30/0x30 [21956.240307] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21956.249393] kthread+0x1d1/0x200 [21956.254712] ? set_kthread_struct+0x70/0x70 [21956.259451] ret_from_fork+0x1f/0x30 [21968.058818] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [21968.067435] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21968.095710] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21968.101135] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21968.103606] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21968.106294] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21968.108395] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21968.113994] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21968.116123] RAX: 000000000000402d RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21968.118286] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [21968.120535] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21968.122931] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21968.124994] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21968.127348] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21968.135259] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21968.137781] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21968.141012] Call Trace: [21968.141957] [21968.142949] ? show_regs.cold.9+0x22/0x2f [21968.146209] ? watchdog_timer_fn.cold.6+0x62/0x10a [21968.153104] ? __hrtimer_run_queues+0x1c2/0x4d0 [21968.155137] ? watchdog+0x40/0x40 [21968.157065] ? hrtimer_interrupt+0x128/0x2c0 [21968.158246] ? smp_apic_timer_interrupt+0xa6/0x270 [21968.159528] ? apic_timer_interrupt+0xf/0x20 [21968.161505] [21968.162613] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21968.164746] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21968.166309] ? newidle_balance.isra.93+0x207/0x580 [21968.171298] do_raw_spin_lock+0x177/0x190 [21968.173141] _raw_spin_lock+0x21/0x30 [21968.174697] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21968.176962] process_one_work+0x2c8/0x700 [21968.178914] worker_thread+0x296/0x6e0 [21968.180348] ? rescuer_thread+0x570/0x570 [21968.181787] kthread+0x1d1/0x200 [21968.186884] ? set_kthread_struct+0x70/0x70 [21968.188544] ret_from_fork+0x1f/0x30 [21970.914023] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 19333s! [21970.922222] Showing busy workqueues and worker pools: [21970.925184] workqueue events: flags=0x0 [21970.927294] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [21970.930604] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [21970.937746] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21970.941347] pending: cpu_latency_work [ptlrpc] [21970.944168] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [21970.947119] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [21970.951685] workqueue events_long: flags=0x0 [21970.953317] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [21970.956476] pending: lu_objects_flush [obdclass] [21970.959577] workqueue events_unbound: flags=0x2 [21970.961907] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [21970.964972] pending: flush_memcg_stats_dwork, idle_cull_fn [21970.967138] workqueue events_power_efficient: flags=0x80 [21970.969203] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21970.972156] pending: check_lifetime, do_cache_clean [21970.974562] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21970.977874] pending: neigh_periodic_work, neigh_periodic_work [21970.980435] workqueue mm_percpu_wq: flags=0x8 [21970.982244] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21970.984927] pending: vmstat_update, lru_add_drain_per_cpu [21970.988313] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [21970.991002] pending: vmstat_update, lru_add_drain_per_cpu [21970.993707] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [21970.997351] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [21970.999534] workqueue xprtiod: flags=0xa [21971.002439] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [21971.005561] pending: xprt_autoclose, xs_error_handle [21971.008547] workqueue ipv6_addrconf: flags=0x40008 [21971.010419] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [21971.013092] pending: addrconf_verify_work [21971.015255] workqueue knbd0-recv: flags=0x1a [21971.017341] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [21971.020167] in-flight: 651:recv_work [21971.022896] workqueue ptlrpc_pinger: flags=0x2 [21971.025151] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [21971.027572] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [21971.030121] pool 8: cpus=0-3 flags=0x4 nice=0 hung=19333s workers=4 idle: 15979 31824 8 [21971.032652] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21961s workers=2 idle: 654 [21980.055092] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [21980.057257] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21980.061725] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [21980.069353] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21980.069367] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21980.069370] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21980.069383] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21980.069388] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21980.069394] RAX: 0000000000003487 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21980.069397] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [21980.069399] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [21980.069401] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [21980.069404] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [21980.069407] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [21980.069411] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21980.069413] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [21980.069421] Call Trace: [21980.069426] [21980.069433] ? show_regs.cold.9+0x22/0x2f [21980.069443] ? watchdog_timer_fn.cold.6+0x62/0x10a [21980.069449] ? __hrtimer_run_queues+0x1c2/0x4d0 [21980.069454] ? watchdog+0x40/0x40 [21980.069458] ? hrtimer_interrupt+0x128/0x2c0 [21980.069462] ? smp_apic_timer_interrupt+0xa6/0x270 [21980.069472] ? apic_timer_interrupt+0xf/0x20 [21980.069476] [21980.069479] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21980.069487] do_raw_spin_lock+0x177/0x190 [21980.069492] _raw_spin_lock+0x21/0x30 [21980.069498] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [21980.069777] ? do_raw_spin_unlock+0x75/0x190 [21980.069783] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [21980.069982] ? do_wait_intr+0xf0/0xf0 [21980.069991] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [21980.070188] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [21980.070403] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21980.070629] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21980.070738] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21980.070777] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [21980.070805] mdc_intent_lock+0x353/0x7e0 [mdc] [21980.070829] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21980.070909] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [21980.071120] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [21980.071146] lmv_intent_open+0x338/0x1150 [lmv] [21980.071168] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21980.071250] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21980.071333] lmv_intent_lock+0x1dd/0x4c0 [lmv] [21980.071355] ll_intent_lock+0xc5/0xaa0 [lustre] [21980.071442] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [21980.071527] ll_lookup_it+0xe0e/0x2270 [lustre] [21980.071616] ? do_raw_spin_unlock+0x75/0x190 [21980.071621] ? _raw_spin_unlock+0x12/0x30 [21980.071626] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [21980.071646] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [21980.071663] ll_atomic_open+0x36a/0xd70 [lustre] [21980.071750] lookup_open+0x356/0xb40 [21980.071759] do_last+0x313/0xfc0 [21980.071764] ? nd_jump_root+0xe5/0x160 [21980.071769] ? path_init+0x437/0x520 [21980.071773] path_openat+0xf7/0x500 [21980.071778] ? do_raw_spin_unlock+0x75/0x190 [21980.071783] do_filp_open+0x99/0x140 [21980.071790] ? getname_flags+0x6e/0x330 [21980.071794] ? __check_object_size+0xff/0x256 [21980.071801] ? do_raw_spin_unlock+0x75/0x190 [21980.071806] ? _raw_spin_unlock+0x12/0x30 [21980.071811] do_sys_openat2+0x2b4/0x410 [21980.071815] do_sys_open+0x73/0xa0 [21980.071820] __x64_sys_openat+0x24/0x30 [21980.071824] do_syscall_64+0xc1/0x440 [21980.071831] entry_SYSCALL_64_after_hwframe+0x49/0xae [21980.071837] RIP: 0033:0x7f53425a5332 [21980.071844] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [21980.071848] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [21980.071852] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [21980.071854] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [21980.071857] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [21980.071859] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [21980.071860] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [21980.440291] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21980.484205] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21980.498296] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21980.508086] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [21980.516369] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [21980.538345] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21980.545810] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [21980.556944] RDX: 0000000000007857 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [21980.563272] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [21980.569385] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [21980.573773] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [21980.577345] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [21980.582520] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21980.585237] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [21980.589302] Call Trace: [21980.590094] [21980.590737] ? show_regs.cold.9+0x22/0x2f [21980.591883] ? watchdog_timer_fn.cold.6+0x62/0x10a [21980.593549] ? __hrtimer_run_queues+0x1c2/0x4d0 [21980.595434] ? watchdog+0x40/0x40 [21980.596355] ? hrtimer_interrupt+0x128/0x2c0 [21980.598346] ? smp_apic_timer_interrupt+0xa6/0x270 [21980.600281] ? apic_timer_interrupt+0xf/0x20 [21980.602252] [21980.603134] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [21980.605855] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [21980.608267] do_raw_spin_lock+0x177/0x190 [21980.609863] _raw_spin_lock+0x21/0x30 [21980.612103] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [21980.614697] obd_mod_free+0x46/0x140 [ptlrpc] [21980.616986] mdc_commit_open+0x91/0xb0 [mdc] [21980.619668] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [21980.622810] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [21980.628771] after_reply+0xe2c/0x1780 [ptlrpc] [21980.634050] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [21980.636736] ? __next_timer_interrupt+0x160/0x160 [21980.639421] ptlrpcd+0xa82/0xe40 [ptlrpc] [21980.646426] ? do_wait_intr+0xf0/0xf0 [21980.647888] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [21980.651513] kthread+0x1d1/0x200 [21980.655345] ? set_kthread_struct+0x70/0x70 [21980.662553] ret_from_fork+0x1f/0x30 [21984.058017] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [21984.063939] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21984.099098] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21984.104557] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21984.108218] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21984.110253] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21984.128911] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [21984.132782] RAX: 0000000000006b0e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21984.136838] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [21984.139832] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [21984.149285] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [21984.152036] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [21984.158299] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [21984.173784] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21984.183236] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [21984.185730] Call Trace: [21984.193203] [21984.194227] ? show_regs.cold.9+0x22/0x2f [21984.196178] ? watchdog_timer_fn.cold.6+0x62/0x10a [21984.200635] ? __hrtimer_run_queues+0x1c2/0x4d0 [21984.202236] ? watchdog+0x40/0x40 [21984.219782] ? hrtimer_interrupt+0x128/0x2c0 [21984.223395] ? smp_apic_timer_interrupt+0xa6/0x270 [21984.228762] ? apic_timer_interrupt+0xf/0x20 [21984.230833] [21984.232870] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21984.237743] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21984.244613] do_raw_spin_lock+0x177/0x190 [21984.247078] _raw_spin_lock+0x21/0x30 [21984.251317] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [21984.257802] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [21984.262038] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [21984.266371] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [21984.272185] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [21984.279389] ? woken_wake_function+0x30/0x30 [21984.281947] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [21984.287953] kthread+0x1d1/0x200 [21984.289068] ? set_kthread_struct+0x70/0x70 [21984.298187] ret_from_fork+0x1f/0x30 [21996.052377] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [21996.055988] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [21996.073674] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [21996.082581] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [21996.087775] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [21996.094486] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21996.096773] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [21996.111884] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [21996.118854] RAX: 0000000000006296 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [21996.125429] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [21996.127421] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [21996.133165] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [21996.135861] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [21996.138214] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [21996.140786] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [21996.142223] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [21996.145712] Call Trace: [21996.147733] [21996.148778] ? show_regs.cold.9+0x22/0x2f [21996.152935] ? watchdog_timer_fn.cold.6+0x62/0x10a [21996.154637] ? __hrtimer_run_queues+0x1c2/0x4d0 [21996.156033] ? watchdog+0x40/0x40 [21996.157643] ? hrtimer_interrupt+0x128/0x2c0 [21996.158847] ? smp_apic_timer_interrupt+0xa6/0x270 [21996.160608] ? apic_timer_interrupt+0xf/0x20 [21996.164669] [21996.166264] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [21996.170393] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [21996.172879] ? newidle_balance.isra.93+0x207/0x580 [21996.177539] do_raw_spin_lock+0x177/0x190 [21996.178932] _raw_spin_lock+0x21/0x30 [21996.184670] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [21996.187364] process_one_work+0x2c8/0x700 [21996.189043] worker_thread+0x296/0x6e0 [21996.190731] ? rescuer_thread+0x570/0x570 [21996.192163] kthread+0x1d1/0x200 [21996.196289] ? set_kthread_struct+0x70/0x70 [21996.197678] ret_from_fork+0x1f/0x30 [22001.122703] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 19364s! [22001.130459] Showing busy workqueues and worker pools: [22001.133309] workqueue events: flags=0x0 [22001.138341] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [22001.144801] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [22001.154164] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [22001.161695] pending: cpu_latency_work [ptlrpc] [22001.168576] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [22001.175699] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [22001.183833] workqueue events_long: flags=0x0 [22001.186391] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [22001.192844] pending: lu_objects_flush [obdclass] [22001.199658] workqueue events_unbound: flags=0x2 [22001.202767] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [22001.209547] pending: flush_memcg_stats_dwork, idle_cull_fn [22001.214446] workqueue events_power_efficient: flags=0x80 [22001.218303] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22001.224793] pending: check_lifetime, do_cache_clean [22001.229765] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22001.236502] pending: neigh_periodic_work, neigh_periodic_work [22001.241808] workqueue mm_percpu_wq: flags=0x8 [22001.243734] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22001.250555] pending: vmstat_update, lru_add_drain_per_cpu [22001.256600] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22001.263305] pending: vmstat_update, lru_add_drain_per_cpu [22001.268317] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [22001.272478] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [22001.279352] workqueue xprtiod: flags=0xa [22001.284908] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [22001.291586] pending: xprt_autoclose, xs_error_handle [22001.294416] workqueue ipv6_addrconf: flags=0x40008 [22001.296643] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [22001.303214] pending: addrconf_verify_work [22001.306639] workqueue knbd0-recv: flags=0x1a [22001.310300] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [22001.316306] in-flight: 651:recv_work [22001.318033] workqueue ptlrpc_pinger: flags=0x2 [22001.320694] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [22001.323198] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [22001.326025] pool 8: cpus=0-3 flags=0x4 nice=0 hung=19364s workers=4 idle: 15979 31824 8 [22001.333074] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=21991s workers=2 idle: 654 [22008.057049] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [22008.057674] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [22008.068771] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22008.068855] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22008.068862] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22008.068865] RIP: 0010:native_safe_halt+0x2e/0x50 [22008.068878] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [22008.068882] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [22008.068886] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [22008.068889] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [22008.068891] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [22008.068894] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [22008.068896] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [22008.068900] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [22008.068903] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22008.068905] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [22008.068914] Call Trace: [22008.068920] [22008.068925] ? show_regs.cold.9+0x22/0x2f [22008.068935] ? watchdog_timer_fn.cold.6+0x62/0x10a [22008.068941] ? __hrtimer_run_queues+0x1c2/0x4d0 [22008.068945] ? watchdog+0x40/0x40 [22008.068949] ? hrtimer_interrupt+0x128/0x2c0 [22008.068953] ? smp_apic_timer_interrupt+0xa6/0x270 [22008.068960] ? apic_timer_interrupt+0xf/0x20 [22008.068965] [22008.068968] ? native_safe_halt+0x2e/0x50 [22008.068972] kvm_wait+0xcd/0xe0 [22008.075080] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22008.075166] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22008.075174] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22008.075176] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22008.075192] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [22008.075196] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [22008.075201] RAX: 0000000000003c15 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [22008.075203] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [22008.075205] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [22008.075206] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [22008.075209] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [22008.075212] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [22008.075214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22008.075216] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [22008.075225] Call Trace: [22008.075231] [22008.075236] ? show_regs.cold.9+0x22/0x2f [22008.075246] ? watchdog_timer_fn.cold.6+0x62/0x10a [22008.075252] ? __hrtimer_run_queues+0x1c2/0x4d0 [22008.075256] ? watchdog+0x40/0x40 [22008.075260] ? hrtimer_interrupt+0x128/0x2c0 [22008.075264] ? smp_apic_timer_interrupt+0xa6/0x270 [22008.075272] ? apic_timer_interrupt+0xf/0x20 [22008.075277] [22008.075280] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22008.075287] do_raw_spin_lock+0x177/0x190 [22008.075291] _raw_spin_lock+0x21/0x30 [22008.075296] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [22008.075652] ? do_raw_spin_unlock+0x75/0x190 [22008.075658] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [22008.095531] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [22008.104007] ? do_wait_intr+0xf0/0xf0 [22008.104031] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [22008.104334] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [22008.104797] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [22008.105611] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22008.105721] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [22008.105757] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [22008.105788] mdc_intent_lock+0x353/0x7e0 [mdc] [22008.106167] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22008.106256] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [22008.113435] do_raw_spin_lock+0x177/0x190 [22008.113453] _raw_spin_lock+0x21/0x30 [22008.113463] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [22008.115374] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [22008.115618] lmv_intent_open+0x338/0x1150 [lmv] [22008.115650] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22008.115755] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22008.115849] lmv_intent_lock+0x1dd/0x4c0 [lmv] [22008.115873] ll_intent_lock+0xc5/0xaa0 [lustre] [22008.115960] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22008.116047] ll_lookup_it+0xe0e/0x2270 [lustre] [22008.116142] ? do_raw_spin_unlock+0x75/0x190 [22008.116151] ? _raw_spin_unlock+0x12/0x30 [22008.116157] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [22008.116186] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [22008.116207] ll_atomic_open+0x36a/0xd70 [lustre] [22008.116295] lookup_open+0x356/0xb40 [22008.116304] do_last+0x313/0xfc0 [22008.116308] ? nd_jump_root+0xe5/0x160 [22008.116313] ? path_init+0x437/0x520 [22008.116317] path_openat+0xf7/0x500 [22008.116322] ? do_raw_spin_unlock+0x75/0x190 [22008.116328] do_filp_open+0x99/0x140 [22008.116334] ? getname_flags+0x6e/0x330 [22008.116339] ? __check_object_size+0xff/0x256 [22008.116346] ? do_raw_spin_unlock+0x75/0x190 [22008.116349] ? _raw_spin_unlock+0x12/0x30 [22008.116354] do_sys_openat2+0x2b4/0x410 [22008.116358] do_sys_open+0x73/0xa0 [22008.116363] __x64_sys_openat+0x24/0x30 [22008.116367] do_syscall_64+0xc1/0x440 [22008.116373] entry_SYSCALL_64_after_hwframe+0x49/0xae [22008.116393] RIP: 0033:0x7f53425a5332 [22008.116400] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [22008.116404] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [22008.116409] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [22008.116411] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [22008.116413] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [22008.116415] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [22008.116418] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [22008.588186] obd_mod_free+0x46/0x140 [ptlrpc] [22008.591895] mdc_commit_open+0x91/0xb0 [mdc] [22008.597941] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [22008.599789] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [22008.607529] after_reply+0xe2c/0x1780 [ptlrpc] [22008.618219] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [22008.622629] ? __next_timer_interrupt+0x160/0x160 [22008.628682] ptlrpcd+0xa82/0xe40 [ptlrpc] [22008.635401] ? do_wait_intr+0xf0/0xf0 [22008.642196] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [22008.656277] kthread+0x1d1/0x200 [22008.663031] ? set_kthread_struct+0x70/0x70 [22008.665068] ret_from_fork+0x1f/0x30 [22012.058740] watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ldlm_bl_02:16828] [22012.065537] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22012.106573] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22012.121398] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22012.131527] RIP: 0010:native_safe_halt+0x2e/0x50 [22012.136335] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [22012.151771] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [22012.158018] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [22012.162655] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [22012.168682] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [22012.174668] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [22012.181611] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [22012.187596] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [22012.194386] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22012.199282] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [22012.205495] Call Trace: [22012.206889] [22012.208586] ? show_regs.cold.9+0x22/0x2f [22012.212728] ? watchdog_timer_fn.cold.6+0x62/0x10a [22012.218154] ? __hrtimer_run_queues+0x1c2/0x4d0 [22012.220848] ? watchdog+0x40/0x40 [22012.228763] ? hrtimer_interrupt+0x128/0x2c0 [22012.230476] ? smp_apic_timer_interrupt+0xa6/0x270 [22012.236122] ? apic_timer_interrupt+0xf/0x20 [22012.238400] [22012.241979] ? native_safe_halt+0x2e/0x50 [22012.245444] kvm_wait+0xcd/0xe0 [22012.249956] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [22012.252704] do_raw_spin_lock+0x177/0x190 [22012.255159] _raw_spin_lock+0x21/0x30 [22012.259274] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [22012.263119] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [22012.267085] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [22012.272354] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [22012.275560] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [22012.278628] ? woken_wake_function+0x30/0x30 [22012.280744] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [22012.284022] kthread+0x1d1/0x200 [22012.285809] ? set_kthread_struct+0x70/0x70 [22012.288071] ret_from_fork+0x1f/0x30 [22028.054390] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [22028.070117] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22028.115055] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22028.122445] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22028.135274] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [22028.141944] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22028.147466] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [22028.158112] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [22028.162005] RAX: 0000000000007c0b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [22028.171929] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [22028.177086] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [22028.182853] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [22028.188763] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [22028.193329] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [22028.198876] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22028.205450] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [22028.207740] Call Trace: [22028.215086] [22028.217931] ? show_regs.cold.9+0x22/0x2f [22028.220915] ? watchdog_timer_fn.cold.6+0x62/0x10a [22028.226295] ? __hrtimer_run_queues+0x1c2/0x4d0 [22028.230987] ? watchdog+0x40/0x40 [22028.232355] ? hrtimer_interrupt+0x128/0x2c0 [22028.234612] ? smp_apic_timer_interrupt+0xa6/0x270 [22028.237302] ? apic_timer_interrupt+0xf/0x20 [22028.239923] [22028.241051] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22028.243935] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [22028.246482] ? newidle_balance.isra.93+0x207/0x580 [22028.248709] do_raw_spin_lock+0x177/0x190 [22028.250347] _raw_spin_lock+0x21/0x30 [22028.258369] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [22028.262183] process_one_work+0x2c8/0x700 [22028.267003] worker_thread+0x296/0x6e0 [22028.270380] ? rescuer_thread+0x570/0x570 [22028.271651] kthread+0x1d1/0x200 [22028.273917] ? set_kthread_struct+0x70/0x70 [22028.276858] ret_from_fork+0x1f/0x30 [22031.840037] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 19394s! [22031.843601] Showing busy workqueues and worker pools: [22031.847127] workqueue events: flags=0x0 [22031.851793] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [22031.855787] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [22031.859885] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [22031.862079] pending: cpu_latency_work [ptlrpc] [22031.870500] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [22031.875610] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [22031.887027] workqueue events_long: flags=0x0 [22031.888424] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [22031.898200] pending: lu_objects_flush [obdclass] [22031.903222] workqueue events_unbound: flags=0x2 [22031.904453] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [22031.908892] pending: flush_memcg_stats_dwork, idle_cull_fn [22031.915913] workqueue events_power_efficient: flags=0x80 [22031.918506] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22031.923437] pending: check_lifetime, do_cache_clean [22031.924982] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22031.928253] pending: neigh_periodic_work, neigh_periodic_work [22031.931233] workqueue mm_percpu_wq: flags=0x8 [22031.932857] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22031.938881] pending: vmstat_update, lru_add_drain_per_cpu [22031.941266] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22031.945842] pending: vmstat_update, lru_add_drain_per_cpu [22031.950207] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [22031.956296] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [22031.958614] workqueue xprtiod: flags=0xa [22031.959847] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [22031.963410] pending: xprt_autoclose, xs_error_handle [22031.967640] workqueue ipv6_addrconf: flags=0x40008 [22031.971863] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [22031.974704] pending: addrconf_verify_work [22031.976182] workqueue knbd0-recv: flags=0x1a [22031.979806] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [22031.983088] in-flight: 651:recv_work [22031.984939] workqueue ptlrpc_pinger: flags=0x2 [22031.987010] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [22031.991395] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [22031.993550] pool 8: cpus=0-3 flags=0x4 nice=0 hung=19394s workers=4 idle: 15979 31824 8 [22031.997177] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=22022s workers=2 idle: 654 [22036.055011] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [22036.058672] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22036.059103] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [22036.077048] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22036.077061] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22036.077063] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22036.077077] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [22036.077081] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [22036.077087] RAX: 0000000000002168 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [22036.079927] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22036.080000] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22036.080007] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22036.080010] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [22036.080023] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [22036.080027] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [22036.080031] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [22036.080033] RDX: 00000000000079aa RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [22036.080035] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [22036.080037] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [22036.080040] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [22036.080043] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [22036.080045] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22036.080047] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [22036.080056] Call Trace: [22036.080060] [22036.080066] ? show_regs.cold.9+0x22/0x2f [22036.080075] ? watchdog_timer_fn.cold.6+0x62/0x10a [22036.080081] ? __hrtimer_run_queues+0x1c2/0x4d0 [22036.080086] ? watchdog+0x40/0x40 [22036.080090] ? hrtimer_interrupt+0x128/0x2c0 [22036.080094] ? smp_apic_timer_interrupt+0xa6/0x270 [22036.080101] ? apic_timer_interrupt+0xf/0x20 [22036.080106] [22036.080108] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [22036.080113] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [22036.080117] do_raw_spin_lock+0x177/0x190 [22036.080121] _raw_spin_lock+0x21/0x30 [22036.080127] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [22036.080515] obd_mod_free+0x46/0x140 [ptlrpc] [22036.080779] mdc_commit_open+0x91/0xb0 [mdc] [22036.080817] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [22036.081073] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [22036.085251] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [22036.085259] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [22036.085261] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [22036.085264] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [22036.085268] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [22036.085271] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22036.085274] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [22036.085282] Call Trace: [22036.090854] after_reply+0xe2c/0x1780 [ptlrpc] [22036.091172] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [22036.091395] ? __next_timer_interrupt+0x160/0x160 [22036.091406] ptlrpcd+0xa82/0xe40 [ptlrpc] [22036.091747] ? do_wait_intr+0xf0/0xf0 [22036.091757] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [22036.091955] kthread+0x1d1/0x200 [22036.091961] ? set_kthread_struct+0x70/0x70 [22036.091964] ret_from_fork+0x1f/0x30 [22036.264483] [22036.265129] ? show_regs.cold.9+0x22/0x2f [22036.266483] ? watchdog_timer_fn.cold.6+0x62/0x10a [22036.268694] ? __hrtimer_run_queues+0x1c2/0x4d0 [22036.270503] ? watchdog+0x40/0x40 [22036.271781] ? hrtimer_interrupt+0x128/0x2c0 [22036.273512] ? smp_apic_timer_interrupt+0xa6/0x270 [22036.277363] ? apic_timer_interrupt+0xf/0x20 [22036.279126] [22036.279814] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22036.281786] do_raw_spin_lock+0x177/0x190 [22036.283390] _raw_spin_lock+0x21/0x30 [22036.285008] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [22036.287519] ? do_raw_spin_unlock+0x75/0x190 [22036.289361] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [22036.291684] ? do_wait_intr+0xf0/0xf0 [22036.293430] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [22036.295319] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [22036.297536] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [22036.300166] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22036.302106] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [22036.304235] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [22036.305968] mdc_intent_lock+0x353/0x7e0 [mdc] [22036.307537] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22036.308989] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [22036.311724] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [22036.314069] lmv_intent_open+0x338/0x1150 [lmv] [22036.315479] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22036.317727] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22036.319937] lmv_intent_lock+0x1dd/0x4c0 [lmv] [22036.321596] ll_intent_lock+0xc5/0xaa0 [lustre] [22036.323760] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22036.326207] ll_lookup_it+0xe0e/0x2270 [lustre] [22036.328736] ? do_raw_spin_unlock+0x75/0x190 [22036.330095] ? _raw_spin_unlock+0x12/0x30 [22036.331800] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [22036.333625] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [22036.335896] ll_atomic_open+0x36a/0xd70 [lustre] [22036.337902] lookup_open+0x356/0xb40 [22036.339586] do_last+0x313/0xfc0 [22036.341008] ? nd_jump_root+0xe5/0x160 [22036.342795] ? path_init+0x437/0x520 [22036.344154] path_openat+0xf7/0x500 [22036.345615] ? do_raw_spin_unlock+0x75/0x190 [22036.347521] do_filp_open+0x99/0x140 [22036.348939] ? getname_flags+0x6e/0x330 [22036.350597] ? __check_object_size+0xff/0x256 [22036.352415] ? do_raw_spin_unlock+0x75/0x190 [22036.353941] ? _raw_spin_unlock+0x12/0x30 [22036.355533] do_sys_openat2+0x2b4/0x410 [22036.357060] do_sys_open+0x73/0xa0 [22036.358368] __x64_sys_openat+0x24/0x30 [22036.360057] do_syscall_64+0xc1/0x440 [22036.361560] entry_SYSCALL_64_after_hwframe+0x49/0xae [22036.363426] RIP: 0033:0x7f53425a5332 [22036.364950] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [22036.372542] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [22036.375676] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [22036.378141] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [22036.380977] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [22036.384646] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [22036.387619] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [22040.058008] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [22040.061815] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22040.083344] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22040.088021] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22040.092399] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22040.094876] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [22040.102636] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [22040.105716] RAX: 00000000000067f2 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [22040.108921] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [22040.111686] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [22040.114693] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [22040.120906] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [22040.128067] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [22040.135775] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22040.139360] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [22040.143490] Call Trace: [22040.145202] [22040.146021] ? show_regs.cold.9+0x22/0x2f [22040.147740] ? watchdog_timer_fn.cold.6+0x62/0x10a [22040.153533] ? __hrtimer_run_queues+0x1c2/0x4d0 [22040.158061] ? watchdog+0x40/0x40 [22040.159702] ? hrtimer_interrupt+0x128/0x2c0 [22040.162076] ? smp_apic_timer_interrupt+0xa6/0x270 [22040.164071] ? apic_timer_interrupt+0xf/0x20 [22040.173643] [22040.174470] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22040.180287] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [22040.182307] do_raw_spin_lock+0x177/0x190 [22040.187252] _raw_spin_lock+0x21/0x30 [22040.189232] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [22040.195660] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [22040.200043] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [22040.202378] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [22040.207698] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [22040.211822] ? woken_wake_function+0x30/0x30 [22040.214260] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [22040.219387] kthread+0x1d1/0x200 [22040.225446] ? set_kthread_struct+0x70/0x70 [22040.228072] ret_from_fork+0x1f/0x30 [22056.052050] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [22056.057618] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22056.106329] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22056.115808] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22056.124739] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [22056.133454] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22056.139101] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [22056.156871] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [22056.164046] RAX: 0000000000006214 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [22056.170659] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [22056.177591] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [22056.183972] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [22056.194030] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [22056.202404] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [22056.205935] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22056.208630] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [22056.212715] Call Trace: [22056.213845] [22056.214766] ? show_regs.cold.9+0x22/0x2f [22056.216638] ? watchdog_timer_fn.cold.6+0x62/0x10a [22056.218793] ? __hrtimer_run_queues+0x1c2/0x4d0 [22056.220835] ? watchdog+0x40/0x40 [22056.222053] ? hrtimer_interrupt+0x128/0x2c0 [22056.223848] ? smp_apic_timer_interrupt+0xa6/0x270 [22056.225904] ? apic_timer_interrupt+0xf/0x20 [22056.227895] [22056.228879] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22056.231335] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [22056.233669] ? newidle_balance.isra.93+0x207/0x580 [22056.235706] do_raw_spin_lock+0x177/0x190 [22056.236844] _raw_spin_lock+0x21/0x30 [22056.237910] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [22056.240656] process_one_work+0x2c8/0x700 [22056.242415] worker_thread+0x296/0x6e0 [22056.244181] ? rescuer_thread+0x570/0x570 [22056.245990] kthread+0x1d1/0x200 [22056.247762] ? set_kthread_struct+0x70/0x70 [22056.250021] ret_from_fork+0x1f/0x30 [22062.048035] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 19424s! [22062.050090] Showing busy workqueues and worker pools: [22062.051627] workqueue events: flags=0x0 [22062.052852] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [22062.054820] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [22062.058294] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [22062.060412] pending: cpu_latency_work [ptlrpc] [22062.061834] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [22062.063546] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [22062.066318] workqueue events_long: flags=0x0 [22062.067422] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [22062.069115] pending: lu_objects_flush [obdclass] [22062.070466] workqueue events_unbound: flags=0x2 [22062.071827] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [22062.074244] pending: flush_memcg_stats_dwork, idle_cull_fn [22062.076384] workqueue events_power_efficient: flags=0x80 [22062.078142] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22062.080814] pending: check_lifetime, do_cache_clean [22062.082464] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22062.084746] pending: neigh_periodic_work, neigh_periodic_work [22062.086759] workqueue mm_percpu_wq: flags=0x8 [22062.088087] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22062.090140] pending: vmstat_update, lru_add_drain_per_cpu [22062.092017] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22062.094095] pending: vmstat_update, lru_add_drain_per_cpu [22062.095850] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [22062.097795] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [22062.099820] workqueue xprtiod: flags=0xa [22062.101321] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [22062.103324] pending: xprt_autoclose, xs_error_handle [22062.105051] workqueue ipv6_addrconf: flags=0x40008 [22062.106703] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [22062.108929] pending: addrconf_verify_work [22062.110353] workqueue knbd0-recv: flags=0x1a [22062.114142] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [22062.117125] in-flight: 651:recv_work [22062.118349] workqueue ptlrpc_pinger: flags=0x2 [22062.119791] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [22062.121702] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [22062.123964] pool 8: cpus=0-3 flags=0x4 nice=0 hung=19425s workers=4 idle: 15979 31824 8 [22062.126578] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=22052s workers=2 idle: 654 [22064.057558] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [22064.057964] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [22064.069309] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22064.069512] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22064.069519] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22064.069523] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22064.069537] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [22064.069541] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [22064.069546] RAX: 0000000000007442 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [22064.069549] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [22064.069552] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [22064.069554] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [22064.069556] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [22064.069559] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [22064.069562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22064.069565] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [22064.069573] Call Trace: [22064.069578] [22064.069583] ? show_regs.cold.9+0x22/0x2f [22064.069593] ? watchdog_timer_fn.cold.6+0x62/0x10a [22064.069599] ? __hrtimer_run_queues+0x1c2/0x4d0 [22064.069603] ? watchdog+0x40/0x40 [22064.069607] ? hrtimer_interrupt+0x128/0x2c0 [22064.069610] ? smp_apic_timer_interrupt+0xa6/0x270 [22064.069619] ? apic_timer_interrupt+0xf/0x20 [22064.069624] [22064.069626] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22064.069633] do_raw_spin_lock+0x177/0x190 [22064.069637] _raw_spin_lock+0x21/0x30 [22064.069641] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [22064.070582] ? do_raw_spin_unlock+0x75/0x190 [22064.070589] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [22064.071531] ? do_wait_intr+0xf0/0xf0 [22064.071544] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [22064.071760] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [22064.082986] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22064.085181] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22064.085189] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22064.085193] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [22064.085206] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [22064.085211] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [22064.085216] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [22064.085218] RDX: 0000000000007f80 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [22064.085221] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [22064.085223] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [22064.085226] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [22064.085229] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [22064.085232] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22064.085234] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [22064.085243] Call Trace: [22064.085249] [22064.085255] ? show_regs.cold.9+0x22/0x2f [22064.085264] ? watchdog_timer_fn.cold.6+0x62/0x10a [22064.085271] ? __hrtimer_run_queues+0x1c2/0x4d0 [22064.085276] ? watchdog+0x40/0x40 [22064.085281] ? hrtimer_interrupt+0x128/0x2c0 [22064.085285] ? smp_apic_timer_interrupt+0xa6/0x270 [22064.085292] ? apic_timer_interrupt+0xf/0x20 [22064.085296] [22064.085299] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [22064.085303] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [22064.085307] do_raw_spin_lock+0x177/0x190 [22064.085311] _raw_spin_lock+0x21/0x30 [22064.085316] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [22064.085601] obd_mod_free+0x46/0x140 [ptlrpc] [22064.085811] mdc_commit_open+0x91/0xb0 [mdc] [22064.085852] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [22064.088528] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [22064.088759] after_reply+0xe2c/0x1780 [ptlrpc] [22064.088959] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [22064.089175] ? __next_timer_interrupt+0x160/0x160 [22064.089187] ptlrpcd+0xa82/0xe40 [ptlrpc] [22064.089409] ? do_wait_intr+0xf0/0xf0 [22064.089419] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [22064.089639] kthread+0x1d1/0x200 [22064.089645] ? set_kthread_struct+0x70/0x70 [22064.089649] ret_from_fork+0x1f/0x30 [22064.583755] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [22064.587748] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22064.598525] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [22064.602181] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [22064.605522] mdc_intent_lock+0x353/0x7e0 [mdc] [22064.608361] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22064.611728] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [22064.619588] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [22064.627511] lmv_intent_open+0x338/0x1150 [lmv] [22064.631694] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22064.636356] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22064.639912] lmv_intent_lock+0x1dd/0x4c0 [lmv] [22064.644312] ll_intent_lock+0xc5/0xaa0 [lustre] [22064.647343] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22064.654977] ll_lookup_it+0xe0e/0x2270 [lustre] [22064.660777] ? do_raw_spin_unlock+0x75/0x190 [22064.663784] ? _raw_spin_unlock+0x12/0x30 [22064.668137] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [22064.669718] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [22064.675182] ll_atomic_open+0x36a/0xd70 [lustre] [22064.678746] lookup_open+0x356/0xb40 [22064.683013] do_last+0x313/0xfc0 [22064.684643] ? nd_jump_root+0xe5/0x160 [22064.686953] ? path_init+0x437/0x520 [22064.690386] path_openat+0xf7/0x500 [22064.693978] ? do_raw_spin_unlock+0x75/0x190 [22064.695445] do_filp_open+0x99/0x140 [22064.696604] ? getname_flags+0x6e/0x330 [22064.702340] ? __check_object_size+0xff/0x256 [22064.705381] ? do_raw_spin_unlock+0x75/0x190 [22064.707788] ? _raw_spin_unlock+0x12/0x30 [22064.709631] do_sys_openat2+0x2b4/0x410 [22064.711732] do_sys_open+0x73/0xa0 [22064.717549] __x64_sys_openat+0x24/0x30 [22064.721709] do_syscall_64+0xc1/0x440 [22064.724402] entry_SYSCALL_64_after_hwframe+0x49/0xae [22064.730392] RIP: 0033:0x7f53425a5332 [22064.734799] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [22064.748176] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [22064.752988] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [22064.756796] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [22064.762612] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [22064.769316] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [22064.781039] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [22068.058011] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [22068.061719] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22068.081998] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22068.087294] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22068.090446] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22068.092250] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [22068.100474] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [22068.102867] RAX: 00000000000049fe RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [22068.105224] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [22068.107905] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [22068.111704] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [22068.115459] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [22068.122483] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [22068.131789] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22068.136635] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [22068.143966] Call Trace: [22068.145543] [22068.147451] ? show_regs.cold.9+0x22/0x2f [22068.150869] ? watchdog_timer_fn.cold.6+0x62/0x10a [22068.153862] ? __hrtimer_run_queues+0x1c2/0x4d0 [22068.156687] ? watchdog+0x40/0x40 [22068.158342] ? hrtimer_interrupt+0x128/0x2c0 [22068.159885] ? smp_apic_timer_interrupt+0xa6/0x270 [22068.161832] ? apic_timer_interrupt+0xf/0x20 [22068.163522] [22068.164391] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22068.166440] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [22068.168376] do_raw_spin_lock+0x177/0x190 [22068.169932] _raw_spin_lock+0x21/0x30 [22068.171328] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [22068.173718] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [22068.175729] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [22068.177852] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [22068.180041] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [22068.182362] ? woken_wake_function+0x30/0x30 [22068.184015] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [22068.186511] kthread+0x1d1/0x200 [22068.187693] ? set_kthread_struct+0x70/0x70 [22068.189333] ret_from_fork+0x1f/0x30 [22088.052031] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:4:36042] [22088.055289] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22088.089908] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22088.096547] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22088.102671] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [22088.107670] RIP: 0010:native_safe_halt+0x2e/0x50 [22088.109671] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [22088.123565] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [22088.129207] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [22088.132424] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [22088.136506] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [22088.139394] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [22088.143051] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [22088.146105] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [22088.149821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22088.152894] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [22088.156302] Call Trace: [22088.159296] [22088.160340] ? show_regs.cold.9+0x22/0x2f [22088.161597] ? watchdog_timer_fn.cold.6+0x62/0x10a [22088.162843] ? __hrtimer_run_queues+0x1c2/0x4d0 [22088.164991] ? watchdog+0x40/0x40 [22088.166380] ? hrtimer_interrupt+0x128/0x2c0 [22088.168138] ? smp_apic_timer_interrupt+0xa6/0x270 [22088.169909] ? apic_timer_interrupt+0xf/0x20 [22088.171536] [22088.174450] ? native_safe_halt+0x2e/0x50 [22088.176225] kvm_wait+0xcd/0xe0 [22088.177545] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [22088.180177] ? newidle_balance.isra.93+0x207/0x580 [22088.182228] do_raw_spin_lock+0x177/0x190 [22088.183982] _raw_spin_lock+0x21/0x30 [22088.185871] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [22088.188855] process_one_work+0x2c8/0x700 [22088.190558] worker_thread+0x296/0x6e0 [22088.193178] ? rescuer_thread+0x570/0x570 [22088.195240] kthread+0x1d1/0x200 [22088.197539] ? set_kthread_struct+0x70/0x70 [22088.200431] ret_from_fork+0x1f/0x30 [22092.055326] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [22092.057049] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [22092.060320] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22092.060592] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22092.060600] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22092.060604] RIP: 0010:native_safe_halt+0x2e/0x50 [22092.060616] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [22092.060620] RSP: 0018:ffffbb53815034f0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [22092.060625] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [22092.060627] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7020b4154 [22092.060638] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [22092.060641] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [22092.060643] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [22092.060646] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [22092.060649] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22092.060652] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [22092.060659] Call Trace: [22092.060665] [22092.060672] ? show_regs.cold.9+0x22/0x2f [22092.060681] ? watchdog_timer_fn.cold.6+0x62/0x10a [22092.060688] ? __hrtimer_run_queues+0x1c2/0x4d0 [22092.060693] ? watchdog+0x40/0x40 [22092.060697] ? hrtimer_interrupt+0x128/0x2c0 [22092.060702] ? smp_apic_timer_interrupt+0xa6/0x270 [22092.060709] ? apic_timer_interrupt+0xf/0x20 [22092.060714] [22092.060717] ? native_safe_halt+0x2e/0x50 [22092.060731] kvm_wait+0xcd/0xe0 [22092.060738] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [22092.060747] do_raw_spin_lock+0x177/0x190 [22092.060752] _raw_spin_lock+0x21/0x30 [22092.060756] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [22092.061022] ? do_raw_spin_unlock+0x75/0x190 [22092.061028] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [22092.061214] ? do_wait_intr+0xf0/0xf0 [22092.061223] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [22092.065391] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) [22092.090787] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [22092.099382] ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 [22092.103197] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [22092.106136] dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22092.106175] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22092.106183] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22092.106186] RIP: 0010:native_safe_halt+0x2e/0x50 [22092.106200] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [22092.106205] RSP: 0018:ffffbb5381ff7bb8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [22092.106210] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [22092.106213] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [22092.106215] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [22092.106217] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [22092.106219] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [22092.106222] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [22092.106226] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22092.106229] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [22092.106238] Call Trace: [22092.106243] [22092.106249] ? show_regs.cold.9+0x22/0x2f [22092.106259] ? watchdog_timer_fn.cold.6+0x62/0x10a [22092.115254] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22092.118313] ? __hrtimer_run_queues+0x1c2/0x4d0 [22092.120727] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [22092.120774] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [22092.120805] mdc_intent_lock+0x353/0x7e0 [mdc] [22092.120846] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22092.120960] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [22092.121230] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [22092.121257] lmv_intent_open+0x338/0x1150 [lmv] [22092.121310] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22092.121394] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22092.124281] ? watchdog+0x40/0x40 [22092.124300] ? hrtimer_interrupt+0x128/0x2c0 [22092.124307] ? smp_apic_timer_interrupt+0xa6/0x270 [22092.124315] ? apic_timer_interrupt+0xf/0x20 [22092.124320] [22092.124325] ? native_safe_halt+0x2e/0x50 [22092.124329] kvm_wait+0xcd/0xe0 [22092.127247] lmv_intent_lock+0x1dd/0x4c0 [lmv] [22092.129421] __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [22092.132113] ll_intent_lock+0xc5/0xaa0 [lustre] [22092.139244] do_raw_spin_lock+0x177/0x190 [22092.142902] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22092.143023] ll_lookup_it+0xe0e/0x2270 [lustre] [22092.143107] ? do_raw_spin_unlock+0x75/0x190 [22092.143115] ? _raw_spin_unlock+0x12/0x30 [22092.143122] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [22092.143146] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [22092.143165] ll_atomic_open+0x36a/0xd70 [lustre] [22092.143245] lookup_open+0x356/0xb40 [22092.143255] do_last+0x313/0xfc0 [22092.143259] ? nd_jump_root+0xe5/0x160 [22092.143264] ? path_init+0x437/0x520 [22092.143268] path_openat+0xf7/0x500 [22092.143273] ? do_raw_spin_unlock+0x75/0x190 [22092.143277] do_filp_open+0x99/0x140 [22092.143283] ? getname_flags+0x6e/0x330 [22092.143287] ? __check_object_size+0xff/0x256 [22092.143293] ? do_raw_spin_unlock+0x75/0x190 [22092.143297] ? _raw_spin_unlock+0x12/0x30 [22092.143300] do_sys_openat2+0x2b4/0x410 [22092.143305] do_sys_open+0x73/0xa0 [22092.143309] __x64_sys_openat+0x24/0x30 [22092.143312] do_syscall_64+0xc1/0x440 [22092.147244] _raw_spin_lock+0x21/0x30 [22092.149280] entry_SYSCALL_64_after_hwframe+0x49/0xae [22092.149294] RIP: 0033:0x7f53425a5332 [22092.149339] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [22092.150111] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [22092.152328] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 [22092.154106] obd_mod_free+0x46/0x140 [ptlrpc] [22092.156275] ORIG_RAX: 0000000000000101 [22092.156281] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [22092.156283] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [22092.156285] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [22092.156288] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [22092.156290] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [22092.306206] mdc_commit_open+0x91/0xb0 [mdc] [22092.307541] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [22092.309553] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [22092.311218] after_reply+0xe2c/0x1780 [ptlrpc] [22092.312835] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [22092.315134] ? __next_timer_interrupt+0x160/0x160 [22092.316427] ptlrpcd+0xa82/0xe40 [ptlrpc] [22092.317890] ? do_wait_intr+0xf0/0xf0 [22092.318980] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [22092.320829] kthread+0x1d1/0x200 [22092.321778] ? set_kthread_struct+0x70/0x70 [22092.322952] ret_from_fork+0x1f/0x30 [22092.324124] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 19455s! [22092.326333] Showing busy workqueues and worker pools: [22092.328018] workqueue events: flags=0x0 [22092.329181] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [22092.331166] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [22092.335216] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [22092.337267] pending: cpu_latency_work [ptlrpc] [22092.339175] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [22092.341293] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [22092.351326] workqueue events_long: flags=0x0 [22092.352679] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [22092.355459] pending: lu_objects_flush [obdclass] [22092.357266] workqueue events_unbound: flags=0x2 [22092.358537] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [22092.360791] pending: flush_memcg_stats_dwork, idle_cull_fn [22092.362483] workqueue events_power_efficient: flags=0x80 [22092.364105] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22092.366177] pending: check_lifetime, do_cache_clean [22092.367988] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22092.370108] pending: neigh_periodic_work, neigh_periodic_work [22092.372391] workqueue mm_percpu_wq: flags=0x8 [22092.373748] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22092.376161] pending: vmstat_update, lru_add_drain_per_cpu [22092.377800] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22092.379712] pending: vmstat_update, lru_add_drain_per_cpu [22092.381308] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [22092.383693] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [22092.385481] workqueue xprtiod: flags=0xa [22092.386769] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [22092.388748] pending: xprt_autoclose, xs_error_handle [22092.390606] workqueue ipv6_addrconf: flags=0x40008 [22092.392398] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [22092.394597] pending: addrconf_verify_work [22092.396174] workqueue knbd0-recv: flags=0x1a [22092.397522] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [22092.399538] in-flight: 651:recv_work [22092.400720] workqueue ptlrpc_pinger: flags=0x2 [22092.402026] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [22092.403799] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [22092.405967] pool 8: cpus=0-3 flags=0x4 nice=0 hung=19455s workers=4 idle: 15979 31824 8 [22092.408796] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=22082s workers=2 idle: 654 [22096.058242] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [22096.061127] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22096.082279] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22096.087833] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22096.091183] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22096.094076] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [22096.102113] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [22096.105730] RAX: 000000000000010f RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [22096.109618] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [22096.113106] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [22096.116394] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [22096.120267] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [22096.123304] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [22096.126108] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22096.128034] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [22096.130490] Call Trace: [22096.131318] [22096.132042] ? show_regs.cold.9+0x22/0x2f [22096.133554] ? watchdog_timer_fn.cold.6+0x62/0x10a [22096.135440] ? __hrtimer_run_queues+0x1c2/0x4d0 [22096.137083] ? watchdog+0x40/0x40 [22096.138247] ? hrtimer_interrupt+0x128/0x2c0 [22096.139698] ? smp_apic_timer_interrupt+0xa6/0x270 [22096.141159] ? apic_timer_interrupt+0xf/0x20 [22096.142670] [22096.143277] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22096.145176] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [22096.146984] do_raw_spin_lock+0x177/0x190 [22096.148410] _raw_spin_lock+0x21/0x30 [22096.149626] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [22096.151974] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [22096.153983] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [22096.156063] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [22096.158121] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [22096.160443] ? woken_wake_function+0x30/0x30 [22096.161907] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [22096.164184] kthread+0x1d1/0x200 [22096.165458] ? set_kthread_struct+0x70/0x70 [22096.166815] ret_from_fork+0x1f/0x30 [22120.057265] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ptlrpcd_rcv:2391] [22120.057608] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [22120.063885] Modules linked in: lustre(O) [22120.064377] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [writemany:37227] [22120.064383] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22120.064530] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22120.064535] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22120.064538] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22120.064559] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [22120.064563] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [22120.064568] RAX: 0000000000007e85 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [22120.064570] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [22120.064573] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [22120.064575] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [22120.064578] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [22120.064581] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [22120.064584] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22120.064587] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [22120.064595] Call Trace: [22120.064606] [22120.064611] ? show_regs.cold.9+0x22/0x2f [22120.064620] ? watchdog_timer_fn.cold.6+0x62/0x10a [22120.064626] ? __hrtimer_run_queues+0x1c2/0x4d0 [22120.064631] ? watchdog+0x40/0x40 [22120.064635] ? hrtimer_interrupt+0x128/0x2c0 [22120.064639] ? smp_apic_timer_interrupt+0xa6/0x270 [22120.064646] ? apic_timer_interrupt+0xf/0x20 [22120.064650] [22120.064654] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22120.064660] do_raw_spin_lock+0x177/0x190 [22120.064664] _raw_spin_lock+0x21/0x30 [22120.064669] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [22120.065198] ? do_raw_spin_unlock+0x75/0x190 [22120.065205] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [22120.065405] ? do_wait_intr+0xf0/0xf0 [22120.065412] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [22120.065954] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [22120.066661] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [22120.066941] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22120.067332] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [22120.067371] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [22120.067400] mdc_intent_lock+0x353/0x7e0 [mdc] [22120.067613] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22120.067689] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22120.067695] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22120.067702] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [22120.067955] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22120.067964] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [22120.067969] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [22120.067973] RAX: 0000000000007f08 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [22120.067975] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [22120.067978] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [22120.067980] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [22120.067982] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [22120.067985] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [22120.067988] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22120.067991] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [22120.068000] Call Trace: [22120.068004] [22120.068007] ? show_regs.cold.9+0x22/0x2f [22120.068015] ? watchdog_timer_fn.cold.6+0x62/0x10a [22120.068127] ? __hrtimer_run_queues+0x1c2/0x4d0 [22120.068134] ? watchdog+0x40/0x40 [22120.068205] ? hrtimer_interrupt+0x128/0x2c0 [22120.068210] ? smp_apic_timer_interrupt+0xa6/0x270 [22120.068217] ? apic_timer_interrupt+0xf/0x20 [22120.068222] [22120.068225] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22120.068230] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [22120.068233] ? newidle_balance.isra.93+0x207/0x580 [22120.068239] do_raw_spin_lock+0x177/0x190 [22120.068246] _raw_spin_lock+0x21/0x30 [22120.068250] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [22120.068412] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22120.068824] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [22120.069830] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [22120.069869] lmv_intent_open+0x338/0x1150 [lmv] [22120.069900] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22120.070167] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22120.070251] lmv_intent_lock+0x1dd/0x4c0 [lmv] [22120.070275] ll_intent_lock+0xc5/0xaa0 [lustre] [22120.070359] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22120.070800] ll_lookup_it+0xe0e/0x2270 [lustre] [22120.070887] ? do_raw_spin_unlock+0x75/0x190 [22120.070894] ? _raw_spin_unlock+0x12/0x30 [22120.070901] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [22120.070930] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [22120.070953] ll_atomic_open+0x36a/0xd70 [lustre] [22120.071516] lookup_open+0x356/0xb40 [22120.071525] do_last+0x313/0xfc0 [22120.071530] ? nd_jump_root+0xe5/0x160 [22120.071535] ? path_init+0x437/0x520 [22120.071539] path_openat+0xf7/0x500 [22120.071544] ? do_raw_spin_unlock+0x75/0x190 [22120.071549] do_filp_open+0x99/0x140 [22120.071556] ? getname_flags+0x6e/0x330 [22120.071561] ? __check_object_size+0xff/0x256 [22120.071568] ? do_raw_spin_unlock+0x75/0x190 [22120.071572] ? _raw_spin_unlock+0x12/0x30 [22120.071576] do_sys_openat2+0x2b4/0x410 [22120.071580] do_sys_open+0x73/0xa0 [22120.071585] __x64_sys_openat+0x24/0x30 [22120.071589] do_syscall_64+0xc1/0x440 [22120.071596] entry_SYSCALL_64_after_hwframe+0x49/0xae [22120.071602] RIP: 0033:0x7f53425a5332 [22120.071650] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [22120.071656] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [22120.071661] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [22120.071663] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [22120.071666] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [22120.071668] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [22120.071670] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [22120.072739] osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22120.072811] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22120.072819] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22120.072821] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [22120.072832] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [22120.072836] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [22120.072841] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [22120.072844] RDX: 0000000000000a5e RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [22120.072846] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [22120.072848] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [22120.072850] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [22120.072853] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [22120.072856] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22120.072859] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [22120.072867] Call Trace: [22120.072871] [22120.072875] ? show_regs.cold.9+0x22/0x2f [22120.072883] ? watchdog_timer_fn.cold.6+0x62/0x10a [22120.072891] ? __hrtimer_run_queues+0x1c2/0x4d0 [22120.072896] ? watchdog+0x40/0x40 [22120.072900] ? hrtimer_interrupt+0x128/0x2c0 [22120.072904] ? smp_apic_timer_interrupt+0xa6/0x270 [22120.072910] ? apic_timer_interrupt+0xf/0x20 [22120.072915] [22120.072918] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [22120.072923] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [22120.072928] do_raw_spin_lock+0x177/0x190 [22120.072933] _raw_spin_lock+0x21/0x30 [22120.072939] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [22120.073483] obd_mod_free+0x46/0x140 [ptlrpc] [22120.073787] mdc_commit_open+0x91/0xb0 [mdc] [22120.073822] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [22120.074018] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [22120.074476] after_reply+0xe2c/0x1780 [ptlrpc] [22120.074687] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [22120.074884] ? __next_timer_interrupt+0x160/0x160 [22120.074894] ptlrpcd+0xa82/0xe40 [ptlrpc] [22120.075479] ? do_wait_intr+0xf0/0xf0 [22120.075488] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [22120.075747] kthread+0x1d1/0x200 [22120.075754] ? set_kthread_struct+0x70/0x70 [22120.075758] ret_from_fork+0x1f/0x30 [22120.731122] process_one_work+0x2c8/0x700 [22120.733334] worker_thread+0x296/0x6e0 [22120.735572] ? rescuer_thread+0x570/0x570 [22120.736994] kthread+0x1d1/0x200 [22120.738388] ? set_kthread_struct+0x70/0x70 [22120.740015] ret_from_fork+0x1f/0x30 [22122.466567] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 19485s! [22122.473831] Showing busy workqueues and worker pools: [22122.476788] workqueue events: flags=0x0 [22122.479630] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [22122.482846] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [22122.493702] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [22122.495906] pending: cpu_latency_work [ptlrpc] [22122.499368] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [22122.504989] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [22122.509578] workqueue events_long: flags=0x0 [22122.512725] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [22122.516005] pending: lu_objects_flush [obdclass] [22122.518765] workqueue events_unbound: flags=0x2 [22122.520000] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [22122.529718] pending: flush_memcg_stats_dwork, idle_cull_fn [22122.533233] workqueue events_power_efficient: flags=0x80 [22122.540052] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22122.542881] pending: check_lifetime, do_cache_clean [22122.545037] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22122.551896] pending: neigh_periodic_work, neigh_periodic_work [22122.556061] workqueue mm_percpu_wq: flags=0x8 [22122.558283] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22122.562692] pending: vmstat_update, lru_add_drain_per_cpu [22122.564732] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22122.570408] pending: vmstat_update, lru_add_drain_per_cpu [22122.573818] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [22122.577877] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [22122.585213] workqueue xprtiod: flags=0xa [22122.586443] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [22122.605567] pending: xprt_autoclose, xs_error_handle [22122.607407] workqueue ipv6_addrconf: flags=0x40008 [22122.610092] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [22122.617005] pending: addrconf_verify_work [22122.619310] workqueue knbd0-recv: flags=0x1a [22122.621351] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [22122.624205] in-flight: 651:recv_work [22122.626918] workqueue ptlrpc_pinger: flags=0x2 [22122.634263] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [22122.639036] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [22122.641955] pool 8: cpus=0-3 flags=0x4 nice=0 hung=19485s workers=4 idle: 15979 31824 8 [22122.646068] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=22113s workers=2 idle: 654 [22124.058106] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [22124.061276] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22124.132327] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22124.138944] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22124.144995] RIP: 0010:native_safe_halt+0x2e/0x50 [22124.147454] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [22124.155837] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 [22124.159397] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [22124.162191] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [22124.165149] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [22124.167543] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [22124.170124] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [22124.172852] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [22124.176040] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22124.178222] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [22124.180168] Call Trace: [22124.180790] [22124.181314] ? show_regs.cold.9+0x22/0x2f [22124.182433] ? watchdog_timer_fn.cold.6+0x62/0x10a [22124.184366] ? __hrtimer_run_queues+0x1c2/0x4d0 [22124.185907] ? watchdog+0x40/0x40 [22124.187036] ? hrtimer_interrupt+0x128/0x2c0 [22124.188446] ? smp_apic_timer_interrupt+0xa6/0x270 [22124.189901] ? apic_timer_interrupt+0xf/0x20 [22124.191368] [22124.192048] ? native_safe_halt+0x2e/0x50 [22124.194235] kvm_wait+0xcd/0xe0 [22124.195286] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [22124.197528] do_raw_spin_lock+0x177/0x190 [22124.199051] _raw_spin_lock+0x21/0x30 [22124.200410] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [22124.202975] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [22124.204899] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [22124.206964] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [22124.209273] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [22124.211451] ? woken_wake_function+0x30/0x30 [22124.213351] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [22124.215725] kthread+0x1d1/0x200 [22124.216940] ? set_kthread_struct+0x70/0x70 [22124.218491] ret_from_fork+0x1f/0x30 [22134.926688] rcu: INFO: rcu_sched self-detected stall on CPU [22134.944868] rcu: 2-....: (11585131 ticks this GP) idle=b22/1/0x4000000000000002 softirq=137717/137717 fqs=96983 [22134.970590] (t=20040500 jiffies g=274185 q=69033) [22134.981081] rcu: rcu_sched kthread starved for 19498866 jiffies! g274185 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 [22134.998808] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [22135.011583] rcu: RCU grace-period kthread stack dump: [22135.015527] task:rcu_sched state:R running task stack:0 pid:13 ppid:2 flags:0x80004000 [22135.026761] Call Trace: [22135.034120] __schedule+0x351/0xcb0 [22135.035448] schedule+0xc0/0x180 [22135.036470] schedule_timeout+0xb4/0x190 [22135.044840] ? do_raw_spin_unlock+0x75/0x190 [22135.050459] ? __next_timer_interrupt+0x160/0x160 [22135.058151] ? prepare_to_swait_event+0xf3/0x140 [22135.065265] rcu_gp_kthread+0x8dd/0x15b0 [22135.068586] ? force_qs_rnp+0x2c0/0x2c0 [22135.071399] kthread+0x1d1/0x200 [22135.072502] ? set_kthread_struct+0x70/0x70 [22135.073874] ret_from_fork+0x1f/0x30 [22135.086353] rcu: Stack dump where RCU GP kthread last ran: [22135.095633] Sending NMI from CPU 2 to CPUs 0: [22135.116194] NMI backtrace for cpu 0 [22135.116202] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22135.116203] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22135.116205] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [22135.116208] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22135.116209] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [22135.116211] RSP: 0018:ffffbb5380d7fdd8 EFLAGS: 00000216 [22135.116213] RAX: 0000000000004a9e RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [22135.116214] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a702034154 [22135.116215] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [22135.116217] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [22135.116218] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [22135.116220] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [22135.116221] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22135.116222] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [22135.116223] Call Trace: [22135.116224] [22135.116225] ? show_regs.cold.9+0x22/0x2f [22135.116226] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [22135.116227] ? nmi_cpu_backtrace_handler+0x15/0x40 [22135.116228] ? nmi_handle+0xbd/0x210 [22135.116229] ? default_do_nmi+0x4b/0x1b0 [22135.116230] ? do_nmi+0x280/0x380 [22135.116231] ? end_repeat_nmi+0x16/0x1f [22135.116232] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22135.116233] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22135.116235] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22135.116236] [22135.116237] ? newidle_balance.isra.93+0x207/0x580 [22135.116238] do_raw_spin_lock+0x177/0x190 [22135.116239] _raw_spin_lock+0x21/0x30 [22135.116240] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [22135.116241] process_one_work+0x2c8/0x700 [22135.116242] worker_thread+0x296/0x6e0 [22135.116243] ? rescuer_thread+0x570/0x570 [22135.116244] kthread+0x1d1/0x200 [22135.116245] ? set_kthread_struct+0x70/0x70 [22135.116246] ret_from_fork+0x1f/0x30 [22135.122947] Sending NMI from CPU 2 to CPUs 1: [22135.416945] NMI backtrace for cpu 1 [22135.416953] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22135.416956] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22135.416958] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22135.416960] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [22135.416962] RSP: 0018:ffffbb5381503500 EFLAGS: 00000202 [22135.416965] RAX: 0000000000006415 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [22135.416967] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [22135.416969] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [22135.416970] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [22135.416972] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [22135.416973] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [22135.416975] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22135.416976] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [22135.416978] Call Trace: [22135.416979] [22135.416979] ? show_regs.cold.9+0x22/0x2f [22135.416981] ? nmi_cpu_backtrace.cold.8+0x98/0xa2 [22135.416982] ? nmi_cpu_backtrace_handler+0x15/0x40 [22135.416983] ? nmi_handle+0xbd/0x210 [22135.416984] ? default_do_nmi+0x4b/0x1b0 [22135.416985] ? do_nmi+0x280/0x380 [22135.416986] ? end_repeat_nmi+0x16/0x1f [22135.416987] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22135.416988] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22135.416989] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22135.416990] [22135.416991] do_raw_spin_lock+0x177/0x190 [22135.416991] _raw_spin_lock+0x21/0x30 [22135.416992] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [22135.416993] ? do_raw_spin_unlock+0x75/0x190 [22135.416994] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [22135.416995] ? do_wait_intr+0xf0/0xf0 [22135.416996] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [22135.416997] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [22135.416999] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [22135.417000] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22135.417001] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [22135.417002] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [22135.417003] mdc_intent_lock+0x353/0x7e0 [mdc] [22135.417004] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22135.417005] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [22135.417006] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [22135.417007] lmv_intent_open+0x338/0x1150 [lmv] [22135.417008] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22135.417009] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22135.417011] lmv_intent_lock+0x1dd/0x4c0 [lmv] [22135.417012] ll_intent_lock+0xc5/0xaa0 [lustre] [22135.417013] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22135.417015] ll_lookup_it+0xe0e/0x2270 [lustre] [22135.417016] ? do_raw_spin_unlock+0x75/0x190 [22135.417017] ? _raw_spin_unlock+0x12/0x30 [22135.417018] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [22135.417019] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [22135.417020] ll_atomic_open+0x36a/0xd70 [lustre] [22135.417021] lookup_open+0x356/0xb40 [22135.417022] do_last+0x313/0xfc0 [22135.417023] ? nd_jump_root+0xe5/0x160 [22135.417024] ? path_init+0x437/0x520 [22135.417025] path_openat+0xf7/0x500 [22135.417026] ? do_raw_spin_unlock+0x75/0x190 [22135.417027] do_filp_open+0x99/0x140 [22135.417028] ? getname_flags+0x6e/0x330 [22135.417029] ? __check_object_size+0xff/0x256 [22135.417031] ? do_raw_spin_unlock+0x75/0x190 [22135.417032] ? _raw_spin_unlock+0x12/0x30 [22135.417033] do_sys_openat2+0x2b4/0x410 [22135.417034] do_sys_open+0x73/0xa0 [22135.417035] __x64_sys_openat+0x24/0x30 [22135.417036] do_syscall_64+0xc1/0x440 [22135.417037] entry_SYSCALL_64_after_hwframe+0x49/0xae [22135.417038] RIP: 0033:0x7f53425a5332 [22135.417040] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [22135.417042] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [22135.417045] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [22135.417046] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [22135.417048] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [22135.417049] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [22135.417050] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [22135.422565] NMI backtrace for cpu 2 [22136.047444] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22136.073927] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22136.087583] Call Trace: [22136.092566] [22136.096027] ? dump_stack+0xbb/0x10e [22136.097276] ? lapic_can_unplug_cpu.cold.28+0x16/0x55 [22136.112556] ? nmi_cpu_backtrace.cold.8+0x24/0xa2 [22136.126198] ? lapic_can_unplug_cpu.cold.28+0x55/0x55 [22136.132224] ? nmi_trigger_cpumask_backtrace+0x1d7/0x1f1 [22136.147343] ? arch_trigger_cpumask_backtrace+0x1d/0x30 [22136.148503] ? rcu_dump_cpu_stacks+0x11b/0x171 [22136.158459] ? rcu_sched_clock_irq.cold.51+0xdb/0x307 [22136.161300] ? update_process_times+0x6d/0xc0 [22136.169125] ? tick_sched_handle.isra.18+0x27/0x90 [22136.176539] ? tick_sched_timer+0x53/0xb0 [22136.180493] ? __hrtimer_run_queues+0x1c2/0x4d0 [22136.184076] ? tick_sched_handle.isra.18+0x90/0x90 [22136.188217] ? hrtimer_interrupt+0x128/0x2c0 [22136.195255] ? smp_apic_timer_interrupt+0xa6/0x270 [22136.196853] ? apic_timer_interrupt+0xf/0x20 [22136.199443] [22136.200055] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [22136.203268] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [22136.209842] ? do_raw_spin_lock+0x177/0x190 [22136.213333] ? _raw_spin_lock+0x21/0x30 [22136.215291] ? __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [22136.217218] ? obd_mod_free+0x46/0x140 [ptlrpc] [22136.220169] ? mdc_commit_open+0x91/0xb0 [mdc] [22136.222359] ? ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [22136.225261] ? ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [22136.228592] ? after_reply+0xe2c/0x1780 [ptlrpc] [22136.231430] ? ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [22136.234233] ? __next_timer_interrupt+0x160/0x160 [22136.236385] ? ptlrpcd+0xa82/0xe40 [ptlrpc] [22136.238316] ? do_wait_intr+0xf0/0xf0 [22136.239955] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [22136.242475] ? kthread+0x1d1/0x200 [22136.245275] ? set_kthread_struct+0x70/0x70 [22136.249363] ? ret_from_fork+0x1f/0x30 [22152.061404] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [22152.069400] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22152.108278] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22152.123591] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22152.130185] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22152.132594] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [22152.152220] RSP: 0018:ffffbb5381c2bd58 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [22152.164296] RAX: 0000000000007eb0 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [22152.173713] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7021b4154 [22152.179793] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [22152.188312] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [22152.192454] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [22152.196863] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [22152.208422] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22152.215264] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [22152.220971] Call Trace: [22152.222309] [22152.223486] ? show_regs.cold.9+0x22/0x2f [22152.226084] ? watchdog_timer_fn.cold.6+0x62/0x10a [22152.228905] ? __hrtimer_run_queues+0x1c2/0x4d0 [22152.231044] ? watchdog+0x40/0x40 [22152.233467] ? hrtimer_interrupt+0x128/0x2c0 [22152.235422] ? smp_apic_timer_interrupt+0xa6/0x270 [22152.238848] ? apic_timer_interrupt+0xf/0x20 [22152.240163] [22152.240706] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22152.242873] ? __pv_queued_spin_lock_slowpath+0x451/0x6d0 [22152.246057] do_raw_spin_lock+0x177/0x190 [22152.247781] _raw_spin_lock+0x21/0x30 [22152.249193] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [22152.251796] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [22152.254937] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [22152.258462] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [22152.262364] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [22152.264374] ? woken_wake_function+0x30/0x30 [22152.266328] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [22152.268748] kthread+0x1d1/0x200 [22152.270159] ? set_kthread_struct+0x70/0x70 [22152.271840] ret_from_fork+0x1f/0x30 [22152.671212] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 19515s! [22152.676401] Showing busy workqueues and worker pools: [22152.683354] workqueue events: flags=0x0 [22152.686119] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [22152.690224] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [22152.701651] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [22152.705084] pending: cpu_latency_work [ptlrpc] [22152.709362] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [22152.719779] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [22152.725291] workqueue events_long: flags=0x0 [22152.729400] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [22152.733859] pending: lu_objects_flush [obdclass] [22152.740512] workqueue events_unbound: flags=0x2 [22152.743428] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [22152.749448] pending: flush_memcg_stats_dwork, idle_cull_fn [22152.755413] workqueue events_power_efficient: flags=0x80 [22152.765800] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22152.771369] pending: check_lifetime, do_cache_clean [22152.777079] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22152.785368] pending: neigh_periodic_work, neigh_periodic_work [22152.793175] workqueue mm_percpu_wq: flags=0x8 [22152.797364] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22152.801506] pending: vmstat_update, lru_add_drain_per_cpu [22152.804167] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22152.808070] pending: vmstat_update, lru_add_drain_per_cpu [22152.813208] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [22152.815976] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [22152.818260] workqueue xprtiod: flags=0xa [22152.832430] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [22152.838479] pending: xprt_autoclose, xs_error_handle [22152.842591] workqueue ipv6_addrconf: flags=0x40008 [22152.848419] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [22152.855409] pending: addrconf_verify_work [22152.858501] workqueue knbd0-recv: flags=0x1a [22152.863696] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [22152.867261] in-flight: 651:recv_work [22152.871309] workqueue ptlrpc_pinger: flags=0x2 [22152.876557] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [22152.879591] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [22152.887366] pool 8: cpus=0-3 flags=0x4 nice=0 hung=19515s workers=4 idle: 15979 31824 8 [22152.892416] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=22143s workers=2 idle: 654 [22160.055010] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [22160.063187] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [22160.064001] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22160.064807] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22160.064813] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22160.064816] RIP: 0010:__pv_queued_spin_lock_slowpath+0x328/0x6d0 [22160.064831] Code: d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 83 e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 05 18 d4 dc 02 01 e0 48 83 05 06 d4 dc 02 01 48 83 05 be d5 dc 02 01 8b 45 08 85 [22160.064835] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [22160.064839] RAX: 0000000000005e5b RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [22160.064841] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [22160.064843] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [22160.064845] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [22160.064847] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [22160.064851] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [22160.064854] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22160.064856] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [22160.064865] Call Trace: [22160.064872] [22160.064877] ? show_regs.cold.9+0x22/0x2f [22160.064886] ? watchdog_timer_fn.cold.6+0x62/0x10a [22160.064893] ? __hrtimer_run_queues+0x1c2/0x4d0 [22160.064898] ? watchdog+0x40/0x40 [22160.064902] ? hrtimer_interrupt+0x128/0x2c0 [22160.064907] ? smp_apic_timer_interrupt+0xa6/0x270 [22160.064914] ? apic_timer_interrupt+0xf/0x20 [22160.064919] [22160.064922] ? __pv_queued_spin_lock_slowpath+0x328/0x6d0 [22160.064929] do_raw_spin_lock+0x177/0x190 [22160.064933] _raw_spin_lock+0x21/0x30 [22160.064938] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [22160.070964] Modules linked in: lustre(O) osp(O) [22160.086004] ? do_raw_spin_unlock+0x75/0x190 [22160.086022] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [22160.086298] ? do_wait_intr+0xf0/0xf0 [22160.086307] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [22160.086612] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [22160.087087] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [22160.087374] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22160.087531] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [22160.087578] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [22160.087611] mdc_intent_lock+0x353/0x7e0 [mdc] [22160.087642] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22160.087836] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [22160.088144] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [22160.088178] lmv_intent_open+0x338/0x1150 [lmv] [22160.088208] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22160.088374] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22160.088524] lmv_intent_lock+0x1dd/0x4c0 [lmv] [22160.088556] ll_intent_lock+0xc5/0xaa0 [lustre] [22160.088657] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22160.088830] ll_lookup_it+0xe0e/0x2270 [lustre] [22160.088935] ? do_raw_spin_unlock+0x75/0x190 [22160.088942] ? _raw_spin_unlock+0x12/0x30 [22160.088950] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [22160.088980] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [22160.089000] ll_atomic_open+0x36a/0xd70 [lustre] [22160.089102] lookup_open+0x356/0xb40 [22160.089112] do_last+0x313/0xfc0 [22160.089117] ? nd_jump_root+0xe5/0x160 [22160.089122] ? path_init+0x437/0x520 [22160.089132] path_openat+0xf7/0x500 [22160.089138] ? do_raw_spin_unlock+0x75/0x190 [22160.089143] do_filp_open+0x99/0x140 [22160.089149] ? getname_flags+0x6e/0x330 [22160.089154] ? __check_object_size+0xff/0x256 [22160.089161] ? do_raw_spin_unlock+0x75/0x190 [22160.089165] ? _raw_spin_unlock+0x12/0x30 [22160.089170] do_sys_openat2+0x2b4/0x410 [22160.089175] do_sys_open+0x73/0xa0 [22160.089179] __x64_sys_openat+0x24/0x30 [22160.089183] do_syscall_64+0xc1/0x440 [22160.089190] entry_SYSCALL_64_after_hwframe+0x49/0xae [22160.089196] RIP: 0033:0x7f53425a5332 [22160.089252] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [22160.089256] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [22160.089262] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [22160.089265] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [22160.089267] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [22160.089270] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [22160.089272] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000 [22160.466697] ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22160.505866] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22160.510354] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22160.517239] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [22160.522374] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [22160.536189] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [22160.540973] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [22160.547211] RDX: 0000000000007bee RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [22160.556945] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [22160.561592] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [22160.565411] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [22160.568977] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [22160.573098] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22160.576632] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [22160.580186] Call Trace: [22160.581494] [22160.583722] ? show_regs.cold.9+0x22/0x2f [22160.586237] ? watchdog_timer_fn.cold.6+0x62/0x10a [22160.593099] ? __hrtimer_run_queues+0x1c2/0x4d0 [22160.595800] ? watchdog+0x40/0x40 [22160.597231] ? hrtimer_interrupt+0x128/0x2c0 [22160.599252] ? smp_apic_timer_interrupt+0xa6/0x270 [22160.604338] ? apic_timer_interrupt+0xf/0x20 [22160.608983] [22160.610497] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [22160.615057] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [22160.617274] do_raw_spin_lock+0x177/0x190 [22160.619948] _raw_spin_lock+0x21/0x30 [22160.622936] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [22160.629964] obd_mod_free+0x46/0x140 [ptlrpc] [22160.633290] mdc_commit_open+0x91/0xb0 [mdc] [22160.634975] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [22160.637694] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [22160.642928] after_reply+0xe2c/0x1780 [ptlrpc] [22160.647093] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [22160.651157] ? __next_timer_interrupt+0x160/0x160 [22160.653275] ptlrpcd+0xa82/0xe40 [ptlrpc] [22160.655671] ? do_wait_intr+0xf0/0xf0 [22160.657015] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [22160.659284] kthread+0x1d1/0x200 [22160.660588] ? set_kthread_struct+0x70/0x70 [22160.662181] ret_from_fork+0x1f/0x30 [22180.052072] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:4:36042] [22180.057163] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22180.058096] watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ldlm_bl_02:16828] [22180.078382] CPU: 0 PID: 36042 Comm: kworker/u8:4 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22180.082051] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) [22180.086761] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22180.086777] Workqueue: ptlrpc_pinger ptlrpc_pinger_main [ptlrpc] [22180.094205] ksocklnd(O) [22180.097776] [22180.100038] lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22180.100070] CPU: 3 PID: 16828 Comm: ldlm_bl_02 Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22180.100075] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22180.101217] RIP: 0010:native_safe_halt+0x2e/0x50 [22180.101233] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [22180.101237] RSP: 0018:ffffbb5380d7fdc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [22180.101243] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [22180.101245] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a702034154 [22180.101247] RBP: ffff98a702034140 R08: 0000000000000000 R09: 0000000000073b1c [22180.101250] R10: 0000000000000000 R11: 000000000000c161 R12: 0000000000040000 [22180.101252] R13: ffff98a7020b4140 R14: 000000000001e480 R15: ffff98a70201e480 [22180.101255] FS: 0000000000000000(0000) GS:ffff98a702000000(0000) knlGS:0000000000000000 [22180.101258] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22180.101261] CR2: 000056464d48872c CR3: 0000000084616002 CR4: 0000000000170ef0 [22180.101271] Call Trace: [22180.101276] [22180.101281] ? show_regs.cold.9+0x22/0x2f [22180.101292] ? watchdog_timer_fn.cold.6+0x62/0x10a [22180.101299] ? __hrtimer_run_queues+0x1c2/0x4d0 [22180.101303] ? watchdog+0x40/0x40 [22180.101308] ? hrtimer_interrupt+0x128/0x2c0 [22180.101312] ? smp_apic_timer_interrupt+0xa6/0x270 [22180.101319] ? apic_timer_interrupt+0xf/0x20 [22180.101325] [22180.101328] ? native_safe_halt+0x2e/0x50 [22180.101331] kvm_wait+0xcd/0xe0 [22180.101339] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [22180.101347] ? newidle_balance.isra.93+0x207/0x580 [22180.101354] do_raw_spin_lock+0x177/0x190 [22180.101358] _raw_spin_lock+0x21/0x30 [22180.101363] ptlrpc_pinger_main+0x1cd/0xf30 [ptlrpc] [22180.101647] process_one_work+0x2c8/0x700 [22180.101655] worker_thread+0x296/0x6e0 [22180.101660] ? rescuer_thread+0x570/0x570 [22180.101664] kthread+0x1d1/0x200 [22180.101668] ? set_kthread_struct+0x70/0x70 [22180.101672] ret_from_fork+0x1f/0x30 [22180.208412] RIP: 0010:native_safe_halt+0x2e/0x50 [22180.209514] Code: ed f1 01 01 e9 2a 00 00 00 48 83 05 3b ed f1 01 01 48 83 05 3b ed f1 01 01 0f 00 2d 44 44 43 00 48 83 05 34 ed f1 01 01 fb f4 <48> 83 05 32 ed f1 01 01 c3 48 83 05 09 ed f1 01 01 eb eb 66 66 2e [22180.220774] RSP: 0018:ffffbb5381c2bd48 EFLAGS: 00000216 ORIG_RAX: ffffffffffffff13 [22180.226541] RAX: 0000000000000001 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [22180.229656] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff98a7021b4154 [22180.232297] RBP: ffff98a7021b4140 R08: 0000000000000000 R09: 0000000000000000 [22180.234681] R10: 0000000003dcdaad R11: ffff98a6c7fe8000 R12: 0000000000100000 [22180.237397] R13: ffff98a702034140 R14: 000000000001e480 R15: ffff98a70219e480 [22180.238938] FS: 0000000000000000(0000) GS:ffff98a702180000(0000) knlGS:0000000000000000 [22180.240923] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22180.242068] CR2: 00007f09a0fe3000 CR3: 0000000084616006 CR4: 0000000000170ee0 [22180.243565] Call Trace: [22180.244294] [22180.244677] ? show_regs.cold.9+0x22/0x2f [22180.245661] ? watchdog_timer_fn.cold.6+0x62/0x10a [22180.247446] ? __hrtimer_run_queues+0x1c2/0x4d0 [22180.250542] ? watchdog+0x40/0x40 [22180.251618] ? hrtimer_interrupt+0x128/0x2c0 [22180.252589] ? smp_apic_timer_interrupt+0xa6/0x270 [22180.255330] ? apic_timer_interrupt+0xf/0x20 [22180.263061] [22180.263537] ? native_safe_halt+0x2e/0x50 [22180.264498] kvm_wait+0xcd/0xe0 [22180.265365] __pv_queued_spin_lock_slowpath+0x451/0x6d0 [22180.277356] do_raw_spin_lock+0x177/0x190 [22180.279038] _raw_spin_lock+0x21/0x30 [22180.280751] ptlrpc_request_alloc_internal+0x438/0x630 [ptlrpc] [22180.289851] ptlrpc_request_alloc+0x17/0x20 [ptlrpc] [22180.291936] ldlm_cli_cancel_req+0x12c/0xc20 [ptlrpc] [22180.297075] ldlm_cli_cancel_list+0x461/0x5f0 [ptlrpc] [22180.300095] ldlm_bl_thread_main+0xae9/0xcb0 [ptlrpc] [22180.301817] ? woken_wake_function+0x30/0x30 [22180.307650] ? ldlm_handle_bl_callback+0x520/0x520 [ptlrpc] [22180.310897] kthread+0x1d1/0x200 [22180.312067] ? set_kthread_struct+0x70/0x70 [22180.318663] ret_from_fork+0x1f/0x30 [22183.398026] BUG: workqueue lockup - pool cpus=0-3 flags=0x4 nice=0 stuck for 19546s! [22183.405392] Showing busy workqueues and worker pools: [22183.409532] workqueue events: flags=0x0 [22183.410969] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=4/256 refcnt=5 [22183.419607] pending: kfree_rcu_monitor, ldlm_pools_recalc_task [ptlrpc], nfs4_renew_state [nfsv4], push_to_pool [22183.428204] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [22183.431473] pending: cpu_latency_work [ptlrpc] [22183.434385] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=3/256 refcnt=4 [22183.437562] pending: vmstat_shepherd, osc_grant_work_handler [osc], sec_gc_main [ptlrpc] [22183.441563] workqueue events_long: flags=0x0 [22183.443279] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2 [22183.445809] pending: lu_objects_flush [obdclass] [22183.447942] workqueue events_unbound: flags=0x2 [22183.449453] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/512 refcnt=4 [22183.452355] pending: flush_memcg_stats_dwork, idle_cull_fn [22183.453674] workqueue events_power_efficient: flags=0x80 [22183.457179] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22183.460251] pending: check_lifetime, do_cache_clean [22183.461870] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22183.464884] pending: neigh_periodic_work, neigh_periodic_work [22183.467501] workqueue mm_percpu_wq: flags=0x8 [22183.469352] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22183.472116] pending: vmstat_update, lru_add_drain_per_cpu [22183.474350] pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=2/256 refcnt=3 [22183.477312] pending: vmstat_update, lru_add_drain_per_cpu [22183.479550] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256 refcnt=4 [22183.482364] pending: vmstat_update, lru_add_drain_per_cpu BAR(48) [22183.484906] workqueue xprtiod: flags=0xa [22183.489829] pwq 8: cpus=0-3 flags=0x4 nice=0 active=2/256 refcnt=4 [22183.494442] pending: xprt_autoclose, xs_error_handle [22183.497290] workqueue ipv6_addrconf: flags=0x40008 [22183.499160] pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/1 refcnt=2 [22183.501086] pending: addrconf_verify_work [22183.502303] workqueue knbd0-recv: flags=0x1a [22183.503609] pwq 9: cpus=0-3 flags=0x4 nice=-20 active=1/256 refcnt=3 [22183.505502] in-flight: 651:recv_work [22183.506687] workqueue ptlrpc_pinger: flags=0x2 [22183.508183] pwq 8: cpus=0-3 flags=0x4 nice=0 active=1/1 refcnt=3 [22183.514191] in-flight: 36042:ptlrpc_pinger_main [ptlrpc] [22183.516247] pool 8: cpus=0-3 flags=0x4 nice=0 hung=19546s workers=4 idle: 15979 31824 8 [22183.520129] pool 9: cpus=0-3 flags=0x4 nice=-20 hung=22173s workers=2 idle: 654 [22188.056630] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [writemany:37227] [22188.061145] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22188.064446] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ptlrpcd_rcv:2391] [22188.088857] CPU: 1 PID: 37227 Comm: writemany Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22188.088871] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22188.088874] RIP: 0010:__pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22188.088888] Code: 4c 03 2c c5 80 58 48 92 49 89 6d 00 b8 00 80 00 00 eb 20 41 0f b6 55 14 48 83 05 46 d4 dc 02 01 84 d2 0f 85 db 00 00 00 f3 90 <83> e8 01 0f 84 3b 02 00 00 8b 55 08 85 d2 75 0e 84 c0 74 d5 48 83 [22188.088892] RSP: 0018:ffffbb5381503500 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [22188.088898] RAX: 0000000000006487 RBX: ffff98a6d1bc8ae0 RCX: 0000000000000001 [22188.088900] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff98a7020b4154 [22188.088902] RBP: ffff98a7020b4140 R08: 0000000000000000 R09: 0000000000000000 [22188.088904] R10: 0000000003cff4ce R11: ffff98a6d007d000 R12: 0000000000080000 [22188.088907] R13: ffff98a702134140 R14: 000000000001e480 R15: ffff98a70209e480 [22188.088910] FS: 00007f5342c76740(0000) GS:ffff98a702080000(0000) knlGS:0000000000000000 [22188.088913] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22188.088916] CR2: 00005583af4bb0f8 CR3: 00000001201fa005 CR4: 0000000000170ee0 [22188.088925] Call Trace: [22188.088930] [22188.088936] ? show_regs.cold.9+0x22/0x2f [22188.088946] ? watchdog_timer_fn.cold.6+0x62/0x10a [22188.088969] ? __hrtimer_run_queues+0x1c2/0x4d0 [22188.088973] ? watchdog+0x40/0x40 [22188.088977] ? hrtimer_interrupt+0x128/0x2c0 [22188.088982] ? smp_apic_timer_interrupt+0xa6/0x270 [22188.104963] Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ata_piix serio_raw [22188.109874] ? apic_timer_interrupt+0xf/0x20 [22188.109896] [22188.109901] ? __pv_queued_spin_lock_slowpath+0x30c/0x6d0 [22188.109912] do_raw_spin_lock+0x177/0x190 [22188.109918] _raw_spin_lock+0x21/0x30 [22188.109923] ptlrpc_check_set+0x11c5/0x30b0 [ptlrpc] [22188.116262] libata [22188.118718] ? do_raw_spin_unlock+0x75/0x190 [22188.127331] dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [22188.127355] CPU: 2 PID: 2391 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O L -------- - - 4.18.0rh8.10-debug #2 [22188.127363] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [22188.127367] RIP: 0010:__pv_queued_spin_lock_slowpath+0x174/0x6d0 [22188.127384] Code: 0c 00 00 00 00 41 be 01 00 00 00 41 bf 00 01 00 00 65 48 ff 05 ed 8a 47 6f c6 45 14 00 ba 00 80 00 00 c6 43 01 01 eb 13 f3 90 <48> 83 05 5c d6 dc 02 01 83 ea 01 0f 84 ef 03 00 00 0f b6 03 84 c0 [22188.127388] RSP: 0018:ffffbb5381ff7bc8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 [22188.127393] RAX: 0000000000000003 RBX: ffff98a6d1bc8ae0 RCX: ffff98a70211e480 [22188.127395] RDX: 0000000000002beb RSI: 0000000000000003 RDI: ffff98a6d1bc8ae0 [22188.127401] RBP: ffff98a702134140 R08: 0000000000000008 R09: 0000000000000088 [22188.127403] R10: ffffffff92608b80 R11: ffffbb5381ff78f0 R12: 00000000000c0000 [22188.127405] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000100 [22188.127409] FS: 0000000000000000(0000) GS:ffff98a702100000(0000) knlGS:0000000000000000 [22188.127412] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22188.127414] CR2: 00007fc42797d670 CR3: 0000000084616001 CR4: 0000000000170ee0 [22188.127422] Call Trace: [22188.127430] [22188.127436] ? show_regs.cold.9+0x22/0x2f [22188.127445] ? watchdog_timer_fn.cold.6+0x62/0x10a [22188.127452] ? __hrtimer_run_queues+0x1c2/0x4d0 [22188.127459] ? watchdog+0x40/0x40 [22188.127464] ? hrtimer_interrupt+0x128/0x2c0 [22188.127469] ? smp_apic_timer_interrupt+0xa6/0x270 [22188.127477] ? apic_timer_interrupt+0xf/0x20 [22188.127482] [22188.127488] ? __pv_queued_spin_lock_slowpath+0x174/0x6d0 [22188.127493] ? __pv_queued_spin_lock_slowpath+0x5e7/0x6d0 [22188.127498] do_raw_spin_lock+0x177/0x190 [22188.127502] _raw_spin_lock+0x21/0x30 [22188.127509] __ptlrpc_req_put+0x6af/0xa50 [ptlrpc] [22188.127836] obd_mod_free+0x46/0x140 [ptlrpc] [22188.132437] ptlrpc_set_wait+0x38e/0xa50 [ptlrpc] [22188.132779] ? do_wait_intr+0xf0/0xf0 [22188.132790] ptlrpc_queue_wait+0xb1/0x2d0 [ptlrpc] [22188.133099] ldlm_cli_enqueue+0x43f/0xe40 [ptlrpc] [22188.133458] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [22188.133863] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22188.134047] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [22188.134090] mdc_enqueue_base+0x2c5/0x1dd0 [mdc] [22188.134116] mdc_intent_lock+0x353/0x7e0 [mdc] [22188.134143] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22188.134579] ? ldlm_cli_enqueue_local+0xbd0/0xbd0 [ptlrpc] [22188.135026] ? mdc_changelog_cdev_finish+0x2c0/0x2c0 [mdc] [22188.138572] mdc_commit_open+0x91/0xb0 [mdc] [22188.138619] ptlrpc_free_request+0x4b/0xc0 [ptlrpc] [22188.139004] ptlrpc_free_committed+0x15c/0xd70 [ptlrpc] [22188.139321] after_reply+0xe2c/0x1780 [ptlrpc] [22188.139523] ptlrpc_check_set+0x2575/0x30b0 [ptlrpc] [22188.139788] ? __next_timer_interrupt+0x160/0x160 [22188.139800] ptlrpcd+0xa82/0xe40 [ptlrpc] [22188.140026] ? do_wait_intr+0xf0/0xf0 [22188.140035] ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] [22188.140572] kthread+0x1d1/0x200 [22188.140579] ? set_kthread_struct+0x70/0x70 [22188.140582] ret_from_fork+0x1f/0x30 [22188.343220] lmv_intent_open+0x338/0x1150 [lmv] [22188.344647] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22188.346465] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22188.348556] lmv_intent_lock+0x1dd/0x4c0 [lmv] [22188.350238] ll_intent_lock+0xc5/0xaa0 [lustre] [22188.352140] ? ll_lock_cancel_bits+0xe10/0xe10 [lustre] [22188.354292] ll_lookup_it+0xe0e/0x2270 [lustre] [22188.356408] ? do_raw_spin_unlock+0x75/0x190 [22188.358005] ? _raw_spin_unlock+0x12/0x30 [22188.359512] ? cfs_trace_unlock_tcd+0x28/0xa0 [libcfs] [22188.361027] ? libcfs_debug_msg+0xcf4/0x1200 [libcfs] [22188.362556] ll_atomic_open+0x36a/0xd70 [lustre] [22188.364371] lookup_open+0x356/0xb40 [22188.365483] do_last+0x313/0xfc0 [22188.366422] ? nd_jump_root+0xe5/0x160 [22188.367877] ? path_init+0x437/0x520 [22188.369132] path_openat+0xf7/0x500 [22188.370255] ? do_raw_spin_unlock+0x75/0x190 [22188.371403] do_filp_open+0x99/0x140 [22188.372371] ? getname_flags+0x6e/0x330 [22188.373584] ? __check_object_size+0xff/0x256 [22188.374924] ? do_raw_spin_unlock+0x75/0x190 [22188.376713] ? _raw_spin_unlock+0x12/0x30 [22188.378199] do_sys_openat2+0x2b4/0x410 [22188.379598] do_sys_open+0x73/0xa0 [22188.380801] __x64_sys_openat+0x24/0x30 [22188.382228] do_syscall_64+0xc1/0x440 [22188.383637] entry_SYSCALL_64_after_hwframe+0x49/0xae [22188.385348] RIP: 0033:0x7f53425a5332 [22188.386788] Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 35 84 2a 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 [22188.392605] RSP: 002b:00007ffff8ec3140 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [22188.395231] RAX: ffffffffffffffda RBX: 000000000000a950 RCX: 00007f53425a5332 [22188.397271] RDX: 0000000000000042 RSI: 00007ffff8ec3210 RDI: 00000000ffffff9c [22188.399787] RBP: 00007ffff8ec3610 R08: 0000000000000000 R09: 0000000000000000 [22188.401999] R10: 00000000000001b6 R11: 0000000000000246 R12: 00007ffff8ec3210 [22188.404274] R13: 00000000ffffffff R14: 0000000000000010 R15: 0000000000000000